mARTin-B78
f34266c0d2
Pooled over three hand-corrected novels (~54k spoken words): agreed 82% right, disputed 48%, Unknown ~0%, unverified ~76%. The old heuristic bands claimed 16% for lines that were 62% right and tag/role signals did not discriminate in span casts; classic casts keep the old model. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
151 lines
8.6 KiB
Markdown
151 lines
8.6 KiB
Markdown
# SOP — Audiobook speaker attribution (span mode)
|
||
|
||
Status: **implemented, default on.** Switch off with `localStorage['ab-span-mode'] = '0'`
|
||
(the classic whole-passage engine then runs). A book with no recognisable quotation
|
||
style automatically uses the classic engine.
|
||
|
||
## Why it changed
|
||
|
||
The classic engine sends ~3,000-character passages to an LLM and asks it to re-emit
|
||
the whole passage as JSON segments (`/api/attribute-dialogue`). That has three costs:
|
||
|
||
* **Text corruption.** The model fuses, duplicates or drops text; a stack of
|
||
post-processing passes (dedupe, recover-dropped-text, prefix-dup cut, orphan-quote
|
||
repair) exists only to undo it. Measured on a hand-corrected book: narration/dialogue
|
||
typing agreed on 93.6 % of words.
|
||
* **Hours per book.** Output tokens ≈ input tokens (a 4 h run for one novel).
|
||
* **Unmeasurable.** Exact-text matching counts any boundary difference as an error.
|
||
|
||
## Design
|
||
|
||
```
|
||
book text ──► segmentByQuotes() deterministic (static/js/audiobook-spans.js)
|
||
narration / dialogue spans with exact source offsets
|
||
──► buildWindows() ~2.4–3.6k chars per LLM call, cut at paragraphs
|
||
──► POST /api/attribute-spans LLM answers ONLY {id, speaker, emotion} per dialogue span
|
||
──► assemble segments text is taken from the spans, never from the LLM
|
||
──► existing finalisation phantom drop, role→name aliases, name consolidation,
|
||
confidence, autosave, Rehearser save, auto-repair
|
||
```
|
||
|
||
The LLM can no longer change, drop, duplicate or fuse a single word; it can only choose
|
||
a speaker, so its output is a few tokens per dialogue line.
|
||
|
||
### Segmenter rules (`audiobook-spans.js`)
|
||
|
||
* One quote style per book, chosen by counting well-formed pairs (`»…«` German inward,
|
||
`„…“`, `“…”`, `«…»` French). No style with ≥ 5 pairs → classic engine.
|
||
* A closing mark with no opener (opener lost in PDF extraction): the speech starts at the
|
||
earliest run of speech-like sentences directly before it (1st/2nd-person pronouns, `?`,
|
||
short `!`).
|
||
* An opener whose closer is lost: the speech ends after the leading run of speech-like
|
||
sentences.
|
||
* A speech continues across paragraph breaks while a closing mark still follows before the
|
||
next opener (some typesetting re-opens every paragraph, some only closes the last).
|
||
* A quote that is a term/title inside a sentence (no sentence-final punctuation, lower-case
|
||
continuation) is narration.
|
||
* Adjacent narration merges; each quote stays its own dialogue span.
|
||
|
||
### `POST /api/attribute-spans`
|
||
|
||
Request: `{spans:[{id, kind, text}], known_characters, recent, following, language, llm_url,
|
||
model, timeout_seconds, want_reasoning}`. Ids are `N<n>` / `D<n>`; `kind` must match the id
|
||
prefix; ≤ 400 spans, ≤ 20 000 chars per span, ≤ 60 000 total, at least one dialogue span
|
||
(HTTP 400 otherwise).
|
||
Response: `{speakers:{id:{speaker, emotion}}, missing:[ids], characters:[names]}`, or
|
||
`{failed:true, error, missing:[…]}` (HTTP 200) when the LLM is unavailable — the client then
|
||
falls back to speech-tag guessing. The text is never returned.
|
||
|
||
Behaviour: thinking is disabled by default (`chat_template_kwargs.enable_thinking=false`,
|
||
retried without the key on HTTP 400/422). Measured: reasoning mode was ~25× slower with no
|
||
accuracy gain on the same windows. An answer that parses to nothing is retried once with a
|
||
larger token budget; lines the model skipped are retried once on their own. Calls are
|
||
serialised with every other attribution call.
|
||
|
||
Naming rule: the model answers with the label the text uses ("Der Fremde", "Baronin",
|
||
"Oberst") when a person is only referred to by role in the scene, instead of guessing another
|
||
known character's name. Role → name resolution then happens book-wide in the existing
|
||
`_audiobookResolveRoleAliases` / `_audiobookConsolidateSpeakerAliases` passes.
|
||
|
||
## Measuring
|
||
|
||
`scripts/score_attribution.py <manual.json> <auto.json> [--characters <manual chars>]` aligns
|
||
the two casts word by word (insensitive to chunking) and reports coverage, narration/dialogue
|
||
type agreement, dialogue speaker accuracy (strict and alias-tolerant) and the largest
|
||
confusions. **Read it with care:** the human references contain label noise (inline narration
|
||
folded into the neighbouring dialogue, inconsistent handling of very long monologues,
|
||
occasional wrong speakers), so differences of ~1–2 points are noise.
|
||
|
||
Book 1 (*Der Sturm*), same reference, same scorer:
|
||
|
||
| | classic (Run 3) | span mode |
|
||
|---|---|---|
|
||
| type agreement | 93.6 % | 97.4 % |
|
||
| dialogue words, right speaker (tolerant) | 60.6 % | 75.4 % |
|
||
| overall every word right (tolerant) | 87.0 % | 92.2 % |
|
||
| wall time | ≈ 4 h | ≈ 11 min |
|
||
| Unknown-speaker words | 16 | 114 |
|
||
|
||
## Verify and recast (span casts)
|
||
|
||
Workflow step 3 (Recast unknown), step 4 (2nd Quality Run) and auto-repair all enter through
|
||
`audiobookRecastUnknown`; for a span cast they are routed to `audiobookVerifySpans`
|
||
(`static/js/audiobook-cast-spans.js`) instead of the classic passes, which would have the LLM re-emit
|
||
text. Segments carry their `spanId`, so a cast maps back to the deterministic spans; a cast that does
|
||
not (classic cast, or a different text) is declined and the classic path runs.
|
||
|
||
* **2nd Quality Run (Verify) is advisory.** One extra no-thinking pass with different window boundaries
|
||
(5000/7000 chars vs 2400/3600), about as long as the first cast (~10 min per novel). It **never
|
||
changes a speaker**. Lines both passes agree on get `verified: 'agreed'`; where they disagree the
|
||
first answer is kept, the line gets `verified: 'disputed'` and the other pass's answer is stored as
|
||
`altSpeaker` (shown in the confidence explanation); a line the first pass left Unknown stays Unknown
|
||
(`unresolved`, with the other pass's suggestion). Nothing is
|
||
applied until the pass finished, so a stopped or failed run leaves the cast unchanged.
|
||
* **Recast unknown** (`unknownOnly`): the grammar pre-pass first, then one reasoning question per
|
||
cluster of remaining Unknown lines (fuller context, `attribute_only` so the rest of the window is
|
||
context). A line the model still cannot place stays Unknown — never invented. Auto-repair after a
|
||
span cast is off: Unknown lines stay visible for review.
|
||
|
||
### Confidence for span casts
|
||
|
||
The displayed confidence of a span-cast dialogue line comes from agreement alone
|
||
(`_abSpanConfidence` in `audiobook.js`), calibrated on three hand-corrected novels (~54 000 spoken
|
||
words, alias-tolerant): **agreed 82 %** (2 013 lines), **disputed 48 %** (472 lines), **Unknown ~0 %**,
|
||
first pass not yet verified **~76 %**. The old heuristic model (speech-tag evidence, role labels, quote
|
||
marks) is kept only for classic casts: in span casts it did not discriminate (lines whose speech tag names
|
||
the speaker were 83 % right versus 84 % without) and its calibrated bands were far too pessimistic (a band
|
||
displayed as 16 % was 62 % right). Unverified lines sit just under the 80 % review threshold on purpose:
|
||
running Verify shrinks the review queue to the disputed and Unknown lines.
|
||
|
||
### What was measured, and what was rejected
|
||
|
||
Book 1, alias-tolerant, real in-app runs on the KONTROLLE document:
|
||
|
||
| | words | first pass | after |
|
||
|---|---|---|---|
|
||
| lines both passes agreed on | 18 442 | 82.0 % | 82.0 % (unchanged) |
|
||
| lines the passes disputed, overruled by a reasoning question restricted to the two candidates | 3 203 | 66.5 % | **61.4 %** |
|
||
| Unknown lines "resolved" by the second pass or reasoning | 128 | 0 % | 3 % |
|
||
|
||
An earlier offline experiment suggested adjudication helped (56 % → 72 % on a sample of disagreeing
|
||
lines) but compared against a biased baseline; on the real pipeline it made the lines it changed worse,
|
||
so the adjudication step (and the `candidates` request field) was removed. The only reliable signal is
|
||
agreement itself: agreed lines are right 82 % of the time, contested first answers 67 %.
|
||
Sampling matters: windows of ~1 200 characters were clearly worse (−4 points) and ~5 000 about equal to
|
||
the default, so the default stays.
|
||
|
||
Request additions: `attribute_only` `[ids]` — the other spans are context; unrequested lines are never
|
||
reported missing.
|
||
|
||
## Security / privacy
|
||
|
||
Book text is sent to the configured LLM exactly as before (same endpoint, same settings);
|
||
the server never returns text. Inputs are validated at the boundary (see limits above). No
|
||
new credentials, storage or network destinations.
|
||
|
||
## Rollback
|
||
|
||
`localStorage['ab-span-mode'] = '0'` restores the classic engine per browser; removing the
|
||
dispatch at the top of `audiobookCast()` (`static/js/audiobook-pipeline.js`) removes it for
|
||
everyone. Casts made by either engine are the same shape and remain readable by the other.
|