tts-voice-creator-clone-and-design-2/architecture/audiobook-attribution.md

SOP — Audiobook speaker attribution (span mode)

Status: implemented, default on. Switch off with localStorage['ab-span-mode'] = '0' (the classic whole-passage engine then runs). A book with no recognisable quotation style automatically uses the classic engine.

Why it changed

The classic engine sends ~3,000-character passages to an LLM and asks it to re-emit the whole passage as JSON segments (/api/attribute-dialogue). That has three costs:

• Text corruption. The model fuses, duplicates or drops text; a stack of post-processing passes (dedupe, recover-dropped-text, prefix-dup cut, orphan-quote repair) exists only to undo it. Measured on a hand-corrected book: narration/dialogue typing agreed on 93.6 % of words.
• Hours per book. Output tokens ≈ input tokens (a 4 h run for one novel).
• Unmeasurable. Exact-text matching counts any boundary difference as an error.

Design

book text ──► segmentByQuotes()      deterministic (static/js/audiobook-spans.js)
              narration / dialogue spans with exact source offsets
          ──► buildWindows()         ~2.4–3.6k chars per LLM call, cut at paragraphs
          ──► POST /api/attribute-spans   LLM answers ONLY {id, speaker, emotion} per dialogue span
          ──► assemble segments      text is taken from the spans, never from the LLM
          ──► existing finalisation  phantom drop, role→name aliases, name consolidation,
                                     confidence, autosave, Rehearser save, auto-repair

The LLM can no longer change, drop, duplicate or fuse a single word; it can only choose a speaker, so its output is a few tokens per dialogue line.

Segmenter rules (audiobook-spans.js)

• One quote style per book, chosen by counting well-formed pairs (»…« German inward, „…“, “…”, «…» French). No style with ≥ 5 pairs → classic engine.
• A closing mark with no opener (opener lost in PDF extraction): the speech starts at the earliest run of speech-like sentences directly before it (1st/2nd-person pronouns, ?, short !).
• An opener whose closer is lost: the speech ends after the leading run of speech-like sentences.
• A speech continues across paragraph breaks while a closing mark still follows before the next opener (some typesetting re-opens every paragraph, some only closes the last).
• A quote that is a term/title inside a sentence (no sentence-final punctuation, lower-case continuation) is narration.
• Adjacent narration merges; each quote stays its own dialogue span.

POST /api/attribute-spans

Request: {spans:[{id, kind, text}], known_characters, recent, following, language, llm_url, model, timeout_seconds, want_reasoning}. Ids are N<n> / D<n>; kind must match the id prefix; ≤ 400 spans, ≤ 20 000 chars per span, ≤ 60 000 total, at least one dialogue span (HTTP 400 otherwise). Response: {speakers:{id:{speaker, emotion}}, missing:[ids], characters:[names]}, or {failed:true, error, missing:[…]} (HTTP 200) when the LLM is unavailable — the client then falls back to speech-tag guessing. The text is never returned.

Behaviour: thinking is disabled by default (chat_template_kwargs.enable_thinking=false, retried without the key on HTTP 400/422). Measured: reasoning mode was ~25× slower with no accuracy gain on the same windows. An answer that parses to nothing is retried once with a larger token budget; lines the model skipped are retried once on their own. Calls are serialised with every other attribution call.

A model warm-up request (/api/attribute-dialogue, "Wake up.") runs first in both the cast and the verify, as in the classic engine, so a cold-booting model server does not time out the first passage. The custom attribution prompt (Settings) is part of the classic engine only; the span prompt is fixed in core/attribution_spans.py.

Naming rule: the model answers with the label the text uses ("Der Fremde", "Baronin", "Oberst") when a person is only referred to by role in the scene, instead of guessing another known character's name. Role → name resolution then happens book-wide in the existing _audiobookResolveRoleAliases / _audiobookConsolidateSpeakerAliases passes.

Measuring

scripts/score_attribution.py <manual.json> <auto.json> [--characters <manual chars>] aligns the two casts word by word (insensitive to chunking) and reports coverage, narration/dialogue type agreement, dialogue speaker accuracy (strict and alias-tolerant) and the largest confusions. Read it with care: the human references contain label noise (inline narration folded into the neighbouring dialogue, inconsistent handling of very long monologues, occasional wrong speakers), so differences of ~1–2 points are noise.

Book 1 (Der Sturm), same reference, same scorer:

	classic (Run 3)	span mode
type agreement	93.6 %	97.4 %
dialogue words, right speaker (tolerant)	60.6 %	75.4 %
overall every word right (tolerant)	87.0 %	92.2 %
wall time	≈ 4 h	≈ 11 min
Unknown-speaker words	16	114

Verify and recast (span casts)

Workflow step 3 (Recast unknown), step 4 (2nd Quality Run) and auto-repair all enter through audiobookRecastUnknown; for a span cast they are routed to audiobookVerifySpans (static/js/audiobook-cast-spans.js) instead of the classic passes, which would have the LLM re-emit text. Segments carry their spanId, so a cast maps back to the deterministic spans; a cast that does not (classic cast, or a different text) is declined and the classic path runs.

• 2nd Quality Run (Verify) is advisory. One extra no-thinking pass with different window boundaries (5000/7000 chars vs 2400/3600), about as long as the first cast (~10 min per novel). It never changes a speaker. Lines both passes agree on get verified: 'agreed'; where they disagree the first answer is kept, the line gets verified: 'disputed' and the other pass's answer is stored as altSpeaker (shown in the confidence explanation); a line the first pass left Unknown stays Unknown (unresolved, with the other pass's suggestion). Nothing is applied until the pass finished, so a stopped or failed run leaves the cast unchanged.
• Recast unknown (unknownOnly): the grammar pre-pass first, then one reasoning question per cluster of remaining Unknown lines (fuller context, attribute_only so the rest of the window is context). A line the model still cannot place stays Unknown — never invented. Auto-repair after a span cast is off: Unknown lines stay visible for review.

Confidence for span casts

The displayed confidence of a span-cast dialogue line comes from agreement alone (_abSpanConfidence in audiobook.js), calibrated on three hand-corrected novels (~54 000 spoken words, alias-tolerant): agreed 82 % (2 013 lines), disputed 48 % (472 lines), Unknown ~0 %, first pass not yet verified ~76 %. The old heuristic model (speech-tag evidence, role labels, quote marks) is kept only for classic casts: in span casts it did not discriminate (lines whose speech tag names the speaker were 83 % right versus 84 % without) and its calibrated bands were far too pessimistic (a band displayed as 16 % was 62 % right). Unverified lines sit just under the 80 % review threshold on purpose: running Verify shrinks the review queue to the disputed and Unknown lines.

What was measured, and what was rejected

Book 1, alias-tolerant, real in-app runs on the KONTROLLE document:

	words	first pass	after
lines both passes agreed on	18 442	82.0 %	82.0 % (unchanged)
lines the passes disputed, overruled by a reasoning question restricted to the two candidates	3 203	66.5 %	61.4 %
Unknown lines "resolved" by the second pass or reasoning	128	0 %	3 %

An earlier offline experiment suggested adjudication helped (56 % → 72 % on a sample of disagreeing lines) but compared against a biased baseline; on the real pipeline it made the lines it changed worse, so the adjudication step (and the candidates request field) was removed. The only reliable signal is agreement itself: agreed lines are right 82 % of the time, contested first answers 67 %. Sampling matters: windows of ~1 200 characters were clearly worse (−4 points) and ~5 000 about equal to the default, so the default stays. A scene-cast hint (which known characters are named in the passage or just before it) gave +0.3 points — noise — and was not adopted.

Request additions: attribute_only [ids] — the other spans are context; unrequested lines are never reported missing.

Emotions

/api/attribute-spans returns {speaker, emotion} per dialogue id (1-2 words in the book's language; narration ids never get one). Path to audio: segment emotion -> audiobookBuildScript / rehState.lines[].emotion -> audiobookLineVoice -> _buildInstruct (English clause; German words via _rehEmotionEnglishTag) -> fetchTtsPreviewBlob -> payload["instruct"] (core/tts_helpers.py). The line audio cache key includes the instruct, so an edited emotion never serves stale audio. Fish-Speech uses _rehInlineTone ([tag]) instead.

Limits: engine support differs (routes/tts.py _backend_capabilities: voice_clone style_aware false, instruct may be ignored; voice_design strong). Span-mode LLM failure leaves emotion empty. A custom attribution prompt is ignored in span mode. The merge pass keeps one emotion per block. Tests: tests/audiobook-emotion.spec.js, tests/test_tts_instruct_payload.py. No per-voice UI hint yet: the voice is only chosen later in the Rehearser, which already shows a tone warning for non-style-aware backends.

Security / privacy

Book text is sent to the configured LLM exactly as before (same endpoint, same settings); the server never returns text. Inputs are validated at the boundary (see limits above). No new credentials, storage or network destinations.

Rollback

localStorage['ab-span-mode'] = '0' restores the classic engine per browser; removing the dispatch at the top of audiobookCast() (static/js/audiobook-pipeline.js) removes it for everyone. Casts made by either engine are the same shape and remain readable by the other.