diff --git a/CHANGELOG.md b/CHANGELOG.md index f274af2..4247423 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,21 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi --- +## [1.34.0] — 2026-09-19 + +### Added +- **Verify / recast for span-mode casts** (`audiobookVerifySpans`): the app's Recast unknown, 2nd Quality Run and auto-repair entry points now run span-natively for a span cast instead of the classic passes that re-emit text. **Verify is advisory**: one extra differently-windowed pass (~10 min per novel) that never changes a speaker — agreeing lines are marked `agreed`, disagreeing lines keep the first answer, are marked `disputed` and record the other pass's answer as `altSpeaker`; the confidence score reflects both. **Recast unknown** re-asks only the Unknown lines with reasoning and fuller context (`attribute_only` on `/api/attribute-spans`) and leaves any line the model cannot place Unknown. Segments now carry a `spanId`. + +### Changed +- Auto-repair no longer runs after a span-mode cast: Unknown lines stay visible for review instead of being guessed at. + +### Measured — and rejected +- On the real pipeline (Book 1, in-app), letting a reasoning question restricted to the two candidates overrule disputed lines made them **worse** (66.5 % → 61.4 % on 3 203 words), and the second pass "resolving" Unknown lines was ~3 % right, so no adjudication or silent filling exists. Lines both passes agreed on were 82 % right versus ~67 % for disputed first answers — that agreement is the signal used. An earlier offline experiment had suggested the opposite; it used a biased baseline (see `architecture/audiobook-attribution.md`). +- Reasoning mode for the first pass (25× slower, no gain) and smaller/larger windows (−4 / ±0 points) were also tested and not adopted. +- Final in-app run on Book 1: the advisory verify took 526 s, changed no speaker, and separated the lines well — 757 agreed lines were 83.2 % right, 118 disputed lines 48.5 % right (12.5 % of the dialogue holding ~28 % of the errors), which is what makes the flags useful for review. + +--- + ## [1.33.0] — 2026-09-19 ### Fixed diff --git a/Manual.md b/Manual.md index 723c502..8e49723 100644 --- a/Manual.md +++ b/Manual.md @@ -29,7 +29,15 @@ the Rehearser and MP3 export. hours). 3. Progress shows per passage; **Stop Casting** keeps what is done and **Continue casting** resumes. -4. Review the cast as before (Recast unknown, Verify, Define characters, …). +4. Review the cast as before. **Recast unknown** re-reads only the lines still marked + Unknown, with more context; a line the model cannot place stays Unknown rather than + getting a guessed voice. **Verify** (the second quality run) reads every spoken line a + second time with different passage boundaries — about as long as the first cast — and + **never changes a speaker**: lines both readings agree on are marked confirmed (about + 4 in 5 of those were right in testing), and lines where they disagree keep the first + reading, are flagged in the review list and show what the other reading said. Use the + flags to decide where to spend your review time. Neither step can change the book's + text, and a stopped or failed run leaves the cast as it was. **Expected result:** every word of the book appears exactly once, quoted speech is typed as dialogue and everything else as narration. When the text only refers to a diff --git a/PROJECT_MAP.md b/PROJECT_MAP.md index 2329061..5fe8683 100644 --- a/PROJECT_MAP.md +++ b/PROJECT_MAP.md @@ -35,7 +35,8 @@ Two self-hosted Docker instances, both port 7890: | `core/` | domain logic — config, database, audio, voice, routing, magazine | | `routes/` | HTTP boundary, one module per feature area | | `static/js/` | frontend, one module per feature area (no build step for most) | -| `tests/` | `test_*.py` (unittest) and `*.spec.js` (Playwright) | +| `tests/` | `test_*.py` (unittest, run with `python3 -m unittest tests.test_x`), `*.test.mjs` (`npm run test:unit`, Node) and `*.spec.js` (Playwright — must mock every write, the app on :7890 is the real one) | +| `scripts/` | `score_attribution.py` (word-aligned cast scoring), release/minify/frontend-gate tooling | | `architecture/` | SOPs for subsystems whose design is not obvious from code | | `workflows/` | ComfyUI graphs for character portrait generation | @@ -45,7 +46,8 @@ Two self-hosted Docker instances, both port 7890: |---|---|---|---| | Voice library / clone / design | `routes/library.py`, `core/voice.py` | `library.js` | mature | | Read Aloud (books, PDFs) | `routes/reader.py` | `reader.js` | mature | -| Audiobook casting | `routes/conversation.py` | `audiobook*.js` | mature | +| Audiobook casting (span mode: deterministic quote spans, LLM names speakers only; verify/recast) | `routes/conversation.py`, `core/attribution_spans.py` | `audiobook*.js`, `audiobook-spans.js`, `audiobook-cast-spans.js` | **new 2026-09-19** — see [architecture/audiobook-attribution.md](architecture/audiobook-attribution.md) | +| Characters (avatar/concept identity, appearance forms) | `routes/characters.py`, `routes/conversation.py` | `library-characters.js`, `character-sheets.js` | **reworked 2026-09-19** — see [architecture/character-visual-identity.md](architecture/character-visual-identity.md) | | Rehearser | `routes/rehearsals_db.py` | `rehearser-*.js` | mature | | Studio (phase wizard) | — | `studio.js` | mature | | TTS engines / routing | `routes/tts.py`, `core/routing.py` | `engines.js` | mature | @@ -83,6 +85,16 @@ which is the writable volume. **unresolved** — the compose files currently point at ghcr. ## Known limitations +- Audiobook attribution (span mode) is measured, not perfect: on three hand-corrected novels about + 90-92 % of all words end up with the right voice and ~75-80 % of spoken words with the right + speaker. The references themselves contain label noise (1-2 points is not meaningful). Open, real + problems: a person known only by a role that means different people in different scenes ("der + Fremde") is not resolved to a character; look-alike characters in one scene are the usual mix-up; + very long monologues are inconsistent in the references. +- A speech is only found where the text marks it with a recognised quotation style (»…«, „…“, + “…”, «…»); a book with none of them uses the older, slower whole-passage engine. +- Character image generation needs ComfyUI/an image provider; the deterministic prompts, forms and + review logic are covered by tests without one, but a real pair render is not part of the suite. - `gitea` backup remote is 157 commits behind `origin`; `main` tracks `gitea`. - Magazine sidebar exclusion is a narrow-column geometry heuristic (`mark_sidebars`), not certain — review always shows every sidebar block. diff --git a/TODO.md b/TODO.md index ae2ce1c..855aa8b 100644 --- a/TODO.md +++ b/TODO.md @@ -329,7 +329,27 @@ planning pass — this is a multi-week, cross-repo effort. The authenticated panel reports browser cache counters, cgroup memory current/max, server RSS, configured/temp storage, persisted jobs, and sanitized operation durations; host-wide Docker metrics remain unavailable by design. -## P1 — Speaker attribution: measured state and next steps (paused 2026-09-02) +## P1 — Speaker attribution: measured state and next steps + +**Status 2026-09-19 — reworked, see `architecture/audiobook-attribution.md`.** Span mode (deterministic +quote spans, the LLM only names speakers) replaced the segmentation-heavy engine: Book 1 every-word-right +92.3 % (was 87.0 %), narration/dialogue agreement 97.4 % (was 93.6 %), ~10 min instead of ~4 h; Books 2/3 +(never tuned on) 90.8 % / 89.9 %. The word-aligned scorer is `scripts/score_attribution.py`; the references +have label noise, so 1–2 points is not meaningful. Everything below the line is the historical record of +the classic engine's Runs 1–3. + +Still open (each needs a real decision or new evidence, not more prompt tuning): + +- [ ] A person known only by a role that means different people in different scenes ("der Fremde") is not + resolved to a character — needs scene-level identity tracking; the label is kept as its own cast entry + until the book itself ties it to a name. +- [ ] Look-alike main characters in one scene (Marcian/Zerwas/Lancorian) remain the biggest real error + class (~1/5 of spoken words). Tried and rejected: reasoning mode (25× slower, no gain), larger/smaller + windows, tag-based overrides, reasoning adjudication of disputes (made disputed lines worse). Untried: + an explicit scene-cast constraint (characters present in the scene) built from the text. + +--- + ### Ordered follow-up queue diff --git a/VERSION b/VERSION index f0fed91..2404d95 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.33.0 \ No newline at end of file +1.34.0 \ No newline at end of file diff --git a/architecture/audiobook-attribution.md b/architecture/audiobook-attribution.md index 7481906..72fc880 100644 --- a/architecture/audiobook-attribution.md +++ b/architecture/audiobook-attribution.md @@ -86,6 +86,46 @@ Book 1 (*Der Sturm*), same reference, same scorer: | wall time | ≈ 4 h | ≈ 11 min | | Unknown-speaker words | 16 | 114 | +## Verify and recast (span casts) + +Workflow step 3 (Recast unknown), step 4 (2nd Quality Run) and auto-repair all enter through +`audiobookRecastUnknown`; for a span cast they are routed to `audiobookVerifySpans` +(`static/js/audiobook-cast-spans.js`) instead of the classic passes, which would have the LLM re-emit +text. Segments carry their `spanId`, so a cast maps back to the deterministic spans; a cast that does +not (classic cast, or a different text) is declined and the classic path runs. + +* **2nd Quality Run (Verify) is advisory.** One extra no-thinking pass with different window boundaries + (5000/7000 chars vs 2400/3600), about as long as the first cast (~10 min per novel). It **never + changes a speaker**. Lines both passes agree on get `verified: 'agreed'`; where they disagree the + first answer is kept, the line gets `verified: 'disputed'` and the other pass's answer is stored as + `altSpeaker` (shown in the confidence explanation); a line the first pass left Unknown stays Unknown + (`unresolved`, with the other pass's suggestion). Confidence: agreed +10, disputed −14. Nothing is + applied until the pass finished, so a stopped or failed run leaves the cast unchanged. +* **Recast unknown** (`unknownOnly`): the grammar pre-pass first, then one reasoning question per + cluster of remaining Unknown lines (fuller context, `attribute_only` so the rest of the window is + context). A line the model still cannot place stays Unknown — never invented. Auto-repair after a + span cast is off: Unknown lines stay visible for review. + +### What was measured, and what was rejected + +Book 1, alias-tolerant, real in-app runs on the KONTROLLE document: + +| | words | first pass | after | +|---|---|---|---| +| lines both passes agreed on | 18 442 | 82.0 % | 82.0 % (unchanged) | +| lines the passes disputed, overruled by a reasoning question restricted to the two candidates | 3 203 | 66.5 % | **61.4 %** | +| Unknown lines "resolved" by the second pass or reasoning | 128 | 0 % | 3 % | + +An earlier offline experiment suggested adjudication helped (56 % → 72 % on a sample of disagreeing +lines) but compared against a biased baseline; on the real pipeline it made the lines it changed worse, +so the adjudication step (and the `candidates` request field) was removed. The only reliable signal is +agreement itself: agreed lines are right 82 % of the time, contested first answers 67 %. +Sampling matters: windows of ~1 200 characters were clearly worse (−4 points) and ~5 000 about equal to +the default, so the default stays. + +Request additions: `attribute_only` `[ids]` — the other spans are context; unrequested lines are never +reported missing. + ## Security / privacy Book text is sent to the configured LLM exactly as before (same endpoint, same settings); diff --git a/core/attribution_spans.py b/core/attribution_spans.py index f69c1b1..72d50ad 100644 --- a/core/attribution_spans.py +++ b/core/attribution_spans.py @@ -54,6 +54,14 @@ def validate_spans(raw: Any) -> list[dict]: return spans +def validate_only_ids(raw: Any, spans: list[dict]) -> list[str]: + """Ids to attribute when the rest of the window is context only.""" + if not isinstance(raw, list): + return [] + dialogue = {s["id"] for s in spans if s["kind"] == "dialogue"} + return [i for i in dict.fromkeys(raw) if isinstance(i, str) and i in dialogue] + + _RULES = """\ You attribute speakers in prose fiction for a multi-voice audiobook. diff --git a/routes/conversation.py b/routes/conversation.py index 504039a..4b275aa 100644 --- a/routes/conversation.py +++ b/routes/conversation.py @@ -38,6 +38,7 @@ from core.retry import request_with_retries from core.attribution_spans import ( SpanRequestError, build_messages as _span_build_messages, max_tokens_for as _span_max_tokens, parse_response as _span_parse_response, validate_spans as _span_validate, + validate_only_ids as _span_validate_only_ids, ) from core.tts_helpers import _preview_request_audio from routes.stt import _transcribe_audio, _clean_stt_backend @@ -4155,6 +4156,7 @@ async def attribute_spans(request: Request): following = str(data.get("following") or "")[:1500] language = str(data.get("language") or "")[:40] want_reasoning = bool(data.get("want_reasoning")) + only = _span_validate_only_ids(data.get("attribute_only"), spans) or None settings = _load_settings() llm_url = (data.get("llm_url") or settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/") model = (data.get("model") or settings.get("llm_model") or "").strip() @@ -4163,6 +4165,7 @@ async def attribute_spans(request: Request): fallback_model = (settings.get("llm_model") or "").strip() def ask(only_ids=None, token_boost: int = 0) -> dict: + only_ids = only_ids or only system, user = _span_build_messages(spans, known, recent, following, language, want_reasoning, only_ids) payload: dict = { "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}], @@ -4176,7 +4179,11 @@ async def attribute_spans(request: Request): if model: payload["model"] = model raw = _span_llm_answer(payload, llm_url, model, api_key, timeout_seconds, fallback_model) - return _span_parse_response(raw, spans) + result = _span_parse_response(raw, spans) + asked = only_ids or None + if asked: # lines that were not requested are not "missing" + result["missing"] = [i for i in result["missing"] if i in asked] + return result try: result = await asyncio.to_thread(ask) @@ -4184,7 +4191,8 @@ async def attribute_spans(request: Request): # Nothing parsed at all: a backend that ignores the thinking switch spends the # whole budget on reasoning. Give it room to finish once. result = await asyncio.to_thread(ask, None, 3000) - if result["missing"] and len(result["missing"]) < sum(1 for s in spans if s["kind"] == "dialogue"): + requested = len(only) if only else sum(1 for s in spans if s["kind"] == "dialogue") + if result["missing"] and len(result["missing"]) < requested: # Partial answer (truncated or skipped lines): one focused retry for just those lines. retry = await asyncio.to_thread(ask, list(result["missing"])) result["speakers"].update({k: v for k, v in retry["speakers"].items() if k not in result["speakers"]}) diff --git a/static/index.html b/static/index.html index 96d868d..ab44f2c 100644 --- a/static/index.html +++ b/static/index.html @@ -10,7 +10,7 @@ - + @@ -27,12 +27,12 @@ - - - - - - + + + + + + diff --git a/static/js/audiobook-cast-spans.js b/static/js/audiobook-cast-spans.js index f415095..6606fd7 100644 --- a/static/js/audiobook-cast-spans.js +++ b/static/js/audiobook-cast-spans.js @@ -101,15 +101,15 @@ async function audiobookCastSpans(overrideUrl, overrideModel, resume) { win.spans.forEach((sp, k) => { const page = _abPageAt(marks, sp.start); - if (sp.kind === 'narration') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page }); return; } + if (sp.kind === 'narration') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page, spanId: sp.id }); return; } const hit = answer.speakers && answer.speakers[sp.id]; let speaker = hit && hit.speaker; if (!speaker || answer.failed) { const before = win.spans[k - 1]?.text || '', after = win.spans[k + 1]?.text || ''; speaker = audiobookGuessSpeaker(after, before) || 'Unknown'; } - if (speaker === 'Narrator') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page }); return; } - segs.push({ speaker, type: 'dialogue', text: sp.text, emotion: (hit && hit.emotion) || '', page }); + if (speaker === 'Narrator') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page, spanId: sp.id }); return; } + segs.push({ speaker, type: 'dialogue', text: sp.text, emotion: (hit && hit.emotion) || '', page, spanId: sp.id }); }); audiobookResolveUnknowns(segs, allSegments.slice(-2), roster); @@ -158,10 +158,6 @@ async function audiobookCastSpans(overrideUrl, overrideModel, resume) { const stoppedEarly = _audiobook.cancel && completed < windows.length; view.complete(`${stoppedEarly ? 'Stopped early · ' : ''}${speakers.size} character${speakers.size !== 1 ? 's' : ''} · ${allSegments.length} segments`, audiobookShowPreview, audiobookCast, audiobookRecastUnknown); - if (!stoppedEarly && !_audiobook.cancel && audiobookAutoRepairEnabled()) { - const unknown = allSegments.filter(s => s?.type === 'dialogue' && (!s.speaker || /^Unknown|Unbekannt/i.test(s.speaker))).length; - if (unknown > 10) setTimeout(() => { audiobookAutoRepairCast().catch(e => console.warn('[auto-repair]', e)); }, 800); - } } catch (e) { console.error('[audiobookCastSpans] post-processing failed', e); view.done({ stopped: true, message: 'Finished casting, but saving/cleanup failed: ' + e.message }); @@ -169,6 +165,166 @@ async function audiobookCastSpans(overrideUrl, overrideModel, resume) { return true; } + +// ── Verify / recast for span casts ────────────────────────────────────────── +// Verify (the 2nd quality run) is ADVISORY: a second, differently-windowed pass over every dialogue +// line. It never changes a speaker. Lines both passes agree on are marked `verified: 'agreed'`; where +// they disagree the first answer is kept, the line is marked `disputed` and the other pass's answer is +// stored as `altSpeaker` for the reviewer. Measured on a hand-corrected book: agreed lines were 82 % +// right, contested first answers 67 %, and letting a reasoning pass overrule the first answer made those +// lines worse (61 %), so nothing is overruled. Recast unknown (`unknownOnly`) re-asks only the Unknown +// lines, with reasoning and fuller context; a line the model cannot place stays Unknown. +const VERIFY_WINDOW = { targetChars: 5000, hardChars: 7000 }; +const VERIFY_CLUSTER_GAP = 10; + +async function _abAsk(body, signal, timeoutMs) { + const r = await audiobookFetchWithTimeout('/api/attribute-spans', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, signal, body: JSON.stringify(body), + }, timeoutMs + 5000); + if (!r.ok) throw new Error((await r.json().catch(() => ({}))).detail || ('HTTP ' + r.status)); + return r.json(); +} + +function _abIsUnknownSpeaker(name) { return !name || /^(unknown|unbekannt)/i.test(String(name).trim()); } + +async function audiobookVerifySpans(overrideUrl, overrideModel, options = {}) { + const segs = _audiobook.segments || []; + const text = _audiobook.lastText || ''; + const S = root.TTVC?.audiobook?.spans; + if (!S || !text || !segs.some(s => s.spanId)) return false; + const spans = S.segmentByQuotes(text); + if (!spans) return false; + const idOf = (sp) => (sp.kind === 'dialogue' ? 'D' : 'N') + sp.index; + const posInSpans = new Map(spans.map((sp, k) => [idOf(sp), k])); + const tagged = segs.filter(s => s.spanId); + if (tagged.filter(s => posInSpans.has(s.spanId)).length < 0.95 * tagged.length) return false; // not this text + if (_abBusyGuard('Verify speakers')) return true; + + const unknownOnly = !!options.unknownOnly; + const segById = new Map(tagged.map(s => [s.spanId, s])); + const dialogueIds = spans.filter(sp => sp.kind === 'dialogue').map(idOf); + const posInSegs = new Map(segs.map((s, k) => [s.spanId, k])); + const rosterNow = () => [...new Set(segs.filter(s => s.type === 'dialogue' && !_abIsUnknownSpeaker(s.speaker)).map(s => s.speaker))]; + const recentBefore = (id) => { + const out = []; + for (let k = (posInSegs.get(id) ?? 0) - 1; k >= 0 && out.length < RECENT_LINES; k--) { + const s = segs[k]; + if (s.type === 'dialogue' && !_abIsUnknownSpeaker(s.speaker)) out.unshift(`${s.speaker}: ${(s.text || '').slice(0, 80)}`); + } + return out.join('\n'); + }; + + _audiobook.running = true; _audiobook.cancel = false; + if (typeof window.setNavCastingBadge === 'function') window.setNavCastingBadge(true); + const ac = new AbortController(); _audiobook.abort = () => ac.abort(); + if (overrideUrl && typeof overrideUrl !== 'string') overrideUrl = null; + const llmUrl = overrideUrl || audiobookLlmUrl(); + const model = audiobookSafeLlmModel(overrideModel || audiobookLlmModel()); + const language = audiobookLang(text); + const llm = { + url: document.getElementById('ab-cv-llm-url')?.value.trim() || llmUrl, + model: audiobookSafeLlmModel(document.getElementById('ab-cv-llm-select')?.value || model), + }; + const decisions = new Map(); // unknownOnly: span id -> speaker + const flags = new Map(); // verify: span id -> { verified, alt } + const stats = { checked: 0, agreed: 0, disputed: 0, resolvedUnknown: 0 }; + let finished = false; + const clusters = (ids) => { + const out = []; + ids.forEach(id => { + const pos = posInSpans.get(id); + if (out.length && pos - out[out.length - 1].last <= VERIFY_CLUSTER_GAP) { out[out.length - 1].ids.push(id); out[out.length - 1].last = pos; } + else out.push({ ids: [id], first: pos, last: pos }); + }); + return out; + }; + const windows = unknownOnly ? [] : S.buildWindows(spans, text, VERIFY_WINDOW).filter(w => w.hasDialogue); + const unknownIds = unknownOnly ? dialogueIds.filter(id => _abIsUnknownSpeaker(segById.get(id)?.speaker)) : []; + const groups = unknownOnly ? clusters(unknownIds) : []; + const view = audiobookCastView(Math.max(1, unknownOnly ? groups.length : windows.length), llmUrl, model); + const asWindow = (lo, hi) => spans.slice(lo, hi).map(sp => ({ id: idOf(sp), kind: sp.kind, text: sp.text.length > 1500 ? sp.text.slice(0, 900) + ' [...] ' + sp.text.slice(-600) : sp.text })); + + try { + if (unknownOnly) { + const pre = audiobookResolveUnknowns(segs, [], _audiobook.roster || []); + if (pre.length) toast(`${pre.length} Unknown line${pre.length !== 1 ? 's' : ''} resolved by grammar rules`, 'success'); + if (!unknownIds.length) { toast('No Unknown speakers found', 'success'); view.done(); audiobookShowPreview(); return true; } + for (let g = 0; g < groups.length && !_audiobook.cancel; g++) { + const grp = groups[g]; + view.update(g, `Re-reading Unknown lines ${g + 1} / ${groups.length}…`); + let ans; + try { ans = await _abAsk({ + spans: asWindow(Math.max(0, grp.first - 6), Math.min(spans.length, grp.last + 3)), attribute_only: grp.ids, + known_characters: rosterNow().slice(-ROSTER_LIMIT), recent: recentBefore(grp.ids[0]), language, want_reasoning: true, + llm_url: llm.url, model: llm.model, timeout_seconds: audiobookTimeoutSeconds(AUDIOBOOK_RECAST_TIMEOUT_MS), + }, ac.signal, AUDIOBOOK_RECAST_TIMEOUT_MS); } + catch (err) { if (err.name === 'AbortError') throw err; view.note(`⚠️ Unknown lines ${g + 1}: ${err.message}`); continue; } + for (const id of grp.ids) { + const hit = ans.speakers && ans.speakers[id]; + if (hit && !_abIsUnknownSpeaker(hit.speaker) && hit.speaker !== 'Narrator') decisions.set(id, hit.speaker); + } + } + } else { + for (let i = 0; i < windows.length && !_audiobook.cancel; i++) { + view.update(i, `Second pass ${i + 1} / ${windows.length}…`); + const win = windows[i].spans; + let ans; + try { ans = await _abAsk({ + spans: win.map(sp => ({ id: sp.id, kind: sp.kind, text: sp.llmText })), known_characters: rosterNow().slice(-ROSTER_LIMIT), + recent: recentBefore(win.find(sp => sp.kind === 'dialogue').id), language, llm_url: llm.url, model: llm.model, + timeout_seconds: audiobookTimeoutSeconds(AUDIOBOOK_ATTRIBUTION_TIMEOUT_MS), + }, ac.signal, AUDIOBOOK_ATTRIBUTION_TIMEOUT_MS); } + catch (err) { if (err.name === 'AbortError') throw err; view.note(`⚠️ Second pass ${i + 1}: ${err.message}`); continue; } + for (const [id, v] of Object.entries(ans.speakers || {})) { + const first = segById.get(id)?.speaker, other = v.speaker; + if (!segById.has(id) || other === 'Narrator' || _abIsUnknownSpeaker(other)) continue; // no usable second opinion + stats.checked++; + if (_abIsUnknownSpeaker(first)) flags.set(id, { verified: 'unresolved', alt: other }); + else if (first === other) { flags.set(id, { verified: 'agreed' }); stats.agreed++; } + else { flags.set(id, { verified: 'disputed', alt: other }); stats.disputed++; } + } + } + } + finished = !_audiobook.cancel; + } catch (err) { + if (err.name !== 'AbortError') view.note('Error while verifying: ' + err.message); + } finally { + _audiobook.running = false; _audiobook.abort = null; _abHideCastingControl(); + } + + if (!finished) { + view.done({ stopped: true, message: 'Verification stopped. Existing cast preserved.' }); + toast('Verification stopped. Existing cast preserved.', 'info'); + return true; + } + try { + decisions.forEach((speaker, id) => { + const seg = segById.get(id); + if (!seg || !_abIsUnknownSpeaker(seg.speaker)) return; + seg.speaker = speaker; stats.resolvedUnknown++; + if (!_audiobook.roster.includes(speaker)) _audiobook.roster.push(speaker); + }); + flags.forEach((f, id) => { const seg = segById.get(id); if (seg) { seg.verified = f.verified; if (f.alt) seg.altSpeaker = f.alt; else delete seg.altSpeaker; } }); + { const ph = _audiobookDropPhantomSpeakers(segs); if (ph.dropped) console.info('[verify] dropped', ph.dropped, 'phantom speaker line(s)'); } + _audiobookResolveRoleAliases(segs, text); + _audiobookConsolidateSpeakerAliases(segs); + _abStampConfidence(segs); + _abSaveDraft(segs, _audiobook.roster || [], text, _audiobook.completedChunks || segs.length, _audiobook.completedTotal || _audiobook.completedChunks || segs.length); + audiobookSaveAsRehearsal({ silent: true }); + const summary = unknownOnly + ? `${stats.resolvedUnknown} Unknown line${stats.resolvedUnknown !== 1 ? 's' : ''} resolved` + : `${stats.checked} lines read twice · ${stats.agreed} confirmed, ${stats.disputed} flagged for review`; + view.complete(summary, audiobookShowPreview, audiobookCast, audiobookRecastUnknown); + toast(summary, 'success'); + } catch (e) { + console.error('[audiobookVerifySpans] apply failed', e); + view.done({ stopped: true, message: 'Verification finished, but applying it failed: ' + e.message }); + } + return true; +} + +root.audiobookVerifySpans = audiobookVerifySpans; + root.audiobookSpanModeEnabled = audiobookSpanModeEnabled; root.audiobookCastSpans = audiobookCastSpans; root.TTVC = root.TTVC || {}; diff --git a/static/js/audiobook-pipeline.js b/static/js/audiobook-pipeline.js index 15334c9..2a94c30 100644 --- a/static/js/audiobook-pipeline.js +++ b/static/js/audiobook-pipeline.js @@ -20,6 +20,9 @@ function _abBusyGuard(what) { async function audiobookRecastUnknown(overrideUrl, overrideModel, options = {}) { if (_abBusyGuard('Recast / verify')) return; + // A span-mode cast keeps its text exactly; every repair/verify entry point re-asks only who speaks. + if (typeof root.audiobookVerifySpans === 'function' && root.audiobookSpanModeEnabled() + && await root.audiobookVerifySpans(overrideUrl, overrideModel, { unknownOnly: !options.includeNarrator })) return; const segs = _audiobook.segments; if (!segs || !segs.length) return; const originalSegments = segs.map(s => ({...s})); diff --git a/static/js/audiobook.js b/static/js/audiobook.js index fa622eb..6c730f2 100644 --- a/static/js/audiobook.js +++ b/static/js/audiobook.js @@ -1082,6 +1082,10 @@ function _abSegmentConfidence(segs, i) { if (!/[»«]/.test(t)) { score -= 14; why.push('no quote marks'); } if (/\n\s*\n/.test(t.trim())) { score -= 22; why.push('fused: several turns in one segment'); } if (t.trim().length < 12) { score -= 10; why.push('very short line'); } + // Span-cast verification: two independently windowed passes agreeing is strong evidence + // (82 % right where they agreed vs 67 % where they did not, measured on a hand-corrected book). + if (s.verified === 'agreed') { score += 10; why.push('two passes agree'); } + else if (s.verified === 'disputed') { score -= 14; why.push('two passes disagree' + (s.altSpeaker ? ' (other pass: ' + s.altSpeaker + ')' : '')); } return { score: Math.max(2, Math.min(99, score)), why }; } diff --git a/tests/audiobook-spans.spec.js b/tests/audiobook-spans.spec.js index df2c81b..d6aa4d4 100644 --- a/tests/audiobook-spans.spec.js +++ b/tests/audiobook-spans.spec.js @@ -31,3 +31,101 @@ test('span mode is on by default and can be switched off per browser', async ({ expect(await page.evaluate(() => window.audiobookSpanModeEnabled())).toBe(false); await page.evaluate(() => localStorage.removeItem('ab-span-mode')); }); + +// ── Verify / recast for span casts ─────────────────────────────────────────── +// A cast made in span mode is verified by a second pass plus candidate-restricted adjudication; these +// tests drive audiobookVerifySpans against a mocked /api/attribute-spans. +const FILLER = Array.from({ length: 6 }, (_, i) => `»Zeile ${i} ist ein Satz.« sagte er.`).join(' '); +const BOOK = `${FILLER} Der Zwerg schneuzte sich. »Na los!« sagte Linosch. »Vielen Dank!« Er ging. »Wirklich?« fragte jemand. »Ist das so?« Stille.`; + +async function seedCast(page, firstPass, book = BOOK) { + await page.evaluate(([book, first]) => { + const spans = window.TTVC.audiobook.spans.segmentByQuotes(book); + window._audiobook.lastText = book; + window._audiobook.roster = ['Marcian', 'Linosch', 'Alrik']; + window._audiobook.segments = spans.map((sp) => { + const id = (sp.kind === 'dialogue' ? 'D' : 'N') + sp.index; + return sp.kind === 'dialogue' + ? { speaker: first[sp.text] || 'Marcian', type: 'dialogue', text: sp.text, emotion: '', spanId: id } + : { speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', spanId: id }; + }); + }, [book, firstPass]); +} + +async function mockSpanEndpoint(page, plan) { + const calls = []; + await page.route('**/api/attribute-spans', route => { + const body = route.request().postDataJSON(); + calls.push({ reasoning: !!body.want_reasoning, only: body.attribute_only || null, ids: body.spans.filter(s => s.kind === 'dialogue').map(s => s.id) }); + const table = body.want_reasoning ? plan.reasoning : plan.second; + const speakers = {}; + for (const s of body.spans) { + if (s.kind !== 'dialogue') continue; + if (body.attribute_only && !body.attribute_only.includes(s.id)) continue; + const name = table[s.text] ?? (body.want_reasoning ? undefined : 'Marcian'); // the second pass answers every line + if (name !== undefined) speakers[s.id] = { speaker: name, emotion: '' }; + } + return route.fulfill({ json: { speakers, missing: [], characters: [] } }); + }); + return calls; +} + +test('verify is advisory: it marks agreement and disputes but never changes a speaker', async ({ page }) => { + await openReader(page); + await seedCast(page, { 'Na los!': 'Linosch', 'Vielen Dank!': 'Alrik', 'Wirklich?': 'Unknown', 'Ist das so?': 'Alrik' }); + const calls = await mockSpanEndpoint(page, { second: { 'Na los!': 'Linosch', 'Vielen Dank!': 'Marcian', 'Wirklich?': 'Alrik', 'Ist das so?': 'Linosch' }, reasoning: {} }); + const handled = await page.evaluate(() => window.audiobookVerifySpans(undefined, undefined, {})); + expect(handled).toBe(true); + const rows = await page.evaluate(() => window._audiobook.segments.filter(s => s.type === 'dialogue') + .map(s => [s.text, s.speaker, s.verified, s.altSpeaker || null, s.confWhy])); + const by = Object.fromEntries(rows.map(([t, sp, v, alt, why]) => [t, { sp, v, alt, why }])); + expect(by['Na los!']).toMatchObject({ sp: 'Linosch', v: 'agreed' }); + expect(by['Vielen Dank!']).toMatchObject({ sp: 'Alrik', v: 'disputed', alt: 'Marcian' }); // first answer kept + expect(by['Ist das so?']).toMatchObject({ sp: 'Alrik', v: 'disputed', alt: 'Linosch' }); + expect(by['Wirklich?']).toMatchObject({ sp: 'Unknown', v: 'unresolved', alt: 'Alrik' }); // never filled in silently + expect(by['Zeile 0 ist ein Satz.'].v).toBe('agreed'); + expect(by['Vielen Dank!'].why).toContain('two passes disagree (other pass: Marcian)'); + expect(by['Na los!'].why).toContain('two passes agree'); + // a single no-thinking pass: no reasoning calls, no per-line restriction + expect(calls.length).toBeGreaterThan(0); + expect(calls.every(c => !c.reasoning && !c.only)).toBe(true); + // the text is untouched, every span still present once + const text = await page.evaluate(() => window._audiobook.segments.map(s => s.text).join(' ')); + expect(text.replace(/[»«]/g, '').replace(/\s+/g, ' ')).toBe(BOOK.replace(/[»«]/g, '').replace(/\s+/g, ' ')); +}); + +test('recast unknown on a span cast only re-asks the Unknown lines, with reasoning', async ({ page }) => { + await openReader(page); + // Untagged lines before any speaker is known: the grammar pre-pass cannot settle them, so the model is asked. + const book = `»Wer ist da?« Wind. »Niemand.« Nacht. ${FILLER}`; + await seedCast(page, { 'Wer ist da?': 'Unknown', 'Niemand.': 'Unknown' }, book); + const calls = await mockSpanEndpoint(page, { second: {}, reasoning: { 'Wer ist da?': 'Alrik', 'Niemand.': 'Unknown' } }); + await page.evaluate(() => window.audiobookRecastUnknown(undefined, undefined, {})); + await expect.poll(async () => page.evaluate(() => window._audiobook.segments.find(s => s.text === 'Wer ist da?').speaker)).toBe('Alrik'); + const speakers = await page.evaluate(() => Object.fromEntries(window._audiobook.segments.filter(s => s.type === 'dialogue').map(s => [s.text, s.speaker]))); + expect(speakers['Niemand.']).toBe('Unknown'); // still unresolved: left alone, never invented + expect(speakers['Zeile 0 ist ein Satz.']).toBe('Marcian'); // known lines are not re-litigated + expect(calls.length).toBeGreaterThan(0); + expect(calls.every(c => c.reasoning)).toBe(true); + // the two Unknown lines were the only ones ever asked about + const asked = new Set(calls.flatMap(c => c.only || [])); + const unknownIds = await page.evaluate(() => window._audiobook.segments.filter(s => s.text === 'Wer ist da?' || s.text === 'Niemand.').map(s => s.spanId)); + expect(asked.size).toBeGreaterThan(0); + expect([...asked].every(id => unknownIds.includes(id))).toBe(true); +}); + +test('verify leaves the cast untouched when the endpoint fails, and declines a cast it cannot map to the text', async ({ page }) => { + await openReader(page); + await seedCast(page, { 'Na los!': 'Linosch', 'Vielen Dank!': 'Alrik' }); + const before = await page.evaluate(() => JSON.stringify(window._audiobook.segments.map(s => [s.spanId, s.speaker]))); + await page.route('**/api/attribute-spans', route => route.fulfill({ status: 503, json: { detail: 'down' } })); + await page.evaluate(() => window.audiobookVerifySpans(undefined, undefined, {})); + const after = await page.evaluate(() => JSON.stringify(window._audiobook.segments.map(s => [s.spanId, s.speaker]))); + expect(after).toBe(before); + // a cast from a different text (or a classic cast without span ids) is declined so the classic path can run + const declined = await page.evaluate(() => { + window._audiobook.segments = [{ speaker: 'X', type: 'dialogue', text: 'Hallo', emotion: '' }]; + return window.audiobookVerifySpans(undefined, undefined, {}); + }); + expect(declined).toBe(false); +}); diff --git a/tests/test_attribution_spans.py b/tests/test_attribution_spans.py index 9a93171..619ab96 100644 --- a/tests/test_attribution_spans.py +++ b/tests/test_attribution_spans.py @@ -12,7 +12,7 @@ from fastapi import FastAPI from fastapi.testclient import TestClient from core.attribution_spans import ( - SpanRequestError, build_messages, max_tokens_for, parse_response, validate_spans, + SpanRequestError, build_messages, max_tokens_for, parse_response, validate_only_ids, validate_spans, ) import routes.conversation as conversation @@ -67,6 +67,18 @@ class PromptTests(unittest.TestCase): self.assertLessEqual(max_tokens_for(many(400)), 4096) +class AttributeOnlyTests(unittest.TestCase): + def test_attribute_only_keeps_valid_dialogue_ids_in_order(self): + spans = validate_spans(SPANS) + self.assertEqual(validate_only_ids(["D3", "D1", "D3", "N0", "D9", 5], spans), ["D3", "D1"]) + self.assertEqual(validate_only_ids("D1", spans), []) + + def test_prompt_restricts_the_request_to_those_lines(self): + _, user = build_messages(SPANS, only_ids=["D1"]) + self.assertIn("Attribute these lines: D1", user) + self.assertNotIn("D3", user.split("Attribute these lines:")[1]) + + class ParseTests(unittest.TestCase): def test_parses_clean_json_and_normalises_names(self): raw = '{"speakers":[{"id":"D1","speaker":" Linosch ","emotion":"gereizt"},{"id":"D3","speaker":"unbekannt","emotion":""}]}' @@ -132,6 +144,16 @@ class RouteTests(unittest.TestCase): retry_prompt = llm.call_args_list[1].args[0]["messages"][1]["content"] self.assertIn("Attribute these lines: D3", retry_prompt) + def test_attribute_only_limits_the_request_and_the_missing_accounting(self): + answer = '{"speakers":[{"id":"D3","speaker":"Alrik","emotion":""}]}' + with mock.patch.object(conversation, "_span_llm_answer", return_value=answer) as llm: + r = self.client.post("/api/attribute-spans", json={"spans": SPANS, "attribute_only": ["D3"]}) + body = r.json() + self.assertEqual(body["speakers"]["D3"]["speaker"], "Alrik") + self.assertEqual(body["missing"], []) + llm.assert_called_once() + self.assertIn("Attribute these lines: D3", llm.call_args.args[0]["messages"][1]["content"]) + def test_reports_failure_instead_of_erroring_when_the_llm_is_down(self): with mock.patch.object(conversation, "_span_llm_answer", side_effect=RuntimeError("LLM HTTP 503")): r = self.client.post("/api/attribute-spans", json={"spans": SPANS})