feat(audiobook): advisory verify and span-native recast for span casts
Verify / recast now run span-natively (no text re-emission): a second differently-windowed pass marks lines agreed/disputed and records the other answer, without changing any speaker (~9 min vs ~49 min). Measured on the real pipeline, overruling disputed lines with a reasoning adjudication made them worse (66.5% -> 61.4%), so that step was removed; agreed lines were 83% right vs 48.5% for disputed ones, which is the signal used for confidence and review. Auto-repair no longer runs after a span cast; segments carry a spanId. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
a920f67008
commit
37e2622d4e
15
CHANGELOG.md
15
CHANGELOG.md
@ -5,6 +5,21 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
||||
|
||||
---
|
||||
|
||||
## [1.34.0] — 2026-09-19
|
||||
|
||||
### Added
|
||||
- **Verify / recast for span-mode casts** (`audiobookVerifySpans`): the app's Recast unknown, 2nd Quality Run and auto-repair entry points now run span-natively for a span cast instead of the classic passes that re-emit text. **Verify is advisory**: one extra differently-windowed pass (~10 min per novel) that never changes a speaker — agreeing lines are marked `agreed`, disagreeing lines keep the first answer, are marked `disputed` and record the other pass's answer as `altSpeaker`; the confidence score reflects both. **Recast unknown** re-asks only the Unknown lines with reasoning and fuller context (`attribute_only` on `/api/attribute-spans`) and leaves any line the model cannot place Unknown. Segments now carry a `spanId`.
|
||||
|
||||
### Changed
|
||||
- Auto-repair no longer runs after a span-mode cast: Unknown lines stay visible for review instead of being guessed at.
|
||||
|
||||
### Measured — and rejected
|
||||
- On the real pipeline (Book 1, in-app), letting a reasoning question restricted to the two candidates overrule disputed lines made them **worse** (66.5 % → 61.4 % on 3 203 words), and the second pass "resolving" Unknown lines was ~3 % right, so no adjudication or silent filling exists. Lines both passes agreed on were 82 % right versus ~67 % for disputed first answers — that agreement is the signal used. An earlier offline experiment had suggested the opposite; it used a biased baseline (see `architecture/audiobook-attribution.md`).
|
||||
- Reasoning mode for the first pass (25× slower, no gain) and smaller/larger windows (−4 / ±0 points) were also tested and not adopted.
|
||||
- Final in-app run on Book 1: the advisory verify took 526 s, changed no speaker, and separated the lines well — 757 agreed lines were 83.2 % right, 118 disputed lines 48.5 % right (12.5 % of the dialogue holding ~28 % of the errors), which is what makes the flags useful for review.
|
||||
|
||||
---
|
||||
|
||||
## [1.33.0] — 2026-09-19
|
||||
|
||||
### Fixed
|
||||
|
||||
10
Manual.md
10
Manual.md
@ -29,7 +29,15 @@ the Rehearser and MP3 export.
|
||||
hours).
|
||||
3. Progress shows per passage; **Stop Casting** keeps what is done and **Continue
|
||||
casting** resumes.
|
||||
4. Review the cast as before (Recast unknown, Verify, Define characters, …).
|
||||
4. Review the cast as before. **Recast unknown** re-reads only the lines still marked
|
||||
Unknown, with more context; a line the model cannot place stays Unknown rather than
|
||||
getting a guessed voice. **Verify** (the second quality run) reads every spoken line a
|
||||
second time with different passage boundaries — about as long as the first cast — and
|
||||
**never changes a speaker**: lines both readings agree on are marked confirmed (about
|
||||
4 in 5 of those were right in testing), and lines where they disagree keep the first
|
||||
reading, are flagged in the review list and show what the other reading said. Use the
|
||||
flags to decide where to spend your review time. Neither step can change the book's
|
||||
text, and a stopped or failed run leaves the cast as it was.
|
||||
|
||||
**Expected result:** every word of the book appears exactly once, quoted speech is
|
||||
typed as dialogue and everything else as narration. When the text only refers to a
|
||||
|
||||
@ -35,7 +35,8 @@ Two self-hosted Docker instances, both port 7890:
|
||||
| `core/` | domain logic — config, database, audio, voice, routing, magazine |
|
||||
| `routes/` | HTTP boundary, one module per feature area |
|
||||
| `static/js/` | frontend, one module per feature area (no build step for most) |
|
||||
| `tests/` | `test_*.py` (unittest) and `*.spec.js` (Playwright) |
|
||||
| `tests/` | `test_*.py` (unittest, run with `python3 -m unittest tests.test_x`), `*.test.mjs` (`npm run test:unit`, Node) and `*.spec.js` (Playwright — must mock every write, the app on :7890 is the real one) |
|
||||
| `scripts/` | `score_attribution.py` (word-aligned cast scoring), release/minify/frontend-gate tooling |
|
||||
| `architecture/` | SOPs for subsystems whose design is not obvious from code |
|
||||
| `workflows/` | ComfyUI graphs for character portrait generation |
|
||||
|
||||
@ -45,7 +46,8 @@ Two self-hosted Docker instances, both port 7890:
|
||||
|---|---|---|---|
|
||||
| Voice library / clone / design | `routes/library.py`, `core/voice.py` | `library.js` | mature |
|
||||
| Read Aloud (books, PDFs) | `routes/reader.py` | `reader.js` | mature |
|
||||
| Audiobook casting | `routes/conversation.py` | `audiobook*.js` | mature |
|
||||
| Audiobook casting (span mode: deterministic quote spans, LLM names speakers only; verify/recast) | `routes/conversation.py`, `core/attribution_spans.py` | `audiobook*.js`, `audiobook-spans.js`, `audiobook-cast-spans.js` | **new 2026-09-19** — see [architecture/audiobook-attribution.md](architecture/audiobook-attribution.md) |
|
||||
| Characters (avatar/concept identity, appearance forms) | `routes/characters.py`, `routes/conversation.py` | `library-characters.js`, `character-sheets.js` | **reworked 2026-09-19** — see [architecture/character-visual-identity.md](architecture/character-visual-identity.md) |
|
||||
| Rehearser | `routes/rehearsals_db.py` | `rehearser-*.js` | mature |
|
||||
| Studio (phase wizard) | — | `studio.js` | mature |
|
||||
| TTS engines / routing | `routes/tts.py`, `core/routing.py` | `engines.js` | mature |
|
||||
@ -83,6 +85,16 @@ which is the writable volume.
|
||||
**unresolved** — the compose files currently point at ghcr.
|
||||
## Known limitations
|
||||
|
||||
- Audiobook attribution (span mode) is measured, not perfect: on three hand-corrected novels about
|
||||
90-92 % of all words end up with the right voice and ~75-80 % of spoken words with the right
|
||||
speaker. The references themselves contain label noise (1-2 points is not meaningful). Open, real
|
||||
problems: a person known only by a role that means different people in different scenes ("der
|
||||
Fremde") is not resolved to a character; look-alike characters in one scene are the usual mix-up;
|
||||
very long monologues are inconsistent in the references.
|
||||
- A speech is only found where the text marks it with a recognised quotation style (»…«, „…“,
|
||||
“…”, «…»); a book with none of them uses the older, slower whole-passage engine.
|
||||
- Character image generation needs ComfyUI/an image provider; the deterministic prompts, forms and
|
||||
review logic are covered by tests without one, but a real pair render is not part of the suite.
|
||||
- `gitea` backup remote is 157 commits behind `origin`; `main` tracks `gitea`.
|
||||
- Magazine sidebar exclusion is a narrow-column geometry heuristic
|
||||
(`mark_sidebars`), not certain — review always shows every sidebar block.
|
||||
|
||||
22
TODO.md
22
TODO.md
@ -329,7 +329,27 @@ planning pass — this is a multi-week, cross-repo effort.
|
||||
The authenticated panel reports browser cache counters, cgroup memory current/max, server RSS, configured/temp
|
||||
storage, persisted jobs, and sanitized operation durations; host-wide Docker metrics remain unavailable by design.
|
||||
|
||||
## P1 — Speaker attribution: measured state and next steps (paused 2026-09-02)
|
||||
## P1 — Speaker attribution: measured state and next steps
|
||||
|
||||
**Status 2026-09-19 — reworked, see `architecture/audiobook-attribution.md`.** Span mode (deterministic
|
||||
quote spans, the LLM only names speakers) replaced the segmentation-heavy engine: Book 1 every-word-right
|
||||
92.3 % (was 87.0 %), narration/dialogue agreement 97.4 % (was 93.6 %), ~10 min instead of ~4 h; Books 2/3
|
||||
(never tuned on) 90.8 % / 89.9 %. The word-aligned scorer is `scripts/score_attribution.py`; the references
|
||||
have label noise, so 1–2 points is not meaningful. Everything below the line is the historical record of
|
||||
the classic engine's Runs 1–3.
|
||||
|
||||
Still open (each needs a real decision or new evidence, not more prompt tuning):
|
||||
|
||||
- [ ] A person known only by a role that means different people in different scenes ("der Fremde") is not
|
||||
resolved to a character — needs scene-level identity tracking; the label is kept as its own cast entry
|
||||
until the book itself ties it to a name.
|
||||
- [ ] Look-alike main characters in one scene (Marcian/Zerwas/Lancorian) remain the biggest real error
|
||||
class (~1/5 of spoken words). Tried and rejected: reasoning mode (25× slower, no gain), larger/smaller
|
||||
windows, tag-based overrides, reasoning adjudication of disputes (made disputed lines worse). Untried:
|
||||
an explicit scene-cast constraint (characters present in the scene) built from the text.
|
||||
|
||||
---
|
||||
|
||||
|
||||
### Ordered follow-up queue
|
||||
|
||||
|
||||
@ -86,6 +86,46 @@ Book 1 (*Der Sturm*), same reference, same scorer:
|
||||
| wall time | ≈ 4 h | ≈ 11 min |
|
||||
| Unknown-speaker words | 16 | 114 |
|
||||
|
||||
## Verify and recast (span casts)
|
||||
|
||||
Workflow step 3 (Recast unknown), step 4 (2nd Quality Run) and auto-repair all enter through
|
||||
`audiobookRecastUnknown`; for a span cast they are routed to `audiobookVerifySpans`
|
||||
(`static/js/audiobook-cast-spans.js`) instead of the classic passes, which would have the LLM re-emit
|
||||
text. Segments carry their `spanId`, so a cast maps back to the deterministic spans; a cast that does
|
||||
not (classic cast, or a different text) is declined and the classic path runs.
|
||||
|
||||
* **2nd Quality Run (Verify) is advisory.** One extra no-thinking pass with different window boundaries
|
||||
(5000/7000 chars vs 2400/3600), about as long as the first cast (~10 min per novel). It **never
|
||||
changes a speaker**. Lines both passes agree on get `verified: 'agreed'`; where they disagree the
|
||||
first answer is kept, the line gets `verified: 'disputed'` and the other pass's answer is stored as
|
||||
`altSpeaker` (shown in the confidence explanation); a line the first pass left Unknown stays Unknown
|
||||
(`unresolved`, with the other pass's suggestion). Confidence: agreed +10, disputed −14. Nothing is
|
||||
applied until the pass finished, so a stopped or failed run leaves the cast unchanged.
|
||||
* **Recast unknown** (`unknownOnly`): the grammar pre-pass first, then one reasoning question per
|
||||
cluster of remaining Unknown lines (fuller context, `attribute_only` so the rest of the window is
|
||||
context). A line the model still cannot place stays Unknown — never invented. Auto-repair after a
|
||||
span cast is off: Unknown lines stay visible for review.
|
||||
|
||||
### What was measured, and what was rejected
|
||||
|
||||
Book 1, alias-tolerant, real in-app runs on the KONTROLLE document:
|
||||
|
||||
| | words | first pass | after |
|
||||
|---|---|---|---|
|
||||
| lines both passes agreed on | 18 442 | 82.0 % | 82.0 % (unchanged) |
|
||||
| lines the passes disputed, overruled by a reasoning question restricted to the two candidates | 3 203 | 66.5 % | **61.4 %** |
|
||||
| Unknown lines "resolved" by the second pass or reasoning | 128 | 0 % | 3 % |
|
||||
|
||||
An earlier offline experiment suggested adjudication helped (56 % → 72 % on a sample of disagreeing
|
||||
lines) but compared against a biased baseline; on the real pipeline it made the lines it changed worse,
|
||||
so the adjudication step (and the `candidates` request field) was removed. The only reliable signal is
|
||||
agreement itself: agreed lines are right 82 % of the time, contested first answers 67 %.
|
||||
Sampling matters: windows of ~1 200 characters were clearly worse (−4 points) and ~5 000 about equal to
|
||||
the default, so the default stays.
|
||||
|
||||
Request additions: `attribute_only` `[ids]` — the other spans are context; unrequested lines are never
|
||||
reported missing.
|
||||
|
||||
## Security / privacy
|
||||
|
||||
Book text is sent to the configured LLM exactly as before (same endpoint, same settings);
|
||||
|
||||
@ -54,6 +54,14 @@ def validate_spans(raw: Any) -> list[dict]:
|
||||
return spans
|
||||
|
||||
|
||||
def validate_only_ids(raw: Any, spans: list[dict]) -> list[str]:
|
||||
"""Ids to attribute when the rest of the window is context only."""
|
||||
if not isinstance(raw, list):
|
||||
return []
|
||||
dialogue = {s["id"] for s in spans if s["kind"] == "dialogue"}
|
||||
return [i for i in dict.fromkeys(raw) if isinstance(i, str) and i in dialogue]
|
||||
|
||||
|
||||
_RULES = """\
|
||||
You attribute speakers in prose fiction for a multi-voice audiobook.
|
||||
|
||||
|
||||
@ -38,6 +38,7 @@ from core.retry import request_with_retries
|
||||
from core.attribution_spans import (
|
||||
SpanRequestError, build_messages as _span_build_messages, max_tokens_for as _span_max_tokens,
|
||||
parse_response as _span_parse_response, validate_spans as _span_validate,
|
||||
validate_only_ids as _span_validate_only_ids,
|
||||
)
|
||||
from core.tts_helpers import _preview_request_audio
|
||||
from routes.stt import _transcribe_audio, _clean_stt_backend
|
||||
@ -4155,6 +4156,7 @@ async def attribute_spans(request: Request):
|
||||
following = str(data.get("following") or "")[:1500]
|
||||
language = str(data.get("language") or "")[:40]
|
||||
want_reasoning = bool(data.get("want_reasoning"))
|
||||
only = _span_validate_only_ids(data.get("attribute_only"), spans) or None
|
||||
settings = _load_settings()
|
||||
llm_url = (data.get("llm_url") or settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/")
|
||||
model = (data.get("model") or settings.get("llm_model") or "").strip()
|
||||
@ -4163,6 +4165,7 @@ async def attribute_spans(request: Request):
|
||||
fallback_model = (settings.get("llm_model") or "").strip()
|
||||
|
||||
def ask(only_ids=None, token_boost: int = 0) -> dict:
|
||||
only_ids = only_ids or only
|
||||
system, user = _span_build_messages(spans, known, recent, following, language, want_reasoning, only_ids)
|
||||
payload: dict = {
|
||||
"messages": [{"role": "system", "content": system}, {"role": "user", "content": user}],
|
||||
@ -4176,7 +4179,11 @@ async def attribute_spans(request: Request):
|
||||
if model:
|
||||
payload["model"] = model
|
||||
raw = _span_llm_answer(payload, llm_url, model, api_key, timeout_seconds, fallback_model)
|
||||
return _span_parse_response(raw, spans)
|
||||
result = _span_parse_response(raw, spans)
|
||||
asked = only_ids or None
|
||||
if asked: # lines that were not requested are not "missing"
|
||||
result["missing"] = [i for i in result["missing"] if i in asked]
|
||||
return result
|
||||
|
||||
try:
|
||||
result = await asyncio.to_thread(ask)
|
||||
@ -4184,7 +4191,8 @@ async def attribute_spans(request: Request):
|
||||
# Nothing parsed at all: a backend that ignores the thinking switch spends the
|
||||
# whole budget on reasoning. Give it room to finish once.
|
||||
result = await asyncio.to_thread(ask, None, 3000)
|
||||
if result["missing"] and len(result["missing"]) < sum(1 for s in spans if s["kind"] == "dialogue"):
|
||||
requested = len(only) if only else sum(1 for s in spans if s["kind"] == "dialogue")
|
||||
if result["missing"] and len(result["missing"]) < requested:
|
||||
# Partial answer (truncated or skipped lines): one focused retry for just those lines.
|
||||
retry = await asyncio.to_thread(ask, list(result["missing"]))
|
||||
result["speakers"].update({k: v for k, v in retry["speakers"].items() if k not in result["speakers"]})
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta name="format-detection" content="telephone=no">
|
||||
<meta name="color-scheme" content="light dark">
|
||||
<meta name="theme-color" content="#2563EB">
|
||||
<meta name="app-version" content="1.33.0">
|
||||
<meta name="app-version" content="1.34.0">
|
||||
<link rel="manifest" href="/manifest.webmanifest">
|
||||
<link rel="icon" href="/static/icon.svg" type="image/svg+xml">
|
||||
<link rel="apple-touch-icon" href="/static/icon.svg">
|
||||
@ -27,12 +27,12 @@
|
||||
|
||||
<!-- ── Core styles (local — no CDN dependency for first paint) ────────── -->
|
||||
<link rel="stylesheet" href="/static/vendor/mdi/materialdesignicons.min.css">
|
||||
<link rel="stylesheet" href="/static/style.css?v=1.33.0">
|
||||
<link rel="stylesheet" href="/static/css/resource-diagnostics.css?v=1.33.0">
|
||||
<link rel="stylesheet" href="/static/css/background-jobs.css?v=1.33.0">
|
||||
<link rel="stylesheet" href="/static/css/voice-design.css?v=1.33.0">
|
||||
<link rel="stylesheet" href="/static/css/rehearser.css?v=1.33.0">
|
||||
<link rel="stylesheet" href="/static/css/audiobook.css?v=1.33.0">
|
||||
<link rel="stylesheet" href="/static/style.css?v=1.34.0">
|
||||
<link rel="stylesheet" href="/static/css/resource-diagnostics.css?v=1.34.0">
|
||||
<link rel="stylesheet" href="/static/css/background-jobs.css?v=1.34.0">
|
||||
<link rel="stylesheet" href="/static/css/voice-design.css?v=1.34.0">
|
||||
<link rel="stylesheet" href="/static/css/rehearser.css?v=1.34.0">
|
||||
<link rel="stylesheet" href="/static/css/audiobook.css?v=1.34.0">
|
||||
|
||||
|
||||
<!-- ── Flag icons — non-blocking (loaded async, icons appear after JS) ── -->
|
||||
|
||||
@ -101,15 +101,15 @@ async function audiobookCastSpans(overrideUrl, overrideModel, resume) {
|
||||
|
||||
win.spans.forEach((sp, k) => {
|
||||
const page = _abPageAt(marks, sp.start);
|
||||
if (sp.kind === 'narration') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page }); return; }
|
||||
if (sp.kind === 'narration') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page, spanId: sp.id }); return; }
|
||||
const hit = answer.speakers && answer.speakers[sp.id];
|
||||
let speaker = hit && hit.speaker;
|
||||
if (!speaker || answer.failed) {
|
||||
const before = win.spans[k - 1]?.text || '', after = win.spans[k + 1]?.text || '';
|
||||
speaker = audiobookGuessSpeaker(after, before) || 'Unknown';
|
||||
}
|
||||
if (speaker === 'Narrator') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page }); return; }
|
||||
segs.push({ speaker, type: 'dialogue', text: sp.text, emotion: (hit && hit.emotion) || '', page });
|
||||
if (speaker === 'Narrator') { segs.push({ speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', page, spanId: sp.id }); return; }
|
||||
segs.push({ speaker, type: 'dialogue', text: sp.text, emotion: (hit && hit.emotion) || '', page, spanId: sp.id });
|
||||
});
|
||||
|
||||
audiobookResolveUnknowns(segs, allSegments.slice(-2), roster);
|
||||
@ -158,10 +158,6 @@ async function audiobookCastSpans(overrideUrl, overrideModel, resume) {
|
||||
const stoppedEarly = _audiobook.cancel && completed < windows.length;
|
||||
view.complete(`${stoppedEarly ? 'Stopped early · ' : ''}${speakers.size} character${speakers.size !== 1 ? 's' : ''} · ${allSegments.length} segments`,
|
||||
audiobookShowPreview, audiobookCast, audiobookRecastUnknown);
|
||||
if (!stoppedEarly && !_audiobook.cancel && audiobookAutoRepairEnabled()) {
|
||||
const unknown = allSegments.filter(s => s?.type === 'dialogue' && (!s.speaker || /^Unknown|Unbekannt/i.test(s.speaker))).length;
|
||||
if (unknown > 10) setTimeout(() => { audiobookAutoRepairCast().catch(e => console.warn('[auto-repair]', e)); }, 800);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('[audiobookCastSpans] post-processing failed', e);
|
||||
view.done({ stopped: true, message: 'Finished casting, but saving/cleanup failed: ' + e.message });
|
||||
@ -169,6 +165,166 @@ async function audiobookCastSpans(overrideUrl, overrideModel, resume) {
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
// ── Verify / recast for span casts ──────────────────────────────────────────
|
||||
// Verify (the 2nd quality run) is ADVISORY: a second, differently-windowed pass over every dialogue
|
||||
// line. It never changes a speaker. Lines both passes agree on are marked `verified: 'agreed'`; where
|
||||
// they disagree the first answer is kept, the line is marked `disputed` and the other pass's answer is
|
||||
// stored as `altSpeaker` for the reviewer. Measured on a hand-corrected book: agreed lines were 82 %
|
||||
// right, contested first answers 67 %, and letting a reasoning pass overrule the first answer made those
|
||||
// lines worse (61 %), so nothing is overruled. Recast unknown (`unknownOnly`) re-asks only the Unknown
|
||||
// lines, with reasoning and fuller context; a line the model cannot place stays Unknown.
|
||||
const VERIFY_WINDOW = { targetChars: 5000, hardChars: 7000 };
|
||||
const VERIFY_CLUSTER_GAP = 10;
|
||||
|
||||
async function _abAsk(body, signal, timeoutMs) {
|
||||
const r = await audiobookFetchWithTimeout('/api/attribute-spans', {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' }, signal, body: JSON.stringify(body),
|
||||
}, timeoutMs + 5000);
|
||||
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).detail || ('HTTP ' + r.status));
|
||||
return r.json();
|
||||
}
|
||||
|
||||
function _abIsUnknownSpeaker(name) { return !name || /^(unknown|unbekannt)/i.test(String(name).trim()); }
|
||||
|
||||
async function audiobookVerifySpans(overrideUrl, overrideModel, options = {}) {
|
||||
const segs = _audiobook.segments || [];
|
||||
const text = _audiobook.lastText || '';
|
||||
const S = root.TTVC?.audiobook?.spans;
|
||||
if (!S || !text || !segs.some(s => s.spanId)) return false;
|
||||
const spans = S.segmentByQuotes(text);
|
||||
if (!spans) return false;
|
||||
const idOf = (sp) => (sp.kind === 'dialogue' ? 'D' : 'N') + sp.index;
|
||||
const posInSpans = new Map(spans.map((sp, k) => [idOf(sp), k]));
|
||||
const tagged = segs.filter(s => s.spanId);
|
||||
if (tagged.filter(s => posInSpans.has(s.spanId)).length < 0.95 * tagged.length) return false; // not this text
|
||||
if (_abBusyGuard('Verify speakers')) return true;
|
||||
|
||||
const unknownOnly = !!options.unknownOnly;
|
||||
const segById = new Map(tagged.map(s => [s.spanId, s]));
|
||||
const dialogueIds = spans.filter(sp => sp.kind === 'dialogue').map(idOf);
|
||||
const posInSegs = new Map(segs.map((s, k) => [s.spanId, k]));
|
||||
const rosterNow = () => [...new Set(segs.filter(s => s.type === 'dialogue' && !_abIsUnknownSpeaker(s.speaker)).map(s => s.speaker))];
|
||||
const recentBefore = (id) => {
|
||||
const out = [];
|
||||
for (let k = (posInSegs.get(id) ?? 0) - 1; k >= 0 && out.length < RECENT_LINES; k--) {
|
||||
const s = segs[k];
|
||||
if (s.type === 'dialogue' && !_abIsUnknownSpeaker(s.speaker)) out.unshift(`${s.speaker}: ${(s.text || '').slice(0, 80)}`);
|
||||
}
|
||||
return out.join('\n');
|
||||
};
|
||||
|
||||
_audiobook.running = true; _audiobook.cancel = false;
|
||||
if (typeof window.setNavCastingBadge === 'function') window.setNavCastingBadge(true);
|
||||
const ac = new AbortController(); _audiobook.abort = () => ac.abort();
|
||||
if (overrideUrl && typeof overrideUrl !== 'string') overrideUrl = null;
|
||||
const llmUrl = overrideUrl || audiobookLlmUrl();
|
||||
const model = audiobookSafeLlmModel(overrideModel || audiobookLlmModel());
|
||||
const language = audiobookLang(text);
|
||||
const llm = {
|
||||
url: document.getElementById('ab-cv-llm-url')?.value.trim() || llmUrl,
|
||||
model: audiobookSafeLlmModel(document.getElementById('ab-cv-llm-select')?.value || model),
|
||||
};
|
||||
const decisions = new Map(); // unknownOnly: span id -> speaker
|
||||
const flags = new Map(); // verify: span id -> { verified, alt }
|
||||
const stats = { checked: 0, agreed: 0, disputed: 0, resolvedUnknown: 0 };
|
||||
let finished = false;
|
||||
const clusters = (ids) => {
|
||||
const out = [];
|
||||
ids.forEach(id => {
|
||||
const pos = posInSpans.get(id);
|
||||
if (out.length && pos - out[out.length - 1].last <= VERIFY_CLUSTER_GAP) { out[out.length - 1].ids.push(id); out[out.length - 1].last = pos; }
|
||||
else out.push({ ids: [id], first: pos, last: pos });
|
||||
});
|
||||
return out;
|
||||
};
|
||||
const windows = unknownOnly ? [] : S.buildWindows(spans, text, VERIFY_WINDOW).filter(w => w.hasDialogue);
|
||||
const unknownIds = unknownOnly ? dialogueIds.filter(id => _abIsUnknownSpeaker(segById.get(id)?.speaker)) : [];
|
||||
const groups = unknownOnly ? clusters(unknownIds) : [];
|
||||
const view = audiobookCastView(Math.max(1, unknownOnly ? groups.length : windows.length), llmUrl, model);
|
||||
const asWindow = (lo, hi) => spans.slice(lo, hi).map(sp => ({ id: idOf(sp), kind: sp.kind, text: sp.text.length > 1500 ? sp.text.slice(0, 900) + ' [...] ' + sp.text.slice(-600) : sp.text }));
|
||||
|
||||
try {
|
||||
if (unknownOnly) {
|
||||
const pre = audiobookResolveUnknowns(segs, [], _audiobook.roster || []);
|
||||
if (pre.length) toast(`${pre.length} Unknown line${pre.length !== 1 ? 's' : ''} resolved by grammar rules`, 'success');
|
||||
if (!unknownIds.length) { toast('No Unknown speakers found', 'success'); view.done(); audiobookShowPreview(); return true; }
|
||||
for (let g = 0; g < groups.length && !_audiobook.cancel; g++) {
|
||||
const grp = groups[g];
|
||||
view.update(g, `Re-reading Unknown lines ${g + 1} / ${groups.length}…`);
|
||||
let ans;
|
||||
try { ans = await _abAsk({
|
||||
spans: asWindow(Math.max(0, grp.first - 6), Math.min(spans.length, grp.last + 3)), attribute_only: grp.ids,
|
||||
known_characters: rosterNow().slice(-ROSTER_LIMIT), recent: recentBefore(grp.ids[0]), language, want_reasoning: true,
|
||||
llm_url: llm.url, model: llm.model, timeout_seconds: audiobookTimeoutSeconds(AUDIOBOOK_RECAST_TIMEOUT_MS),
|
||||
}, ac.signal, AUDIOBOOK_RECAST_TIMEOUT_MS); }
|
||||
catch (err) { if (err.name === 'AbortError') throw err; view.note(`⚠️ Unknown lines ${g + 1}: ${err.message}`); continue; }
|
||||
for (const id of grp.ids) {
|
||||
const hit = ans.speakers && ans.speakers[id];
|
||||
if (hit && !_abIsUnknownSpeaker(hit.speaker) && hit.speaker !== 'Narrator') decisions.set(id, hit.speaker);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for (let i = 0; i < windows.length && !_audiobook.cancel; i++) {
|
||||
view.update(i, `Second pass ${i + 1} / ${windows.length}…`);
|
||||
const win = windows[i].spans;
|
||||
let ans;
|
||||
try { ans = await _abAsk({
|
||||
spans: win.map(sp => ({ id: sp.id, kind: sp.kind, text: sp.llmText })), known_characters: rosterNow().slice(-ROSTER_LIMIT),
|
||||
recent: recentBefore(win.find(sp => sp.kind === 'dialogue').id), language, llm_url: llm.url, model: llm.model,
|
||||
timeout_seconds: audiobookTimeoutSeconds(AUDIOBOOK_ATTRIBUTION_TIMEOUT_MS),
|
||||
}, ac.signal, AUDIOBOOK_ATTRIBUTION_TIMEOUT_MS); }
|
||||
catch (err) { if (err.name === 'AbortError') throw err; view.note(`⚠️ Second pass ${i + 1}: ${err.message}`); continue; }
|
||||
for (const [id, v] of Object.entries(ans.speakers || {})) {
|
||||
const first = segById.get(id)?.speaker, other = v.speaker;
|
||||
if (!segById.has(id) || other === 'Narrator' || _abIsUnknownSpeaker(other)) continue; // no usable second opinion
|
||||
stats.checked++;
|
||||
if (_abIsUnknownSpeaker(first)) flags.set(id, { verified: 'unresolved', alt: other });
|
||||
else if (first === other) { flags.set(id, { verified: 'agreed' }); stats.agreed++; }
|
||||
else { flags.set(id, { verified: 'disputed', alt: other }); stats.disputed++; }
|
||||
}
|
||||
}
|
||||
}
|
||||
finished = !_audiobook.cancel;
|
||||
} catch (err) {
|
||||
if (err.name !== 'AbortError') view.note('Error while verifying: ' + err.message);
|
||||
} finally {
|
||||
_audiobook.running = false; _audiobook.abort = null; _abHideCastingControl();
|
||||
}
|
||||
|
||||
if (!finished) {
|
||||
view.done({ stopped: true, message: 'Verification stopped. Existing cast preserved.' });
|
||||
toast('Verification stopped. Existing cast preserved.', 'info');
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
decisions.forEach((speaker, id) => {
|
||||
const seg = segById.get(id);
|
||||
if (!seg || !_abIsUnknownSpeaker(seg.speaker)) return;
|
||||
seg.speaker = speaker; stats.resolvedUnknown++;
|
||||
if (!_audiobook.roster.includes(speaker)) _audiobook.roster.push(speaker);
|
||||
});
|
||||
flags.forEach((f, id) => { const seg = segById.get(id); if (seg) { seg.verified = f.verified; if (f.alt) seg.altSpeaker = f.alt; else delete seg.altSpeaker; } });
|
||||
{ const ph = _audiobookDropPhantomSpeakers(segs); if (ph.dropped) console.info('[verify] dropped', ph.dropped, 'phantom speaker line(s)'); }
|
||||
_audiobookResolveRoleAliases(segs, text);
|
||||
_audiobookConsolidateSpeakerAliases(segs);
|
||||
_abStampConfidence(segs);
|
||||
_abSaveDraft(segs, _audiobook.roster || [], text, _audiobook.completedChunks || segs.length, _audiobook.completedTotal || _audiobook.completedChunks || segs.length);
|
||||
audiobookSaveAsRehearsal({ silent: true });
|
||||
const summary = unknownOnly
|
||||
? `${stats.resolvedUnknown} Unknown line${stats.resolvedUnknown !== 1 ? 's' : ''} resolved`
|
||||
: `${stats.checked} lines read twice · ${stats.agreed} confirmed, ${stats.disputed} flagged for review`;
|
||||
view.complete(summary, audiobookShowPreview, audiobookCast, audiobookRecastUnknown);
|
||||
toast(summary, 'success');
|
||||
} catch (e) {
|
||||
console.error('[audiobookVerifySpans] apply failed', e);
|
||||
view.done({ stopped: true, message: 'Verification finished, but applying it failed: ' + e.message });
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
root.audiobookVerifySpans = audiobookVerifySpans;
|
||||
|
||||
root.audiobookSpanModeEnabled = audiobookSpanModeEnabled;
|
||||
root.audiobookCastSpans = audiobookCastSpans;
|
||||
root.TTVC = root.TTVC || {};
|
||||
|
||||
@ -20,6 +20,9 @@ function _abBusyGuard(what) {
|
||||
|
||||
async function audiobookRecastUnknown(overrideUrl, overrideModel, options = {}) {
|
||||
if (_abBusyGuard('Recast / verify')) return;
|
||||
// A span-mode cast keeps its text exactly; every repair/verify entry point re-asks only who speaks.
|
||||
if (typeof root.audiobookVerifySpans === 'function' && root.audiobookSpanModeEnabled()
|
||||
&& await root.audiobookVerifySpans(overrideUrl, overrideModel, { unknownOnly: !options.includeNarrator })) return;
|
||||
const segs = _audiobook.segments;
|
||||
if (!segs || !segs.length) return;
|
||||
const originalSegments = segs.map(s => ({...s}));
|
||||
|
||||
@ -1082,6 +1082,10 @@ function _abSegmentConfidence(segs, i) {
|
||||
if (!/[»«]/.test(t)) { score -= 14; why.push('no quote marks'); }
|
||||
if (/\n\s*\n/.test(t.trim())) { score -= 22; why.push('fused: several turns in one segment'); }
|
||||
if (t.trim().length < 12) { score -= 10; why.push('very short line'); }
|
||||
// Span-cast verification: two independently windowed passes agreeing is strong evidence
|
||||
// (82 % right where they agreed vs 67 % where they did not, measured on a hand-corrected book).
|
||||
if (s.verified === 'agreed') { score += 10; why.push('two passes agree'); }
|
||||
else if (s.verified === 'disputed') { score -= 14; why.push('two passes disagree' + (s.altSpeaker ? ' (other pass: ' + s.altSpeaker + ')' : '')); }
|
||||
return { score: Math.max(2, Math.min(99, score)), why };
|
||||
}
|
||||
|
||||
|
||||
@ -31,3 +31,101 @@ test('span mode is on by default and can be switched off per browser', async ({
|
||||
expect(await page.evaluate(() => window.audiobookSpanModeEnabled())).toBe(false);
|
||||
await page.evaluate(() => localStorage.removeItem('ab-span-mode'));
|
||||
});
|
||||
|
||||
// ── Verify / recast for span casts ───────────────────────────────────────────
|
||||
// A cast made in span mode is verified by a second pass plus candidate-restricted adjudication; these
|
||||
// tests drive audiobookVerifySpans against a mocked /api/attribute-spans.
|
||||
const FILLER = Array.from({ length: 6 }, (_, i) => `»Zeile ${i} ist ein Satz.« sagte er.`).join(' ');
|
||||
const BOOK = `${FILLER} Der Zwerg schneuzte sich. »Na los!« sagte Linosch. »Vielen Dank!« Er ging. »Wirklich?« fragte jemand. »Ist das so?« Stille.`;
|
||||
|
||||
async function seedCast(page, firstPass, book = BOOK) {
|
||||
await page.evaluate(([book, first]) => {
|
||||
const spans = window.TTVC.audiobook.spans.segmentByQuotes(book);
|
||||
window._audiobook.lastText = book;
|
||||
window._audiobook.roster = ['Marcian', 'Linosch', 'Alrik'];
|
||||
window._audiobook.segments = spans.map((sp) => {
|
||||
const id = (sp.kind === 'dialogue' ? 'D' : 'N') + sp.index;
|
||||
return sp.kind === 'dialogue'
|
||||
? { speaker: first[sp.text] || 'Marcian', type: 'dialogue', text: sp.text, emotion: '', spanId: id }
|
||||
: { speaker: 'Narrator', type: 'narration', text: sp.text, emotion: '', spanId: id };
|
||||
});
|
||||
}, [book, firstPass]);
|
||||
}
|
||||
|
||||
async function mockSpanEndpoint(page, plan) {
|
||||
const calls = [];
|
||||
await page.route('**/api/attribute-spans', route => {
|
||||
const body = route.request().postDataJSON();
|
||||
calls.push({ reasoning: !!body.want_reasoning, only: body.attribute_only || null, ids: body.spans.filter(s => s.kind === 'dialogue').map(s => s.id) });
|
||||
const table = body.want_reasoning ? plan.reasoning : plan.second;
|
||||
const speakers = {};
|
||||
for (const s of body.spans) {
|
||||
if (s.kind !== 'dialogue') continue;
|
||||
if (body.attribute_only && !body.attribute_only.includes(s.id)) continue;
|
||||
const name = table[s.text] ?? (body.want_reasoning ? undefined : 'Marcian'); // the second pass answers every line
|
||||
if (name !== undefined) speakers[s.id] = { speaker: name, emotion: '' };
|
||||
}
|
||||
return route.fulfill({ json: { speakers, missing: [], characters: [] } });
|
||||
});
|
||||
return calls;
|
||||
}
|
||||
|
||||
test('verify is advisory: it marks agreement and disputes but never changes a speaker', async ({ page }) => {
|
||||
await openReader(page);
|
||||
await seedCast(page, { 'Na los!': 'Linosch', 'Vielen Dank!': 'Alrik', 'Wirklich?': 'Unknown', 'Ist das so?': 'Alrik' });
|
||||
const calls = await mockSpanEndpoint(page, { second: { 'Na los!': 'Linosch', 'Vielen Dank!': 'Marcian', 'Wirklich?': 'Alrik', 'Ist das so?': 'Linosch' }, reasoning: {} });
|
||||
const handled = await page.evaluate(() => window.audiobookVerifySpans(undefined, undefined, {}));
|
||||
expect(handled).toBe(true);
|
||||
const rows = await page.evaluate(() => window._audiobook.segments.filter(s => s.type === 'dialogue')
|
||||
.map(s => [s.text, s.speaker, s.verified, s.altSpeaker || null, s.confWhy]));
|
||||
const by = Object.fromEntries(rows.map(([t, sp, v, alt, why]) => [t, { sp, v, alt, why }]));
|
||||
expect(by['Na los!']).toMatchObject({ sp: 'Linosch', v: 'agreed' });
|
||||
expect(by['Vielen Dank!']).toMatchObject({ sp: 'Alrik', v: 'disputed', alt: 'Marcian' }); // first answer kept
|
||||
expect(by['Ist das so?']).toMatchObject({ sp: 'Alrik', v: 'disputed', alt: 'Linosch' });
|
||||
expect(by['Wirklich?']).toMatchObject({ sp: 'Unknown', v: 'unresolved', alt: 'Alrik' }); // never filled in silently
|
||||
expect(by['Zeile 0 ist ein Satz.'].v).toBe('agreed');
|
||||
expect(by['Vielen Dank!'].why).toContain('two passes disagree (other pass: Marcian)');
|
||||
expect(by['Na los!'].why).toContain('two passes agree');
|
||||
// a single no-thinking pass: no reasoning calls, no per-line restriction
|
||||
expect(calls.length).toBeGreaterThan(0);
|
||||
expect(calls.every(c => !c.reasoning && !c.only)).toBe(true);
|
||||
// the text is untouched, every span still present once
|
||||
const text = await page.evaluate(() => window._audiobook.segments.map(s => s.text).join(' '));
|
||||
expect(text.replace(/[»«]/g, '').replace(/\s+/g, ' ')).toBe(BOOK.replace(/[»«]/g, '').replace(/\s+/g, ' '));
|
||||
});
|
||||
|
||||
test('recast unknown on a span cast only re-asks the Unknown lines, with reasoning', async ({ page }) => {
|
||||
await openReader(page);
|
||||
// Untagged lines before any speaker is known: the grammar pre-pass cannot settle them, so the model is asked.
|
||||
const book = `»Wer ist da?« Wind. »Niemand.« Nacht. ${FILLER}`;
|
||||
await seedCast(page, { 'Wer ist da?': 'Unknown', 'Niemand.': 'Unknown' }, book);
|
||||
const calls = await mockSpanEndpoint(page, { second: {}, reasoning: { 'Wer ist da?': 'Alrik', 'Niemand.': 'Unknown' } });
|
||||
await page.evaluate(() => window.audiobookRecastUnknown(undefined, undefined, {}));
|
||||
await expect.poll(async () => page.evaluate(() => window._audiobook.segments.find(s => s.text === 'Wer ist da?').speaker)).toBe('Alrik');
|
||||
const speakers = await page.evaluate(() => Object.fromEntries(window._audiobook.segments.filter(s => s.type === 'dialogue').map(s => [s.text, s.speaker])));
|
||||
expect(speakers['Niemand.']).toBe('Unknown'); // still unresolved: left alone, never invented
|
||||
expect(speakers['Zeile 0 ist ein Satz.']).toBe('Marcian'); // known lines are not re-litigated
|
||||
expect(calls.length).toBeGreaterThan(0);
|
||||
expect(calls.every(c => c.reasoning)).toBe(true);
|
||||
// the two Unknown lines were the only ones ever asked about
|
||||
const asked = new Set(calls.flatMap(c => c.only || []));
|
||||
const unknownIds = await page.evaluate(() => window._audiobook.segments.filter(s => s.text === 'Wer ist da?' || s.text === 'Niemand.').map(s => s.spanId));
|
||||
expect(asked.size).toBeGreaterThan(0);
|
||||
expect([...asked].every(id => unknownIds.includes(id))).toBe(true);
|
||||
});
|
||||
|
||||
test('verify leaves the cast untouched when the endpoint fails, and declines a cast it cannot map to the text', async ({ page }) => {
|
||||
await openReader(page);
|
||||
await seedCast(page, { 'Na los!': 'Linosch', 'Vielen Dank!': 'Alrik' });
|
||||
const before = await page.evaluate(() => JSON.stringify(window._audiobook.segments.map(s => [s.spanId, s.speaker])));
|
||||
await page.route('**/api/attribute-spans', route => route.fulfill({ status: 503, json: { detail: 'down' } }));
|
||||
await page.evaluate(() => window.audiobookVerifySpans(undefined, undefined, {}));
|
||||
const after = await page.evaluate(() => JSON.stringify(window._audiobook.segments.map(s => [s.spanId, s.speaker])));
|
||||
expect(after).toBe(before);
|
||||
// a cast from a different text (or a classic cast without span ids) is declined so the classic path can run
|
||||
const declined = await page.evaluate(() => {
|
||||
window._audiobook.segments = [{ speaker: 'X', type: 'dialogue', text: 'Hallo', emotion: '' }];
|
||||
return window.audiobookVerifySpans(undefined, undefined, {});
|
||||
});
|
||||
expect(declined).toBe(false);
|
||||
});
|
||||
|
||||
@ -12,7 +12,7 @@ from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from core.attribution_spans import (
|
||||
SpanRequestError, build_messages, max_tokens_for, parse_response, validate_spans,
|
||||
SpanRequestError, build_messages, max_tokens_for, parse_response, validate_only_ids, validate_spans,
|
||||
)
|
||||
import routes.conversation as conversation
|
||||
|
||||
@ -67,6 +67,18 @@ class PromptTests(unittest.TestCase):
|
||||
self.assertLessEqual(max_tokens_for(many(400)), 4096)
|
||||
|
||||
|
||||
class AttributeOnlyTests(unittest.TestCase):
|
||||
def test_attribute_only_keeps_valid_dialogue_ids_in_order(self):
|
||||
spans = validate_spans(SPANS)
|
||||
self.assertEqual(validate_only_ids(["D3", "D1", "D3", "N0", "D9", 5], spans), ["D3", "D1"])
|
||||
self.assertEqual(validate_only_ids("D1", spans), [])
|
||||
|
||||
def test_prompt_restricts_the_request_to_those_lines(self):
|
||||
_, user = build_messages(SPANS, only_ids=["D1"])
|
||||
self.assertIn("Attribute these lines: D1", user)
|
||||
self.assertNotIn("D3", user.split("Attribute these lines:")[1])
|
||||
|
||||
|
||||
class ParseTests(unittest.TestCase):
|
||||
def test_parses_clean_json_and_normalises_names(self):
|
||||
raw = '{"speakers":[{"id":"D1","speaker":" Linosch ","emotion":"gereizt"},{"id":"D3","speaker":"unbekannt","emotion":""}]}'
|
||||
@ -132,6 +144,16 @@ class RouteTests(unittest.TestCase):
|
||||
retry_prompt = llm.call_args_list[1].args[0]["messages"][1]["content"]
|
||||
self.assertIn("Attribute these lines: D3", retry_prompt)
|
||||
|
||||
def test_attribute_only_limits_the_request_and_the_missing_accounting(self):
|
||||
answer = '{"speakers":[{"id":"D3","speaker":"Alrik","emotion":""}]}'
|
||||
with mock.patch.object(conversation, "_span_llm_answer", return_value=answer) as llm:
|
||||
r = self.client.post("/api/attribute-spans", json={"spans": SPANS, "attribute_only": ["D3"]})
|
||||
body = r.json()
|
||||
self.assertEqual(body["speakers"]["D3"]["speaker"], "Alrik")
|
||||
self.assertEqual(body["missing"], [])
|
||||
llm.assert_called_once()
|
||||
self.assertIn("Attribute these lines: D3", llm.call_args.args[0]["messages"][1]["content"])
|
||||
|
||||
def test_reports_failure_instead_of_erroring_when_the_llm_is_down(self):
|
||||
with mock.patch.object(conversation, "_span_llm_answer", side_effect=RuntimeError("LLM HTTP 503")):
|
||||
r = self.client.post("/api/attribute-spans", json={"spans": SPANS})
|
||||
|
||||
Loading…
Reference in New Issue
Block a user