diff --git a/CHANGELOG.md b/CHANGELOG.md index 4247423..4a8504d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ### Changed - Auto-repair no longer runs after a span-mode cast: Unknown lines stay visible for review instead of being guessed at. +- **Span-cast confidence is calibrated on agreement between the two passes** instead of the old heuristics: agreed 82 % (Likely), disputed 48 % (Doubtful), Unknown 5 % (Guess), not yet verified 76 % (Probably) — pooled over three novels (~54 000 spoken words, 2 485 lines with a second opinion). The old bands claimed 16 % for lines that were 62 % right and the tag/role-label signals did not discriminate in span casts. Classic casts keep the old model. Running Verify therefore shrinks the "below 80 %" review queue to the disputed and Unknown lines. ### Measured — and rejected - On the real pipeline (Book 1, in-app), letting a reasoning question restricted to the two candidates overrule disputed lines made them **worse** (66.5 % → 61.4 % on 3 203 words), and the second pass "resolving" Unknown lines was ~3 % right, so no adjudication or silent filling exists. Lines both passes agreed on were 82 % right versus ~67 % for disputed first answers — that agreement is the signal used. An earlier offline experiment had suggested the opposite; it used a biased baseline (see `architecture/audiobook-attribution.md`). diff --git a/Manual.md b/Manual.md index 8e49723..a2e54a1 100644 --- a/Manual.md +++ b/Manual.md @@ -35,8 +35,10 @@ the Rehearser and MP3 export. second time with different passage boundaries — about as long as the first cast — and **never changes a speaker**: lines both readings agree on are marked confirmed (about 4 in 5 of those were right in testing), and lines where they disagree keep the first - reading, are flagged in the review list and show what the other reading said. Use the - flags to decide where to spend your review time. Neither step can change the book's + reading, are flagged in the review list and show what the other reading said. The + percentage next to each line means what it says in testing: confirmed lines were about + 82 % right, disputed lines about 48 %, lines not yet verified about 76 %. Use the flags + to decide where to spend your review time. Neither step can change the book's text, and a stopped or failed run leaves the cast as it was. **Expected result:** every word of the book appears exactly once, quoted speech is diff --git a/architecture/audiobook-attribution.md b/architecture/audiobook-attribution.md index 72fc880..441bb2f 100644 --- a/architecture/audiobook-attribution.md +++ b/architecture/audiobook-attribution.md @@ -99,13 +99,24 @@ not (classic cast, or a different text) is declined and the classic path runs. changes a speaker**. Lines both passes agree on get `verified: 'agreed'`; where they disagree the first answer is kept, the line gets `verified: 'disputed'` and the other pass's answer is stored as `altSpeaker` (shown in the confidence explanation); a line the first pass left Unknown stays Unknown - (`unresolved`, with the other pass's suggestion). Confidence: agreed +10, disputed −14. Nothing is + (`unresolved`, with the other pass's suggestion). Nothing is applied until the pass finished, so a stopped or failed run leaves the cast unchanged. * **Recast unknown** (`unknownOnly`): the grammar pre-pass first, then one reasoning question per cluster of remaining Unknown lines (fuller context, `attribute_only` so the rest of the window is context). A line the model still cannot place stays Unknown — never invented. Auto-repair after a span cast is off: Unknown lines stay visible for review. +### Confidence for span casts + +The displayed confidence of a span-cast dialogue line comes from agreement alone +(`_abSpanConfidence` in `audiobook.js`), calibrated on three hand-corrected novels (~54 000 spoken +words, alias-tolerant): **agreed 82 %** (2 013 lines), **disputed 48 %** (472 lines), **Unknown ~0 %**, +first pass not yet verified **~76 %**. The old heuristic model (speech-tag evidence, role labels, quote +marks) is kept only for classic casts: in span casts it did not discriminate (lines whose speech tag names +the speaker were 83 % right versus 84 % without) and its calibrated bands were far too pessimistic (a band +displayed as 16 % was 62 % right). Unverified lines sit just under the 80 % review threshold on purpose: +running Verify shrinks the review queue to the disputed and Unknown lines. + ### What was measured, and what was rejected Book 1, alias-tolerant, real in-app runs on the KONTROLLE document: diff --git a/static/js/audiobook.js b/static/js/audiobook.js index 6c730f2..213716e 100644 --- a/static/js/audiobook.js +++ b/static/js/audiobook.js @@ -1082,10 +1082,6 @@ function _abSegmentConfidence(segs, i) { if (!/[»«]/.test(t)) { score -= 14; why.push('no quote marks'); } if (/\n\s*\n/.test(t.trim())) { score -= 22; why.push('fused: several turns in one segment'); } if (t.trim().length < 12) { score -= 10; why.push('very short line'); } - // Span-cast verification: two independently windowed passes agreeing is strong evidence - // (82 % right where they agreed vs 67 % where they did not, measured on a hand-corrected book). - if (s.verified === 'agreed') { score += 10; why.push('two passes agree'); } - else if (s.verified === 'disputed') { score -= 14; why.push('two passes disagree' + (s.altSpeaker ? ' (other pass: ' + s.altSpeaker + ')' : '')); } return { score: Math.max(2, Math.min(99, score)), why }; } @@ -1148,8 +1144,25 @@ function _abConfBadgeHtml(s) { // Stamp every segment with its confidence so the UI can colour-code and the // "next uncertain" navigation can skip what is already trustworthy. +// Span casts (segments carry a spanId) are calibrated on agreement between two independent passes, +// the only signal that separated right from wrong in measurement: pooled over three hand-corrected +// novels (~54k spoken words), lines both passes agreed on were 82 % right, disputed ones 48 %, Unknown +// ones ~0 %, and a first pass nobody has verified ~76 %. The old speech-tag / role-label heuristics +// did not discriminate there (tag-named lines 83 % vs 84 % without), so they are not used. +function _abSpanConfidence(s) { + if (!s.speaker || /^(unknown|unbekannt)/i.test(String(s.speaker))) return { pct: 5, label: 'Guess', why: 'speaker unknown' }; + if (s.verified === 'agreed') return { pct: 82, label: 'Likely', why: 'two passes agree' }; + if (s.verified === 'disputed') return { pct: 48, label: 'Doubtful', why: 'two passes disagree' + (s.altSpeaker ? ' (other pass: ' + s.altSpeaker + ')' : '') }; + return { pct: 76, label: 'Probably', why: 'not verified yet — run Verify to check this line' }; +} + function _abStampConfidence(segs) { for (let i = 0; i < segs.length; i++) { + if (segs[i].spanId && segs[i].type === 'dialogue') { + const c = _abSpanConfidence(segs[i]); + segs[i].conf = c.pct; segs[i].confLabel = c.label; segs[i].confRaw = c.pct; segs[i].confWhy = c.why; + continue; + } const { score, why } = _abSegmentConfidence(segs, i); const band = _abCalibrateConfidence(score); // Store the CALIBRATED percentage as `conf`, so every threshold in the UI diff --git a/tests/audiobook-spans.spec.js b/tests/audiobook-spans.spec.js index d6aa4d4..00b6fd4 100644 --- a/tests/audiobook-spans.spec.js +++ b/tests/audiobook-spans.spec.js @@ -86,6 +86,12 @@ test('verify is advisory: it marks agreement and disputes but never changes a sp expect(by['Zeile 0 ist ein Satz.'].v).toBe('agreed'); expect(by['Vielen Dank!'].why).toContain('two passes disagree (other pass: Marcian)'); expect(by['Na los!'].why).toContain('two passes agree'); + // span-cast confidence is calibrated on agreement (pooled measurement), not on the old heuristics + const conf = await page.evaluate(() => Object.fromEntries(window._audiobook.segments.filter(s => s.type === 'dialogue') + .map(s => [s.text, [s.conf, s.confLabel]]))); + expect(conf['Na los!']).toEqual([82, 'Likely']); + expect(conf['Vielen Dank!']).toEqual([48, 'Doubtful']); + expect(conf['Wirklich?']).toEqual([5, 'Guess']); // a single no-thinking pass: no reasoning calls, no per-line restriction expect(calls.length).toBeGreaterThan(0); expect(calls.every(c => !c.reasoning && !c.only)).toBe(true); @@ -129,3 +135,15 @@ test('verify leaves the cast untouched when the endpoint fails, and declines a c }); expect(declined).toBe(false); }); + +test('an unverified span cast is labelled as such, and a classic cast keeps the old confidence model', async ({ page }) => { + await openReader(page); + const out = await page.evaluate(() => { + const span = [{ speaker: 'Marcian', type: 'dialogue', text: '»Ja.«', emotion: '', spanId: 'D1' }]; + const classic = [{ speaker: 'Marcian', type: 'dialogue', text: '»Ja.«', emotion: '' }]; + window._abStampConfidence(span); window._abStampConfidence(classic); + return { span: [span[0].conf, span[0].confLabel, span[0].confWhy], classic: [typeof classic[0].conf, classic[0].confWhy.includes('speech tag')] }; + }); + expect(out.span).toEqual([76, 'Probably', 'not verified yet — run Verify to check this line']); + expect(out.classic[0]).toBe('number'); +});