feat(audiobook): calibrate span-cast confidence on pass agreement

Pooled over three hand-corrected novels (~54k spoken words): agreed 82% right, disputed 48%,
Unknown ~0%, unverified ~76%. The old heuristic bands claimed 16% for lines that were 62% right
and tag/role signals did not discriminate in span casts; classic casts keep the old model.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -12,6 +12,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
 
 ### Changed
 - Auto-repair no longer runs after a span-mode cast: Unknown lines stay visible for review instead of being guessed at.
+- **Span-cast confidence is calibrated on agreement between the two passes** instead of the old heuristics: agreed 82 % (Likely), disputed 48 % (Doubtful), Unknown 5 % (Guess), not yet verified 76 % (Probably) — pooled over three novels (~54 000 spoken words, 2 485 lines with a second opinion). The old bands claimed 16 % for lines that were 62 % right and the tag/role-label signals did not discriminate in span casts. Classic casts keep the old model. Running Verify therefore shrinks the "below 80 %" review queue to the disputed and Unknown lines.
 
 ### Measured — and rejected
 - On the real pipeline (Book 1, in-app), letting a reasoning question restricted to the two candidates overrule disputed lines made them **worse** (66.5 % → 61.4 % on 3 203 words), and the second pass "resolving" Unknown lines was ~3 % right, so no adjudication or silent filling exists. Lines both passes agreed on were 82 % right versus ~67 % for disputed first answers — that agreement is the signal used. An earlier offline experiment had suggested the opposite; it used a biased baseline (see `architecture/audiobook-attribution.md`).

Manual.md

@@ -35,8 +35,10 @@ the Rehearser and MP3 export.
    second time with different passage boundaries — about as long as the first cast — and
    **never changes a speaker**: lines both readings agree on are marked confirmed (about
    4 in 5 of those were right in testing), and lines where they disagree keep the first
-   reading, are flagged in the review list and show what the other reading said. Use the
+   reading, are flagged in the review list and show what the other reading said. The
-   flags to decide where to spend your review time. Neither step can change the book's
+   percentage next to each line means what it says in testing: confirmed lines were about
+   82 % right, disputed lines about 48 %, lines not yet verified about 76 %. Use the flags
+   to decide where to spend your review time. Neither step can change the book's
    text, and a stopped or failed run leaves the cast as it was.
 
 **Expected result:** every word of the book appears exactly once, quoted speech is

architecture/audiobook-attribution.md

@@ -99,13 +99,24 @@ not (classic cast, or a different text) is declined and the classic path runs.
   changes a speaker**. Lines both passes agree on get `verified: 'agreed'`; where they disagree the
   first answer is kept, the line gets `verified: 'disputed'` and the other pass's answer is stored as
   `altSpeaker` (shown in the confidence explanation); a line the first pass left Unknown stays Unknown
-  (`unresolved`, with the other pass's suggestion). Confidence: agreed +10, disputed −14. Nothing is
+  (`unresolved`, with the other pass's suggestion). Nothing is
   applied until the pass finished, so a stopped or failed run leaves the cast unchanged.
 * **Recast unknown** (`unknownOnly`): the grammar pre-pass first, then one reasoning question per
   cluster of remaining Unknown lines (fuller context, `attribute_only` so the rest of the window is
   context). A line the model still cannot place stays Unknown — never invented. Auto-repair after a
   span cast is off: Unknown lines stay visible for review.
 
+### Confidence for span casts
+
+The displayed confidence of a span-cast dialogue line comes from agreement alone
+(`_abSpanConfidence` in `audiobook.js`), calibrated on three hand-corrected novels (~54 000 spoken
+words, alias-tolerant): **agreed 82 %** (2 013 lines), **disputed 48 %** (472 lines), **Unknown ~0 %**,
+first pass not yet verified **~76 %**. The old heuristic model (speech-tag evidence, role labels, quote
+marks) is kept only for classic casts: in span casts it did not discriminate (lines whose speech tag names
+the speaker were 83 % right versus 84 % without) and its calibrated bands were far too pessimistic (a band
+displayed as 16 % was 62 % right). Unverified lines sit just under the 80 % review threshold on purpose:
+running Verify shrinks the review queue to the disputed and Unknown lines.
+
 ### What was measured, and what was rejected
 
 Book 1, alias-tolerant, real in-app runs on the KONTROLLE document:

static/js/audiobook.js

@@ -1082,10 +1082,6 @@ function _abSegmentConfidence(segs, i) {
   if (!/[»«]/.test(t)) { score -= 14; why.push('no quote marks'); }
   if (/\n\s*\n/.test(t.trim())) { score -= 22; why.push('fused: several turns in one segment'); }
   if (t.trim().length < 12) { score -= 10; why.push('very short line'); }
-  // Span-cast verification: two independently windowed passes agreeing is strong evidence
-  // (82 % right where they agreed vs 67 % where they did not, measured on a hand-corrected book).
-  if (s.verified === 'agreed') { score += 10; why.push('two passes agree'); }
-  else if (s.verified === 'disputed') { score -= 14; why.push('two passes disagree' + (s.altSpeaker ? ' (other pass: ' + s.altSpeaker + ')' : '')); }
   return { score: Math.max(2, Math.min(99, score)), why };
 }
 
@@ -1148,8 +1144,25 @@ function _abConfBadgeHtml(s) {
 
 // Stamp every segment with its confidence so the UI can colour-code and the
 // "next uncertain" navigation can skip what is already trustworthy.
+// Span casts (segments carry a spanId) are calibrated on agreement between two independent passes,
+// the only signal that separated right from wrong in measurement: pooled over three hand-corrected
+// novels (~54k spoken words), lines both passes agreed on were 82 % right, disputed ones 48 %, Unknown
+// ones ~0 %, and a first pass nobody has verified ~76 %. The old speech-tag / role-label heuristics
+// did not discriminate there (tag-named lines 83 % vs 84 % without), so they are not used.
+function _abSpanConfidence(s) {
+  if (!s.speaker || /^(unknown|unbekannt)/i.test(String(s.speaker))) return { pct: 5, label: 'Guess', why: 'speaker unknown' };
+  if (s.verified === 'agreed') return { pct: 82, label: 'Likely', why: 'two passes agree' };
+  if (s.verified === 'disputed') return { pct: 48, label: 'Doubtful', why: 'two passes disagree' + (s.altSpeaker ? ' (other pass: ' + s.altSpeaker + ')' : '') };
+  return { pct: 76, label: 'Probably', why: 'not verified yet — run Verify to check this line' };
+}
+
 function _abStampConfidence(segs) {
   for (let i = 0; i < segs.length; i++) {
+    if (segs[i].spanId && segs[i].type === 'dialogue') {
+      const c = _abSpanConfidence(segs[i]);
+      segs[i].conf = c.pct; segs[i].confLabel = c.label; segs[i].confRaw = c.pct; segs[i].confWhy = c.why;
+      continue;
+    }
     const { score, why } = _abSegmentConfidence(segs, i);
     const band = _abCalibrateConfidence(score);
     // Store the CALIBRATED percentage as `conf`, so every threshold in the UI

tests/audiobook-spans.spec.js

@@ -86,6 +86,12 @@ test('verify is advisory: it marks agreement and disputes but never changes a sp
   expect(by['Zeile 0 ist ein Satz.'].v).toBe('agreed');
   expect(by['Vielen Dank!'].why).toContain('two passes disagree (other pass: Marcian)');
   expect(by['Na los!'].why).toContain('two passes agree');
+  // span-cast confidence is calibrated on agreement (pooled measurement), not on the old heuristics
+  const conf = await page.evaluate(() => Object.fromEntries(window._audiobook.segments.filter(s => s.type === 'dialogue')
+    .map(s => [s.text, [s.conf, s.confLabel]])));
+  expect(conf['Na los!']).toEqual([82, 'Likely']);
+  expect(conf['Vielen Dank!']).toEqual([48, 'Doubtful']);
+  expect(conf['Wirklich?']).toEqual([5, 'Guess']);
   // a single no-thinking pass: no reasoning calls, no per-line restriction
   expect(calls.length).toBeGreaterThan(0);
   expect(calls.every(c => !c.reasoning && !c.only)).toBe(true);
@@ -129,3 +135,15 @@ test('verify leaves the cast untouched when the endpoint fails, and declines a c
   });
   expect(declined).toBe(false);
 });
+
+test('an unverified span cast is labelled as such, and a classic cast keeps the old confidence model', async ({ page }) => {
+  await openReader(page);
+  const out = await page.evaluate(() => {
+    const span = [{ speaker: 'Marcian', type: 'dialogue', text: '»Ja.«', emotion: '', spanId: 'D1' }];
+    const classic = [{ speaker: 'Marcian', type: 'dialogue', text: '»Ja.«', emotion: '' }];
+    window._abStampConfidence(span); window._abStampConfidence(classic);
+    return { span: [span[0].conf, span[0].confLabel, span[0].confWhy], classic: [typeof classic[0].conf, classic[0].confWhy.includes('speech tag')] };
+  });
+  expect(out.span).toEqual([76, 'Probably', 'not verified yet — run Verify to check this line']);
+  expect(out.classic[0]).toBe('number');
+});