docs(magazine): record D-E re-run results with both G fixes applied

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-09-29 09:18:11 +02:00
parent ac1153b782
commit 6f2dfca89a
3 changed files with 6546 additions and 4 deletions

23
TODO.md
View File

@ -120,10 +120,25 @@ rules beat what was built up by hand, and to feed every difference back into the
backup itself is not clean either -- its next Auto-fix/quality run will self-heal now). backup itself is not clean either -- its next Auto-fix/quality run will self-heal now).
Regression tests: `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`, Regression tests: `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`,
`tests/test_magazine_routes.py::QualityRunJobTests::test_a_duplicate_created_by_recovering_missing_text_is_still_caught`, `tests/test_magazine_routes.py::QualityRunJobTests::test_a_duplicate_created_by_recovering_missing_text_is_still_caught`,
`tests/test_magazine_routes.py::IngestSyncDuplicateRecheckTests`. **Still open:** re-run D→E with both `tests/test_magazine_routes.py::IngestSyncDuplicateRecheckTests`.
fixes in place and see whether the gap to the backup narrows; the remaining worst-page cases not yet
root-caused (e.g. page 44's screenshot-in-article OCR) need a closer look before deciding whether they are **D→E re-run with both fixes, 2026-09-29** (re-running Auto-fix on the same clone; report at
general bugs or page-specific noise not worth a rule for. `architecture/evidence/compare-345b2a84-vs-backup-20260929-after-fixes.{md,json}`): pages needing a
decision 60→53, overlapping pairs 188→14, clipped edges 127→96 -- real improvement. Narratable word
count also dropped slightly (68680→64946), consistent with duplicate text no longer being counted twice.
Caveat: this re-run's own "re-reading suspicious words" step calls the vision engine again on flagged
boxes, and that engine is not deterministic -- pages 82 and 87 got *worse* on this pass (re-hallucinated
differently) for reasons unrelated to either fix, which is why some individual pages regressed even as
the totals improved. A from-scratch re-ingest (true D) would be a cleaner test but costs ~2h27m; not run
again this pass.
**Still open:** page 44's screenshot-in-article OCR (garbled but partly real transcription of a mockup
image) was looked at and deliberately left alone -- a known-word-ratio filter would also catch genuine
quoted text, and no reliable rule to tell them apart was found; page 3's biggest "difference" turned out
to be baseline swallowing real editorial text into a figure's alt text, i.e. new is better there, not a
regression to fix. A from-scratch re-ingest to test the ingest-time ordering fix directly, and a decision
on whether to run the healed quality pass on the live production magazine (6 residual duplicates found
there, not yet corrected), are both still open.
### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison ### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,70 @@
# Magazine comparison
* baseline: `config/magazine_backups/bd871c7c22f0449984d3b0e3d603d6d6-after-autofix-20260928-1810`
* new: `config/reader_library/345b2a841f104b03b3cdf61faceb66cc`
| measure | value |
|---|---|
| pages | 204 |
| narratable words (baseline / new) | 65509 / 64946 |
| text similarity (mean per page) | 0.9345 |
| pages with text differences | 91 |
| box recall / precision | 0.8778 / 0.8479 |
| type agreement | 0.9546 |
| reading-order agreement | 0.969 |
| figures (baseline / new) | 183 / 218 |
| articles (baseline / new / matched) | 62 / 56 / 51 |
| matched articles with the same page span | 39 |
| pages flagged | 137 |
## Quality report
| | baseline | new |
|---|---|---|
| pages needing a decision | 27 | 53 |
| unboxed areas | 14 | 34 |
| clipped edges | 39 | 96 |
| overlapping pairs | 22 | 14 |
| words to check | 77 | 138 |
## The 15 pages that differ most
| page | flags | text | boxes b/n | order |
|---|---|---|---|---|
| 82 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.0 | 2/31 | None |
| 87 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.0 | 6/1 | None |
| 202 | text, extra_boxes_in_new | 0.0 | 0/8 | None |
| 184 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.1135 | 9/6 | 1.0 |
| 124 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.1648 | 31/10 | 1.0 |
| 44 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.5254 | 10/5 | None |
| 3 | text, boxes_missing_in_new, extra_boxes_in_new, types, order | 0.5743 | 6/11 | 0.9 |
| 148 | text, figures | 0.6479 | 3/3 | 1.0 |
| 152 | text, boxes_missing_in_new, extra_boxes_in_new | 0.6905 | 4/7 | 1.0 |
| 164 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.6992 | 17/8 | 1.0 |
| 75 | text, boxes_missing_in_new, extra_boxes_in_new, types, order, figures | 0.7411 | 7/11 | 0.6667 |
| 136 | text, boxes_missing_in_new, extra_boxes_in_new | 0.7744 | 7/10 | 1.0 |
| 31 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.7876 | 13/7 | 1.0 |
| 84 | text, extra_boxes_in_new | 0.7976 | 6/8 | 1.0 |
| 28 | text, boxes_missing_in_new, extra_boxes_in_new | 0.8 | 5/5 | 1.0 |
## Articles only in the baseline (11)
* 'Wie Kl Enbwickler-Jobs verändert' pages [32] (623 words)
* 'Durchstarten, bis die KI-Blase platzt?' pages [53] (320 words)
* 'Verbraulicher Butler gesucht' pages [59] (405 words)
* 'Wie KI die Softwareentwicklung erobert' pages [65, 66, 67] (1277 words)
* 'Page 63' pages [64] (30 words)
* 'Page 34' pages [34] (5 words)
* 'KI nutzen und nicht verdummen' pages [34] (104 words)
* 'Bequeme Mitarbeiter ersetzen sich selbst' pages [38] (476 words)
* 'KI-Einkaufsagenten: Neue Risiken' pages [184] (151 words)
* 'Page 193' pages [193] (1 words)
* 'Wer haftet, wenn KI einkauft?' pages [190] (118 words)
## Articles only in the new import (5)
* 'Durchstarten, bis die KI-Blase platzt?' pages [48, 49, 50, 51, 52, 53] (1742 words)
* 'Wie die Plattformen die Kreisläufe kontrollieren' pages [152, 153, 154, 155] (1220 words)
* 'Wall Street Star' pages [161] (478 words)
* 'KI-Einkaufsagenten: Neue Risiken' pages [184, 185, 186] (1151 words)
* "Vorschau: c't Digital vorsorgen" pages [202] (190 words)