docs(magazine): record D-E re-run results with both G fixes applied
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
ac1153b782
commit
6f2dfca89a
23
TODO.md
23
TODO.md
@ -120,10 +120,25 @@ rules beat what was built up by hand, and to feed every difference back into the
|
|||||||
backup itself is not clean either -- its next Auto-fix/quality run will self-heal now).
|
backup itself is not clean either -- its next Auto-fix/quality run will self-heal now).
|
||||||
Regression tests: `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`,
|
Regression tests: `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`,
|
||||||
`tests/test_magazine_routes.py::QualityRunJobTests::test_a_duplicate_created_by_recovering_missing_text_is_still_caught`,
|
`tests/test_magazine_routes.py::QualityRunJobTests::test_a_duplicate_created_by_recovering_missing_text_is_still_caught`,
|
||||||
`tests/test_magazine_routes.py::IngestSyncDuplicateRecheckTests`. **Still open:** re-run D→E with both
|
`tests/test_magazine_routes.py::IngestSyncDuplicateRecheckTests`.
|
||||||
fixes in place and see whether the gap to the backup narrows; the remaining worst-page cases not yet
|
|
||||||
root-caused (e.g. page 44's screenshot-in-article OCR) need a closer look before deciding whether they are
|
**D→E re-run with both fixes, 2026-09-29** (re-running Auto-fix on the same clone; report at
|
||||||
general bugs or page-specific noise not worth a rule for.
|
`architecture/evidence/compare-345b2a84-vs-backup-20260929-after-fixes.{md,json}`): pages needing a
|
||||||
|
decision 60→53, overlapping pairs 188→14, clipped edges 127→96 -- real improvement. Narratable word
|
||||||
|
count also dropped slightly (68680→64946), consistent with duplicate text no longer being counted twice.
|
||||||
|
Caveat: this re-run's own "re-reading suspicious words" step calls the vision engine again on flagged
|
||||||
|
boxes, and that engine is not deterministic -- pages 82 and 87 got *worse* on this pass (re-hallucinated
|
||||||
|
differently) for reasons unrelated to either fix, which is why some individual pages regressed even as
|
||||||
|
the totals improved. A from-scratch re-ingest (true D) would be a cleaner test but costs ~2h27m; not run
|
||||||
|
again this pass.
|
||||||
|
|
||||||
|
**Still open:** page 44's screenshot-in-article OCR (garbled but partly real transcription of a mockup
|
||||||
|
image) was looked at and deliberately left alone -- a known-word-ratio filter would also catch genuine
|
||||||
|
quoted text, and no reliable rule to tell them apart was found; page 3's biggest "difference" turned out
|
||||||
|
to be baseline swallowing real editorial text into a figure's alt text, i.e. new is better there, not a
|
||||||
|
regression to fix. A from-scratch re-ingest to test the ingest-time ordering fix directly, and a decision
|
||||||
|
on whether to run the healed quality pass on the live production magazine (6 residual duplicates found
|
||||||
|
there, not yet corrected), are both still open.
|
||||||
|
|
||||||
### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison
|
### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,70 @@
|
|||||||
|
# Magazine comparison
|
||||||
|
|
||||||
|
* baseline: `config/magazine_backups/bd871c7c22f0449984d3b0e3d603d6d6-after-autofix-20260928-1810`
|
||||||
|
* new: `config/reader_library/345b2a841f104b03b3cdf61faceb66cc`
|
||||||
|
|
||||||
|
| measure | value |
|
||||||
|
|---|---|
|
||||||
|
| pages | 204 |
|
||||||
|
| narratable words (baseline / new) | 65509 / 64946 |
|
||||||
|
| text similarity (mean per page) | 0.9345 |
|
||||||
|
| pages with text differences | 91 |
|
||||||
|
| box recall / precision | 0.8778 / 0.8479 |
|
||||||
|
| type agreement | 0.9546 |
|
||||||
|
| reading-order agreement | 0.969 |
|
||||||
|
| figures (baseline / new) | 183 / 218 |
|
||||||
|
| articles (baseline / new / matched) | 62 / 56 / 51 |
|
||||||
|
| matched articles with the same page span | 39 |
|
||||||
|
| pages flagged | 137 |
|
||||||
|
|
||||||
|
## Quality report
|
||||||
|
|
||||||
|
| | baseline | new |
|
||||||
|
|---|---|---|
|
||||||
|
| pages needing a decision | 27 | 53 |
|
||||||
|
| unboxed areas | 14 | 34 |
|
||||||
|
| clipped edges | 39 | 96 |
|
||||||
|
| overlapping pairs | 22 | 14 |
|
||||||
|
| words to check | 77 | 138 |
|
||||||
|
|
||||||
|
## The 15 pages that differ most
|
||||||
|
|
||||||
|
| page | flags | text | boxes b/n | order |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 82 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.0 | 2/31 | None |
|
||||||
|
| 87 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.0 | 6/1 | None |
|
||||||
|
| 202 | text, extra_boxes_in_new | 0.0 | 0/8 | None |
|
||||||
|
| 184 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.1135 | 9/6 | 1.0 |
|
||||||
|
| 124 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.1648 | 31/10 | 1.0 |
|
||||||
|
| 44 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.5254 | 10/5 | None |
|
||||||
|
| 3 | text, boxes_missing_in_new, extra_boxes_in_new, types, order | 0.5743 | 6/11 | 0.9 |
|
||||||
|
| 148 | text, figures | 0.6479 | 3/3 | 1.0 |
|
||||||
|
| 152 | text, boxes_missing_in_new, extra_boxes_in_new | 0.6905 | 4/7 | 1.0 |
|
||||||
|
| 164 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.6992 | 17/8 | 1.0 |
|
||||||
|
| 75 | text, boxes_missing_in_new, extra_boxes_in_new, types, order, figures | 0.7411 | 7/11 | 0.6667 |
|
||||||
|
| 136 | text, boxes_missing_in_new, extra_boxes_in_new | 0.7744 | 7/10 | 1.0 |
|
||||||
|
| 31 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.7876 | 13/7 | 1.0 |
|
||||||
|
| 84 | text, extra_boxes_in_new | 0.7976 | 6/8 | 1.0 |
|
||||||
|
| 28 | text, boxes_missing_in_new, extra_boxes_in_new | 0.8 | 5/5 | 1.0 |
|
||||||
|
|
||||||
|
## Articles only in the baseline (11)
|
||||||
|
|
||||||
|
* 'Wie Kl Enbwickler-Jobs verändert' pages [32] (623 words)
|
||||||
|
* 'Durchstarten, bis die KI-Blase platzt?' pages [53] (320 words)
|
||||||
|
* 'Verbraulicher Butler gesucht' pages [59] (405 words)
|
||||||
|
* 'Wie KI die Softwareentwicklung erobert' pages [65, 66, 67] (1277 words)
|
||||||
|
* 'Page 63' pages [64] (30 words)
|
||||||
|
* 'Page 34' pages [34] (5 words)
|
||||||
|
* 'KI nutzen und nicht verdummen' pages [34] (104 words)
|
||||||
|
* 'Bequeme Mitarbeiter ersetzen sich selbst' pages [38] (476 words)
|
||||||
|
* 'KI-Einkaufsagenten: Neue Risiken' pages [184] (151 words)
|
||||||
|
* 'Page 193' pages [193] (1 words)
|
||||||
|
* 'Wer haftet, wenn KI einkauft?' pages [190] (118 words)
|
||||||
|
|
||||||
|
## Articles only in the new import (5)
|
||||||
|
|
||||||
|
* 'Durchstarten, bis die KI-Blase platzt?' pages [48, 49, 50, 51, 52, 53] (1742 words)
|
||||||
|
* 'Wie die Plattformen die Kreisläufe kontrollieren' pages [152, 153, 154, 155] (1220 words)
|
||||||
|
* 'Wall Street Star' pages [161] (478 words)
|
||||||
|
* 'KI-Einkaufsagenten: Neue Risiken' pages [184, 185, 186] (1151 words)
|
||||||
|
* "Vorschau: c't Digital vorsorgen" pages [202] (190 words)
|
||||||
Loading…
Reference in New Issue
Block a user