mARTin-B78/tts-voice-creator-clone-and-design-2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(magazine): exclude advertisement pages from magazine-comparison statistics

Browse Source 
A page either side marks as a full-page ad is skipped by that side's quality
run, so comparing its near-empty boxes against the same page freshly OCR'd
on the other side (ad-marking is a one-time manual step) was a review-state
difference, not a pipeline one, skewing whole-issue numbers. Verified: text
similarity 0.837->0.870, box precision 0.62->0.66 on the real re-import.

Also records the clean from-scratch re-ingest results: the ingest-time
duplicate-check ordering fix is proven in isolation (0 additional duplicates
found afterward, vs 13 before the fix), and the auto-chained Auto-fix ran
end to end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-09-29 13:11:00 +02:00
parent f0b5d5aa68
commit faa3c16a91
6 changed files with 7467 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -9,6 +9,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
### Added ### Added
- `scripts/compare_magazines.py`: read-only comparison of two magazine folders (words, boxes, reading order, articles, quality numbers) for re-import checks. - `scripts/compare_magazines.py`: read-only comparison of two magazine folders (words, boxes, reading order, articles, quality numbers) for re-import checks.
### Changed (compare_magazines.py)
- A page either side marks as a full-page advertisement is now excluded from every difference statistic and the worst-pages list, reported separately instead (`pages_skipped_as_advertisement`). Being marked an ad is a one-time manual step a fresh re-import has usually not had yet, so without this a page the baseline skips entirely (near-empty) was compared against the same page fully OCR'd in the new import -- a difference in review state, not pipeline quality, that was skewing the whole-issue numbers.
### Changed ### Changed
- Spelling agent and acronym resolver call the language model with thinking off (about 4 s instead of about 27 s per call) and retry without the option if the server rejects it. - Spelling agent and acronym resolver call the language model with thinking off (about 4 s instead of about 27 s per call) and retry without the option if the server rejects it.
- The spelling agent treats run-together words (`dasich`) as errors and may split them at a space when both halves are real words and the letters are unchanged. - The spelling agent treats run-together words (`dasich`) as errors and may split them at a space when both halves are real words and the letters are unchanged.

31
TODO.md
View File

@ -136,9 +136,34 @@ rules beat what was built up by hand, and to feed every difference back into the
image) was looked at and deliberately left alone -- a known-word-ratio filter would also catch genuine image) was looked at and deliberately left alone -- a known-word-ratio filter would also catch genuine
quoted text, and no reliable rule to tell them apart was found; page 3's biggest "difference" turned out quoted text, and no reliable rule to tell them apart was found; page 3's biggest "difference" turned out
to be baseline swallowing real editorial text into a figure's alt text, i.e. new is better there, not a to be baseline swallowing real editorial text into a figure's alt text, i.e. new is better there, not a
regression to fix. A from-scratch re-ingest to test the ingest-time ordering fix directly, and a decision regression to fix. A decision on whether to run the healed quality pass on the live production magazine
on whether to run the healed quality pass on the live production magazine (6 residual duplicates found (6 residual duplicates found there, not yet corrected) is still open.
there, not yet corrected), are both still open.
**Ingest now runs Auto-fix itself, 2026-09-29.** A fresh import used to stop laid out but not worked,
leaving Auto-fix for a separate manual click; `_ingest_sync` now runs `_quality_run_sync` itself under
the same job (`routes/magazine.py`), so a fresh import opens already-fixed. An Auto-fix failure is
reported as an ingest failure without losing the ingest's own result. Regression tests:
`tests/test_magazine_routes.py::IngestSyncAutoQualityRunTests`.
**Clean from-scratch re-ingest, 2026-09-29** (doc `64fe2b4def774c29a44c14a5c4d35571`, both fixes and the
auto-chained Auto-fix in place; ~2h39m combined; report at
`architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.{md,json}`): the ingest-time
ordering fix is proven in isolation -- re-running `clean_up_regions` on the finished layout found **0**
additional duplicates (13 were found this way on the previous, unfixed clone). The hallucination filter
also fired identically (29 + 122 regions discarded on pages 82/87, matching the first run). But the
whole-magazine comparison against the backup came back worse than the previous clone's (76 pages needing
a decision vs. 53, overlapping pairs 170 vs. 14) for two traced, non-regression reasons: (1) the vision
OCR engine is not deterministic across independent full-magazine passes -- already seen with pages 82/87
specifically; (2) `scripts/compare_magazines.py` was comparing pages the backup has hand-marked as
full-page advertisements (skipped by its quality run) against the same pages freshly OCR'd in the new
import, which had never had that one-time manual step -- a review-state difference, not a pipeline one.
Fixed (2): the compare tool now excludes any page either side calls an advertisement from every
difference statistic, reporting the count separately instead (`pages_skipped_as_advertisement`; 18 on
this pair). Regression tests: `tests/test_compare_magazines.py` (advertisement handling). Re-run with the
fix: text similarity 0.837→0.870, box precision 0.62→0.66 -- a real but partial improvement, confirming
(1) still accounts for most of the remaining gap. Not fixable here: engine non-determinism is a property
of the external OCR service, not this pipeline; a fair A/B would need the same fixed OCR responses on
both sides, which the current tooling has no way to freeze.
### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison ### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison

7273
architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.json Normal file
View File

File diff suppressed because it is too large Load Diff

85
architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.md Normal file
View File

@ -0,0 +1,85 @@
# Magazine comparison
* baseline: `config/magazine_backups/bd871c7c22f0449984d3b0e3d603d6d6-after-autofix-20260928-1810`
* new: `config/reader_library/64fe2b4def774c29a44c14a5c4d35571`
| measure | value |
|---|---|
| pages | 204 |
| pages skipped as advertisement (either side) | 18 |
| narratable words (baseline / new) | 65509 / 61815 |
| text similarity (mean per page) | 0.8695 |
| pages with text differences | 127 |
| box recall / precision | 0.8355 / 0.6571 |
| type agreement | 0.8009 |
| reading-order agreement | 0.9627 |
| figures (baseline / new) | 183 / 268 |
| articles (baseline / new / matched) | 62 / 70 / 51 |
| matched articles with the same page span | 27 |
| pages flagged | 175 |
## Quality report
| | baseline | new |
|---|---|---|
| pages needing a decision | 27 | 76 |
| unboxed areas | 14 | 36 |
| clipped edges | 39 | 171 |
| overlapping pairs | 22 | 170 |
| words to check | 77 | 338 |
## The 15 pages that differ most
| page | flags | text | boxes b/n | order |
|---|---|---|---|---|
| 82 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.0 | 2/30 | None |
| 87 | text, boxes_missing_in_new, figures | 0.0 | 6/0 | None |
| 124 | text, boxes_missing_in_new, figures | 0.0 | 31/0 | None |
| 184 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.1052 | 9/7 | 1.0 |
| 167 | text, boxes_missing_in_new, extra_boxes_in_new, types | 0.161 | 4/20 | None |
| 144 | text, boxes_missing_in_new, extra_boxes_in_new, types, order | 0.2834 | 7/40 | 0.8333 |
| 95 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.3513 | 6/27 | 1.0 |
| 170 | text, boxes_missing_in_new, extra_boxes_in_new, types | 0.3834 | 5/10 | 1.0 |
| 15 | text, boxes_missing_in_new, extra_boxes_in_new, types, order, figures | 0.4404 | 9/28 | 0.7143 |
| 71 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.4599 | 7/16 | 1.0 |
| 123 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.4828 | 4/14 | 1.0 |
| 158 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.4845 | 6/9 | 1.0 |
| 135 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.4895 | 4/28 | 1.0 |
| 148 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.5 | 3/3 | 1.0 |
| 44 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.534 | 10/54 | 1.0 |
## Articles only in the baseline (11)
* 'Wie Kl Enbwickler-Jobs verändert' pages [32] (623 words)
* 'Durchstarten, bis die KI-Blase platzt?' pages [53] (320 words)
* 'Europa-KI: zwischen Mistral und Lovable' pages [54, 55, 56, 57, 58] (1337 words)
* 'Verbraulicher Butler gesucht' pages [59] (405 words)
* 'Page 63' pages [64] (30 words)
* 'Page 34' pages [34] (5 words)
* 'KI nutzen und nicht verdummen' pages [34] (104 words)
* 'Bequeme Mitarbeiter ersetzen sich selbst' pages [38] (476 words)
* 'KI-Einkaufsagenten: Neue Risiken' pages [184] (151 words)
* 'Page 193' pages [193] (1 words)
* 'Wer haftet, wenn KI einkauft?' pages [190] (118 words)
## Articles only in the new import (19)
* 'Kritische Analysen Er kreative Praxis' pages [1] (39 words)
* 'SLOP, KOMMERZ & MALWARE' pages [3] (102 words)
* 'Gedächtnis verwalten' pages [25, 26, 27] (1009 words)
* 'Literatur' pages [27] (261 words)
* "50®/o günstige' —_— me Exklusiver Sonderra" pages [33] (36 words)
* 'WIR TEILEN KEIN HALBWISSEN. WIR SCHAFFEN FACHWISSEN.' pages [37] (99 words)
* 'Durchstarten, bis die KI-Blase platzt?' pages [48, 49, 50, 51, 52, 53] (1779 words)
* 'Claude Code Tracker' pages [66, 67] (610 words)
* 'Sbimmb das?' pages [104] (67 words)
* 'Verwaltungsprozesse sicher digitalisieren' pages [107, 108] (250 words)
* 'Photovoltaik für Einsteiger' pages [109] (114 words)
* 'Workshops für IT-Professionals' pages [113] (108 words)
* 'Quanten-KI im Realitätscheck' pages [128, 129, 130] (795 words)
* 'ii imrl Mnx/omhor* • M^nrinpjm' pages [133] (68 words)
* 'Fazit' pages [146] (354 words)
* 'Wie die Plattformen die Kreisläufe kontrollieren' pages [152, 153, 154, 155] (1226 words)
* 'Wall Street Star' pages [161] (478 words)
* 'Was Plattformen tun' pages [170, 171] (486 words)
* 'KI-Einkaufsagenten: Neue Risiken' pages [184, 185, 186] (1140 words)

53
scripts/compare_magazines.py
View File

@ -49,6 +49,23 @@ def words(text: str) -> list[str]:
return [w.casefold().strip("-'’") for w in _WORD.findall(text or "") if w.strip("-'’")] return [w.casefold().strip("-'’") for w in _WORD.findall(text or "") if w.strip("-'’")]
def page_is_advertisement(layout: dict[str, Any], page: int) -> bool:
"""Same rule as core.magazine.page_is_advertisement, kept independent of it so this
script has no import on the app: flagged by hand, or detected ad regions and nothing
narratable. A page either side calls an advertisement is not a fair comparison -- being
marked one is itself a one-time manual step a fresh re-import has usually not had yet,
so the side that has not skips it fully instead (real OCR text where the other side has
almost none), which is a difference in review state, not in pipeline quality."""
meta = next((p for p in layout.get("pages", []) if p.get("page") == page), None)
if meta is None:
return False
if meta.get("advertisement"):
return True
on_page = [b for b in layout.get("blocks", []) if b.get("page") == page]
return (any(b.get("type") == "ad" for b in on_page)
and not any(b.get("type") in NARRATED_TYPES for b in on_page))
def page_words(layout: dict[str, Any], page: int) -> list[str]: def page_words(layout: dict[str, Any], page: int) -> list[str]:
blocks = sorted((b for b in layout.get("blocks", []) if b["page"] == page and b["type"] in NARRATED_TYPES), blocks = sorted((b for b in layout.get("blocks", []) if b["page"] == page and b["type"] in NARRATED_TYPES),
key=lambda b: b.get("reading_order", 0)) key=lambda b: b.get("reading_order", 0))
@ -129,8 +146,10 @@ def compare_pages(a: dict[str, Any], b: dict[str, Any]) -> list[dict[str, Any]]:
images_a = [i for i in a["layout"].get("images", []) if i["page"] == page] images_a = [i for i in a["layout"].get("images", []) if i["page"] == page]
images_b = [i for i in b["layout"].get("images", []) if i["page"] == page] images_b = [i for i in b["layout"].get("images", []) if i["page"] == page]
figures = match_boxes([{"bbox": i["bbox"]} for i in images_a], [{"bbox": i["bbox"]} for i in images_b]) figures = match_boxes([{"bbox": i["bbox"]} for i in images_a], [{"bbox": i["bbox"]} for i in images_b])
advertisement = page_is_advertisement(a["layout"], page) or page_is_advertisement(b["layout"], page)
row = { row = {
"page": page, "page": page,
"advertisement": advertisement,
"text_similarity": text["similarity"], "words_a": text["words_a"], "words_b": text["words_b"], "text_similarity": text["similarity"], "words_a": text["words_a"], "words_b": text["words_b"],
"only_in_a": " ".join(text["only_in_a"][:40]), "only_in_b": " ".join(text["only_in_b"][:40]), "only_in_a": " ".join(text["only_in_a"][:40]), "only_in_b": " ".join(text["only_in_b"][:40]),
"boxes_a": len(blocks_a), "boxes_b": len(blocks_b), "boxes_matched": len(matched), "boxes_a": len(blocks_a), "boxes_b": len(blocks_b), "boxes_matched": len(matched),
@ -143,6 +162,14 @@ def compare_pages(a: dict[str, Any], b: dict[str, Any]) -> list[dict[str, Any]]:
"figures_a": len(images_a), "figures_b": len(images_b), "figures_matched": len(figures), "figures_a": len(images_a), "figures_b": len(images_b), "figures_matched": len(figures),
"page_only_in": "a" if page not in pages_b else ("b" if page not in pages_a else ""), "page_only_in": "a" if page not in pages_b else ("b" if page not in pages_a else ""),
} }
# A page either side calls a full-page advertisement is skipped by the quality run on
# that side, so its own boxes/text are not comparable pipeline output -- flagging it
# "advertisement" instead of running the usual checks keeps it out of the difference
# counts and the worst-pages list without hiding that it exists (see markdown()).
if advertisement:
row["flags"] = ["advertisement"]
rows.append(row)
continue
flags = [] flags = []
if row["text_similarity"] < _SIMILAR_PAGE: if row["text_similarity"] < _SIMILAR_PAGE:
flags.append("text") flags.append("text")
@ -237,20 +264,26 @@ def mean(values: list[float]) -> float | None:
def compare(a: dict[str, Any], b: dict[str, Any]) -> dict[str, Any]: def compare(a: dict[str, Any], b: dict[str, Any]) -> dict[str, Any]:
pages = compare_pages(a, b) pages = compare_pages(a, b)
articles = compare_articles(a, b) articles = compare_articles(a, b)
# Pages either side calls a full-page advertisement are counted (see pages/words below,
# so the totals still add up) but left out of every difference statistic -- being marked
# one is a one-time manual step, not pipeline output, so comparing it is comparing review
# state, not quality. Reported separately instead of silently dropped.
compared = [p for p in pages if not p["advertisement"]]
words_a = sum(p["words_a"] for p in pages) words_a = sum(p["words_a"] for p in pages)
words_b = sum(p["words_b"] for p in pages) words_b = sum(p["words_b"] for p in pages)
summary = { summary = {
"baseline": a["folder"], "new": b["folder"], "baseline": a["folder"], "new": b["folder"],
"pages": len(pages), "pages": len(pages),
"pages_skipped_as_advertisement": sum(p["advertisement"] for p in pages),
"words": {"baseline": words_a, "new": words_b}, "words": {"baseline": words_a, "new": words_b},
"text_similarity_mean": mean([p["text_similarity"] for p in pages]), "text_similarity_mean": mean([p["text_similarity"] for p in compared]),
"pages_with_text_differences": sum("text" in p["flags"] for p in pages), "pages_with_text_differences": sum("text" in p["flags"] for p in compared),
"box_recall_mean": mean([p["box_recall"] for p in pages]), "box_recall_mean": mean([p["box_recall"] for p in compared]),
"box_precision_mean": mean([p["box_precision"] for p in pages]), "box_precision_mean": mean([p["box_precision"] for p in compared]),
"type_agreement_mean": mean([p["type_agreement"] for p in pages]), "type_agreement_mean": mean([p["type_agreement"] for p in compared]),
"order_agreement_mean": mean([p["order_agreement"] for p in pages]), "order_agreement_mean": mean([p["order_agreement"] for p in compared]),
"pages_flagged": sum(bool(p["flags"]) for p in pages), "pages_flagged": sum(bool(p["flags"]) for p in compared),
"figures": {"baseline": sum(p["figures_a"] for p in pages), "new": sum(p["figures_b"] for p in pages)}, "figures": {"baseline": sum(p["figures_a"] for p in compared), "new": sum(p["figures_b"] for p in compared)},
"articles": {"baseline": len(a["articles"]), "new": len(b["articles"]), "articles": {"baseline": len(a["articles"]), "new": len(b["articles"]),
"matched": len(articles["matched"]), "only_in_baseline": len(articles["only_in_a"]), "matched": len(articles["matched"]), "only_in_baseline": len(articles["only_in_a"]),
"only_in_new": len(articles["only_in_b"]), "only_in_new": len(articles["only_in_b"]),
@ -266,6 +299,7 @@ def markdown(report: dict[str, Any], worst: int = 15) -> str:
lines = ["# Magazine comparison", "", f"* baseline: `{s['baseline']}`", f"* new: `{s['new']}`", "", lines = ["# Magazine comparison", "", f"* baseline: `{s['baseline']}`", f"* new: `{s['new']}`", "",
"| measure | value |", "|---|---|", "| measure | value |", "|---|---|",
f"| pages | {s['pages']} |", f"| pages | {s['pages']} |",
f"| pages skipped as advertisement (either side) | {s['pages_skipped_as_advertisement']} |",
f"| narratable words (baseline / new) | {s['words']['baseline']} / {s['words']['new']} |", f"| narratable words (baseline / new) | {s['words']['baseline']} / {s['words']['new']} |",
f"| text similarity (mean per page) | {s['text_similarity_mean']} |", f"| text similarity (mean per page) | {s['text_similarity_mean']} |",
f"| pages with text differences | {s['pages_with_text_differences']} |", f"| pages with text differences | {s['pages_with_text_differences']} |",
@ -279,7 +313,8 @@ def markdown(report: dict[str, Any], worst: int = 15) -> str:
"| | baseline | new |", "|---|---|---|"] "| | baseline | new |", "|---|---|---|"]
for key in s["quality"]["baseline"]: for key in s["quality"]["baseline"]:
lines.append(f"| {key.replace('_', ' ')} | {s['quality']['baseline'][key]} | {s['quality']['new'][key]} |") lines.append(f"| {key.replace('_', ' ')} | {s['quality']['baseline'][key]} | {s['quality']['new'][key]} |")
flagged = sorted((p for p in report["pages"] if p["flags"]), key=lambda p: (p["text_similarity"], p["page"]))[:worst] flagged = sorted((p for p in report["pages"] if p["flags"] and not p["advertisement"]),
key=lambda p: (p["text_similarity"], p["page"]))[:worst]
lines += ["", f"## The {len(flagged)} pages that differ most", "", "| page | flags | text | boxes b/n | order |", "|---|---|---|---|---|"] lines += ["", f"## The {len(flagged)} pages that differ most", "", "| page | flags | text | boxes b/n | order |", "|---|---|---|---|---|"]
for p in flagged: for p in flagged:
lines.append(f"| {p['page']} | {', '.join(p['flags'])} | {p['text_similarity']} | {p['boxes_a']}/{p['boxes_b']} | {p['order_agreement']} |") lines.append(f"| {p['page']} | {', '.join(p['flags'])} | {p['text_similarity']} | {p['boxes_a']}/{p['boxes_b']} | {p['order_agreement']} |")

34
tests/test_compare_magazines.py
View File

@ -95,6 +95,40 @@ class CompareTests(unittest.TestCase):
self.assertIn("# Magazine comparison", text) self.assertIn("# Magazine comparison", text)
self.assertIn("| clipped edges | 0 | 2 |", text) self.assertIn("| clipped edges | 0 | 2 |", text)
def test_a_page_marked_advertisement_on_either_side_is_skipped_not_flagged(self):
"""Baseline marks page 1 a full-page ad (skipped by the quality run, so it was never
OCR'd further); the new import has not had that manual step yet and actually OCR'd
real (if unrelated) text there. Comparing the two would otherwise read as a bad text
match and extra boxes -- a difference in review state, not in pipeline quality."""
base = self.folder(
[blk("ad", 1, "ad", "", [0, 0, 500, 700], 0), blk("c", 2, "body", "Und die zweite Seite geht weiter.", [10, 10, 90, 100], 0)],
[art("article-001", "Der Butler", [2], ["c"])])
json.loads((base / "layout.json").read_text()) # sanity: valid JSON written
layout = json.loads((base / "layout.json").read_text())
layout["pages"][0]["advertisement"] = True
(base / "layout.json").write_text(json.dumps(layout))
new = self.folder(
[blk("x", 1, "body", "Ein ganz anderer Text der hier plotzlich auftaucht.", [10, 10, 300, 300], 0),
blk("c", 2, "body", "Und die zweite Seite geht weiter.", [10, 10, 90, 100], 0)],
[art("article-001", "Der Butler", [2], ["c"])])
report = cm.compare(cm.load_magazine(base), cm.load_magazine(new))
page1 = next(p for p in report["pages"] if p["page"] == 1)
self.assertTrue(page1["advertisement"])
self.assertEqual(page1["flags"], ["advertisement"])
self.assertEqual(report["summary"]["pages_skipped_as_advertisement"], 1)
self.assertEqual(report["summary"]["pages_flagged"], 0) # not counted as a difference
self.assertNotIn(page1, [p for p in report["pages"] if p["flags"] and not p["advertisement"]])
text = cm.markdown(report)
self.assertIn("| pages skipped as advertisement (either side) | 1 |", text)
self.assertNotIn("| 1 | advertisement", text) # never in the worst-pages table
def test_the_advertisement_rule_also_fires_from_ad_regions_with_no_manual_flag(self):
base = self.folder([blk("ad", 1, "ad", "Werbung", [0, 0, 500, 700], 0)], [])
self.assertTrue(cm.page_is_advertisement(cm.load_magazine(base)["layout"], 1))
mixed = self.folder([blk("ad", 1, "ad", "Werbung", [0, 0, 200, 700], 0),
blk("t", 1, "body", "Aber hier steht auch echter Text.", [200, 0, 300, 700], 1)], [])
self.assertFalse(cm.page_is_advertisement(cm.load_magazine(mixed)["layout"], 1)) # narratable text present
def test_the_command_line_writes_json_and_markdown(self): def test_the_command_line_writes_json_and_markdown(self):
out = Path(tempfile.mkdtemp(prefix="ttvc-cmp-out-")) out = Path(tempfile.mkdtemp(prefix="ttvc-cmp-out-"))
self.addCleanup(shutil.rmtree, out, True) self.addCleanup(shutil.rmtree, out, True)