mARTin-B78/tts-voice-creator-clone-and-design-2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(magazine): exclude advertisement pages from magazine-comparison statistics

Browse Source 
A page either side marks as a full-page ad is skipped by that side's quality
run, so comparing its near-empty boxes against the same page freshly OCR'd
on the other side (ad-marking is a one-time manual step) was a review-state
difference, not a pipeline one, skewing whole-issue numbers. Verified: text
similarity 0.837->0.870, box precision 0.62->0.66 on the real re-import.

Also records the clean from-scratch re-ingest results: the ingest-time
duplicate-check ordering fix is proven in isolation (0 additional duplicates
found afterward, vs 13 before the fix), and the auto-chained Auto-fix ran
end to end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-09-29 13:11:00 +02:00
parent f0b5d5aa68
commit faa3c16a91
6 changed files with 7467 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -9,6 +9,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
### Added
- `scripts/compare_magazines.py`: read-only comparison of two magazine folders (words, boxes, reading order, articles, quality numbers) for re-import checks.
### Changed (compare_magazines.py)
- A page either side marks as a full-page advertisement is now excluded from every difference statistic and the worst-pages list, reported separately instead (`pages_skipped_as_advertisement`). Being marked an ad is a one-time manual step a fresh re-import has usually not had yet, so without this a page the baseline skips entirely (near-empty) was compared against the same page fully OCR'd in the new import -- a difference in review state, not pipeline quality, that was skewing the whole-issue numbers.
### Changed
- Spelling agent and acronym resolver call the language model with thinking off (about 4 s instead of about 27 s per call) and retry without the option if the server rejects it.
- The spelling agent treats run-together words (`dasich`) as errors and may split them at a space when both halves are real words and the letters are unchanged.

31
TODO.md
View File

@ -136,9 +136,34 @@ rules beat what was built up by hand, and to feed every difference back into the
image) was looked at and deliberately left alone -- a known-word-ratio filter would also catch genuine
quoted text, and no reliable rule to tell them apart was found; page 3's biggest "difference" turned out
to be baseline swallowing real editorial text into a figure's alt text, i.e. new is better there, not a
regression to fix. A from-scratch re-ingest to test the ingest-time ordering fix directly, and a decision
on whether to run the healed quality pass on the live production magazine (6 residual duplicates found
there, not yet corrected), are both still open.
regression to fix. A decision on whether to run the healed quality pass on the live production magazine
(6 residual duplicates found there, not yet corrected) is still open.
**Ingest now runs Auto-fix itself, 2026-09-29.** A fresh import used to stop laid out but not worked,
leaving Auto-fix for a separate manual click; `_ingest_sync` now runs `_quality_run_sync` itself under
the same job (`routes/magazine.py`), so a fresh import opens already-fixed. An Auto-fix failure is
reported as an ingest failure without losing the ingest's own result. Regression tests:
`tests/test_magazine_routes.py::IngestSyncAutoQualityRunTests`.
**Clean from-scratch re-ingest, 2026-09-29** (doc `64fe2b4def774c29a44c14a5c4d35571`, both fixes and the
auto-chained Auto-fix in place; ~2h39m combined; report at
`architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.{md,json}`): the ingest-time
ordering fix is proven in isolation -- re-running `clean_up_regions` on the finished layout found **0**
additional duplicates (13 were found this way on the previous, unfixed clone). The hallucination filter
also fired identically (29 + 122 regions discarded on pages 82/87, matching the first run). But the
whole-magazine comparison against the backup came back worse than the previous clone's (76 pages needing
a decision vs. 53, overlapping pairs 170 vs. 14) for two traced, non-regression reasons: (1) the vision
OCR engine is not deterministic across independent full-magazine passes -- already seen with pages 82/87
specifically; (2) `scripts/compare_magazines.py` was comparing pages the backup has hand-marked as
full-page advertisements (skipped by its quality run) against the same pages freshly OCR'd in the new
import, which had never had that one-time manual step -- a review-state difference, not a pipeline one.
Fixed (2): the compare tool now excludes any page either side calls an advertisement from every
difference statistic, reporting the count separately instead (`pages_skipped_as_advertisement`; 18 on
this pair). Regression tests: `tests/test_compare_magazines.py` (advertisement handling). Re-run with the
fix: text similarity 0.837→0.870, box precision 0.62→0.66 -- a real but partial improvement, confirming
(1) still accounts for most of the remaining gap. Not fixable here: engine non-determinism is a property
of the external OCR service, not this pipeline; a fair A/B would need the same fixed OCR responses on
both sides, which the current tooling has no way to freeze.
### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison

7273
architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.json Normal file
View File

File diff suppressed because it is too large Load Diff

85
architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.md Normal file
View File

@ -0,0 +1,85 @@
# Magazine comparison
* baseline: `config/magazine_backups/bd871c7c22f0449984d3b0e3d603d6d6-after-autofix-20260928-1810`
* new: `config/reader_library/64fe2b4def774c29a44c14a5c4d35571`
| measure | value |
|---|---|
| pages | 204 |
| pages skipped as advertisement (either side) | 18 |
| narratable words (baseline / new) | 65509 / 61815 |
| text similarity (mean per page) | 0.8695 |
| pages with text differences | 127 |
| box recall / precision | 0.8355 / 0.6571 |
| type agreement | 0.8009 |
| reading-order agreement | 0.9627 |
| figures (baseline / new) | 183 / 268 |
| articles (baseline / new / matched) | 62 / 70 / 51 |
| matched articles with the same page span | 27 |
| pages flagged | 175 |
## Quality report
| | baseline | new |
|---|---|---|
| pages needing a decision | 27 | 76 |
| unboxed areas | 14 | 36 |
| clipped edges | 39 | 171 |
| overlapping pairs | 22 | 170 |
| words to check | 77 | 338 |
## The 15 pages that differ most
| page | flags | text | boxes b/n | order |
|---|---|---|---|---|
| 82 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.0 | 2/30 | None |
| 87 | text, boxes_missing_in_new, figures | 0.0 | 6/0 | None |
| 124 | text, boxes_missing_in_new, figures | 0.0 | 31/0 | None |
| 184 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.1052 | 9/7 | 1.0 |
| 167 | text, boxes_missing_in_new, extra_boxes_in_new, types | 0.161 | 4/20 | None |
| 144 | text, boxes_missing_in_new, extra_boxes_in_new, types, order | 0.2834 | 7/40 | 0.8333 |
| 95 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.3513 | 6/27 | 1.0 |
| 170 | text, boxes_missing_in_new, extra_boxes_in_new, types | 0.3834 | 5/10 | 1.0 |
| 15 | text, boxes_missing_in_new, extra_boxes_in_new, types, order, figures | 0.4404 | 9/28 | 0.7143 |
| 71 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.4599 | 7/16 | 1.0 |
| 123 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.4828 | 4/14 | 1.0 |
| 158 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.4845 | 6/9 | 1.0 |
| 135 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.4895 | 4/28 | 1.0 |
| 148 | text, boxes_missing_in_new, extra_boxes_in_new, figures | 0.5 | 3/3 | 1.0 |
| 44 | text, boxes_missing_in_new, extra_boxes_in_new, types, figures | 0.534 | 10/54 | 1.0 |
## Articles only in the baseline (11)
* 'Wie Kl Enbwickler-Jobs verändert' pages [32] (623 words)
* 'Durchstarten, bis die KI-Blase platzt?' pages [53] (320 words)
* 'Europa-KI: zwischen Mistral und Lovable' pages [54, 55, 56, 57, 58] (1337 words)
* 'Verbraulicher Butler gesucht' pages [59] (405 words)
* 'Page 63' pages [64] (30 words)
* 'Page 34' pages [34] (5 words)
* 'KI nutzen und nicht verdummen' pages [34] (104 words)
* 'Bequeme Mitarbeiter ersetzen sich selbst' pages [38] (476 words)
* 'KI-Einkaufsagenten: Neue Risiken' pages [184] (151 words)
* 'Page 193' pages [193] (1 words)
* 'Wer haftet, wenn KI einkauft?' pages [190] (118 words)
## Articles only in the new import (19)
* 'Kritische Analysen Er kreative Praxis' pages [1] (39 words)
* 'SLOP, KOMMERZ & MALWARE' pages [3] (102 words)
* 'Gedächtnis verwalten' pages [25, 26, 27] (1009 words)
* 'Literatur' pages [27] (261 words)
* "50®/o günstige' —_— me Exklusiver Sonderra" pages [33] (36 words)
* 'WIR TEILEN KEIN HALBWISSEN. WIR SCHAFFEN FACHWISSEN.' pages [37] (99 words)
* 'Durchstarten, bis die KI-Blase platzt?' pages [48, 49, 50, 51, 52, 53] (1779 words)
* 'Claude Code Tracker' pages [66, 67] (610 words)
* 'Sbimmb das?' pages [104] (67 words)
* 'Verwaltungsprozesse sicher digitalisieren' pages [107, 108] (250 words)
* 'Photovoltaik für Einsteiger' pages [109] (114 words)
* 'Workshops für IT-Professionals' pages [113] (108 words)
* 'Quanten-KI im Realitätscheck' pages [128, 129, 130] (795 words)
* 'ii imrl Mnx/omhor* • M^nrinpjm' pages [133] (68 words)
* 'Fazit' pages [146] (354 words)
* 'Wie die Plattformen die Kreisläufe kontrollieren' pages [152, 153, 154, 155] (1226 words)
* 'Wall Street Star' pages [161] (478 words)
* 'Was Plattformen tun' pages [170, 171] (486 words)
* 'KI-Einkaufsagenten: Neue Risiken' pages [184, 185, 186] (1140 words)

53
scripts/compare_magazines.py
View File

@ -49,6 +49,23 @@ def words(text: str) -> list[str]:
return [w.casefold().strip("-'’") for w in _WORD.findall(text or "") if w.strip("-'’")]
def page_is_advertisement(layout: dict[str, Any], page: int) -> bool:
"""Same rule as core.magazine.page_is_advertisement, kept independent of it so this
script has no import on the app: flagged by hand, or detected ad regions and nothing
narratable. A page either side calls an advertisement is not a fair comparison -- being
marked one is itself a one-time manual step a fresh re-import has usually not had yet,
so the side that has not skips it fully instead (real OCR text where the other side has
almost none), which is a difference in review state, not in pipeline quality."""
meta = next((p for p in layout.get("pages", []) if p.get("page") == page), None)
if meta is None:
return False
if meta.get("advertisement"):
return True
on_page = [b for b in layout.get("blocks", []) if b.get("page") == page]
return (any(b.get("type") == "ad" for b in on_page)
and not any(b.get("type") in NARRATED_TYPES for b in on_page))
def page_words(layout: dict[str, Any], page: int) -> list[str]:
blocks = sorted((b for b in layout.get("blocks", []) if b["page"] == page and b["type"] in NARRATED_TYPES),
key=lambda b: b.get("reading_order", 0))
@ -129,8 +146,10 @@ def compare_pages(a: dict[str, Any], b: dict[str, Any]) -> list[dict[str, Any]]:
images_a = [i for i in a["layout"].get("images", []) if i["page"] == page]
images_b = [i for i in b["layout"].get("images", []) if i["page"] == page]
figures = match_boxes([{"bbox": i["bbox"]} for i in images_a], [{"bbox": i["bbox"]} for i in images_b])
advertisement = page_is_advertisement(a["layout"], page) or page_is_advertisement(b["layout"], page)
row = {
"page": page,
"advertisement": advertisement,
"text_similarity": text["similarity"], "words_a": text["words_a"], "words_b": text["words_b"],
"only_in_a": " ".join(text["only_in_a"][:40]), "only_in_b": " ".join(text["only_in_b"][:40]),
"boxes_a": len(blocks_a), "boxes_b": len(blocks_b), "boxes_matched": len(matched),
@ -143,6 +162,14 @@ def compare_pages(a: dict[str, Any], b: dict[str, Any]) -> list[dict[str, Any]]:
"figures_a": len(images_a), "figures_b": len(images_b), "figures_matched": len(figures),
"page_only_in": "a" if page not in pages_b else ("b" if page not in pages_a else ""),
}
# A page either side calls a full-page advertisement is skipped by the quality run on
# that side, so its own boxes/text are not comparable pipeline output -- flagging it
# "advertisement" instead of running the usual checks keeps it out of the difference
# counts and the worst-pages list without hiding that it exists (see markdown()).
if advertisement:
row["flags"] = ["advertisement"]
rows.append(row)
continue
flags = []
if row["text_similarity"] < _SIMILAR_PAGE:
flags.append("text")
@ -237,20 +264,26 @@ def mean(values: list[float]) -> float | None:
def compare(a: dict[str, Any], b: dict[str, Any]) -> dict[str, Any]:
pages = compare_pages(a, b)
articles = compare_articles(a, b)
# Pages either side calls a full-page advertisement are counted (see pages/words below,
# so the totals still add up) but left out of every difference statistic -- being marked
# one is a one-time manual step, not pipeline output, so comparing it is comparing review
# state, not quality. Reported separately instead of silently dropped.
compared = [p for p in pages if not p["advertisement"]]
words_a = sum(p["words_a"] for p in pages)
words_b = sum(p["words_b"] for p in pages)
summary = {
"baseline": a["folder"], "new": b["folder"],
"pages": len(pages),
"pages_skipped_as_advertisement": sum(p["advertisement"] for p in pages),
"words": {"baseline": words_a, "new": words_b},
"text_similarity_mean": mean([p["text_similarity"] for p in pages]),
"pages_with_text_differences": sum("text" in p["flags"] for p in pages),
"box_recall_mean": mean([p["box_recall"] for p in pages]),
"box_precision_mean": mean([p["box_precision"] for p in pages]),
"type_agreement_mean": mean([p["type_agreement"] for p in pages]),
"order_agreement_mean": mean([p["order_agreement"] for p in pages]),
"pages_flagged": sum(bool(p["flags"]) for p in pages),
"figures": {"baseline": sum(p["figures_a"] for p in pages), "new": sum(p["figures_b"] for p in pages)},
"text_similarity_mean": mean([p["text_similarity"] for p in compared]),
"pages_with_text_differences": sum("text" in p["flags"] for p in compared),
"box_recall_mean": mean([p["box_recall"] for p in compared]),
"box_precision_mean": mean([p["box_precision"] for p in compared]),
"type_agreement_mean": mean([p["type_agreement"] for p in compared]),
"order_agreement_mean": mean([p["order_agreement"] for p in compared]),
"pages_flagged": sum(bool(p["flags"]) for p in compared),
"figures": {"baseline": sum(p["figures_a"] for p in compared), "new": sum(p["figures_b"] for p in compared)},
"articles": {"baseline": len(a["articles"]), "new": len(b["articles"]),
"matched": len(articles["matched"]), "only_in_baseline": len(articles["only_in_a"]),
"only_in_new": len(articles["only_in_b"]),
@ -266,6 +299,7 @@ def markdown(report: dict[str, Any], worst: int = 15) -> str:
lines = ["# Magazine comparison", "", f"* baseline: `{s['baseline']}`", f"* new: `{s['new']}`", "",
"| measure | value |", "|---|---|",
f"| pages | {s['pages']} |",
f"| pages skipped as advertisement (either side) | {s['pages_skipped_as_advertisement']} |",
f"| narratable words (baseline / new) | {s['words']['baseline']} / {s['words']['new']} |",
f"| text similarity (mean per page) | {s['text_similarity_mean']} |",
f"| pages with text differences | {s['pages_with_text_differences']} |",
@ -279,7 +313,8 @@ def markdown(report: dict[str, Any], worst: int = 15) -> str:
"| | baseline | new |", "|---|---|---|"]
for key in s["quality"]["baseline"]:
lines.append(f"| {key.replace('_', ' ')} | {s['quality']['baseline'][key]} | {s['quality']['new'][key]} |")
flagged = sorted((p for p in report["pages"] if p["flags"]), key=lambda p: (p["text_similarity"], p["page"]))[:worst]
flagged = sorted((p for p in report["pages"] if p["flags"] and not p["advertisement"]),
key=lambda p: (p["text_similarity"], p["page"]))[:worst]
lines += ["", f"## The {len(flagged)} pages that differ most", "", "| page | flags | text | boxes b/n | order |", "|---|---|---|---|---|"]
for p in flagged:
lines.append(f"| {p['page']} | {', '.join(p['flags'])} | {p['text_similarity']} | {p['boxes_a']}/{p['boxes_b']} | {p['order_agreement']} |")

34
tests/test_compare_magazines.py
View File

@ -95,6 +95,40 @@ class CompareTests(unittest.TestCase):
self.assertIn("# Magazine comparison", text)
self.assertIn("| clipped edges | 0 | 2 |", text)
def test_a_page_marked_advertisement_on_either_side_is_skipped_not_flagged(self):
"""Baseline marks page 1 a full-page ad (skipped by the quality run, so it was never
OCR'd further); the new import has not had that manual step yet and actually OCR'd
real (if unrelated) text there. Comparing the two would otherwise read as a bad text
match and extra boxes -- a difference in review state, not in pipeline quality."""
base = self.folder(
[blk("ad", 1, "ad", "", [0, 0, 500, 700], 0), blk("c", 2, "body", "Und die zweite Seite geht weiter.", [10, 10, 90, 100], 0)],
[art("article-001", "Der Butler", [2], ["c"])])
json.loads((base / "layout.json").read_text()) # sanity: valid JSON written
layout = json.loads((base / "layout.json").read_text())
layout["pages"][0]["advertisement"] = True
(base / "layout.json").write_text(json.dumps(layout))
new = self.folder(
[blk("x", 1, "body", "Ein ganz anderer Text der hier plotzlich auftaucht.", [10, 10, 300, 300], 0),
blk("c", 2, "body", "Und die zweite Seite geht weiter.", [10, 10, 90, 100], 0)],
[art("article-001", "Der Butler", [2], ["c"])])
report = cm.compare(cm.load_magazine(base), cm.load_magazine(new))
page1 = next(p for p in report["pages"] if p["page"] == 1)
self.assertTrue(page1["advertisement"])
self.assertEqual(page1["flags"], ["advertisement"])
self.assertEqual(report["summary"]["pages_skipped_as_advertisement"], 1)
self.assertEqual(report["summary"]["pages_flagged"], 0) # not counted as a difference
self.assertNotIn(page1, [p for p in report["pages"] if p["flags"] and not p["advertisement"]])
text = cm.markdown(report)
self.assertIn("| pages skipped as advertisement (either side) | 1 |", text)
self.assertNotIn("| 1 | advertisement", text) # never in the worst-pages table
def test_the_advertisement_rule_also_fires_from_ad_regions_with_no_manual_flag(self):
base = self.folder([blk("ad", 1, "ad", "Werbung", [0, 0, 500, 700], 0)], [])
self.assertTrue(cm.page_is_advertisement(cm.load_magazine(base)["layout"], 1))
mixed = self.folder([blk("ad", 1, "ad", "Werbung", [0, 0, 200, 700], 0),
blk("t", 1, "body", "Aber hier steht auch echter Text.", [200, 0, 300, 700], 1)], [])
self.assertFalse(cm.page_is_advertisement(cm.load_magazine(mixed)["layout"], 1)) # narratable text present
def test_the_command_line_writes_json_and_markdown(self):
out = Path(tempfile.mkdtemp(prefix="ttvc-cmp-out-"))
self.addCleanup(shutil.rmtree, out, True)