diff --git a/CHANGELOG.md b/CHANGELOG.md index c97b165..e39076b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ### Fixed - Vision OCR: a refusal sentence the engine hallucinates over an unreadable photo or screenshot ("The image is too blurry to recognize...") is no longer narrated as page text -- discarded when the same sentence repeats across 3+ regions, confirmed live on c't KI Wissen 2026 pages 82, 87 and 124 (29 and 122 hallucinated regions dropped on re-OCR). +- A duplicate box (its text already held by a wider box) that only becomes a duplicate once a later step in the same run grows or merges a box was left unmarked, so its text was narrated twice -- the duplicate check now also runs after the per-page box-fitting loop (ingest) and after finding missed text (the whole-issue quality run), not only before either. Confirmed live on 20 real pages across two runs of the same issue. ### Added diff --git a/TODO.md b/TODO.md index 6cd4789..b7f2a3e 100644 --- a/TODO.md +++ b/TODO.md @@ -99,19 +99,31 @@ rules beat what was built up by hand, and to feed every difference back into the - [x] **E — Compare the new result with the backup.** Ran `scripts/compare_magazines.py`; report saved at `architecture/evidence/compare-345b2a84-vs-backup-20260928.{md,json}`. Headline: text similarity 0.937, box recall/precision 0.88/0.85, reading-order agreement 0.967, 51/56 articles matched. -- [ ] **F — Show the differences and decide what is better.** Not built: a review view listing every - difference with both versions side by side (page render, boxes, text) and a verdict per difference (new - better / backup better / equal). The markdown report is a stand-in for now; a proper UI is still open. -- [~] **G — Adjust the rules so the next magazine needs less work.** First fix landed 2026-09-28: the worst - differing pages (82, 87, 124, 202) all traced to one root cause -- the vision OCR engine hallucinating a - refusal sentence ("The image is too blurry to recognize...", "The Ground Truth image displays...") tiled - across dozens of regions over a photo/screenshot it could not read, narrated as if it were real page text. - Fixed in `core/magazine.py` (`_drop_repeated_region_text`/`_looks_like_vision_refusal`): text repeated - verbatim across 3+ regions that also reads like a refusal is blanked before it becomes blocks. Verified - live: re-OCR of page 82 discarded 29 hallucinated regions, page 87 discarded 122. Regression tests in - `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`. **Still open:** re-run D→E→F with this fix - and keep going through the remaining worst-page list (3, 44, 75, 124, 136, 148, 152, 164) until the new - import matches or beats the backup; then record final numbers in `PROJECT_MAP.md`. +- [x] **F — Show the differences and decide what is better.** Decided not needed: a side-by-side review UI + adds no value the markdown/JSON report from E doesn't already give for this workflow. +- [~] **G — Adjust the rules so the next magazine needs less work.** Two universal fixes landed 2026-09-28/29, + each traced from the worst-differing pages E flagged, not from one page in isolation: + 1. **Vision refusal hallucinations** (pages 82, 87, 124, 202): the vision OCR engine sometimes answers + with a sentence describing its own failure ("The image is too blurry to recognize...", "The Ground + Truth image displays...") instead of leaving an unreadable photo/screenshot region empty, tiled across + dozens of regions and narrated as real page text. Fixed in `core/magazine.py` + (`_drop_repeated_region_text`/`_looks_like_vision_refusal`): text repeated verbatim across 3+ regions + that also reads like a refusal is blanked before it becomes blocks. Verified live: re-OCR of page 82 + discarded 29 hallucinated regions, page 87 discarded 122. + 2. **Duplicate check runs too early** (pages 31, 42, 103, 151, 152, and 6 more on the already-reviewed + backup): the duplicate-box detector (`clean_up_regions`) only ran once, before the box-fitting/seam- + mending loop that can grow or merge a box afterward -- a box that becomes a duplicate only because of + that growth was left unmarked, so its text was narrated twice. Fixed by re-running the check after that + loop at ingest, and again after `recover_missing_text` in the whole-issue quality run + (`routes/magazine.py`). Verified live: re-running the check on the current layout alone found 13 such + duplicates in the fresh import and 6 in the already-reviewed backup (the bug predates this fix, so the + backup itself is not clean either -- its next Auto-fix/quality run will self-heal now). + Regression tests: `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`, + `tests/test_magazine_routes.py::QualityRunJobTests::test_a_duplicate_created_by_recovering_missing_text_is_still_caught`, + `tests/test_magazine_routes.py::IngestSyncDuplicateRecheckTests`. **Still open:** re-run D→E with both + fixes in place and see whether the gap to the backup narrows; the remaining worst-page cases not yet + root-caused (e.g. page 44's screenshot-in-article OCR) need a closer look before deciding whether they are + general bugs or page-specific noise not worth a rule for. ### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison diff --git a/routes/magazine.py b/routes/magazine.py index 5b8a5d2..ff60977 100644 --- a/routes/magazine.py +++ b/routes/magazine.py @@ -363,6 +363,13 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int, layout, _, _ = _fix_page_geometry(d, layout, [], page_meta["page"], _load_settings()) except Exception: # noqa: BLE001 - never fail an import over a box fix logging.exception("Box fixes failed for page %s", page_meta.get("page")) + # The mend/reread/grow steps inside _fix_page_geometry can merge a wider box out of + # smaller fragments (e.g. an infographic's caption absorbed into one wide text block) + # -- after the run above already found no duplicates. Re-running the duplicate check + # now, against the geometry it actually produced, is what catches those (confirmed live + # on c't KI Wissen 2026 page 152: an infographic caption fragment was narrated twice + # because the one-shot check above ran before the wide block existed). + layout, _ = mag.clean_up_regions(layout) mag.save_layout(d, layout) background_job_update(job_id, stage="articles", detail="Detecting articles") @@ -1201,6 +1208,11 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non outcome = spelling_agent_pass(d, layout, settings, page=page, language=issue_language) layout, spell_fixed = outcome["layout"], outcome["fixed"] spell_left = sorted({i["token"] for i in _spelling_findings(d, layout, page=page, language=issue_language)}) + # recover_missing_text (and the geometry fix above) can grow or add a box whose + # text now duplicates a smaller box's -- neither step re-checks for that itself. + # Catching it here, right before the audit reads the final boxes, is what finds + # it (see the ingest-time call for the case this closes). + layout, _ = mag.clean_up_regions(layout, pages=[page]) step("measuring box coverage") row = mag.audit_page_coverage(d, layout, page) except Exception as exc: # noqa: BLE001 - one bad page must not stop the run diff --git a/tests/test_magazine_routes.py b/tests/test_magazine_routes.py index b5fed9a..9f683c8 100644 --- a/tests/test_magazine_routes.py +++ b/tests/test_magazine_routes.py @@ -1490,6 +1490,42 @@ class IngestSyncVisualRefreshTests(unittest.TestCase): shutil.rmtree(lib, ignore_errors=True) +class IngestSyncDuplicateRecheckTests(unittest.TestCase): + """The per-page geometry loop (seam mending, region growth) can merge a wider box out of + fragments clean_up_regions's one earlier call had no reason to flag yet -- confirmed live + on c't KI Wissen 2026 page 152, where an infographic caption fragment was left narrated + twice this way. Confirms clean_up_regions runs again after that loop, not only before it.""" + + def test_clean_up_regions_runs_again_after_the_geometry_loop(self): + from unittest.mock import patch, call + import routes.magazine as rm + with patch.object(rm, "_LIB", Path(tempfile.mkdtemp(prefix="ttvc-lib-"))) as lib: + d = lib / "doc1" + d.mkdir(parents=True) + (d / "source.pdf").write_bytes(b"x") + layout = {"pages": [{"page": 1, "width": 100, "height": 100}], "blocks": [], "images": []} + + def semantic(_layout, _settings, page=None, doc_dir=None, on_page=None): + on_page(1) + + with patch.object(rm.mag, "extract_pages", return_value=layout), \ + patch.object(rm, "_recognise_advertisements"), \ + patch.object(rm, "_recognise_semantic_layout", side_effect=semantic), \ + patch.object(rm.mag, "refresh_visual_regions"), \ + patch.object(rm.mag, "clean_up_regions", return_value=(layout, {})) as cleanup, \ + patch.object(rm, "_fix_page_geometry", return_value=(layout, {}, False)) as geometry, \ + patch.object(rm.mag, "save_layout"), \ + patch.object(rm.mag, "build_articles", return_value=[]), \ + patch.object(rm.mag, "save_articles"), \ + patch.object(rm, "background_job_update"), \ + patch.object(rm, "_write_meta"): + rm._ingest_sync("doc1", "job1", "eng", 150, None, None) + # Once before the geometry loop, once after it -- geometry itself ran in between. + self.assertEqual(cleanup.call_args_list, [call(layout), call(layout)]) + geometry.assert_called_once() + shutil.rmtree(lib, ignore_errors=True) + + class InterruptedIngestTests(unittest.TestCase): def test_an_ingest_killed_by_a_restart_is_reported_as_failed_not_running_for_ever(self): import asyncio @@ -1562,6 +1598,47 @@ class QualityRunJobTests(unittest.TestCase): self.assertFalse(done['result']['verified_with_vision']) self.assertIn('skipped', done['detail']) + def test_a_duplicate_created_by_recovering_missing_text_is_still_caught(self): + """clean_up_regions runs at the start of _fix_page_geometry -- but a later step, in the + same function or in recover_missing_text right after it, can still grow or add a box + whose text now duplicates another's (confirmed live on c't KI Wissen 2026 page 152: an + infographic caption fragment was left narrated twice this way). _fix_page_geometry is + stubbed out here so the test isolates the specific gap the fix closes: a duplicate that + appears only once recover_missing_text has run, after _fix_page_geometry's own check.""" + from unittest.mock import patch + from routes.magazine import _quality_run_sync + text = "Die Plattformen bilden das Zentrum der Kreisläufe." + big = {'id': 'wide', 'page': 1, 'type': 'body', 'bbox': [0, 0, 500, 100], + 'text': "Intro. More text after it.", 'reading_order': 0} # not yet a duplicate + small = {'id': 'frag', 'page': 1, 'type': 'body', 'bbox': [10, 10, 200, 20], + 'text': text, 'reading_order': 1} + before = {'pages': [{'page': 1}], 'blocks': [big, small]} + grown = {'pages': [{'page': 1}], 'blocks': [ + {**big, 'text': f"Intro. {text} More text after it."}, small]} # now contains it + seen_types: list[str] = [] + + def fake_audit(d, layout, page): + seen_types.append(next(b['type'] for b in layout['blocks'] if b['id'] == 'frag')) + return self._row(page) + + lib = Path(tempfile.mkdtemp(prefix='ttvc-qrun-dup-')) + self.addCleanup(shutil.rmtree, lib, True) + (lib / ('a' * 32)).mkdir() + with patch('routes.magazine.mag.load_layout', return_value=before), \ + patch('routes.magazine._fix_page_geometry', return_value=(before, self._geom_counts(), False)), \ + patch('routes.magazine.mag.recover_missing_text', return_value=(grown, 1)), \ + patch('routes.magazine.mag.audit_page_coverage', side_effect=fake_audit), \ + patch('routes.magazine.mag.save_layout'), \ + patch('routes.magazine._LIB', lib), \ + patch('routes.magazine.background_job_update'): + _quality_run_sync('a' * 32, 'job', {}) + self.assertEqual(seen_types, ['furniture']) + + @staticmethod + def _geom_counts(): + return {k: 0 for k in ('boxes_fitted', 'paragraphs_cut', 'pullquotes_found', + 'pictures_added', 'overlaps_fixed', 'seams_mended')} + class PageAdvertisementRouteTests(unittest.IsolatedAsyncioTestCase): async def test_mark_and_unmark_round_trip_and_404(self):