diff --git a/CHANGELOG.md b/CHANGELOG.md index d27ce0f..564b62c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ## [Unreleased] +## [1.42.4] — 2026-09-22 + +### Fixed +- **Magazine ingestion: a proposed picture, chart or table sometimes wasn't one.** The semantic-layout pass can invent a new picture region for a graphic OCR missed; on a real issue it sometimes drew that box over a column gap and the text in it instead (a table-of-contents heading and intro paragraph in one case), burying real paragraphs under an empty "Figure" placeholder. A proposed region is now checked against the page render's own ink density before it is created — a real photo, chart or table is solidly inked, a text area is not — confirmed against the actual false positives and true positives found on that issue (0.10-0.14 vs 0.56-0.97 mean ink, no overlap). QA also spot-checked headline, paragraph, column, page-number and advertisement recognition across a sample of pages of that issue; no other errors found. + ## [1.42.3] — 2026-09-22 ### Fixed diff --git a/Manual.md b/Manual.md index 6048a14..e0d2432 100644 --- a/Manual.md +++ b/Manual.md @@ -113,7 +113,12 @@ turned into a podcast discussion between two or three voices you pick. real edges of the picture for long. The recognised-text panel shows the actual cropped picture, chart or table for the same reason, rather than a plain "Figure" placeholder, as soon as it has been cropped. The page - scan itself (a scanned magazine is one picture per page) is not framed. If a run is interrupted + scan itself (a scanned magazine is one picture per page) is not framed. + A proposed picture, chart or table is checked against the page render + itself before it is created, so a text area or column gap is not shown + as a picture. A real picture can still get a loose or shifted box in + some cases; use *Refine figures* (or re-run detection on the page) to + try snapping it again, or fix the box by hand on the review page. If a run is interrupted (for example when the server restarts), the status now says so instead of spinning forever; start it again with *Re-run detection*. 3. **Review the articles.** This step is not optional and cannot be skipped. diff --git a/VERSION b/VERSION index 9754920..6f7c8cd 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.42.3 +1.42.4 diff --git a/architecture/magazine-ingestion.md b/architecture/magazine-ingestion.md index 2f52395..ab3c5db 100644 --- a/architecture/magazine-ingestion.md +++ b/architecture/magazine-ingestion.md @@ -615,6 +615,40 @@ rendered. Confirmed live: the page now shows both engines as ready-made local cards (Connect/Start/Restart/Use as OCR/Show code snippet), matching the LLM page, with no "Add Custom" dialog needed for either. +### A new figure the model invents is screened against the page's own ink + +The semantic-layout call is allowed to propose a picture/chart/table region +with `block_ids: []` -- a graphic OCR never produced a block for. Confirmed +on a real issue: it sometimes drew that box over a column gap and the text +sitting in it instead -- a table-of-contents heading and its intro paragraph +in one case, a photo caption's column in another -- which buried real, +already-recognised paragraphs under an empty "Figure" placeholder rather +than adding a picture that was actually missing. + +`region_looks_like_a_picture()` (core/magazine.py) screens every such +invented region against `page_ink_grid_points()` before it is created: +mean ink coverage over the proposed box's own area must reach +`_INVENTED_VISUAL_MIN_INK` (0.4). Measured on the confirmed false positives +and true positives of the same issue, the two cases do not come close to +overlapping -- 0.10-0.14 for the invented regions that were actually text, +0.56-0.97 for real photos, charts and tables -- so this is a wide floor, not +a fine-tuned one. `apply_semantic_regions()` takes `page_ink` as an optional +parameter (`None` skips the check, e.g. in a unit test with no page render +to check against); `_recognise_semantic_layout()` computes it once per page +from the same render already loaded for the vision prompt. + +This is a *different* signal from `snap_visual_bboxes()` above: that one +corrects a real graphic's loose boundary once OCR/the model has already +agreed something is there. This one decides whether the "something" should +have existed at all. A region that survives this check can still get a loose +box from the snap pass afterwards -- confirmed on the same issue, several +real photos kept a box shifted or oversized enough to miss half the actual +picture, which the snap pass's own stability check declined to correct +rather than risk publishing a box the next run would replace (see the +"pulled onto one graphic" fence above). That remaining imprecision is a +candidate for the model-based layout detection already on the TODO list +(PP-DocLayout gives a native picture bbox instead of an LLM's guess). + ### A workflow-pipeline nav bar was bleeding into unrelated pages `_wfUpdateHeaderNav` (utils.js) injects the audiobook pipeline's "Previous / diff --git a/core/magazine.py b/core/magazine.py index 5215792..b511abd 100644 --- a/core/magazine.py +++ b/core/magazine.py @@ -1527,6 +1527,24 @@ def page_ink_grid(png_path: Path, cell: int = _SNAP_CELL_PX): rows, cell, cols, cell).mean(axis=(1, 3)) +def page_ink_grid_points(png_path: Path, page_meta: dict, cell: int = _SNAP_CELL_PX): + """page_ink_grid(), plus each cell's size in the page's own point units (as bboxes are stored + in), for a caller that has no reason to know the render's pixel size -- see + region_looks_like_a_picture, which screens a proposed visual region against this.""" + grid = page_ink_grid(png_path, cell) + if grid is None: + return None + try: + from PIL import Image + with Image.open(png_path) as im: + w_px, h_px = im.size + except Exception: + return None + if not w_px or not h_px or not page_meta.get("width") or not page_meta.get("height"): + return None + return grid, page_meta["width"] * cell / w_px, page_meta["height"] * cell / h_px + + _SNAP_STABLE_IOU = 0.7 # a result must reproduce at least this much of # itself (intersection over union) when fed back # in as its own seed, or it is discarded -- see @@ -2385,7 +2403,34 @@ def reocr_page(doc_dir: Path, layout: dict[str, Any], page_no: int, return layout -def apply_semantic_regions(blocks: list[dict], regions: list[dict], page_meta: dict | None = None) -> list[dict]: +_INVENTED_VISUAL_MIN_INK = 0.4 # see the guard below apply_semantic_regions; a real photo, chart + # or table measured 0.56-0.97 on confirmed real pages of the same + # issue, a text area the model mistook for one 0.10-0.14 -- wide + # margin either side of this floor + + +def region_looks_like_a_picture(bbox: Sequence[float], page_ink) -> bool: + """Whether a NEW visual region (see apply_semantic_regions) is inked densely enough, over its + own area, to plausibly be a photo, chart or table rather than a text area or column gap the + model mistook for one. `page_ink` is (coverage_grid, cell_w, cell_h) in the same page-point + units as `bbox`, as produced by page_ink_grid() -- or None to skip the check (no page render to + check against, e.g. in a unit test), which always passes. + """ + if page_ink is None: + return True + grid, cell_w, cell_h = page_ink + if grid is None or not cell_w or not cell_h: + return True + x, y, w, h = bbox + r0, r1 = max(0, int(y / cell_h)), min(grid.shape[0], int(math.ceil((y + h) / cell_h))) + c0, c1 = max(0, int(x / cell_w)), min(grid.shape[1], int(math.ceil((x + w) / cell_w))) + if r1 <= r0 or c1 <= c0: + return True # too small to sample a page-render cell -- err toward keeping it + return float(grid[r0:r1, c0:c1].mean()) >= _INVENTED_VISUAL_MIN_INK + + +def apply_semantic_regions(blocks: list[dict], regions: list[dict], page_meta: dict | None = None, + page_ink=None) -> list[dict]: """Group existing OCR fragments without allowing invented IDs or text.""" known = {b["id"]: b for b in blocks} used = set() @@ -2409,6 +2454,14 @@ def apply_semantic_regions(blocks: list[dict], regions: list[dict], page_meta: d visual_box = [x0 / 1000 * page_meta["width"], y0 / 1000 * page_meta["height"], (x1 - x0) / 1000 * page_meta["width"], (y1 - y0) / 1000 * page_meta["height"]] if not ids and kind in VISUAL_TYPES and visual_box: + # A "new" picture the model invents (block_ids: []) is meant for a graphic OCR never + # produced a block for. Confirmed on a real issue: it sometimes instead draws that box + # over a column gap and the text below it (a table-of-contents heading and its intro + # paragraph in one case) -- which would otherwise bury real paragraphs under an empty + # "Figure" placeholder. A real photo/chart/table is a densely inked area of the page + # render; that case is not, so page_ink screens it out before it is ever created. + if not region_looks_like_a_picture(visual_box, page_ink): + continue out.append(Block(id=f"p{page_meta['page']:04d}-visual-{len(out)}", page=page_meta['page'], bbox=visual_box, type=kind, text="", reading_order=len(out), source="llm_layout", region_label=label).to_json()) diff --git a/routes/magazine.py b/routes/magazine.py index 618701d..17a182d 100644 --- a/routes/magazine.py +++ b/routes/magazine.py @@ -720,6 +720,7 @@ def _recognise_semantic_layout(layout: dict, settings: dict, page: int | None = for b in batch], ensure_ascii=False)}]} page_meta = next((p for p in layout.get("pages", []) if p["page"] == page_no), None) png = doc_dir / "pages" / f"{page_no:04d}.png" if doc_dir else None + page_ink = mag.page_ink_grid_points(png, page_meta) if png and png.is_file() and page_meta else None if png and png.is_file(): prompt = payload["messages"][1]["content"] payload["messages"][1]["content"] = [ @@ -740,7 +741,7 @@ def _recognise_semantic_layout(layout: dict, settings: dict, page: int | None = isinstance(confidence, (int, float)) and .9 <= confidence <= 1: batch = [dict(b, type=mag.BLOCK_AD) for b in batch] regions = [{"type": "advertisement", "block_ids": [b["id"] for b in batch], "bbox": [0,0,1000,1000]}] - grouped = mag.apply_semantic_regions(batch, regions, page_meta) + grouped = mag.apply_semantic_regions(batch, regions, page_meta, page_ink=page_ink) ids = {b["id"] for b in batch} layout["blocks"] = [b for b in layout["blocks"] if b["id"] not in ids] + grouped except Exception: diff --git a/tests/test_magazine.py b/tests/test_magazine.py index b003b0d..569cec2 100644 --- a/tests/test_magazine.py +++ b/tests/test_magazine.py @@ -1825,6 +1825,90 @@ class DeleteRegionTests(unittest.TestCase): self.assertEqual(len(layout["blocks"]), 1) +class InventedVisualRegionScreenTests(unittest.TestCase): + """A semantic-layout call can propose a brand-new picture/chart/table region (block_ids: []) + for a graphic OCR never produced a block for. Confirmed on a real issue (c't KI Wissen 2026): + it sometimes instead draws that box over a column gap and the text below it, which would bury + real paragraphs under an empty "Figure" placeholder. region_looks_like_a_picture and + apply_semantic_regions' page_ink screen that out using the page render's own ink density -- + a real photo/chart/table is solid, a text area is not (confirmed: 0.56-0.97 vs 0.10-0.14 on + the actual false positives and true positives found on that issue).""" + + PAGE_W, PAGE_H = 582.0, 678.0 + PX_W, PX_H = 890, 1037 + FIGURE_PX = (400, 100, 850, 400) + TEXT_PX = (60, 100, 350, 500) + + def setUp(self): + from PIL import Image, ImageDraw + self.dir = pathlib.Path(tempfile.mkdtemp()) + (self.dir / "pages").mkdir() + page = Image.new("RGB", (self.PX_W, self.PX_H), (255, 255, 255)) + draw = ImageDraw.Draw(page) + draw.rectangle(self.FIGURE_PX, fill=(40, 90, 200)) + x0, y0, x1, y1 = self.TEXT_PX + for y in range(y0, y1, 12): + draw.rectangle((x0, y, x1, y + 3), fill=(20, 20, 20)) + self.png = self.dir / "pages" / "0001.png" + page.save(self.png) + self.meta = {"page": 1, "width": self.PAGE_W, "height": self.PAGE_H} + + def tearDown(self): + shutil.rmtree(self.dir, ignore_errors=True) + + def _pt(self, px_box): + sx, sy = self.PAGE_W / self.PX_W, self.PAGE_H / self.PX_H + x0, y0, x1, y1 = px_box + return [x0 * sx, y0 * sy, (x1 - x0) * sx, (y1 - y0) * sy] + + def test_a_dense_region_looks_like_a_picture(self): + from core.magazine import region_looks_like_a_picture, page_ink_grid_points + page_ink = page_ink_grid_points(self.png, self.meta) + self.assertIsNotNone(page_ink) + self.assertTrue(region_looks_like_a_picture(self._pt(self.FIGURE_PX), page_ink)) + + def test_a_sparse_text_region_does_not_look_like_a_picture(self): + from core.magazine import region_looks_like_a_picture, page_ink_grid_points + page_ink = page_ink_grid_points(self.png, self.meta) + self.assertFalse(region_looks_like_a_picture(self._pt(self.TEXT_PX), page_ink)) + + def test_no_page_ink_always_passes(self): + from core.magazine import region_looks_like_a_picture + self.assertTrue(region_looks_like_a_picture(self._pt(self.TEXT_PX), None)) + + def test_apply_semantic_regions_keeps_a_real_picture_and_drops_a_text_area(self): + from core.magazine import apply_semantic_regions, page_ink_grid_points + page_ink = page_ink_grid_points(self.png, self.meta) + + def permille(px_box): + x, y, w, h = self._pt(px_box) + return [x / self.PAGE_W * 1000, y / self.PAGE_H * 1000, + (x + w) / self.PAGE_W * 1000, (y + h) / self.PAGE_H * 1000] + + grouped = apply_semantic_regions([], [ + {"type": "image", "block_ids": [], "bbox": permille(self.FIGURE_PX)}, + {"type": "image", "block_ids": [], "bbox": permille(self.TEXT_PX)}, + ], self.meta, page_ink=page_ink) + self.assertEqual(len(grouped), 1) + self.assertEqual(grouped[0]["type"], "figure") + for got, want in zip(grouped[0]["bbox"], self._pt(self.FIGURE_PX)): + self.assertAlmostEqual(got, want, delta=1.0) + + def test_without_page_ink_both_are_kept_as_before(self): + from core.magazine import apply_semantic_regions + + def permille(px_box): + x, y, w, h = self._pt(px_box) + return [x / self.PAGE_W * 1000, y / self.PAGE_H * 1000, + (x + w) / self.PAGE_W * 1000, (y + h) / self.PAGE_H * 1000] + + grouped = apply_semantic_regions([], [ + {"type": "image", "block_ids": [], "bbox": permille(self.FIGURE_PX)}, + {"type": "image", "block_ids": [], "bbox": permille(self.TEXT_PX)}, + ], self.meta) + self.assertEqual(len(grouped), 2) + + class SnapVisualBboxesTests(unittest.TestCase): """A vision OCR engine grounds a figure loosely; the render is the truth.