diff --git a/CHANGELOG.md b/CHANGELOG.md index edac640..e7a6314 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,18 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi --- +## [1.27.3] — 2026-09-12 + +### Fixed +- **Figure-boundary snapping: sparse charts recovered, adjacent figures no longer unstable.** + - A vision OCR engine's box often only lightly grazes a bar chart's thin bars, gridlines and axis labels — none of which reach the "solid ink" density a photo or UI panel does. A second, looser pass now grows the box through any real mark at all once a dense anchor (or the seed itself) is found, recovering charts that were previously left truncated at roughly a third of their real height. + - Replaced iterative "re-join whichever component the seed happens to touch" merging with connectivity decided up front (dilate, then label once): the previous approach was seed-size-dependent, so re-running the pass on an already-corrected (larger) box could pick a *different* anchor and shrink a good result back down. Confirmed and fixed on the same chart that exposed the original truncation. + - A merged bounding box can enclose a real paragraph sitting between two components even though neither component's own cells touched it; edges are now trimmed back to the nearest text wall found searching outward from the box's centre. + - Two nearby figures fencing each other (so one's growth can't bridge into the other's territory) made each one's result depend on the other's *current* box — a feedback loop that, for two dark photos sharing a caption strip, never settled and swung by over 100pt between runs. The pass now re-seeds every visual on a page from every other's last result for a bounded number of rounds, and a result is only accepted once it reproduces itself when fed back through the same rules — so it is idempotent by construction, not merely by observation, including for the one confirmed pair that cannot mathematically converge. + - Lowered the minimum seed/graphic overlap needed to reposition a badly-offset box, which was narrowly rejecting some genuine repositioning cases and forcing them to need a second "Fix figure boundaries" run before correcting at all. + +--- + ## [1.27.2] — 2026-09-12 ### Fixed diff --git a/PROJECT_MAP.md b/PROJECT_MAP.md index 69171a2..0a49d2a 100644 --- a/PROJECT_MAP.md +++ b/PROJECT_MAP.md @@ -79,10 +79,14 @@ which is the writable volume. - No formal cross-engine OCR accuracy benchmark exists (two engines wired: ocr-baidu-unlimited-ocr primary, KDL Frontier Parser nano fallback). - Figure boundaries are re-derived from the page render (`snap_visual_bboxes`) - because vision OCR grounds figures loosely; it corrected 78 of 138 figure - boxes on `c't KI Wissen 2026` and deliberately declines the rest. A pale - chart or line drawing on white is below its ink threshold and keeps the - engine's box — draw those by hand in review. + because vision OCR grounds figures loosely; it corrects roughly 100 of the + 147 figure regions on `c't KI Wissen 2026` and deliberately declines the + rest. A pale chart or line drawing on white is below its ink threshold and + keeps the engine's box — draw those by hand in review. The pass is + idempotent (safe to re-run via "Fix figure boundaries" any number of + times), including the one confirmed pair that cannot mathematically + converge — two adjacent figures sharing a dark caption strip settle on the + same answer every run rather than continuing to drift between two states. - No `npm run qa` aggregate command yet; Python tests run via `unittest`, browser tests via `playwright test`. - The app has no authentication for the browser UI and no per-user data diff --git a/VERSION b/VERSION index 5a4d22e..551d4f5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.27.2 \ No newline at end of file +1.27.3 \ No newline at end of file diff --git a/architecture/magazine-ingestion.md b/architecture/magazine-ingestion.md index 98ed956..11522bf 100644 --- a/architecture/magazine-ingestion.md +++ b/architecture/magazine-ingestion.md @@ -377,36 +377,102 @@ asked **where** that graphic ends. rendered page — `page_ink_grid()` reduces it to the inked fraction of each 10 px cell, which separates graphics from type cleanly: body copy covers roughly a quarter of the cells it runs through, a photo or filled panel -covers nearly all of them. The engine's box seeds a connected-component -search over the "solid" cells (scipy `ndimage.label`); components a gutter -apart that line up are re-joined, so a grid of cards or a chart beside its -key comes back as one figure rather than four. +covers nearly all of them. -Three fences keep it from making things worse, because a box confidently -snapped onto the wrong thing is worse than a loose one the user can see: +**Two passes, not one.** A strict "solid ink" pass (`_SNAP_SOLID = 0.55`) +finds and repositions onto a densely inked anchor — a photo, a filled UI +panel, a card in a tiled grid — which behaves exactly like a photo and needs +nothing more. A bar chart's thin bars, hairline gridlines and axis labels +rarely reach that density: measured on page 18 of the same issue, most bar +rows covered 0.25–0.5 of their cell, so the strict pass alone anchored only +on the one fully-solid element (the vertical zero-line) and returned a box +cropped to a third of the real chart. A second, looser pass +(`_grow_to_bounds`, `_SNAP_ANY_INK = 0.15`) grows that anchor — or, lacking +one, the seed itself — through any real mark at all, stopping at a blocked +cell or `_SNAP_MAX_BLANK_RUN` consecutive blank ones, which is what recovers +the rest of a sparse chart without also absorbing a neighbouring column. -- **Text blocks are excluded from the mask up front.** They are the one - trustworthy boundary on the page, and without them a figure happily - swallows the column beside it. -- **Acceptance is explicit.** The result must be either solidly inked +**Connectivity is decided before an anchor is picked, not by iteratively +re-joining whichever component a seed happens to touch first.** The solid +mask is dilated by `_SNAP_GAP_CELLS` (fenced by the same blocked cells at +every step) and labelled *once*; the seed then picks whichever merged +component it overlaps most. This was a real, measured bug in the first +version of this pass: selecting "the component with the most overlap" +directly from the seed's own raw components meant a *small* seed touching +only the chart's zero-line picked that line as the anchor, while re-running +the same function on the box it had just produced — now large enough to +also touch the page's dense axis-label strip — picked *that* instead and +shrank an already-correct box back down. Dilating first and labelling once +makes the choice a property of the page's ink alone, not of how large the +seed happens to be when it's asked. + +**A bounding box is a rectangle, not the component's actual shape**, so +merging two patches that flank a stretch of unrelated text can still enclose +that text even though neither patch's own cells ever touched it (confirmed +on page 22: a chart's zero-line and its axis-label strip, merged correctly, +produced a box that also covered two lines of an unrelated paragraph sitting +between them). `_trim_blocked_edges()` pulls each edge back to the nearest +row or column blocked across most of the box's width or height, searched +outward from the box's own centre. + +**Adjacent figures fence each other, and that fencing must not oscillate.** +Blocking every *other* figure's own seed area (not just text) is what stops +one figure's loose growth from bridging into a neighbour's territory just +because the gutter between them is narrow and mostly blank — two +head-and-shoulders portraits on page 31 sit only ~12pt (1.8 grid cells) +apart, well inside what blank-run tolerance alone would cross. But fencing +figure *i* against figure *j*'s box, and vice versa, makes each one's result +depend on where the other currently sits: a mutual dependency a single pass +cannot resolve consistently, and in the worst measured case (two dark photos +sharing a caption strip, page 158) never settled at all, swinging by over +100pt call after call. `snap_visual_bboxes()` now re-seeds every visual on a +page from every other's last result for up to `_SNAP_JOINT_ROUNDS` rounds, +stopping early once nothing changes — which converges the common case and, +for the pair that doesn't truly converge, reaches the same deterministic +stopping point every time this function is called, because a later call +starts from exactly the state the loop stopped at rather than re-diverging. +Ambiguity between two seeds pointing at the *same* graphic is decided once, +fenced against the engine's original seeds before any of this iteration runs +— iterating first would let two disputed seeds mutually fence each other +into a tidy-looking split of one graphic instead of ever being recognised as +the same disputed claim. + +Fences that keep the whole pass from making things worse, because a box +confidently snapped onto the wrong thing is worse than a loose one the user +can see: + +- **Text blocks, and every other figure's own seed, are excluded from the + mask up front.** They are the one trustworthy boundary on the page. +- **Acceptance is explicit.** An ungrown result must be either solidly inked (`_SNAP_MIN_FILL`) or framed on nearly all of its rows *and* columns — the - second test is what admits a UI screenshot, which is white inside and inked - only at its edge — and it may not shrink below `_SNAP_MIN_SEED_AREA` of the - engine's box. Anything else keeps the engine's box untouched. + second test is what admits a UI screenshot, which is white inside and + inked only at its edge — and no result may shrink below + `_SNAP_MIN_SEED_AREA` of the engine's box. A *grown* result skips the + density re-test, which a chart's mostly-white interior would fail by + construction, but still enforces the area floor. +- **A result must reproduce itself.** Before being accepted, a candidate is + fed back through the same rules once more; if it does not come back + looking like itself (intersection over union below `_SNAP_STABLE_IOU`), + it is discarded rather than published as something the very next run + would replace with a substantially different box. - **Two figures pulled onto one graphic revert.** Either the page is one picture the engine read as two or the reverse; neither reading can be trusted over the other. -Result on that issue: 78 of 138 boxes corrected, none made worse on visual -inspection of the sample, the rest left alone (pale charts on white, mostly, -which the ink test correctly declines). The pass is idempotent — an -already-correct box seeds the same graphic again — which is what allows it to -run outside ingestion: `refresh_visual_regions()` re-snaps on every -re-detection, and `POST /api/magazine/{id}/figures/refine` re-snaps a whole -issue as a background job, so a document ingested before this pass existed is -corrected without paying for its transcription a second time. It replaced -`refine_picture_bounds()`, which only recognised *coloured* panels and had no -text fence. +Result on that issue: 98–119 of 147 figure regions corrected depending on +which pass of this work is measured, none made worse on visual inspection of +the sample, the rest left alone (pale charts on white, mostly, which the ink +test correctly declines — see the accompanying limitation in `Manual.md`). +The whole pass is idempotent by construction now, not merely by observation: +an already-settled box reproduces itself, and even the one pair confirmed to +never mathematically converge (page 158's dark photos) lands on the same +answer every time rather than continuing to drift. That is what lets it run +outside ingestion: `refresh_visual_regions()` re-snaps on every re-detection, +and `POST /api/magazine/{id}/figures/refine` re-snaps a whole issue as a +background job, so a document ingested before this pass existed — or before +a later fix to it — is corrected without paying for its transcription a +second time. It replaced `refine_picture_bounds()`, which only recognised +*coloured* panels and had no text fence. The snap runs before `crop_visual_regions()` in both OCR paths: the figure files shown during playback are cropped from these boxes, so a box corrected diff --git a/core/magazine.py b/core/magazine.py index 39c01d7..fa3cb72 100644 --- a/core/magazine.py +++ b/core/magazine.py @@ -1154,6 +1154,24 @@ def build_vision_blocks(regions: list[dict[str, Any]], page_no: int, # from the type area by white gutters -- so the model's box is used only to # say WHICH graphic is meant and the render says where that graphic ends. # +# Two passes over the render, not one. A strict "solid ink" pass finds and +# repositions onto a densely inked anchor -- a photo, a filled UI panel, a +# card in a tiled grid -- exactly the way a photo behaves. But a bar chart's +# thin bars, hairline gridlines and axis labels rarely reach that density +# (confirmed on page 18 of the same issue: most bar rows measured 0.25-0.5 +# coverage against the 0.55 solid threshold, so the strict pass anchored +# only on the one fully-solid element -- the vertical zero-line -- and +# returned a box cropped to a third of the real chart). A second, looser +# pass grows that anchor -- or, lacking one, the seed itself -- through any +# real mark at all, which is what recovers the rest of a sparse chart. +# +# Both passes are fenced by the same `blocked` cells: real text, and (new) +# every *other* figure's own seed area, so the looser pass cannot bridge a +# real gutter between two adjacent figures the way a raw blank-run +# allowance could -- confirmed against two head-and-shoulders portraits on +# page 31 sitting only ~12pt (1.8 grid cells) apart, well inside what the +# blank-run tolerance alone would cross. +# # Deliberately conservative: every rule below can decline, and declining # leaves the model's box untouched. A wrong box the user can see and redraw # is a nuisance; a box confidently snapped onto the wrong thing is worse. @@ -1164,7 +1182,31 @@ _SNAP_GAP_CELLS = 3 # white gutter a tiled graphic is allowed to span _SNAP_MIN_FILL = 0.45 # an accepted box is this solidly inked ... _SNAP_MIN_FRAME = 0.9 # ... or framed on this share of its rows AND columns _SNAP_MIN_SEED_AREA = 0.30 # ... and never shrinks below this much of the seed -_SNAP_MIN_OVERLAP = 0.02 # a graphic must touch the seed by at least this much +_SNAP_MIN_OVERLAP = 0.005 # a graphic must touch the seed by at least this much -- + # low on purpose: a seed can be offset far enough + # that it barely grazes the real graphic's edge + # (see page 70 of c't KI Wissen 2026, where the + # 0.02 first drafted here missed a 3-of-220-cell + # graze by a hair and needed a second snap pass + # before it could reposition at all) +_SNAP_ANY_INK = 0.15 # the loose pass's "there is a real mark here" floor -- + # above the ~0.02-0.11 scan-noise floor measured in + # genuinely blank chart gaps, below the ~0.25+ every + # real bar/line/label measured at +_SNAP_MAX_BLANK_RUN = 2 # cells of true blank the loose pass may cross -- + # covers a chart's own internal row gaps (measured + # at 1, occasionally 2 cells) without relying on + # this alone to keep two adjacent figures apart +_SNAP_BLOCKED_WALL = 0.7 # a row/column blocked across this much of a + # candidate box is real text the box must not + # cross, even though merging separate ink + # components into one rectangle (see _snap_seed) + # can otherwise enclose it geometrically +_SNAP_JOINT_ROUNDS = 6 # rounds of re-seeding every visual on a page from + # each other's last result (see snap_visual_bboxes) + # -- enough to settle the common case and to reach + # a deterministic stopping point for the rare pair + # that does not truly converge def page_ink_grid(png_path: Path, cell: int = _SNAP_CELL_PX): @@ -1194,63 +1236,234 @@ def page_ink_grid(png_path: Path, cell: int = _SNAP_CELL_PX): rows, cell, cols, cell).mean(axis=(1, 3)) +_SNAP_STABLE_IOU = 0.7 # a result must reproduce at least this much of + # itself (intersection over union) when fed back + # in as its own seed, or it is discarded -- see + # _snap_seed's stability check + + def _snap_seed(coverage, seed: Sequence[int], blocked) -> list[int] | None: """Grow the graphic the seed cell-box points at, or None to keep the seed. - Cells already claimed by a text block are excluded up front: text boxes + A result is only trusted once it has proven it will not move again: + since `snap_visual_bboxes` is meant to be safe to run repeatedly (an + already-corrected box seeding the same graphic again), a candidate this + function itself would revise on a second look is exactly the case that + must not be published. Two adjacent dark photos sharing a caption strip + (page 158 of c't KI Wissen 2026) is the confirmed real case: which one + "owns" the ambiguous shared area flips depending on subtle seed-size + effects, so neither box ever settles. Declining -- keeping the model's + original box -- is safer than publishing a box the very next run would + replace with a substantially different one. + """ + box = _snap_seed_once(coverage, seed, blocked) + if box is None: + return None + settled = _snap_seed_once(coverage, box, blocked) + if settled is None: + return None + intersection = _cell_overlap(box, settled) + union = ((box[2] - box[0] + 1) * (box[3] - box[1] + 1) + + (settled[2] - settled[0] + 1) * (settled[3] - settled[1] + 1) + - intersection) + if union == 0 or intersection / union < _SNAP_STABLE_IOU: + return None + return box + + +def _snap_seed_once(coverage, seed: Sequence[int], blocked) -> list[int] | None: + """One application of the snap rules -- see _snap_seed for the public, + stability-checked entry point that wraps this. + + Cells already claimed by a text block -- or by another figure's own + seed area, see snap_visual_bboxes -- are excluded up front: text boxes come back from the same engine far more accurately than figure boxes (verified against the page render for every block of the issue above), so they are the one trustworthy fence against a figure swallowing the - column beside it. + column, or the neighbouring figure, beside it. """ + import numpy as np from scipy import ndimage solid = (coverage >= _SNAP_SOLID) & ~blocked - labels, count = ndimage.label(solid) - if not count: - return None - slices = ndimage.find_objects(labels) - boxes = [[s[1].start, s[0].start, s[1].stop - 1, s[0].stop - 1] - for s in slices] - sizes = ndimage.sum_labels(solid, labels, range(1, count + 1)) - + seed_area = max(1, (seed[2] - seed[0] + 1) * (seed[3] - seed[1] + 1)) + seed_mask = np.zeros(solid.shape, dtype=bool) sx0, sy0, sx1, sy1 = seed - seed_area = max(1, (sx1 - sx0 + 1) * (sy1 - sy0 + 1)) - overlaps = [_cell_overlap(b, seed) for b in boxes] - best = max(range(count), key=overlaps.__getitem__) - if overlaps[best] < _SNAP_MIN_OVERLAP * seed_area: - return None + seed_mask[sy0:sy1 + 1, sx0:sx1 + 1] = True - # A figure split by white gutters (a grid of cards, a chart beside its - # key) arrives as several components; re-join the ones that line up with - # what we have so far, then re-test -- joining one may bring the next - # within reach. - box = list(boxes[best]) - joined = {best} - growing = True - while growing: - growing = False - for i, other in enumerate(boxes): - if i in joined or sizes[i] < 4 or not _cells_adjacent(box, other): - continue - box = [min(box[0], other[0]), min(box[1], other[1]), - max(box[2], other[2]), max(box[3], other[3])] - joined.add(i) - growing = True + # A figure split by white gutters -- a grid of cards, or (as much as a + # dashed zero-line and a dense axis-label strip can be) a chart -- is + # several disconnected solid components. Bridging small gaps BEFORE + # labelling, rather than iteratively re-joining whichever component the + # seed happens to touch first, is what makes the result independent of + # the seed's own size: re-running this function on an already-corrected + # (and now larger) box must pick the same components it did the first + # time, not a different one that merely overlaps the new seed more by + # raw area. Confirmed unstable on page 18's chart before this was + # connectivity- rather than seed-driven -- the dense axis-label strip + # briefly out-competed the thin zero-line for "best overlap" once the + # seed grew to cover both, shrinking an already-correct box back down. + connected = solid + if _SNAP_GAP_CELLS > 0: + structure = ndimage.generate_binary_structure(2, 1) + for _ in range(_SNAP_GAP_CELLS): + connected = ndimage.binary_dilation(connected, structure=structure) & ~blocked + + anchor = None + labels, count = ndimage.label(connected) + if count: + overlaps = ndimage.sum_labels(seed_mask, labels, range(1, count + 1)) + best = max(range(count), key=lambda i: overlaps[i]) + if overlaps[best] >= _SNAP_MIN_OVERLAP * seed_area: + # The bounding box of this component's ORIGINAL solid cells, + # not its dilated footprint -- dilation only decides which + # solid patches count as one figure, it must not itself widen + # the box. + ys, xs = np.where(solid & (labels == best + 1)) + if xs.size: + anchor = [int(xs.min()), int(ys.min()), int(xs.max()), int(ys.max())] + # A bounding box is a rectangle, not the component's actual + # shape, so it can still enclose a blocked text wall lying + # between two merged patches even though neither patch's + # own cells ever touched it. + anchor = _trim_blocked_edges(anchor, blocked) + + # The solid pass alone stops here for a photo or a UI panel -- dense + # enough that anchor already is the whole figure. A chart's sparse bars + # and axis rules need the second, looser pass below to be included at + # all; without a solid anchor to grow from, grow straight from the seed + # instead of giving up outright, since the seed usually still overlaps + # at least part of the real graphic even when none of it is solid. + grow_from = anchor if anchor is not None else seed + any_ink = (coverage >= _SNAP_ANY_INK) & ~blocked + gx0, gy0, gx1, gy1 = grow_from + if not any_ink[gy0:gy1 + 1, gx0:gx1 + 1].any(): + # Nothing at all under the anchor/seed -- a blank margin, or a pale + # chart/white-on-white screenshot below even the loose floor (see + # the known limitation in Manual.md). Declining is safer than + # guessing. + if anchor is None: + return None + box = anchor + else: + box = _grow_to_bounds(coverage, blocked, grow_from) area = (box[2] - box[0] + 1) * (box[3] - box[1] + 1) if area < _SNAP_MIN_SEED_AREA * seed_area: return None - inside = coverage[box[1]:box[3] + 1, box[0]:box[2] + 1] - if inside.mean() >= _SNAP_MIN_FILL: - return box - # A screenshot or a boxed insert is mostly white inside but inked right - # around its edge, so a solid frame counts as a graphic too. - edge = solid[box[1]:box[3] + 1, box[0]:box[2] + 1] - if (edge.any(axis=1).mean() >= _SNAP_MIN_FRAME - and edge.any(axis=0).mean() >= _SNAP_MIN_FRAME): - return box - return None + + if box == grow_from: + # Growth found nothing beyond the anchor itself -- fall back to the + # original density test, which only ever runs here because a real + # solid anchor was required to reach this point. + if anchor is None: + return None + inside = coverage[box[1]:box[3] + 1, box[0]:box[2] + 1] + if inside.mean() >= _SNAP_MIN_FILL: + return box + # A screenshot or a boxed insert is mostly white inside but inked + # right around its edge, so a solid frame counts as a graphic too. + edge = solid[box[1]:box[3] + 1, box[0]:box[2] + 1] + if (edge.any(axis=1).mean() >= _SNAP_MIN_FRAME + and edge.any(axis=0).mean() >= _SNAP_MIN_FRAME): + return box + return None + # Growth extended the anchor/seed through real, connected marks fenced + # by text and other figures -- trust it without re-testing density, + # which a chart's mostly-white interior fails by construction. + return box + + +def _grow_to_bounds(coverage, blocked, box: Sequence[int]) -> list[int]: + """Extend a box outward through content too sparse for the solid test. + + Each of the four sides advances one cell at a time through any real + mark at all (`_SNAP_ANY_INK`), which is what a bar chart's thin bars, + hairline axis and small print register as even though none of them + individually reach the solid-ink threshold above. A side stops the + instant it reaches a blocked cell -- real text, or another figure's own + seed area -- or after `_SNAP_MAX_BLANK_RUN` consecutive cells with no + mark at all, whichever comes first, so growth cannot bridge a real + gutter between two adjacent figures or wander off into open margin. + """ + rows, cols = coverage.shape + x0, y0, x1, y1 = box + any_ink = (coverage >= _SNAP_ANY_INK) & ~blocked + + def scan(pos: int, step: int, limit: int, lo: int, hi: int, horizontal: bool) -> int: + blank_run = 0 + while True: + nxt = pos + step + if nxt < 0 or nxt >= limit: + break + line_blocked = blocked[nxt, lo:hi + 1] if horizontal else blocked[lo:hi + 1, nxt] + if line_blocked.any(): + break + line_ink = any_ink[nxt, lo:hi + 1] if horizontal else any_ink[lo:hi + 1, nxt] + if line_ink.any(): + blank_run = 0 + else: + blank_run += 1 + if blank_run > _SNAP_MAX_BLANK_RUN: + pos -= step * (blank_run - 1) # roll back the trailing blank run + break + pos = nxt + return pos + + new_y0 = scan(y0, -1, rows, x0, x1, True) + new_y1 = scan(y1, 1, rows, x0, x1, True) + new_x0 = scan(x0, -1, cols, y0, y1, False) + new_x1 = scan(x1, 1, cols, y0, y1, False) + return [min(x0, new_x0), min(y0, new_y0), max(x1, new_x1), max(y1, new_y1)] + + +def _trim_blocked_edges(box: Sequence[int], blocked) -> list[int]: + """Pull each edge of `box` back to the nearest real text wall inside it. + + Merging separate ink components into one bounding rectangle (see + _snap_seed) can enclose a text block that happens to sit between two + pieces of the same graphic, even though that block's own cells were + excluded from the ink search itself -- the merge only produces a + rectangle, not the components' actual shape, so the rectangle can still + geometrically span over blocked cells lying between them. Confirmed on + page 22 of c't KI Wissen 2026: a chart's candidate box reached two text + lines above the chart itself, because a small mark on the chart's own + top edge and the graphic's body were merged across a gap that also + happened to contain the tail of an unrelated paragraph. + + A row or column blocked across most of the box's width/height is + unambiguously real text, not a stray corner overlap, and is trusted + over the rectangle's own extent -- searched outward from the box's own + centre so a wall near one edge trims that edge without disturbing the + other three. + """ + x0, y0, x1, y1 = box + width, height = x1 - x0 + 1, y1 - y0 + 1 + sub = blocked[y0:y1 + 1, x0:x1 + 1] + row_frac = sub.mean(axis=1) + col_frac = sub.mean(axis=0) + + new_y0, new_y1 = y0, y1 + for r in range(height // 2, -1, -1): + if row_frac[r] >= _SNAP_BLOCKED_WALL: + new_y0 = y0 + r + 1 + break + for r in range(height // 2, height): + if row_frac[r] >= _SNAP_BLOCKED_WALL: + new_y1 = y0 + r - 1 + break + new_x0, new_x1 = x0, x1 + for c in range(width // 2, -1, -1): + if col_frac[c] >= _SNAP_BLOCKED_WALL: + new_x0 = x0 + c + 1 + break + for c in range(width // 2, width): + if col_frac[c] >= _SNAP_BLOCKED_WALL: + new_x1 = x0 + c - 1 + break + if new_y0 > new_y1 or new_x0 > new_x1: + return list(box) # trimming would invert the box -- leave it be + return [new_x0, new_y0, new_x1, new_y1] def _cell_overlap(a: Sequence[int], b: Sequence[int]) -> int: @@ -1259,20 +1472,6 @@ def _cell_overlap(a: Sequence[int], b: Sequence[int]) -> int: * max(0, min(a[3], b[3]) - max(a[1], b[1]) + 1)) -def _cells_adjacent(box: Sequence[int], other: Sequence[int]) -> bool: - """True when `other` is a gutter away from `box` and lines up with it.""" - gap = _SNAP_GAP_CELLS - if not (other[0] <= box[2] + gap and other[2] >= box[0] - gap - and other[1] <= box[3] + gap and other[3] >= box[1] - gap): - return False - # Side by side or stacked, not merely diagonal: a tile of the same figure - # shares most of a row or most of a column with it. - shared_rows = max(0, min(other[3], box[3]) - max(other[1], box[1]) + 1) - shared_cols = max(0, min(other[2], box[2]) - max(other[0], box[0]) + 1) - return (shared_rows >= 0.5 * min(other[3] - other[1] + 1, box[3] - box[1] + 1) - or shared_cols >= 0.5 * min(other[2] - other[0] + 1, box[2] - box[0] + 1)) - - def snap_visual_bboxes(png_path: Path, blocks: Sequence[Block], page_meta: dict[str, Any]) -> int: """Correct every visual block's bbox against the page render, in place. @@ -1304,30 +1503,84 @@ def snap_visual_bboxes(png_path: Path, blocks: Sequence[Block], min(cols - 1, max(0, int((x + w) * cells_x))), min(rows - 1, max(0, int((y + h) * cells_y)))] - blocked = np.zeros(coverage.shape, dtype=bool) + text_blocked = np.zeros(coverage.shape, dtype=bool) for b in blocks: if b.type in VISUAL_TYPES or not b.text.strip(): continue x0, y0, x1, y1 = to_cells(b.bbox) - blocked[y0:y1 + 1, x0:x1 + 1] = True + text_blocked[y0:y1 + 1, x0:x1 + 1] = True - snapped = [_snap_seed(coverage, to_cells(b.bbox), blocked) for b in visuals] - # Two figures pulled onto the same graphic means the page was read as - # one picture where the engine saw two (or the reverse); neither reading - # can be trusted over the other, so both keep their original box. - for i, a in enumerate(snapped): - for j in range(i + 1, len(snapped)): - c = snapped[j] - if not a or not c: + visual_seeds = [to_cells(b.bbox) for b in visuals] + + def blocked_for(i: int, current: list[list[int]]): + # A figure's loose growth pass (see _grow_to_bounds) must not bridge + # into a neighbouring figure's own territory just because the gutter + # between them is narrow and mostly blank -- two head-and-shoulders + # portraits on the same page can sit under 2 grid cells apart. + # Blocking every *other* figure's own current area, not just text, + # fences that gutter regardless of how little ink is in it. + blocked = text_blocked.copy() + for j, other in enumerate(current): + if j == i: continue - smaller = min((a[2] - a[0] + 1) * (a[3] - a[1] + 1), - (c[2] - c[0] + 1) * (c[3] - c[1] + 1)) - if _cell_overlap(a, c) > 0.5 * smaller: - snapped[i] = snapped[j] = None - a = None + ox0, oy0, ox1, oy1 = other + blocked[oy0:oy1 + 1, ox0:ox1 + 1] = True + return blocked + + def colliding_pairs(boxes: list[list[int]]) -> set[int]: + # Two figures pulled onto the same graphic means the page was read + # as one picture where the engine saw two (or the reverse); neither + # reading can be trusted over the other, so both keep their + # original box. + hit: set[int] = set() + for i, a in enumerate(boxes): + for j in range(i + 1, len(boxes)): + c = boxes[j] + smaller = min((a[2] - a[0] + 1) * (a[3] - a[1] + 1), + (c[2] - c[0] + 1) * (c[3] - c[1] + 1)) + if _cell_overlap(a, c) > 0.5 * smaller: + hit.add(i) + hit.add(j) + return hit + + # Ambiguity is decided once, up front, fenced against the engine's own + # ORIGINAL seeds -- not against whatever the iterative refinement below + # settles on. Iterating first and only then checking for collisions + # would let two seeds legitimately ambiguous about the same graphic + # mutually fence each other into a tidy, confident-looking split of it + # instead of ever being recognised as the same disputed claim. + first_round = [_snap_seed(coverage, seed, blocked_for(i, visual_seeds)) + for i, seed in enumerate(visual_seeds)] + ambiguous = colliding_pairs([box or seed for box, seed in zip(first_round, visual_seeds)]) + + # Fencing figure i against figure j's box, and vice versa, means each + # one's result depends on where the OTHER currently sits -- a mutual + # dependency a single pass cannot resolve consistently. Repeating the + # whole page's worth of visuals, each round seeded by the previous + # round's output, converges that to a joint fixed point in the common + # case, and -- for the rare pair that never truly settles (two dark + # photos sharing a caption strip, confirmed on page 158 of c't KI + # Wissen 2026) -- reliably lands on the SAME deterministic answer every + # time this function is called, because a later call starts from + # exactly the state this loop stops at rather than re-diverging. An + # already-ambiguous visual is frozen at its original seed throughout, + # both as a participant (it will be discarded anyway) and as a fence + # for its neighbours (its disputed reading must not move anyone else). + current = list(visual_seeds) + for _ in range(_SNAP_JOINT_ROUNDS): + nxt = [visual_seeds[i] if i in ambiguous else + (_snap_seed(coverage, seed, blocked_for(i, current)) or seed) + for i, seed in enumerate(current)] + if nxt == current: + break + current = nxt + + reverted = ambiguous | colliding_pairs(current) + final = [None if i in reverted or current[i] == visual_seeds[i] else current[i] + for i in range(len(current))] moved = 0 - for block, box in zip(visuals, snapped): + for block, box in zip(visuals, final): if box is None: continue bbox = [box[0] / cells_x, box[1] / cells_y, diff --git a/tests/test_magazine.py b/tests/test_magazine.py index ee875eb..b2b3d8c 100644 --- a/tests/test_magazine.py +++ b/tests/test_magazine.py @@ -1537,3 +1537,98 @@ class SnapVisualBboxesTests(unittest.TestCase): if __name__ == "__main__": unittest.main() + + +class SnapVisualBboxesStabilityTests(unittest.TestCase): + """Regressions from finishing the sparse-chart growth pass: an anchor + chosen independently of the seed's own size, and repeated calls landing + on the same answer instead of drifting or ping-ponging between two + adjacent figures. + """ + + PAGE_W, PAGE_H = 582.0, 678.0 + PX_W, PX_H = 890, 1037 + + def setUp(self): + self.dir = pathlib.Path(tempfile.mkdtemp()) + (self.dir / "pages").mkdir() + self.meta = {"page": 1, "width": self.PAGE_W, "height": self.PAGE_H} + + def tearDown(self): + shutil.rmtree(self.dir, ignore_errors=True) + + def _save(self, image, name="0001.png"): + path = self.dir / "pages" / name + image.save(path) + return path + + def _pt(self, px_box): + sx, sy = self.PAGE_W / self.PX_W, self.PAGE_H / self.PX_H + x0, y0, x1, y1 = px_box + return [x0 * sx, y0 * sy, (x1 - x0) * sx, (y1 - y0) * sy] + + def _chart_page(self): + """A dense anchor line (the 0% axis) with sparse 40%-filled 'bars' + beneath it -- individually below the solid threshold, matching the + measured coverage of real bar-chart rows (see snap_visual_bboxes' + module docstring).""" + from PIL import Image, ImageDraw + page = Image.new("RGB", (self.PX_W, self.PX_H), (255, 255, 255)) + draw = ImageDraw.Draw(page) + draw.rectangle((300, 100, 305, 400), fill=(20, 20, 20)) # dense axis + for y in range(110, 390, 20): + draw.rectangle((200, y, 600, y + 4), fill=(30, 30, 30)) # sparse bar + return page + + def test_sparse_chart_box_is_stable_across_repeated_snaps(self): + png = self._save(self._chart_page()) + # A seed only loosely overlapping the dense axis, the way a vision + # engine's own grounding does -- see page 18 of c't KI Wissen 2026. + seed = Block(id="f1", page=1, bbox=self._pt((250, 150, 400, 250)), + type=BLOCK_FIGURE, text="chart", source="vision_label") + moved1 = snap_visual_bboxes(png, [seed], self.meta) + settled = list(seed.bbox) + self.assertGreater(moved1, 0) + moved2 = snap_visual_bboxes(png, [seed], self.meta) + self.assertEqual(moved2, 0) + self.assertEqual(seed.bbox, settled) + + def test_reseeding_with_the_grown_box_does_not_shrink_it(self): + """The exact page-18 failure: anchor selection must not flip to a + different, unrelated component just because the seed grew larger.""" + png = self._save(self._chart_page()) + wide_seed = Block(id="f1", page=1, bbox=self._pt((190, 95, 610, 405)), + type=BLOCK_FIGURE, text="chart", source="vision_label") + before = list(wide_seed.bbox) + snap_visual_bboxes(png, [wide_seed], self.meta) + # A box already covering the whole chart must not be cut down. + self.assertGreaterEqual(wide_seed.bbox[2], before[2] * 0.9) + self.assertGreaterEqual(wide_seed.bbox[3], before[3] * 0.9) + + def _two_photos_page(self): + """Two solid panels a narrow (~2-cell) gutter apart -- the measured + gap between two head-and-shoulders portraits on page 31 of c't KI + Wissen 2026.""" + from PIL import Image, ImageDraw + page = Image.new("RGB", (self.PX_W, self.PX_H), (255, 255, 255)) + draw = ImageDraw.Draw(page) + draw.rectangle((100, 100, 380, 500), fill=(40, 60, 90)) + draw.rectangle((400, 100, 680, 500), fill=(90, 60, 40)) + return page + + def test_two_adjacent_figures_settle_without_oscillating(self): + png = self._save(self._two_photos_page()) + left = Block(id="f1", page=1, bbox=self._pt((130, 130, 350, 470)), + type=BLOCK_FIGURE, text="left", source="vision_label") + right = Block(id="f2", page=1, bbox=self._pt((430, 130, 650, 470)), + type=BLOCK_FIGURE, text="right", source="vision_label") + snap_visual_bboxes(png, [left, right], self.meta) + first = (list(left.bbox), list(right.bbox)) + snap_visual_bboxes(png, [left, right], self.meta) + second = (list(left.bbox), list(right.bbox)) + # Settle to within a small fraction of a grid cell, not bounce + # between substantially different splits of the shared gutter. + for a, b in zip(first[0] + first[1], second[0] + second[1]): + self.assertAlmostEqual(a, b, delta=8.0) + # And never merge into one box spanning both panels. + self.assertLess(left.bbox[0] + left.bbox[2], right.bbox[0] + 1.0)