42 KiB
SOP — Magazine ingestion
Status: P0 complete. P1 complete. Studio -> Magazines has the full library/contents/review/read/podcast flow, including figures shown during playback and a cross-article play-all queue. Automatic sidebar exclusion, word-level highlighting (estimated, not measured) and driving mode -- originally P2 items -- are also in. Remaining P2: image descriptions, table/chart handling, EPUB 3 export, and a formal multi-engine OCR benchmark.
Why magazines are not books
The audiobook pipeline assumes one continuous text in reading order. A magazine breaks every part of that assumption: several unrelated articles share a page, columns run in parallel, adverts and sidebars interrupt body copy, and a feature routinely jumps twenty pages to its tail. Narrating a magazine page-by-page produces audio that walks out of an article mid-sentence and into an advert.
The canonical object is therefore the article, not the page. Everything below exists to reconstruct articles from a flat PDF.
Pipeline
Implemented in core/magazine.py; driven by
routes/magazine.py.
PDF
→ extract_pages() PyMuPDF: text blocks with bbox, font size/name, weight
→ vision_ocr_page() connected vision OCR engine, when a page needs OCR or its
text scores poorly (ocr_quality_score) -- native or
tesseract alike. Labels footer/title/page_number/caption
directly. Off unless configured; never fatal.
→ ocr_page() tesseract TSV fallback: no vision engine configured, or
the vision call failed/timed out/was unreachable
→ build_vision_blocks() vision regions -> Blocks; a direct label bypasses
classify_blocks entirely (see below)
→ snap_visual_bboxes() figure/chart/table/code boxes re-derived from the page
render, because grounding is the weakest part of a
vision engine's answer (see below)
→ classify_blocks() title / deck / byline / body / caption / pullquote /
sidebar / furniture / ad / figure -- for native/tesseract
blocks, and any vision region left ambiguous ("text")
→ mark_advert_pages() spread an "ADVERTISEMENT" label across its whole page
→ mark_pullquotes() demote display text that merely repeats body copy
→ assign_reading_order() column detection, then top-to-bottom within a column
→ cluster_running_furniture() fuzzy-match near-edge repeats tesseract's noisy
OCR defeated exact matching on (see below)
→ score/parse_contents() Stage A: locate and parse the table of contents
→ group_articles() Stage C: group blocks, link "continued on page N"
→ human review Stage D, in the UI — nothing is narrated until approved
The semantic grouping stage keeps a logical paragraph split into separate physical regions when it flows from one text column into the next. Joining the text is semantically correct, but joining the rectangles creates a large union box over unrelated copy between the two pieces (seen repeatedly on pages 111 and 114 of the measured c't issue). Fragments within one column are still merged; only a genuine column transition remains as two boxes.
Why server-side, not in the browser
Text extraction for plain PDFs runs client-side (PDF.js), and that stays. Layout analysis does not, for three reasons: PyMuPDF exposes real font metrics and embedded-image rectangles that PDF.js does not; OCR coordinates from tesseract merge with those bboxes in one coordinate space; and the result is expensive enough (minutes for an issue) that it must be computed once and persisted, not recomputed per browser session.
PDF.js remains responsible for rendering in the PDF view.
Docling/Unstructured were considered and rejected: they pull torch plus layout model weights (multi-GB, ARM64 wheel risk on the DGX Spark, GPU contention with the TTS stack) to improve on heuristics that already work on the cases we hit. Revisit only if real issues show grouping failures the heuristics cannot fix.
Decisions that are easy to get wrong
Font baseline is document-wide, never per page. classify_blocks judges
size as a ratio against base_size. A page carrying a full-bleed photograph and
one caption has no body copy of its own, so a per-page baseline makes that
caption its own reference and classifies it as body text. extract_pages
measures the baseline once across every block in the issue.
Reading order is column-major. Sorting blocks by y interleaves parallel
columns, splicing the left column's sentence into the right column's. Columns
are inferred from vertical whitespace (where blocks are absent), because
magazines change column count per page — a fixed grid is wrong. Blocks spanning
60% of the page width sort ahead of both columns, so a full-width headline precedes the body it introduces.
Column boundary detection excludes unusually wide blocks, and the gutter
threshold is narrower than it looks. Confirmed live on a real scanned c't
page, using the split-screen review's numbered boxes (built specifically to
make exactly this kind of error visible): a photo caption at 49.9% of the
page width — under the 60% "spans the page" threshold above, so not given
special handling — sat in the whitespace gap between two genuinely separate
columns and merged them into one detected column. Every block on the page
came back column=0, and reading order fell back to pure top-to-bottom,
splicing the two real columns' sentences together — the box numbered 6 in
review (a caption physically above the box numbered 1) was read after it.
Column boundaries are now detected from blocks close to the page's median
width only, and the real gutter measured on that page was itself only ~2.1%
of the page width, narrower than the 3.5% detect_columns started with — now
1.5%. A single block much narrower than the surrounding column (a stray
caption fragment) can still form a spurious extra column of its own; the
main failure (two real columns interleaved) is fixed, that residual
narrower one is not.
Furniture is identified by repetition, not position. A running head and a
headline both sit at the top of a page. What separates them is that the running
head recurs on many pages at the same edge; find_repeated_lines needs the
whole document, so classification waits until every page has been read.
Exact-match repetition is not enough once OCR is involved. Confirmed live
on a scanned c't magazine: the running footer "c't Wissen 2026" was read
differently by tesseract on nearly every page (cE KI-WISSEN, ce Wissen 2136, cA Wissen 2036, …), so find_repeated_lines's exact match never
recognised them as the same line — each became its own spurious title and
started a new article, taking a 28-page scan from a handful of real sections
to 26 near-1-per-page "articles". cluster_running_furniture fuzzy-matches
near-edge blocks (digit-stripped, difflib.SequenceMatcher similarity ≥ 0.6,
requiring ≥3 occurrences) as a second pass over the whole document. A vision
OCR engine that labels the footer directly (see below) makes this pass
unnecessary for the pages it covers, but tesseract-sourced pages still need
it.
OCR font size is a measurement for native text, an estimate for OCR text.
PyMuPDF gives an exact digital font size per span. Tesseract only gives a line
bounding box, and its own line SEGMENTATION is unreliable on a scanned
magazine page — ocr_page groups by paragraph (matching PyMuPDF's own
grouping) and uses the MEDIAN of the paragraph's line heights, not its total
bbox height, which would scale with line count instead of type size. A vision
OCR engine gives one box per paragraph with no per-line data at all, so its
"text"-labeled (ambiguous) blocks get a synthesized estimate instead
(estimate_font_size_from_area, derived from the box's area and the text's
length) — good enough to feed the same title/deck/pullquote thresholds, not a
real measurement.
A vision-engine label is trusted outright, never re-derived. When the
engine labels a region footer/title/page_number/caption directly
(build_vision_blocks), classify_blocks skips it (source == "vision_label") rather than reclassifying it from an estimated font size —
that would throw away a more direct signal than any heuristic can reconstruct
after the fact. Only an ambiguous "text" label (or one the engine invents)
goes through the normal classification pass.
A page is claimed only if it contributes. An advert between two features starts no article, so without an explicit check it gets swallowed by the preceding one and highlighted as part of it in the PDF view.
Pull quotes are demoted. Narrating one reads the same sentence twice a few seconds apart. A display-sized block whose text already appears in the body copy is a pull quote, not a deck.
Confidence, and why review is mandatory
| Signal | Confidence |
|---|---|
| Anchored by a contents entry | 0.90 |
| Started by a headline, no contents entry | 0.55 |
| Article whose blocks contain no body copy | ×0.4 |
Nothing is narrated until a human sets status: "approved". This is deliberate:
a five-second correction beats a 30-minute recording with a sidebar in the
middle. POST …/podcast refuses unapproved articles outright.
Storage
Magazines live in the Read Aloud store and are distinguished by
meta.json → kind: "magazine", so the existing library listing, cover, progress
and audio endpoints keep working.
<CONFIG_DIR>/reader_library/<doc_id>/
meta.json kind, title, issue, ingest status, default_voice,
podcast hosts, per-article progress
source.pdf the original
layout.json every classified block, with bbox + reading order
articles.json detected articles; status gates narration
pages/NNNN.png rendered pages (PDF view + OCR input)
images/*.png extracted figures ≥64px
podcast/<id>.json generated host scripts
audio/<idx>.mp3 synthesised audio (existing reader endpoints)
layout.json is the join table between audio, text, image and PDF coordinates:
every block carries page + bbox, so a sentence in the reflowed view can jump
to its source region on the rendered page.
Podcast mode
Per article, alongside narration — not instead of it. An approved article's
narratable blocks are sent to the configured LLM (llm_url / llm_model,
reusing _post_llm_chat_completion from routes/conversation.py) with a system
prompt that forbids inventing facts and demands strict JSON. The parser tolerates
a markdown fence and surrounding prose, and reassigns unknown host labels rather
than dropping the turn — an unrecognised label would otherwise leave a line with
no voice. 2–3 hosts, each bound to a voice the user picks; synthesis reuses the
multi-speaker path the audiobook casting already uses.
Frontend
static/js/magazine.js + static/sections/s-magazine.html, wired into the same
lazy-module system as the other sections (static/module-manifest.json,
static/loader.js, static/nav.js). Two independent section lists gate
visibility (loader.js fetches the HTML, nav.js decides navTo will reveal
it) — both needed updating; missing either one leaves the section perpetually
hidden even though its HTML loaded correctly.
State machine: library -> contents -> review -> read, plus a pane toggle
(reflow | pdf) within read. Sentences are split from narratable blocks
client-side, each tagged with its source blockId/page/bbox, so clicking a
sentence or switching to the PDF pane draws the highlight rectangle from the
same coordinate the layout pipeline recorded — no separate sync data.
Reuses /api/tts-preview for both narration and podcast turns; no new audio
pipeline. magazine_articles marks each article has_podcast so the read view
skips a doomed status probe when nothing was generated, rather than a routine
404 on every open.
The pane toggle is a button group with aria-pressed, not role="tab" —
this app's tab pattern would additionally require aria-selected, roving
tabindex and arrow-key navigation, none of which plain click switching
provides. role="tab" without that whole pattern fails WCAG 2.2 AA guidance
and, in practice, hides the buttons from getByRole('tab')-oriented tooling.
Images during playback
build_vision_blocks aside, a figure has no sentence of its own to attach
to. assign_images_to_articles (server) links each extracted image to the
article whose column it falls in, using the nearer article's blocks when two
share a page; image_reading_position gives it an approximate slot in the
page's reading order. The client (currentImageIndex in magazine.js) then
shows "the last image reached so far" as playback advances past its
position — no separate sync data beyond that one reading_order number.
Play all
magPlayAll builds state.queue from every approved article in listing
order and starts the first; magPlay's completion path advances the queue
automatically rather than stopping, so listening runs across articles
without returning to the contents screen. magRead(id, {fromStart: true})
is what makes this a fresh start rather than resuming a queued article's own
saved progress, which would otherwise interrupt a continuous listen.
Word-level highlighting is estimated, not measured
No forced-alignment or TTS timestamp source exists in this pipeline.
estimateWordTimings distributes a sentence's real, measured audio
duration (audio.duration, once loaded) across its words proportional to
character length — a common karaoke approximation, not true alignment, and
it can drift on a sentence with heavy punctuation-driven pauses. A checkbox
(#mag-karaoke-cb) lets it be turned off; sentence-level highlighting
(measured, not estimated — it just waits for audio.onended) always still
works regardless.
Driving mode
#mag-driving is a full-screen overlay, not a separate playback path: it
reuses whatever is already playing or queued, so entering it never
interrupts an article in progress. Two oversized controls (skip, play/pause)
and no other interactive surface, matching the "no interaction required"
behaviour the manual originally specified as a P2 goal.
Automatic sidebar/advert exclusion
mark_sidebars (server) reclassifies body text sitting in a column much
narrower than the page's regular grid as BLOCK_SIDEBAR; group_articles
excludes it from an article's narrated blocks the same way it already
excludes BLOCK_FURNITURE/BLOCK_AD. Full-width elements (a spanning
headline) are excluded from the column-width analysis itself — left in, one
bridges two genuinely separate columns into a single merged span and hides
a real narrow sidebar column entirely (confirmed by a failing test before
the fix). This is a geometry heuristic, not a certainty: review still shows
every sidebar block (unticked by default) and it can be included with one
click — nothing is silently discarded.
Reading order (2026-09-21)
assign_reading_order reads a page the way a person does:
- Columns come from where column-width blocks start (
column_anchors: clustered left edges), not from whitespace between columns. The whitespace signal failed on real scanned pages: OCR boxes are loose, so a gutter shrank to 4-6 pt (page 38 of issue 204), and a pull-quote laid across the gutter bridged the columns outright -- every block came back as column 0 and the two columns were read interleaved. The column width is measured on body text only (a page-wide photo pushed it to 317 pt against real 210 pt columns), and columns never overlap. - Bands. Text across two or more columns in a strip of its own (a headline, a standfirst) is read before what lies under it. A box lying over the column boxes (a duplicate region) does not count. A photo with no running text beside it also separates the text above from the text below (page 176: a caption beside a screenshot above the columns). Paragraph gaps are never used as breaks: in neighbouring columns they often line up, and treating them as breaks interleaves the columns again (tried, measured, reverted).
- Short pull-quotes (≤ 260 chars, wrapped in quotation marks, or typed pullquote and short) are read after the page's running text; types alone are not trusted (on page 38 the real pull-quote was typed body, and a body paragraph was typed pullquote). Furniture last.
Measured over all 204 pages of a real issue with a text-only score -- a block that ends mid-sentence must be followed by one that continues it in lower case, a split word ("hun-" / "dertmal") by its other half: confirmed continuations 85 (old) → 89 (new). The two pages that score lower both carry duplicate boxes stacked over the same text, where no order is right until the boxes are fixed. The old detector had also counted full-page adverts and photo columns as text columns.
Existing issues get the new order with Fix reading order (POST .../reading-order): order
only, no OCR; text, boxes, types, articles and approvals unchanged. A page with order_locked (set
when the reader orders a page by hand) is always left alone -- by this action, by a type change and by
any other recompute.
Reading order by hand
PUT .../pages/{page}/reading-order {order: [ids]} sets a page's order (text blocks and figures in one
sequence, which is also where a figure appears during playback; ids left out follow in their current
order) and sets order_locked on the page. DELETE on the same path clears the lock and recomputes.
While locked: Fix reading order and type changes skip the page, a newly drawn region is appended last,
and re-OCR of the page clears the lock (its boxes are all new). UI: an editable position number per row,
and an Order by clicking mode (clicked boxes are numbered live; clicking does not tick/untick them).
Box clean-up (clean_up_regions)
Sized on a real 204-page issue (2 038 boxes) before any rule was written; every rule re-types, none
deletes (furniture + region_label in_picture / ocr_noise / duplicate: not narrated, visible,
reversible in the type picker).
| rule | found | left alone on purpose |
|---|---|---|
text ≤ 80 chars, ≥ 90% inside a picture → in_picture |
71 | longer text in a picture: included real running text caught in a loose frame |
≤ 2 alphanumeric characters → ocr_noise |
25 | longer short strings: a phone number, a price, a citation |
box ≥ 80% inside a bigger text box whose text contains ≥ 90% of it → duplicate |
38 | the other 333 overlapping pairs hold different text |
| author line running into the first sentence → split | 5 (4 split, 1 already on page) | a two-author byline; a sentence starting with "Von" |
The split sentence becomes its own paragraph (bbox: lower half of the author box), not text glued onto
"the next paragraph": on the five real cases the next paragraph in reading order was right once. The
new paragraph is inserted after the author line in every article that holds it; the author block keeps
original_text. Runs on import, on page re-OCR (that page), and via Clean up boxes (POST .../boxes/clean-up, writes layout.backup-<ts>.json first). Idempotent. _reorder_page now writes
only position/column back, so fields the Block model does not carry (e.g. original_text) survive.
Quick corrections from the review list
PATCH .../regions/{id} also takes text (trusted verbatim, like a hand-drawn region's typed text).
POST .../regions/{id}/merge-next joins a text region with the next one in reading order (first keeps id,
type and place; hyphenated line-end words are closed up; bbox is the union) and drops the folded region
from articles. POST .../regions/{id}/split {at} splits at a character position; the box is cut
horizontally in proportion to the text on each side, the second part gets a new id, is read directly
after the first and joins the same articles. Keyboard: ↑/↓ (j/k), Space, Enter (edit), R, M, Delete;
ignored while typing in the editor, a select or a position box; focus returns to the same row after the
list is redrawn.
Many boxes at once, and undo instead of confirmation
POST .../pages/{page}/regions/bulk {ids, action: delete|type, type?} applies one action to a selection
(≤ 500 ids, all on the page; validated before anything changes) with one undo point, and moves article
membership like the single-region routes. Every destructive route (delete, type change, text edit, merge,
split, bulk) writes undo.json = {token, label, layout, articles} after validating and before saving
(the files on disk still hold the old state), and returns the token; POST .../undo {token} restores both
files only if that token is still the latest (409 otherwise), so an old Undo can never roll back a newer
change. One level. After the change is saved the snapshot is sealed with a digest of layout.json +
articles.json (_seal_undo); undo also requires the current state to match that digest, so a later change that
records no undo point (drawing, fix reading order, clean-up, approve) makes the old Undo invalid instead of being
rolled back with it. UI: Select boxes mode (click or drag a rectangle; a box ≥ 50% inside is selected),
a bulk bar, and an Undo bar replacing the per-delete confirmation dialog (per NN/g-style guidance: frequent,
reversible actions get Undo, not "are you sure?").
fold_text_into_picture: when a region becomes a picture (drawn, or changed into one), text ≤ 80 chars and
≥ 90% inside it becomes in_picture -- the same limits as the clean-up. A real chart page had 13 legend and
axis-label boxes that had to be deleted one by one after the chart was marked as an image.
Redraw race: the review list redraws asynchronously after a change; actions that build UI in the list (the text editor) wait for a redraw in flight, and focus goes to the row focused when the redraw happens.
Region types from the review list
Every region in the review list has a type picker and a delete button. PATCH .../regions/{id} takes
type (REGION_TYPES: body, title, heading, deck, byline, caption, pullquote, sidebar, page_number,
source, ad, image). Between text kinds only the type changes. Text → image crops the box into a figure;
image → text reads the box with OCR (placeholder if nothing is recognised). Crossing between text and
image gives the region a new id, and the route moves its article membership over (figures and text
are listed separately on an article), so a converted figure stays in the article being reviewed. The
page's order is recomputed afterwards (unless hand-ordered), since the type decides where a region is read.
Live region boxes in the ingest preview
ingest_preview.json now carries stage ("reading" | "labelling") and boxes: each region in percent of the
page, its type (page_number for page-number furniture) and its reading-order number; pictures come from the
page's images (_ingest_preview_boxes, capped at 400). One writer, _write_ingest_preview, serves both stages:
the reader's progress hook (it now also passes the page size) and the labelling passes, which call on_page
after every page in a try/finally, so a page whose model call fails is still reported. The labelling pass holds
blocks as dicts; they are adapted to the attributes the text renderer reads (it previously failed silently on
dicts and wrote nothing). The browser fits the box layer to the part of the frame the image covers ("contain"
scaling leaves bars) and re-fits on resize; new boxes fade in (not with reduced motion); the panel stays
aria-hidden -- the status line carries the progress for screen readers.
Interrupted runs. A server restart kills the ingest job but left the magazine marked running, so the page
spun for ever. GET .../ingest/status now turns running + a finished job with an error into failed (with the
error), and the page offers a re-run.
Tests
tests/test_magazine.py— 100 tests on the pure geometry/classification logic: column reading order, the document-baseline regression, furniture, pull quotes, contents parsing, continuation linking, advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, the vision OCR quality score / region parsing / block building / HTTP call (network mocked), sidebar detection (including the full-width-headline regression), image-to-article assignment, the whole-page vision fallback, and the multi-engine fallback chain.tests/test_magazine_routes.py— 10 tests: path traversal rejection, review gating, LLM script parsing including malformed output.tests/magazine.spec.js— Playwright, backend mocked to the real response contract: library listing, review pre-selection, approval, sentence highlighting + PDF-page linking, the podcast approval gate, and a full-flow console-error check.
Both run without PyMuPDF: the I/O functions import it lazily so the logic worth testing needs no native dependency or sample PDF.
Verified end-to-end against a synthetic 8-page issue (contents page, two-column feature with a byline and a page-8 jump, full-page advert, second feature, reviews) and against an image-only rendering of the same file to exercise OCR.
Vision OCR escalation
Engines → OCR & Document Parsing lists layout-aware parsers (KDL Frontier Parser
nano, Baidu Unlimited-OCR) as connection-probed cards; those cards are still
presentation/health-check only. Ingestion itself now dispatches to whichever
OpenAI-compatible vision engine is configured (vision_ocr_url /
vision_ocr_model in Settings, vision_ocr_enabled gate) — vision_ocr_page
sends the rendered page image with a plain transcription prompt and parses the
engine's own labeled-region reply (parse_vision_ocr_regions). Escalation
triggers per page, not for the whole document: a page needing OCR at all
(native_chars < 120), or one whose already-extracted text — native
or tesseract — scores below ocr_quality_score's coarse garbage threshold.
Any failure (unreachable engine, timeout, unparseable output, or a decoding
loop — _looks_like_repetition_loop treats a short phrase repeated 8+ times
back-to-back as a failed call, confirmed live against a Qwen VL engine that
transcribed a chart as "das Produkt, das Produkt, ..." for its entire
response budget) returns None and the page falls back to tesseract exactly
as if vision OCR were not configured; ingestion never fails because of it.
Verified live against ocr-baidu-unlimited-ocr on a real 28-page scanned
c't magazine excerpt (Epson ES-580W, no scanner-side OCR) — every garbled
title tesseract had produced was replaced by the real headline:
| Before (tesseract) | After (vision OCR) |
|---|---|
K | soll Arbeit abnehmen… |
1. Irrtum: KI ist ein allgemeiner Jobkiller |
. | |
(furniture, correctly excluded) |
@)X |
(furniture, correctly excluded) |
Owns mee |
Das Team: „Hive Mind" frisst Konzentration |
Wie KI Entwickler-Jobs verandert |
Wie KI Entwickler-Jobs verändert |
Furniture detection went from 44 to 151 blocks on the same document, because
the engine labels running headers/footers/page numbers directly instead of
cluster_running_furniture having to recover them after the fact from noisy
OCR variants. 26 of 28 pages used the vision path; the remaining 2 fell back
to tesseract automatically (a transient failure or timeout — not investigated
further, since the fallback is exactly the intended behaviour). Article count
did not drop much (26 → 25): the residual "over-fragmentation" is now clearly
attributable to the document itself — a dossier of numbered "1. Irrtum" /
"2. Irrtum" sub-sections and an interview with c't: … question headers, all
of which read as headline-fallback article starts absent a contents page —
not to OCR quality, which the review screen is exactly the tool for.
ocr_quality_score is deliberately coarse: it discriminates "mostly symbol
noise" from "plausible prose" well enough to gate an expensive vision-OCR
call, but a single plausible-looking misread inside an otherwise normal
sentence (confirmed live: tesseract's "KI" → "K |" substitution, 7 of 8 words
still genuine German) scores just as high as clean text. Catching that class
of error needs a dictionary or language-model check, out of scope here.
Second engine and fallback ladder
vision_ocr_page_chain tries each configured (url, model) pair in order,
first usable result wins — primary (vision_ocr_url/_model), then fallback
(vision_ocr_fallback_url/_model), then tesseract if every vision engine
fails or is unreachable. KDL Frontier Parser nano is wired as the fallback,
confirmed live: it answers markedly faster than Baidu (~12s vs ~40s per page
on the same test page) but returns plain text with no coordinate markers
at all, unlike Baidu's labeled text [x0,y0,x1,y1]content regions.
_whole_page_region is what makes that still usable rather than discarded:
the whole response becomes one ambiguous full-page "text" region (bbox =
the entire rendered page, read via PIL), gated by the same
ocr_quality_score threshold used for native/tesseract text so a genuinely
bad KDL response isn't accepted either. Column and reading-order structure
is lost for a page handled this way — a real text improvement over
tesseract, but a worse layout result than Baidu's, which is why it is the
fallback and not the default.
A formal cross-engine accuracy benchmark remains TODO P2.
Figure bounds come from the render, not the engine
A vision OCR engine grounds a figure far less precisely than it transcribes
one. Measured over all 138 figure regions of c't KI Wissen 2026 (204 pages,
Baidu Unlimited-OCR): text boxes matched the page render closely enough to
use as-is, while figure boxes were routinely offset, undersized, or landed on
the neighbouring text column — page 23's card grid was reported at
x 0.24–0.54 of the page where the graphic actually runs 0.41–0.95, and
page 20's full-width illustration was cut off at a third of its height. The
transcribed contents of those same regions were right, which is the useful
signal: the model reliably says which graphic it means, so the render is
asked where that graphic ends.
snap_visual_bboxes() (core/magazine.py) does that, on a coarse grid of the
rendered page — page_ink_grid() reduces it to the inked fraction of each
10 px cell, which separates graphics from type cleanly: body copy covers
roughly a quarter of the cells it runs through, a photo or filled panel
covers nearly all of them.
Two passes, not one. A strict "solid ink" pass (_SNAP_SOLID = 0.55)
finds and repositions onto a densely inked anchor — a photo, a filled UI
panel, a card in a tiled grid — which behaves exactly like a photo and needs
nothing more. A bar chart's thin bars, hairline gridlines and axis labels
rarely reach that density: measured on page 18 of the same issue, most bar
rows covered 0.25–0.5 of their cell, so the strict pass alone anchored only
on the one fully-solid element (the vertical zero-line) and returned a box
cropped to a third of the real chart. A second, looser pass
(_grow_to_bounds, _SNAP_ANY_INK = 0.15) grows that anchor — or, lacking
one, the seed itself — through any real mark at all, stopping at a blocked
cell or _SNAP_MAX_BLANK_RUN consecutive blank ones, which is what recovers
the rest of a sparse chart without also absorbing a neighbouring column.
Connectivity is decided before an anchor is picked, not by iteratively
re-joining whichever component a seed happens to touch first. The solid
mask is dilated by _SNAP_GAP_CELLS (fenced by the same blocked cells at
every step) and labelled once; the seed then picks whichever merged
component it overlaps most. This was a real, measured bug in the first
version of this pass: selecting "the component with the most overlap"
directly from the seed's own raw components meant a small seed touching
only the chart's zero-line picked that line as the anchor, while re-running
the same function on the box it had just produced — now large enough to
also touch the page's dense axis-label strip — picked that instead and
shrank an already-correct box back down. Dilating first and labelling once
makes the choice a property of the page's ink alone, not of how large the
seed happens to be when it's asked.
A bounding box is a rectangle, not the component's actual shape, so
merging two patches that flank a stretch of unrelated text can still enclose
that text even though neither patch's own cells ever touched it (confirmed
on page 22: a chart's zero-line and its axis-label strip, merged correctly,
produced a box that also covered two lines of an unrelated paragraph sitting
between them). _trim_blocked_edges() pulls each edge back to the nearest
row or column blocked across most of the box's width or height, searched
outward from the box's own centre.
Adjacent figures fence each other, and that fencing must not oscillate.
Blocking every other figure's own seed area (not just text) is what stops
one figure's loose growth from bridging into a neighbour's territory just
because the gutter between them is narrow and mostly blank — two
head-and-shoulders portraits on page 31 sit only ~12pt (1.8 grid cells)
apart, well inside what blank-run tolerance alone would cross. But fencing
figure i against figure j's box, and vice versa, makes each one's result
depend on where the other currently sits: a mutual dependency a single pass
cannot resolve consistently, and in the worst measured case (two dark photos
sharing a caption strip, page 158) never settled at all, swinging by over
100pt call after call. snap_visual_bboxes() now re-seeds every visual on a
page from every other's last result for up to _SNAP_JOINT_ROUNDS rounds,
stopping early once nothing changes — which converges the common case and,
for the pair that doesn't truly converge, reaches the same deterministic
stopping point every time this function is called, because a later call
starts from exactly the state the loop stopped at rather than re-diverging.
Ambiguity between two seeds pointing at the same graphic is decided once,
fenced against the engine's original seeds before any of this iteration runs
— iterating first would let two disputed seeds mutually fence each other
into a tidy-looking split of one graphic instead of ever being recognised as
the same disputed claim.
Fences that keep the whole pass from making things worse, because a box confidently snapped onto the wrong thing is worse than a loose one the user can see:
- Text blocks, and every other figure's own seed, are excluded from the mask up front. They are the one trustworthy boundary on the page.
- Acceptance is explicit. An ungrown result must be either solidly inked
(
_SNAP_MIN_FILL) or framed on nearly all of its rows and columns — the second test is what admits a UI screenshot, which is white inside and inked only at its edge — and no result may shrink below_SNAP_MIN_SEED_AREAof the engine's box. A grown result skips the density re-test, which a chart's mostly-white interior would fail by construction, but still enforces the area floor. - A result must reproduce itself. Before being accepted, a candidate is
fed back through the same rules once more; if it does not come back
looking like itself (intersection over union below
_SNAP_STABLE_IOU), it is discarded rather than published as something the very next run would replace with a substantially different box. - Two figures pulled onto one graphic revert. Either the page is one picture the engine read as two or the reverse; neither reading can be trusted over the other.
Result on that issue: 98–119 of 147 figure regions corrected depending on
which pass of this work is measured, none made worse on visual inspection of
the sample, the rest left alone (pale charts on white, mostly, which the ink
test correctly declines — see the accompanying limitation in Manual.md).
The whole pass is idempotent by construction now, not merely by observation:
an already-settled box reproduces itself, and even the one pair confirmed to
never mathematically converge (page 158's dark photos) lands on the same
answer every time rather than continuing to drift. That is what lets it run
outside ingestion: refresh_visual_regions() re-snaps on every re-detection,
and POST /api/magazine/{id}/figures/refine re-snaps a whole issue as a
background job, so a document ingested before this pass existed — or before
a later fix to it — is corrected without paying for its transcription a
second time. It replaced refine_picture_bounds(), which only recognised
coloured panels and had no text fence.
The snap runs before crop_visual_regions() in both OCR paths: the figure
files shown during playback are cropped from these boxes, so a box corrected
late would leave a mis-framed image on disk.
The Active OCR Engine panel (Settings → Engines → OCR & Document Parsing)
mirrors the Active Language Model panel exactly — same endpoint/model
selects, same /api/conversation/llm-models fetch (it just proxies to
whatever URL it's given, not hardcoded to the LLM setting) — because an OCR
engine here is the same shape of problem: an OpenAI-compatible endpoint plus
a model name, just applied to vision_ocr_url/vision_ocr_model instead of
llm_url/llm_model. Local engine cards (#dc-grid-ocr) were already
supported by renderLocalContainers — that generic code handles all four
roles (tts/stt/llm/ocr) identically — but the OCR page had no
.s-engines-page[data-page="ocr"] markup at all until this change, so
$('dc-grid-ocr') was always null and the whole card grid silently never
rendered. Confirmed live: the page now shows both engines as ready-made
local cards (Connect/Start/Restart/Use as OCR/Show code snippet), matching
the LLM page, with no "Add Custom" dialog needed for either.
A new figure the model invents is screened against the page's own ink
The semantic-layout call is allowed to propose a picture/chart/table region
with block_ids: [] -- a graphic OCR never produced a block for. Confirmed
on a real issue: it sometimes drew that box over a column gap and the text
sitting in it instead -- a table-of-contents heading and its intro paragraph
in one case, a photo caption's column in another -- which buried real,
already-recognised paragraphs under an empty "Figure" placeholder rather
than adding a picture that was actually missing.
region_looks_like_a_picture() (core/magazine.py) screens every such
invented region against page_ink_grid_points() before it is created:
mean ink coverage over the proposed box's own area must reach
_INVENTED_VISUAL_MIN_INK (0.4). Measured on the confirmed false positives
and true positives of the same issue, the two cases do not come close to
overlapping -- 0.10-0.14 for the invented regions that were actually text,
0.56-0.97 for real photos, charts and tables -- so this is a wide floor, not
a fine-tuned one. apply_semantic_regions() takes page_ink as an optional
parameter (None skips the check, e.g. in a unit test with no page render
to check against); _recognise_semantic_layout() computes it once per page
from the same render already loaded for the vision prompt.
This is a different signal from snap_visual_bboxes() above: that one
corrects a real graphic's loose boundary once OCR/the model has already
agreed something is there. This one decides whether the "something" should
have existed at all. A region that survives this check can still get a loose
box from the snap pass afterwards -- confirmed on the same issue, several
real photos kept a box shifted or oversized enough to miss half the actual
picture, which the snap pass's own stability check declined to correct
rather than risk publishing a box the next run would replace (see the
"pulled onto one graphic" fence above). That remaining imprecision is a
candidate for the model-based layout detection already on the TODO list
(PP-DocLayout gives a native picture bbox instead of an LLM's guess).
A workflow-pipeline nav bar was bleeding into unrelated pages
_wfUpdateHeaderNav (utils.js) injects the audiobook pipeline's "Previous /
Next" bar and step hint into every .section-head in the document, with
Studio as the only named exclusion — confirmed live: the Engines page (LLM/
STT/TTS/OCR/Image Generation all share one .section-head) showed a
"Previous … Cast Audiobook ›" bar and a "STEP 1/8" hint that made no sense
there, since the pipeline's three destinations are s-reader, s-library
and s-rehearser only. Changed from a blocklist (name each unrelated
section as it's noticed) to an allowlist (head.closest('#s-reader, #s-library, #s-rehearser')) so the same bleed-through cannot recur on
Magazines, Settings, or any future section.
Known limitations
- Sidebar detection (
mark_sidebars) is geometry-only (an anomalously narrow column) — it has no notion of a boxed border or background tint, so a false positive/negative is possible on an unusual layout. Review always shows every sidebar block and lets it be ticked back in with one click. - Word-level highlighting is an estimate (proportional to character length within the sentence's real measured audio duration), not true forced alignment — see "Word-level highlighting is estimated, not measured" above.
continued from page Nis parsed but not used for linking — only the forwardcontinued on page Nreference drives it.- Reviews-style pages with no headline are absorbed by the preceding article when the contents page does not list them.
- KDL Frontier Parser nano (the OCR fallback engine) returns no layout coordinates, so a page it handles loses column/reading-order structure even though its text is usable — see "Second engine and fallback ladder".
- No formal cross-engine OCR accuracy benchmark exists; the engine comparison above is one confirmed live test, not a systematic evaluation.
- Image descriptions, table/chart handling, and EPUB 3 Media Overlay export remain P2 and are not started.