tts-voice-creator-clone-and-design-2/architecture/magazine-ingestion.md

# SOP — Magazine ingestion

Status: **P0 complete. P1 complete.** Studio -> Magazines has the full
library/contents/review/read/podcast flow, including figures shown during
playback and a cross-article play-all queue. Automatic sidebar exclusion,
word-level highlighting (estimated, not measured) and driving mode --
originally P2 items -- are also in. Remaining P2: image descriptions,
table/chart handling, EPUB 3 export, and a formal multi-engine OCR benchmark.

## Why magazines are not books

The audiobook pipeline assumes one continuous text in reading order. A magazine
breaks every part of that assumption: several unrelated articles share a page,
columns run in parallel, adverts and sidebars interrupt body copy, and a feature
routinely jumps twenty pages to its tail. Narrating a magazine page-by-page
produces audio that walks out of an article mid-sentence and into an advert.

**The canonical object is therefore the article, not the page.** Everything
below exists to reconstruct articles from a flat PDF.

## Pipeline

Implemented in [`core/magazine.py`](../core/magazine.py); driven by
[`routes/magazine.py`](../routes/magazine.py).

```
PDF
 → extract_pages()        PyMuPDF: text blocks with bbox, font size/name, weight
 → vision_ocr_page()      connected vision OCR engine, when a page needs OCR or its
                          text scores poorly (ocr_quality_score) -- native or
                          tesseract alike. Labels footer/title/page_number/caption
                          directly. Off unless configured; never fatal.
 → ocr_page()             tesseract TSV fallback: no vision engine configured, or
                          the vision call failed/timed out/was unreachable
 → build_vision_blocks()  vision regions -> Blocks; a direct label bypasses
                          classify_blocks entirely (see below)
 → classify_blocks()      title / deck / byline / body / caption / pullquote /
                          sidebar / furniture / ad / figure -- for native/tesseract
                          blocks, and any vision region left ambiguous ("text")
 → mark_advert_pages()    spread an "ADVERTISEMENT" label across its whole page
 → mark_pullquotes()      demote display text that merely repeats body copy
 → assign_reading_order() column detection, then top-to-bottom within a column
 → cluster_running_furniture()  fuzzy-match near-edge repeats tesseract's noisy
                          OCR defeated exact matching on (see below)
 → score/parse_contents() Stage A: locate and parse the table of contents
 → group_articles()       Stage C: group blocks, link "continued on page N"
 → human review           Stage D, in the UI — nothing is narrated until approved
```

### Why server-side, not in the browser

Text extraction for plain PDFs runs client-side (PDF.js), and that stays. Layout
analysis does not, for three reasons: PyMuPDF exposes real font metrics and
embedded-image rectangles that PDF.js does not; OCR coordinates from tesseract
merge with those bboxes in one coordinate space; and the result is expensive
enough (minutes for an issue) that it must be computed once and persisted, not
recomputed per browser session.

PDF.js remains responsible for *rendering* in the PDF view.

Docling/Unstructured were considered and rejected: they pull torch plus layout
model weights (multi-GB, ARM64 wheel risk on the DGX Spark, GPU contention with
the TTS stack) to improve on heuristics that already work on the cases we hit.
Revisit only if real issues show grouping failures the heuristics cannot fix.

## Decisions that are easy to get wrong

**Font baseline is document-wide, never per page.** `classify_blocks` judges
size as a ratio against `base_size`. A page carrying a full-bleed photograph and
one caption has no body copy of its own, so a per-page baseline makes that
caption its own reference and classifies it as body text. `extract_pages`
measures the baseline once across every block in the issue.

**Reading order is column-major.** Sorting blocks by `y` interleaves parallel
columns, splicing the left column's sentence into the right column's. Columns
are inferred from vertical whitespace (where blocks are *absent*), because
magazines change column count per page — a fixed grid is wrong. Blocks spanning
>60% of the page width sort ahead of both columns, so a full-width headline
precedes the body it introduces.

**Column boundary detection excludes unusually wide blocks, and the gutter
threshold is narrower than it looks.** Confirmed live on a real scanned c't
page, using the split-screen review's numbered boxes (built specifically to
make exactly this kind of error visible): a photo caption at 49.9% of the
page width — under the 60% "spans the page" threshold above, so not given
special handling — sat in the whitespace gap between two genuinely separate
columns and merged them into one detected column. Every block on the page
came back `column=0`, and reading order fell back to pure top-to-bottom,
splicing the two real columns' sentences together — the box numbered 6 in
review (a caption physically above the box numbered 1) was read *after* it.
Column boundaries are now detected from blocks close to the page's median
width only, and the real gutter measured on that page was itself only ~2.1%
of the page width, narrower than the 3.5% `detect_columns` started with — now
1.5%. A single block much narrower than the surrounding column (a stray
caption fragment) can still form a spurious extra column of its own; the
main failure (two real columns interleaved) is fixed, that residual
narrower one is not.

**Furniture is identified by repetition, not position.** A running head and a
headline both sit at the top of a page. What separates them is that the running
head recurs on many pages at the same edge; `find_repeated_lines` needs the
whole document, so classification waits until every page has been read.

**Exact-match repetition is not enough once OCR is involved.** Confirmed live
on a scanned c't magazine: the running footer "c't Wissen 2026" was read
differently by tesseract on nearly every page (`cE KI-WISSEN`, `ce Wissen
2136`, `cA Wissen 2036`, …), so `find_repeated_lines`'s exact match never
recognised them as the same line — each became its own spurious title and
started a new article, taking a 28-page scan from a handful of real sections
to 26 near-1-per-page "articles". `cluster_running_furniture` fuzzy-matches
near-edge blocks (digit-stripped, `difflib.SequenceMatcher` similarity ≥ 0.6,
requiring ≥3 occurrences) as a second pass over the whole document. A vision
OCR engine that labels the footer directly (see below) makes this pass
unnecessary for the pages it covers, but tesseract-sourced pages still need
it.

**OCR font size is a measurement for native text, an estimate for OCR text.**
PyMuPDF gives an exact digital font size per span. Tesseract only gives a line
bounding box, and its own line SEGMENTATION is unreliable on a scanned
magazine page — `ocr_page` groups by paragraph (matching PyMuPDF's own
grouping) and uses the MEDIAN of the paragraph's line heights, not its total
bbox height, which would scale with line count instead of type size. A vision
OCR engine gives one box per paragraph with no per-line data at all, so its
"text"-labeled (ambiguous) blocks get a synthesized estimate instead
(`estimate_font_size_from_area`, derived from the box's area and the text's
length) — good enough to feed the same title/deck/pullquote thresholds, not a
real measurement.

**A vision-engine label is trusted outright, never re-derived.** When the
engine labels a region `footer`/`title`/`page_number`/`caption` directly
(`build_vision_blocks`), `classify_blocks` skips it (`source ==
"vision_label"`) rather than reclassifying it from an estimated font size —
that would throw away a more direct signal than any heuristic can reconstruct
after the fact. Only an ambiguous `"text"` label (or one the engine invents)
goes through the normal classification pass.

**A page is claimed only if it contributes.** An advert between two features
starts no article, so without an explicit check it gets swallowed by the
preceding one and highlighted as part of it in the PDF view.

**Pull quotes are demoted.** Narrating one reads the same sentence twice a few
seconds apart. A display-sized block whose text already appears in the body copy
is a pull quote, not a deck.

## Confidence, and why review is mandatory

| Signal | Confidence |
|---|---|
| Anchored by a contents entry | 0.90 |
| Started by a headline, no contents entry | 0.55 |
| Article whose blocks contain no body copy | ×0.4 |

Nothing is narrated until a human sets `status: "approved"`. This is deliberate:
a five-second correction beats a 30-minute recording with a sidebar in the
middle. `POST …/podcast` refuses unapproved articles outright.

## Storage

Magazines live in the Read Aloud store and are distinguished by
`meta.json → kind: "magazine"`, so the existing library listing, cover, progress
and audio endpoints keep working.

```
<CONFIG_DIR>/reader_library/<doc_id>/
    meta.json           kind, title, issue, ingest status, default_voice,
                        podcast hosts, per-article progress
    source.pdf          the original
    layout.json         every classified block, with bbox + reading order
    articles.json       detected articles; status gates narration
    pages/NNNN.png      rendered pages (PDF view + OCR input)
    images/*.png        extracted figures ≥64px
    podcast/<id>.json   generated host scripts
    audio/<idx>.mp3     synthesised audio (existing reader endpoints)
```

`layout.json` is the join table between audio, text, image and PDF coordinates:
every block carries `page` + `bbox`, so a sentence in the reflowed view can jump
to its source region on the rendered page.

## Podcast mode

Per article, alongside narration — not instead of it. An approved article's
narratable blocks are sent to the configured LLM (`llm_url` / `llm_model`,
reusing `_post_llm_chat_completion` from `routes/conversation.py`) with a system
prompt that forbids inventing facts and demands strict JSON. The parser tolerates
a markdown fence and surrounding prose, and reassigns unknown host labels rather
than dropping the turn — an unrecognised label would otherwise leave a line with
no voice. 2–3 hosts, each bound to a voice the user picks; synthesis reuses the
multi-speaker path the audiobook casting already uses.

## Frontend

`static/js/magazine.js` + `static/sections/s-magazine.html`, wired into the same
lazy-module system as the other sections (`static/module-manifest.json`,
`static/loader.js`, `static/nav.js`). Two independent section lists gate
visibility (`loader.js` fetches the HTML, `nav.js` decides `navTo` will reveal
it) — both needed updating; missing either one leaves the section perpetually
hidden even though its HTML loaded correctly.

State machine: `library -> contents -> review -> read`, plus a `pane` toggle
(`reflow` | `pdf`) within `read`. Sentences are split from narratable blocks
client-side, each tagged with its source `blockId`/`page`/`bbox`, so clicking a
sentence or switching to the PDF pane draws the highlight rectangle from the
same coordinate the layout pipeline recorded — no separate sync data.

Reuses `/api/tts-preview` for both narration and podcast turns; no new audio
pipeline. `magazine_articles` marks each article `has_podcast` so the read view
skips a doomed status probe when nothing was generated, rather than a routine
404 on every open.

The pane toggle is a button *group* with `aria-pressed`, not `role="tab"` —
this app's tab pattern would additionally require `aria-selected`, roving
`tabindex` and arrow-key navigation, none of which plain click switching
provides. `role="tab"` without that whole pattern fails WCAG 2.2 AA guidance
and, in practice, hides the buttons from `getByRole('tab')`-oriented tooling.

### Images during playback

`build_vision_blocks` aside, a figure has no sentence of its own to attach
to. `assign_images_to_articles` (server) links each extracted image to the
article whose column it falls in, using the nearer article's blocks when two
share a page; `image_reading_position` gives it an approximate slot in the
page's reading order. The client (`currentImageIndex` in magazine.js) then
shows "the last image reached so far" as playback advances past its
position — no separate sync data beyond that one `reading_order` number.

### Play all

`magPlayAll` builds `state.queue` from every *approved* article in listing
order and starts the first; `magPlay`'s completion path advances the queue
automatically rather than stopping, so listening runs across articles
without returning to the contents screen. `magRead(id, {fromStart: true})`
is what makes this a fresh start rather than resuming a queued article's own
saved progress, which would otherwise interrupt a continuous listen.

### Word-level highlighting is estimated, not measured

No forced-alignment or TTS timestamp source exists in this pipeline.
`estimateWordTimings` distributes a sentence's *real, measured* audio
duration (`audio.duration`, once loaded) across its words proportional to
character length — a common karaoke approximation, not true alignment, and
it can drift on a sentence with heavy punctuation-driven pauses. A checkbox
(`#mag-karaoke-cb`) lets it be turned off; sentence-level highlighting
(measured, not estimated — it just waits for `audio.onended`) always still
works regardless.

### Driving mode

`#mag-driving` is a full-screen overlay, not a separate playback path: it
reuses whatever is already playing or queued, so entering it never
interrupts an article in progress. Two oversized controls (skip, play/pause)
and no other interactive surface, matching the "no interaction required"
behaviour the manual originally specified as a P2 goal.

### Automatic sidebar/advert exclusion

`mark_sidebars` (server) reclassifies body text sitting in a column much
narrower than the page's regular grid as `BLOCK_SIDEBAR`; `group_articles`
excludes it from an article's narrated blocks the same way it already
excludes `BLOCK_FURNITURE`/`BLOCK_AD`. Full-width elements (a spanning
headline) are excluded from the column-width analysis itself — left in, one
bridges two genuinely separate columns into a single merged span and hides
a real narrow sidebar column entirely (confirmed by a failing test before
the fix). This is a geometry heuristic, not a certainty: review still shows
every sidebar block (unticked by default) and it can be included with one
click — nothing is silently discarded.

## Tests

- [`tests/test_magazine.py`](../tests/test_magazine.py) — 100 tests on the pure
  geometry/classification logic: column reading order, the document-baseline
  regression, furniture, pull quotes, contents parsing, continuation linking,
  advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, the
  vision OCR quality score / region parsing / block building / HTTP call
  (network mocked), sidebar detection (including the full-width-headline
  regression), image-to-article assignment, the whole-page vision fallback,
  and the multi-engine fallback chain.
- [`tests/test_magazine_routes.py`](../tests/test_magazine_routes.py) — 10 tests: path
  traversal rejection, review gating, LLM script parsing including malformed
  output.
- [`tests/magazine.spec.js`](../tests/magazine.spec.js) — Playwright, backend mocked to the
  real response contract: library listing, review pre-selection, approval,
  sentence highlighting + PDF-page linking, the podcast approval gate, and a
  full-flow console-error check.

Both run without PyMuPDF: the I/O functions import it lazily so the logic worth
testing needs no native dependency or sample PDF.

Verified end-to-end against a synthetic 8-page issue (contents page, two-column
feature with a byline and a page-8 jump, full-page advert, second feature,
reviews) and against an image-only rendering of the same file to exercise OCR.

## Vision OCR escalation

Engines → OCR & Document Parsing lists layout-aware parsers (KDL Frontier Parser
nano, Baidu Unlimited-OCR) as connection-probed cards; those cards are still
presentation/health-check only. Ingestion itself now dispatches to whichever
OpenAI-compatible vision engine is configured (`vision_ocr_url` /
`vision_ocr_model` in Settings, `vision_ocr_enabled` gate) — `vision_ocr_page`
sends the rendered page image with a plain transcription prompt and parses the
engine's own labeled-region reply (`parse_vision_ocr_regions`). Escalation
triggers per page, not for the whole document: a page needing OCR at all
(`native_chars < 120`), or one whose already-extracted text — native
*or* tesseract — scores below `ocr_quality_score`'s coarse garbage threshold.
Any failure (unreachable engine, timeout, unparseable output) returns `None`
and the page falls back to tesseract exactly as if vision OCR were not
configured; ingestion never fails because of it.

**Verified live** against `ocr-baidu-unlimited-ocr` on a real 28-page scanned
c't magazine excerpt (Epson ES-580W, no scanner-side OCR) — every garbled
title tesseract had produced was replaced by the real headline:

| Before (tesseract) | After (vision OCR) |
|---|---|
| `K \| soll Arbeit abnehmen…` | `1. Irrtum: KI ist ein allgemeiner Jobkiller` |
| `. \|` | *(furniture, correctly excluded)* |
| `@)X` | *(furniture, correctly excluded)* |
| `Owns mee` | `Das Team: „Hive Mind" frisst Konzentration` |
| `Wie KI Entwickler-Jobs verandert` | `Wie KI Entwickler-Jobs verändert` |

Furniture detection went from 44 to 151 blocks on the same document, because
the engine labels running headers/footers/page numbers directly instead of
`cluster_running_furniture` having to recover them after the fact from noisy
OCR variants. 26 of 28 pages used the vision path; the remaining 2 fell back
to tesseract automatically (a transient failure or timeout — not investigated
further, since the fallback is exactly the intended behaviour). Article count
did not drop much (26 → 25): the residual "over-fragmentation" is now clearly
attributable to the document itself — a dossier of numbered "1. Irrtum" /
"2. Irrtum" sub-sections and an interview with `c't: …` question headers, all
of which read as headline-fallback article starts absent a contents page —
not to OCR quality, which the review screen is exactly the tool for.

`ocr_quality_score` is deliberately coarse: it discriminates "mostly symbol
noise" from "plausible prose" well enough to gate an expensive vision-OCR
call, but a single plausible-looking misread inside an otherwise normal
sentence (confirmed live: tesseract's "KI" → "K |" substitution, 7 of 8 words
still genuine German) scores just as high as clean text. Catching that class
of error needs a dictionary or language-model check, out of scope here.

### Second engine and fallback ladder

`vision_ocr_page_chain` tries each configured (url, model) pair in order,
first usable result wins — primary (`vision_ocr_url`/`_model`), then fallback
(`vision_ocr_fallback_url`/`_model`), then tesseract if every vision engine
fails or is unreachable. KDL Frontier Parser nano is wired as the fallback,
confirmed live: it answers markedly faster than Baidu (~12s vs ~40s per page
on the same test page) but returns **plain text with no coordinate markers
at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content` regions.
`_whole_page_region` is what makes that still usable rather than discarded:
the whole response becomes one ambiguous full-page "text" region (bbox =
the entire rendered page, read via PIL), gated by the same
`ocr_quality_score` threshold used for native/tesseract text so a genuinely
bad KDL response isn't accepted either. Column and reading-order structure
is lost for a page handled this way — a real text improvement over
tesseract, but a worse layout result than Baidu's, which is why it is the
fallback and not the default.

A formal cross-engine accuracy benchmark remains TODO P2.

The Active OCR Engine panel (Settings → Engines → OCR & Document Parsing)
mirrors the Active Language Model panel exactly — same endpoint/model
selects, same `/api/conversation/llm-models` fetch (it just proxies to
whatever URL it's given, not hardcoded to the LLM setting) — because an OCR
engine here is the same shape of problem: an OpenAI-compatible endpoint plus
a model name, just applied to `vision_ocr_url`/`vision_ocr_model` instead of
`llm_url`/`llm_model`. Local engine cards (`#dc-grid-ocr`) were already
supported by `renderLocalContainers` — that generic code handles all four
roles (`tts`/`stt`/`llm`/`ocr`) identically — but the OCR page had no
`.s-engines-page[data-page="ocr"]` markup at all until this change, so
`$('dc-grid-ocr')` was always null and the whole card grid silently never
rendered. Confirmed live: the page now shows both engines as ready-made
local cards (Connect/Start/Restart/Use as OCR/Show code snippet), matching
the LLM page, with no "Add Custom" dialog needed for either.

### A workflow-pipeline nav bar was bleeding into unrelated pages

`_wfUpdateHeaderNav` (utils.js) injects the audiobook pipeline's "Previous /
Next" bar and step hint into *every* `.section-head` in the document, with
Studio as the only named exclusion — confirmed live: the Engines page (LLM/
STT/TTS/OCR/Image Generation all share one `.section-head`) showed a
"Previous … Cast Audiobook ›" bar and a "STEP 1/8" hint that made no sense
there, since the pipeline's three destinations are `s-reader`, `s-library`
and `s-rehearser` only. Changed from a blocklist (name each unrelated
section as it's noticed) to an allowlist (`head.closest('#s-reader,
#s-library, #s-rehearser')`) so the same bleed-through cannot recur on
Magazines, Settings, or any future section.

## Known limitations

- Sidebar detection (`mark_sidebars`) is geometry-only (an anomalously narrow
  column) — it has no notion of a boxed border or background tint, so a
  false positive/negative is possible on an unusual layout. Review always
  shows every sidebar block and lets it be ticked back in with one click.
- Word-level highlighting is an *estimate* (proportional to character length
  within the sentence's real measured audio duration), not true forced
  alignment — see "Word-level highlighting is estimated, not measured" above.
- `continued from page N` is parsed but not used for linking — only the forward
  `continued on page N` reference drives it.
- Reviews-style pages with no headline are absorbed by the preceding article
  when the contents page does not list them.
- KDL Frontier Parser nano (the OCR fallback engine) returns no layout
  coordinates, so a page it handles loses column/reading-order structure
  even though its text is usable — see "Second engine and fallback ladder".
- No formal cross-engine OCR accuracy benchmark exists; the engine
  comparison above is one confirmed live test, not a systematic evaluation.
- Image descriptions, table/chart handling, and EPUB 3 Media Overlay export
  remain P2 and are not started.