# SOP — Magazine ingestion Status: **P0 complete. P1 complete.** Studio -> Magazines has the full library/contents/review/read/podcast flow, including figures shown during playback and a cross-article play-all queue. Automatic sidebar exclusion, word-level highlighting (estimated, not measured) and driving mode -- originally P2 items -- are also in. Remaining P2: image descriptions, table/chart handling, EPUB 3 export, and a formal multi-engine OCR benchmark. ## Why magazines are not books The audiobook pipeline assumes one continuous text in reading order. A magazine breaks every part of that assumption: several unrelated articles share a page, columns run in parallel, adverts and sidebars interrupt body copy, and a feature routinely jumps twenty pages to its tail. Narrating a magazine page-by-page produces audio that walks out of an article mid-sentence and into an advert. **The canonical object is therefore the article, not the page.** Everything below exists to reconstruct articles from a flat PDF. ## Pipeline Implemented in [`core/magazine.py`](../core/magazine.py); driven by [`routes/magazine.py`](../routes/magazine.py). ``` PDF → extract_pages() PyMuPDF: text blocks with bbox, font size/name, weight → vision_ocr_page() connected vision OCR engine, when a page needs OCR or its text scores poorly (ocr_quality_score) -- native or tesseract alike. Labels footer/title/page_number/caption directly. Off unless configured; never fatal. → ocr_page() tesseract TSV fallback: no vision engine configured, or the vision call failed/timed out/was unreachable → build_vision_blocks() vision regions -> Blocks; a direct label bypasses classify_blocks entirely (see below) → snap_visual_bboxes() figure/chart/table/code boxes re-derived from the page render, because grounding is the weakest part of a vision engine's answer (see below) → classify_blocks() title / deck / byline / body / caption / pullquote / sidebar / furniture / ad / figure -- for native/tesseract blocks, and any vision region left ambiguous ("text") → mark_advert_pages() spread an "ADVERTISEMENT" label across its whole page → mark_pullquotes() demote display text that merely repeats body copy → assign_reading_order() column detection, then top-to-bottom within a column → cluster_running_furniture() fuzzy-match near-edge repeats tesseract's noisy OCR defeated exact matching on (see below) → score/parse_contents() Stage A: locate and parse the table of contents → group_articles() Stage C: group blocks, link "continued on page N" → human review Stage D, in the UI — nothing is narrated until approved ``` The semantic grouping stage keeps a logical paragraph split into separate physical regions when it flows from one text column into the next. Joining the text is semantically correct, but joining the rectangles creates a large union box over unrelated copy between the two pieces (seen repeatedly on pages 111 and 114 of the measured c't issue). Fragments within one column are still merged; only a genuine column transition remains as two boxes. ### Narration text: fusion, language, acronyms, spelling `article_text(..., fuse=True, glossary=...)` is the single source of narratable text for the reader, the podcast source and the MP3 export. - **Fusion** (`fuse_paragraph_blocks`). A body paragraph that flows into another column or page is separate regions; joining them before sentence splitting stops narration at the seam. `paragraph_continues` joins when the previous fragment has no sentence-ending punctuation and the next starts lower-case, the previous ends in a hyphen or comma, or it ends on an article/preposition/conjunction (German nouns are capitalised, so lower-case alone is not enough). Titles, headings, bylines, interview turns (`Label: ...`) and finished paragraphs never fuse. Captions and figure descriptions between fragments are emitted after the paragraph. A fused block keeps the first region's id/page/bbox plus `parts` (id, page, bbox, start, end offsets) so the player highlights and jumps to the region each sentence began in. A hyphen at the seam is dropped for a printed line break and kept for a real compound (`KI-` + `Modelle`), and the rebuilt word goes through the umlaut repair again. - **Language** is decided once per article (`detect_article_language`), then overridden per block only when that block is decisively the other language (`language_for_block`: at least 12 words and three hits for one language, none for the other). - **Acronyms.** `core/pronunciations.json` holds per-language expansions and letter-spelled lists; an article's `glossary` (in `articles.json`) is applied on top and wins. Unknown all-caps tokens (`find_unknown_acronyms`) are shown with their sentence; the LLM classifies them as expansion / letters / word (`parse_acronym_decisions`): only `expansion` carries model text, `letters` and `word` are produced server-side, and answers for acronyms that were not asked about are dropped. The displayed text is never altered — only `spoken_text`. - **Spelling** (`core/spelling.py`). Findings, not rewrites: a word unknown in its language and in the other one, not a compound of known parts (`is_known_compound`), not an acronym or camelCase brand, not in the issue's `dictionary.json`, and not recurring three times. `accept_agent_fix` is the only path by which a model's answer edits text: the replacement must be a dictionary suggestion or a single known word within two edits, and offsets are re-verified against the current text (`apply_fix`). The whole-issue quality run applies the same agent per page and reports remaining words per page (`spelling`, `spelling_fixed`). ### Why server-side, not in the browser Text extraction for plain PDFs runs client-side (PDF.js), and that stays. Layout analysis does not, for three reasons: PyMuPDF exposes real font metrics and embedded-image rectangles that PDF.js does not; OCR coordinates from tesseract merge with those bboxes in one coordinate space; and the result is expensive enough (minutes for an issue) that it must be computed once and persisted, not recomputed per browser session. PDF.js remains responsible for *rendering* in the PDF view. Docling/Unstructured were considered and rejected: they pull torch plus layout model weights (multi-GB, ARM64 wheel risk on the DGX Spark, GPU contention with the TTS stack) to improve on heuristics that already work on the cases we hit. Revisit only if real issues show grouping failures the heuristics cannot fix. ## Decisions that are easy to get wrong **Font baseline is document-wide, never per page.** `classify_blocks` judges size as a ratio against `base_size`. A page carrying a full-bleed photograph and one caption has no body copy of its own, so a per-page baseline makes that caption its own reference and classifies it as body text. `extract_pages` measures the baseline once across every block in the issue. **Reading order is column-major.** Sorting blocks by `y` interleaves parallel columns, splicing the left column's sentence into the right column's. Columns are inferred from vertical whitespace (where blocks are *absent*), because magazines change column count per page — a fixed grid is wrong. Blocks spanning >60% of the page width sort ahead of both columns, so a full-width headline precedes the body it introduces. **Column boundary detection excludes unusually wide blocks, and the gutter threshold is narrower than it looks.** Confirmed live on a real scanned c't page, using the split-screen review's numbered boxes (built specifically to make exactly this kind of error visible): a photo caption at 49.9% of the page width — under the 60% "spans the page" threshold above, so not given special handling — sat in the whitespace gap between two genuinely separate columns and merged them into one detected column. Every block on the page came back `column=0`, and reading order fell back to pure top-to-bottom, splicing the two real columns' sentences together — the box numbered 6 in review (a caption physically above the box numbered 1) was read *after* it. Column boundaries are now detected from blocks close to the page's median width only, and the real gutter measured on that page was itself only ~2.1% of the page width, narrower than the 3.5% `detect_columns` started with — now 1.5%. A single block much narrower than the surrounding column (a stray caption fragment) can still form a spurious extra column of its own; the main failure (two real columns interleaved) is fixed, that residual narrower one is not. **Furniture is identified by repetition, not position.** A running head and a headline both sit at the top of a page. What separates them is that the running head recurs on many pages at the same edge; `find_repeated_lines` needs the whole document, so classification waits until every page has been read. **Exact-match repetition is not enough once OCR is involved.** Confirmed live on a scanned c't magazine: the running footer "c't Wissen 2026" was read differently by tesseract on nearly every page (`cE KI-WISSEN`, `ce Wissen 2136`, `cA Wissen 2036`, …), so `find_repeated_lines`'s exact match never recognised them as the same line — each became its own spurious title and started a new article, taking a 28-page scan from a handful of real sections to 26 near-1-per-page "articles". `cluster_running_furniture` fuzzy-matches near-edge blocks (digit-stripped, `difflib.SequenceMatcher` similarity ≥ 0.6, requiring ≥3 occurrences) as a second pass over the whole document. A vision OCR engine that labels the footer directly (see below) makes this pass unnecessary for the pages it covers, but tesseract-sourced pages still need it. **OCR font size is a measurement for native text, an estimate for OCR text.** PyMuPDF gives an exact digital font size per span. Tesseract only gives a line bounding box, and its own line SEGMENTATION is unreliable on a scanned magazine page — `ocr_page` groups by paragraph (matching PyMuPDF's own grouping) and uses the MEDIAN of the paragraph's line heights, not its total bbox height, which would scale with line count instead of type size. A vision OCR engine gives one box per paragraph with no per-line data at all, so its "text"-labeled (ambiguous) blocks get a synthesized estimate instead (`estimate_font_size_from_area`, derived from the box's area and the text's length) — good enough to feed the same title/deck/pullquote thresholds, not a real measurement. **A vision-engine label is trusted outright, never re-derived.** When the engine labels a region `footer`/`title`/`page_number`/`caption` directly (`build_vision_blocks`), `classify_blocks` skips it (`source == "vision_label"`) rather than reclassifying it from an estimated font size — that would throw away a more direct signal than any heuristic can reconstruct after the fact. Only an ambiguous `"text"` label (or one the engine invents) goes through the normal classification pass. **A page is claimed only if it contributes.** An advert between two features starts no article, so without an explicit check it gets swallowed by the preceding one and highlighted as part of it in the PDF view. **Pull quotes are demoted.** Narrating one reads the same sentence twice a few seconds apart. A display-sized block whose text already appears in the body copy is a pull quote, not a deck. ## Confidence, and why review is mandatory | Signal | Confidence | |---|---| | Anchored by a contents entry | 0.90 | | Started by a headline, no contents entry | 0.55 | | Article whose blocks contain no body copy | ×0.4 | Nothing is narrated until a human sets `status: "approved"`. This is deliberate: a five-second correction beats a 30-minute recording with a sidebar in the middle. `POST …/podcast` refuses unapproved articles outright. ## Storage Magazines live in the Read Aloud store and are distinguished by `meta.json → kind: "magazine"`, so the existing library listing, cover, progress and audio endpoints keep working. ``` /reader_library// meta.json kind, title, issue, ingest status, default_voice, podcast hosts, per-article progress source.pdf the original layout.json every classified block, with bbox + reading order articles.json detected articles; status gates narration pages/NNNN.png rendered pages (PDF view + OCR input) images/*.png extracted figures ≥64px podcast/.json generated host scripts audio/.mp3 synthesised audio (existing reader endpoints) ``` `layout.json` is the join table between audio, text, image and PDF coordinates: every block carries `page` + `bbox`, so a sentence in the reflowed view can jump to its source region on the rendered page. ## Podcast mode Per article, alongside narration — not instead of it. An approved article's narratable blocks are sent to the configured LLM (`llm_url` / `llm_model`, reusing `_post_llm_chat_completion` from `routes/conversation.py`) with a system prompt that forbids inventing facts and demands strict JSON. The parser tolerates a markdown fence and surrounding prose, and reassigns unknown host labels rather than dropping the turn — an unrecognised label would otherwise leave a line with no voice. 2–3 hosts, each bound to a voice the user picks; synthesis reuses the multi-speaker path the audiobook casting already uses. ## Frontend `static/js/magazine.js` + `static/sections/s-magazine.html`, wired into the same lazy-module system as the other sections (`static/module-manifest.json`, `static/loader.js`, `static/nav.js`). Two independent section lists gate visibility (`loader.js` fetches the HTML, `nav.js` decides `navTo` will reveal it) — both needed updating; missing either one leaves the section perpetually hidden even though its HTML loaded correctly. State machine: `library -> contents -> review -> read`, plus a `pane` toggle (`reflow` | `pdf`) within `read`. Sentences are split from narratable blocks client-side, each tagged with its source `blockId`/`page`/`bbox`, so clicking a sentence or switching to the PDF pane draws the highlight rectangle from the same coordinate the layout pipeline recorded — no separate sync data. Reuses `/api/tts-preview` for both narration and podcast turns; no new audio pipeline. `magazine_articles` marks each article `has_podcast` so the read view skips a doomed status probe when nothing was generated, rather than a routine 404 on every open. The pane toggle is a button *group* with `aria-pressed`, not `role="tab"` — this app's tab pattern would additionally require `aria-selected`, roving `tabindex` and arrow-key navigation, none of which plain click switching provides. `role="tab"` without that whole pattern fails WCAG 2.2 AA guidance and, in practice, hides the buttons from `getByRole('tab')`-oriented tooling. ### Images during playback `build_vision_blocks` aside, a figure has no sentence of its own to attach to. `assign_images_to_articles` (server) links each extracted image to the article whose column it falls in, using the nearer article's blocks when two share a page; `image_reading_position` gives it an approximate slot in the page's reading order. The client (`currentImageIndex` in magazine.js) then shows "the last image reached so far" as playback advances past its position — no separate sync data beyond that one `reading_order` number. ### Play all `magPlayAll` builds `state.queue` from every *approved* article in listing order and starts the first; `magPlay`'s completion path advances the queue automatically rather than stopping, so listening runs across articles without returning to the contents screen. `magRead(id, {fromStart: true})` is what makes this a fresh start rather than resuming a queued article's own saved progress, which would otherwise interrupt a continuous listen. ### Word-level highlighting is estimated, not measured No forced-alignment or TTS timestamp source exists in this pipeline. `estimateWordTimings` distributes a sentence's *real, measured* audio duration (`audio.duration`, once loaded) across its words proportional to character length — a common karaoke approximation, not true alignment, and it can drift on a sentence with heavy punctuation-driven pauses. A checkbox (`#mag-karaoke-cb`) lets it be turned off; sentence-level highlighting (measured, not estimated — it just waits for `audio.onended`) always still works regardless. ### EPUB 3 export and sentence timings MP3 export renders one TTS call per spoken sentence and joins them, so every sentence's position is known exactly. `synthesize_article_audio` now records it (`write_audio_timings`) in `audio/.timings.json`: per narrated segment its character count and the `[start_ms, end_ms]` of each of its sentences, plus the total duration and whether the audio used interview voices. No text is stored. Older MP3s have no record. `core/epub_export.py` (pure packaging, no layout or OCR code) builds the book. `plan_sentences` splits each narrated block with the same sentence rule the MP3 and the reader use and assigns times in three tiers, reported per article: **exact** (record present, segment count and every segment's character count match the current text, and the sentence counts agree), **anchored** (paragraph start/end measured, its sentences share that span by length), **estimated** (no usable record, changed text, or interview audio: each sentence's share of the real duration by spoken length). The package: `mimetype` stored first, `container.xml`, an OPF 3.0 with `media:duration` per overlay and the active class, a nav document, one XHTML + one SMIL + the stored MP3 per article. All text is XML-escaped. `GET /api/magazine/{id}/epub[?article=id]` includes approved articles that have audio, builds into a temp file (removed after sending) and returns `application/epub+zip`. Validated with W3C EPUBCheck 5.2.1: 0 fatals, 0 errors, 0 warnings on a real two-article issue. `mp3_duration_seconds` uses mutagen and falls back to ffprobe. ID3 tagging failing because mutagen is missing no longer discards a finished synthesis. ### Sidebars in the reading view `article_asides(layout, article, all_articles)` lists the sidebar regions on an article's own pages that are not part of its narrated blocks; `GET .../articles/{id}/text` returns them as `asides` (`id, page, bbox, text, after`), where `after` is the last narrated region before the sidebar in reading order (None = before the first on its page). A page that any other article also lists is skipped, as are advertisement pages and empty boxes; jump cues are stripped. The client inserts each as an `