tts-voice-creator-clone-and-design-2/architecture/ocr-porting-prompt.md

Prompt: bring the tts-voice-creator OCR pipeline's hard-won lessons into another app

Copy everything below the line into the other project's agent. It is written for an AI coding agent that has never seen this repository. Reference implementation (read it, do not blindly copy it): /home/sparky/Docker/tts-voice-creator-clone-and-design-2 — core/magazine.py (pure logic, ~5,400 lines), routes/magazine.py (API and jobs), static/js/magazine.js (review UI), architecture/magazine-ingestion.md (the reasoning), scripts/bench_ocr.py (benchmark), tests/test_magazine.py (behaviour pinned by tests).

---

Your task

This app already runs OCR. Improve it by adopting the approach below where it fits this app's documents — first measure, then change. Do not port code wholesale: the reference app OCRs scanned German-language magazines for audio narration, so some rules are magazine-specific (articles, columns, sidebars) and some are general (everything about trusting, verifying, fitting and reviewing OCR output). Sort each item below into applies / does not apply / needs a decision for this app and tell me before starting the big ones.

Follow this project's own CLAUDE.md and conventions. Inspect the repo and current OCR code first; preserve existing behaviour that works; update docs and tests in the same change; commit per feature; never push without being asked.

Step 0 — measure before touching anything

Build a benchmark first, or every later claim is opinion.

1. Ground truth without hand labelling. Pick documents that have a real text layer (born-digital PDFs), render pages to images at the resolution the app really uses, run each OCR engine on the image, and score against the PDF's own text layer. Score on lower-cased word tokens (join line-end hyphenation first):
   • bag F1 — order-insensitive overlap of tokens;
   • sequence similarity — difflib.SequenceMatcher ratio over the token sequence (also rewards correct reading order);
   • seconds per page, and the engine's memory footprint (docker stats for a remote engine; process RSS for a library). Reference: scripts/bench_ocr.py. It scores ~0.87–0.88 F1 for every decent engine on clean pages: the ceiling is below 1.0 because a text layer contains running heads and sidebars. Compare engines to each other, never to 1.0.
2. A second benchmark for structure (boxes, reading order, region types) needs labelled pages; build a small one (10–20 pages) from a document you can correct by hand. Transcription accuracy and layout quality are different problems and were solved by different means here.
3. Record results as JSON (metrics only — no copyrighted or private page text) next to a short README of what was measured, when, on what.

Lesson from our run (2026-09-29, 6 pages of one issue): tesseract deu+eng F1 0.880 at 2.2 s/page; Baidu Unlimited-OCR (GPU vision model) 0.865 at 44 s/page and returned nothing on 1 of 6 pages; docling on CPU 0.867 at 8.9 s; PaddleOCR 3.x on CPU 0.881 at 36 s. Plain transcription accuracy was a wash across all of them, and the cheapest was 20× faster. What the heavy engines add is structure (region labels, furniture detection, layout order), not better characters. Check whether that holds on your documents before paying for a heavier engine.

The pipeline shape that worked

render page (110 dpi) ─► native text if the PDF has it ─► OCR only where needed ─► regions with bbox
   ─► trust engine labels ─► fit boxes to the ink ─► classify the ambiguous ─► reading order
   ─► clean-up (re-type, never delete) ─► quality audit ─► human review ─► only then use the text

1. Escalate per page, not per document

Use native text when the page has it. Escalate a page to OCR when it has almost no native text (we used < 120 characters) or when its existing text scores badly on a coarse plausibility score (ocr_quality_score: share of letters vs symbol noise, word shape). The score is deliberately coarse — it separates "mostly symbol noise" from "plausible prose" to gate an expensive call; it will not catch a single misread inside a good sentence (KI → K |). Do not pretend it does.

2. A fallback ladder, and every step may fail

Order: primary vision/layout engine → second engine → local classical OCR (tesseract) → keep whatever text exists. Any failure (unreachable, timeout, HTTP error, unparseable, repetition loop) returns None and the next rung runs. OCR must never be the reason a whole import fails. A second engine that returns plain text with no coordinates is still useful: wrap it as one full-page "text" region (_whole_page_region), gated by the same plausibility score, and accept that column structure is lost for that page.

3. Distrust the model's output in specific, testable ways

Each of these happened on real pages; each is a small function with tests:

• Repetition loops. A vision model transcribed a chart as "das Produkt, das Produkt, …" for its whole token budget. A short phrase repeated ≥ 8× back-to-back = failed call (_looks_like_repetition_loop), fall through.
• Hallucinated refusals. Over unreadable photos/screenshots an engine emits "The image is too blurry to recognize…". Drop a sentence that repeats across ≥ 3 regions (_looks_like_vision_refusal); on our issue this removed 29 and 122 hallucinated regions on re-OCR of three pages.
• Invented figures. A layout model may propose a picture/chart/table box where OCR found nothing. Screen every invented region against the page's ink density before creating it (region_looks_like_a_picture, mean ink ≥ 0.4). Measured: invented regions that were really text had 0.10–0.14 mean ink, real photos/charts/tables 0.56–0.97 — a wide gap, so the floor is not fine-tuned.
• Grounding vs transcription. Engines transcribe far better than they localise. See "figure bounds" below.
• Treat a direct engine label (footer, title, page number, caption) as authoritative and skip heuristic re-classification for it; only ambiguous "text" regions go through your own classifier.

4. Get figure bounds from the render, not from the engine

Measured over 138 figure regions: text boxes matched the render closely enough to use as-is, figure boxes were routinely offset, undersized, or on the neighbouring column — but the contents were right. The model knows which graphic; the render knows where it ends. snap_visual_bboxes:

• reduce the render to an ink grid (10 px cells → inked fraction per cell; body text ≈ 0.25, a photo ≈ 1.0);
• two passes: a strict solid-ink pass (cell ≥ 0.55) anchors on photos/panels; a loose pass (any mark ≥ 0.15, ≤ 2 blank cells crossed) grows sparse charts (thin bars, gridlines) that the strict pass would crop;
• decide connectivity once (dilate the solid mask by 3 cells, label components, seed picks the component it overlaps most) — picking "the best component of the seed's own raw pieces" made the result depend on seed size and was a real bug (a correct box shrank on re-run);
• a rectangle is not the component's shape: trim edges back to the nearest row/column blocked across ≥ 70% of the width/height, or the box swallows unrelated text between two patches;
• text blocks and every other figure's seed are fences (excluded from the mask up front); neighbours re-seed each other for up to 6 rounds so the result stops oscillating and is deterministic;
• accept a result only if it is solidly inked (≥ 0.45) or framed on ~90% of its rows and columns (a UI screenshot is white inside), never shrinks below 30% of the seed, and reproduces itself when fed back (IoU ≥ 0.7);
• two figures pulled onto one graphic → revert both. A confidently wrong box is worse than a loose one the user sees. Result: 98–119 of 147 figure regions corrected, none worse on visual inspection; pale charts on white left alone. Make it idempotent and re-runnable on existing documents without re-paying for transcription.

5. Fit text boxes to their ink (small, bounded, geometry-only)

snap_text_boxes: if a side cuts through letters (ink continues just outside it where no other box is) grow it until the text ends, by at most ~16 px sideways (70 for headlines, 90 downward); trim empty margin back to the first inked line, leaving 2 px of air. Never touch a box a human drew or resized. Never change text. On the measured issue only 5 of 658 boxes (pages 1–59) needed it — which is the point: measure first, and keep it from ever making a good box worse. Related: mend_column_seams moves the shared boundary between two vertically stacked boxes to the nearest white gap of ≥ 2 blank pixel rows and reads only the newly covered strip into the box that gained it; re-check for duplicates afterwards because fitting can turn a neighbour into a duplicate.

6. Clean-up rules: re-type, never delete — and size each rule on real data first

clean_up_regions measured 2,038 boxes before any rule was written:

rule	found	left alone on purpose
text ≤ 80 chars and ≥ 90% inside a picture → in_picture	71	longer text in a picture: it was real running text caught in a loose frame
≤ 2 alphanumeric characters → ocr_noise	25	longer short strings: a phone number, a price, a citation
box ≥ 80% inside a bigger box whose text contains ≥ 90% of it → duplicate	38	the other 333 overlapping pairs hold different text — hiding one loses text
author line running into the first sentence → split	5	two-author bylines; a sentence starting with "Von"
Every rule re-types (not narrated/used, still visible, reversible in the UI) and is idempotent, runs on import and on
re-OCR, writes a timestamped backup first. Overlapping boxes with different text are **surfaced for a human, not
auto-fixed** — we generate a review list with a suggested action per pair.

7. Reading order (only if your documents have columns)

Sorting by y interleaves parallel columns. What worked: columns come from where column-width blocks start (clustered left edges), not from whitespace between columns (OCR boxes are loose; a 2 pt gutter and a pull quote across it defeated the whitespace method); measure column width on body text only; bands — a headline/standfirst spanning columns in a strip of its own is read first; never use paragraph gaps as breaks (they line up across columns and re-interleave them: tried, measured, reverted). Measured with a text-only score (a block ending mid-sentence must be followed by its continuation; a split word by its other half) over 204 pages: confirmed continuations 85 → 89. Let users lock a hand-ordered page (order_locked) so no recompute overwrites it.

8. Running furniture (headers, footers, page numbers)

Identify by repetition, not position (a running head and a headline both sit at the top). Exact-match repetition fails once OCR is involved (c't Wissen 2026 came back as cE KI-WISSEN, ce Wissen 2136, …), so add a fuzzy pass: near-edge blocks, digits stripped, SequenceMatcher ≥ 0.6, ≥ 3 occurrences. Without it one scan produced 26 bogus "articles" from footers. An engine that labels footers directly makes this unnecessary for its pages.

9. A whole-document quality pass that reports, and fixes only what is provable

Per page, read-only audit from the render: uncovered ink blobs outside every box (ignore page-edge margins ~3%, dust < 10 cells, rules and dotted leaders < 6 ink rows, unboxed ink within 10 pt of a picture); overlaps (text boxes sharing ≥ 30% of the smaller); clipped edges (ink in a 14 px strip just outside a box's side, ignoring what another box covers, ignoring strips that are pictures — ink density > 0.7); suspect_blocks. Independent completeness check: run local OCR as a second opinion over the page (recover_missing_text) and add missing text as new unticked regions, never replace boxes. Re-read dictionary-suspicious regions with the image engine and accept only a better reading (verify_ocr_recognition). Spelling: findings, not rewrites — a model may propose a fix only if it is a dictionary suggestion or a single known word within two edits, offsets re-verified against the current text. Run the whole thing automatically as the last import step under the same progress job; report failures instead of hiding them.

10. Human review is part of the design, and the UI must make errors visible

Nothing downstream (audio, export, search) uses unreviewed text. Review screen essentials, all of which paid off: region boxes drawn on the page image, colour-coded by type with a legend; numbered in reading order; per-region type picker, delete, merge-with-next, split-at-cursor, per-region re-OCR, inline text edit; keyboard navigation; bulk actions with undo instead of confirmation dialogs (one level, tokenised, sealed with a digest so a stale Undo can never roll back newer work); draw-a-missed-region; two-page spread; a live ingest preview that draws boxes as they are found (purely decorative — progress numbers live in a separate status line for screen readers). WCAG 2.2 AA: icon + text, never colour alone; labelled controls; keyboard alternative for every drag; reduced-motion respected.

Engineering rules that kept this maintainable

• Pure logic in one module, I/O at the edges. Geometry/classification takes plain data, so tests need no PDF, no engine, no network. Import PyMuPDF lazily. Mock HTTP for engine tests.
• Constants are named and commented with the measurement that set them (_SNAP_SOLID = 0.55 # ...). When you change one, re-measure.
• Every destructive action writes a backup or undo point first; writes are atomic; jobs are restart-safe and a "running" state whose job died is reported as failed, not spun forever.
• Idempotence is a feature: re-running any fix on an already-fixed document changes nothing.
• Per-document data stays out of git (copyrighted/private pages, quality reports quoting text).
• Security at the boundary: validate ids (server-generated hex only), bound upload/download sizes, timeouts on every engine call, no credentials in URLs or logs.
• Test both directions: for each heuristic add a test for the case it fixes and the case it must leave alone (the "left alone on purpose" column above came from failing tests, not foresight).

Order of work I recommend

1. Benchmark + baseline of the current engine(s) (Step 0). Report the numbers before changing anything.
2. Cheap, high-yield robustness: per-page escalation, fallback ladder, repetition-loop and refusal filters, engine labels trusted, plausibility score.
3. Box quality: fit text boxes to ink, figure bounds from the render, clean-up rules — each sized on real data first.
4. Audit + review list (overlaps for humans), then the review UI improvements that fit this app.
5. Reading order / furniture / articles — only if this app's documents have that structure.
6. Re-benchmark; report before/after with the same metrics, and list what did not help.

Deliverables

• A short plan classifying each item above (applies / does not apply / needs decision), with expected effort.
• Benchmark script + results (metrics only) + README in the repo.
• Each adopted item: implementation, unit tests (positive and negative), docs/manual/changelog updated, one commit.
• A final before/after table and an honest list of limitations.

What not to do

• Do not add a heavy OCR/vision engine because it is fashionable; our measurements found no accuracy gain over tesseract on clean pages and a 4–20× speed cost. Prove the gain on this app's documents.
• Do not auto-delete or auto-merge overlapping boxes with different text.
• Do not let a model's output edit text unless a deterministic check confirms the edit.
• Do not trust the engine's bounding boxes for graphics.
• Do not run installs of multi-GB models on a machine without checking free memory first (GPU and system memory can be one pool; a kill can take down unrelated services).