tts-voice-creator-clone-and-design-2/architecture/ocr-porting-prompt.md

# Prompt: bring the tts-voice-creator OCR pipeline's hard-won lessons into another app

Copy everything below the line into the other project's agent. It is written for an AI coding agent that has
never seen this repository. Reference implementation (read it, do not blindly copy it):
`/home/sparky/Docker/tts-voice-creator-clone-and-design-2` — `core/magazine.py` (pure logic, ~5,400 lines),
`routes/magazine.py` (API and jobs), `static/js/magazine.js` (review UI), `architecture/magazine-ingestion.md`
(the reasoning), `scripts/bench_ocr.py` (benchmark), `tests/test_magazine.py` (behaviour pinned by tests).

---

## Your task

This app already runs OCR. Improve it by adopting the approach below **where it fits this app's documents** — first
measure, then change. Do not port code wholesale: the reference app OCRs *scanned German-language magazines* for
audio narration, so some rules are magazine-specific (articles, columns, sidebars) and some are general (everything
about trusting, verifying, fitting and reviewing OCR output). Sort each item below into *applies / does not apply /
needs a decision* for this app and tell me before starting the big ones.

Follow this project's own CLAUDE.md and conventions. Inspect the repo and current OCR code first; preserve
existing behaviour that works; update docs and tests in the same change; commit per feature; never push without
being asked.

## Step 0 — measure before touching anything

Build a benchmark first, or every later claim is opinion.

1. **Ground truth without hand labelling.** Pick documents that have a real text layer (born-digital PDFs), render
   pages to images at the resolution the app really uses, run each OCR engine on the *image*, and score against the
   PDF's own text layer. Score on lower-cased word tokens (join line-end hyphenation first):
   - *bag F1* — order-insensitive overlap of tokens;
   - *sequence similarity* — `difflib.SequenceMatcher` ratio over the token sequence (also rewards correct reading order);
   - seconds per page, and the engine's memory footprint (`docker stats` for a remote engine; process RSS for a library).
   Reference: `scripts/bench_ocr.py`. It scores ~0.87–0.88 F1 for every decent engine on clean pages: the ceiling is
   below 1.0 because a text layer contains running heads and sidebars. **Compare engines to each other, never to 1.0.**
2. **A second benchmark for structure** (boxes, reading order, region types) needs labelled pages; build a small one
   (10–20 pages) from a document you can correct by hand. Transcription accuracy and layout quality are different
   problems and were solved by different means here.
3. Record results as JSON (metrics only — no copyrighted or private page text) next to a short README of what was
   measured, when, on what.

**Lesson from our run (2026-09-29, 6 pages of one issue):** tesseract deu+eng F1 0.880 at 2.2 s/page; Baidu
Unlimited-OCR (GPU vision model) 0.865 at 44 s/page and returned nothing on 1 of 6 pages; docling on CPU 0.867 at 8.9 s;
PaddleOCR 3.x on CPU 0.881 at 36 s. **Plain transcription accuracy was a wash across all of them, and the cheapest was
20× faster.** What the heavy engines add is structure (region labels, furniture detection, layout order), not better
characters. Check whether that holds on your documents before paying for a heavier engine.

## The pipeline shape that worked

```
render page (110 dpi) ─► native text if the PDF has it ─► OCR only where needed ─► regions with bbox
   ─► trust engine labels ─► fit boxes to the ink ─► classify the ambiguous ─► reading order
   ─► clean-up (re-type, never delete) ─► quality audit ─► human review ─► only then use the text
```

### 1. Escalate per page, not per document
Use native text when the page has it. Escalate a page to OCR when it has almost no native text (we used < 120
characters) **or** when its existing text scores badly on a coarse plausibility score (`ocr_quality_score`: share of
letters vs symbol noise, word shape). The score is deliberately coarse — it separates "mostly symbol noise" from
"plausible prose" to gate an expensive call; it will **not** catch a single misread inside a good sentence
(`KI` → `K |`). Do not pretend it does.

### 2. A fallback ladder, and every step may fail
Order: primary vision/layout engine → second engine → local classical OCR (tesseract) → keep whatever text exists.
Any failure (unreachable, timeout, HTTP error, unparseable, repetition loop) returns `None` and the *next* rung runs.
OCR must never be the reason a whole import fails. A second engine that returns plain text with no coordinates is
still useful: wrap it as one full-page "text" region (`_whole_page_region`), gated by the same plausibility score, and
accept that column structure is lost for that page.

### 3. Distrust the model's output in specific, testable ways
Each of these happened on real pages; each is a small function with tests:
- **Repetition loops.** A vision model transcribed a chart as "das Produkt, das Produkt, …" for its whole token
  budget. A short phrase repeated ≥ 8× back-to-back = failed call (`_looks_like_repetition_loop`), fall through.
- **Hallucinated refusals.** Over unreadable photos/screenshots an engine emits "The image is too blurry to
  recognize…". Drop a sentence that repeats across ≥ 3 regions (`_looks_like_vision_refusal`); on our issue this
  removed 29 and 122 hallucinated regions on re-OCR of three pages.
- **Invented figures.** A layout model may propose a picture/chart/table box where OCR found nothing. Screen every
  invented region against the page's ink density before creating it (`region_looks_like_a_picture`, mean ink ≥ 0.4).
  Measured: invented regions that were really text had 0.10–0.14 mean ink, real photos/charts/tables 0.56–0.97 — a
  wide gap, so the floor is not fine-tuned.
- **Grounding vs transcription.** Engines transcribe far better than they localise. See "figure bounds" below.
- Treat a **direct engine label** (footer, title, page number, caption) as authoritative and skip heuristic
  re-classification for it; only ambiguous "text" regions go through your own classifier.

### 4. Get figure bounds from the render, not from the engine
Measured over 138 figure regions: text boxes matched the render closely enough to use as-is, figure boxes were
routinely offset, undersized, or on the neighbouring column — but the *contents* were right. The model knows **which**
graphic; the render knows **where it ends**. `snap_visual_bboxes`:
- reduce the render to an ink grid (10 px cells → inked fraction per cell; body text ≈ 0.25, a photo ≈ 1.0);
- **two passes**: a strict solid-ink pass (cell ≥ 0.55) anchors on photos/panels; a loose pass (any mark ≥ 0.15,
  ≤ 2 blank cells crossed) grows sparse charts (thin bars, gridlines) that the strict pass would crop;
- decide connectivity **once** (dilate the solid mask by 3 cells, label components, seed picks the component it
  overlaps most) — picking "the best component of the seed's own raw pieces" made the result depend on seed size and
  was a real bug (a correct box shrank on re-run);
- a rectangle is not the component's shape: trim edges back to the nearest row/column blocked across ≥ 70% of the
  width/height, or the box swallows unrelated text between two patches;
- text blocks and every *other* figure's seed are fences (excluded from the mask up front); neighbours re-seed each
  other for up to 6 rounds so the result stops oscillating and is deterministic;
- accept a result only if it is solidly inked (≥ 0.45) **or** framed on ~90% of its rows and columns (a UI screenshot
  is white inside), never shrinks below 30% of the seed, and **reproduces itself** when fed back (IoU ≥ 0.7);
- two figures pulled onto one graphic → revert both. A confidently wrong box is worse than a loose one the user sees.
Result: 98–119 of 147 figure regions corrected, none worse on visual inspection; pale charts on white left alone.
Make it **idempotent** and re-runnable on existing documents without re-paying for transcription.

### 5. Fit text boxes to their ink (small, bounded, geometry-only)
`snap_text_boxes`: if a side cuts through letters (ink continues just outside it where no other box is) grow it until
the text ends, by at most ~16 px sideways (70 for headlines, 90 downward); trim empty margin back to the first inked
line, leaving 2 px of air. Never touch a box a human drew or resized. Never change text. On the measured issue only 5
of 658 boxes (pages 1–59) needed it — which is the point: measure first, and keep it from ever making a good box worse.
Related: `mend_column_seams` moves the shared boundary between two vertically stacked boxes to the nearest white gap
of ≥ 2 blank pixel rows and reads *only the newly covered strip* into the box that gained it; re-check for duplicates
afterwards because fitting can turn a neighbour into a duplicate.

### 6. Clean-up rules: re-type, never delete — and size each rule on real data first
`clean_up_regions` measured 2,038 boxes before any rule was written:
| rule | found | left alone on purpose |
|---|---|---|
| text ≤ 80 chars and ≥ 90% inside a picture → `in_picture` | 71 | longer text in a picture: it was real running text caught in a loose frame |
| ≤ 2 alphanumeric characters → `ocr_noise` | 25 | longer short strings: a phone number, a price, a citation |
| box ≥ 80% inside a bigger box whose text contains ≥ 90% of it → `duplicate` | 38 | the other **333 overlapping pairs hold different text** — hiding one loses text |
| author line running into the first sentence → split | 5 | two-author bylines; a sentence starting with "Von" |
Every rule *re-types* (not narrated/used, still visible, reversible in the UI) and is idempotent, runs on import and on
re-OCR, writes a timestamped backup first. Overlapping boxes with different text are **surfaced for a human, not
auto-fixed** — we generate a review list with a suggested action per pair.

### 7. Reading order (only if your documents have columns)
Sorting by y interleaves parallel columns. What worked: columns come from where column-width blocks *start*
(clustered left edges), not from whitespace between columns (OCR boxes are loose; a 2 pt gutter and a pull quote across
it defeated the whitespace method); measure column width on body text only; bands — a headline/standfirst spanning
columns in a strip of its own is read first; **never** use paragraph gaps as breaks (they line up across columns and
re-interleave them: tried, measured, reverted). Measured with a text-only score (a block ending mid-sentence must be
followed by its continuation; a split word by its other half) over 204 pages: confirmed continuations 85 → 89.
Let users lock a hand-ordered page (`order_locked`) so no recompute overwrites it.

### 8. Running furniture (headers, footers, page numbers)
Identify by **repetition**, not position (a running head and a headline both sit at the top). Exact-match repetition
fails once OCR is involved (`c't Wissen 2026` came back as `cE KI-WISSEN`, `ce Wissen 2136`, …), so add a fuzzy pass:
near-edge blocks, digits stripped, `SequenceMatcher` ≥ 0.6, ≥ 3 occurrences. Without it one scan produced 26 bogus
"articles" from footers. An engine that labels footers directly makes this unnecessary for its pages.

### 9. A whole-document quality pass that reports, and fixes only what is provable
Per page, read-only audit from the render: `uncovered` ink blobs outside every box (ignore page-edge margins ~3%,
dust < 10 cells, rules and dotted leaders < 6 ink rows, unboxed ink within 10 pt of a picture); `overlaps` (text boxes
sharing ≥ 30% of the smaller); `clipped` edges (ink in a 14 px strip just outside a box's side, ignoring what another
box covers, ignoring strips that are pictures — ink density > 0.7); `suspect_blocks`. Independent completeness check:
run local OCR as a second opinion over the page (`recover_missing_text`) and *add* missing text as new **unticked**
regions, never replace boxes. Re-read dictionary-suspicious regions with the image engine and accept **only** a
better reading (`verify_ocr_recognition`). Spelling: findings, not rewrites — a model may propose a fix only if it is a
dictionary suggestion or a single known word within two edits, offsets re-verified against the current text.
Run the whole thing automatically as the last import step under the same progress job; report failures instead of
hiding them.

### 10. Human review is part of the design, and the UI must make errors visible
Nothing downstream (audio, export, search) uses unreviewed text. Review screen essentials, all of which paid off:
region boxes drawn on the page image, colour-coded by type with a legend; numbered in reading order; per-region type
picker, delete, merge-with-next, split-at-cursor, per-region re-OCR, inline text edit; keyboard navigation; bulk actions
with **undo instead of confirmation dialogs** (one level, tokenised, sealed with a digest so a stale Undo can never roll
back newer work); draw-a-missed-region; two-page spread; a live ingest preview that draws boxes as they are found
(purely decorative — progress numbers live in a separate status line for screen readers). WCAG 2.2 AA: icon + text,
never colour alone; labelled controls; keyboard alternative for every drag; reduced-motion respected.

## Engineering rules that kept this maintainable
- **Pure logic in one module, I/O at the edges.** Geometry/classification takes plain data, so tests need no PDF,
  no engine, no network. Import PyMuPDF lazily. Mock HTTP for engine tests.
- **Constants are named and commented with the measurement that set them** (`_SNAP_SOLID = 0.55 # ...`). When you
  change one, re-measure.
- **Every destructive action writes a backup or undo point first;** writes are atomic; jobs are restart-safe and a
  "running" state whose job died is reported as failed, not spun forever.
- **Idempotence is a feature:** re-running any fix on an already-fixed document changes nothing.
- **Per-document data stays out of git** (copyrighted/private pages, quality reports quoting text).
- **Security at the boundary:** validate ids (server-generated hex only), bound upload/download sizes, timeouts on
  every engine call, no credentials in URLs or logs.
- **Test both directions:** for each heuristic add a test for the case it fixes *and* the case it must leave alone
  (the "left alone on purpose" column above came from failing tests, not foresight).

## Order of work I recommend
1. Benchmark + baseline of the current engine(s) (Step 0). Report the numbers before changing anything.
2. Cheap, high-yield robustness: per-page escalation, fallback ladder, repetition-loop and refusal filters, engine
   labels trusted, plausibility score.
3. Box quality: fit text boxes to ink, figure bounds from the render, clean-up rules — each sized on real data first.
4. Audit + review list (overlaps for humans), then the review UI improvements that fit this app.
5. Reading order / furniture / articles — only if this app's documents have that structure.
6. Re-benchmark; report before/after with the same metrics, and list what did **not** help.

## Deliverables
- A short plan classifying each item above (applies / does not apply / needs decision), with expected effort.
- Benchmark script + results (metrics only) + README in the repo.
- Each adopted item: implementation, unit tests (positive and negative), docs/manual/changelog updated, one commit.
- A final before/after table and an honest list of limitations.

## What not to do
- Do not add a heavy OCR/vision engine because it is fashionable; our measurements found no accuracy gain over
  tesseract on clean pages and a 4–20× speed cost. Prove the gain on *this* app's documents.
- Do not auto-delete or auto-merge overlapping boxes with different text.
- Do not let a model's output edit text unless a deterministic check confirms the edit.
- Do not trust the engine's bounding boxes for graphics.
- Do not run installs of multi-GB models on a machine without checking free memory first (GPU and system memory can
  be one pool; a kill can take down unrelated services).