tts-voice-creator-clone-and-design-2/architecture/ocr-porting-prompt-general.md

# Prompt: make this app's OCR more accurate, robust and reviewable (general documents)

Copy everything below the line into the agent of the app that processes documents, letters, papers, forms, invoices
or similar. It is written for an AI coding agent that has never seen the source repository.

The lessons come from a different app (tts-voice-creator-clone-and-design-2) that OCRs **scanned German-language
magazines** and has run it on a 204-page issue for weeks. Its reference code is on the same machine, if you want to read
how a rule was implemented (read it, do not blindly copy it):
`/home/sparky/Docker/tts-voice-creator-clone-and-design-2` — `core/magazine.py` (pure logic), `routes/magazine.py`
(API/jobs), `static/js/magazine.js` (review UI), `scripts/bench_ocr.py` (benchmark), `tests/test_magazine.py`,
`architecture/magazine-ingestion.md` (the reasoning, including every measured number quoted below).

**How to read the evidence.** Each claim below is tagged:
- **[measured]** — measured on magazine pages, with the number given.
- **[principle]** — a general rule the magazine work made clear; expected to transfer.
- **[untested here]** — a recommendation for document types the reference app never processed. Treat it as a hypothesis
  and check it on your own documents before relying on it.

---

## Your task

This app already runs OCR on documents such as letters, papers and forms. Improve its accuracy, robustness and
reviewability by adopting the approach below **where it fits this app's documents** — first measure, then change.
Do not port code wholesale. Sort every numbered item into *applies / does not apply / needs a decision* for this app
and show me the plan before starting the larger ones.

Follow this project's own CLAUDE.md and conventions. Inspect the repository and the current OCR code first; preserve
behaviour that works; update docs and tests in the same change; commit per feature; do not push unless asked.

## 0. Privacy first (letters, papers and forms are personal data)

**[principle]** Documents like these contain names, addresses, account numbers, health or legal content. Before any
other work, find out where the current pipeline sends pages and decide with me:
- Prefer **local** OCR. Do not add a cloud OCR/vision API without checking its licence, cost, rate limits, data
  retention and privacy terms, and without my explicit approval.
- Keep page images and recognised text out of logs, error messages, analytics, git, test fixtures and benchmark output.
  Benchmark results and bug reports contain **metrics only**, never page text.
- Bound retention: say where originals, renders and text are stored, and provide a way to delete a document completely.
- Redaction/PII: if the app shows or exports text, consider a review-time option to mask identifiers; do not build
  PII detection unless asked.

## 1. Measure before touching anything

Build a benchmark first, or every later claim is opinion.

1. **Ground truth without hand labelling [measured].** Use documents that already have a real text layer (born-digital
   PDFs: papers, generated letters, invoices). Render pages to images at the resolution the app really uses, run each OCR
   engine on the *image*, and score against the PDF's own text layer. Score lower-cased word tokens (join line-end
   hyphenation first): **bag F1** (order-insensitive), **sequence similarity** (`difflib.SequenceMatcher` over tokens,
   which also rewards correct reading order), seconds per page, and memory footprint. Reference: `scripts/bench_ocr.py`.
   The score is capped below 1.0 by things a text layer has that a transcription need not (headers, footers, footnotes),
   so **compare engines to each other, never to 1.0**.
2. **Also test what the text layer cannot grade [principle].** Real letters and photographed documents are scans or
   phone photos: render some born-digital pages, then *degrade* them on purpose (rotate 1–3°, blur, add noise, JPEG-compress
   hard, lower dpi, uneven lighting) and measure again. Add a handful of genuinely scanned or photographed documents
   (yours, with permission) that you correct by hand; keep them outside git.
3. **Structure needs its own small labelled set [principle].** Transcription accuracy and layout quality are separate
   problems. Label 10–20 pages (boxes, reading order, region types, and for forms the key fields) and score them
   separately.
4. **Field-level accuracy, if the app extracts fields [untested here].** For letters/invoices/forms the useful metric is
   "did we get the date / total / sender / IBAN right", exact-match per field, not character accuracy.

**What we found [measured, magazines, 6 clean pages, 2026-09-29]:** tesseract deu+eng F1 0.880 at 2.2 s/page; a GPU
vision model (Baidu Unlimited-OCR) 0.865 at 44 s/page and returned nothing on 1 of 6 pages; docling on CPU 0.867 at
8.9 s; PaddleOCR 3.x on CPU 0.881 at 36 s. **Plain transcription accuracy was a wash; the cheapest engine was 20×
faster.** What the heavier engines add is *structure*, not better characters. **This may not hold for degraded scans,
handwriting, stamps, tables or non-Latin scripts — test it on your documents** before paying for a heavier engine.

## 2. The pipeline shape that worked

```
render page (fixed dpi) ─► use native text if the page has it ─► OCR only where needed ─► regions with bbox
   ─► trust engine labels ─► fit boxes to the ink ─► classify the ambiguous ─► reading order
   ─► clean-up (re-type, never delete) ─► quality audit ─► human review ─► only then use the text
```

### 2.1 Pre-processing for real scans [untested here]
The magazine scans were clean. Letters and photos are not. Before OCR, evaluate (each measured on your benchmark, kept
only if it helps): deskew (estimate skew from text-line angle), orientation detection (0/90/180/270), border/shadow
removal, contrast normalisation/binarisation for uneven lighting, and dpi normalisation (too low hurts, too high is
slow). **Never overwrite the original**; keep the processed render separate so any step can be undone and re-run.
Fixed page rotation is a common hidden cause of "OCR is bad".

### 2.2 Escalate per page, not per document [measured]
Use native text when present. Escalate a page to OCR when it has almost no native text (we used < 120 characters) **or**
its existing text scores badly on a coarse plausibility score (share of letters vs symbol noise, word shape). The score is
deliberately coarse: it gates an expensive call; it will **not** catch one misread in a good sentence (`KI` → `K |`).

### 2.3 A fallback ladder where every rung may fail [measured]
Primary vision/layout engine → second engine → local classical OCR (tesseract) → keep whatever text exists. Any failure
(unreachable, timeout, HTTP error, unparseable, repetition loop) returns `None` and the next rung runs. **OCR must never be
the reason a whole import fails.** An engine that returns plain text with no coordinates is still useful as one
full-page "text" region, gated by the same plausibility score, accepting that layout is lost for that page.

### 2.4 Distrust model output in specific, testable ways [measured]
Each of these happened on real pages; each is a small function with tests:
- **Repetition loops:** a vision model transcribed a chart as the same phrase for its whole token budget. A short phrase
  repeated ≥ 8× back-to-back = failed call, fall through to the next rung.
- **Hallucinated refusals:** over unreadable regions an engine writes "the image is too blurry to recognise…". Drop a
  sentence that repeats across ≥ 3 regions (29 and 122 such regions removed on re-OCR of three pages).
- **Invented graphics:** a layout model may propose a picture/table box where OCR found nothing. Screen each against the
  page's own ink density before creating it: invented boxes that were really text had 0.10–0.14 mean ink, real
  photos/charts/tables 0.56–0.97 — a wide gap.
- **Engines transcribe better than they localise:** see 2.5.
- A **direct engine label** (header, footer, page number, caption, title) is authoritative: skip heuristic
  re-classification for it; only ambiguous "text" regions go through your own classifier.
- **A model may never edit text unless a deterministic check confirms the edit** (for spelling: the replacement must be a
  dictionary suggestion or a single known word within two edits, with offsets re-verified against the current text).

### 2.5 Get graphic/table bounds from the render, not from the engine [measured]
Over 138 figure regions the engine's text boxes matched the render closely enough, but its figure boxes were routinely
offset, undersized or on the neighbouring column — while the *contents* were right. The model knows **which** graphic;
the render knows **where it ends**. Approach (`snap_visual_bboxes`): reduce the render to an ink grid (10 px cells →
inked fraction; body text ≈ 0.25, a photo ≈ 1.0); a strict solid-ink pass (cell ≥ 0.55) anchors on photos/panels and a loose
pass (any mark ≥ 0.15, ≤ 2 blank cells crossed) grows sparse charts; decide connectivity **once** (dilate, label, pick the
component the seed overlaps most — re-picking per seed made results depend on seed size); trim edges back to the nearest
row/column blocked across ≥ 70% of the width/height; fence off text blocks and other figures; accept only if solidly
inked (≥ 0.45) or framed on ~90% of rows and columns, never shrunk below 30% of the seed, and **reproducing itself** when
fed back (IoU ≥ 0.7); two figures pulled onto one graphic → revert. 98–119 of 147 regions corrected, none worse. This
matters for papers (figures, charts) and tables; it is **[untested here]** for ruled tables and forms, where line detection
(rows/columns from long horizontal/vertical strokes) is the better anchor — evaluate it.
Make it idempotent and re-runnable on existing documents without paying for transcription again.

### 2.6 Fit text boxes to their ink [measured]
Small, bounded, geometry-only: if a side cuts through letters (ink continues just outside where no other box is) grow it
until the text ends, by at most ~16 px sideways (70 for large type, 90 downward); trim empty margin back to the first
inked line, leaving 2 px of air. Never touch a box a human drew or resized; never change text. On the measured issue only 5
of 658 boxes needed it — measure first, and make sure it can never make a good box worse. Re-check for duplicates
afterwards, because fitting can turn a neighbour into one.

### 2.7 Clean-up rules: re-type, never delete; size each on real data first [measured]
The reference measured 2,038 boxes before writing any rule: text ≤ 80 chars and ≥ 90% inside a picture → `in_picture`
(71 found; longer text was real running text in a loose frame); ≤ 2 alphanumeric characters → `ocr_noise` (25; longer short
strings included a phone number, a price, a citation); a box ≥ 80% inside a bigger one whose text contains ≥ 90% of it →
`duplicate` (38). **The other 333 overlapping pairs held different text, and hiding one loses text.** So every rule
*re-types* (kept, visible, reversible), is idempotent, runs on import and on re-OCR, and writes a timestamped backup first.
**Overlaps with different text go to a human review list with a suggested action per pair, not to an auto-fix.**

### 2.8 Reading order [measured for multi-column pages]
Sorting by y interleaves parallel columns. What worked: columns from where column-width blocks *start* (clustered left
edges), not from whitespace between columns (OCR boxes are loose; a 2 pt gutter or a block across it defeated the
whitespace method); measure column width on body text only; a headline/abstract/title spanning columns in a strip of its
own is read first; **never** use paragraph gaps as breaks (they line up across columns and re-interleave them — tried,
measured, reverted). Score it with a text-only check (a block ending mid-sentence must be followed by its continuation; a
split word by its other half): 85 → 89 confirmed continuations over 204 pages. Let the user lock a hand-ordered page.
- **Papers [untested here]:** two-column body, full-width title/abstract, footnotes and running headers at the edges,
  references as a list, figure/table captions that belong to their float, equations as their own region. The column logic
  above should apply; footnote/caption/reference handling needs its own rules and tests.
- **Letters [untested here]:** mostly single column, but with a fixed semantic layout: letterhead, sender line,
  recipient block, date, subject line, salutation, body, closing/signature, footer (bank details, registration). Reading
  order is simple; **classifying these blocks** is the value (often more so than transcription) and is where a layout
  model or simple position+keyword rules can pay off. Stamps, handwritten notes and signatures overlap text — treat them
  as their own region types, not as text.
- **Forms and invoices [untested here]:** reading order is secondary to key-value pairing (label ↔ value) and table rows.
  Don't force them through a prose pipeline.

### 2.9 Running headers/footers/page numbers [measured]
Identify by **repetition**, not position (a running head and a title both sit at the top). Exact-match repetition fails
once OCR is involved (`c't Wissen 2026` came back as `cE KI-WISSEN`, `ce Wissen 2136`…), so add a fuzzy pass: near-edge
blocks, digits stripped, `SequenceMatcher` ≥ 0.6, ≥ 3 occurrences. Without it, footers produced 26 bogus sections in one
scan. Matters for multi-page letters, contracts, reports and papers.

### 2.10 A whole-document quality pass that reports, and fixes only what is provable [measured]
Per page, read-only, from the render: ink blobs outside every box (ignore page-edge margins ~3%, dust < 10 cells, rules and
dotted leaders < 6 ink rows, unboxed ink within 10 pt of a picture); overlapping text boxes (sharing ≥ 30% of the smaller);
boxes whose edge clips letters (ink in a 14 px strip just outside a side; skip strips that are pictures, ink > 0.7); and
suspect blocks. An independent completeness check: run a second, local OCR over the page and **add** missing text as new
**unticked** regions, never replace boxes. Re-read dictionary-suspicious regions with the image engine and accept **only** a
better reading. Run it automatically as the last import step under the same progress job; report failures instead of
hiding them.
- **Language/dictionary [principle]:** a spell check needs the document's language. Letters and papers are often
  multilingual or full of names, codes and jargon: decide language per document (and per block only when decisively
  different), keep a per-document dictionary of accepted words, and treat spelling as *findings*, not rewrites. For
  identifiers (IBAN, invoice numbers, dates, amounts) prefer **checksum/format validation** (IBAN mod-97, date parsing,
  sum checks) over dictionaries **[untested here]** — a wrong digit is the costly error in this domain.

### 2.11 Handwriting [untested here]
Classical OCR and most of the engines above do poorly on handwriting. Do not assume otherwise. If the documents contain
any: detect it (a region type), keep it as an image with an editable transcription, and evaluate a handwriting-capable
model on your own samples against a manual transcription before promising anything.

### 2.12 Human review is part of the design [measured]
Nothing downstream (search, export, automation) should use unreviewed text for high-stakes fields. Review-screen essentials
that paid off: region boxes drawn on the page, colour-coded by type with a legend; numbered in reading order; per-region type
picker, delete, merge-with-next, split-at-cursor, per-region re-OCR, inline text edit; keyboard navigation; bulk actions with
**undo instead of confirmation dialogs** (one level, tokenised, sealed with a digest so a stale Undo can never roll back
newer work); draw-a-missed-region; a live ingest preview that draws boxes as they are found (decorative — progress lives in a
separate status line for screen readers). Show low-confidence words/fields visibly. WCAG 2.2 AA: icon plus text, never
colour alone; labelled controls; a keyboard alternative for every drag; reduced motion respected.

## 3. Engineering rules that kept this maintainable [principle]
- **Pure logic in one module, I/O at the edges.** Geometry and classification take plain data, so tests need no PDF, no
  engine, no network. Import heavy libraries lazily. Mock HTTP for engine tests.
- **Constants are named and commented with the measurement that set them**; when you change one, re-measure.
- **Every destructive action writes a backup or an undo point first;** writes are atomic; jobs are restart-safe and a
  "running" state whose job died is reported as failed, not spun forever.
- **Idempotence is a feature:** re-running any fix on an already-fixed document changes nothing.
- **Security at the boundary:** validate ids, bound upload and download sizes, set timeouts on every engine call, no
  credentials in URLs or logs, validate MIME type and scan uploads, protect against path traversal and SSRF.
- **Test both directions:** for each heuristic add a test for the case it fixes *and* the case it must leave alone (the
  "left alone on purpose" cases above came from failing tests, not foresight).

## 4. Order of work I recommend
1. Privacy review of the current pipeline (section 0) and the benchmark + baseline (section 1). Report the numbers before
   changing anything, including results on degraded renders.
2. Pre-processing for real scans (2.1), evaluated one step at a time.
3. Cheap, high-yield robustness: per-page escalation, fallback ladder, repetition/refusal filters, trusted engine labels.
4. Box and table quality: fit to ink, graphic/table bounds from the render, clean-up rules — each sized on real data first.
5. Document-type handling: papers (columns, captions, references), letters (block classification), forms (key-value,
   tables). Start with the type most common in this app; build a small labelled set for it.
6. Quality audit, review list for the ambiguous cases, and the review-UI improvements that fit.
7. Re-benchmark with the same metrics; report before/after and list what did **not** help.

## 5. Deliverables
- A short plan classifying each item (applies / does not apply / needs decision) with expected effort and risk.
- The benchmark script, results (metrics only) and a README saying what was measured, when, on what.
- Each adopted item: implementation, unit tests (positive and negative), docs/manual/changelog updated, one commit.
- A final before/after table and an honest list of limitations and of things tried that did not help.

## 6. What not to do
- Do not add a heavy OCR/vision engine because it is fashionable: on clean magazine pages it gave no accuracy gain over
  tesseract at a 4–20× cost. Prove the gain on *this* app's documents, especially degraded ones.
- Do not send private documents to a cloud service without my explicit approval.
- Do not auto-delete or auto-merge overlapping boxes that hold different text.
- Do not let a model edit text without a deterministic confirmation.
- Do not trust an engine's bounding boxes for graphics or tables.
- Do not put page text into logs, tests, fixtures, benchmarks or git.
- Do not install multi-GB models without checking free memory first (GPU and system memory can be one pool; an
  out-of-memory kill can take down unrelated services).