From 0d3178f733f2e8efcc6c0126b1864c5c3b94734d Mon Sep 17 00:00:00 2001 From: mARTin-B78 Date: Wed, 30 Sep 2026 10:10:59 +0200 Subject: [PATCH] docs: general-purpose OCR porting brief for letters, papers and forms Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 2 + architecture/ocr-porting-prompt-general.md | 230 +++++++++++++++++++++ 2 files changed, 232 insertions(+) create mode 100644 architecture/ocr-porting-prompt-general.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 58edd95..03db407 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi --- ## [Unreleased] +### Added +- `architecture/ocr-porting-prompt-general.md`: the OCR porting brief rewritten for general documents (letters, papers, forms, invoices). Adds privacy, pre-processing of skewed/photographed scans, document-type handling, field-level accuracy and handwriting, and tags every claim as measured on magazines, a general principle, or untested for these document types. --- diff --git a/architecture/ocr-porting-prompt-general.md b/architecture/ocr-porting-prompt-general.md new file mode 100644 index 0000000..15e2c81 --- /dev/null +++ b/architecture/ocr-porting-prompt-general.md @@ -0,0 +1,230 @@ +# Prompt: make this app's OCR more accurate, robust and reviewable (general documents) + +Copy everything below the line into the agent of the app that processes documents, letters, papers, forms, invoices +or similar. It is written for an AI coding agent that has never seen the source repository. + +The lessons come from a different app (tts-voice-creator-clone-and-design-2) that OCRs **scanned German-language +magazines** and has run it on a 204-page issue for weeks. Its reference code is on the same machine, if you want to read +how a rule was implemented (read it, do not blindly copy it): +`/home/sparky/Docker/tts-voice-creator-clone-and-design-2` — `core/magazine.py` (pure logic), `routes/magazine.py` +(API/jobs), `static/js/magazine.js` (review UI), `scripts/bench_ocr.py` (benchmark), `tests/test_magazine.py`, +`architecture/magazine-ingestion.md` (the reasoning, including every measured number quoted below). + +**How to read the evidence.** Each claim below is tagged: +- **[measured]** — measured on magazine pages, with the number given. +- **[principle]** — a general rule the magazine work made clear; expected to transfer. +- **[untested here]** — a recommendation for document types the reference app never processed. Treat it as a hypothesis + and check it on your own documents before relying on it. + +--- + +## Your task + +This app already runs OCR on documents such as letters, papers and forms. Improve its accuracy, robustness and +reviewability by adopting the approach below **where it fits this app's documents** — first measure, then change. +Do not port code wholesale. Sort every numbered item into *applies / does not apply / needs a decision* for this app +and show me the plan before starting the larger ones. + +Follow this project's own CLAUDE.md and conventions. Inspect the repository and the current OCR code first; preserve +behaviour that works; update docs and tests in the same change; commit per feature; do not push unless asked. + +## 0. Privacy first (letters, papers and forms are personal data) + +**[principle]** Documents like these contain names, addresses, account numbers, health or legal content. Before any +other work, find out where the current pipeline sends pages and decide with me: +- Prefer **local** OCR. Do not add a cloud OCR/vision API without checking its licence, cost, rate limits, data + retention and privacy terms, and without my explicit approval. +- Keep page images and recognised text out of logs, error messages, analytics, git, test fixtures and benchmark output. + Benchmark results and bug reports contain **metrics only**, never page text. +- Bound retention: say where originals, renders and text are stored, and provide a way to delete a document completely. +- Redaction/PII: if the app shows or exports text, consider a review-time option to mask identifiers; do not build + PII detection unless asked. + +## 1. Measure before touching anything + +Build a benchmark first, or every later claim is opinion. + +1. **Ground truth without hand labelling [measured].** Use documents that already have a real text layer (born-digital + PDFs: papers, generated letters, invoices). Render pages to images at the resolution the app really uses, run each OCR + engine on the *image*, and score against the PDF's own text layer. Score lower-cased word tokens (join line-end + hyphenation first): **bag F1** (order-insensitive), **sequence similarity** (`difflib.SequenceMatcher` over tokens, + which also rewards correct reading order), seconds per page, and memory footprint. Reference: `scripts/bench_ocr.py`. + The score is capped below 1.0 by things a text layer has that a transcription need not (headers, footers, footnotes), + so **compare engines to each other, never to 1.0**. +2. **Also test what the text layer cannot grade [principle].** Real letters and photographed documents are scans or + phone photos: render some born-digital pages, then *degrade* them on purpose (rotate 1–3°, blur, add noise, JPEG-compress + hard, lower dpi, uneven lighting) and measure again. Add a handful of genuinely scanned or photographed documents + (yours, with permission) that you correct by hand; keep them outside git. +3. **Structure needs its own small labelled set [principle].** Transcription accuracy and layout quality are separate + problems. Label 10–20 pages (boxes, reading order, region types, and for forms the key fields) and score them + separately. +4. **Field-level accuracy, if the app extracts fields [untested here].** For letters/invoices/forms the useful metric is + "did we get the date / total / sender / IBAN right", exact-match per field, not character accuracy. + +**What we found [measured, magazines, 6 clean pages, 2026-09-29]:** tesseract deu+eng F1 0.880 at 2.2 s/page; a GPU +vision model (Baidu Unlimited-OCR) 0.865 at 44 s/page and returned nothing on 1 of 6 pages; docling on CPU 0.867 at +8.9 s; PaddleOCR 3.x on CPU 0.881 at 36 s. **Plain transcription accuracy was a wash; the cheapest engine was 20× +faster.** What the heavier engines add is *structure*, not better characters. **This may not hold for degraded scans, +handwriting, stamps, tables or non-Latin scripts — test it on your documents** before paying for a heavier engine. + +## 2. The pipeline shape that worked + +``` +render page (fixed dpi) ─► use native text if the page has it ─► OCR only where needed ─► regions with bbox + ─► trust engine labels ─► fit boxes to the ink ─► classify the ambiguous ─► reading order + ─► clean-up (re-type, never delete) ─► quality audit ─► human review ─► only then use the text +``` + +### 2.1 Pre-processing for real scans [untested here] +The magazine scans were clean. Letters and photos are not. Before OCR, evaluate (each measured on your benchmark, kept +only if it helps): deskew (estimate skew from text-line angle), orientation detection (0/90/180/270), border/shadow +removal, contrast normalisation/binarisation for uneven lighting, and dpi normalisation (too low hurts, too high is +slow). **Never overwrite the original**; keep the processed render separate so any step can be undone and re-run. +Fixed page rotation is a common hidden cause of "OCR is bad". + +### 2.2 Escalate per page, not per document [measured] +Use native text when present. Escalate a page to OCR when it has almost no native text (we used < 120 characters) **or** +its existing text scores badly on a coarse plausibility score (share of letters vs symbol noise, word shape). The score is +deliberately coarse: it gates an expensive call; it will **not** catch one misread in a good sentence (`KI` → `K |`). + +### 2.3 A fallback ladder where every rung may fail [measured] +Primary vision/layout engine → second engine → local classical OCR (tesseract) → keep whatever text exists. Any failure +(unreachable, timeout, HTTP error, unparseable, repetition loop) returns `None` and the next rung runs. **OCR must never be +the reason a whole import fails.** An engine that returns plain text with no coordinates is still useful as one +full-page "text" region, gated by the same plausibility score, accepting that layout is lost for that page. + +### 2.4 Distrust model output in specific, testable ways [measured] +Each of these happened on real pages; each is a small function with tests: +- **Repetition loops:** a vision model transcribed a chart as the same phrase for its whole token budget. A short phrase + repeated ≥ 8× back-to-back = failed call, fall through to the next rung. +- **Hallucinated refusals:** over unreadable regions an engine writes "the image is too blurry to recognise…". Drop a + sentence that repeats across ≥ 3 regions (29 and 122 such regions removed on re-OCR of three pages). +- **Invented graphics:** a layout model may propose a picture/table box where OCR found nothing. Screen each against the + page's own ink density before creating it: invented boxes that were really text had 0.10–0.14 mean ink, real + photos/charts/tables 0.56–0.97 — a wide gap. +- **Engines transcribe better than they localise:** see 2.5. +- A **direct engine label** (header, footer, page number, caption, title) is authoritative: skip heuristic + re-classification for it; only ambiguous "text" regions go through your own classifier. +- **A model may never edit text unless a deterministic check confirms the edit** (for spelling: the replacement must be a + dictionary suggestion or a single known word within two edits, with offsets re-verified against the current text). + +### 2.5 Get graphic/table bounds from the render, not from the engine [measured] +Over 138 figure regions the engine's text boxes matched the render closely enough, but its figure boxes were routinely +offset, undersized or on the neighbouring column — while the *contents* were right. The model knows **which** graphic; +the render knows **where it ends**. Approach (`snap_visual_bboxes`): reduce the render to an ink grid (10 px cells → +inked fraction; body text ≈ 0.25, a photo ≈ 1.0); a strict solid-ink pass (cell ≥ 0.55) anchors on photos/panels and a loose +pass (any mark ≥ 0.15, ≤ 2 blank cells crossed) grows sparse charts; decide connectivity **once** (dilate, label, pick the +component the seed overlaps most — re-picking per seed made results depend on seed size); trim edges back to the nearest +row/column blocked across ≥ 70% of the width/height; fence off text blocks and other figures; accept only if solidly +inked (≥ 0.45) or framed on ~90% of rows and columns, never shrunk below 30% of the seed, and **reproducing itself** when +fed back (IoU ≥ 0.7); two figures pulled onto one graphic → revert. 98–119 of 147 regions corrected, none worse. This +matters for papers (figures, charts) and tables; it is **[untested here]** for ruled tables and forms, where line detection +(rows/columns from long horizontal/vertical strokes) is the better anchor — evaluate it. +Make it idempotent and re-runnable on existing documents without paying for transcription again. + +### 2.6 Fit text boxes to their ink [measured] +Small, bounded, geometry-only: if a side cuts through letters (ink continues just outside where no other box is) grow it +until the text ends, by at most ~16 px sideways (70 for large type, 90 downward); trim empty margin back to the first +inked line, leaving 2 px of air. Never touch a box a human drew or resized; never change text. On the measured issue only 5 +of 658 boxes needed it — measure first, and make sure it can never make a good box worse. Re-check for duplicates +afterwards, because fitting can turn a neighbour into one. + +### 2.7 Clean-up rules: re-type, never delete; size each on real data first [measured] +The reference measured 2,038 boxes before writing any rule: text ≤ 80 chars and ≥ 90% inside a picture → `in_picture` +(71 found; longer text was real running text in a loose frame); ≤ 2 alphanumeric characters → `ocr_noise` (25; longer short +strings included a phone number, a price, a citation); a box ≥ 80% inside a bigger one whose text contains ≥ 90% of it → +`duplicate` (38). **The other 333 overlapping pairs held different text, and hiding one loses text.** So every rule +*re-types* (kept, visible, reversible), is idempotent, runs on import and on re-OCR, and writes a timestamped backup first. +**Overlaps with different text go to a human review list with a suggested action per pair, not to an auto-fix.** + +### 2.8 Reading order [measured for multi-column pages] +Sorting by y interleaves parallel columns. What worked: columns from where column-width blocks *start* (clustered left +edges), not from whitespace between columns (OCR boxes are loose; a 2 pt gutter or a block across it defeated the +whitespace method); measure column width on body text only; a headline/abstract/title spanning columns in a strip of its +own is read first; **never** use paragraph gaps as breaks (they line up across columns and re-interleave them — tried, +measured, reverted). Score it with a text-only check (a block ending mid-sentence must be followed by its continuation; a +split word by its other half): 85 → 89 confirmed continuations over 204 pages. Let the user lock a hand-ordered page. +- **Papers [untested here]:** two-column body, full-width title/abstract, footnotes and running headers at the edges, + references as a list, figure/table captions that belong to their float, equations as their own region. The column logic + above should apply; footnote/caption/reference handling needs its own rules and tests. +- **Letters [untested here]:** mostly single column, but with a fixed semantic layout: letterhead, sender line, + recipient block, date, subject line, salutation, body, closing/signature, footer (bank details, registration). Reading + order is simple; **classifying these blocks** is the value (often more so than transcription) and is where a layout + model or simple position+keyword rules can pay off. Stamps, handwritten notes and signatures overlap text — treat them + as their own region types, not as text. +- **Forms and invoices [untested here]:** reading order is secondary to key-value pairing (label ↔ value) and table rows. + Don't force them through a prose pipeline. + +### 2.9 Running headers/footers/page numbers [measured] +Identify by **repetition**, not position (a running head and a title both sit at the top). Exact-match repetition fails +once OCR is involved (`c't Wissen 2026` came back as `cE KI-WISSEN`, `ce Wissen 2136`…), so add a fuzzy pass: near-edge +blocks, digits stripped, `SequenceMatcher` ≥ 0.6, ≥ 3 occurrences. Without it, footers produced 26 bogus sections in one +scan. Matters for multi-page letters, contracts, reports and papers. + +### 2.10 A whole-document quality pass that reports, and fixes only what is provable [measured] +Per page, read-only, from the render: ink blobs outside every box (ignore page-edge margins ~3%, dust < 10 cells, rules and +dotted leaders < 6 ink rows, unboxed ink within 10 pt of a picture); overlapping text boxes (sharing ≥ 30% of the smaller); +boxes whose edge clips letters (ink in a 14 px strip just outside a side; skip strips that are pictures, ink > 0.7); and +suspect blocks. An independent completeness check: run a second, local OCR over the page and **add** missing text as new +**unticked** regions, never replace boxes. Re-read dictionary-suspicious regions with the image engine and accept **only** a +better reading. Run it automatically as the last import step under the same progress job; report failures instead of +hiding them. +- **Language/dictionary [principle]:** a spell check needs the document's language. Letters and papers are often + multilingual or full of names, codes and jargon: decide language per document (and per block only when decisively + different), keep a per-document dictionary of accepted words, and treat spelling as *findings*, not rewrites. For + identifiers (IBAN, invoice numbers, dates, amounts) prefer **checksum/format validation** (IBAN mod-97, date parsing, + sum checks) over dictionaries **[untested here]** — a wrong digit is the costly error in this domain. + +### 2.11 Handwriting [untested here] +Classical OCR and most of the engines above do poorly on handwriting. Do not assume otherwise. If the documents contain +any: detect it (a region type), keep it as an image with an editable transcription, and evaluate a handwriting-capable +model on your own samples against a manual transcription before promising anything. + +### 2.12 Human review is part of the design [measured] +Nothing downstream (search, export, automation) should use unreviewed text for high-stakes fields. Review-screen essentials +that paid off: region boxes drawn on the page, colour-coded by type with a legend; numbered in reading order; per-region type +picker, delete, merge-with-next, split-at-cursor, per-region re-OCR, inline text edit; keyboard navigation; bulk actions with +**undo instead of confirmation dialogs** (one level, tokenised, sealed with a digest so a stale Undo can never roll back +newer work); draw-a-missed-region; a live ingest preview that draws boxes as they are found (decorative — progress lives in a +separate status line for screen readers). Show low-confidence words/fields visibly. WCAG 2.2 AA: icon plus text, never +colour alone; labelled controls; a keyboard alternative for every drag; reduced motion respected. + +## 3. Engineering rules that kept this maintainable [principle] +- **Pure logic in one module, I/O at the edges.** Geometry and classification take plain data, so tests need no PDF, no + engine, no network. Import heavy libraries lazily. Mock HTTP for engine tests. +- **Constants are named and commented with the measurement that set them**; when you change one, re-measure. +- **Every destructive action writes a backup or an undo point first;** writes are atomic; jobs are restart-safe and a + "running" state whose job died is reported as failed, not spun forever. +- **Idempotence is a feature:** re-running any fix on an already-fixed document changes nothing. +- **Security at the boundary:** validate ids, bound upload and download sizes, set timeouts on every engine call, no + credentials in URLs or logs, validate MIME type and scan uploads, protect against path traversal and SSRF. +- **Test both directions:** for each heuristic add a test for the case it fixes *and* the case it must leave alone (the + "left alone on purpose" cases above came from failing tests, not foresight). + +## 4. Order of work I recommend +1. Privacy review of the current pipeline (section 0) and the benchmark + baseline (section 1). Report the numbers before + changing anything, including results on degraded renders. +2. Pre-processing for real scans (2.1), evaluated one step at a time. +3. Cheap, high-yield robustness: per-page escalation, fallback ladder, repetition/refusal filters, trusted engine labels. +4. Box and table quality: fit to ink, graphic/table bounds from the render, clean-up rules — each sized on real data first. +5. Document-type handling: papers (columns, captions, references), letters (block classification), forms (key-value, + tables). Start with the type most common in this app; build a small labelled set for it. +6. Quality audit, review list for the ambiguous cases, and the review-UI improvements that fit. +7. Re-benchmark with the same metrics; report before/after and list what did **not** help. + +## 5. Deliverables +- A short plan classifying each item (applies / does not apply / needs decision) with expected effort and risk. +- The benchmark script, results (metrics only) and a README saying what was measured, when, on what. +- Each adopted item: implementation, unit tests (positive and negative), docs/manual/changelog updated, one commit. +- A final before/after table and an honest list of limitations and of things tried that did not help. + +## 6. What not to do +- Do not add a heavy OCR/vision engine because it is fashionable: on clean magazine pages it gave no accuracy gain over + tesseract at a 4–20× cost. Prove the gain on *this* app's documents, especially degraded ones. +- Do not send private documents to a cloud service without my explicit approval. +- Do not auto-delete or auto-merge overlapping boxes that hold different text. +- Do not let a model edit text without a deterministic confirmation. +- Do not trust an engine's bounding boxes for graphics or tables. +- Do not put page text into logs, tests, fixtures, benchmarks or git. +- Do not install multi-GB models without checking free memory first (GPU and system memory can be one pool; an + out-of-memory kill can take down unrelated services).