# OCR candidate benchmark — 2026-09-29

Harness: `scripts/bench_ocr.py`. Corpus: *c't KI Wissen 2026* (native text layer, 204 pages). Six evenly spaced
pages with at least 250 words (5, 32, 67, 101, 138, 169), rendered at 110 dpi (the app's ingest default). The
PDF's own text layer is the ground truth. Raw numbers: `bench_*.json` (metrics only, no magazine text).

| Engine | Where it ran | Mean word F1 | Seq. similarity | Median s/page | Memory | Licence |
|---|---|---:|---:|---:|---|---|
| tesseract deu+eng (current last resort) | CPU, app container | 0.880 | 0.841 | 2.2 | negligible | Apache-2.0 |
| baidu/Unlimited-OCR (current primary) | GPU container, port 8098 | 0.865 (5/6 pages) | 0.865 | 44.1 | see `docker stats` | as published |
| docling (default pipeline + RapidOCR) | CPU, venv | 0.867 | 0.783 | 8.9 (first page 66 s warm-up) | 2.4 GB RSS | MIT |
| PaddleOCR 3.2 (PP-OCRv6 medium, `de`) | CPU, venv | 0.881 | 0.551 | 36.5 | 1.4 GB RSS | Apache-2.0 |
| DeepSeek-OCR | **not run** | | | | ~7 GB weights | MIT |
| Qwen3-VL-30B-A3B-Instruct-FP8 | **not run** | | | | ~30 GB weights | Apache-2.0 |
| KDL-Frontier-Parser-nano (current fallback) | **not run**: container exited (137) 6 days ago | | | | | |

Metrics: *word F1* is order-insensitive overlap of lower-cased word tokens; *sequence similarity* is a difflib ratio
over the token sequence and so also rewards correct reading order. Both are capped below 1.0 by things a text layer
contains that a page transcription need not (running heads, hyphenation, sidebars), so compare engines to each other,
not to 1.0.

## Reading
- **Plain transcription accuracy is a wash.** Every engine lands at F1 0.865–0.881 on these pages; the differences are
  within page-to-page variation (0.77–0.96). Nothing here justifies replacing tesseract for text quality alone.
- **Speed differs by 20×.** Tesseract 2.2 s/page beats the GPU vision engine (44 s/page) and both CPU candidates.
  The vision engine also returned nothing for one of six pages (page 101), which the app's fallback chain absorbs.
- **What the heavier engines add is structure, not characters:** Unlimited-OCR labels regions and furniture,
  docling returns layout-ordered text (sequence similarity 0.78 vs PaddleOCR's 0.55, which is raw line order).
  Neither is scored here; the app's own layout stage already does this work, so the case for a candidate rests on
  region/table/figure quality, which needs a different benchmark (labelled boxes), not this one.
- **Open-ended figure description** (the harder requirement in TODO) is untested: none of the engines run here is a
  captioning model, and the two that are (DeepSeek-OCR, Qwen3-VL) could not be loaded — see below.

## Why two candidates were not run
The host has 121 GB of unified memory with about 107 GB in use (the running sglang model dominates; GPU and system
memory are the same pool on this machine), leaving roughly 14 GB. Qwen3-VL-30B FP8 (~30 GB) cannot be loaded without
stopping another model, and DeepSeek-OCR (~7 GB, pinned dependencies on a GB10 GPU) would leave almost no margin: an
out-of-memory kill there could take down the running stack. Both need a decision from the owner: which running model to
stop for a test window.

## Caveats
Six pages of one German-language, native-text magazine; CPU timings for docling and PaddleOCR (both would be faster on
GPU); one run each. Enough to rule out an obvious accuracy win, not to pick a winner for scanned or image-heavy issues —
this magazine's pages are born-digital, so a scanned issue would separate the engines more.