45 lines
3.5 KiB
Markdown
45 lines
3.5 KiB
Markdown
# OCR candidate benchmark — 2026-09-29
|
||
|
||
Harness: `scripts/bench_ocr.py`. Corpus: *c't KI Wissen 2026* (native text layer, 204 pages). Six evenly spaced
|
||
pages with at least 250 words (5, 32, 67, 101, 138, 169), rendered at 110 dpi (the app's ingest default). The
|
||
PDF's own text layer is the ground truth. Raw numbers: `bench_*.json` (metrics only, no magazine text).
|
||
|
||
| Engine | Where it ran | Mean word F1 | Seq. similarity | Median s/page | Memory | Licence |
|
||
|---|---|---:|---:|---:|---|---|
|
||
| tesseract deu+eng (current last resort) | CPU, app container | 0.880 | 0.841 | 2.2 | negligible | Apache-2.0 |
|
||
| baidu/Unlimited-OCR (current primary) | GPU container, port 8098 | 0.865 (5/6 pages) | 0.865 | 44.1 | see `docker stats` | as published |
|
||
| docling (default pipeline + RapidOCR) | CPU, venv | 0.867 | 0.783 | 8.9 (first page 66 s warm-up) | 2.4 GB RSS | MIT |
|
||
| PaddleOCR 3.2 (PP-OCRv6 medium, `de`) | CPU, venv | 0.881 | 0.551 | 36.5 | 1.4 GB RSS | Apache-2.0 |
|
||
| DeepSeek-OCR | **not run** | | | | ~7 GB weights | MIT |
|
||
| Qwen3-VL-30B-A3B-Instruct-FP8 | **not run** | | | | ~30 GB weights | Apache-2.0 |
|
||
| KDL-Frontier-Parser-nano (current fallback) | **not run**: container exited (137) 6 days ago | | | | | |
|
||
|
||
Metrics: *word F1* is order-insensitive overlap of lower-cased word tokens; *sequence similarity* is a difflib ratio
|
||
over the token sequence and so also rewards correct reading order. Both are capped below 1.0 by things a text layer
|
||
contains that a page transcription need not (running heads, hyphenation, sidebars), so compare engines to each other,
|
||
not to 1.0.
|
||
|
||
## Reading
|
||
- **Plain transcription accuracy is a wash.** Every engine lands at F1 0.865–0.881 on these pages; the differences are
|
||
within page-to-page variation (0.77–0.96). Nothing here justifies replacing tesseract for text quality alone.
|
||
- **Speed differs by 20×.** Tesseract 2.2 s/page beats the GPU vision engine (44 s/page) and both CPU candidates.
|
||
The vision engine also returned nothing for one of six pages (page 101), which the app's fallback chain absorbs.
|
||
- **What the heavier engines add is structure, not characters:** Unlimited-OCR labels regions and furniture,
|
||
docling returns layout-ordered text (sequence similarity 0.78 vs PaddleOCR's 0.55, which is raw line order).
|
||
Neither is scored here; the app's own layout stage already does this work, so the case for a candidate rests on
|
||
region/table/figure quality, which needs a different benchmark (labelled boxes), not this one.
|
||
- **Open-ended figure description** (the harder requirement in TODO) is untested: none of the engines run here is a
|
||
captioning model, and the two that are (DeepSeek-OCR, Qwen3-VL) could not be loaded — see below.
|
||
|
||
## Why two candidates were not run
|
||
The host has 121 GB of unified memory with about 107 GB in use (the running sglang model dominates; GPU and system
|
||
memory are the same pool on this machine), leaving roughly 14 GB. Qwen3-VL-30B FP8 (~30 GB) cannot be loaded without
|
||
stopping another model, and DeepSeek-OCR (~7 GB, pinned dependencies on a GB10 GPU) would leave almost no margin: an
|
||
out-of-memory kill there could take down the running stack. Both need a decision from the owner: which running model to
|
||
stop for a test window.
|
||
|
||
## Caveats
|
||
Six pages of one German-language, native-text magazine; CPU timings for docling and PaddleOCR (both would be faster on
|
||
GPU); one run each. Enough to rule out an obvious accuracy win, not to pick a winner for scanned or image-heavy issues —
|
||
this magazine's pages are born-digital, so a scanned issue would separate the engines more.
|