diff --git a/CHANGELOG.md b/CHANGELOG.md index 1eca82c..2a46691 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ### Fixed - **Voice benchmark failures now say why.** Designed voices are benchmarked through the Voice Design engine; when that service is down the row showed a bare "-" and the raw "Max retries exceeded" text. The error now names the service and the container to start, the Factor and WPM columns show `ERR` beside the time column, and a batch stops after three identical "unreachable" failures instead of retrying every remaining voice. ### Added +- `scripts/bench_ocr.py`: engine-agnostic OCR benchmark that scores an engine's transcription of rendered pages against a native-text PDF's own text layer (word F1, sequence similarity, seconds per page). First results (tesseract, Unlimited-OCR, docling, PaddleOCR) in `architecture/evidence/ocr-bench-2026-09-29/`: transcription accuracy is equal within noise (F1 0.865-0.881), tesseract is 4-20x faster; DeepSeek-OCR and Qwen3-VL-30B were not run (no memory headroom). - **Find an issue online (magazines).** A search panel in the magazine library finds issues in the Internet Archive and in any OPDS catalogue you configure (Calibre-Web, Kavita, Komga), shows the source's licence, and imports the PDF straight into a new magazine project, then starts ingestion. SSRF-safe (downloads only from the provider's own host), size/time bounded, feeds with DTDs refused. See `architecture/magazine-sources.md`. - **Magazine contents list shows synthesis state.** Each article row says "Synthesised · m:ss" (the MP3's exact length) or "Not synthesised" (approved, no MP3 yet), and has an inline play/pause button that streams the MP3 without opening the article. Icon plus words, screen-reader labelled, works at phone width. - `scripts/compare_magazines.py`: read-only comparison of two magazine folders (words, boxes, reading order, articles, quality numbers) for re-import checks. diff --git a/TODO.md b/TODO.md index 48b2719..e418865 100644 --- a/TODO.md +++ b/TODO.md @@ -732,10 +732,10 @@ casting. The original PDF remains the visual source of truth while approved arti Candidates to install as Docker containers alongside the existing engines, each added through the Engines tab (`routes/docker.py`'s `_LOCAL_CONTAINER_DEFS`, role `ocr`) so they're selectable/connectable/benchmarkable like any other engine, not hardcoded: - - [ ] [docling](https://github.com/docling-project/docling) — document conversion/layout toolkit. - - [ ] [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) — established general-purpose OCR. - - [ ] [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) — vision-language OCR model. - - [ ] Qwen3-VL-30B-A3B-Instruct-FP8 — general vision-language model; check whether it fits the RAM/VRAM + - [x] [docling](https://github.com/docling-project/docling) — document conversion/layout toolkit. *(2026-09-29: benchmarked on CPU, F1 0.867, 8.9 s/page, 2.4 GB; see architecture/evidence/ocr-bench-2026-09-29/)* + - [x] [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) — established general-purpose OCR. *(2026-09-29: benchmarked on CPU, F1 0.881, 36.5 s/page, 1.4 GB; not yet wired as an engine)* + - [ ] [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) — vision-language OCR model. *(not run 2026-09-29: only ~14 GB of the unified memory is free; needs a running model stopped for a test window)* + - [ ] *(not run 2026-09-29: ~30 GB does not fit beside the running sglang model)* Qwen3-VL-30B-A3B-Instruct-FP8 — general vision-language model; check whether it fits the RAM/VRAM budget at 30B-A3B FP8 (likely the heaviest candidate — weigh against the "low footprint" goal specifically) and whether it's meaningfully better at open-ended description than the nano-class candidates above. diff --git a/architecture/evidence/ocr-bench-2026-09-29/README.md b/architecture/evidence/ocr-bench-2026-09-29/README.md new file mode 100644 index 0000000..bb09cb7 --- /dev/null +++ b/architecture/evidence/ocr-bench-2026-09-29/README.md @@ -0,0 +1,44 @@ +# OCR candidate benchmark — 2026-09-29 + +Harness: `scripts/bench_ocr.py`. Corpus: *c't KI Wissen 2026* (native text layer, 204 pages). Six evenly spaced +pages with at least 250 words (5, 32, 67, 101, 138, 169), rendered at 110 dpi (the app's ingest default). The +PDF's own text layer is the ground truth. Raw numbers: `bench_*.json` (metrics only, no magazine text). + +| Engine | Where it ran | Mean word F1 | Seq. similarity | Median s/page | Memory | Licence | +|---|---|---:|---:|---:|---|---| +| tesseract deu+eng (current last resort) | CPU, app container | 0.880 | 0.841 | 2.2 | negligible | Apache-2.0 | +| baidu/Unlimited-OCR (current primary) | GPU container, port 8098 | 0.865 (5/6 pages) | 0.865 | 44.1 | see `docker stats` | as published | +| docling (default pipeline + RapidOCR) | CPU, venv | 0.867 | 0.783 | 8.9 (first page 66 s warm-up) | 2.4 GB RSS | MIT | +| PaddleOCR 3.2 (PP-OCRv6 medium, `de`) | CPU, venv | 0.881 | 0.551 | 36.5 | 1.4 GB RSS | Apache-2.0 | +| DeepSeek-OCR | **not run** | | | | ~7 GB weights | MIT | +| Qwen3-VL-30B-A3B-Instruct-FP8 | **not run** | | | | ~30 GB weights | Apache-2.0 | +| KDL-Frontier-Parser-nano (current fallback) | **not run**: container exited (137) 6 days ago | | | | | | + +Metrics: *word F1* is order-insensitive overlap of lower-cased word tokens; *sequence similarity* is a difflib ratio +over the token sequence and so also rewards correct reading order. Both are capped below 1.0 by things a text layer +contains that a page transcription need not (running heads, hyphenation, sidebars), so compare engines to each other, +not to 1.0. + +## Reading +- **Plain transcription accuracy is a wash.** Every engine lands at F1 0.865–0.881 on these pages; the differences are + within page-to-page variation (0.77–0.96). Nothing here justifies replacing tesseract for text quality alone. +- **Speed differs by 20×.** Tesseract 2.2 s/page beats the GPU vision engine (44 s/page) and both CPU candidates. + The vision engine also returned nothing for one of six pages (page 101), which the app's fallback chain absorbs. +- **What the heavier engines add is structure, not characters:** Unlimited-OCR labels regions and furniture, + docling returns layout-ordered text (sequence similarity 0.78 vs PaddleOCR's 0.55, which is raw line order). + Neither is scored here; the app's own layout stage already does this work, so the case for a candidate rests on + region/table/figure quality, which needs a different benchmark (labelled boxes), not this one. +- **Open-ended figure description** (the harder requirement in TODO) is untested: none of the engines run here is a + captioning model, and the two that are (DeepSeek-OCR, Qwen3-VL) could not be loaded — see below. + +## Why two candidates were not run +The host has 121 GB of unified memory with about 107 GB in use (the running sglang model dominates; GPU and system +memory are the same pool on this machine), leaving roughly 14 GB. Qwen3-VL-30B FP8 (~30 GB) cannot be loaded without +stopping another model, and DeepSeek-OCR (~7 GB, pinned dependencies on a GB10 GPU) would leave almost no margin: an +out-of-memory kill there could take down the running stack. Both need a decision from the owner: which running model to +stop for a test window. + +## Caveats +Six pages of one German-language, native-text magazine; CPU timings for docling and PaddleOCR (both would be faster on +GPU); one run each. Enough to rule out an obvious accuracy win, not to pick a winner for scanned or image-heavy issues — +this magazine's pages are born-digital, so a scanned issue would separate the engines more. diff --git a/architecture/evidence/ocr-bench-2026-09-29/bench_baseline.json b/architecture/evidence/ocr-bench-2026-09-29/bench_baseline.json new file mode 100644 index 0000000..89e5f07 --- /dev/null +++ b/architecture/evidence/ocr-bench-2026-09-29/bench_baseline.json @@ -0,0 +1,136 @@ +{ + "pdf": "source.pdf", + "pages": [ + 5, + 32, + 67, + 101, + 138, + 169 + ], + "dpi": 110, + "results": [ + { + "engine": "tesseract", + "pages": [ + { + "page": 5, + "sec": 2.2, + "ref_words": 709, + "hyp_words": 687, + "f1": 0.8954, + "seq": 0.884, + "error": null + }, + { + "page": 32, + "sec": 2.3, + "ref_words": 681, + "hyp_words": 659, + "f1": 0.8791, + "seq": 0.6925, + "error": null + }, + { + "page": 67, + "sec": 1.3, + "ref_words": 457, + "hyp_words": 376, + "f1": 0.7899, + "seq": 0.7731, + "error": null + }, + { + "page": 101, + "sec": 1.2, + "ref_words": 365, + "hyp_words": 360, + "f1": 0.9572, + "seq": 0.9517, + "error": null + }, + { + "page": 138, + "sec": 2.3, + "ref_words": 653, + "hyp_words": 615, + "f1": 0.8864, + "seq": 0.877, + "error": null + }, + { + "page": 169, + "sec": 2.2, + "ref_words": 667, + "hyp_words": 631, + "f1": 0.8737, + "seq": 0.8706, + "error": null + } + ], + "load_sec": 0.0, + "process_rss_growth_mb": 0 + }, + { + "engine": "vision:http://host.docker.internal:8098|baidu/Unlimited-OCR", + "pages": [ + { + "page": 5, + "sec": 50.6, + "ref_words": 709, + "hyp_words": 691, + "f1": 0.8886, + "seq": 0.8886, + "error": null + }, + { + "page": 32, + "sec": 46.3, + "ref_words": 681, + "hyp_words": 652, + "f1": 0.8792, + "seq": 0.8762, + "error": null + }, + { + "page": 67, + "sec": 26.7, + "ref_words": 457, + "hyp_words": 361, + "f1": 0.7971, + "seq": 0.7971, + "error": null + }, + { + "page": 101, + "sec": 3.5, + "ref_words": 365, + "hyp_words": 0, + "f1": 0.0, + "seq": 0.0, + "error": "RuntimeError: engine returned nothing" + }, + { + "page": 138, + "sec": 44.1, + "ref_words": 653, + "hyp_words": 621, + "f1": 0.8917, + "seq": 0.8917, + "error": null + }, + { + "page": 169, + "sec": 37.3, + "ref_words": 667, + "hyp_words": 640, + "f1": 0.8707, + "seq": 0.8707, + "error": null + } + ], + "load_sec": 0.1, + "process_rss_growth_mb": 13 + } + ] +} \ No newline at end of file diff --git a/architecture/evidence/ocr-bench-2026-09-29/bench_docling.json b/architecture/evidence/ocr-bench-2026-09-29/bench_docling.json new file mode 100644 index 0000000..a8d14be --- /dev/null +++ b/architecture/evidence/ocr-bench-2026-09-29/bench_docling.json @@ -0,0 +1,75 @@ +{ + "pdf": "source.pdf", + "pages": [ + 5, + 32, + 67, + 101, + 138, + 169 + ], + "dpi": 110, + "results": [ + { + "engine": "docling", + "pages": [ + { + "page": 5, + "sec": 66.2, + "ref_words": 709, + "hyp_words": 679, + "f1": 0.8847, + "seq": 0.7695, + "error": null + }, + { + "page": 32, + "sec": 9.9, + "ref_words": 681, + "hyp_words": 653, + "f1": 0.8861, + "seq": 0.6762, + "error": null + }, + { + "page": 67, + "sec": 6.7, + "ref_words": 457, + "hyp_words": 340, + "f1": 0.7729, + "seq": 0.7704, + "error": null + }, + { + "page": 101, + "sec": 6.2, + "ref_words": 365, + "hyp_words": 350, + "f1": 0.9231, + "seq": 0.9175, + "error": null + }, + { + "page": 138, + "sec": 7.9, + "ref_words": 653, + "hyp_words": 607, + "f1": 0.8825, + "seq": 0.7873, + "error": null + }, + { + "page": 169, + "sec": 12.4, + "ref_words": 667, + "hyp_words": 622, + "f1": 0.8534, + "seq": 0.7742, + "error": null + } + ], + "load_sec": 3.5, + "process_rss_growth_mb": 2397 + } + ] +} \ No newline at end of file diff --git a/architecture/evidence/ocr-bench-2026-09-29/bench_paddle.json b/architecture/evidence/ocr-bench-2026-09-29/bench_paddle.json new file mode 100644 index 0000000..2841883 --- /dev/null +++ b/architecture/evidence/ocr-bench-2026-09-29/bench_paddle.json @@ -0,0 +1,75 @@ +{ + "pdf": "source.pdf", + "pages": [ + 5, + 32, + 67, + 101, + 138, + 169 + ], + "dpi": 110, + "results": [ + { + "engine": "paddleocr", + "pages": [ + { + "page": 5, + "sec": 49.0, + "ref_words": 709, + "hyp_words": 688, + "f1": 0.889, + "seq": 0.4839, + "error": null + }, + { + "page": 32, + "sec": 46.4, + "ref_words": 681, + "hyp_words": 664, + "f1": 0.8907, + "seq": 0.4848, + "error": null + }, + { + "page": 67, + "sec": 29.7, + "ref_words": 457, + "hyp_words": 466, + "f1": 0.7822, + "seq": 0.4225, + "error": null + }, + { + "page": 101, + "sec": 27.9, + "ref_words": 365, + "hyp_words": 363, + "f1": 0.9533, + "seq": 0.9533, + "error": null + }, + { + "page": 138, + "sec": 41.1, + "ref_words": 653, + "hyp_words": 622, + "f1": 0.9035, + "seq": 0.4863, + "error": null + }, + { + "page": 169, + "sec": 31.8, + "ref_words": 667, + "hyp_words": 638, + "f1": 0.869, + "seq": 0.4736, + "error": null + } + ], + "load_sec": 13.7, + "process_rss_growth_mb": 1355 + } + ] +} \ No newline at end of file diff --git a/scripts/bench_ocr.py b/scripts/bench_ocr.py new file mode 100644 index 0000000..c1f2673 --- /dev/null +++ b/scripts/bench_ocr.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 +"""Benchmark OCR engines on pages of a native-text PDF, using its text layer as ground truth. + + python3 scripts/bench_ocr.py --pdf source.pdf --pages 12,20,30 --engine tesseract \ + --engine "vision:http://localhost:8098|baidu/Unlimited-OCR" --engine docling --engine paddleocr + +The PDF's own text layer is what a perfect transcription of the rendered page would say, so no +hand-labelled data is needed. Two scores per page, both on lower-cased word tokens: + * bag F1 -- order-insensitive overlap (robust to differing column reading order) + * seq sim -- difflib ratio over the token sequence (penalises scrambled order) +Also wall-clock seconds per page and the peak resident memory of THIS process (a remote engine's own +footprint is not visible here; read it from `docker stats`). + +Engines are optional: one whose library or server is missing is reported as unavailable, not an error. +Read-only: renders into a temp directory and writes nothing else unless --out is given. +""" +from __future__ import annotations + +import argparse +import difflib +import json +import re +import resource +import statistics +import sys +import tempfile +import time +from collections import Counter +from pathlib import Path + +_TOKEN = re.compile(r"[\wÄÖÜäöüß]+", re.UNICODE) + + +def tokens(text: str) -> list[str]: + # Hyphenation at line ends ("zu-\nführen") is joined so it does not count against an engine + # that correctly re-joins it (or for one that keeps the break). + text = re.sub(r"-\s*\n\s*", "", text or "") + return [t.lower() for t in _TOKEN.findall(text)] + + +def bag_f1(ref: list[str], hyp: list[str]) -> float: + if not ref or not hyp: + return 0.0 + overlap = sum((Counter(ref) & Counter(hyp)).values()) + if not overlap: + return 0.0 + p, r = overlap / len(hyp), overlap / len(ref) + return 2 * p * r / (p + r) + + +def seq_sim(ref: list[str], hyp: list[str]) -> float: + if not ref or not hyp: + return 0.0 + return difflib.SequenceMatcher(None, ref, hyp, autojunk=False).ratio() + + +def render(pdf: Path, page: int, dpi: int, out: Path) -> Path: + import pymupdf + with pymupdf.open(str(pdf)) as doc: + doc[page - 1].get_pixmap(dpi=dpi).save(str(out)) + return out + + +def truth(pdf: Path, page: int) -> str: + import pymupdf + with pymupdf.open(str(pdf)) as doc: + return doc[page - 1].get_text() + + +def pick_pages(pdf: Path, count: int, min_words: int) -> list[int]: + """Evenly spaced pages with enough text to score meaningfully (adverts and photo pages drop out).""" + import pymupdf + with pymupdf.open(str(pdf)) as doc: + eligible = [n + 1 for n in range(len(doc)) if len(tokens(doc[n].get_text())) >= min_words] + if len(eligible) <= count: + return eligible + step = len(eligible) / count + return [eligible[int(i * step)] for i in range(count)] + + +# ── engines: each is a callable (png_path) -> transcribed text, or raises Unavailable ───────────── + +class Unavailable(Exception): + pass + + +def engine_tesseract(lang: str = "deu+eng"): + import shutil + import subprocess + if not shutil.which("tesseract"): + raise Unavailable("tesseract is not installed here") + + def run(png: Path) -> str: + return subprocess.run(["tesseract", str(png), "stdout", "-l", lang], capture_output=True, + text=True, timeout=180, check=True).stdout + return run + + +def engine_vision(spec: str): + """`vision:|` -- an OpenAI-compatible vision server, called exactly as the app calls it.""" + try: + base, model = spec.split("|", 1) + except ValueError: + raise Unavailable("use vision:|") + sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + try: + from core import magazine as mag + except Exception as e: # the app's own module needs its environment + raise Unavailable(f"cannot import the app's OCR client ({type(e).__name__})") + import requests + try: + requests.get(base.rstrip("/") + ("/v1/models" if "/v1" not in base else "/models"), timeout=5).raise_for_status() + except Exception as e: + raise Unavailable(f"server not reachable ({type(e).__name__})") + + def run(png: Path) -> str: + regions = mag.vision_ocr_page(png, base, model, timeout=300.0) + if regions is None: + raise RuntimeError("engine returned nothing") + return "\n".join(str(r.get("text") or "") for r in regions) + return run + + +def engine_docling(): + try: + from docling.document_converter import DocumentConverter + except Exception as e: + raise Unavailable(f"docling not installed ({type(e).__name__})") + conv = DocumentConverter() + + def run(png: Path) -> str: + return conv.convert(str(png)).document.export_to_text() + return run + + +def engine_paddleocr(lang: str = "de"): + try: + from paddleocr import PaddleOCR + except Exception as e: + raise Unavailable(f"paddleocr not installed ({type(e).__name__})") + try: # 3.x: page orientation/unwarping models are extra work a straight scan does not need + ocr = PaddleOCR(lang=lang, use_doc_orientation_classify=False, use_doc_unwarping=False, + use_textline_orientation=False) + except TypeError: # 2.x + ocr = PaddleOCR(lang=lang, use_angle_cls=False) + + def run(png: Path) -> str: + lines: list[str] = [] + res = ocr.predict(str(png)) if hasattr(ocr, "predict") else ocr.ocr(str(png)) + for page in res or []: + if isinstance(page, dict) or hasattr(page, "get"): # 3.x result object + lines.extend(str(t) for t in (page.get("rec_texts") or [])) + continue + for item in page or []: # 2.x [[box, (text, score)], ...] + try: + lines.append(item[1][0]) + except (IndexError, TypeError): + pass + return "\n".join(lines) + return run + + +def build_engine(spec: str): + if spec == "tesseract": + return engine_tesseract() + if spec.startswith("vision:"): + return engine_vision(spec[len("vision:"):]) + if spec == "docling": + return engine_docling() + if spec == "paddleocr": + return engine_paddleocr() + raise Unavailable(f"unknown engine {spec!r}") + + +def peak_rss_mb() -> float: + return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.0 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--pdf", required=True, type=Path) + ap.add_argument("--pages", default="", help="comma-separated 1-based page numbers (default: evenly spaced)") + ap.add_argument("--count", type=int, default=8, help="pages to sample when --pages is not given") + ap.add_argument("--min-words", type=int, default=250) + ap.add_argument("--dpi", type=int, default=110, help="110 matches the app's ingest default") + ap.add_argument("--engine", action="append", required=True) + ap.add_argument("--out", type=Path, help="write the full results as JSON") + args = ap.parse_args() + + pages = [int(p) for p in args.pages.split(",") if p.strip()] or pick_pages(args.pdf, args.count, args.min_words) + print(f"pages: {pages}") + results = [] + with tempfile.TemporaryDirectory(prefix="bench-ocr-") as tmp: + pngs = {p: render(args.pdf, p, args.dpi, Path(tmp) / f"p{p}.png") for p in pages} + refs = {p: tokens(truth(args.pdf, p)) for p in pages} + for spec in args.engine: + entry = {"engine": spec, "pages": []} + try: + base_rss = peak_rss_mb() + t0 = time.perf_counter() + run = build_engine(spec) + entry["load_sec"] = round(time.perf_counter() - t0, 1) + except Unavailable as e: + entry["unavailable"] = str(e) + results.append(entry) + print(f"{spec}: unavailable — {e}") + continue + for p in pages: + t0 = time.perf_counter() + try: + hyp = tokens(run(pngs[p])) + err = None + except Exception as e: + hyp, err = [], f"{type(e).__name__}: {e}"[:200] + sec = time.perf_counter() - t0 + entry["pages"].append({ + "page": p, "sec": round(sec, 1), "ref_words": len(refs[p]), "hyp_words": len(hyp), + "f1": round(bag_f1(refs[p], hyp), 4), "seq": round(seq_sim(refs[p], hyp), 4), "error": err}) + print(f" {spec} p{p}: f1={entry['pages'][-1]['f1']:.3f} seq={entry['pages'][-1]['seq']:.3f} {sec:.1f}s" + + (f" ERROR {err}" if err else "")) + entry["process_rss_growth_mb"] = round(peak_rss_mb() - base_rss) + results.append(entry) + + print("\n| engine | pages ok | mean F1 | mean seq | median s/page | RSS growth MB |") + print("|---|---:|---:|---:|---:|---:|") + for e in results: + ok = [r for r in e.get("pages", []) if not r["error"]] + if not ok: + print(f"| {e['engine']} | 0/{len(e.get('pages', []))} | – | – | – | – | {e.get('unavailable', '')}") + continue + print(f"| {e['engine']} | {len(ok)}/{len(e['pages'])} | {statistics.mean(r['f1'] for r in ok):.3f} | " + f"{statistics.mean(r['seq'] for r in ok):.3f} | {statistics.median(r['sec'] for r in ok):.1f} | " + f"{e.get('process_rss_growth_mb', 0)} |") + if args.out: + args.out.write_text(json.dumps({"pdf": args.pdf.name, "pages": pages, "dpi": args.dpi, "results": results}, + ensure_ascii=False, indent=1), "utf-8") + return 0 + + +if __name__ == "__main__": + sys.exit(main())