diff --git a/CHANGELOG.md b/CHANGELOG.md
index 77fb842..a0ede42 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,11 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
- **Text flow indicator.** Review draws a green arrow where an article's text carries on to the next box below or the next column, and chips where it continues on another page (*continues on page 111 ›*, *‹ from page 110*) plus *article starts / ends*; a *Show text flow* switch hides them. The reading text shows small markers (`↓`, `col ›`, `p.111 ›`) at the exact word where a paragraph crosses a box, column or page. Built from the same fusion the narration uses (`GET .../articles/{id}/flow`), so the arrows show what is read as one paragraph. On the measured issue: 97 column, 29 page and 21 below continuations.
- **The whole-magazine button is now named *Auto-fix all pages*** (it fixes boxes, adds pull quotes, finds missed text, re-reads words and fixes provable misspellings on every page) and keeps a layout backup first.
+- **Continuation pages are linked into their articles automatically.** Detection often found an article's headline page and left the pages that follow as one-page stand-ins ("Page 61") that had to be approved one by one (79 of 132 articles on the measured issue). Text no article owns is now attached to the article running from the page before, unless that article already ended (a signed end such as `(hag)`), it is more than two pages back, or a headline with a byline/deck opens a new article there. Approved articles that gain text go back to *detected* (nothing unreviewed is ever narrated); stand-ins whose text moved are removed. Runs at import, in **Auto-fix all pages**, and on its own (`POST .../articles/auto-link`, undoable).
+- **Link an article to a page by hand:** *Link to page N — Article continues there / starts there* in the review panel picks the text that belongs (the paragraph that carries the sentence on, up to the next headline) and adds it ticked; undoable.
+- **Flow arrows follow the reading order:** a green arrow for every jump to another column and for a sentence running across boxes (dashed when a new paragraph starts, solid when a sentence continues), chips for page jumps.
+- **Much less noise:** the *Auto-fix all pages* report now lists what was fixed and only the pages that need a decision about their boxes (with a *Start with page N* button and compact page pills); pages with only words to check are collapsed and optional. The per-box "words look mis-read" marks are gone (spelling keeps its red wavy underline). The spellchecker knows ~130 company/product names, treats a word printed twice anywhere in the issue as vocabulary (missing-umlaut words such as `fur` stay flagged) and treats two unknown capitalised words in a row as a person's name: 871 → 625 findings on the measured issue.
+
### Fixed
- **The review list shows each region's whole text**, not the first 220 characters, so every misspelling gets its red wavy underline (like a word processor). The engine's `[Non-Text]` marker and hairline boxes (under 3 pt tall) are now treated as OCR noise by *Clean up boxes* and after a page re-OCR, so they no longer show up as near-invisible regions. Fitted boxes keep a little air on grown sides.
- **The quality check no longer flags correct text as "suspect OCR".** The word check guessed the language per fragment (`iPhones statt Token` read as English) and flagged valid German; it now uses the page's language, the same lenient German rules as the spellchecker, and treats words that recur on the page (names, brands) as vocabulary. Page 61 of the measured issue went from 11 suspect regions to 3 (one a genuinely truncated word), and the issue from 758 to 610 before the language fix's full effect.
diff --git a/Manual.md b/Manual.md
index cb5a244..3ccda0e 100644
--- a/Manual.md
+++ b/Manual.md
@@ -291,6 +291,11 @@ unreviewed boundaries may include a neighbouring article or an advert.
**Whole magazine:** **Auto-fix all pages** (contents view, formerly *Quality check / run*) applies the same fixes to every page and then lists the pages that still need a look (purple badges).
**Errors:** a failed fix shows the reason and changes nothing. **Offline / privacy:** the box fixes and the audit are local; the OCR re-read and spelling steps use only the engines you configured.
+### Less manual work: what happens automatically
+
+At import, and whenever you press **Auto-fix all pages**: boxes are fitted, inset pull quotes found, wrapped paragraphs cut, continuation pages **linked into the article running from the page before**, missed text added, suspicious words re-read and (with a language model) provable misspellings fixed. The report then lists only the pages that need a *decision about their boxes*, with **Start with page N**; pages that only have words to check are collapsed and optional. Articles that gained pages go back to *detected*, so re-approve them (the batch tools in *Select* do many at once). A backup of the layout is kept.
+**When the automatic link misses:** open the article in review and use **Link to page N** in the panel: *Article continues there* adds the text that belongs on that page (from the paragraph that carries the sentence on, up to the next headline), *Article starts there* adds the opening. Added text is ticked, so untick what does not belong. **Undo** reverses it.
+
### Clean up boxes: fitting boxes to the text
**Clean up boxes** (review toolbar) also fits each text box to the ink of its text: a side that cuts through letters grows a little, empty margin is trimmed. Boxes you drew or resized are never changed; a backup of the layout is kept and the message reports how many boxes were fitted.
diff --git a/core/known_names.json b/core/known_names.json
new file mode 100644
index 0000000..ac0ce25
--- /dev/null
+++ b/core/known_names.json
@@ -0,0 +1,15 @@
+{
+ "names": [
+ "Nvidia", "OpenAI", "Anthropic", "Claude", "Gemini", "ChatGPT", "GPT", "Copilot", "Microsoft", "Google", "Apple", "Siri",
+ "Meta", "Llama", "Mistral", "DeepSeek", "Qwen", "Grok", "xAI", "Tesla", "Amazon", "Alexa", "Samsung", "Huawei", "Intel",
+ "AMD", "Qualcomm", "Databricks", "Salesforce", "Zapier", "Coinbase", "GitHub", "GitLab", "Docker", "Kubernetes", "Linux",
+ "Ubuntu", "Debian", "Windows", "Android", "iOS", "macOS", "iPhone", "iPad", "WhatsApp", "Telegram", "Signal", "LinkedIn",
+ "YouTube", "Instagram", "TikTok", "Spotify", "Netflix", "Nemotron", "PyTorch", "TensorFlow", "Hugging", "Midjourney",
+ "Perplexity", "Heise", "Brave", "Firefox", "Chrome", "Chromium", "Safari", "Edge", "Bing", "Azure", "AWS", "Oracle", "SAP",
+ "IBM", "Cisco", "Dell", "Lenovo", "Asus", "Acer", "Sony", "Nintendo", "Steam", "Valve", "Adobe", "Figma", "Notion", "Slack",
+ "Zoom", "Teams", "Excel", "Word", "PowerPoint", "Outlook", "OneDrive", "Dropbox", "Cloudflare", "Nextcloud", "Proxmox",
+ "Raspberry", "Arduino", "Python", "Rust", "Java", "JavaScript", "TypeScript", "Kotlin", "Swift", "Golang", "Docker",
+ "Ollama", "LiteLLM", "Whisper", "Stable", "Diffusion", "Flux", "ComfyUI", "Mixtral", "Phi", "Gemma", "Kimi", "Moonshot",
+ "Baidu", "Alibaba", "Tencent", "ByteDance", "Xiaomi", "Yandex", "Palantir", "Snowflake", "Stripe", "PayPal", "Klarna"
+ ]
+}
diff --git a/core/magazine.py b/core/magazine.py
index 60f6206..1839382 100644
--- a/core/magazine.py
+++ b/core/magazine.py
@@ -4621,32 +4621,189 @@ def fuse_paragraph_blocks(blocks: list[dict[str, Any]], language: str = "auto",
return out
-def article_flow(layout: dict[str, Any], article: dict[str, Any]) -> dict[str, Any]:
- """Where an article's text carries on from one region to the next.
+def _flow_relation(a: dict[str, Any], b: dict[str, Any]) -> str:
+ """How the eye travels from region a to region b: below / column / page."""
+ if a["page"] != b["page"]:
+ return "page"
+ ax, aw = a["bbox"][0], a["bbox"][2]
+ bx, bw = b["bbox"][0], b["bbox"][2]
+ overlap = max(0.0, min(ax + aw, bx + bw) - max(ax, bx))
+ same_column = overlap >= 0.5 * min(aw, bw) # also a wide headline above its column
+ return "below" if same_column and b["bbox"][1] >= a["bbox"][1] else "column"
- Built from the same fusion the narration uses, so what is drawn is exactly what is read as one
- paragraph. Each link says how the reader's eye has to travel: `below` (the next region under
- it, same column), `column` (top of another column on the same page) or `page` (another page).
+
+def article_flow(layout: dict[str, Any], article: dict[str, Any]) -> dict[str, Any]:
+ """Where an article's text carries on, region to region, in reading order.
+
+ Every consecutive pair of narrated regions is a link. `relation` says how the eye travels:
+ `below` (the next region under it, same column), `column` (top of another column on the same
+ page) or `page` (another page). `sentence` is true when a sentence runs on across the link
+ (the same fusion narration uses), false when the next region simply starts a new paragraph.
Also returns the article's first and last regions.
"""
+ plain = article_text(layout, article)
fused = article_text(layout, article, fuse=True)
- links: list[dict[str, Any]] = []
+ running: set[tuple[str, str]] = set()
for block in fused:
parts = block.get("parts") or []
- for a, b in zip(parts, parts[1:]):
- if a["page"] != b["page"]:
- relation = "page"
- else:
- ax, aw = a["bbox"][0], a["bbox"][2]
- bx, bw = b["bbox"][0], b["bbox"][2]
- same_column = abs((ax + aw / 2) - (bx + bw / 2)) <= 0.5 * min(aw, bw)
- relation = "below" if same_column and b["bbox"][1] >= a["bbox"][1] else "column"
- links.append({"from": a["id"], "to": b["id"], "relation": relation,
- "from_page": a["page"], "to_page": b["page"]})
- narrated = [b for b in fused if b.get("type") in NARRATABLE]
- return {"links": links,
- "first": (narrated[0].get("parts") or [{"id": narrated[0]["id"]}])[0]["id"] if narrated else None,
- "last": (narrated[-1].get("parts") or [{"id": narrated[-1]["id"]}])[-1]["id"] if narrated else None}
+ running.update((a["id"], b["id"]) for a, b in zip(parts, parts[1:]))
+ narrated = [b for b in plain if b.get("type") in NARRATABLE]
+ links = [{"from": a["id"], "to": b["id"], "relation": _flow_relation(a, b),
+ "from_page": a["page"], "to_page": b["page"], "sentence": (a["id"], b["id"]) in running}
+ for a, b in zip(narrated, narrated[1:])]
+ return {"links": links, "first": narrated[0]["id"] if narrated else None,
+ "last": narrated[-1]["id"] if narrated else None}
+
+
+_ARTICLE_END_RE = re.compile(r"\(\s*[A-Za-zÄÖÜäöü]{2,4}\s*\)\s*\S{0,4}\s*$") # "(hag)", "(atr) c't": a signed end
+_PLACEHOLDER_TITLE = re.compile(r"Page \d+")
+
+
+def _is_placeholder_article(article: dict[str, Any]) -> bool:
+ return bool(_PLACEHOLDER_TITLE.fullmatch((article.get("title") or "").strip()))
+
+
+def auto_link_pages(layout: dict[str, Any], articles: list[dict[str, Any]],
+ max_gap_pages: int = 2) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+ """Attach text that no real article owns to the article running from the page before.
+
+ Detection often finds an article's headline page and leaves its continuation pages as
+ one-page stand-ins ("Page 61"). Reading in page order, text no real article owns is:
+ a NEW article when a headline followed by a byline or deck starts it; otherwise a
+ CONTINUATION of the article running just before, unless that article already ended (a signed
+ end such as ``(hag)``) or is more than `max_gap_pages` pages back -- then it stays unlinked
+ for a human. Stand-in articles whose text all moved are dropped; an approved article that
+ gains text goes back to *detected* so nothing unreviewed is ever narrated.
+ Returns the articles and a report {linked, created, unlinked, dropped}.
+ """
+ articles = [dict(a, blocks=list(a.get("blocks") or []), pages=list(a.get("pages") or [])) for a in articles]
+ real = [a for a in articles if not _is_placeholder_article(a)]
+ owner = {b: a for a in real for b in a["blocks"]}
+ blocks = {b["id"]: b for b in layout.get("blocks", [])}
+ by_page: dict[int, list[dict[str, Any]]] = {}
+ for b in layout.get("blocks", []):
+ if b["type"] in NARRATABLE and (b.get("text") or "").strip():
+ by_page.setdefault(b["page"], []).append(b)
+ report = {"linked": 0, "created": 0, "unlinked": [], "dropped": 0}
+ taken = {int(m.group(1)) for a in articles if (m := re.fullmatch(r"article-(\d+)", a["id"]))}
+ moved: set[str] = set()
+ running: dict[str, Any] | None = None
+ running_page = 0
+ running_tail = ""
+
+ def next_id() -> str:
+ n = max(taken, default=0) + 1
+ taken.add(n)
+ return f"article-{n:03d}"
+
+ for page in sorted(by_page):
+ ordered = sorted(by_page[page], key=lambda b: b.get("reading_order", 0))
+ i = 0
+ while i < len(ordered):
+ block = ordered[i]
+ if block["id"] in owner:
+ running, running_page, running_tail = owner[block["id"]], page, block["text"]
+ i += 1
+ continue
+ j = i # collect this run of unowned text
+ while j < len(ordered) and ordered[j]["id"] not in owner:
+ j += 1
+ segment = ordered[i:j]
+ i = j
+ cut = [0]
+ for k, b in enumerate(segment): # a headline + byline/deck opens a new piece
+ nxt = [x["type"] for x in segment[k + 1:k + 3]]
+ if k and b["type"] in (BLOCK_TITLE, BLOCK_HEADING) and (BLOCK_BYLINE in nxt or BLOCK_DECK in nxt):
+ cut.append(k)
+ for start, end in zip(cut, cut[1:] + [len(segment)]):
+ part = segment[start:end]
+ head = part[0]
+ opens = head["type"] in (BLOCK_TITLE, BLOCK_HEADING) and any(
+ x["type"] in (BLOCK_BYLINE, BLOCK_DECK) for x in part[1:3])
+ if opens:
+ byline = next((x for x in part[1:4] if x["type"] == BLOCK_BYLINE), None)
+ fresh = Article(id=next_id(), title=" ".join(head["text"].split())[:300],
+ author=re.sub(r"^(von|by)\s+", "", (byline or {}).get("text", ""), flags=re.I).strip()[:120],
+ pages=sorted({x["page"] for x in part}), blocks=[x["id"] for x in part],
+ confidence=0.55).to_json()
+ articles.append(fresh)
+ real.append(fresh)
+ owner.update({b: fresh for b in fresh["blocks"]})
+ moved.update(fresh["blocks"])
+ running, running_page, running_tail = fresh, page, part[-1]["text"]
+ report["created"] += 1
+ elif (running is not None and page - running_page <= max_gap_pages
+ and not _ARTICLE_END_RE.search(running_tail.strip())):
+ running["blocks"].extend(x["id"] for x in part)
+ running["pages"] = sorted(set(running["pages"]) | {x["page"] for x in part})
+ if running.get("status") == "approved":
+ running["status"] = "detected"
+ owner.update({x["id"]: running for x in part})
+ moved.update(x["id"] for x in part)
+ running_page, running_tail = page, part[-1]["text"]
+ report["linked"] += len(part)
+ else:
+ report["unlinked"].append(part[0]["id"])
+ absorbed_by_page: dict[int, dict[str, Any]] = {}
+ for article in real:
+ for b in article["blocks"]:
+ if b in moved and b in blocks:
+ absorbed_by_page[blocks[b]["page"]] = article
+ kept = []
+ for a in articles:
+ if _is_placeholder_article(a):
+ # text that moved, or that a real article owns anyway, no longer needs a stand-in
+ a["blocks"] = [b for b in a["blocks"] if b not in moved and b not in owner]
+ still_text = any(blocks.get(b, {}).get("type") in NARRATABLE for b in a["blocks"])
+ if not still_text:
+ target = next((absorbed_by_page[p] for p in a.get("pages", []) if p in absorbed_by_page), None)
+ if target is not None and a.get("images"): # its pictures follow the text
+ target["images"] = list(dict.fromkeys([*target.get("images", []), *a["images"]]))
+ report["dropped"] += 1
+ continue
+ kept.append(a)
+ return kept, report
+
+
+_LINK_BOUNDARY_TYPES = (BLOCK_TITLE, BLOCK_HEADING, BLOCK_DECK, BLOCK_BYLINE)
+
+
+def blocks_to_link(layout: dict[str, Any], article: dict[str, Any], page: int, where: str) -> list[str]:
+ """Regions on `page` to add so the article continues there (`after`) or starts there (`before`).
+
+ after: from the first body region that carries the article on (the sentence runs on, or
+ simply the page's first body region) down to, but not including, the next headline, deck or
+ byline -- that is a different piece. before: from the page's last headline (the article's
+ opening) to the end, or all its body text when it has none.
+ """
+ own = set(article.get("blocks") or [])
+ on_page = sorted((b for b in layout.get("blocks", [])
+ if b["page"] == page and b["type"] in NARRATABLE and (b.get("text") or "").strip()
+ and b["id"] not in own), key=lambda b: b.get("reading_order", 0))
+ if where == "before":
+ starts = [i for i, b in enumerate(on_page) if b["type"] in _LINK_BOUNDARY_TYPES]
+ chosen = on_page[starts[-1]:] if starts else [b for b in on_page if b["type"] == BLOCK_BODY]
+ return [b["id"] for b in chosen]
+ narrated = [b for b in article_text(layout, article) if b.get("type") in NARRATABLE]
+ tail = (narrated[-1]["text"] if narrated else "")
+ first = next((i for i, b in enumerate(on_page) if b["type"] == BLOCK_BODY), None)
+ if first is None:
+ return []
+ # The paragraph that carries the article on: one starting lower-case is the surest sign of a
+ # sentence running over; failing that, one the fusion rules accept; failing that, the first.
+ bodies = [(i, normalize_ocr_text(strip_continuation_markers(b["text"])))
+ for i, b in enumerate(on_page) if b["type"] == BLOCK_BODY]
+ carry = next((i for i, t in bodies if t[:1].islower()), None)
+ if carry is None:
+ carry = next((i for i, t in bodies if paragraph_continues(tail, t)), None)
+ first = first if carry is None else carry
+ chosen = []
+ for b in on_page[first:]:
+ if b["type"] in _LINK_BOUNDARY_TYPES:
+ break
+ if b["type"] in (BLOCK_BODY, BLOCK_QUOTE):
+ chosen.append(b["id"])
+ return chosen
# ── Standalone audio (Stage E, optional) ─────────────────────────────────────
diff --git a/core/spelling.py b/core/spelling.py
index 077ad0a..a884c1f 100644
--- a/core/spelling.py
+++ b/core/spelling.py
@@ -8,7 +8,10 @@ under the guards in ``accept_agent_fix``.
"""
from __future__ import annotations
+import json
import re
+from functools import lru_cache
+from pathlib import Path
from typing import Iterable
from core import pronunciations as _pron
@@ -30,6 +33,16 @@ _MAX_ISSUES_PER_TEXT = 60
_RECURRING = 3
+@lru_cache(maxsize=1)
+def known_names() -> frozenset[str]:
+ """Company, product and technology names the dictionaries cannot know (core/known_names.json)."""
+ try:
+ data = json.loads((Path(__file__).with_name("known_names.json")).read_text("utf-8"))
+ return frozenset(str(n).casefold() for n in data.get("names", []))
+ except (OSError, ValueError):
+ return frozenset()
+
+
def _checker(language: str):
return _pron._german_spellchecker() if language == "de" else _pron._english_spellchecker()
@@ -184,7 +197,8 @@ def _close_line_breaks(text: str) -> tuple[str, list[int]]:
def find_spelling_issues(text: str, language: str = "auto",
- ignore: Iterable[str] = (), other_language: bool = True) -> list[dict]:
+ ignore: Iterable[str] = (), other_language: bool = True,
+ vocabulary: Iterable[str] = ()) -> list[dict]:
"""Dictionary-unknown words in `text`, with position and suggestions.
A word is only a finding when it is unknown in the text's language, is not
@@ -198,7 +212,8 @@ def find_spelling_issues(text: str, language: str = "auto",
if checker is None:
return []
other = _checker("en" if lang == "de" else "de") if other_language else None
- ignored = {str(w).casefold() for w in ignore}
+ ignored = {str(w).casefold() for w in ignore} | known_names()
+ vocab = {str(w).casefold() for w in vocabulary}
issues: list[dict] = []
raw = text or ""
text, origin = _close_line_breaks(raw)
@@ -218,6 +233,9 @@ def find_spelling_issues(text: str, language: str = "auto",
continue
if lang != "de" and part[:1].isupper() and not _starts_sentence(text, match.start()):
continue # mid-sentence capital: a name or product
+ if part.casefold() in vocab and not (
+ lang == "de" and any(v in checker for v in _umlaut_variants(part.casefold()))):
+ continue # printed repeatedly: a name or term, not OCR noise
if _known_form(part, checker, lang) or (other is not None and _known_form(part, other, "en" if lang == "de" else "de")):
continue
if lang == "de" and is_known_compound(part, checker):
@@ -233,12 +251,16 @@ def find_spelling_issues(text: str, language: str = "auto",
"suggestions": _suggestions(part, checker, lang),
})
if len(issues) >= _MAX_ISSUES_PER_TEXT:
- return issues
- return issues
+ break
+ # Two unknown capitalised words side by side (``Juliane Jarke``) are a person's name.
+ starts = {i["start"] for i in issues if i["token"][:1].isupper()}
+ ends = {i["end"] + 1 for i in issues if i["token"][:1].isupper()}
+ return [i for i in issues
+ if not (i["token"][:1].isupper() and (i["start"] in ends or i["end"] + 1 in starts))]
def block_issues(blocks: list[dict], article_language: str | None = None,
- ignore: Iterable[str] = ()) -> list[dict]:
+ ignore: Iterable[str] = (), vocabulary: Iterable[str] = ()) -> list[dict]:
"""Findings for a run of blocks, each with its block id, page and context."""
ignore = set(ignore)
language = article_language or detect_article_language(b.get("text", "") for b in blocks)
@@ -246,7 +268,7 @@ def block_issues(blocks: list[dict], article_language: str | None = None,
for block in blocks:
text = block.get("text") or ""
lang = language_for_block(text, language)
- for issue in find_spelling_issues(text, lang, ignore):
+ for issue in find_spelling_issues(text, lang, ignore, vocabulary=vocabulary):
lo = max(0, issue["start"] - 60)
hi = min(len(text), issue["end"] + 60)
out.append(dict(issue, block_id=block["id"], page=block.get("page"),
diff --git a/routes/magazine.py b/routes/magazine.py
index dc2cc3e..9826d19 100644
--- a/routes/magazine.py
+++ b/routes/magazine.py
@@ -46,6 +46,7 @@ from core import spelling
from core.pronunciations import (
clean_glossary,
detect_article_language,
+ recurring_tokens,
find_unknown_acronyms,
parse_acronym_decisions,
)
@@ -356,10 +357,17 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
except Exception:
logging.exception("Failed to refresh visual regions for %s", doc_id)
layout, _ = mag.clean_up_regions(layout)
+ background_job_update(job_id, stage="articles", detail="Fitting boxes and finding inset quotes")
+ for page_meta in layout.get("pages", []):
+ try:
+ layout, _, _ = _fix_page_geometry(d, layout, [], page_meta["page"], _load_settings())
+ except Exception: # noqa: BLE001 - never fail an import over a box fix
+ logging.exception("Box fixes failed for page %s", page_meta.get("page"))
mag.save_layout(d, layout)
background_job_update(job_id, stage="articles", detail="Detecting articles")
articles = mag.build_articles(layout)
+ articles, _ = mag.auto_link_pages(layout, articles)
mag.save_articles(d, articles)
meta = _read_meta(d)
@@ -623,6 +631,53 @@ async def magazine_article_text(doc_id: str, article_id: str,
return {"article": target, "blocks": blocks}
+@router.post("/api/magazine/{doc_id}/articles/auto-link")
+async def magazine_auto_link(doc_id: str):
+ """Link continuation pages into the articles they belong to (see mag.auto_link_pages). Undoable."""
+ d, _ = _require_magazine(doc_id)
+ layout = mag.load_layout(d)
+ if not layout:
+ raise HTTPException(404, "No layout yet -- run ingestion first")
+ articles = mag.load_articles(d)
+ token = _snapshot_for_undo(d, "Linked pages into articles")
+ linked, report = await asyncio.to_thread(mag.auto_link_pages, layout, articles)
+ mag.save_articles(d, linked)
+ _seal_undo(d, token)
+ return {"ok": True, "undo": token, "before": len(articles), "after": len(linked),
+ "text_blocks_linked": report["linked"], "articles_created": report["created"],
+ "stand_ins_removed": report["dropped"], "left_for_you": len(report["unlinked"])}
+
+
+@router.post("/api/magazine/{doc_id}/articles/{article_id}/link-page")
+async def magazine_link_page(doc_id: str, article_id: str, request: Request):
+ """Link one page to this article by hand. Body: {page, where: "after" | "before"}.
+
+ after: the article carries on there; before: the article opens there. The text that belongs
+ is chosen for you (mag.blocks_to_link) and added ticked, so review shows it and it can be
+ unticked. Other articles keep their own copy of those regions until you reject them.
+ """
+ d, _ = _require_magazine(doc_id)
+ articles, target = _article_or_404(d, article_id)
+ body = await request.json()
+ try:
+ page = int(body["page"])
+ except (KeyError, TypeError, ValueError):
+ raise HTTPException(400, "page is required")
+ where = "before" if isinstance(body, dict) and body.get("where") == "before" else "after"
+ layout = mag.load_layout(d)
+ if not any(p["page"] == page for p in layout.get("pages", [])):
+ raise HTTPException(404, "Page not found")
+ ids = mag.blocks_to_link(layout, target, page, where)
+ if not ids:
+ raise HTTPException(409, f"Nothing on page {page} could be linked -- draw or tick its regions by hand")
+ token = _snapshot_for_undo(d, f"Linked page {page}")
+ target["blocks"] = list(dict.fromkeys([*(target.get("blocks") or []), *ids]))
+ target["pages"] = sorted(set(target.get("pages") or []) | {page})
+ mag.save_articles(d, articles)
+ _seal_undo(d, token)
+ return {"ok": True, "undo": token, "added": ids, "page": page}
+
+
@router.get("/api/magazine/{doc_id}/articles/{article_id}/flow")
async def magazine_article_flow(doc_id: str, article_id: str):
"""How the article's text continues between regions (below / next column / next page)."""
@@ -1094,14 +1149,18 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non
"suspect_blocks": [], "overlaps": [], "clipped": [], "measured": False, "error": str(exc)[:200]}
row.update(added=added, checked=checked, corrected=corrected,
spelling=spell_left, spelling_fixed=spell_fixed)
- row["needs_review"] = bool(row["uncovered"] or row["suspect_blocks"] or row["overlaps"]
- or row["clipped"] or row.get("error") or spell_left)
+ # Two kinds of "look at this": boxes that are wrong (someone must decide) and words
+ # that may be wrong (a long tail, mostly names). Only the first sends you to a page.
+ row["needs_review"] = bool(row["uncovered"] or row["overlaps"] or row["clipped"] or row.get("error"))
+ row["words_to_check"] = bool(row["suspect_blocks"] or spell_left)
rows.append(row)
if i % 5 == 0:
mag.save_layout(d, layout)
mag.save_layout(d, layout)
- if articles_changed:
- mag.save_articles(d, articles)
+ step_text = "linking pages into articles"
+ background_job_update(job_id, stage="quality-run", detail=f"Checking page {len(pages)} of {len(pages)}: {step_text}")
+ articles, link_report = mag.auto_link_pages(layout, articles)
+ mag.save_articles(d, articles)
duplicates = mag.duplicate_article_groups(articles)
by_id = {a["id"]: a for a in articles}
for row in rows:
@@ -1114,12 +1173,16 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non
"pages": len(rows),
"verified_with_vision": verify,
"boxes_fixed": fixed_boxes,
+ "pages_linked": link_report["linked"],
+ "articles_created": link_report["created"],
+ "standins_removed": link_report["dropped"],
"regions_added": sum(r["added"] for r in rows),
"regions_corrected": sum(r["corrected"] for r in rows),
"spelling_fixed": sum(r.get("spelling_fixed", 0) for r in rows),
"spelling_marked": sum(len(r.get("spelling") or []) for r in rows),
"spelling_agent": agent,
"pages_needing_review": [r["page"] for r in rows if r["needs_review"]],
+ "pages_with_words": [r["page"] for r in rows if r.get("words_to_check") and not r["needs_review"]],
"duplicate_article_groups": len(duplicates),
}
(d / _QUALITY_REPORT).write_text(
@@ -1127,7 +1190,8 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non
flagged = len(summary["pages_needing_review"])
background_job_update(
job_id, stage="done", done=True, result=summary,
- detail=(f"{summary['boxes_fixed']} boxes fixed, {summary['regions_added']} missed regions added, "
+ detail=(f"{summary['boxes_fixed']} boxes fixed, {summary['pages_linked']} text blocks linked into articles, "
+ f"{summary['regions_added']} missed regions added, "
f"{summary['regions_corrected']} words corrected, "
f"{summary['spelling_fixed']} misspellings fixed, {summary['spelling_marked']} marked; "
f"{flagged} page{'s' if flagged != 1 else ''} still need a look"
@@ -2129,6 +2193,29 @@ def _spelling_blocks(d: Path, layout: dict[str, Any], article_id: str | None = N
return sorted(blocks, key=lambda b: (b["page"], b.get("reading_order", 0)))
+_VOCAB_CACHE: dict[str, tuple[tuple[float, int], frozenset[str]]] = {}
+
+
+def _issue_vocabulary(d: Path, layout: dict[str, Any]) -> frozenset[str]:
+ """Words printed at least twice anywhere in the issue: names, brands and terms of its subject.
+
+ OCR mistakes differ every time they happen; a word that keeps coming back is real. Cached
+ until the layout file changes.
+ """
+ try:
+ stat = (d / "layout.json").stat()
+ key = (stat.st_mtime, stat.st_size)
+ except OSError:
+ key = (0.0, 0)
+ cached = _VOCAB_CACHE.get(str(d))
+ if cached and cached[0] == key:
+ return cached[1]
+ vocab = frozenset(recurring_tokens((b.get("text") or "" for b in layout.get("blocks", [])
+ if b.get("type") in _SPELL_TYPES), 2))
+ _VOCAB_CACHE[str(d)] = (key, vocab)
+ return vocab
+
+
def _issue_language(d: Path, layout: dict[str, Any]) -> str:
"""One language for the whole issue: a single page is too little text to decide."""
return detect_article_language(b["text"] for b in _spelling_blocks(d, layout))
@@ -2139,7 +2226,7 @@ def _spelling_findings(d: Path, layout: dict[str, Any], article_id: str | None =
blocks = _spelling_blocks(d, layout, article_id, page)
if language is None:
language = _issue_language(d, layout) if page is not None and not article_id else None
- return spelling.block_issues(blocks, language, _load_dictionary(d))
+ return spelling.block_issues(blocks, language, _load_dictionary(d), _issue_vocabulary(d, layout))
def _agent_check_block(settings: dict[str, Any], language: str, text: str,
@@ -2178,7 +2265,7 @@ def spelling_agent_pass(d: Path, layout: dict[str, Any], settings: dict[str, Any
language = language or (_issue_language(d, layout) if page is not None and not article_id
else detect_article_language(b["text"] for b in blocks))
by_block: dict[str, list[dict[str, Any]]] = {}
- for issue in spelling.block_issues(blocks, language, dictionary):
+ for issue in spelling.block_issues(blocks, language, dictionary, _issue_vocabulary(d, layout)):
by_block.setdefault(issue["block_id"], []).append(issue)
fixed = kept = failed = 0
texts = {b["id"]: b["text"] for b in blocks}
diff --git a/static/js/magazine.js b/static/js/magazine.js
index 35cb55b..170211d 100644
--- a/static/js/magazine.js
+++ b/static/js/magazine.js
@@ -3044,32 +3044,43 @@ async function magRunFigureRefine(page) {
function magRenderQualityReport(report) {
const box = $('mag-quality-report');
if (!box) return;
- const flagged = (report.pages || []).filter((p) => p.needs_review);
- if (!flagged.length) {
- box.hidden = false;
- box.textContent = 'Quality check: every page is fully covered and no suspicious words remain.';
- return;
- }
+ const sm = report.summary || {};
+ const pages = report.pages || [];
+ const boxes = pages.filter((p) => p.needs_review);
+ const words = pages.filter((p) => !p.needs_review && p.words_to_check);
+ const did = [
+ [sm.pages_linked, 'text blocks linked into articles'], [sm.boxes_fixed, 'boxes fixed'],
+ [sm.regions_added, 'missed regions added'], [sm.regions_corrected, 'words corrected'],
+ [sm.spelling_fixed, 'misspellings fixed']].filter(([n]) => n).map(([n, w]) => n + ' ' + w);
+ const tip = (p) => {
+ const bits = [];
+ if (p.uncovered.length) bits.push(p.uncovered.length + ' text area' + (p.uncovered.length === 1 ? '' : 's') + ' outside any box');
+ if (p.overlaps.length) bits.push(p.overlaps.length + ' overlapping box pair' + (p.overlaps.length === 1 ? '' : 's'));
+ if (p.clipped.length) bits.push(p.clipped.length + ' box edge' + (p.clipped.length === 1 ? '' : 's') + ' cutting text');
+ if ((p.duplicates || []).length) bits.push('duplicate articles');
+ if (p.error) bits.push('check failed');
+ if (p.words_to_check) bits.push('words to check');
+ return bits.join(', ');
+ };
+ const pill = (p) => '';
box.hidden = false;
- box.innerHTML = '' + flagged.length + ' page' + (flagged.length === 1 ? '' : 's') +
- ' still need a look: ' + flagged.map((p) => {
- const bits = [];
- if (p.uncovered.length) bits.push(p.uncovered.length + ' uncovered area' + (p.uncovered.length === 1 ? '' : 's'));
- if (p.suspect_blocks.length) bits.push(p.suspect_blocks.length + ' suspect text');
- if ((p.spelling || []).length) bits.push(p.spelling.length + ' spelling');
- if (p.overlaps.length) bits.push(p.overlaps.length + ' overlapping');
- if (p.clipped.length) bits.push(p.clipped.length + ' clipped');
- if ((p.duplicates || []).length) bits.push('duplicate articles');
- if (p.error) bits.push('check failed');
- return '';
- }).join(' ');
+ box.innerHTML =
+ '