feat(magazine): auto-link continuation pages, manual page linking, flow arrows by reading order, compact report, names/brands in spellcheck
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
d0fba6a16f
commit
05bdecd58a
@ -24,6 +24,11 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
||||
- **Text flow indicator.** Review draws a green arrow where an article's text carries on to the next box below or the next column, and chips where it continues on another page (*continues on page 111 ›*, *‹ from page 110*) plus *article starts / ends*; a *Show text flow* switch hides them. The reading text shows small markers (`↓`, `col ›`, `p.111 ›`) at the exact word where a paragraph crosses a box, column or page. Built from the same fusion the narration uses (`GET .../articles/{id}/flow`), so the arrows show what is read as one paragraph. On the measured issue: 97 column, 29 page and 21 below continuations.
|
||||
- **The whole-magazine button is now named *Auto-fix all pages*** (it fixes boxes, adds pull quotes, finds missed text, re-reads words and fixes provable misspellings on every page) and keeps a layout backup first.
|
||||
|
||||
- **Continuation pages are linked into their articles automatically.** Detection often found an article's headline page and left the pages that follow as one-page stand-ins ("Page 61") that had to be approved one by one (79 of 132 articles on the measured issue). Text no article owns is now attached to the article running from the page before, unless that article already ended (a signed end such as `(hag)`), it is more than two pages back, or a headline with a byline/deck opens a new article there. Approved articles that gain text go back to *detected* (nothing unreviewed is ever narrated); stand-ins whose text moved are removed. Runs at import, in **Auto-fix all pages**, and on its own (`POST .../articles/auto-link`, undoable).
|
||||
- **Link an article to a page by hand:** *Link to page N — Article continues there / starts there* in the review panel picks the text that belongs (the paragraph that carries the sentence on, up to the next headline) and adds it ticked; undoable.
|
||||
- **Flow arrows follow the reading order:** a green arrow for every jump to another column and for a sentence running across boxes (dashed when a new paragraph starts, solid when a sentence continues), chips for page jumps.
|
||||
- **Much less noise:** the *Auto-fix all pages* report now lists what was fixed and only the pages that need a decision about their boxes (with a *Start with page N* button and compact page pills); pages with only words to check are collapsed and optional. The per-box "words look mis-read" marks are gone (spelling keeps its red wavy underline). The spellchecker knows ~130 company/product names, treats a word printed twice anywhere in the issue as vocabulary (missing-umlaut words such as `fur` stay flagged) and treats two unknown capitalised words in a row as a person's name: 871 → 625 findings on the measured issue.
|
||||
|
||||
### Fixed
|
||||
- **The review list shows each region's whole text**, not the first 220 characters, so every misspelling gets its red wavy underline (like a word processor). The engine's `[Non-Text]` marker and hairline boxes (under 3 pt tall) are now treated as OCR noise by *Clean up boxes* and after a page re-OCR, so they no longer show up as near-invisible regions. Fitted boxes keep a little air on grown sides.
|
||||
- **The quality check no longer flags correct text as "suspect OCR".** The word check guessed the language per fragment (`iPhones statt Token` read as English) and flagged valid German; it now uses the page's language, the same lenient German rules as the spellchecker, and treats words that recur on the page (names, brands) as vocabulary. Page 61 of the measured issue went from 11 suspect regions to 3 (one a genuinely truncated word), and the issue from 758 to 610 before the language fix's full effect.
|
||||
|
||||
@ -291,6 +291,11 @@ unreviewed boundaries may include a neighbouring article or an advert.
|
||||
**Whole magazine:** **Auto-fix all pages** (contents view, formerly *Quality check / run*) applies the same fixes to every page and then lists the pages that still need a look (purple badges).
|
||||
**Errors:** a failed fix shows the reason and changes nothing. **Offline / privacy:** the box fixes and the audit are local; the OCR re-read and spelling steps use only the engines you configured.
|
||||
|
||||
### Less manual work: what happens automatically
|
||||
|
||||
At import, and whenever you press **Auto-fix all pages**: boxes are fitted, inset pull quotes found, wrapped paragraphs cut, continuation pages **linked into the article running from the page before**, missed text added, suspicious words re-read and (with a language model) provable misspellings fixed. The report then lists only the pages that need a *decision about their boxes*, with **Start with page N**; pages that only have words to check are collapsed and optional. Articles that gained pages go back to *detected*, so re-approve them (the batch tools in *Select* do many at once). A backup of the layout is kept.
|
||||
**When the automatic link misses:** open the article in review and use **Link to page N** in the panel: *Article continues there* adds the text that belongs on that page (from the paragraph that carries the sentence on, up to the next headline), *Article starts there* adds the opening. Added text is ticked, so untick what does not belong. **Undo** reverses it.
|
||||
|
||||
### Clean up boxes: fitting boxes to the text
|
||||
|
||||
**Clean up boxes** (review toolbar) also fits each text box to the ink of its text: a side that cuts through letters grows a little, empty margin is trimmed. Boxes you drew or resized are never changed; a backup of the layout is kept and the message reports how many boxes were fitted.
|
||||
|
||||
15
core/known_names.json
Normal file
15
core/known_names.json
Normal file
@ -0,0 +1,15 @@
|
||||
{
|
||||
"names": [
|
||||
"Nvidia", "OpenAI", "Anthropic", "Claude", "Gemini", "ChatGPT", "GPT", "Copilot", "Microsoft", "Google", "Apple", "Siri",
|
||||
"Meta", "Llama", "Mistral", "DeepSeek", "Qwen", "Grok", "xAI", "Tesla", "Amazon", "Alexa", "Samsung", "Huawei", "Intel",
|
||||
"AMD", "Qualcomm", "Databricks", "Salesforce", "Zapier", "Coinbase", "GitHub", "GitLab", "Docker", "Kubernetes", "Linux",
|
||||
"Ubuntu", "Debian", "Windows", "Android", "iOS", "macOS", "iPhone", "iPad", "WhatsApp", "Telegram", "Signal", "LinkedIn",
|
||||
"YouTube", "Instagram", "TikTok", "Spotify", "Netflix", "Nemotron", "PyTorch", "TensorFlow", "Hugging", "Midjourney",
|
||||
"Perplexity", "Heise", "Brave", "Firefox", "Chrome", "Chromium", "Safari", "Edge", "Bing", "Azure", "AWS", "Oracle", "SAP",
|
||||
"IBM", "Cisco", "Dell", "Lenovo", "Asus", "Acer", "Sony", "Nintendo", "Steam", "Valve", "Adobe", "Figma", "Notion", "Slack",
|
||||
"Zoom", "Teams", "Excel", "Word", "PowerPoint", "Outlook", "OneDrive", "Dropbox", "Cloudflare", "Nextcloud", "Proxmox",
|
||||
"Raspberry", "Arduino", "Python", "Rust", "Java", "JavaScript", "TypeScript", "Kotlin", "Swift", "Golang", "Docker",
|
||||
"Ollama", "LiteLLM", "Whisper", "Stable", "Diffusion", "Flux", "ComfyUI", "Mixtral", "Phi", "Gemma", "Kimi", "Moonshot",
|
||||
"Baidu", "Alibaba", "Tencent", "ByteDance", "Xiaomi", "Yandex", "Palantir", "Snowflake", "Stripe", "PayPal", "Klarna"
|
||||
]
|
||||
}
|
||||
197
core/magazine.py
197
core/magazine.py
@ -4621,32 +4621,189 @@ def fuse_paragraph_blocks(blocks: list[dict[str, Any]], language: str = "auto",
|
||||
return out
|
||||
|
||||
|
||||
def article_flow(layout: dict[str, Any], article: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Where an article's text carries on from one region to the next.
|
||||
def _flow_relation(a: dict[str, Any], b: dict[str, Any]) -> str:
|
||||
"""How the eye travels from region a to region b: below / column / page."""
|
||||
if a["page"] != b["page"]:
|
||||
return "page"
|
||||
ax, aw = a["bbox"][0], a["bbox"][2]
|
||||
bx, bw = b["bbox"][0], b["bbox"][2]
|
||||
overlap = max(0.0, min(ax + aw, bx + bw) - max(ax, bx))
|
||||
same_column = overlap >= 0.5 * min(aw, bw) # also a wide headline above its column
|
||||
return "below" if same_column and b["bbox"][1] >= a["bbox"][1] else "column"
|
||||
|
||||
Built from the same fusion the narration uses, so what is drawn is exactly what is read as one
|
||||
paragraph. Each link says how the reader's eye has to travel: `below` (the next region under
|
||||
it, same column), `column` (top of another column on the same page) or `page` (another page).
|
||||
|
||||
def article_flow(layout: dict[str, Any], article: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Where an article's text carries on, region to region, in reading order.
|
||||
|
||||
Every consecutive pair of narrated regions is a link. `relation` says how the eye travels:
|
||||
`below` (the next region under it, same column), `column` (top of another column on the same
|
||||
page) or `page` (another page). `sentence` is true when a sentence runs on across the link
|
||||
(the same fusion narration uses), false when the next region simply starts a new paragraph.
|
||||
Also returns the article's first and last regions.
|
||||
"""
|
||||
plain = article_text(layout, article)
|
||||
fused = article_text(layout, article, fuse=True)
|
||||
links: list[dict[str, Any]] = []
|
||||
running: set[tuple[str, str]] = set()
|
||||
for block in fused:
|
||||
parts = block.get("parts") or []
|
||||
for a, b in zip(parts, parts[1:]):
|
||||
if a["page"] != b["page"]:
|
||||
relation = "page"
|
||||
else:
|
||||
ax, aw = a["bbox"][0], a["bbox"][2]
|
||||
bx, bw = b["bbox"][0], b["bbox"][2]
|
||||
same_column = abs((ax + aw / 2) - (bx + bw / 2)) <= 0.5 * min(aw, bw)
|
||||
relation = "below" if same_column and b["bbox"][1] >= a["bbox"][1] else "column"
|
||||
links.append({"from": a["id"], "to": b["id"], "relation": relation,
|
||||
"from_page": a["page"], "to_page": b["page"]})
|
||||
narrated = [b for b in fused if b.get("type") in NARRATABLE]
|
||||
return {"links": links,
|
||||
"first": (narrated[0].get("parts") or [{"id": narrated[0]["id"]}])[0]["id"] if narrated else None,
|
||||
"last": (narrated[-1].get("parts") or [{"id": narrated[-1]["id"]}])[-1]["id"] if narrated else None}
|
||||
running.update((a["id"], b["id"]) for a, b in zip(parts, parts[1:]))
|
||||
narrated = [b for b in plain if b.get("type") in NARRATABLE]
|
||||
links = [{"from": a["id"], "to": b["id"], "relation": _flow_relation(a, b),
|
||||
"from_page": a["page"], "to_page": b["page"], "sentence": (a["id"], b["id"]) in running}
|
||||
for a, b in zip(narrated, narrated[1:])]
|
||||
return {"links": links, "first": narrated[0]["id"] if narrated else None,
|
||||
"last": narrated[-1]["id"] if narrated else None}
|
||||
|
||||
|
||||
_ARTICLE_END_RE = re.compile(r"\(\s*[A-Za-zÄÖÜäöü]{2,4}\s*\)\s*\S{0,4}\s*$") # "(hag)", "(atr) c't": a signed end
|
||||
_PLACEHOLDER_TITLE = re.compile(r"Page \d+")
|
||||
|
||||
|
||||
def _is_placeholder_article(article: dict[str, Any]) -> bool:
|
||||
return bool(_PLACEHOLDER_TITLE.fullmatch((article.get("title") or "").strip()))
|
||||
|
||||
|
||||
def auto_link_pages(layout: dict[str, Any], articles: list[dict[str, Any]],
|
||||
max_gap_pages: int = 2) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
||||
"""Attach text that no real article owns to the article running from the page before.
|
||||
|
||||
Detection often finds an article's headline page and leaves its continuation pages as
|
||||
one-page stand-ins ("Page 61"). Reading in page order, text no real article owns is:
|
||||
a NEW article when a headline followed by a byline or deck starts it; otherwise a
|
||||
CONTINUATION of the article running just before, unless that article already ended (a signed
|
||||
end such as ``(hag)``) or is more than `max_gap_pages` pages back -- then it stays unlinked
|
||||
for a human. Stand-in articles whose text all moved are dropped; an approved article that
|
||||
gains text goes back to *detected* so nothing unreviewed is ever narrated.
|
||||
Returns the articles and a report {linked, created, unlinked, dropped}.
|
||||
"""
|
||||
articles = [dict(a, blocks=list(a.get("blocks") or []), pages=list(a.get("pages") or [])) for a in articles]
|
||||
real = [a for a in articles if not _is_placeholder_article(a)]
|
||||
owner = {b: a for a in real for b in a["blocks"]}
|
||||
blocks = {b["id"]: b for b in layout.get("blocks", [])}
|
||||
by_page: dict[int, list[dict[str, Any]]] = {}
|
||||
for b in layout.get("blocks", []):
|
||||
if b["type"] in NARRATABLE and (b.get("text") or "").strip():
|
||||
by_page.setdefault(b["page"], []).append(b)
|
||||
report = {"linked": 0, "created": 0, "unlinked": [], "dropped": 0}
|
||||
taken = {int(m.group(1)) for a in articles if (m := re.fullmatch(r"article-(\d+)", a["id"]))}
|
||||
moved: set[str] = set()
|
||||
running: dict[str, Any] | None = None
|
||||
running_page = 0
|
||||
running_tail = ""
|
||||
|
||||
def next_id() -> str:
|
||||
n = max(taken, default=0) + 1
|
||||
taken.add(n)
|
||||
return f"article-{n:03d}"
|
||||
|
||||
for page in sorted(by_page):
|
||||
ordered = sorted(by_page[page], key=lambda b: b.get("reading_order", 0))
|
||||
i = 0
|
||||
while i < len(ordered):
|
||||
block = ordered[i]
|
||||
if block["id"] in owner:
|
||||
running, running_page, running_tail = owner[block["id"]], page, block["text"]
|
||||
i += 1
|
||||
continue
|
||||
j = i # collect this run of unowned text
|
||||
while j < len(ordered) and ordered[j]["id"] not in owner:
|
||||
j += 1
|
||||
segment = ordered[i:j]
|
||||
i = j
|
||||
cut = [0]
|
||||
for k, b in enumerate(segment): # a headline + byline/deck opens a new piece
|
||||
nxt = [x["type"] for x in segment[k + 1:k + 3]]
|
||||
if k and b["type"] in (BLOCK_TITLE, BLOCK_HEADING) and (BLOCK_BYLINE in nxt or BLOCK_DECK in nxt):
|
||||
cut.append(k)
|
||||
for start, end in zip(cut, cut[1:] + [len(segment)]):
|
||||
part = segment[start:end]
|
||||
head = part[0]
|
||||
opens = head["type"] in (BLOCK_TITLE, BLOCK_HEADING) and any(
|
||||
x["type"] in (BLOCK_BYLINE, BLOCK_DECK) for x in part[1:3])
|
||||
if opens:
|
||||
byline = next((x for x in part[1:4] if x["type"] == BLOCK_BYLINE), None)
|
||||
fresh = Article(id=next_id(), title=" ".join(head["text"].split())[:300],
|
||||
author=re.sub(r"^(von|by)\s+", "", (byline or {}).get("text", ""), flags=re.I).strip()[:120],
|
||||
pages=sorted({x["page"] for x in part}), blocks=[x["id"] for x in part],
|
||||
confidence=0.55).to_json()
|
||||
articles.append(fresh)
|
||||
real.append(fresh)
|
||||
owner.update({b: fresh for b in fresh["blocks"]})
|
||||
moved.update(fresh["blocks"])
|
||||
running, running_page, running_tail = fresh, page, part[-1]["text"]
|
||||
report["created"] += 1
|
||||
elif (running is not None and page - running_page <= max_gap_pages
|
||||
and not _ARTICLE_END_RE.search(running_tail.strip())):
|
||||
running["blocks"].extend(x["id"] for x in part)
|
||||
running["pages"] = sorted(set(running["pages"]) | {x["page"] for x in part})
|
||||
if running.get("status") == "approved":
|
||||
running["status"] = "detected"
|
||||
owner.update({x["id"]: running for x in part})
|
||||
moved.update(x["id"] for x in part)
|
||||
running_page, running_tail = page, part[-1]["text"]
|
||||
report["linked"] += len(part)
|
||||
else:
|
||||
report["unlinked"].append(part[0]["id"])
|
||||
absorbed_by_page: dict[int, dict[str, Any]] = {}
|
||||
for article in real:
|
||||
for b in article["blocks"]:
|
||||
if b in moved and b in blocks:
|
||||
absorbed_by_page[blocks[b]["page"]] = article
|
||||
kept = []
|
||||
for a in articles:
|
||||
if _is_placeholder_article(a):
|
||||
# text that moved, or that a real article owns anyway, no longer needs a stand-in
|
||||
a["blocks"] = [b for b in a["blocks"] if b not in moved and b not in owner]
|
||||
still_text = any(blocks.get(b, {}).get("type") in NARRATABLE for b in a["blocks"])
|
||||
if not still_text:
|
||||
target = next((absorbed_by_page[p] for p in a.get("pages", []) if p in absorbed_by_page), None)
|
||||
if target is not None and a.get("images"): # its pictures follow the text
|
||||
target["images"] = list(dict.fromkeys([*target.get("images", []), *a["images"]]))
|
||||
report["dropped"] += 1
|
||||
continue
|
||||
kept.append(a)
|
||||
return kept, report
|
||||
|
||||
|
||||
_LINK_BOUNDARY_TYPES = (BLOCK_TITLE, BLOCK_HEADING, BLOCK_DECK, BLOCK_BYLINE)
|
||||
|
||||
|
||||
def blocks_to_link(layout: dict[str, Any], article: dict[str, Any], page: int, where: str) -> list[str]:
|
||||
"""Regions on `page` to add so the article continues there (`after`) or starts there (`before`).
|
||||
|
||||
after: from the first body region that carries the article on (the sentence runs on, or
|
||||
simply the page's first body region) down to, but not including, the next headline, deck or
|
||||
byline -- that is a different piece. before: from the page's last headline (the article's
|
||||
opening) to the end, or all its body text when it has none.
|
||||
"""
|
||||
own = set(article.get("blocks") or [])
|
||||
on_page = sorted((b for b in layout.get("blocks", [])
|
||||
if b["page"] == page and b["type"] in NARRATABLE and (b.get("text") or "").strip()
|
||||
and b["id"] not in own), key=lambda b: b.get("reading_order", 0))
|
||||
if where == "before":
|
||||
starts = [i for i, b in enumerate(on_page) if b["type"] in _LINK_BOUNDARY_TYPES]
|
||||
chosen = on_page[starts[-1]:] if starts else [b for b in on_page if b["type"] == BLOCK_BODY]
|
||||
return [b["id"] for b in chosen]
|
||||
narrated = [b for b in article_text(layout, article) if b.get("type") in NARRATABLE]
|
||||
tail = (narrated[-1]["text"] if narrated else "")
|
||||
first = next((i for i, b in enumerate(on_page) if b["type"] == BLOCK_BODY), None)
|
||||
if first is None:
|
||||
return []
|
||||
# The paragraph that carries the article on: one starting lower-case is the surest sign of a
|
||||
# sentence running over; failing that, one the fusion rules accept; failing that, the first.
|
||||
bodies = [(i, normalize_ocr_text(strip_continuation_markers(b["text"])))
|
||||
for i, b in enumerate(on_page) if b["type"] == BLOCK_BODY]
|
||||
carry = next((i for i, t in bodies if t[:1].islower()), None)
|
||||
if carry is None:
|
||||
carry = next((i for i, t in bodies if paragraph_continues(tail, t)), None)
|
||||
first = first if carry is None else carry
|
||||
chosen = []
|
||||
for b in on_page[first:]:
|
||||
if b["type"] in _LINK_BOUNDARY_TYPES:
|
||||
break
|
||||
if b["type"] in (BLOCK_BODY, BLOCK_QUOTE):
|
||||
chosen.append(b["id"])
|
||||
return chosen
|
||||
|
||||
|
||||
# ── Standalone audio (Stage E, optional) ─────────────────────────────────────
|
||||
|
||||
@ -8,7 +8,10 @@ under the guards in ``accept_agent_fix``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from core import pronunciations as _pron
|
||||
@ -30,6 +33,16 @@ _MAX_ISSUES_PER_TEXT = 60
|
||||
_RECURRING = 3
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def known_names() -> frozenset[str]:
|
||||
"""Company, product and technology names the dictionaries cannot know (core/known_names.json)."""
|
||||
try:
|
||||
data = json.loads((Path(__file__).with_name("known_names.json")).read_text("utf-8"))
|
||||
return frozenset(str(n).casefold() for n in data.get("names", []))
|
||||
except (OSError, ValueError):
|
||||
return frozenset()
|
||||
|
||||
|
||||
def _checker(language: str):
|
||||
return _pron._german_spellchecker() if language == "de" else _pron._english_spellchecker()
|
||||
|
||||
@ -184,7 +197,8 @@ def _close_line_breaks(text: str) -> tuple[str, list[int]]:
|
||||
|
||||
|
||||
def find_spelling_issues(text: str, language: str = "auto",
|
||||
ignore: Iterable[str] = (), other_language: bool = True) -> list[dict]:
|
||||
ignore: Iterable[str] = (), other_language: bool = True,
|
||||
vocabulary: Iterable[str] = ()) -> list[dict]:
|
||||
"""Dictionary-unknown words in `text`, with position and suggestions.
|
||||
|
||||
A word is only a finding when it is unknown in the text's language, is not
|
||||
@ -198,7 +212,8 @@ def find_spelling_issues(text: str, language: str = "auto",
|
||||
if checker is None:
|
||||
return []
|
||||
other = _checker("en" if lang == "de" else "de") if other_language else None
|
||||
ignored = {str(w).casefold() for w in ignore}
|
||||
ignored = {str(w).casefold() for w in ignore} | known_names()
|
||||
vocab = {str(w).casefold() for w in vocabulary}
|
||||
issues: list[dict] = []
|
||||
raw = text or ""
|
||||
text, origin = _close_line_breaks(raw)
|
||||
@ -218,6 +233,9 @@ def find_spelling_issues(text: str, language: str = "auto",
|
||||
continue
|
||||
if lang != "de" and part[:1].isupper() and not _starts_sentence(text, match.start()):
|
||||
continue # mid-sentence capital: a name or product
|
||||
if part.casefold() in vocab and not (
|
||||
lang == "de" and any(v in checker for v in _umlaut_variants(part.casefold()))):
|
||||
continue # printed repeatedly: a name or term, not OCR noise
|
||||
if _known_form(part, checker, lang) or (other is not None and _known_form(part, other, "en" if lang == "de" else "de")):
|
||||
continue
|
||||
if lang == "de" and is_known_compound(part, checker):
|
||||
@ -233,12 +251,16 @@ def find_spelling_issues(text: str, language: str = "auto",
|
||||
"suggestions": _suggestions(part, checker, lang),
|
||||
})
|
||||
if len(issues) >= _MAX_ISSUES_PER_TEXT:
|
||||
return issues
|
||||
return issues
|
||||
break
|
||||
# Two unknown capitalised words side by side (``Juliane Jarke``) are a person's name.
|
||||
starts = {i["start"] for i in issues if i["token"][:1].isupper()}
|
||||
ends = {i["end"] + 1 for i in issues if i["token"][:1].isupper()}
|
||||
return [i for i in issues
|
||||
if not (i["token"][:1].isupper() and (i["start"] in ends or i["end"] + 1 in starts))]
|
||||
|
||||
|
||||
def block_issues(blocks: list[dict], article_language: str | None = None,
|
||||
ignore: Iterable[str] = ()) -> list[dict]:
|
||||
ignore: Iterable[str] = (), vocabulary: Iterable[str] = ()) -> list[dict]:
|
||||
"""Findings for a run of blocks, each with its block id, page and context."""
|
||||
ignore = set(ignore)
|
||||
language = article_language or detect_article_language(b.get("text", "") for b in blocks)
|
||||
@ -246,7 +268,7 @@ def block_issues(blocks: list[dict], article_language: str | None = None,
|
||||
for block in blocks:
|
||||
text = block.get("text") or ""
|
||||
lang = language_for_block(text, language)
|
||||
for issue in find_spelling_issues(text, lang, ignore):
|
||||
for issue in find_spelling_issues(text, lang, ignore, vocabulary=vocabulary):
|
||||
lo = max(0, issue["start"] - 60)
|
||||
hi = min(len(text), issue["end"] + 60)
|
||||
out.append(dict(issue, block_id=block["id"], page=block.get("page"),
|
||||
|
||||
@ -46,6 +46,7 @@ from core import spelling
|
||||
from core.pronunciations import (
|
||||
clean_glossary,
|
||||
detect_article_language,
|
||||
recurring_tokens,
|
||||
find_unknown_acronyms,
|
||||
parse_acronym_decisions,
|
||||
)
|
||||
@ -356,10 +357,17 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
|
||||
except Exception:
|
||||
logging.exception("Failed to refresh visual regions for %s", doc_id)
|
||||
layout, _ = mag.clean_up_regions(layout)
|
||||
background_job_update(job_id, stage="articles", detail="Fitting boxes and finding inset quotes")
|
||||
for page_meta in layout.get("pages", []):
|
||||
try:
|
||||
layout, _, _ = _fix_page_geometry(d, layout, [], page_meta["page"], _load_settings())
|
||||
except Exception: # noqa: BLE001 - never fail an import over a box fix
|
||||
logging.exception("Box fixes failed for page %s", page_meta.get("page"))
|
||||
mag.save_layout(d, layout)
|
||||
|
||||
background_job_update(job_id, stage="articles", detail="Detecting articles")
|
||||
articles = mag.build_articles(layout)
|
||||
articles, _ = mag.auto_link_pages(layout, articles)
|
||||
mag.save_articles(d, articles)
|
||||
|
||||
meta = _read_meta(d)
|
||||
@ -623,6 +631,53 @@ async def magazine_article_text(doc_id: str, article_id: str,
|
||||
return {"article": target, "blocks": blocks}
|
||||
|
||||
|
||||
@router.post("/api/magazine/{doc_id}/articles/auto-link")
|
||||
async def magazine_auto_link(doc_id: str):
|
||||
"""Link continuation pages into the articles they belong to (see mag.auto_link_pages). Undoable."""
|
||||
d, _ = _require_magazine(doc_id)
|
||||
layout = mag.load_layout(d)
|
||||
if not layout:
|
||||
raise HTTPException(404, "No layout yet -- run ingestion first")
|
||||
articles = mag.load_articles(d)
|
||||
token = _snapshot_for_undo(d, "Linked pages into articles")
|
||||
linked, report = await asyncio.to_thread(mag.auto_link_pages, layout, articles)
|
||||
mag.save_articles(d, linked)
|
||||
_seal_undo(d, token)
|
||||
return {"ok": True, "undo": token, "before": len(articles), "after": len(linked),
|
||||
"text_blocks_linked": report["linked"], "articles_created": report["created"],
|
||||
"stand_ins_removed": report["dropped"], "left_for_you": len(report["unlinked"])}
|
||||
|
||||
|
||||
@router.post("/api/magazine/{doc_id}/articles/{article_id}/link-page")
|
||||
async def magazine_link_page(doc_id: str, article_id: str, request: Request):
|
||||
"""Link one page to this article by hand. Body: {page, where: "after" | "before"}.
|
||||
|
||||
after: the article carries on there; before: the article opens there. The text that belongs
|
||||
is chosen for you (mag.blocks_to_link) and added ticked, so review shows it and it can be
|
||||
unticked. Other articles keep their own copy of those regions until you reject them.
|
||||
"""
|
||||
d, _ = _require_magazine(doc_id)
|
||||
articles, target = _article_or_404(d, article_id)
|
||||
body = await request.json()
|
||||
try:
|
||||
page = int(body["page"])
|
||||
except (KeyError, TypeError, ValueError):
|
||||
raise HTTPException(400, "page is required")
|
||||
where = "before" if isinstance(body, dict) and body.get("where") == "before" else "after"
|
||||
layout = mag.load_layout(d)
|
||||
if not any(p["page"] == page for p in layout.get("pages", [])):
|
||||
raise HTTPException(404, "Page not found")
|
||||
ids = mag.blocks_to_link(layout, target, page, where)
|
||||
if not ids:
|
||||
raise HTTPException(409, f"Nothing on page {page} could be linked -- draw or tick its regions by hand")
|
||||
token = _snapshot_for_undo(d, f"Linked page {page}")
|
||||
target["blocks"] = list(dict.fromkeys([*(target.get("blocks") or []), *ids]))
|
||||
target["pages"] = sorted(set(target.get("pages") or []) | {page})
|
||||
mag.save_articles(d, articles)
|
||||
_seal_undo(d, token)
|
||||
return {"ok": True, "undo": token, "added": ids, "page": page}
|
||||
|
||||
|
||||
@router.get("/api/magazine/{doc_id}/articles/{article_id}/flow")
|
||||
async def magazine_article_flow(doc_id: str, article_id: str):
|
||||
"""How the article's text continues between regions (below / next column / next page)."""
|
||||
@ -1094,14 +1149,18 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non
|
||||
"suspect_blocks": [], "overlaps": [], "clipped": [], "measured": False, "error": str(exc)[:200]}
|
||||
row.update(added=added, checked=checked, corrected=corrected,
|
||||
spelling=spell_left, spelling_fixed=spell_fixed)
|
||||
row["needs_review"] = bool(row["uncovered"] or row["suspect_blocks"] or row["overlaps"]
|
||||
or row["clipped"] or row.get("error") or spell_left)
|
||||
# Two kinds of "look at this": boxes that are wrong (someone must decide) and words
|
||||
# that may be wrong (a long tail, mostly names). Only the first sends you to a page.
|
||||
row["needs_review"] = bool(row["uncovered"] or row["overlaps"] or row["clipped"] or row.get("error"))
|
||||
row["words_to_check"] = bool(row["suspect_blocks"] or spell_left)
|
||||
rows.append(row)
|
||||
if i % 5 == 0:
|
||||
mag.save_layout(d, layout)
|
||||
mag.save_layout(d, layout)
|
||||
if articles_changed:
|
||||
mag.save_articles(d, articles)
|
||||
step_text = "linking pages into articles"
|
||||
background_job_update(job_id, stage="quality-run", detail=f"Checking page {len(pages)} of {len(pages)}: {step_text}")
|
||||
articles, link_report = mag.auto_link_pages(layout, articles)
|
||||
mag.save_articles(d, articles)
|
||||
duplicates = mag.duplicate_article_groups(articles)
|
||||
by_id = {a["id"]: a for a in articles}
|
||||
for row in rows:
|
||||
@ -1114,12 +1173,16 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non
|
||||
"pages": len(rows),
|
||||
"verified_with_vision": verify,
|
||||
"boxes_fixed": fixed_boxes,
|
||||
"pages_linked": link_report["linked"],
|
||||
"articles_created": link_report["created"],
|
||||
"standins_removed": link_report["dropped"],
|
||||
"regions_added": sum(r["added"] for r in rows),
|
||||
"regions_corrected": sum(r["corrected"] for r in rows),
|
||||
"spelling_fixed": sum(r.get("spelling_fixed", 0) for r in rows),
|
||||
"spelling_marked": sum(len(r.get("spelling") or []) for r in rows),
|
||||
"spelling_agent": agent,
|
||||
"pages_needing_review": [r["page"] for r in rows if r["needs_review"]],
|
||||
"pages_with_words": [r["page"] for r in rows if r.get("words_to_check") and not r["needs_review"]],
|
||||
"duplicate_article_groups": len(duplicates),
|
||||
}
|
||||
(d / _QUALITY_REPORT).write_text(
|
||||
@ -1127,7 +1190,8 @@ def _quality_run_sync(doc_id: str, job_id: str, settings: dict[str, Any]) -> Non
|
||||
flagged = len(summary["pages_needing_review"])
|
||||
background_job_update(
|
||||
job_id, stage="done", done=True, result=summary,
|
||||
detail=(f"{summary['boxes_fixed']} boxes fixed, {summary['regions_added']} missed regions added, "
|
||||
detail=(f"{summary['boxes_fixed']} boxes fixed, {summary['pages_linked']} text blocks linked into articles, "
|
||||
f"{summary['regions_added']} missed regions added, "
|
||||
f"{summary['regions_corrected']} words corrected, "
|
||||
f"{summary['spelling_fixed']} misspellings fixed, {summary['spelling_marked']} marked; "
|
||||
f"{flagged} page{'s' if flagged != 1 else ''} still need a look"
|
||||
@ -2129,6 +2193,29 @@ def _spelling_blocks(d: Path, layout: dict[str, Any], article_id: str | None = N
|
||||
return sorted(blocks, key=lambda b: (b["page"], b.get("reading_order", 0)))
|
||||
|
||||
|
||||
_VOCAB_CACHE: dict[str, tuple[tuple[float, int], frozenset[str]]] = {}
|
||||
|
||||
|
||||
def _issue_vocabulary(d: Path, layout: dict[str, Any]) -> frozenset[str]:
|
||||
"""Words printed at least twice anywhere in the issue: names, brands and terms of its subject.
|
||||
|
||||
OCR mistakes differ every time they happen; a word that keeps coming back is real. Cached
|
||||
until the layout file changes.
|
||||
"""
|
||||
try:
|
||||
stat = (d / "layout.json").stat()
|
||||
key = (stat.st_mtime, stat.st_size)
|
||||
except OSError:
|
||||
key = (0.0, 0)
|
||||
cached = _VOCAB_CACHE.get(str(d))
|
||||
if cached and cached[0] == key:
|
||||
return cached[1]
|
||||
vocab = frozenset(recurring_tokens((b.get("text") or "" for b in layout.get("blocks", [])
|
||||
if b.get("type") in _SPELL_TYPES), 2))
|
||||
_VOCAB_CACHE[str(d)] = (key, vocab)
|
||||
return vocab
|
||||
|
||||
|
||||
def _issue_language(d: Path, layout: dict[str, Any]) -> str:
|
||||
"""One language for the whole issue: a single page is too little text to decide."""
|
||||
return detect_article_language(b["text"] for b in _spelling_blocks(d, layout))
|
||||
@ -2139,7 +2226,7 @@ def _spelling_findings(d: Path, layout: dict[str, Any], article_id: str | None =
|
||||
blocks = _spelling_blocks(d, layout, article_id, page)
|
||||
if language is None:
|
||||
language = _issue_language(d, layout) if page is not None and not article_id else None
|
||||
return spelling.block_issues(blocks, language, _load_dictionary(d))
|
||||
return spelling.block_issues(blocks, language, _load_dictionary(d), _issue_vocabulary(d, layout))
|
||||
|
||||
|
||||
def _agent_check_block(settings: dict[str, Any], language: str, text: str,
|
||||
@ -2178,7 +2265,7 @@ def spelling_agent_pass(d: Path, layout: dict[str, Any], settings: dict[str, Any
|
||||
language = language or (_issue_language(d, layout) if page is not None and not article_id
|
||||
else detect_article_language(b["text"] for b in blocks))
|
||||
by_block: dict[str, list[dict[str, Any]]] = {}
|
||||
for issue in spelling.block_issues(blocks, language, dictionary):
|
||||
for issue in spelling.block_issues(blocks, language, dictionary, _issue_vocabulary(d, layout)):
|
||||
by_block.setdefault(issue["block_id"], []).append(issue)
|
||||
fixed = kept = failed = 0
|
||||
texts = {b["id"]: b["text"] for b in blocks}
|
||||
|
||||
@ -3044,32 +3044,43 @@ async function magRunFigureRefine(page) {
|
||||
function magRenderQualityReport(report) {
|
||||
const box = $('mag-quality-report');
|
||||
if (!box) return;
|
||||
const flagged = (report.pages || []).filter((p) => p.needs_review);
|
||||
if (!flagged.length) {
|
||||
box.hidden = false;
|
||||
box.textContent = 'Quality check: every page is fully covered and no suspicious words remain.';
|
||||
return;
|
||||
}
|
||||
const sm = report.summary || {};
|
||||
const pages = report.pages || [];
|
||||
const boxes = pages.filter((p) => p.needs_review);
|
||||
const words = pages.filter((p) => !p.needs_review && p.words_to_check);
|
||||
const did = [
|
||||
[sm.pages_linked, 'text blocks linked into articles'], [sm.boxes_fixed, 'boxes fixed'],
|
||||
[sm.regions_added, 'missed regions added'], [sm.regions_corrected, 'words corrected'],
|
||||
[sm.spelling_fixed, 'misspellings fixed']].filter(([n]) => n).map(([n, w]) => n + ' ' + w);
|
||||
const tip = (p) => {
|
||||
const bits = [];
|
||||
if (p.uncovered.length) bits.push(p.uncovered.length + ' text area' + (p.uncovered.length === 1 ? '' : 's') + ' outside any box');
|
||||
if (p.overlaps.length) bits.push(p.overlaps.length + ' overlapping box pair' + (p.overlaps.length === 1 ? '' : 's'));
|
||||
if (p.clipped.length) bits.push(p.clipped.length + ' box edge' + (p.clipped.length === 1 ? '' : 's') + ' cutting text');
|
||||
if ((p.duplicates || []).length) bits.push('duplicate articles');
|
||||
if (p.error) bits.push('check failed');
|
||||
if (p.words_to_check) bits.push('words to check');
|
||||
return bits.join(', ');
|
||||
};
|
||||
const pill = (p) => '<button type="button" class="btn-sm mag-page-pill" onclick="magReviewPage(' + Number(p.page) + ')" title="' + esc(tip(p)) + '">' + Number(p.page) + '</button>';
|
||||
box.hidden = false;
|
||||
box.innerHTML = '<strong>' + flagged.length + ' page' + (flagged.length === 1 ? '' : 's') +
|
||||
' still need a look:</strong> ' + flagged.map((p) => {
|
||||
const bits = [];
|
||||
if (p.uncovered.length) bits.push(p.uncovered.length + ' uncovered area' + (p.uncovered.length === 1 ? '' : 's'));
|
||||
if (p.suspect_blocks.length) bits.push(p.suspect_blocks.length + ' suspect text');
|
||||
if ((p.spelling || []).length) bits.push(p.spelling.length + ' spelling');
|
||||
if (p.overlaps.length) bits.push(p.overlaps.length + ' overlapping');
|
||||
if (p.clipped.length) bits.push(p.clipped.length + ' clipped');
|
||||
if ((p.duplicates || []).length) bits.push('duplicate articles');
|
||||
if (p.error) bits.push('check failed');
|
||||
return '<button type="button" class="btn-sm" onclick="magReviewPage(' + Number(p.page) + ')" title="' +
|
||||
esc(bits.join(', ')) + '">Page ' + Number(p.page) + ' (' + esc(bits.join(', ')) + ')</button>';
|
||||
}).join(' ');
|
||||
box.innerHTML =
|
||||
'<div><strong>' + (did.length ? 'Fixed automatically: ' + esc(did.join(', ')) + '.' : 'Nothing needed fixing automatically.') + '</strong></div>' +
|
||||
(boxes.length
|
||||
? '<div class="mag-report-group"><span>' + boxes.length + ' page' + (boxes.length === 1 ? '' : 's') + ' need a decision (boxes):</span> ' +
|
||||
'<button type="button" class="btn-sm primary" onclick="magReviewPage(' + Number(boxes[0].page) + ')">Start with page ' + Number(boxes[0].page) + '</button> ' +
|
||||
boxes.map(pill).join(' ') + '</div>'
|
||||
: '<div class="mag-report-group">✓ No page needs a decision about its boxes.</div>') +
|
||||
(words.length
|
||||
? '<details class="mag-report-group"><summary>' + words.length + ' more page' + (words.length === 1 ? '' : 's') +
|
||||
' have only words to check (mostly names and terms) — optional</summary>' + words.map(pill).join(' ') + '</details>'
|
||||
: '');
|
||||
}
|
||||
|
||||
async function magQualityRun() {
|
||||
if (!state.doc) return;
|
||||
const ok = await confirmDialog(
|
||||
'Fix every page automatically (fit boxes, find inset pull quotes, add missed text, re-read suspicious words, fix provable misspellings) and list what still needs a look? This can take a while on a long issue. ' +
|
||||
'Fix every page automatically (link continuation pages into their articles, fit boxes, find inset pull quotes, add missed text, re-read suspicious words, fix provable misspellings) and list only what still needs your decision? This can take a while on a long issue; a backup of the layout is kept. ' +
|
||||
'Added regions start unticked, so nothing new is read aloud until you approve it.',
|
||||
{ title: 'Auto-fix and check the whole magazine?', okLabel: 'Run' });
|
||||
if (!ok) return;
|
||||
@ -3228,7 +3239,9 @@ function magFlowPaint() {
|
||||
if (l.relation === 'page') {
|
||||
if (l.from_page === page) chip(l.from, 'bottom', 'continues on page ' + l.to_page + ' ›', 'is-page');
|
||||
if (l.to_page === page) chip(l.to, 'top', '‹ from page ' + l.from_page, 'is-page');
|
||||
} else if (l.from_page === page) {
|
||||
} else if (l.from_page === page && (l.relation === 'column' || l.sentence)) {
|
||||
// A jump to another column always gets an arrow; a step to the box below only when a
|
||||
// sentence runs across it (a plain new paragraph below is obvious and would be clutter).
|
||||
arrows.push(l);
|
||||
}
|
||||
});
|
||||
@ -3253,14 +3266,14 @@ function magFlowPaint() {
|
||||
path.setAttribute('d', l.relation === 'column'
|
||||
? 'M' + ax + ',' + ay + ' C' + ax + ',' + (ay + bend) + ' ' + bx + ',' + (by - bend) + ' ' + bx + ',' + by
|
||||
: 'M' + ax + ',' + ay + ' L' + bx + ',' + by);
|
||||
path.setAttribute('class', 'mag-flow-path');
|
||||
path.setAttribute('class', 'mag-flow-path' + (l.sentence ? '' : ' is-paragraph'));
|
||||
path.setAttribute('marker-end', 'url(#mag-flow-head)');
|
||||
svg.appendChild(path);
|
||||
const label = document.createElementNS(svgNS, 'text');
|
||||
label.setAttribute('class', 'mag-flow-label');
|
||||
label.setAttribute('x', (ax + bx) / 2 + 6);
|
||||
label.setAttribute('y', (ay + by) / 2);
|
||||
label.textContent = l.relation === 'column' ? 'next column' : '↓';
|
||||
label.textContent = l.relation === 'column' ? (l.sentence ? 'sentence continues in next column' : 'next column') : 'sentence continues';
|
||||
svg.appendChild(label);
|
||||
});
|
||||
wrap.appendChild(svg);
|
||||
@ -3287,9 +3300,6 @@ const MAG_ISSUE_KINDS = [
|
||||
{ key: 'overlaps', cls: 'is-issue-overlap', label: 'Overlaps another box',
|
||||
what: (n) => n + ' pair' + (n === 1 ? '' : 's') + ' of boxes cover the same text, so it may be read twice (purple dashed outline).',
|
||||
fix: null, hint: 'Delete or merge one of the two (Select boxes, then Delete), or run Clean up boxes.' },
|
||||
{ key: 'suspect_blocks', cls: 'is-issue-suspect', label: 'Words look mis-read',
|
||||
what: (n) => n + ' box' + (n === 1 ? ' has' : 'es have') + ' words that look like OCR mistakes (yellow dotted outline).',
|
||||
fix: 'Verify OCR text', run: "magRunPageQuality('recognition')" },
|
||||
{ key: 'spelling', cls: null, label: 'Unknown words',
|
||||
what: (n) => n + ' word' + (n === 1 ? '' : 's') + ' no German or English dictionary knows (red wavy underline in the list).',
|
||||
fix: 'Fix with language model', run: 'magSpellOpenAndFix()' },
|
||||
@ -3321,7 +3331,7 @@ function magAuditPaint() {
|
||||
const audit = state.audit && state.audit.page === state.reviewPage ? state.audit.data : null;
|
||||
document.querySelectorAll('#mag-review .mag-issue-box').forEach((el) => el.remove());
|
||||
document.querySelectorAll('#mag-review .mag-review-box').forEach((el) => {
|
||||
el.classList.remove('is-issue-clipped', 'is-issue-overlap', 'is-issue-suspect',
|
||||
el.classList.remove('is-issue-clipped', 'is-issue-overlap',
|
||||
'side-left', 'side-right', 'side-top', 'side-bottom');
|
||||
el.removeAttribute('data-issue');
|
||||
});
|
||||
@ -3342,7 +3352,6 @@ function magAuditPaint() {
|
||||
if (el) el.classList.add('side-' + c.side);
|
||||
});
|
||||
(audit.overlaps || []).forEach(([a, b]) => { tag(boxOf(a), 'is-issue-overlap', 'overlaps another box'); tag(boxOf(b), 'is-issue-overlap', 'overlaps another box'); });
|
||||
(audit.suspect_blocks || []).forEach((id) => tag(boxOf(id), 'is-issue-suspect', 'words look mis-read'));
|
||||
const meta = (state.layout?.pages || []).find((p) => p.page === state.reviewPage);
|
||||
if (wrap && meta) {
|
||||
(audit.uncovered || []).forEach((b) => {
|
||||
@ -3361,6 +3370,10 @@ function magAuditPaint() {
|
||||
'title="Fits boxes to their text, adds inset pull quotes, cuts paragraphs that wrap around them, finds missed text, re-reads suspicious words and (with a language model) fixes provable misspellings. One Undo brings the page back.">' +
|
||||
'<span class="mdi mdi-auto-fix" aria-hidden="true"></span> Fix this page automatically</button>' +
|
||||
'<span id="mag-issues-status" class="muted" role="status"></span>' +
|
||||
'<span class="mag-link-page" title="Connect this article to a page it was not linked to: the text that belongs is picked for you and added ticked">' +
|
||||
'<label>Link to page <input type="number" id="mag-link-page" min="1" max="' + Number(state.layout?.page_count || 9999) + '" style="width:5em"></label> ' +
|
||||
'<button type="button" class="btn-sm" onclick="magLinkPage(\'after\')">Article continues there</button> ' +
|
||||
'<button type="button" class="btn-sm" onclick="magLinkPage(\'before\')">Article starts there</button></span>' +
|
||||
'<label class="mag-flow-toggle" title="Green arrows show where the article\'s text carries on: below, in the next column or on another page">' +
|
||||
'<input type="checkbox" id="mag-flow-toggle" ' + (state.flowOn ? 'checked' : '') + ' onchange="magFlowToggle(this.checked)"> Show text flow</label></div>';
|
||||
if (total) {
|
||||
@ -3412,6 +3425,29 @@ async function magAutofixPage(mode) {
|
||||
renderReview();
|
||||
}
|
||||
|
||||
async function magLinkPage(where) {
|
||||
const input = $('mag-link-page');
|
||||
const page = Number(input && input.value);
|
||||
if (!page) return say('Type the page number to link first', 'err');
|
||||
try {
|
||||
const r = await api('/api/magazine/' + state.doc.id + '/articles/' + state.article.id + '/link-page', {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ page, where }),
|
||||
});
|
||||
say('Linked ' + r.added.length + ' region' + (r.added.length === 1 ? '' : 's') + ' from page ' + page + ' — review them on that page', 'ok');
|
||||
state.undo = { token: r.undo, label: 'Linked page ' + page };
|
||||
} catch (e) {
|
||||
return say(e.message, 'err');
|
||||
}
|
||||
const articleId = state.article.id;
|
||||
await magRefreshArticles();
|
||||
state.article = state.articles.find((a) => a.id === articleId) || state.article;
|
||||
_magReviewChecked = null;
|
||||
state.reviewPage = page;
|
||||
state.layout = null;
|
||||
await ensureLayout();
|
||||
renderReview();
|
||||
}
|
||||
|
||||
async function magSpellOpenAndFix() {
|
||||
if (!state.spellOpen) magSpellToggle();
|
||||
await magSpellAgent();
|
||||
@ -4946,6 +4982,7 @@ root.magDrivingToggleText = magDrivingToggleText;
|
||||
root.magDrivingToggleImages = magDrivingToggleImages;
|
||||
root.magDrivingPrevious = magDrivingPrevious;
|
||||
root.magFlowToggle = magFlowToggle;
|
||||
root.magLinkPage = magLinkPage;
|
||||
root.magAutofixPage = magAutofixPage;
|
||||
root.magSpellOpenAndFix = magSpellOpenAndFix;
|
||||
root.magSpellToggle = magSpellToggle;
|
||||
|
||||
@ -5516,3 +5516,10 @@ input[list="lang-codes"] {
|
||||
|
||||
.mag-review-page-wrap.has-room .mag-issue-box::after, .mag-review-page-wrap.has-room .mag-review-box[data-issue]::after,
|
||||
.mag-review-box[data-issue]:hover::after { display: block; }
|
||||
|
||||
.mag-flow-path.is-paragraph { stroke-dasharray: 7 5; }
|
||||
.mag-link-page { display: inline-flex; flex-wrap: wrap; align-items: center; gap: 6px; font-size: 13px; }
|
||||
.mag-report-group { margin-top: 8px; display: flex; flex-wrap: wrap; align-items: center; gap: 6px; }
|
||||
details.mag-report-group { display: block; }
|
||||
details.mag-report-group summary { cursor: pointer; margin-bottom: 6px; }
|
||||
.mag-page-pill { min-width: 2.6em; padding: 2px 8px; }
|
||||
|
||||
@ -487,12 +487,12 @@ test.describe('Magazine: what is wrong on this page', () => {
|
||||
await expect(panel).toContainText('Text outside any box');
|
||||
await expect(panel).toContainText('Edge cuts text');
|
||||
await expect(panel).toContainText('Overlaps another box');
|
||||
await expect(panel).toContainText('Words look mis-read');
|
||||
await expect(panel).not.toContainText('mis-read');
|
||||
await expect(panel).toContainText('Unknown words');
|
||||
await expect(page.locator('#mag-review-page-wrap .mag-issue-box.is-uncovered')).toHaveCount(1);
|
||||
await expect(page.locator('#mag-review-page-wrap .mag-review-box.is-issue-clipped.side-right')).toHaveCount(1);
|
||||
await expect(page.locator('#mag-review-page-wrap .mag-review-box.is-issue-overlap')).toHaveCount(2);
|
||||
await expect(page.locator('#mag-review-page-wrap .mag-review-box.is-issue-suspect')).toHaveCount(1);
|
||||
await expect(page.locator('#mag-review-page-wrap .mag-review-box.is-issue-suspect')).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('Fix this page automatically starts the job, offers Undo and reloads the page', async ({ page }) => {
|
||||
@ -568,6 +568,26 @@ test.describe('Magazine: text flow indicator', () => {
|
||||
});
|
||||
});
|
||||
|
||||
test('an article is linked to a page it was not connected to, from the review panel', async ({ page }) => {
|
||||
await mockBackend(page);
|
||||
let body = null;
|
||||
await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/link-page`, (route) => {
|
||||
body = route.request().postDataJSON();
|
||||
return route.fulfill({ json: { ok: true, undo: 'tok', added: ['b9', 'b10'], page: 4 } });
|
||||
});
|
||||
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
|
||||
await navTo(page, 's-magazine');
|
||||
await page.locator('#mag-library-list .mag-card').first().click();
|
||||
await page.locator('#mag-contents-view-toggle').click();
|
||||
await page.locator('.mag-card-wrap').filter({ hasText: 'Page 3' }).getByRole('button', { name: 'Review', exact: true }).click();
|
||||
await page.waitForSelector('#mag-issues-panel .mag-issues-head');
|
||||
await page.getByLabel('Link to page').fill('4');
|
||||
await page.getByRole('button', { name: 'Article continues there' }).click();
|
||||
await expect.poll(() => body).toEqual({ page: 4, where: 'after' });
|
||||
await expect(page.locator('#mag-review .mag-review-pagenav-group strong')).toContainText('Page 4');
|
||||
});
|
||||
|
||||
test('drawing a chart links the crop and excludes enclosed narration on approval', async ({ page }) => {
|
||||
await mockBackend(page);
|
||||
let cropped = false;
|
||||
|
||||
@ -2921,29 +2921,104 @@ class SplitAroundPullquoteTests(unittest.TestCase):
|
||||
|
||||
|
||||
class ArticleFlowTests(unittest.TestCase):
|
||||
def layout(self, rows):
|
||||
blocks = [{"id": i, "page": pg, "type": ty, "text": txt, "reading_order": n, "bbox": bb}
|
||||
for n, (i, pg, ty, txt, bb) in enumerate(rows)]
|
||||
return {"pages": [], "blocks": blocks, "images": []}, {"blocks": [r[0] for r in rows]}
|
||||
|
||||
def flow(self, rows):
|
||||
from core.magazine import article_flow
|
||||
blocks = [{"id": i, "page": pg, "type": t, "text": txt, "reading_order": n, "bbox": bb}
|
||||
for n, (i, pg, t, txt, bb) in enumerate(rows)]
|
||||
return article_flow({"pages": [], "blocks": blocks, "images": []}, {"blocks": [r[0] for r in rows]})
|
||||
return article_flow(*self.layout(rows))
|
||||
|
||||
def test_a_sentence_running_on_says_how_the_eye_travels(self):
|
||||
def test_every_step_between_regions_is_a_link_that_says_how_the_eye_travels(self):
|
||||
flow = self.flow([
|
||||
("t", 1, "title", "Titel", [10, 10, 200, 20]),
|
||||
("a", 1, "body", "Der Satz beginnt in der ersten Spalte und", [10, 40, 90, 100]),
|
||||
("b", 1, "body", "geht in der zweiten Spalte weiter, bis er", [120, 40, 90, 100]),
|
||||
("c", 2, "body", "auf der nächsten Seite endet. Danach ein neuer Absatz.", [10, 40, 90, 100])])
|
||||
self.assertEqual([(l["from"], l["to"], l["relation"]) for l in flow["links"]],
|
||||
[("a", "b", "column"), ("b", "c", "page")])
|
||||
self.assertEqual((flow["first"], flow["last"]), ("t", "c"))
|
||||
("b", 1, "body", "geht in der zweiten Spalte weiter. Fertig.", [120, 40, 90, 100]),
|
||||
("c", 1, "body", "Ein neuer Absatz darunter.", [120, 150, 90, 50]),
|
||||
("d", 2, "body", "Und ein neuer Absatz auf der nächsten Seite.", [10, 40, 90, 100])])
|
||||
self.assertEqual([(l["from"], l["to"], l["relation"], l["sentence"]) for l in flow["links"]],
|
||||
[("t", "a", "below", False), ("a", "b", "column", True),
|
||||
("b", "c", "below", False), ("c", "d", "page", False)])
|
||||
self.assertEqual((flow["first"], flow["last"]), ("t", "d"))
|
||||
|
||||
def test_a_continuation_under_the_same_column_is_below(self):
|
||||
flow = self.flow([("a", 1, "body", "Der Satz beginnt hier und", [10, 40, 90, 50]),
|
||||
("b", 1, "body", "geht direkt darunter weiter.", [10, 100, 90, 50])])
|
||||
self.assertEqual(flow["links"][0]["relation"], "below")
|
||||
def test_a_next_column_link_exists_even_when_the_sentence_ended(self):
|
||||
flow = self.flow([("a", 1, "body", "Ein fertiger Absatz.", [10, 40, 90, 100]),
|
||||
("b", 1, "body", "Der nächste Absatz in der anderen Spalte.", [120, 40, 90, 100])])
|
||||
self.assertEqual((flow["links"][0]["relation"], flow["links"][0]["sentence"]), ("column", False))
|
||||
|
||||
def test_finished_paragraphs_and_headings_have_no_link(self):
|
||||
flow = self.flow([("a", 1, "body", "Ein fertiger Absatz.", [10, 40, 90, 50]),
|
||||
("h", 1, "heading", "Zwischentitel", [10, 100, 90, 10]),
|
||||
("b", 1, "body", "Ein weiterer Absatz.", [10, 120, 90, 50])])
|
||||
self.assertEqual(flow["links"], [])
|
||||
|
||||
class BlocksToLinkTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
def blk(i, pg, ty, txt, n):
|
||||
return {"id": i, "page": pg, "type": ty, "text": txt, "reading_order": n, "bbox": [0, 0, 1, 1]}
|
||||
self.layout = {"pages": [], "images": [], "blocks": [
|
||||
blk("a", 1, "body", "Der Text beginnt hier und", 0),
|
||||
blk("p1", 2, "body", "Ein anderer, fertiger Absatz.", 0),
|
||||
blk("p2", 2, "body", "geht auf dieser Seite weiter und endet.", 1),
|
||||
blk("p3", 2, "body", "Noch ein Absatz desselben Beitrags.", 2),
|
||||
blk("h", 2, "heading", "Ein neuer Beitrag", 3),
|
||||
blk("p4", 2, "body", "Gehört zum neuen Beitrag.", 4)]}
|
||||
self.article = {"blocks": ["a"]}
|
||||
|
||||
def test_linking_forward_starts_at_the_paragraph_that_carries_on_and_stops_at_a_new_headline(self):
|
||||
from core.magazine import blocks_to_link
|
||||
self.assertEqual(blocks_to_link(self.layout, self.article, 2, "after"), ["p2", "p3"])
|
||||
|
||||
def test_without_a_carrying_paragraph_it_starts_at_the_first_body_region(self):
|
||||
from core.magazine import blocks_to_link
|
||||
self.layout["blocks"][0]["text"] = "Der Text ist fertig."
|
||||
self.layout["blocks"][2]["text"] = "Geht auf dieser Seite weiter und endet."
|
||||
self.assertEqual(blocks_to_link(self.layout, self.article, 2, "after"), ["p1", "p2", "p3"])
|
||||
|
||||
def test_linking_backward_takes_the_last_opening_of_that_page(self):
|
||||
from core.magazine import blocks_to_link
|
||||
self.assertEqual(blocks_to_link(self.layout, {"blocks": []}, 2, "before"), ["h", "p4"])
|
||||
|
||||
def test_regions_already_in_the_article_are_not_offered_again(self):
|
||||
from core.magazine import blocks_to_link
|
||||
self.assertNotIn("p2", blocks_to_link(self.layout, {"blocks": ["a", "p2"]}, 2, "after"))
|
||||
|
||||
|
||||
class AutoLinkPagesTests(unittest.TestCase):
|
||||
def blk(self, i, page, ty, text, n=0):
|
||||
return {"id": i, "page": page, "type": ty, "text": text, "reading_order": n, "bbox": [0, 0, 1, 1]}
|
||||
|
||||
def art(self, i, title, pages, blocks, status="detected"):
|
||||
return {"id": i, "title": title, "author": "", "pages": pages, "blocks": blocks, "images": [], "status": status,
|
||||
"confidence": 0.5, "section": "", "continued_to": [], "audio_status": "none"}
|
||||
|
||||
def link(self, blocks, articles):
|
||||
from core.magazine import auto_link_pages
|
||||
return auto_link_pages({"pages": [], "images": [], "blocks": blocks}, articles)
|
||||
|
||||
def test_pages_without_a_headline_continue_the_article_before_and_the_stand_ins_go(self):
|
||||
blocks = [self.blk("t", 1, "title", "Der Butler", 0), self.blk("a", 1, "body", "Der Text beginnt und", 1),
|
||||
self.blk("b", 2, "body", "geht auf Seite zwei weiter.", 0), self.blk("c", 3, "body", "Und endet auf Seite drei. (hag)", 0)]
|
||||
arts = [self.art("article-001", "Der Butler", [1], ["t", "a"], "approved"),
|
||||
self.art("article-002", "Page 2", [2], ["b"], "approved"), self.art("article-003", "Page 3", [3], ["c"], "approved")]
|
||||
out, report = self.link(blocks, arts)
|
||||
self.assertEqual([(a["id"], a["pages"], a["blocks"]) for a in out], [("article-001", [1, 2, 3], ["t", "a", "b", "c"])])
|
||||
self.assertEqual(out[0]["status"], "detected") # gained unreviewed text: back to review
|
||||
self.assertEqual((report["linked"], report["dropped"]), (2, 2))
|
||||
|
||||
def test_a_signed_end_stops_the_article_and_leaves_the_next_page_alone(self):
|
||||
blocks = [self.blk("t", 1, "title", "Kurz", 0), self.blk("a", 1, "body", "Fertig. (atr)", 1),
|
||||
self.blk("b", 2, "body", "Ein ganz anderer Text.", 0)]
|
||||
out, report = self.link(blocks, [self.art("article-001", "Kurz", [1], ["t", "a"]), self.art("article-002", "Page 2", [2], ["b"])])
|
||||
self.assertEqual(len(out), 2)
|
||||
self.assertEqual(report["unlinked"], ["b"])
|
||||
|
||||
def test_a_headline_with_a_byline_opens_a_new_article(self):
|
||||
blocks = [self.blk("t", 1, "title", "Alt", 0), self.blk("a", 1, "body", "Text geht weiter und", 1),
|
||||
self.blk("n", 2, "title", "Neu", 0), self.blk("by", 2, "byline", "Von Anna Beispiel", 1),
|
||||
self.blk("x", 2, "body", "Der neue Beitrag beginnt.", 2)]
|
||||
out, report = self.link(blocks, [self.art("article-001", "Alt", [1], ["t", "a"]), self.art("article-002", "Page 2", [2], ["n", "by", "x"])])
|
||||
new = next(a for a in out if a["title"] == "Neu")
|
||||
self.assertEqual((new["author"], new["blocks"], report["created"]), ("Anna Beispiel", ["n", "by", "x"], 1))
|
||||
|
||||
def test_too_far_from_the_running_article_stays_unlinked(self):
|
||||
blocks = [self.blk("t", 1, "title", "Alt", 0), self.blk("a", 1, "body", "Text geht weiter und", 1),
|
||||
self.blk("z", 6, "body", "weit weg", 0)]
|
||||
out, report = self.link(blocks, [self.art("article-001", "Alt", [1], ["t", "a"]), self.art("article-002", "Page 6", [6], ["z"])])
|
||||
self.assertEqual(report["unlinked"], ["z"])
|
||||
|
||||
@ -1593,7 +1593,7 @@ class SpellingRouteTests(unittest.TestCase):
|
||||
self.dir = Path(tempfile.mkdtemp(prefix="ttvc-spelling-"))
|
||||
mag.save_layout(self.dir, {"pages": [{"page": 1}], "images": [], "blocks": [
|
||||
{"id": "b1", "page": 1, "type": "body", "reading_order": 0, "bbox": [0, 0, 1, 1],
|
||||
"text": "Der Tesst ist gut und die Datenbank wächst nicht. Nvidia hilft."},
|
||||
"text": "Der Tesst ist gut und die Datenbank wächst nicht. Zorbex hilft."},
|
||||
{"id": "ad", "page": 1, "type": "ad", "reading_order": 1, "bbox": [0, 0, 1, 1], "text": "Kaufen Sie Blorp."},
|
||||
]})
|
||||
mag.save_articles(self.dir, [{"id": "a1", "blocks": ["b1"], "images": []}])
|
||||
@ -1624,7 +1624,7 @@ class SpellingRouteTests(unittest.TestCase):
|
||||
def test_findings_are_scoped_to_narratable_text_and_marked_with_suggestions(self):
|
||||
from routes.magazine import magazine_spelling
|
||||
out = self.call(magazine_spelling, page=1)
|
||||
self.assertEqual([i["token"] for i in out["issues"]], ["Tesst", "Nvidia"])
|
||||
self.assertEqual([i["token"] for i in out["issues"]], ["Tesst", "Zorbex"])
|
||||
self.assertEqual(out["issues"][0]["block_id"], "b1")
|
||||
self.assertIn("Test", out["issues"][0]["suggestions"])
|
||||
|
||||
@ -1651,9 +1651,9 @@ class SpellingRouteTests(unittest.TestCase):
|
||||
|
||||
def test_dictionary_entries_silence_a_finding_and_can_be_removed(self):
|
||||
from routes.magazine import magazine_spelling, magazine_spelling_dictionary
|
||||
self.call(magazine_spelling_dictionary, body={"word": "Nvidia"})
|
||||
self.call(magazine_spelling_dictionary, body={"word": "Zorbex"})
|
||||
self.assertEqual([i["token"] for i in self.call(magazine_spelling)["issues"]], ["Tesst"])
|
||||
self.call(magazine_spelling_dictionary, body={"word": "Nvidia", "remove": True})
|
||||
self.call(magazine_spelling_dictionary, body={"word": "Zorbex", "remove": True})
|
||||
self.assertEqual(len(self.call(magazine_spelling)["issues"]), 2)
|
||||
with self.assertRaises(HTTPException):
|
||||
self.call(magazine_spelling_dictionary, body={"word": "two words"})
|
||||
@ -1664,11 +1664,11 @@ class SpellingRouteTests(unittest.TestCase):
|
||||
|
||||
def check(settings, language, text, issues):
|
||||
return [{"token": "Tesst", "action": "replace", "replacement": "Test"},
|
||||
{"token": "Nvidia", "action": "keep"}]
|
||||
{"token": "Zorbex", "action": "keep"}]
|
||||
result = spelling_agent_pass(self.dir, mag.load_layout(self.dir), {}, check=check)
|
||||
self.assertEqual((result["fixed"], result["kept"], result["remaining"]), (1, 1, 0))
|
||||
self.assertIn("Der Test ist gut", result["layout"]["blocks"][0]["text"])
|
||||
self.assertEqual(_load_dictionary(self.dir), {"nvidia": "agent"})
|
||||
self.assertEqual(_load_dictionary(self.dir), {"zorbex": "agent"})
|
||||
|
||||
def test_an_agent_that_rewrites_or_fails_changes_nothing_and_findings_stay_marked(self):
|
||||
from core import magazine as mag
|
||||
@ -1755,3 +1755,53 @@ class PageAuditAndAutofixTests(unittest.TestCase):
|
||||
with patch('routes.magazine._LIB', self.dir.parent), patch('routes.magazine.background_job_update') as job:
|
||||
_autofix_page_sync(self.dir.name, "j", 1, "all", {})
|
||||
self.assertTrue(job.call_args.kwargs["done"] and job.call_args.kwargs["error"])
|
||||
|
||||
|
||||
class ArticleLinkingRouteTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
from core import magazine as mag
|
||||
self.dir = Path(tempfile.mkdtemp(prefix="ttvc-link-"))
|
||||
blk = lambda i, pg, ty, t, n: {"id": i, "page": pg, "type": ty, "text": t, "reading_order": n, "bbox": [0, 0, 1, 1]}
|
||||
mag.save_layout(self.dir, {"pages": [{"page": 1}, {"page": 2}, {"page": 3}], "images": [], "blocks": [
|
||||
blk("t", 1, "title", "Der Butler", 0), blk("a", 1, "body", "Der Text beginnt und", 1),
|
||||
blk("b", 2, "body", "geht weiter und endet. (hag)", 0), blk("z", 3, "body", "Etwas ganz anderes.", 0)]})
|
||||
art = lambda i, ti, pg, bl: {"id": i, "title": ti, "author": "", "pages": pg, "blocks": bl, "images": [], "status": "approved",
|
||||
"confidence": 0.5, "section": "", "continued_to": [], "audio_status": "none"}
|
||||
mag.save_articles(self.dir, [art("article-001", "Der Butler", [1], ["t", "a"]), art("article-002", "Page 2", [2], ["b"])])
|
||||
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.dir, ignore_errors=True)
|
||||
|
||||
def call(self, fn, *args, body=None):
|
||||
import asyncio
|
||||
from unittest.mock import AsyncMock, patch
|
||||
request = AsyncMock(); request.json.return_value = body if body is not None else {}
|
||||
with patch('routes.magazine._require_magazine', return_value=(self.dir, {})), \
|
||||
patch('routes.magazine.asyncio.to_thread', new=_run_sync):
|
||||
return asyncio.run(fn('a' * 32, *args, *([request] if body is not None else [])))
|
||||
|
||||
def test_auto_link_merges_the_stand_in_page_and_can_be_undone(self):
|
||||
from core import magazine as mag
|
||||
from routes.magazine import magazine_auto_link, magazine_undo
|
||||
out = self.call(magazine_auto_link)
|
||||
self.assertEqual((out["before"], out["after"], out["text_blocks_linked"]), (2, 1, 1))
|
||||
self.assertEqual(mag.load_articles(self.dir)[0]["pages"], [1, 2])
|
||||
self.call(magazine_undo, body={"token": out["undo"]})
|
||||
self.assertEqual(len(mag.load_articles(self.dir)), 2)
|
||||
|
||||
def test_linking_a_page_by_hand_adds_the_right_text_ticked_and_is_undoable(self):
|
||||
from core import magazine as mag
|
||||
from routes.magazine import magazine_link_page
|
||||
out = self.call(magazine_link_page, "article-001", body={"page": 2, "where": "after"})
|
||||
self.assertEqual(out["added"], ["b"])
|
||||
article = mag.load_articles(self.dir)[0]
|
||||
self.assertEqual((article["pages"], article["blocks"]), ([1, 2], ["t", "a", "b"]))
|
||||
|
||||
def test_linking_needs_a_real_page_with_something_to_link(self):
|
||||
from routes.magazine import magazine_link_page
|
||||
with self.assertRaises(HTTPException) as ctx:
|
||||
self.call(magazine_link_page, "article-001", body={"page": 99})
|
||||
self.assertEqual(ctx.exception.status_code, 404)
|
||||
with self.assertRaises(HTTPException) as ctx:
|
||||
self.call(magazine_link_page, "article-001", body={})
|
||||
self.assertEqual(ctx.exception.status_code, 400)
|
||||
|
||||
@ -5,7 +5,7 @@ from unittest import mock
|
||||
from core import spelling
|
||||
|
||||
DE = {"die", "software", "entwicklung", "braucht", "grafik", "karte", "jährlich", "modell", "ist", "gut",
|
||||
"das", "und", "test", "tests", "der", "hilft", "nicht", "datenbank", "daten", "bank", "wächst", "jahrlich", "sowie", "laufen", "modell", "sind"}
|
||||
"das", "und", "test", "tests", "der", "hilft", "nicht", "datenbank", "daten", "bank", "wächst", "jahrlich", "sowie", "laufen", "modell", "sind", "sagt", "heute"}
|
||||
EN = {"the", "software", "engineers", "use", "graphics", "card", "message", "receive", "this", "database", "then", "and", "help"}
|
||||
|
||||
|
||||
@ -109,6 +109,29 @@ class LineBreakAndSuspendedCompoundTests(unittest.TestCase):
|
||||
self.assertFalse(spelling.accept_agent_fix(issue, "Datenbank- und mehr"))
|
||||
|
||||
|
||||
class NamesBrandsAndVocabularyTests(unittest.TestCase):
|
||||
@patched
|
||||
def test_built_in_company_and_product_names_are_never_findings(self):
|
||||
self.assertEqual(spelling.find_spelling_issues("Die Nvidia und Anthropic sind gut.", "de"), [])
|
||||
|
||||
@patched
|
||||
def test_two_unknown_capitalised_words_in_a_row_are_a_persons_name(self):
|
||||
self.assertEqual(spelling.find_spelling_issues("Das sagt Juliane Jarke heute.", "de"), [])
|
||||
self.assertEqual([i["token"] for i in spelling.find_spelling_issues("Das sagt Jarke heute.", "de")], ["Jarke"])
|
||||
|
||||
@patched
|
||||
def test_a_word_printed_repeatedly_in_the_issue_is_vocabulary(self):
|
||||
text = "Die Zorbex ist gut."
|
||||
self.assertEqual([i["token"] for i in spelling.find_spelling_issues(text, "de")], ["Zorbex"])
|
||||
self.assertEqual(spelling.find_spelling_issues(text, "de", vocabulary={"zorbex"}), [])
|
||||
|
||||
@patched
|
||||
def test_a_missing_umlaut_stays_flagged_even_though_it_recurs(self):
|
||||
# 'jahrlich' is in the fake German dictionary as its umlaut form 'jährlich'
|
||||
found = spelling.find_spelling_issues("Das Modell ist gut und wachst nicht.", "de", vocabulary={"wachst"})
|
||||
self.assertEqual([i["token"] for i in found], ["wachst"])
|
||||
|
||||
|
||||
class AgentGuardTests(unittest.TestCase):
|
||||
ISSUE = {"token": "Tesst", "language": "de", "suggestions": ["Tests", "Test"]}
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user