fix(magazine): apply column reading order automatically; article-end detection by signed end or following headline; link-next-page chip

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-28 13:00:14 +02:00 · 2026-09-28 13:00:14 +02:00 · 4c1c97400b
commit 4c1c97400b
parent 05bdecd58a
9 changed files with 148 additions and 12 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -29,7 +29,10 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
 - **Flow arrows follow the reading order:** a green arrow for every jump to another column and for a sentence running across boxes (dashed when a new paragraph starts, solid when a sentence continues), chips for page jumps.
 - **Much less noise:** the *Auto-fix all pages* report now lists what was fixed and only the pages that need a decision about their boxes (with a *Start with page N* button and compact page pills); pages with only words to check are collapsed and optional. The per-box "words look mis-read" marks are gone (spelling keeps its red wavy underline). The spellchecker knows ~130 company/product names, treats a word printed twice anywhere in the issue as vocabulary (missing-umlaut words such as `fur` stay flagged) and treats two unknown capitalised words in a row as a person's name: 871 → 625 findings on the measured issue.

+- **"Article ends" is only claimed when it is true.** An article counts as ended when it is signed off (`(rah) et`, `(hag)` — OCR noise after the code is tolerated) or when the next region, on the same page or at the top of the next, is a headline, deck or byline. Otherwise the last box shows a clickable *no end marker — link page N ›* chip that adds the next page's text in one click.
+
 ### Fixed
+- **Reading order is now applied automatically after import, *OCR again* and every per-page fix**, not only by the manual *Fix reading order* button. Until now the order after import was whatever the vision/LLM step returned, so a column's short last box could be read after the whole other column; on the measured issue 101 of 204 pages had a saved order that differs from the column-aware order. Pages whose order you fixed by hand stay untouched, except after *OCR again*, when every region is new and a hand order can no longer apply.
 - **The review list shows each region's whole text**, not the first 220 characters, so every misspelling gets its red wavy underline (like a word processor). The engine's `[Non-Text]` marker and hairline boxes (under 3 pt tall) are now treated as OCR noise by *Clean up boxes* and after a page re-OCR, so they no longer show up as near-invisible regions. Fitted boxes keep a little air on grown sides.
 - **The quality check no longer flags correct text as "suspect OCR".** The word check guessed the language per fragment (`iPhones statt Token` read as English) and flagged valid German; it now uses the page's language, the same lenient German rules as the spellchecker, and treats words that recur on the page (names, brands) as vocabulary. Page 61 of the measured issue went from 11 suspect regions to 3 (one a genuinely truncated word), and the issue from 758 to 610 before the language fix's full effect.
 - Suspended compounds (`Programmier- und Hausaufgaben`) no longer lose their hyphen when regions are merged or layouts are grouped; already-ingested `Programmierund` shows in the spelling list with a one-click repair.
--- a/Manual.md
+++ b/Manual.md
@ -287,6 +287,8 @@ unreviewed boundaries may include a neighbouring article or an advert.
 **Purpose:** see at a glance what is wrong with a page and fix it without hunting for the right button.
 **Steps:** open a page in review. The **What is wrong on this page** panel lists each problem in plain language and marks it on the page: **red dashed frame** = printed text outside any box (never read); **orange edge** = a box edge cutting through text; **purple dashed outline** = two boxes over the same text; **yellow dotted outline** = words that look mis-read; **red wavy underline** (in the list) = words no dictionary knows. Each row has its own button (*Find missed text*, *Fit boxes to the text*, *Verify OCR text*, *Fix with language model*). **Fix this page automatically** does all of them in one go: fits boxes, adds an inset pull quote, cuts a paragraph that wraps around it, finds missed text, re-reads suspicious words and, when a language model is configured, fixes provable misspellings.
 **Result:** the page reloads with what was fixed; **Undo** (above the list) restores the page as it was. Overlapping boxes have no automatic fix: delete or merge one.
+**Where an article ends:** the last box says *article ends* only when the article is signed off (`(rah) et`) or a headline/deck/byline follows it (same page or top of the next). Otherwise it shows *no end marker — link page N ›*; click it to add the text of that page that carries the article on (Undo reverses it).
+**Reading order:** applied automatically after import, *OCR again* and *Fix this page automatically*; pages whose order you set by hand keep it (*Automatic order* releases a page).
 **Text flow:** green arrows and chips on the page show where the article carries on: an arrow to the next box below, a curved *next column* arrow, and a chip *continues on page N ›* / *‹ from page N*, plus *article starts / ends*. Tick or untick **Show text flow** to hide them. In the reading view a small marker (`↓`, `col ›`, `p.N ›`) sits at the word where a paragraph crosses a box, column or page. The markers are labelled for screen readers.
 **Whole magazine:** **Auto-fix all pages** (contents view, formerly *Quality check / run*) applies the same fixes to every page and then lists the pages that still need a look (purple badges).
 **Errors:** a failed fix shows the reason and changes nothing. **Offline / privacy:** the box fixes and the audit are local; the OCR re-read and spelling steps use only the engines you configured.
--- a/core/magazine.py
+++ b/core/magazine.py
@ -4130,7 +4130,7 @@ def clean_up_regions(layout: dict[str, Any], pages: Iterable[int] | None = None)
    layout = dict(layout)
    layout["blocks"] = blocks
    for page_no in {x["page"] for x in report["byline_split"] if x["new_id"]}:
-        layout = _reorder_page(layout, page_no)
+        layout = reorder_page(layout, page_no)
    return layout, report


@ -4211,14 +4211,14 @@ def change_region_type(doc_dir: Path, layout: dict[str, Any], page_no: int, regi
        layout = add_manual_region(doc_dir, layout, page_no, block["bbox"], "image")
        new_id = next(im["id"] for im in layout["images"] if im["id"] not in before)
        layout, _ = fold_text_into_picture(delete_region(layout, region_id), page_no, block["bbox"])
-        return _reorder_page(layout, page_no), new_id
+        return reorder_page(layout, page_no), new_id

    block_type, region_label = REGION_TYPES[new_type]
    if block is not None:
        layout = dict(layout)
        layout["blocks"] = [dict(b, type=block_type, region_label=region_label, confidence=1.0)
                            if b["id"] == region_id else b for b in layout.get("blocks", [])]
-        return _reorder_page(layout, page_no), region_id
+        return reorder_page(layout, page_no), region_id

    # Image to text: read the box, then give the new block the chosen type.
    before = {b["id"] for b in layout.get("blocks", [])}
@ -4228,7 +4228,7 @@ def change_region_type(doc_dir: Path, layout: dict[str, Any], page_no: int, regi
    new_id = next(b["id"] for b in layout["blocks"] if b["id"] not in before)
    layout["blocks"] = [dict(b, type=block_type, region_label=region_label) if b["id"] == new_id else b
                        for b in layout["blocks"]]
-    return _reorder_page(layout, page_no), new_id
+    return reorder_page(layout, page_no), new_id


 def _page_text_blocks_in_order(layout: dict[str, Any], page_no: int) -> list[dict]:
@ -4352,7 +4352,7 @@ def clear_page_order(layout: dict[str, Any], page_no: int) -> dict[str, Any]:
    layout = dict(layout)
    layout["pages"] = [{k: v for k, v in p.items() if k != "order_locked"} if p["page"] == page_no else p
                       for p in layout.get("pages", [])]
-    return _reorder_page(layout, page_no)
+    return reorder_page(layout, page_no)


 def recompute_reading_order(layout: dict[str, Any]) -> tuple[dict[str, Any], list[int]]:
@ -4367,7 +4367,7 @@ def recompute_reading_order(layout: dict[str, Any]) -> tuple[dict[str, Any], lis
        page_no = page_meta["page"]
        before = [b["id"] for b in sorted((b for b in layout.get("blocks", []) if b["page"] == page_no),
                                          key=lambda b: b.get("reading_order", 0))]
-        layout = _reorder_page(layout, page_no)
+        layout = reorder_page(layout, page_no)
        after = [b["id"] for b in sorted((b for b in layout.get("blocks", []) if b["page"] == page_no),
                                         key=lambda b: b.get("reading_order", 0))]
        if before != after:
@ -4375,10 +4375,10 @@ def recompute_reading_order(layout: dict[str, Any]) -> tuple[dict[str, Any], lis
    return layout, changed


-def _reorder_page(layout: dict[str, Any], page_no: int) -> dict[str, Any]:
+def reorder_page(layout: dict[str, Any], page_no: int) -> dict[str, Any]:
    """Recompute one page's reading order -- unless the reader fixed that order by hand."""
    page_meta = next((p for p in layout.get("pages", []) if p["page"] == page_no), None)
-    if not page_meta or page_meta.get("order_locked"):
+    if not page_meta or page_meta.get("order_locked") or not page_meta.get("width"):
        return layout
    # Only the position is written back, so fields the Block model does not carry survive.
    fields = Block.__dataclass_fields__
@ -4651,8 +4651,24 @@ def article_flow(layout: dict[str, Any], article: dict[str, Any]) -> dict[str, A
    links = [{"from": a["id"], "to": b["id"], "relation": _flow_relation(a, b),
              "from_page": a["page"], "to_page": b["page"], "sentence": (a["id"], b["id"]) in running}
             for a, b in zip(narrated, narrated[1:])]
+    last = narrated[-1] if narrated else None
+    ended = False
+    if last is not None:
+        # Two signs an article stops: it is signed off ("(rah) ct"), or the next region -- on this
+        # page, else at the top of the next -- is a headline, deck or byline: something new starts.
+        flowing = sorted((b for b in layout.get("blocks", [])
+                          if b["type"] in NARRATABLE and (b.get("text") or "").strip()),
+                         key=lambda b: (b["page"], b.get("reading_order", 0)))
+        at = next((i for i, b in enumerate(flowing) if b["id"] == last["id"]), None)
+        following = flowing[at + 1] if at is not None and at + 1 < len(flowing) else None
+        ended = bool(_ARTICLE_END_RE.search((last.get("text") or "").strip())
+                     or (following is not None and following["type"] in _LINK_BOUNDARY_TYPES
+                         and following["page"] - last["page"] <= 1)
+                     or following is None)
    return {"links": links, "first": narrated[0]["id"] if narrated else None,
-            "last": narrated[-1]["id"] if narrated else None}
+            "last": last["id"] if last else None,
+            "last_page": last["page"] if last else None,
+            "ended": ended}


 _ARTICLE_END_RE = re.compile(r"\(\s*[A-Za-zÄÖÜäöü]{2,4}\s*\)\s*\S{0,4}\s*$")   # "(hag)", "(atr) c't": a signed end
--- a/routes/magazine.py
+++ b/routes/magazine.py
@ -924,6 +924,11 @@ def _reocr_page_sync(doc_id: str, job_id: str, page: int,
        _recognise_semantic_layout(updated, _load_settings(), page, doc_dir=d)
        mag.refresh_visual_regions(d, updated, page)
        updated, cleanup = mag.clean_up_regions(updated, pages=[page])
+        # Every region on the page is new, so an order set by hand for the old ones cannot apply.
+        for meta_ in updated.get("pages", []):
+            if meta_.get("page") == page:
+                meta_.pop("order_locked", None)
+        updated = mag.reorder_page(updated, page)
        updated, _ = mag.detect_inset_pullquotes(d, updated, [page], vision_ocr_url, vision_ocr_model,
                                                 vision_ocr_fallback_url, vision_ocr_fallback_model)
        updated, wrap = mag.split_around_pullquotes(d, updated, [page])
@ -1076,6 +1081,10 @@ def _fix_page_geometry(d: Path, layout: dict[str, Any], articles: list[dict[str,
    layout, wrap = mag.split_around_pullquotes(d, layout, [page])
    cleanup["wrap_split"] = wrap
    layout, snapped = mag.snap_text_boxes(d, layout, pages=[page])
+    # The column-aware reading order is otherwise only applied by the manual "Fix reading order"
+    # button; after import or a re-OCR the order would be whatever the vision/LLM step returned.
+    # A page whose order the reviewer fixed by hand is left alone.
+    layout = mag.reorder_page(layout, page)
    if url and model and any(m.get("lines_added") for m in snapped):
        layout, _ = mag.reread_grown_regions(d, layout, snapped, url, model, fb_url, fb_model)
    changed = _add_split_paragraphs_to_articles(articles, cleanup)
--- a/static/js/magazine.js
+++ b/static/js/magazine.js
@ -3233,7 +3233,23 @@ function magFlowPaint() {
    el.appendChild(c);
  };
  if (state.flow.first) chip(state.flow.first, 'top', 'article starts', 'is-edge');
-  if (state.flow.last) chip(state.flow.last, 'bottom', 'article ends', 'is-edge');
+  if (state.flow.last) {
+    // "Article ends" only when it says so (a signed end such as "(hag)"). Otherwise the next page
+    // may well carry it on: offer to link it with one click.
+    const nextPage = (state.flow.last_page || 0) + 1;
+    const lastBox = boxOf(state.flow.last);
+    if (state.flow.ended || nextPage > Number(state.layout?.page_count || 0)) {
+      chip(state.flow.last, 'bottom', 'article ends', 'is-edge');
+    } else if (lastBox) {
+      const open = document.createElement('button');
+      open.type = 'button';
+      open.className = 'mag-flow-chip is-bottom is-open';
+      open.textContent = 'no end marker — link page ' + nextPage + ' ›';
+      open.title = 'This article has no signed end, so it may continue on page ' + nextPage + '. Click to add the text of that page that carries it on.';
+      open.addEventListener('click', (e) => { e.stopPropagation(); e.preventDefault(); magLinkPage('after', nextPage); });
+      lastBox.appendChild(open);
+    }
+  }
  const arrows = [];
  (state.flow.links || []).forEach((l) => {
    if (l.relation === 'page') {
@ -3425,9 +3441,9 @@ async function magAutofixPage(mode) {
  renderReview();
 }

-async function magLinkPage(where) {
+async function magLinkPage(where, pageOverride) {
  const input = $('mag-link-page');
-  const page = Number(input && input.value);
+  const page = Number(pageOverride || (input && input.value));
  if (!page) return say('Type the page number to link first', 'err');
  try {
    const r = await api('/api/magazine/' + state.doc.id + '/articles/' + state.article.id + '/link-page', {
--- a/static/style.css
+++ b/static/style.css
@ -5523,3 +5523,6 @@ input[list="lang-codes"] {
 details.mag-report-group { display: block; }
 details.mag-report-group summary { cursor: pointer; margin-bottom: 6px; }
 .mag-page-pill { min-width: 2.6em; padding: 2px 8px; }
+
+.mag-flow-chip.is-open { pointer-events: auto; cursor: pointer; background: #b45309; border: 0; font: inherit; font-size: 11px; }
+.mag-flow-chip.is-open:hover, .mag-flow-chip.is-open:focus-visible { background: #92400e; }
--- a/tests/magazine.spec.js
+++ b/tests/magazine.spec.js
@ -547,6 +547,29 @@ test.describe('Magazine: text flow indicator', () => {
    await expect(page.locator('#mag-review-page-wrap .mag-flow-path, #mag-review-page-wrap .mag-flow-chip')).toHaveCount(0);
  });

+  test('an article without a signed end offers to link the next page instead of claiming it ends', async ({ page }) => {
+    await mockBackend(page);
+    let linked = null;
+    await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/flow`, route => route.fulfill({ json: {
+      links: [], first: 'b1', last: 'b3', last_page: 3, ended: false } }));
+    await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/link-page`, (route) => {
+      linked = route.request().postDataJSON();
+      return route.fulfill({ json: { ok: true, undo: 'tok', added: ['b9'], page: 4 } });
+    });
+    await openReview(page);
+    await expect(page.locator('#mag-review-page-wrap .mag-flow-chip', { hasText: 'article ends' })).toHaveCount(0);
+    await page.getByRole('button', { name: /no end marker — link page 4/ }).click();
+    await expect.poll(() => linked).toEqual({ page: 4, where: 'after' });
+  });
+
+  test('a signed end says the article ends', async ({ page }) => {
+    await mockBackend(page);
+    await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/flow`, route => route.fulfill({ json: {
+      links: [], first: 'b1', last: 'b3', last_page: 3, ended: true } }));
+    await openReview(page);
+    await expect(page.locator('#mag-review-page-wrap .mag-flow-chip', { hasText: 'article ends' })).toHaveCount(1);
+  });
+
  test('the reading text marks where a paragraph crosses a column or a page', async ({ page }) => {
    await mockBackend(page, { articleStatus: 'approved' });
    const text = 'Der Satz beginnt hier und geht auf der nächsten Seite weiter, bis er endet.';
--- a/tests/test_magazine.py
+++ b/tests/test_magazine.py
@ -3022,3 +3022,53 @@ class AutoLinkPagesTests(unittest.TestCase):
                  self.blk("z", 6, "body", "weit weg", 0)]
        out, report = self.link(blocks, [self.art("article-001", "Alt", [1], ["t", "a"]), self.art("article-002", "Page 6", [6], ["z"])])
        self.assertEqual(report["unlinked"], ["z"])
+
+
+class FlowEndedAndAutoOrderTests(unittest.TestCase):
+    def layout(self, last_text):
+        blocks = [{"id": "a", "page": 1, "type": "body", "text": "Der Anfang.", "reading_order": 0, "bbox": [10, 10, 90, 50]},
+                  {"id": "b", "page": 1, "type": "body", "text": last_text, "reading_order": 1, "bbox": [10, 70, 90, 50]}]
+        return {"pages": [{"page": 1, "width": 200, "height": 300}], "blocks": blocks, "images": []}, {"blocks": ["a", "b"]}
+
+    def test_only_a_signed_end_says_the_article_ended(self):
+        from core.magazine import article_flow
+        self.assertTrue(article_flow(*self.layout("Und Schluss. (hag) ct"))["ended"])
+        open_layout, article = self.layout("Und die Frage bleibt offen?")
+        open_layout["blocks"].append({"id": "c", "page": 2, "type": "body", "text": "Geht weiter.", "reading_order": 0, "bbox": [10, 10, 90, 50]})
+        self.assertFalse(article_flow(open_layout, article)["ended"])
+        self.assertEqual(article_flow(*self.layout("x"))["last_page"], 1)
+
+    def test_a_locked_page_keeps_its_order_and_an_unlocked_page_gets_the_column_order(self):
+        from core.magazine import reorder_page
+        def blk(i, x, y, n):
+            return {"id": i, "page": 1, "type": "body", "text": "Text " + i, "reading_order": n, "bbox": [x, y, 90, 60]}
+        # right column listed first by the (wrong) engine order; left column second
+        blocks = [blk("r1", 110, 10, 0), blk("r2", 110, 80, 1), blk("l1", 10, 10, 2), blk("l2", 10, 80, 3)]
+        layout = {"pages": [{"page": 1, "width": 210, "height": 300}], "images": [], "blocks": blocks}
+        order = lambda l: [b["id"] for b in sorted(l["blocks"], key=lambda b: b["reading_order"])]
+        self.assertEqual(order(reorder_page(layout, 1)), ["l1", "l2", "r1", "r2"])
+        locked = dict(layout, pages=[{"page": 1, "width": 210, "height": 300, "order_locked": True}])
+        self.assertEqual(order(reorder_page(locked, 1)), ["r1", "r2", "l1", "l2"])
+
+
+class ArticleEndedTests(unittest.TestCase):
+    def ended(self, rows, own):
+        from core.magazine import article_flow
+        blocks = [{"id": i, "page": pg, "type": ty, "text": txt, "reading_order": n, "bbox": [10, 10 + 60 * n, 90, 50]}
+                  for n, (i, pg, ty, txt) in enumerate(rows)]
+        return article_flow({"pages": [], "blocks": blocks, "images": []}, {"blocks": own})["ended"]
+
+    def test_a_signed_end_with_ocr_noise_after_it_ends_the_article(self):
+        self.assertTrue(self.ended([("a", 1, "body", "Der Schluss. (rah) et"), ("z", 1, "body", "weiter geht es")], ["a"]))
+
+    def test_a_headline_as_the_next_block_ends_the_article(self):
+        self.assertTrue(self.ended([("a", 1, "body", "Ohne Kürzel am Ende"), ("h", 1, "title", "Neuer Beitrag")], ["a"]))
+
+    def test_a_headline_at_the_top_of_the_next_page_ends_it_too(self):
+        self.assertTrue(self.ended([("a", 1, "body", "Ohne Kürzel am Ende"), ("h", 2, "heading", "Neuer Beitrag")], ["a"]))
+
+    def test_running_text_next_means_it_probably_continues(self):
+        self.assertFalse(self.ended([("a", 1, "body", "Ohne Kürzel am Ende"), ("b", 2, "body", "Geht auf der nächsten Seite weiter.")], ["a"]))
+
+    def test_a_headline_two_pages_on_does_not_end_it(self):
+        self.assertFalse(self.ended([("a", 1, "body", "Ohne Kürzel am Ende"), ("h", 3, "title", "Viel später")], ["a"]))
--- a/tests/test_magazine_routes.py
+++ b/tests/test_magazine_routes.py
@ -1805,3 +1805,17 @@ class ArticleLinkingRouteTests(unittest.TestCase):
        with self.assertRaises(HTTPException) as ctx:
            self.call(magazine_link_page, "article-001", body={})
        self.assertEqual(ctx.exception.status_code, 400)
+
+
+class BoxFixesReorderThePageTests(unittest.TestCase):
+    def test_the_per_page_fix_applies_the_column_order_the_engine_order_lacked(self):
+        """Regression: after import or a re-OCR the order was whatever the vision/LLM step
+        returned, because the column-aware order only ran from a manual button."""
+        from routes.magazine import _fix_page_geometry
+        d = Path(tempfile.mkdtemp(prefix="ttvc-order-"))
+        self.addCleanup(shutil.rmtree, d, True)
+        blk = lambda i, x, y, n: {"id": i, "page": 1, "type": "body", "text": "Text " + i, "reading_order": n, "bbox": [x, y, 90, 60]}
+        layout = {"pages": [{"page": 1, "width": 210, "height": 300}], "images": [],
+                  "blocks": [blk("r1", 110, 10, 0), blk("l1", 10, 10, 1), blk("r2", 110, 80, 2), blk("l2", 10, 80, 3)]}
+        fixed, _, _ = _fix_page_geometry(d, layout, [], 1, {})
+        self.assertEqual([b["id"] for b in sorted(fixed["blocks"], key=lambda b: b["reading_order"])], ["l1", "l2", "r1", "r2"])