feat(magazine): detect inset pull quotes as their own region

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -14,6 +14,8 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
 
 - **Clean up boxes now also fits text boxes to their text.** A side that cuts through letters grows (at most a letter or two, never into a neighbouring box) and empty margins are trimmed. Boxes you drew or resized by hand are never touched. It also runs automatically after **OCR again (this page)**; on page 61 of the measured issue it widened the left-column paragraphs ~10 pt to cover their text, cutting clipped edges from 7 to 0. A box never grows into a neighbouring box; when it gains whole lines, its text is re-read from the enlarged crop and replaced only by a longer reading that still starts with the old text.
 
+- **Inset pull quotes get a region of their own.** A display-coloured quote set into the text (detection wraps body text around it, so no box covered it) is found from the page image — a muted-colour blob of quote size on plain paper that no headline, figure or existing pull quote covers — and added as a *pull quote* region (never read aloud). Runs after **OCR again (this page)** and in **Clean up boxes**. On the measured 204-page issue it finds the two real quotes (pages 40 and 62) and one infographic (page 139, harmless and deletable).
+
 ### Fixed
 - **The review list shows each region's whole text**, not the first 220 characters, so every misspelling gets its red wavy underline (like a word processor). The engine's `[Non-Text]` marker and hairline boxes (under 3 pt tall) are now treated as OCR noise by *Clean up boxes* and after a page re-OCR, so they no longer show up as near-invisible regions. Fitted boxes keep a little air on grown sides.
 - **The quality check no longer flags correct text as "suspect OCR".** The word check guessed the language per fragment (`iPhones statt Token` read as English) and flagged valid German; it now uses the page's language, the same lenient German rules as the spellchecker, and treats words that recur on the page (names, brands) as vocabulary. Page 61 of the measured issue went from 11 suspect regions to 3 (one a genuinely truncated word), and the issue from 758 to 610 before the language fix's full effect.

Manual.md

@@ -286,6 +286,8 @@ unreviewed boundaries may include a neighbouring article or an advert.
 
 **Clean up boxes** (review toolbar) also fits each text box to the ink of its text: a side that cuts through letters grows a little, empty margin is trimmed. Boxes you drew or resized are never changed; a backup of the layout is kept and the message reports how many boxes were fitted.
 
+**Inset pull quotes:** a quote set into the text in a display colour is added as its own *pull quote* region (unticked, never read aloud) by **OCR again (this page)** and **Clean up boxes**. If a picture or infographic is picked up by mistake, delete the region or change its type.
+
 ### One paragraph, one block — even across columns and pages
 
 **Purpose:** a paragraph that continues into the next column or onto a later page is separate regions in the layout. Reading and the MP3 join them before splitting into sentences, so narration never stops mid-sentence at a seam.

core/magazine.py

@@ -3048,6 +3048,102 @@ def snap_text_boxes(doc_dir: Path, layout: dict[str, Any],
     return layout, moved
 
 
+_PULLQUOTE_MIN_W, _PULLQUOTE_MIN_H = 55.0, 28.0      # points
+_PULLQUOTE_MAX_W, _PULLQUOTE_MAX_H = 230.0, 100.0
+_PULLQUOTE_MAX_INK = 0.25       # share of the box that is coloured ink: text, not a filled chart
+_PULLQUOTE_PAPER = 0.6          # share of the box that is plain paper: a quote sits on the page, a photo/ad does not
+_PULLQUOTE_COVERED = 0.3        # a blob this covered by a headline/figure box is that, not a quote
+
+
+def detect_inset_pullquotes(doc_dir: Path, layout: dict[str, Any], pages: Iterable[int] | None = None,
+                            vision_ocr_url: str | None = None, vision_ocr_model: str | None = None,
+                            vision_ocr_fallback_url: str | None = None,
+                            vision_ocr_fallback_model: str | None = None) -> tuple[dict[str, Any], list[str]]:
+    """Find a coloured display quote inset into the text and give it a pull-quote region of its own.
+
+    Detection layouts wrap body text around such an inset, so no box covers it and the body
+    paragraph's rectangle runs over it. The quote is printed in a display colour (teal on the
+    measured issue) unlike the grey/black body text: a dense-enough blob of that colour, of
+    quote size, that no headline, figure or existing pull quote already covers. The new region is
+    typed pull quote (never read aloud); its text is read from the crop when a vision engine is
+    available. Returns the layout and the new region ids.
+    """
+    try:
+        import numpy as np
+        from PIL import Image
+        from scipy import ndimage
+    except Exception:
+        return layout, []
+    wanted = set(pages) if pages is not None else None
+    layout = dict(layout)
+    blocks = list(layout.get("blocks", []))
+    added: list[str] = []
+    counter = 0
+    for meta in layout.get("pages", []):
+        page_no = meta["page"]
+        if wanted is not None and page_no not in wanted:
+            continue
+        png = doc_dir / "pages" / f"{page_no:04d}.png"
+        if not png.is_file() or page_is_advertisement({**layout, "blocks": blocks}, page_no):
+            continue
+        try:
+            with Image.open(png) as im:
+                rgb = np.asarray(im.convert("RGB"), dtype=np.int16)
+        except Exception:
+            continue
+        height, width, _ = rgb.shape
+        sx, sy = width / max(1.0, float(meta["width"])), height / max(1.0, float(meta["height"]))
+        r, g, b = rgb[..., 0], rgb[..., 1], rgb[..., 2]
+        coloured = (g - r > 12) & (b - r > 12) & (rgb.max(axis=2) < 190)
+        if coloured.sum() < 400:
+            continue
+        labels, _ = ndimage.label(ndimage.binary_dilation(coloured, iterations=10))
+        cover = [bb["bbox"] for bb in blocks if bb["page"] == page_no and bb["type"] in (
+            BLOCK_TITLE, BLOCK_HEADING, BLOCK_DECK, BLOCK_BYLINE, BLOCK_PULLQUOTE, BLOCK_SIDEBAR,
+            BLOCK_CAPTION, *VISUAL_TYPES)]
+        cover += [im_["bbox"] for im_ in layout.get("images", []) if im_["page"] == page_no]
+        found = 0
+        for index, sl in enumerate(ndimage.find_objects(labels), 1):
+            ys, xs = sl
+            w_pt, h_pt = (xs.stop - xs.start) / sx, (ys.stop - ys.start) / sy
+            ink = int((coloured[sl] & (labels[sl] == index)).sum())
+            if ink < 400 or not (_PULLQUOTE_MIN_W <= w_pt <= _PULLQUOTE_MAX_W
+                                 and _PULLQUOTE_MIN_H <= h_pt <= _PULLQUOTE_MAX_H):
+                continue
+            if ink / max(1, (xs.stop - xs.start) * (ys.stop - ys.start)) > _PULLQUOTE_MAX_INK:
+                continue
+            region = rgb[sl]
+            paper = (region.max(axis=2) - region.min(axis=2) < 28) & (region.max(axis=2) > 205)
+            if paper.mean() < _PULLQUOTE_PAPER:
+                continue
+            # A quote is set in one muted display colour; an illustration or infographic on a pale
+            # ground also has strongly saturated areas (red/orange/violet), a quote has none.
+            if ((region.max(axis=2) - region.min(axis=2)) > 70).mean() > 0.01:
+                continue
+            box = [xs.start / sx, ys.start / sy, w_pt, h_pt]
+            if any(_area_inside(box, c) >= _PULLQUOTE_COVERED for c in cover):
+                continue
+            found += 1
+            if found > 2:
+                break
+            counter += 1
+            text = ""
+            if vision_ocr_url and vision_ocr_model:
+                try:
+                    text = " ".join((extract_region_text(doc_dir, layout, page_no, box, vision_ocr_url,
+                                                         vision_ocr_model, vision_ocr_fallback_url,
+                                                         vision_ocr_fallback_model) or "").split())
+                except Exception:
+                    text = ""
+            new_id = f"pq{page_no:04d}-{counter}"
+            blocks.append(Block(id=new_id, page=page_no, bbox=[round(v, 1) for v in box],
+                                type=BLOCK_PULLQUOTE, text=text or "(no text recognised: inset quote)",
+                                confidence=0.6, source="inset_pullquote", region_label="text").to_json())
+            added.append(new_id)
+    layout["blocks"] = blocks
+    return layout, added
+
+
 def reread_grown_regions(doc_dir: Path, layout: dict[str, Any], moved: Sequence[dict[str, Any]],
                          vision_ocr_url: str | None, vision_ocr_model: str | None,
                          vision_ocr_fallback_url: str | None = None,

routes/magazine.py

@@ -858,6 +858,8 @@ def _reocr_page_sync(doc_id: str, job_id: str, page: int,
         _recognise_semantic_layout(updated, _load_settings(), page, doc_dir=d)
         mag.refresh_visual_regions(d, updated, page)
         updated, cleanup = mag.clean_up_regions(updated, pages=[page])
+        updated, _ = mag.detect_inset_pullquotes(d, updated, [page], vision_ocr_url, vision_ocr_model,
+                                                 vision_ocr_fallback_url, vision_ocr_fallback_model)
         updated, snapped = mag.snap_text_boxes(d, updated, pages=[page])   # boxes cover their whole text
         if any(m.get("lines_added") for m in snapped):
             updated, _ = mag.reread_grown_regions(d, updated, snapped, vision_ocr_url, vision_ocr_model,
@@ -1331,6 +1333,12 @@ async def magazine_clean_up_boxes(doc_id: str):
     backup = d / f"layout.backup-{time.strftime('%Y%m%dT%H%M%S')}.json"
     backup.write_text(json.dumps(layout, ensure_ascii=False), "utf-8")
     layout, report = await asyncio.to_thread(mag.clean_up_regions, layout)
+    settings = _load_settings()
+    layout, pullquotes = await asyncio.to_thread(
+        mag.detect_inset_pullquotes, d, layout, None,
+        str(settings.get("vision_ocr_url") or "").strip() or None, str(settings.get("vision_ocr_model") or "").strip() or None,
+        str(settings.get("vision_ocr_fallback_url") or "").strip() or None, str(settings.get("vision_ocr_fallback_model") or "").strip() or None)
+    report["pullquote_added"] = pullquotes
     # Last, so the boxes are fitted after the rules above have settled which ones remain.
     layout, snapped = await asyncio.to_thread(mag.snap_text_boxes, d, layout)
     report["snapped"] = snapped

static/js/magazine.js

@@ -2411,7 +2411,7 @@ async function magCleanUpBoxes() {
     return say('Could not clean up the boxes: ' + e.message, 'err');
   }
   const c = r.counts || {};
-  const parts = [[c.in_picture, 'text in pictures'], [c.ocr_noise, 'OCR noise'], [c.duplicate, 'duplicates'], [c.byline_split, 'author lines split'], [c.snapped, 'boxes fitted to their text']]
+  const parts = [[c.in_picture, 'text in pictures'], [c.ocr_noise, 'OCR noise'], [c.duplicate, 'duplicates'], [c.byline_split, 'author lines split'], [c.snapped, 'boxes fitted to their text'], [c.pullquote_added, 'inset pull quotes found']]
     .filter(([n]) => n).map(([n, what]) => n + ' ' + what);
   say(parts.length ? 'Cleaned up: ' + parts.join(', ') + '. Backup: ' + r.backup : 'Nothing to clean up.', 'ok');
   state.layout = null;

tests/test_magazine.py

@@ -2813,3 +2813,51 @@ class NonTextNoiseTests(unittest.TestCase):
         types = {b["id"]: b["type"] for b in out["blocks"]}
         self.assertEqual((types["n"], types["h"], types["ok"]), ("furniture", "furniture", "body"))
         self.assertEqual(len(report["ocr_noise"]), 2)
+
+
+class InsetPullquoteTests(unittest.TestCase):
+    def setUp(self):
+        from pathlib import Path
+        from PIL import Image, ImageDraw
+        self.dir = Path(tempfile.mkdtemp(prefix="ttvc-pq-"))
+        (self.dir / "pages").mkdir()
+        im = Image.new("RGB", (600, 800), (248, 242, 244))
+        draw = ImageDraw.Draw(im)
+        for y in range(300, 350, 12):                       # muted teal display text, ~140x50 pt
+            for x in range(100, 240, 6):                    # thin dashes, like letter strokes
+                draw.rectangle([x, y, x + 2, y + 5], fill=(100, 122, 121))
+        for y in range(100, 200, 10):                       # grey body text elsewhere
+            draw.rectangle([300, y, 500, y + 4], fill=(93, 91, 92))
+        draw.rectangle([400, 500, 560, 620], fill=(30, 40, 200))   # a strongly coloured picture
+        im.save(self.dir / "pages" / "0001.png")
+
+    def tearDown(self):
+        shutil.rmtree(self.dir, ignore_errors=True)
+
+    def layout(self, blocks=()):
+        return {"pages": [{"page": 1, "width": 600, "height": 800}], "images": [], "blocks": list(blocks)}
+
+    def test_an_uncovered_display_coloured_quote_becomes_a_pullquote_region(self):
+        from core.magazine import detect_inset_pullquotes
+        out, added = detect_inset_pullquotes(self.dir, self.layout())
+        self.assertEqual(len(added), 1)
+        block = out["blocks"][0]
+        self.assertEqual((block["type"], block["source"]), ("pullquote", "inset_pullquote"))
+        x, y, w, h = block["bbox"]
+        self.assertTrue(90 <= x <= 105 and 290 <= y <= 305 and w >= 135, block["bbox"])
+
+    def test_body_text_photos_and_covered_headlines_are_not_quotes(self):
+        from core.magazine import detect_inset_pullquotes
+        covered = [{"id": "t", "page": 1, "type": "title", "bbox": [90, 290, 160, 70], "text": "Headline"}]
+        self.assertEqual(detect_inset_pullquotes(self.dir, self.layout(covered))[1], [])
+
+    def test_running_it_twice_adds_nothing_more(self):
+        from core.magazine import detect_inset_pullquotes
+        once, _ = detect_inset_pullquotes(self.dir, self.layout())
+        self.assertEqual(detect_inset_pullquotes(self.dir, once)[1], [])
+
+    def test_the_quote_text_is_read_from_the_crop_when_a_vision_engine_is_configured(self):
+        from core.magazine import detect_inset_pullquotes
+        with mock.patch('core.magazine.extract_region_text', return_value="Weil virtuelle Butler mächtig werden"):
+            out, _ = detect_inset_pullquotes(self.dir, self.layout(), None, "http://ocr", "m")
+        self.assertEqual(out["blocks"][0]["text"], "Weil virtuelle Butler mächtig werden")