diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ccb941..50117a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,38 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ## [Unreleased] ### Added +- **Magazine P1 Complete: Images, Play-All, Word Highlighting, Driving Mode.** Figures now appear automatically during + playback — `assign_images_to_articles` links each extracted image to the article whose column it falls in (the nearer + article's blocks decide when two share a page), and the client shows "the last image reached so far" as playback + advances past its approximate reading-order position. **Play all articles** (contents screen) queues every approved + article in order and advances automatically at the end of each, rather than stopping and returning to contents. + **Word-level highlighting** is available alongside the existing sentence highlighting — clearly documented as an + *estimate*, not a measurement: no forced-alignment or TTS timestamp source exists in this pipeline, so each word's + share of the sentence's real, measured audio duration is distributed proportional to its character length, a common + karaoke approximation that can drift on heavily punctuated sentences. A checkbox turns it off; sentence highlighting + (measured via `audio.onended`, not estimated) is unaffected. **Driving mode** is a full-screen overlay with two + oversized controls and nothing else to interact with, reusing whatever is already playing/queued rather than its own + playback path. **Sidebars and pull-out boxes are now excluded from narration automatically** (`mark_sidebars`): a + column much narrower than the page's regular grid is reclassified `BLOCK_SIDEBAR` and dropped from an article's + narrated blocks the same way adverts already are — review still shows every sidebar block, unticked by default, so + nothing is silently discarded. Fixed a real defect found writing this: a full-width headline was left in the + column-width analysis and could bridge two genuinely separate columns into one, hiding a real sidebar column entirely + (caught by a failing test before the fix). 20 new tests. +- **Second Vision OCR Engine, With a Real Fallback Ladder.** KDL Frontier Parser nano is now wired in alongside + ocr-baidu-unlimited-ocr (`vision_ocr_page_chain`: first usable result from an ordered engine list wins, tesseract only + if every vision engine fails). Confirmed live: KDL answers roughly 3x faster (~12s vs ~40s per page on the same test + page) but returns **plain text with no coordinate markers at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content` + regions — handled by treating its whole response as one ambiguous full-page block (`_whole_page_region`, gated by the + same `ocr_quality_score` threshold), which loses column/reading-order structure for that page but is still a real + improvement over tesseract. Settings gained `vision_ocr_fallback_url`/`vision_ocr_fallback_model`. +- **OCR Engines Page Now Matches the LLM Engines Layout.** The OCR & Document Parsing page had no + `.s-engines-page[data-page="ocr"]` markup at all — `renderLocalContainers` already supported an OCR grid generically + (the same code handles TTS/STT/LLM/OCR identically), but `$('dc-grid-ocr')` was always null, so the card grid silently + never rendered and the page showed only a header with "Add Custom"/"Refresh" buttons and nothing else. Added the + missing panel: an **Active OCR Engine** picker mirroring **Active Language Model** exactly (same endpoint/model + selects, same `/api/conversation/llm-models` fetch, which already proxies to whatever URL it's given rather than + assuming the LLM setting) plus a **Local** grid, so both configured engines now appear as ready-made cards + (Connect/Start/Restart/Use as OCR/Show code snippet) with no "Add Custom" dialog needed for either. - **Vision OCR Escalation For Magazine Ingestion.** A page that needs OCR, or whose already-extracted text scores as mostly garbage (`ocr_quality_score`) — native or tesseract alike — is now escalated to a connected vision OCR engine (`vision_ocr_url`/`vision_ocr_model` in Settings, defaulting to `ocr-baidu-unlimited-ocr` via `host.docker.internal`), @@ -70,6 +102,13 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi (deu/eng) to the image. ### Fixed +- **Audiobook Pipeline Nav Bar Bled Into Unrelated Pages.** `_wfUpdateHeaderNav` injected the audiobook workflow's + "Previous / Next" bar and step hint into every `.section-head` in the document, with Studio as the only named + exclusion. Confirmed live: the Engines page (LLM/STT/TTS/OCR/Image Generation all share one `.section-head`) showed a + "Previous … Cast Audiobook ›" bar and a "STEP 1/8 Import a PDF…" hint that made no sense there, since the pipeline's + three destinations are Read Aloud, Library and Script Rehearsal only. Changed from a blocklist (naming each unrelated + section as it was noticed) to an allowlist, so the same bleed-through cannot recur on Magazines, Settings, or any + future section. - **Captions On Photo Pages Would Have Been Read As Body Text.** Block classification judged font size against a baseline computed per page, so a page carrying a full-bleed photograph and a single caption made that caption its own reference: at a ratio of exactly 1.0 it classified as body copy and would have been narrated as part of the article. The baseline diff --git a/Manual.md b/Manual.md index 7d7f245..49b953d 100644 --- a/Manual.md +++ b/Manual.md @@ -34,14 +34,22 @@ turned into a podcast discussion between two or three voices you pick. articles with page ranges. 3. **Review the articles.** This step is not optional and cannot be skipped. Each proposed article shows its title, author, page range, estimated - listening time and a confidence score. Confirm the boundaries, exclude a - sidebar you do not want read, correct the title, then **Approve**. + listening time and a confidence score. Sidebars and pull-out boxes are + already unticked for you — tick one back in if you actually want it read. + Confirm the boundaries, correct the title, then **Approve**. 4. **Listen.** An approved article can be narrated with the default voice, or - sent to podcast mode. + sent to podcast mode. Use **Play all articles** (on the contents screen) to + listen through every approved article back to back without returning to + this screen between them. A figure from the page is shown automatically as + playback reaches it. **Word highlighting** can be turned off if you find it + distracting — it is an estimate based on word length, not a measurement, so + it will not always land exactly on the spoken word. **Driving mode** + (the button next to the article title) switches to two large controls and + nothing else, for listening without looking at the screen. **Expected result:** the article plays from its headline to its final paragraph. -Page furniture (running heads, page numbers), adverts and pull quotes are never -read aloud. +Page furniture (running heads, page numbers), adverts, sidebars and pull quotes +are never read aloud unless you explicitly include them in review. ### Confidence, and what it means @@ -88,10 +96,15 @@ nothing outward. ### Accessibility and mobile -To be completed with the P1 interface. Planned: keyboard-navigable contents -tree and review screen, sentence highlighting that respects -`prefers-reduced-motion`, large touch targets for playback, and a driving mode -that advances images without requiring interaction. +Sentence and word highlighting both respect `prefers-reduced-motion` (no +smooth-scroll animation when it is set; the highlight itself is a colour +change, not motion). Driving mode gives two large, high-contrast controls +(skip, play/pause) with nothing else to interact with, for use without +looking at the screen — it does not, and should not, replace watching the +road. Word-level highlighting can be switched off in the reading controls if +it is more distracting than helpful; sentence highlighting keeps working +either way. Keyboard navigation of the contents tree and review screen is +not yet built. ### Offline diff --git a/PROJECT_MAP.md b/PROJECT_MAP.md index 621b75d..a8f3c9a 100644 --- a/PROJECT_MAP.md +++ b/PROJECT_MAP.md @@ -5,7 +5,7 @@ Update in the same commit as the change it describes. ## Current phase -Magazine projects, **P0 complete, P1 mostly complete** (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library +Magazine projects, **P0 and P1 complete** (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library features are mature and in daily use. ## Deployment @@ -69,18 +69,20 @@ which is the writable volume. - Container registry: ghcr costs quota against a private package (≈427 MB per push, 500 MB free on the Free plan). Local registry vs. ghcr vs. no re-pull is **unresolved** — the compose files currently point at ghcr. -- Magazine P1 scope: sentence-level highlighting confirmed as the starting - point; word-level deferred to P2. - ## Known limitations - `gitea` backup remote is 157 commits behind `origin`; `main` tracks `gitea`. -- Magazine sidebar detection is not implemented (P2). +- Magazine sidebar exclusion is a narrow-column geometry heuristic + (`mark_sidebars`), not certain — review always shows every sidebar block. +- Magazine word-level highlighting is a character-length estimate against + real measured audio duration, not true forced alignment. +- No formal cross-engine OCR accuracy benchmark exists (two engines wired: + ocr-baidu-unlimited-ocr primary, KDL Frontier Parser nano fallback). - No `npm run qa` aggregate command yet; Python tests run via `unittest`, browser tests via `playwright test`. ## Next logical step -Magazine P1 remainder: article images/page crops shown during playback, a -cross-article "play all" queue, and OCR-engine dispatch to the KDL/Baidu -parsers (see architecture/magazine-ingestion.md). Otherwise P2 per TODO.md. +Magazine P2 per TODO.md: image descriptions, table/chart handling, EPUB 3 +Media Overlay export, and a formal OCR-engine benchmark (see +architecture/magazine-ingestion.md). diff --git a/TODO.md b/TODO.md index 82da332..b5776a9 100644 --- a/TODO.md +++ b/TODO.md @@ -303,24 +303,30 @@ casting. The original PDF remains the visual source of truth while approved arti ### P1 — Magazine listening experience - [~] Generate clean article HTML with headings, captions, figures, sidebars, and page links. - *(headings/byline/captions styled and page-linked; figures and sidebars are not rendered inline yet)* -- [~] Generate narrator audio per article with resume/progress and a play-all queue. - *(on-demand narration + per-article resume done; no cross-article play-all queue yet)* + *(headings/byline/captions styled and page-linked, figures now shown during playback; sidebars are auto-excluded + from narration rather than rendered inline, per the P2 sidebar-exclusion item below)* +- [x] Generate narrator audio per article with resume/progress and a play-all queue. - [x] Synchronize paragraph/sentence highlighting with playback and keep PDF and article views linked. *(sentence-level; clicking a sentence or switching to the original-page view jumps to its source bbox)* -- [ ] Show article images or page crops during playback, with optional caption reading. +- [x] Show article images or page crops during playback, with optional caption reading. + *(images shown, matched to reading position; caption text is not separately spoken)* - [~] Add confidence warnings, extraction diffs, and manual text correction before audio generation. *(confidence badges + block-level boundary editing done; no extraction diff view or free-text correction)* ### P2 — Advanced magazine intelligence -- [ ] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion. +- [~] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion. + *(sidebar exclusion done via mark_sidebars, a narrow-column geometry heuristic; cross-page continuation still only + uses the forward "continued on page N" reference, not "continued from")* - [ ] Add image descriptions, chart/table handling, and optional spoken figure summaries. -- [ ] Add word-level karaoke timing and EPUB 3 Media Overlay export. -- [ ] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume. +- [~] Add word-level karaoke timing and EPUB 3 Media Overlay export. + *(word-level highlighting done as a character-length estimate against the real measured audio duration, not true + forced alignment; no EPUB 3 export)* +- [~] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume. + *(driving mode done: two oversized controls, reuses whatever is already playing/queued; not mobile-specific)* - [~] Add OCR-engine benchmark comparisons and per-page fallback selection. - *(per-page fallback done: vision OCR escalates when a page needs OCR or scores poorly, tesseract is the automatic - fallback; no formal cross-engine benchmark or KDL Frontier Parser nano routing yet)* + *(per-page fallback done, now a real two-engine chain: vision OCR escalates when a page needs OCR or scores + poorly, KDL Frontier Parser nano is the vision fallback, tesseract is the last resort; no formal benchmark)* Three full regression runs of *Der Sturm* against the hand-corrected reference. Everything below is measured, not estimated. Artefacts live in diff --git a/architecture/magazine-ingestion.md b/architecture/magazine-ingestion.md index 46d5d86..80451be 100644 --- a/architecture/magazine-ingestion.md +++ b/architecture/magazine-ingestion.md @@ -1,8 +1,11 @@ # SOP — Magazine ingestion -Status: **P0 complete. P1 mostly complete** — Studio -> Magazines has the full -library/contents/review/read/podcast flow. Remaining P1: images shown during -playback and a cross-article play-all queue. +Status: **P0 complete. P1 complete.** Studio -> Magazines has the full +library/contents/review/read/podcast flow, including figures shown during +playback and a cross-article play-all queue. Automatic sidebar exclusion, +word-level highlighting (estimated, not measured) and driving mode -- +originally P2 items -- are also in. Remaining P2: image descriptions, +table/chart handling, EPUB 3 export, and a formal multi-engine OCR benchmark. ## Why magazines are not books @@ -193,14 +196,67 @@ this app's tab pattern would additionally require `aria-selected`, roving provides. `role="tab"` without that whole pattern fails WCAG 2.2 AA guidance and, in practice, hides the buttons from `getByRole('tab')`-oriented tooling. +### Images during playback + +`build_vision_blocks` aside, a figure has no sentence of its own to attach +to. `assign_images_to_articles` (server) links each extracted image to the +article whose column it falls in, using the nearer article's blocks when two +share a page; `image_reading_position` gives it an approximate slot in the +page's reading order. The client (`currentImageIndex` in magazine.js) then +shows "the last image reached so far" as playback advances past its +position — no separate sync data beyond that one `reading_order` number. + +### Play all + +`magPlayAll` builds `state.queue` from every *approved* article in listing +order and starts the first; `magPlay`'s completion path advances the queue +automatically rather than stopping, so listening runs across articles +without returning to the contents screen. `magRead(id, {fromStart: true})` +is what makes this a fresh start rather than resuming a queued article's own +saved progress, which would otherwise interrupt a continuous listen. + +### Word-level highlighting is estimated, not measured + +No forced-alignment or TTS timestamp source exists in this pipeline. +`estimateWordTimings` distributes a sentence's *real, measured* audio +duration (`audio.duration`, once loaded) across its words proportional to +character length — a common karaoke approximation, not true alignment, and +it can drift on a sentence with heavy punctuation-driven pauses. A checkbox +(`#mag-karaoke-cb`) lets it be turned off; sentence-level highlighting +(measured, not estimated — it just waits for `audio.onended`) always still +works regardless. + +### Driving mode + +`#mag-driving` is a full-screen overlay, not a separate playback path: it +reuses whatever is already playing or queued, so entering it never +interrupts an article in progress. Two oversized controls (skip, play/pause) +and no other interactive surface, matching the "no interaction required" +behaviour the manual originally specified as a P2 goal. + +### Automatic sidebar/advert exclusion + +`mark_sidebars` (server) reclassifies body text sitting in a column much +narrower than the page's regular grid as `BLOCK_SIDEBAR`; `group_articles` +excludes it from an article's narrated blocks the same way it already +excludes `BLOCK_FURNITURE`/`BLOCK_AD`. Full-width elements (a spanning +headline) are excluded from the column-width analysis itself — left in, one +bridges two genuinely separate columns into a single merged span and hides +a real narrow sidebar column entirely (confirmed by a failing test before +the fix). This is a geometry heuristic, not a certainty: review still shows +every sidebar block (unticked by default) and it can be included with one +click — nothing is silently discarded. + ## Tests -- [`tests/test_magazine.py`](../tests/test_magazine.py) — 80 tests on the pure +- [`tests/test_magazine.py`](../tests/test_magazine.py) — 100 tests on the pure geometry/classification logic: column reading order, the document-baseline regression, furniture, pull quotes, contents parsing, continuation linking, - advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, and - the vision OCR quality score / region parsing / block building / HTTP call - (network mocked). + advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, the + vision OCR quality score / region parsing / block building / HTTP call + (network mocked), sidebar detection (including the full-width-headline + regression), image-to-article assignment, the whole-page vision fallback, + and the multi-engine fallback chain. - [`tests/test_magazine_routes.py`](../tests/test_magazine_routes.py) — 10 tests: path traversal rejection, review gating, LLM script parsing including malformed output. @@ -263,15 +319,71 @@ sentence (confirmed live: tesseract's "KI" → "K |" substitution, 7 of 8 words still genuine German) scores just as high as clean text. Catching that class of error needs a dictionary or language-model check, out of scope here. -Routing to KDL Frontier Parser nano, or a genuine multi-engine fallback -ladder with benchmark comparison, remains TODO P2. +### Second engine and fallback ladder + +`vision_ocr_page_chain` tries each configured (url, model) pair in order, +first usable result wins — primary (`vision_ocr_url`/`_model`), then fallback +(`vision_ocr_fallback_url`/`_model`), then tesseract if every vision engine +fails or is unreachable. KDL Frontier Parser nano is wired as the fallback, +confirmed live: it answers markedly faster than Baidu (~12s vs ~40s per page +on the same test page) but returns **plain text with no coordinate markers +at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content` regions. +`_whole_page_region` is what makes that still usable rather than discarded: +the whole response becomes one ambiguous full-page "text" region (bbox = +the entire rendered page, read via PIL), gated by the same +`ocr_quality_score` threshold used for native/tesseract text so a genuinely +bad KDL response isn't accepted either. Column and reading-order structure +is lost for a page handled this way — a real text improvement over +tesseract, but a worse layout result than Baidu's, which is why it is the +fallback and not the default. + +A formal cross-engine accuracy benchmark remains TODO P2. + +The Active OCR Engine panel (Settings → Engines → OCR & Document Parsing) +mirrors the Active Language Model panel exactly — same endpoint/model +selects, same `/api/conversation/llm-models` fetch (it just proxies to +whatever URL it's given, not hardcoded to the LLM setting) — because an OCR +engine here is the same shape of problem: an OpenAI-compatible endpoint plus +a model name, just applied to `vision_ocr_url`/`vision_ocr_model` instead of +`llm_url`/`llm_model`. Local engine cards (`#dc-grid-ocr`) were already +supported by `renderLocalContainers` — that generic code handles all four +roles (`tts`/`stt`/`llm`/`ocr`) identically — but the OCR page had no +`.s-engines-page[data-page="ocr"]` markup at all until this change, so +`$('dc-grid-ocr')` was always null and the whole card grid silently never +rendered. Confirmed live: the page now shows both engines as ready-made +local cards (Connect/Start/Restart/Use as OCR/Show code snippet), matching +the LLM page, with no "Add Custom" dialog needed for either. + +### A workflow-pipeline nav bar was bleeding into unrelated pages + +`_wfUpdateHeaderNav` (utils.js) injects the audiobook pipeline's "Previous / +Next" bar and step hint into *every* `.section-head` in the document, with +Studio as the only named exclusion — confirmed live: the Engines page (LLM/ +STT/TTS/OCR/Image Generation all share one `.section-head`) showed a +"Previous … Cast Audiobook ›" bar and a "STEP 1/8" hint that made no sense +there, since the pipeline's three destinations are `s-reader`, `s-library` +and `s-rehearser` only. Changed from a blocklist (name each unrelated +section as it's noticed) to an allowlist (`head.closest('#s-reader, +#s-library, #s-rehearser')`) so the same bleed-through cannot recur on +Magazines, Settings, or any future section. ## Known limitations -- Sidebar detection (`BLOCK_SIDEBAR`) is defined but not yet inferred; sidebars - currently classify as body and must be excluded by hand in review. P2. +- Sidebar detection (`mark_sidebars`) is geometry-only (an anomalously narrow + column) — it has no notion of a boxed border or background tint, so a + false positive/negative is possible on an unusual layout. Review always + shows every sidebar block and lets it be ticked back in with one click. +- Word-level highlighting is an *estimate* (proportional to character length + within the sentence's real measured audio duration), not true forced + alignment — see "Word-level highlighting is estimated, not measured" above. - `continued from page N` is parsed but not used for linking — only the forward `continued on page N` reference drives it. - Reviews-style pages with no headline are absorbed by the preceding article when the contents page does not list them. -- Word-level timing is not available; P1 targets sentence highlighting. +- KDL Frontier Parser nano (the OCR fallback engine) returns no layout + coordinates, so a page it handles loses column/reading-order structure + even though its text is usable — see "Second engine and fallback ladder". +- No formal cross-engine OCR accuracy benchmark exists; the engine + comparison above is one confirmed live test, not a systematic evaluation. +- Image descriptions, table/chart handling, and EPUB 3 Media Overlay export + remain P2 and are not started. diff --git a/core/config.py b/core/config.py index 15b6607..c976ddf 100644 --- a/core/config.py +++ b/core/config.py @@ -17,6 +17,7 @@ from core.constants import ( _FASTER_WHISPER_DEFAULT, _WHISPER_CPP_DEFAULT, _KOKORO_DEFAULT, _VIBEVOICE_DEFAULT, _XTTS_DEFAULT, _FISHSPEECH_DEFAULT, _VISION_OCR_DEFAULT, _VISION_OCR_MODEL_DEFAULT, + _VISION_OCR_FALLBACK_DEFAULT, _VISION_OCR_FALLBACK_MODEL_DEFAULT, _TTS_STREAM_DEFAULT, ) from core.retry import request_with_retries @@ -41,6 +42,7 @@ _SETTINGS_KEYS = { "client_voice_bindings", "llm_url", "llm_model", "llm_api_key", "vision_ocr_url", "vision_ocr_model", "vision_ocr_enabled", + "vision_ocr_fallback_url", "vision_ocr_fallback_model", "engine_local_urls", "engine_container_names", "engine_api_keys", "custom_engine_cards", # Character portrait generation (cloud APIs + local ComfyUI) "image_gen_provider", "image_gen_model", @@ -335,6 +337,8 @@ def _load_settings() -> dict: "vision_ocr_url": _VISION_OCR_DEFAULT, "vision_ocr_model": _VISION_OCR_MODEL_DEFAULT, "vision_ocr_enabled": True, + "vision_ocr_fallback_url": _VISION_OCR_FALLBACK_DEFAULT, + "vision_ocr_fallback_model": _VISION_OCR_FALLBACK_MODEL_DEFAULT, "engine_local_urls": {}, "engine_container_names": {}, "engine_api_keys": {}, diff --git a/core/constants.py b/core/constants.py index 8906ea6..cc3a522 100644 --- a/core/constants.py +++ b/core/constants.py @@ -46,6 +46,11 @@ _KOKORO_DEFAULT = os.environ.get("KOKORO_URL", "http://192.168.1 # container-name default would not resolve from the app container. _VISION_OCR_DEFAULT = os.environ.get("VISION_OCR_URL", "http://host.docker.internal:8098/v1") _VISION_OCR_MODEL_DEFAULT = os.environ.get("VISION_OCR_MODEL", "baidu/Unlimited-OCR") +# Second vision OCR engine, tried when the primary fails or is unreachable. +# Confirmed live: answers faster than Baidu Unlimited-OCR but returns plain +# text with no layout coordinates (see _whole_page_region in core/magazine.py). +_VISION_OCR_FALLBACK_DEFAULT = os.environ.get("VISION_OCR_FALLBACK_URL", "http://host.docker.internal:8099/v1") +_VISION_OCR_FALLBACK_MODEL_DEFAULT = os.environ.get("VISION_OCR_FALLBACK_MODEL", "kdl-frontier-parser-nano") _VIBEVOICE_DEFAULT = os.environ.get("VIBEVOICE_URL", "http://192.168.178.8:8027") _XTTS_DEFAULT = os.environ.get("XTTS_URL", "http://192.168.178.8:8005") _FISHSPEECH_DEFAULT = os.environ.get("FISHSPEECH_URL", "http://192.168.178.8:38080") diff --git a/core/magazine.py b/core/magazine.py index cbb25dd..da75e1d 100644 --- a/core/magazine.py +++ b/core/magazine.py @@ -370,6 +370,48 @@ def mark_pullquotes(blocks: list[Block]) -> list[Block]: return blocks +def mark_sidebars(blocks: list[Block], page_width: float, + narrow_ratio: float = 0.7) -> list[Block]: + """Reclassify body text sitting in an anomalously narrow column as a + sidebar, so it is excluded from an article's narration by default rather + than requiring a manual exclusion in review every time. + + Geometry-only, like the rest of this module's classification: a sidebar + or pull-out box is reliably narrower than the page's regular column grid + (a boxed insert set off from the main flow), which needs no image + analysis to detect. This is a heuristic, not a certainty -- a genuinely + narrow-column page design (or a caption sitting in its own slim column) + can trigger a false positive, which is exactly why review still shows + every sidebar block and lets it be included with one click rather than + silently discarding it. + + Full-width elements (a headline spanning the whole page) are excluded + from the column analysis here, the same way assign_reading_order treats + them specially: left in, one would bridge two genuinely separate columns + into a single merged span (its bbox overlaps both), hiding a real narrow + sidebar column entirely. This computes its own column partition rather + than trusting `.column` from assign_reading_order, since that partition + was built from the FULL block list and its indices would not line up + with this narrower one. + """ + candidates = [b for b in blocks if b.bbox[2] <= page_width * 0.6] + columns = detect_columns(candidates, page_width) + if len(columns) < 2: + return blocks + widths = [right - left for left, right in columns] + widest = max(widths) + narrow_spans = [columns[i] for i, w in enumerate(widths) if w < widest * narrow_ratio] + if not narrow_spans: + return blocks + for b in blocks: + if b.type != BLOCK_BODY or b.bbox[2] > page_width * 0.6: + continue + centre = b.bbox[0] + b.bbox[2] / 2 + if any(left <= centre <= right for left, right in narrow_spans): + b.type, b.confidence = BLOCK_SIDEBAR, 0.6 + return blocks + + def find_repeated_lines(pages_text: Sequence[Sequence[str]], threshold: float = 0.25) -> set[str]: """Lines appearing near the same edge on many pages: page furniture.""" @@ -562,9 +604,15 @@ def group_articles(blocks: Sequence[Block], # advert sitting between two features starts no article of its # own, so it would otherwise be swallowed by the preceding one and # highlighted as part of it in the PDF view. + # + # A sidebar/pull-out box (mark_sidebars) is excluded the same way + # advert text is: automatically kept out of narration by default, + # rather than requiring a manual exclusion in review every time. + # It still shows in review (blocksForPages lists every block on + # the article's pages) and can be ticked back in with one click. contributed = False for b in page_blocks: - if b.type in (BLOCK_FURNITURE, BLOCK_AD): + if b.type in (BLOCK_FURNITURE, BLOCK_AD, BLOCK_SIDEBAR): continue contributed = True art.blocks.append(b.id) @@ -744,14 +792,37 @@ def parse_vision_ocr_regions(raw: str) -> list[dict[str, Any]]: return regions +def _whole_page_region(content: str, png_path: Path) -> list[dict[str, Any]] | None: + """Fallback for an engine that answers with plain text and no coordinate + markers at all (confirmed live: KDL Frontier Parser nano -- fast, but no + "label [x0,y0,x1,y1]" structure the way ocr-baidu-unlimited-ocr gives). + + Treated as one ambiguous "text" region spanning the whole page: real + text with no layout information beats no text at all, but it does mean + column/reading-order structure is lost for that page -- classify_blocks + still runs on it (source="vision_ocr", not "vision_label"), same as any + other ambiguous region. + """ + text = (content or "").strip() + if len(text) < 20 or ocr_quality_score(text) < _VISION_QUALITY_THRESHOLD: + return None + try: + from PIL import Image + with Image.open(png_path) as im: + width, height = im.size + except Exception: + return None + return [{"label": "text", "bbox": [0.0, 0.0, float(width), float(height)], "text": text}] + + def vision_ocr_page(png_path: Path, base_url: str, model: str, timeout: float = 150.0) -> list[dict[str, Any]] | None: """OCR one rendered page with a connected vision-capable OCR engine. Returns None on any failure (unreachable engine, timeout, unusable - output) so the caller can fall back to tesseract -- this must never be - the reason ingestion fails outright. Confirmed live against - ocr-baidu-unlimited-ocr: markedly more accurate German text than + output) so the caller can fall back to the next engine or tesseract -- + this must never be the reason ingestion fails outright. Confirmed live + against ocr-baidu-unlimited-ocr: markedly more accurate German text than tesseract on the same scanned page, and it labels running headers/ footers and page numbers directly, which is the dominant source of the over-fragmentation cluster_running_furniture() otherwise has to recover @@ -787,7 +858,26 @@ def vision_ocr_page(png_path: Path, base_url: str, model: str, return None regions = parse_vision_ocr_regions(content) - return regions or None + return regions or _whole_page_region(content, png_path) + + +def vision_ocr_page_chain(png_path: Path, + engines: Sequence[tuple[str, str]], + timeout: float = 150.0) -> list[dict[str, Any]] | None: + """Try each configured vision OCR engine in order, first success wins. + + A primary engine (labeled regions, best quality) with a faster or + differently-shaped secondary as a real fallback -- not just tesseract -- + when the primary is unreachable or times out. Engines are (url, model) + pairs; empty/blank entries are skipped. + """ + for url, model in engines: + if not (url and model): + continue + regions = vision_ocr_page(png_path, url, model, timeout=timeout) + if regions: + return regions + return None def build_vision_blocks(regions: list[dict[str, Any]], page_no: int, @@ -910,6 +1000,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, ocr_lang: str = "deu+eng", vision_ocr_url: str | None = None, vision_ocr_model: str | None = None, + vision_ocr_fallback_url: str | None = None, + vision_ocr_fallback_model: str | None = None, vision_ocr_timeout: float = 150.0, progress: Any = None) -> dict[str, Any]: """Run the full extraction for one PDF, writing page renders and images. @@ -917,13 +1009,17 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, When vision_ocr_url/model are given, a page whose text needs OCR (or whose already-extracted text scores as mostly garbage, native or tesseract alike -- see ocr_quality_score) is escalated to that engine - first, with tesseract only as the fallback if the call fails or the - engine is unreachable. Confirmed live against ocr-baidu-unlimited-ocr: - markedly more accurate transcription of a scanned German magazine than - tesseract, and it labels running headers/footers/page numbers directly - (see build_vision_blocks), which is the dominant source of the - over-fragmentation cluster_running_furniture() otherwise has to recover - after the fact from noisy OCR variants of the same repeating line. + first, with a second vision engine (vision_ocr_fallback_*) tried next if + given, and tesseract only as the last resort if every vision call fails + or every engine is unreachable. Confirmed live against + ocr-baidu-unlimited-ocr: markedly more accurate transcription of a + scanned German magazine than tesseract, and it labels running headers/ + footers/page numbers directly (see build_vision_blocks), which is the + dominant source of the over-fragmentation cluster_running_furniture() + otherwise has to recover after the fact from noisy OCR variants of the + same repeating line. KDL Frontier Parser nano, tested as the fallback + engine, answers faster but returns plain text with no layout coordinates + at all -- handled by _whole_page_region as one ambiguous full-page block. Returns the layout document: every block on every page, classified and in reading order, plus the raw contents-page candidates. @@ -933,6 +1029,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, images_dir = out_dir / "images" pages_dir.mkdir(parents=True, exist_ok=True) images_dir.mkdir(parents=True, exist_ok=True) + vision_engines = [(vision_ocr_url, vision_ocr_model), + (vision_ocr_fallback_url, vision_ocr_fallback_model)] vision_enabled = bool(vision_ocr_url and vision_ocr_model) vision_page_count = 0 @@ -951,14 +1049,27 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, pix.save(str(png)) image_boxes: list[list[float]] = [] + page_images: list[dict[str, Any]] = [] for img_index, info in enumerate(page.get_images(full=True)): + rects: list[list[float]] = [] try: for r in page.get_image_rects(info[0]): - image_boxes.append([r.x0, r.y0, r.width, r.height]) + rects.append([r.x0, r.y0, r.width, r.height]) except Exception: continue - _extract_image(fitz, doc, info[0], images_dir, - f"p{page_no:04d}-{img_index:02d}") + image_boxes.extend(rects) + name = f"p{page_no:04d}-{img_index:02d}" + if _extract_image(fitz, doc, info[0], images_dir, name) and rects: + # A few embedded images repeat (a logo stamped on every + # page); each placement still gets its own bbox so a + # caption or figure position can be matched precisely. + for ri, box in enumerate(rects): + page_images.append({ + "id": name if ri == 0 else f"{name}-{ri}", + "page": page_no, + "bbox": box, + "filename": f"{name}.png", + }) blocks: list[Block] = [] data = page.get_text("dict") @@ -1004,8 +1115,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, vision_blocks: list[Block] | None = None if vision_enabled and needs_escalation: - regions = vision_ocr_page(png, vision_ocr_url, vision_ocr_model, - timeout=vision_ocr_timeout) + regions = vision_ocr_page_chain(png, vision_engines, + timeout=vision_ocr_timeout) if regions: scale_x = rect.width / max(1.0, _png_width(png, fitz)) vision_blocks = build_vision_blocks( @@ -1042,6 +1153,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, "height": rect.height, "blocks": blocks, "image_boxes": image_boxes, + "images": page_images, }) if progress: progress(page_no, doc.page_count) @@ -1066,6 +1178,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, mark_advert_pages(p["blocks"]) mark_pullquotes(p["blocks"]) assign_reading_order(p["blocks"], p["width"]) + mark_sidebars(p["blocks"], p["width"]) all_blocks.extend(p["blocks"]) # Cross-page, so it runs once over every block rather than per page @@ -1084,11 +1197,16 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110, "entries": parse_contents(lines), }) + all_images = [img for p in raw_pages for img in p["images"]] + for img in all_images: + img["reading_order"] = image_reading_position(img, all_blocks) + return { "page_count": doc.page_count, "pages": [{"page": p["page"], "width": p["width"], "height": p["height"]} for p in raw_pages], "blocks": [b.to_json() for b in all_blocks], + "images": all_images, "contents_candidates": contents_candidates, "vision_ocr_pages": vision_page_count, } @@ -1103,19 +1221,87 @@ def _png_width(png: Path, fitz) -> float: return 1.0 -def _extract_image(fitz, doc, xref: int, images_dir: Path, name: str) -> None: - """Save one embedded image, skipping the tiny rules and spacers.""" +def _extract_image(fitz, doc, xref: int, images_dir: Path, name: str) -> bool: + """Save one embedded image, skipping the tiny rules and spacers. + + Returns whether a file now exists at that name, so the caller can decide + whether to record it as a real figure -- without this, a page's rules and + spacers were silently skipped on disk but still implicitly "existed" as + far as anything downstream knew, since nothing checked. + """ try: pix = fitz.Pixmap(doc, xref) if pix.width < 64 or pix.height < 64: - return + return False if pix.n - pix.alpha >= 4: # CMYK needs converting first pix = fitz.Pixmap(fitz.csRGB, pix) target = images_dir / f"{name}.png" if not target.exists(): pix.save(str(target)) + return True except Exception: - return + return False + + +# ── Images: linking a figure to the article it illustrates ────────────────── + +def assign_images_to_articles(images: Sequence[dict[str, Any]], + blocks: Sequence[Block], + articles: Sequence[Article]) -> None: + """Attach each extracted figure to the article whose column it falls in. + + A figure belongs to whichever article occupies its page; when two + articles share a page (a feature ending, a new one starting lower down), + the figure goes to whichever article has the nearer block by vertical + position, so a photo is not shown under the wrong headline. Mutates + articles in place. + """ + by_page: dict[int, list[Article]] = {} + for art in articles: + for page in art.pages: + by_page.setdefault(page, []).append(art) + + blocks_by_page: dict[int, list[Block]] = {} + for b in blocks: + blocks_by_page.setdefault(b.page, []).append(b) + + for img in images: + candidates = by_page.get(img["page"], []) + if not candidates: + continue + if len(candidates) == 1: + candidates[0].images.append(img["id"]) + continue + # Several articles share this page: pick by nearest block vertically, + # since that is what a reader would associate the figure with. + img_y = img["bbox"][1] + best, best_dist = candidates[0], float("inf") + for art in candidates: + own_blocks = [b for b in blocks_by_page.get(img["page"], []) + if b.id in set(art.blocks)] + if not own_blocks: + continue + dist = min(abs(b.bbox[1] - img_y) for b in own_blocks) + if dist < best_dist: + best, best_dist = art, dist + best.images.append(img["id"]) + + +def image_reading_position(image: dict[str, Any], blocks: Sequence[Block]) -> int: + """Approximate reading_order for an image, for ordering it among sentences. + + Uses the nearest block on the same page whose top sits at or above the + image (so a photo is shown once the surrounding text has been reached, + not before), falling back to the closest block on the page if the image + sits above all of them. + """ + same_page = [b for b in blocks if b.page == image["page"]] + if not same_page: + return 0 + above = [b for b in same_page if b.bbox[1] <= image["bbox"][1]] + pool = above or same_page + nearest = min(pool, key=lambda b: abs(b.bbox[1] - image["bbox"][1])) + return nearest.reading_order # ── Persistence ────────────────────────────────────────────────────────────── @@ -1131,6 +1317,7 @@ def build_articles(layout: dict[str, Any]) -> list[dict[str, Any]]: skip = {c["page"] for c in candidates if c.get("score", 0) >= 0.45} articles = group_articles(blocks, entries, layout.get("page_count", 0), skip_pages=skip) + assign_images_to_articles(layout.get("images") or [], blocks, articles) out = [] for art in articles: rec = art.to_json() diff --git a/routes/magazine.py b/routes/magazine.py index c3ec06e..7ab4cc9 100644 --- a/routes/magazine.py +++ b/routes/magazine.py @@ -109,7 +109,9 @@ async def magazine_create(request: Request): # ── Stage A–C: ingestion ───────────────────────────────────────────────────── def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int, - vision_ocr_url: str | None, vision_ocr_model: str | None) -> None: + vision_ocr_url: str | None, vision_ocr_model: str | None, + vision_ocr_fallback_url: str | None = None, + vision_ocr_fallback_model: str | None = None) -> None: """Run the extraction pipeline; recorded as a restart-safe job.""" d = _LIB / doc_id pdf = d / "source.pdf" @@ -127,6 +129,8 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int, layout = mag.extract_pages(pdf, d, render_dpi=render_dpi, ocr_lang=ocr_lang, vision_ocr_url=vision_ocr_url, vision_ocr_model=vision_ocr_model, + vision_ocr_fallback_url=vision_ocr_fallback_url, + vision_ocr_fallback_model=vision_ocr_fallback_model, progress=progress) mag.save_layout(d, layout) @@ -183,6 +187,7 @@ async def magazine_ingest(doc_id: str, request: Request): # approved and only re-detection is wanted. settings = _load_settings() vision_ocr_url = vision_ocr_model = None + vision_ocr_fallback_url = vision_ocr_fallback_model = None want_vision = body.get("vision_ocr") if want_vision is None: want_vision = bool(settings.get("vision_ocr_enabled")) @@ -191,6 +196,12 @@ async def magazine_ingest(doc_id: str, request: Request): vision_ocr_model = str(body.get("vision_ocr_model") or settings.get("vision_ocr_model") or "").strip() if not (vision_ocr_url and vision_ocr_model): vision_ocr_url = vision_ocr_model = None + vision_ocr_fallback_url = str( + body.get("vision_ocr_fallback_url") or settings.get("vision_ocr_fallback_url") or "").strip() + vision_ocr_fallback_model = str( + body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip() + if not (vision_ocr_fallback_url and vision_ocr_fallback_model): + vision_ocr_fallback_url = vision_ocr_fallback_model = None job_id = uuid.uuid4().hex background_job_create(job_id, "magazine-ingest", fingerprint=doc_id) @@ -199,7 +210,8 @@ async def magazine_ingest(doc_id: str, request: Request): asyncio.create_task( asyncio.to_thread(_ingest_sync, doc_id, job_id, ocr_lang, render_dpi, - vision_ocr_url, vision_ocr_model) + vision_ocr_url, vision_ocr_model, + vision_ocr_fallback_url, vision_ocr_fallback_model) ) return {"job_id": job_id} diff --git a/static/js/ai-backends.js b/static/js/ai-backends.js index 48b5703..6958e05 100644 --- a/static/js/ai-backends.js +++ b/static/js/ai-backends.js @@ -427,6 +427,134 @@ })(); })(); + // ── Active OCR Engine (global picker, magazine ingestion) ─────────────── + // Mirrors initActiveLlmPanel above: an OCR engine is the same shape of + // problem (an OpenAI-compatible endpoint plus a model name), just applied + // to vision_ocr_url/vision_ocr_model/vision_ocr_enabled instead of + // llm_url/llm_model. Reuses the same model-list endpoint, since it just + // proxies to whatever URL it's given rather than assuming it's the LLM one. + (function initActiveOcrPanel() { + const urlInp = $('ocr-active-url'); + const modelSel = $('ocr-active-model'); + const refreshBtn = $('ocr-active-refresh'); + const statusEl = $('ocr-active-status'); + const enabledCb = $('ocr-active-enabled'); + if (!urlInp || !modelSel) return; + + function setStatus(msg, cls) { + if (!statusEl) return; + statusEl.textContent = msg || ''; + statusEl.className = 'llm-active-status' + (cls ? ' ' + cls : ''); + } + + async function fetchModels(url) { + url = (url || urlInp.value.trim()); + if (!url) { modelSel.innerHTML = ''; setStatus(''); return; } + const want = (_appSettings && _appSettings.vision_ocr_model) || ''; + setStatus('Loading models…'); + if (refreshBtn) refreshBtn.disabled = true; + try { + const d = await fetch('/api/conversation/llm-models?url=' + encodeURIComponent(url)).then(r => r.json()); + if (d.error) { + let msg = d.error; + try { const j = JSON.parse(d.error); msg = j.error?.message || j.detail || d.error; } catch (_) {} + if (msg.length > 50) msg = msg.substring(0, 47) + '...'; + modelSel.innerHTML = ''; + setStatus(msg, 'err'); + return; + } + const models = d.models || []; + modelSel.innerHTML = models.length + ? models.map(m => ``).join('') + : ''; + if (want && models.includes(want)) modelSel.value = want; + setStatus(models.length ? `${models.length} model${models.length === 1 ? '' : 's'}` : 'No models found', + models.length ? 'ok' : 'err'); + } catch (e) { + modelSel.innerHTML = ''; + setStatus('Could not reach endpoint', 'err'); + } finally { + if (refreshBtn) refreshBtn.disabled = false; + } + } + + function commitUrl() { + const url = urlInp.value.trim(); + if (!url) return; + applyAndSaveSettings({ vision_ocr_url: url }); + if (_appSettings) _appSettings.vision_ocr_url = url; + fetchModels(url); + } + urlInp.addEventListener('blur', commitUrl); + urlInp.addEventListener('keydown', e => { if (e.key === 'Enter') { e.preventDefault(); commitUrl(); } }); + urlInp.addEventListener('change', commitUrl); + + function refreshEndpointOptions(currentUrl) { + if (!urlInp || urlInp.tagName !== 'SELECT') return; + let html = ''; + const added = new Set(); + document.querySelectorAll('#dc-grid-ocr .llm-local-card').forEach(card => { + const uInp = card.querySelector('.dc-url-inp'); + if (!uInp) return; + const u = uInp.value.trim() || uInp.placeholder; + const name = card.querySelector('.llm-local-name')?.textContent.trim() || 'Local Engine'; + if (u && !added.has(u)) { added.add(u); html += ``; } + }); + const activeUrl = currentUrl || urlInp.value; + if (activeUrl && !added.has(activeUrl)) { + html += ``; + } + urlInp.innerHTML = html; + if (activeUrl) urlInp.value = activeUrl; + } + urlInp.addEventListener('focus', () => refreshEndpointOptions()); + + modelSel.addEventListener('change', () => { + applyAndSaveSettings({ vision_ocr_model: modelSel.value }); + if (_appSettings) _appSettings.vision_ocr_model = modelSel.value; + if (modelSel.value) setStatus('Active model: ' + modelSel.value, 'ok'); + }); + + enabledCb?.addEventListener('change', () => { + applyAndSaveSettings({ vision_ocr_enabled: enabledCb.checked }); + if (_appSettings) _appSettings.vision_ocr_enabled = enabledCb.checked; + }); + + refreshBtn?.addEventListener('click', () => fetchModels()); + + // "Use as OCR" card buttons (see DC_ROLE_USE.ocr in engines.js) sync this + // panel the same way "Use as LLM" syncs the Active Language Model one. + window.syncActiveOcrPanel = function (url) { + refreshEndpointOptions(url); + urlInp.value = url || ''; + fetchModels(url); + }; + + (async function () { + let url = (_appSettings && _appSettings.vision_ocr_url) || ''; + let enabled = (_appSettings && _appSettings.vision_ocr_enabled); + if (!url) { + try { + const s = await fetch('/api/settings').then(r => r.json()); + url = s.vision_ocr_url || ''; + enabled = s.vision_ocr_enabled; + if (!_appSettings) _appSettings = s; + else { + _appSettings.vision_ocr_url = s.vision_ocr_url || _appSettings.vision_ocr_url; + _appSettings.vision_ocr_model = s.vision_ocr_model || _appSettings.vision_ocr_model; + _appSettings.vision_ocr_enabled = s.vision_ocr_enabled; + } + } catch (_) {} + } + if (enabledCb) enabledCb.checked = enabled !== false; + setTimeout(() => { + refreshEndpointOptions(url); + urlInp.value = url || ''; + if (url) fetchModels(url); + }, 100); + })(); + })(); + // ── Inline STT quick-test panel ───────────────────────────────────────── (async function initSttTestPanel() { const panel = $('stt-test-panel'); diff --git a/static/js/engines.js b/static/js/engines.js index 7e08d66..6cdb071 100644 --- a/static/js/engines.js +++ b/static/js/engines.js @@ -285,6 +285,7 @@ const DC_ROLE_USE = { tts: { settingKey: 'tts_url', label: 'Use as TTS' }, stt: { settingKey: 'faster_whisper_url', label: 'Use as STT' }, llm: { settingKey: 'llm_url', label: 'Use as LLM' }, + ocr: { settingKey: 'vision_ocr_url', label: 'Use as OCR' }, }; window.openAddEngineDialog = function(role) { @@ -797,6 +798,7 @@ function renderLocalContainers(containers) { if (apiKey && settingKey === 'llm_url') patch.llm_api_key = apiKey; await window.applyAndSaveSettings(patch); if (settingKey === 'llm_url' && window.syncActiveLlmPanel) window.syncActiveLlmPanel(url, apiKey); + if (settingKey === 'vision_ocr_url' && window.syncActiveOcrPanel) window.syncActiveOcrPanel(url); btn.classList.add('active'); toast(`✓ ${btn.textContent.trim()} — now available in the backend dropdown`, 'success'); } diff --git a/static/js/magazine.js b/static/js/magazine.js index 43ec55a..76e7701 100644 --- a/static/js/magazine.js +++ b/static/js/magazine.js @@ -20,6 +20,7 @@ const state = { layout: null, // { page_count, pages[], blocks[] } article: null, // the article being read or reviewed sentences: [], // [{ text, blockId, page, bbox }] + images: [], // current article's figures, in reading order view: 'library', // library | contents | review | read pane: 'reflow', // reflow | pdf playing: false, @@ -28,6 +29,10 @@ const state = { audio: null, voices: [], podcast: null, + queue: [], // article ids for "play all" (empty = not queued) + queueIndex: -1, + driving: false, // large-control, minimal-interaction view + karaoke: true, // word-level highlighting (approximate, see estimateWordTimings) }; MAG.magazine = state; @@ -387,14 +392,60 @@ function splitSentences(blocks) { .map((s) => s.trim()).filter(Boolean); (parts.length ? parts : [b.text]).forEach((text) => { if (text && text.trim()) { - out.push({ text: text.trim(), blockId: b.id, page: b.page, bbox: b.bbox, type: b.type }); + out.push({ + text: text.trim(), blockId: b.id, page: b.page, bbox: b.bbox, type: b.type, + // Used to pick "the image reached so far" during playback (see + // currentImageIndex) -- the backend already computes this per + // block relative to the whole page, not just this article. + readingOrder: b.reading_order || 0, + words: text.trim().split(/\s+/).filter(Boolean), + }); } }); }); return out; } -async function magRead(articleId) { +// The current article's figures, in reading order, resolved from the full +// layout (fetched once) filtered to this article's own image ids. Kept +// separate from state.sentences because an image has no sentence of its +// own -- it is matched to "the most recent sentence at or before it". +function articleImages(article) { + const ids = new Set(article.images || []); + return (state.layout.images || []) + .filter((img) => ids.has(img.id)) + .sort((a, b) => (a.reading_order || 0) - (b.reading_order || 0)); +} + +// The image to show once sentence `i` has been reached: the last image +// whose position in the page's reading order is at or before that +// sentence's own block. Falls back to the first image before any text has +// caught up to it, so a lead photo is not left blank at the very start. +function currentImageIndex(i) { + if (!state.images.length) return -1; + const order = (state.sentences[i] || {}).readingOrder || 0; + let found = -1; + for (let k = 0; k < state.images.length; k++) { + if ((state.images[k].reading_order || 0) <= order) found = k; + else break; + } + return found === -1 ? 0 : found; +} + +function renderCurrentImage(i) { + const host = $('mag-read-image'); + if (!host) return; + const idx = currentImageIndex(i); + if (idx < 0) { host.hidden = true; return; } + const img = state.images[idx]; + if (state._shownImageId === img.id) { host.hidden = false; return; } + state._shownImageId = img.id; + host.hidden = false; + host.innerHTML = 'Figure from page ' + img.page + ''; +} + +async function magRead(articleId, opts) { const a = state.articles.find((x) => x.id === articleId); if (!a) return; state.article = a; @@ -406,7 +457,10 @@ async function magRead(articleId) { } catch (e) { return say('Could not load article text: ' + e.message, 'err'); } - const saved = (state.doc.progress || {})[a.id]; + state.images = articleImages(a); + state._shownImageId = null; + const forceStart = opts && opts.fromStart; + const saved = !forceStart && (state.doc.progress || {})[a.id]; state.idx = saved && saved.sentence < state.sentences.length ? saved.sentence : 0; magShow('read'); renderArticle(); @@ -426,8 +480,13 @@ function renderArticle() { : s.type === 'byline' ? 'p class="mag-byline"' : s.type === 'caption' ? 'p class="mag-caption"' : 'span'; const open = tag.split(' ')[0]; + // Each word gets its own span so playback can highlight it individually + // (see estimateWordTimings) -- sentence-level highlighting alone doesn't + // need this, but word-level does and this markup serves both. + const wordsHtml = s.words.map((w, j) => + '' + esc(w) + '').join(' '); return '<' + tag + ' class="mag-s' + (open === 'span' ? '' : '') + '" data-i="' + i + '" ' + - 'onclick="magSeek(' + i + ')">' + esc(s.text) + ' '; + 'onclick="magSeek(' + i + ')">' + wordsHtml + ' '; }).join(''); highlight(state.idx); } @@ -436,13 +495,54 @@ function highlight(i) { document.querySelectorAll('#mag-article [data-i]').forEach((el) => { el.classList.toggle('is-current', Number(el.dataset.i) === i); }); + clearWordHighlight(); const el = document.querySelector('#mag-article [data-i="' + i + '"]'); if (el && !prefersReducedMotion()) el.scrollIntoView({ block: 'center', behavior: 'smooth' }); else if (el) el.scrollIntoView({ block: 'center' }); const s = state.sentences[i]; if (s && state.pane === 'pdf') showPage(s.page, s.bbox); + renderCurrentImage(i); const counter = $('mag-read-progress'); if (counter) counter.textContent = (i + 1) + ' / ' + state.sentences.length; + const qText = state.queue.length ? 'Article ' + (state.queueIndex + 1) + ' / ' + state.queue.length : ''; + ['mag-queue-progress', 'mag-queue-progress-2'].forEach((id) => { + const el = $(id); + if (el) el.textContent = qText; + }); +} + +function clearWordHighlight() { + document.querySelectorAll('#mag-article .mag-w.is-current').forEach((el) => el.classList.remove('is-current')); +} + +// ── Word-level highlighting (approximate) ──────────────────────────────── +// No forced-alignment or TTS timestamp source exists in this pipeline, so +// per-word timing is ESTIMATED, not measured: each word's share of the +// sentence's real (measured) audio duration is proportional to its own +// character length, with a small fixed share added per word so short words +// are not given an unrealistically instant duration. This is a common +// karaoke approximation, not true alignment -- it can drift on sentences +// with heavy punctuation-driven pauses. +function estimateWordTimings(words, totalSeconds) { + if (!words.length || !totalSeconds) return []; + const FIXED = 0.6; // arbitrary units, shifts weight toward evening out short words + const weights = words.map((w) => Math.max(1, w.replace(/[.,!?;:"'()«»„"]/g, '').length) + FIXED); + const total = weights.reduce((a, b) => a + b, 0); + let t = 0; + return weights.map((w) => { + const dur = totalSeconds * (w / total); + const start = t; + t += dur; + return { start, end: t }; + }); +} + +function highlightWord(sentenceIdx, wordIdx) { + const sentenceEl = document.querySelector('#mag-article [data-i="' + sentenceIdx + '"]'); + if (!sentenceEl) return; + sentenceEl.querySelectorAll('.mag-w').forEach((el) => { + el.classList.toggle('is-current', Number(el.dataset.w) === wordIdx); + }); } function prefersReducedMotion() { @@ -508,46 +608,104 @@ async function magPlay() { state.playing = true; state.cancel = false; syncPlayButton(); + let finishedArticle = false; try { while (state.idx < state.sentences.length && !state.cancel) { highlight(state.idx); - const text = state.sentences[state.idx].text; + const sentence = state.sentences[state.idx]; let blob; try { - blob = await fetchTtsPreviewBlob(voice, text, 'mp3'); + blob = await fetchTtsPreviewBlob(voice, sentence.text, 'mp3'); } catch (e) { say('Synthesis failed: ' + e.message, 'err'); break; } if (state.cancel) break; - await playBlob(blob); + await playBlob(blob, sentence); if (state.cancel) break; state.idx += 1; saveProgress(); } + finishedArticle = state.idx >= state.sentences.length && !state.cancel; } finally { state.playing = false; state.cancel = false; syncPlayButton(); - if (state.idx >= state.sentences.length) { - state.idx = state.sentences.length - 1; + } + if (finishedArticle) { + state.idx = state.sentences.length - 1; + if (state.queue.length && state.queueIndex < state.queue.length - 1) { + // Play-all: move straight to the next approved article rather than + // stopping -- the whole point of a queue is not touching the screen + // between articles. + state.queueIndex += 1; + await magRead(state.queue[state.queueIndex], { fromStart: true }); + magPlay(); + } else { + state.queue = []; + state.queueIndex = -1; say('Article finished', 'ok'); } } } -function playBlob(blob) { +function playBlob(blob, sentence) { return new Promise((resolve) => { const url = URL.createObjectURL(blob); const audio = new Audio(url); state.audio = audio; - const done = () => { URL.revokeObjectURL(url); state.audio = null; resolve(); }; + let timings = null; + const onTimeUpdate = () => { + if (!state.karaoke || !sentence || !sentence.words.length) return; + if (!timings) { + if (!audio.duration || !isFinite(audio.duration)) return; + timings = estimateWordTimings(sentence.words, audio.duration); + } + const t = audio.currentTime; + const wi = timings.findIndex((w) => t >= w.start && t < w.end); + if (wi !== -1) highlightWord(state.idx, wi); + }; + if (state.karaoke) audio.addEventListener('timeupdate', onTimeUpdate); + const done = () => { + audio.removeEventListener('timeupdate', onTimeUpdate); + URL.revokeObjectURL(url); + state.audio = null; + resolve(); + }; audio.onended = done; audio.onerror = done; audio.play().catch(done); }); } +// ── Play all approved articles in sequence ────────────────────────────── +async function magPlayAll() { + const ids = state.articles.filter((a) => a.status === 'approved').map((a) => a.id); + if (!ids.length) return say('Approve at least one article first', 'err'); + state.queue = ids; + state.queueIndex = 0; + await magRead(ids[0], { fromStart: true }); + magPlay(); +} + +function magStopQueue() { + state.queue = []; + state.queueIndex = -1; + ['mag-queue-progress', 'mag-queue-progress-2'].forEach((id) => { + const el = $(id); + if (el) el.textContent = ''; + }); +} + +async function magQueueNext() { + if (!state.queue.length || state.queueIndex >= state.queue.length - 1) return; + const wasPlaying = state.playing; + if (wasPlaying) magPause(); + state.queueIndex += 1; + await magRead(state.queue[state.queueIndex], { fromStart: true }); + if (wasPlaying) magPlay(); +} + function magPause() { state.cancel = true; if (state.audio) { try { state.audio.pause(); } catch (_) {} } @@ -557,11 +715,13 @@ function magPause() { function syncPlayButton() { const btn = $('mag-play'); - if (!btn) return; - btn.innerHTML = state.playing - ? ' Pause' - : ' Listen'; - btn.setAttribute('aria-label', state.playing ? 'Pause narration' : 'Play narration'); + if (btn) { + btn.innerHTML = state.playing + ? ' Pause' + : ' Listen'; + btn.setAttribute('aria-label', state.playing ? 'Pause narration' : 'Play narration'); + } + if (state.driving) syncDrivingPlayButton(); } let _saveTimer = null; @@ -707,10 +867,45 @@ function magShow(view) { function magBackToContents() { magPause(); + magStopQueue(); + state.driving = false; magShow('contents'); magRefreshArticles(); } +// ── Driving mode ────────────────────────────────────────────────────────── +// Large, minimal-interaction controls for listening without looking at the +// screen (see Manual.md's planned driving-mode behaviour). Reuses whatever +// is already playing/queued rather than starting its own playback path -- +// entering driving mode does not interrupt an article already in progress. +function magToggleDriving() { + state.driving = !state.driving; + const overlay = $('mag-driving'); + if (overlay) overlay.hidden = !state.driving; + if (state.driving) renderDriving(); +} + +function renderDriving() { + const a = state.article; + if (!a) return; + const titleEl = $('mag-driving-title'); + if (titleEl) titleEl.textContent = a.title || 'Untitled'; + const byEl = $('mag-driving-by'); + if (byEl) byEl.textContent = a.author ? 'By ' + a.author : ''; + syncDrivingPlayButton(); + const qEl = $('mag-driving-queue'); + if (qEl) qEl.hidden = !state.queue.length; +} + +function syncDrivingPlayButton() { + const btn = $('mag-driving-play'); + if (!btn) return; + btn.innerHTML = state.playing + ? '' + : ''; + btn.setAttribute('aria-label', state.playing ? 'Pause' : 'Play'); +} + // ── Boot ───────────────────────────────────────────────────────────────────── function magInit() { @@ -736,6 +931,10 @@ root.magShow = magShow; root.magBackToContents = magBackToContents; root.magGeneratePodcast = magGeneratePodcast; root.magPlayPodcast = magPlayPodcast; +root.magPlayAll = magPlayAll; +root.magQueueNext = magQueueNext; +root.magStopQueue = magStopQueue; +root.magToggleDriving = magToggleDriving; root.magInit = magInit; MAG.magazineApi = { load: magLoadLibrary, refresh: magRefreshArticles, splitSentences, fmtPages }; diff --git a/static/js/utils.js b/static/js/utils.js index bf6152f..e66b706 100644 --- a/static/js/utils.js +++ b/static/js/utils.js @@ -1112,11 +1112,18 @@ function _wfUpdateHeaderNav(anyState, curIdx) { const nextStep = curIdx >= 0 ? _wfNeighbor(curIdx, 1) : null; const curStep = curIdx >= 0 ? WF_STEPS[curIdx] : null; document.querySelectorAll('.section-head').forEach(head => { - // Studio has its own self-contained 4-phase nav (studio.js) — this - // stepper's Previous/Next buttons are hardcoded to the old s-reader/ - // s-library/s-rehearser sections and don't know about Studio's phases, - // so injecting them here just breaks navigation when clicked. - if (head.closest('#s-caststudio')) { + // This stepper's Previous/Next buttons are hardcoded to the audiobook + // pipeline's three destinations (s-reader, s-library, s-rehearser) — an + // allowlist, not a blocklist, since every OTHER section (Studio has its + // own 4-phase nav; Engines, Magazines, Settings, etc. have nothing to do + // with this pipeline at all) would otherwise need to be named here one + // at a time as each one was noticed showing a "Step 1/8" bar and a + // "Cast Audiobook" button that make no sense on that page. Confirmed + // live: this exact bleed-through appeared on Engines (LLM/STT/TTS/OCR/ + // Image Generation all share one .section-head) before this became an + // allowlist. + const onPipelinePage = !!head.closest('#s-reader, #s-library, #s-rehearser'); + if (!onPipelinePage) { const staleNav = head.querySelector('.wf-header-nav'); if (staleNav) staleNav.hidden = true; const staleHint = head.nextElementSibling?.classList?.contains('wf-step-hint') ? head.nextElementSibling : null; diff --git a/static/sections/s-llms.html b/static/sections/s-llms.html index b4b95c5..613c665 100644 --- a/static/sections/s-llms.html +++ b/static/sections/s-llms.html @@ -807,6 +807,42 @@ +
+ + +
+
+ +
+

Active OCR Engine

+

Used by Magazine ingestion when a page needs OCR, or when its existing text scores poorly. Tesseract (bundled, no setup) is always the fallback if this is unreachable.

+
+
+
+ + + + + + +
+ +
+ +
+

Local

+ +
+
+ +
Two engines were confirmed live against a real scanned magazine: Baidu Unlimited-OCR gives the most accurate text and also labels running headers/footers/page numbers/titles directly (best default). KDL Frontier Parser nano answers faster but returns plain text with no layout coordinates or labels — a usable fallback, not a first choice.
+ +
+
diff --git a/static/sections/s-magazine.html b/static/sections/s-magazine.html index 769acac..7c03eab 100644 --- a/static/sections/s-magazine.html +++ b/static/sections/s-magazine.html @@ -39,6 +39,9 @@

Magazine

+ @@ -70,6 +73,9 @@
+
@@ -81,6 +87,11 @@ Listen + +
@@ -90,6 +101,7 @@
+
@@ -117,3 +129,28 @@
+ + + + diff --git a/static/style.css b/static/style.css index ae77bc1..6397741 100644 --- a/static/style.css +++ b/static/style.css @@ -4816,6 +4816,17 @@ input[list="lang-codes"] { /* The reading highlight has to survive both themes and colour-blindness, so it carries a left rule as well as a tint rather than relying on hue alone. */ .mag-article [data-i].is-current { background: rgba(37,99,235,.16); box-shadow: -3px 0 0 0 var(--accent); } +/* Word-level highlighting is an ESTIMATE (see estimateWordTimings in + magazine.js) -- a distinct, lighter treatment from the sentence highlight + above keeps that difference legible rather than implying frame-accurate + timing. */ +.mag-w.is-current { background: var(--accent); color: var(--on-accent); border-radius: 2px; } + +.mag-karaoke-toggle { display: inline-flex; align-items: center; gap: 6px; font-size: 13px; color: var(--subtext); } + +.mag-read-image { max-width: 68ch; margin: 0 auto 18px; } +.mag-read-image img { width: 100%; height: auto; max-height: 360px; object-fit: contain; + border-radius: var(--radius); border: 1px solid var(--border); background: var(--panel); display: block; } .mag-page-wrap { position: relative; max-width: 900px; margin: 0 auto; } .mag-page-wrap img { width: 100%; height: auto; display: block; border: 1px solid var(--border); border-radius: var(--radius); } @@ -4831,8 +4842,31 @@ input[list="lang-codes"] { .mag-turns li { margin-bottom: 8px; border-radius: 3px; padding: 2px 4px; } .mag-turns li.is-current { background: rgba(37,99,235,.16); box-shadow: -3px 0 0 0 var(--accent); } +/* ── Driving mode ────────────────────────────────────────────────────────── + Full-screen, minimal-interaction overlay: two oversized controls, large + type, nothing else competing for attention. Reuses whatever is already + playing/queued -- it is a view, not a separate playback path. */ +.mag-driving { position: fixed; inset: 0; z-index: 500; background: var(--bg); + display: flex; align-items: center; justify-content: center; } +.mag-driving-exit { position: absolute; top: 20px; right: 20px; width: 52px; height: 52px; + border-radius: 50%; border: 1px solid var(--border); background: var(--surface); color: var(--text); + font-size: 22px; display: flex; align-items: center; justify-content: center; cursor: pointer; } +.mag-driving-exit:focus-visible { outline: 3px solid var(--accent); outline-offset: 2px; } +.mag-driving-body { text-align: center; max-width: 90vw; } +.mag-driving-body h2 { font-size: clamp(28px, 5vw, 48px); margin: 0 0 8px; } +.mag-driving-queue { margin-bottom: 20px; font-size: 18px; } +.mag-driving-controls { display: flex; align-items: center; justify-content: center; gap: 32px; margin-top: 32px; } +.mag-driving-btn { width: 96px; height: 96px; border-radius: 50%; border: none; + background: var(--panel); color: var(--text); font-size: 36px; cursor: pointer; + display: flex; align-items: center; justify-content: center; } +.mag-driving-btn.mag-driving-play { width: 140px; height: 140px; font-size: 56px; + background: var(--accent); color: var(--on-accent); } +.mag-driving-btn:focus-visible { outline: 3px solid var(--accent); outline-offset: 3px; } +.mag-driving-btn:active { transform: scale(0.96); } + @media (prefers-reduced-motion: reduce) { .mag-page-box { transition: none; } + .mag-driving-btn:active { transform: none; } } @media (max-width: 860px) { .mag-new-grid, .mag-review-head { grid-template-columns: 1fr; } @@ -4841,4 +4875,7 @@ input[list="lang-codes"] { .mag-block { grid-template-columns: auto 1fr; } .mag-block-type { display: none; } .mag-article { font-size: 18px; } + .mag-driving-controls { gap: 20px; } + .mag-driving-btn { width: 76px; height: 76px; font-size: 28px; } + .mag-driving-btn.mag-driving-play { width: 110px; height: 110px; font-size: 44px; } } diff --git a/tests/test_magazine.py b/tests/test_magazine.py index 3d2b2f2..551d34e 100644 --- a/tests/test_magazine.py +++ b/tests/test_magazine.py @@ -19,6 +19,7 @@ from core.magazine import ( ocr_page, cluster_running_furniture, looks_like_real_title, + Article, BLOCK_AD, BLOCK_BODY, BLOCK_BYLINE, @@ -26,9 +27,11 @@ from core.magazine import ( BLOCK_DECK, BLOCK_FURNITURE, BLOCK_PULLQUOTE, + BLOCK_SIDEBAR, BLOCK_TITLE, Block, article_text, + assign_images_to_articles, assign_reading_order, body_font_size, build_articles, @@ -39,16 +42,19 @@ from core.magazine import ( estimate_listen_seconds, find_continuation, find_repeated_lines, + image_reading_position, strip_continuation_markers, group_articles, looks_like_byline, mark_advert_pages, mark_pullquotes, + mark_sidebars, ocr_quality_score, parse_contents, parse_vision_ocr_regions, score_contents_page, vision_ocr_page, + vision_ocr_page_chain, ) PAGE_W, PAGE_H = 595.0, 842.0 # A4 in points @@ -831,5 +837,187 @@ class BuildArticlesTests(unittest.TestCase): self.assertGreater(arts[0]["estimated_seconds"], 0) +class SidebarDetectionTests(unittest.TestCase): + """BLOCK_SIDEBAR is geometry-only, like the rest of this module: a boxed + insert is reliably narrower than the page's regular column grid. This is + a heuristic, not a certainty -- review still shows every sidebar block + and lets it be ticked back in with one click.""" + + def test_a_narrow_third_column_is_marked_a_sidebar(self): + blocks = [ + block("main1", 1, 50, 100, 300, 400, "Main body copy. " * 40), + block("side1", 1, 400, 100, 100, 400, "A boxed insert. " * 10), + ] + assign_reading_order(blocks, PAGE_W) + mark_sidebars(blocks, PAGE_W) + self.assertEqual(next(b for b in blocks if b.id == "side1").type, BLOCK_SIDEBAR) + self.assertEqual(next(b for b in blocks if b.id == "main1").type, BLOCK_BODY) + + def test_two_similarly_wide_columns_are_left_alone(self): + """A normal two-column layout must not be mistaken for a sidebar.""" + blocks = [ + block("l", 1, 50, 100, 200, 400, "Left column. " * 30), + block("r", 1, 280, 100, 200, 400, "Right column. " * 30), + ] + assign_reading_order(blocks, PAGE_W) + mark_sidebars(blocks, PAGE_W) + self.assertTrue(all(b.type == BLOCK_BODY for b in blocks)) + + def test_a_single_column_page_has_no_sidebar_candidates(self): + blocks = [block("a", 1, 50, 100, 480, 400, "Full width copy. " * 40)] + assign_reading_order(blocks, PAGE_W) + mark_sidebars(blocks, PAGE_W) + self.assertEqual(blocks[0].type, BLOCK_BODY) + + def test_a_title_or_caption_in_the_narrow_column_is_untouched(self): + """Only BLOCK_BODY is reclassified -- a caption already correctly + typed should not be relabelled just for sitting in a narrow column.""" + blocks = [ + block("main1", 1, 50, 100, 300, 400, "Main body copy. " * 40), + block("cap", 1, 400, 100, 100, 30, "A caption.", btype=BLOCK_CAPTION), + ] + assign_reading_order(blocks, PAGE_W) + mark_sidebars(blocks, PAGE_W) + self.assertEqual(next(b for b in blocks if b.id == "cap").type, BLOCK_CAPTION) + + def test_sidebar_text_is_excluded_from_narration_by_default(self): + """End-to-end: a detected sidebar must not be swept into the + article's narrated blocks the way an advert page already is not.""" + blocks = [ + block("t", 18, 50, 60, 470, 50, "The Future of Batteries", + size=26.0, btype=BLOCK_TITLE), + block("main1", 18, 50, 150, 300, 400, "Main body copy. " * 40), + block("side1", 18, 400, 150, 100, 400, "Battery myths. " * 20), + ] + assign_reading_order(blocks, PAGE_W) + mark_sidebars(blocks, PAGE_W) + arts = group_articles(blocks, [{"title": "The Future of Batteries", + "page": 18, "section": ""}], page_count=20) + self.assertNotIn("side1", arts[0].blocks) + self.assertIn("main1", arts[0].blocks) + + +class ImageAssignmentTests(unittest.TestCase): + """Extracted figures are useless until linked to the article they + illustrate -- assign_images_to_articles is that link.""" + + def test_an_image_on_a_single_article_page_is_assigned_to_it(self): + art = Article(id="a1", title="T", pages=[5], blocks=["b1"]) + blocks = [block("b1", 5, 50, 100, 300, 50, "Some text")] + images = [{"id": "img1", "page": 5, "bbox": [50, 200, 200, 150]}] + assign_images_to_articles(images, blocks, [art]) + self.assertEqual(art.images, ["img1"]) + + def test_an_image_on_a_page_with_no_matching_article_is_skipped(self): + art = Article(id="a1", title="T", pages=[5], blocks=[]) + images = [{"id": "img1", "page": 9, "bbox": [50, 200, 200, 150]}] + assign_images_to_articles(images, [], [art]) + self.assertEqual(art.images, []) + + def test_shared_page_image_goes_to_the_nearer_articles_blocks(self): + """Two articles land on the same page (one ending, one starting) -- + the figure must follow whichever article it actually sits beside.""" + art_a = Article(id="a1", title="First", pages=[6], blocks=["ba"]) + art_b = Article(id="a2", title="Second", pages=[6], blocks=["bb"]) + blocks = [ + block("ba", 6, 50, 50, 300, 100, "End of the first article."), + block("bb", 6, 50, 500, 300, 100, "Start of the second article."), + ] + # The image sits right next to article B's block (y=520 vs y=500). + images = [{"id": "img1", "page": 6, "bbox": [50, 520, 200, 150]}] + assign_images_to_articles(images, blocks, [art_a, art_b]) + self.assertEqual(art_a.images, []) + self.assertEqual(art_b.images, ["img1"]) + + +class ImageReadingPositionTests(unittest.TestCase): + def test_image_takes_the_position_of_the_nearest_block_above_it(self): + blocks = [ + block("b1", 1, 50, 50, 300, 100, "First paragraph."), + block("b2", 1, 50, 300, 300, 100, "Second paragraph."), + ] + blocks[0].reading_order = 0 + blocks[1].reading_order = 1 + image = {"page": 1, "bbox": [50, 320, 200, 150]} + self.assertEqual(image_reading_position(image, blocks), 1) + + def test_image_above_all_text_falls_back_to_the_nearest_block(self): + blocks = [block("b1", 1, 50, 300, 300, 100, "Paragraph.")] + blocks[0].reading_order = 5 + image = {"page": 1, "bbox": [50, 10, 200, 150]} + self.assertEqual(image_reading_position(image, blocks), 5) + + def test_no_blocks_on_the_page_returns_zero(self): + self.assertEqual(image_reading_position({"page": 9, "bbox": [0, 0, 1, 1]}, []), 0) + + +class WholePageVisionFallbackTests(unittest.TestCase): + """KDL Frontier Parser nano, confirmed live, answers with plain text and + no coordinate markers at all -- unlike ocr-baidu-unlimited-ocr's labeled + regions. _whole_page_region is what lets that still produce something + usable rather than being discarded outright.""" + + def test_plain_text_with_no_markers_becomes_one_page_sized_region(self): + from core.magazine import _whole_page_region + with mock.patch("PIL.Image.open") as mopen: + mopen.return_value.__enter__.return_value.size = (897, 1049) + regions = _whole_page_region( + "Battery technology is entering a new phase. " * 5, + pathlib.Path("/fake.png")) + self.assertEqual(len(regions), 1) + self.assertEqual(regions[0]["label"], "text") + self.assertEqual(regions[0]["bbox"], [0.0, 0.0, 897.0, 1049.0]) + + def test_garbage_text_is_rejected_not_wrapped(self): + from core.magazine import _whole_page_region + self.assertIsNone(_whole_page_region(". | @)X", pathlib.Path("/fake.png"))) + + def test_too_short_text_is_rejected(self): + from core.magazine import _whole_page_region + self.assertIsNone(_whole_page_region("Hi.", pathlib.Path("/fake.png"))) + + def test_empty_text_is_rejected(self): + from core.magazine import _whole_page_region + self.assertIsNone(_whole_page_region("", pathlib.Path("/fake.png"))) + + +class VisionEngineChainTests(unittest.TestCase): + """The primary engine (labeled regions, best quality) is tried first; a + differently-shaped secondary is a real fallback, not just tesseract.""" + + def test_primary_success_short_circuits_the_fallback(self): + with mock.patch("core.magazine.vision_ocr_page") as mocked: + mocked.return_value = [{"label": "text", "bbox": [0, 0, 1, 1], "text": "x"}] + out = vision_ocr_page_chain(pathlib.Path("/fake.png"), + [("http://a", "m1"), ("http://b", "m2")]) + self.assertEqual(mocked.call_count, 1) + self.assertIsNotNone(out) + + def test_primary_failure_falls_through_to_the_secondary(self): + with mock.patch("core.magazine.vision_ocr_page") as mocked: + mocked.side_effect = [None, [{"label": "text", "bbox": [0, 0, 1, 1], "text": "x"}]] + out = vision_ocr_page_chain(pathlib.Path("/fake.png"), + [("http://a", "m1"), ("http://b", "m2")]) + self.assertEqual(mocked.call_count, 2) + self.assertIsNotNone(out) + + def test_every_engine_failing_returns_none(self): + with mock.patch("core.magazine.vision_ocr_page", return_value=None): + out = vision_ocr_page_chain(pathlib.Path("/fake.png"), + [("http://a", "m1"), ("http://b", "m2")]) + self.assertIsNone(out) + + def test_blank_engine_entries_are_skipped_without_calling_out(self): + with mock.patch("core.magazine.vision_ocr_page") as mocked: + mocked.return_value = [{"label": "text", "bbox": [0, 0, 1, 1], "text": "x"}] + vision_ocr_page_chain(pathlib.Path("/fake.png"), + [("", ""), ("http://b", "m2")]) + self.assertEqual(mocked.call_count, 1) + self.assertEqual(mocked.call_args[0][1], "http://b") + + def test_empty_engine_list_returns_none(self): + self.assertIsNone(vision_ocr_page_chain(pathlib.Path("/fake.png"), [])) + + if __name__ == "__main__": unittest.main()