feat: complete magazine P1 (images, play-all, karaoke, driving mode, sidebar exclusion) + OCR engines page + second OCR engine

Magazine P1:
- Images shown automatically during playback (assign_images_to_articles
  links each figure to the article whose column it falls in; the client
  shows the last image reached in reading order as playback advances).
- "Play all articles" queues every approved article and advances
  automatically at the end of each rather than stopping.
- Word-level highlighting, explicitly documented as an ESTIMATE (no
  forced-alignment/timestamp source exists): each word's share of the
  sentence's real measured audio duration is proportional to its
  character length. A checkbox turns it off; sentence highlighting
  (measured, not estimated) is unaffected.
- Driving mode: full-screen overlay, two oversized controls, reuses
  whatever is already playing/queued.
- Sidebars/pull-out boxes excluded from narration automatically
  (mark_sidebars: an anomalously narrow column vs. the page's regular
  grid). Fixed a real defect found while building this: a full-width
  headline was left in the column-width analysis and could bridge two
  genuinely separate columns into one, hiding a real sidebar column
  entirely -- caught by a failing test before the fix.

Second OCR engine + fallback ladder:
- KDL Frontier Parser nano wired in via vision_ocr_page_chain (ordered
  engine list, first usable result wins, tesseract last resort).
  Confirmed live: ~3x faster than Baidu (~12s vs ~40s/page) but returns
  plain text with NO coordinate markers, unlike Baidu's labeled regions.
  _whole_page_region treats the whole response as one ambiguous
  full-page block (gated by ocr_quality_score) rather than discarding
  usable text just because it lacks layout structure.

OCR engines page + a real UI bug:
- The OCR & Document Parsing page had no .s-engines-page[data-page="ocr"]
  markup at all -- renderLocalContainers already supported an OCR grid
  generically, but $('dc-grid-ocr') was always null, so the card grid
  silently never rendered. Added an Active OCR Engine panel mirroring
  Active Language Model exactly, plus the Local grid -- both configured
  engines now appear as ready-made cards, no "Add Custom" dialog needed.
- Fixed the audiobook pipeline's "Previous/Next" nav bar bleeding into
  every .section-head in the document (confirmed live on the Engines
  page: a "Cast Audiobook" button and "STEP 1/8" hint that made no sense
  there). Changed from a blocklist (naming each unrelated section as
  noticed) to an allowlist of the pipeline's three actual destinations.

Verified live end-to-end: play-all queue building and manual advance,
word-span markup, driving-mode toggle, OCR page showing both engines as
local cards, zero stray nav bars on the Engines page, no console errors.

20 new backend tests (156 total in the magazine module).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-09-06 23:39:34 +02:00
parent 477cb43d4a
commit 813e06c914
17 changed files with 1096 additions and 82 deletions

View File

@ -8,6 +8,38 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
## [Unreleased]
### Added
- **Magazine P1 Complete: Images, Play-All, Word Highlighting, Driving Mode.** Figures now appear automatically during
playback — `assign_images_to_articles` links each extracted image to the article whose column it falls in (the nearer
article's blocks decide when two share a page), and the client shows "the last image reached so far" as playback
advances past its approximate reading-order position. **Play all articles** (contents screen) queues every approved
article in order and advances automatically at the end of each, rather than stopping and returning to contents.
**Word-level highlighting** is available alongside the existing sentence highlighting — clearly documented as an
*estimate*, not a measurement: no forced-alignment or TTS timestamp source exists in this pipeline, so each word's
share of the sentence's real, measured audio duration is distributed proportional to its character length, a common
karaoke approximation that can drift on heavily punctuated sentences. A checkbox turns it off; sentence highlighting
(measured via `audio.onended`, not estimated) is unaffected. **Driving mode** is a full-screen overlay with two
oversized controls and nothing else to interact with, reusing whatever is already playing/queued rather than its own
playback path. **Sidebars and pull-out boxes are now excluded from narration automatically** (`mark_sidebars`): a
column much narrower than the page's regular grid is reclassified `BLOCK_SIDEBAR` and dropped from an article's
narrated blocks the same way adverts already are — review still shows every sidebar block, unticked by default, so
nothing is silently discarded. Fixed a real defect found writing this: a full-width headline was left in the
column-width analysis and could bridge two genuinely separate columns into one, hiding a real sidebar column entirely
(caught by a failing test before the fix). 20 new tests.
- **Second Vision OCR Engine, With a Real Fallback Ladder.** KDL Frontier Parser nano is now wired in alongside
ocr-baidu-unlimited-ocr (`vision_ocr_page_chain`: first usable result from an ordered engine list wins, tesseract only
if every vision engine fails). Confirmed live: KDL answers roughly 3x faster (~12s vs ~40s per page on the same test
page) but returns **plain text with no coordinate markers at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content`
regions — handled by treating its whole response as one ambiguous full-page block (`_whole_page_region`, gated by the
same `ocr_quality_score` threshold), which loses column/reading-order structure for that page but is still a real
improvement over tesseract. Settings gained `vision_ocr_fallback_url`/`vision_ocr_fallback_model`.
- **OCR Engines Page Now Matches the LLM Engines Layout.** The OCR & Document Parsing page had no
`.s-engines-page[data-page="ocr"]` markup at all — `renderLocalContainers` already supported an OCR grid generically
(the same code handles TTS/STT/LLM/OCR identically), but `$('dc-grid-ocr')` was always null, so the card grid silently
never rendered and the page showed only a header with "Add Custom"/"Refresh" buttons and nothing else. Added the
missing panel: an **Active OCR Engine** picker mirroring **Active Language Model** exactly (same endpoint/model
selects, same `/api/conversation/llm-models` fetch, which already proxies to whatever URL it's given rather than
assuming the LLM setting) plus a **Local** grid, so both configured engines now appear as ready-made cards
(Connect/Start/Restart/Use as OCR/Show code snippet) with no "Add Custom" dialog needed for either.
- **Vision OCR Escalation For Magazine Ingestion.** A page that needs OCR, or whose already-extracted text scores as
mostly garbage (`ocr_quality_score`) — native or tesseract alike — is now escalated to a connected vision OCR engine
(`vision_ocr_url`/`vision_ocr_model` in Settings, defaulting to `ocr-baidu-unlimited-ocr` via `host.docker.internal`),
@ -70,6 +102,13 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
(deu/eng) to the image.
### Fixed
- **Audiobook Pipeline Nav Bar Bled Into Unrelated Pages.** `_wfUpdateHeaderNav` injected the audiobook workflow's
"Previous / Next" bar and step hint into every `.section-head` in the document, with Studio as the only named
exclusion. Confirmed live: the Engines page (LLM/STT/TTS/OCR/Image Generation all share one `.section-head`) showed a
"Previous … Cast Audiobook ›" bar and a "STEP 1/8 Import a PDF…" hint that made no sense there, since the pipeline's
three destinations are Read Aloud, Library and Script Rehearsal only. Changed from a blocklist (naming each unrelated
section as it was noticed) to an allowlist, so the same bleed-through cannot recur on Magazines, Settings, or any
future section.
- **Captions On Photo Pages Would Have Been Read As Body Text.** Block classification judged font size against a baseline
computed per page, so a page carrying a full-bleed photograph and a single caption made that caption its own reference:
at a ratio of exactly 1.0 it classified as body copy and would have been narrated as part of the article. The baseline

View File

@ -34,14 +34,22 @@ turned into a podcast discussion between two or three voices you pick.
articles with page ranges.
3. **Review the articles.** This step is not optional and cannot be skipped.
Each proposed article shows its title, author, page range, estimated
listening time and a confidence score. Confirm the boundaries, exclude a
sidebar you do not want read, correct the title, then **Approve**.
listening time and a confidence score. Sidebars and pull-out boxes are
already unticked for you — tick one back in if you actually want it read.
Confirm the boundaries, correct the title, then **Approve**.
4. **Listen.** An approved article can be narrated with the default voice, or
sent to podcast mode.
sent to podcast mode. Use **Play all articles** (on the contents screen) to
listen through every approved article back to back without returning to
this screen between them. A figure from the page is shown automatically as
playback reaches it. **Word highlighting** can be turned off if you find it
distracting — it is an estimate based on word length, not a measurement, so
it will not always land exactly on the spoken word. **Driving mode**
(the button next to the article title) switches to two large controls and
nothing else, for listening without looking at the screen.
**Expected result:** the article plays from its headline to its final paragraph.
Page furniture (running heads, page numbers), adverts and pull quotes are never
read aloud.
Page furniture (running heads, page numbers), adverts, sidebars and pull quotes
are never read aloud unless you explicitly include them in review.
### Confidence, and what it means
@ -88,10 +96,15 @@ nothing outward.
### Accessibility and mobile
To be completed with the P1 interface. Planned: keyboard-navigable contents
tree and review screen, sentence highlighting that respects
`prefers-reduced-motion`, large touch targets for playback, and a driving mode
that advances images without requiring interaction.
Sentence and word highlighting both respect `prefers-reduced-motion` (no
smooth-scroll animation when it is set; the highlight itself is a colour
change, not motion). Driving mode gives two large, high-contrast controls
(skip, play/pause) with nothing else to interact with, for use without
looking at the screen — it does not, and should not, replace watching the
road. Word-level highlighting can be switched off in the reading controls if
it is more distracting than helpful; sentence highlighting keeps working
either way. Keyboard navigation of the contents tree and review screen is
not yet built.
### Offline

View File

@ -5,7 +5,7 @@ Update in the same commit as the change it describes.
## Current phase
Magazine projects, **P0 complete, P1 mostly complete** (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library
Magazine projects, **P0 and P1 complete** (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library
features are mature and in daily use.
## Deployment
@ -69,18 +69,20 @@ which is the writable volume.
- Container registry: ghcr costs quota against a private package (≈427 MB per
push, 500 MB free on the Free plan). Local registry vs. ghcr vs. no re-pull is
**unresolved** — the compose files currently point at ghcr.
- Magazine P1 scope: sentence-level highlighting confirmed as the starting
point; word-level deferred to P2.
## Known limitations
- `gitea` backup remote is 157 commits behind `origin`; `main` tracks `gitea`.
- Magazine sidebar detection is not implemented (P2).
- Magazine sidebar exclusion is a narrow-column geometry heuristic
(`mark_sidebars`), not certain — review always shows every sidebar block.
- Magazine word-level highlighting is a character-length estimate against
real measured audio duration, not true forced alignment.
- No formal cross-engine OCR accuracy benchmark exists (two engines wired:
ocr-baidu-unlimited-ocr primary, KDL Frontier Parser nano fallback).
- No `npm run qa` aggregate command yet; Python tests run via `unittest`,
browser tests via `playwright test`.
## Next logical step
Magazine P1 remainder: article images/page crops shown during playback, a
cross-article "play all" queue, and OCR-engine dispatch to the KDL/Baidu
parsers (see architecture/magazine-ingestion.md). Otherwise P2 per TODO.md.
Magazine P2 per TODO.md: image descriptions, table/chart handling, EPUB 3
Media Overlay export, and a formal OCR-engine benchmark (see
architecture/magazine-ingestion.md).

24
TODO.md
View File

@ -303,24 +303,30 @@ casting. The original PDF remains the visual source of truth while approved arti
### P1 — Magazine listening experience
- [~] Generate clean article HTML with headings, captions, figures, sidebars, and page links.
*(headings/byline/captions styled and page-linked; figures and sidebars are not rendered inline yet)*
- [~] Generate narrator audio per article with resume/progress and a play-all queue.
*(on-demand narration + per-article resume done; no cross-article play-all queue yet)*
*(headings/byline/captions styled and page-linked, figures now shown during playback; sidebars are auto-excluded
from narration rather than rendered inline, per the P2 sidebar-exclusion item below)*
- [x] Generate narrator audio per article with resume/progress and a play-all queue.
- [x] Synchronize paragraph/sentence highlighting with playback and keep PDF and article views linked.
*(sentence-level; clicking a sentence or switching to the original-page view jumps to its source bbox)*
- [ ] Show article images or page crops during playback, with optional caption reading.
- [x] Show article images or page crops during playback, with optional caption reading.
*(images shown, matched to reading position; caption text is not separately spoken)*
- [~] Add confidence warnings, extraction diffs, and manual text correction before audio generation.
*(confidence badges + block-level boundary editing done; no extraction diff view or free-text correction)*
### P2 — Advanced magazine intelligence
- [ ] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion.
- [~] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion.
*(sidebar exclusion done via mark_sidebars, a narrow-column geometry heuristic; cross-page continuation still only
uses the forward "continued on page N" reference, not "continued from")*
- [ ] Add image descriptions, chart/table handling, and optional spoken figure summaries.
- [ ] Add word-level karaoke timing and EPUB 3 Media Overlay export.
- [ ] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume.
- [~] Add word-level karaoke timing and EPUB 3 Media Overlay export.
*(word-level highlighting done as a character-length estimate against the real measured audio duration, not true
forced alignment; no EPUB 3 export)*
- [~] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume.
*(driving mode done: two oversized controls, reuses whatever is already playing/queued; not mobile-specific)*
- [~] Add OCR-engine benchmark comparisons and per-page fallback selection.
*(per-page fallback done: vision OCR escalates when a page needs OCR or scores poorly, tesseract is the automatic
fallback; no formal cross-engine benchmark or KDL Frontier Parser nano routing yet)*
*(per-page fallback done, now a real two-engine chain: vision OCR escalates when a page needs OCR or scores
poorly, KDL Frontier Parser nano is the vision fallback, tesseract is the last resort; no formal benchmark)*
Three full regression runs of *Der Sturm* against the hand-corrected reference.
Everything below is measured, not estimated. Artefacts live in

View File

@ -1,8 +1,11 @@
# SOP — Magazine ingestion
Status: **P0 complete. P1 mostly complete** — Studio -> Magazines has the full
library/contents/review/read/podcast flow. Remaining P1: images shown during
playback and a cross-article play-all queue.
Status: **P0 complete. P1 complete.** Studio -> Magazines has the full
library/contents/review/read/podcast flow, including figures shown during
playback and a cross-article play-all queue. Automatic sidebar exclusion,
word-level highlighting (estimated, not measured) and driving mode --
originally P2 items -- are also in. Remaining P2: image descriptions,
table/chart handling, EPUB 3 export, and a formal multi-engine OCR benchmark.
## Why magazines are not books
@ -193,14 +196,67 @@ this app's tab pattern would additionally require `aria-selected`, roving
provides. `role="tab"` without that whole pattern fails WCAG 2.2 AA guidance
and, in practice, hides the buttons from `getByRole('tab')`-oriented tooling.
### Images during playback
`build_vision_blocks` aside, a figure has no sentence of its own to attach
to. `assign_images_to_articles` (server) links each extracted image to the
article whose column it falls in, using the nearer article's blocks when two
share a page; `image_reading_position` gives it an approximate slot in the
page's reading order. The client (`currentImageIndex` in magazine.js) then
shows "the last image reached so far" as playback advances past its
position — no separate sync data beyond that one `reading_order` number.
### Play all
`magPlayAll` builds `state.queue` from every *approved* article in listing
order and starts the first; `magPlay`'s completion path advances the queue
automatically rather than stopping, so listening runs across articles
without returning to the contents screen. `magRead(id, {fromStart: true})`
is what makes this a fresh start rather than resuming a queued article's own
saved progress, which would otherwise interrupt a continuous listen.
### Word-level highlighting is estimated, not measured
No forced-alignment or TTS timestamp source exists in this pipeline.
`estimateWordTimings` distributes a sentence's *real, measured* audio
duration (`audio.duration`, once loaded) across its words proportional to
character length — a common karaoke approximation, not true alignment, and
it can drift on a sentence with heavy punctuation-driven pauses. A checkbox
(`#mag-karaoke-cb`) lets it be turned off; sentence-level highlighting
(measured, not estimated — it just waits for `audio.onended`) always still
works regardless.
### Driving mode
`#mag-driving` is a full-screen overlay, not a separate playback path: it
reuses whatever is already playing or queued, so entering it never
interrupts an article in progress. Two oversized controls (skip, play/pause)
and no other interactive surface, matching the "no interaction required"
behaviour the manual originally specified as a P2 goal.
### Automatic sidebar/advert exclusion
`mark_sidebars` (server) reclassifies body text sitting in a column much
narrower than the page's regular grid as `BLOCK_SIDEBAR`; `group_articles`
excludes it from an article's narrated blocks the same way it already
excludes `BLOCK_FURNITURE`/`BLOCK_AD`. Full-width elements (a spanning
headline) are excluded from the column-width analysis itself — left in, one
bridges two genuinely separate columns into a single merged span and hides
a real narrow sidebar column entirely (confirmed by a failing test before
the fix). This is a geometry heuristic, not a certainty: review still shows
every sidebar block (unticked by default) and it can be included with one
click — nothing is silently discarded.
## Tests
- [`tests/test_magazine.py`](../tests/test_magazine.py) — 80 tests on the pure
- [`tests/test_magazine.py`](../tests/test_magazine.py) — 100 tests on the pure
geometry/classification logic: column reading order, the document-baseline
regression, furniture, pull quotes, contents parsing, continuation linking,
advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, and
the vision OCR quality score / region parsing / block building / HTTP call
(network mocked).
advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, the
vision OCR quality score / region parsing / block building / HTTP call
(network mocked), sidebar detection (including the full-width-headline
regression), image-to-article assignment, the whole-page vision fallback,
and the multi-engine fallback chain.
- [`tests/test_magazine_routes.py`](../tests/test_magazine_routes.py) — 10 tests: path
traversal rejection, review gating, LLM script parsing including malformed
output.
@ -263,15 +319,71 @@ sentence (confirmed live: tesseract's "KI" → "K |" substitution, 7 of 8 words
still genuine German) scores just as high as clean text. Catching that class
of error needs a dictionary or language-model check, out of scope here.
Routing to KDL Frontier Parser nano, or a genuine multi-engine fallback
ladder with benchmark comparison, remains TODO P2.
### Second engine and fallback ladder
`vision_ocr_page_chain` tries each configured (url, model) pair in order,
first usable result wins — primary (`vision_ocr_url`/`_model`), then fallback
(`vision_ocr_fallback_url`/`_model`), then tesseract if every vision engine
fails or is unreachable. KDL Frontier Parser nano is wired as the fallback,
confirmed live: it answers markedly faster than Baidu (~12s vs ~40s per page
on the same test page) but returns **plain text with no coordinate markers
at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content` regions.
`_whole_page_region` is what makes that still usable rather than discarded:
the whole response becomes one ambiguous full-page "text" region (bbox =
the entire rendered page, read via PIL), gated by the same
`ocr_quality_score` threshold used for native/tesseract text so a genuinely
bad KDL response isn't accepted either. Column and reading-order structure
is lost for a page handled this way — a real text improvement over
tesseract, but a worse layout result than Baidu's, which is why it is the
fallback and not the default.
A formal cross-engine accuracy benchmark remains TODO P2.
The Active OCR Engine panel (Settings → Engines → OCR & Document Parsing)
mirrors the Active Language Model panel exactly — same endpoint/model
selects, same `/api/conversation/llm-models` fetch (it just proxies to
whatever URL it's given, not hardcoded to the LLM setting) — because an OCR
engine here is the same shape of problem: an OpenAI-compatible endpoint plus
a model name, just applied to `vision_ocr_url`/`vision_ocr_model` instead of
`llm_url`/`llm_model`. Local engine cards (`#dc-grid-ocr`) were already
supported by `renderLocalContainers` — that generic code handles all four
roles (`tts`/`stt`/`llm`/`ocr`) identically — but the OCR page had no
`.s-engines-page[data-page="ocr"]` markup at all until this change, so
`$('dc-grid-ocr')` was always null and the whole card grid silently never
rendered. Confirmed live: the page now shows both engines as ready-made
local cards (Connect/Start/Restart/Use as OCR/Show code snippet), matching
the LLM page, with no "Add Custom" dialog needed for either.
### A workflow-pipeline nav bar was bleeding into unrelated pages
`_wfUpdateHeaderNav` (utils.js) injects the audiobook pipeline's "Previous /
Next" bar and step hint into *every* `.section-head` in the document, with
Studio as the only named exclusion — confirmed live: the Engines page (LLM/
STT/TTS/OCR/Image Generation all share one `.section-head`) showed a
"Previous … Cast Audiobook ›" bar and a "STEP 1/8" hint that made no sense
there, since the pipeline's three destinations are `s-reader`, `s-library`
and `s-rehearser` only. Changed from a blocklist (name each unrelated
section as it's noticed) to an allowlist (`head.closest('#s-reader,
#s-library, #s-rehearser')`) so the same bleed-through cannot recur on
Magazines, Settings, or any future section.
## Known limitations
- Sidebar detection (`BLOCK_SIDEBAR`) is defined but not yet inferred; sidebars
currently classify as body and must be excluded by hand in review. P2.
- Sidebar detection (`mark_sidebars`) is geometry-only (an anomalously narrow
column) — it has no notion of a boxed border or background tint, so a
false positive/negative is possible on an unusual layout. Review always
shows every sidebar block and lets it be ticked back in with one click.
- Word-level highlighting is an *estimate* (proportional to character length
within the sentence's real measured audio duration), not true forced
alignment — see "Word-level highlighting is estimated, not measured" above.
- `continued from page N` is parsed but not used for linking — only the forward
`continued on page N` reference drives it.
- Reviews-style pages with no headline are absorbed by the preceding article
when the contents page does not list them.
- Word-level timing is not available; P1 targets sentence highlighting.
- KDL Frontier Parser nano (the OCR fallback engine) returns no layout
coordinates, so a page it handles loses column/reading-order structure
even though its text is usable — see "Second engine and fallback ladder".
- No formal cross-engine OCR accuracy benchmark exists; the engine
comparison above is one confirmed live test, not a systematic evaluation.
- Image descriptions, table/chart handling, and EPUB 3 Media Overlay export
remain P2 and are not started.

View File

@ -17,6 +17,7 @@ from core.constants import (
_FASTER_WHISPER_DEFAULT, _WHISPER_CPP_DEFAULT,
_KOKORO_DEFAULT, _VIBEVOICE_DEFAULT, _XTTS_DEFAULT, _FISHSPEECH_DEFAULT,
_VISION_OCR_DEFAULT, _VISION_OCR_MODEL_DEFAULT,
_VISION_OCR_FALLBACK_DEFAULT, _VISION_OCR_FALLBACK_MODEL_DEFAULT,
_TTS_STREAM_DEFAULT,
)
from core.retry import request_with_retries
@ -41,6 +42,7 @@ _SETTINGS_KEYS = {
"client_voice_bindings",
"llm_url", "llm_model", "llm_api_key",
"vision_ocr_url", "vision_ocr_model", "vision_ocr_enabled",
"vision_ocr_fallback_url", "vision_ocr_fallback_model",
"engine_local_urls", "engine_container_names", "engine_api_keys", "custom_engine_cards",
# Character portrait generation (cloud APIs + local ComfyUI)
"image_gen_provider", "image_gen_model",
@ -335,6 +337,8 @@ def _load_settings() -> dict:
"vision_ocr_url": _VISION_OCR_DEFAULT,
"vision_ocr_model": _VISION_OCR_MODEL_DEFAULT,
"vision_ocr_enabled": True,
"vision_ocr_fallback_url": _VISION_OCR_FALLBACK_DEFAULT,
"vision_ocr_fallback_model": _VISION_OCR_FALLBACK_MODEL_DEFAULT,
"engine_local_urls": {},
"engine_container_names": {},
"engine_api_keys": {},

View File

@ -46,6 +46,11 @@ _KOKORO_DEFAULT = os.environ.get("KOKORO_URL", "http://192.168.1
# container-name default would not resolve from the app container.
_VISION_OCR_DEFAULT = os.environ.get("VISION_OCR_URL", "http://host.docker.internal:8098/v1")
_VISION_OCR_MODEL_DEFAULT = os.environ.get("VISION_OCR_MODEL", "baidu/Unlimited-OCR")
# Second vision OCR engine, tried when the primary fails or is unreachable.
# Confirmed live: answers faster than Baidu Unlimited-OCR but returns plain
# text with no layout coordinates (see _whole_page_region in core/magazine.py).
_VISION_OCR_FALLBACK_DEFAULT = os.environ.get("VISION_OCR_FALLBACK_URL", "http://host.docker.internal:8099/v1")
_VISION_OCR_FALLBACK_MODEL_DEFAULT = os.environ.get("VISION_OCR_FALLBACK_MODEL", "kdl-frontier-parser-nano")
_VIBEVOICE_DEFAULT = os.environ.get("VIBEVOICE_URL", "http://192.168.178.8:8027")
_XTTS_DEFAULT = os.environ.get("XTTS_URL", "http://192.168.178.8:8005")
_FISHSPEECH_DEFAULT = os.environ.get("FISHSPEECH_URL", "http://192.168.178.8:38080")

View File

@ -370,6 +370,48 @@ def mark_pullquotes(blocks: list[Block]) -> list[Block]:
return blocks
def mark_sidebars(blocks: list[Block], page_width: float,
narrow_ratio: float = 0.7) -> list[Block]:
"""Reclassify body text sitting in an anomalously narrow column as a
sidebar, so it is excluded from an article's narration by default rather
than requiring a manual exclusion in review every time.
Geometry-only, like the rest of this module's classification: a sidebar
or pull-out box is reliably narrower than the page's regular column grid
(a boxed insert set off from the main flow), which needs no image
analysis to detect. This is a heuristic, not a certainty -- a genuinely
narrow-column page design (or a caption sitting in its own slim column)
can trigger a false positive, which is exactly why review still shows
every sidebar block and lets it be included with one click rather than
silently discarding it.
Full-width elements (a headline spanning the whole page) are excluded
from the column analysis here, the same way assign_reading_order treats
them specially: left in, one would bridge two genuinely separate columns
into a single merged span (its bbox overlaps both), hiding a real narrow
sidebar column entirely. This computes its own column partition rather
than trusting `.column` from assign_reading_order, since that partition
was built from the FULL block list and its indices would not line up
with this narrower one.
"""
candidates = [b for b in blocks if b.bbox[2] <= page_width * 0.6]
columns = detect_columns(candidates, page_width)
if len(columns) < 2:
return blocks
widths = [right - left for left, right in columns]
widest = max(widths)
narrow_spans = [columns[i] for i, w in enumerate(widths) if w < widest * narrow_ratio]
if not narrow_spans:
return blocks
for b in blocks:
if b.type != BLOCK_BODY or b.bbox[2] > page_width * 0.6:
continue
centre = b.bbox[0] + b.bbox[2] / 2
if any(left <= centre <= right for left, right in narrow_spans):
b.type, b.confidence = BLOCK_SIDEBAR, 0.6
return blocks
def find_repeated_lines(pages_text: Sequence[Sequence[str]],
threshold: float = 0.25) -> set[str]:
"""Lines appearing near the same edge on many pages: page furniture."""
@ -562,9 +604,15 @@ def group_articles(blocks: Sequence[Block],
# advert sitting between two features starts no article of its
# own, so it would otherwise be swallowed by the preceding one and
# highlighted as part of it in the PDF view.
#
# A sidebar/pull-out box (mark_sidebars) is excluded the same way
# advert text is: automatically kept out of narration by default,
# rather than requiring a manual exclusion in review every time.
# It still shows in review (blocksForPages lists every block on
# the article's pages) and can be ticked back in with one click.
contributed = False
for b in page_blocks:
if b.type in (BLOCK_FURNITURE, BLOCK_AD):
if b.type in (BLOCK_FURNITURE, BLOCK_AD, BLOCK_SIDEBAR):
continue
contributed = True
art.blocks.append(b.id)
@ -744,14 +792,37 @@ def parse_vision_ocr_regions(raw: str) -> list[dict[str, Any]]:
return regions
def _whole_page_region(content: str, png_path: Path) -> list[dict[str, Any]] | None:
"""Fallback for an engine that answers with plain text and no coordinate
markers at all (confirmed live: KDL Frontier Parser nano -- fast, but no
"label [x0,y0,x1,y1]" structure the way ocr-baidu-unlimited-ocr gives).
Treated as one ambiguous "text" region spanning the whole page: real
text with no layout information beats no text at all, but it does mean
column/reading-order structure is lost for that page -- classify_blocks
still runs on it (source="vision_ocr", not "vision_label"), same as any
other ambiguous region.
"""
text = (content or "").strip()
if len(text) < 20 or ocr_quality_score(text) < _VISION_QUALITY_THRESHOLD:
return None
try:
from PIL import Image
with Image.open(png_path) as im:
width, height = im.size
except Exception:
return None
return [{"label": "text", "bbox": [0.0, 0.0, float(width), float(height)], "text": text}]
def vision_ocr_page(png_path: Path, base_url: str, model: str,
timeout: float = 150.0) -> list[dict[str, Any]] | None:
"""OCR one rendered page with a connected vision-capable OCR engine.
Returns None on any failure (unreachable engine, timeout, unusable
output) so the caller can fall back to tesseract -- this must never be
the reason ingestion fails outright. Confirmed live against
ocr-baidu-unlimited-ocr: markedly more accurate German text than
output) so the caller can fall back to the next engine or tesseract --
this must never be the reason ingestion fails outright. Confirmed live
against ocr-baidu-unlimited-ocr: markedly more accurate German text than
tesseract on the same scanned page, and it labels running headers/
footers and page numbers directly, which is the dominant source of the
over-fragmentation cluster_running_furniture() otherwise has to recover
@ -787,7 +858,26 @@ def vision_ocr_page(png_path: Path, base_url: str, model: str,
return None
regions = parse_vision_ocr_regions(content)
return regions or None
return regions or _whole_page_region(content, png_path)
def vision_ocr_page_chain(png_path: Path,
engines: Sequence[tuple[str, str]],
timeout: float = 150.0) -> list[dict[str, Any]] | None:
"""Try each configured vision OCR engine in order, first success wins.
A primary engine (labeled regions, best quality) with a faster or
differently-shaped secondary as a real fallback -- not just tesseract --
when the primary is unreachable or times out. Engines are (url, model)
pairs; empty/blank entries are skipped.
"""
for url, model in engines:
if not (url and model):
continue
regions = vision_ocr_page(png_path, url, model, timeout=timeout)
if regions:
return regions
return None
def build_vision_blocks(regions: list[dict[str, Any]], page_no: int,
@ -910,6 +1000,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
ocr_lang: str = "deu+eng",
vision_ocr_url: str | None = None,
vision_ocr_model: str | None = None,
vision_ocr_fallback_url: str | None = None,
vision_ocr_fallback_model: str | None = None,
vision_ocr_timeout: float = 150.0,
progress: Any = None) -> dict[str, Any]:
"""Run the full extraction for one PDF, writing page renders and images.
@ -917,13 +1009,17 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
When vision_ocr_url/model are given, a page whose text needs OCR (or
whose already-extracted text scores as mostly garbage, native or
tesseract alike -- see ocr_quality_score) is escalated to that engine
first, with tesseract only as the fallback if the call fails or the
engine is unreachable. Confirmed live against ocr-baidu-unlimited-ocr:
markedly more accurate transcription of a scanned German magazine than
tesseract, and it labels running headers/footers/page numbers directly
(see build_vision_blocks), which is the dominant source of the
over-fragmentation cluster_running_furniture() otherwise has to recover
after the fact from noisy OCR variants of the same repeating line.
first, with a second vision engine (vision_ocr_fallback_*) tried next if
given, and tesseract only as the last resort if every vision call fails
or every engine is unreachable. Confirmed live against
ocr-baidu-unlimited-ocr: markedly more accurate transcription of a
scanned German magazine than tesseract, and it labels running headers/
footers/page numbers directly (see build_vision_blocks), which is the
dominant source of the over-fragmentation cluster_running_furniture()
otherwise has to recover after the fact from noisy OCR variants of the
same repeating line. KDL Frontier Parser nano, tested as the fallback
engine, answers faster but returns plain text with no layout coordinates
at all -- handled by _whole_page_region as one ambiguous full-page block.
Returns the layout document: every block on every page, classified and in
reading order, plus the raw contents-page candidates.
@ -933,6 +1029,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
images_dir = out_dir / "images"
pages_dir.mkdir(parents=True, exist_ok=True)
images_dir.mkdir(parents=True, exist_ok=True)
vision_engines = [(vision_ocr_url, vision_ocr_model),
(vision_ocr_fallback_url, vision_ocr_fallback_model)]
vision_enabled = bool(vision_ocr_url and vision_ocr_model)
vision_page_count = 0
@ -951,14 +1049,27 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
pix.save(str(png))
image_boxes: list[list[float]] = []
page_images: list[dict[str, Any]] = []
for img_index, info in enumerate(page.get_images(full=True)):
rects: list[list[float]] = []
try:
for r in page.get_image_rects(info[0]):
image_boxes.append([r.x0, r.y0, r.width, r.height])
rects.append([r.x0, r.y0, r.width, r.height])
except Exception:
continue
_extract_image(fitz, doc, info[0], images_dir,
f"p{page_no:04d}-{img_index:02d}")
image_boxes.extend(rects)
name = f"p{page_no:04d}-{img_index:02d}"
if _extract_image(fitz, doc, info[0], images_dir, name) and rects:
# A few embedded images repeat (a logo stamped on every
# page); each placement still gets its own bbox so a
# caption or figure position can be matched precisely.
for ri, box in enumerate(rects):
page_images.append({
"id": name if ri == 0 else f"{name}-{ri}",
"page": page_no,
"bbox": box,
"filename": f"{name}.png",
})
blocks: list[Block] = []
data = page.get_text("dict")
@ -1004,7 +1115,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
vision_blocks: list[Block] | None = None
if vision_enabled and needs_escalation:
regions = vision_ocr_page(png, vision_ocr_url, vision_ocr_model,
regions = vision_ocr_page_chain(png, vision_engines,
timeout=vision_ocr_timeout)
if regions:
scale_x = rect.width / max(1.0, _png_width(png, fitz))
@ -1042,6 +1153,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
"height": rect.height,
"blocks": blocks,
"image_boxes": image_boxes,
"images": page_images,
})
if progress:
progress(page_no, doc.page_count)
@ -1066,6 +1178,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
mark_advert_pages(p["blocks"])
mark_pullquotes(p["blocks"])
assign_reading_order(p["blocks"], p["width"])
mark_sidebars(p["blocks"], p["width"])
all_blocks.extend(p["blocks"])
# Cross-page, so it runs once over every block rather than per page
@ -1084,11 +1197,16 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
"entries": parse_contents(lines),
})
all_images = [img for p in raw_pages for img in p["images"]]
for img in all_images:
img["reading_order"] = image_reading_position(img, all_blocks)
return {
"page_count": doc.page_count,
"pages": [{"page": p["page"], "width": p["width"], "height": p["height"]}
for p in raw_pages],
"blocks": [b.to_json() for b in all_blocks],
"images": all_images,
"contents_candidates": contents_candidates,
"vision_ocr_pages": vision_page_count,
}
@ -1103,19 +1221,87 @@ def _png_width(png: Path, fitz) -> float:
return 1.0
def _extract_image(fitz, doc, xref: int, images_dir: Path, name: str) -> None:
"""Save one embedded image, skipping the tiny rules and spacers."""
def _extract_image(fitz, doc, xref: int, images_dir: Path, name: str) -> bool:
"""Save one embedded image, skipping the tiny rules and spacers.
Returns whether a file now exists at that name, so the caller can decide
whether to record it as a real figure -- without this, a page's rules and
spacers were silently skipped on disk but still implicitly "existed" as
far as anything downstream knew, since nothing checked.
"""
try:
pix = fitz.Pixmap(doc, xref)
if pix.width < 64 or pix.height < 64:
return
return False
if pix.n - pix.alpha >= 4: # CMYK needs converting first
pix = fitz.Pixmap(fitz.csRGB, pix)
target = images_dir / f"{name}.png"
if not target.exists():
pix.save(str(target))
return True
except Exception:
return
return False
# ── Images: linking a figure to the article it illustrates ──────────────────
def assign_images_to_articles(images: Sequence[dict[str, Any]],
blocks: Sequence[Block],
articles: Sequence[Article]) -> None:
"""Attach each extracted figure to the article whose column it falls in.
A figure belongs to whichever article occupies its page; when two
articles share a page (a feature ending, a new one starting lower down),
the figure goes to whichever article has the nearer block by vertical
position, so a photo is not shown under the wrong headline. Mutates
articles in place.
"""
by_page: dict[int, list[Article]] = {}
for art in articles:
for page in art.pages:
by_page.setdefault(page, []).append(art)
blocks_by_page: dict[int, list[Block]] = {}
for b in blocks:
blocks_by_page.setdefault(b.page, []).append(b)
for img in images:
candidates = by_page.get(img["page"], [])
if not candidates:
continue
if len(candidates) == 1:
candidates[0].images.append(img["id"])
continue
# Several articles share this page: pick by nearest block vertically,
# since that is what a reader would associate the figure with.
img_y = img["bbox"][1]
best, best_dist = candidates[0], float("inf")
for art in candidates:
own_blocks = [b for b in blocks_by_page.get(img["page"], [])
if b.id in set(art.blocks)]
if not own_blocks:
continue
dist = min(abs(b.bbox[1] - img_y) for b in own_blocks)
if dist < best_dist:
best, best_dist = art, dist
best.images.append(img["id"])
def image_reading_position(image: dict[str, Any], blocks: Sequence[Block]) -> int:
"""Approximate reading_order for an image, for ordering it among sentences.
Uses the nearest block on the same page whose top sits at or above the
image (so a photo is shown once the surrounding text has been reached,
not before), falling back to the closest block on the page if the image
sits above all of them.
"""
same_page = [b for b in blocks if b.page == image["page"]]
if not same_page:
return 0
above = [b for b in same_page if b.bbox[1] <= image["bbox"][1]]
pool = above or same_page
nearest = min(pool, key=lambda b: abs(b.bbox[1] - image["bbox"][1]))
return nearest.reading_order
# ── Persistence ──────────────────────────────────────────────────────────────
@ -1131,6 +1317,7 @@ def build_articles(layout: dict[str, Any]) -> list[dict[str, Any]]:
skip = {c["page"] for c in candidates if c.get("score", 0) >= 0.45}
articles = group_articles(blocks, entries, layout.get("page_count", 0),
skip_pages=skip)
assign_images_to_articles(layout.get("images") or [], blocks, articles)
out = []
for art in articles:
rec = art.to_json()

View File

@ -109,7 +109,9 @@ async def magazine_create(request: Request):
# ── Stage A–C: ingestion ─────────────────────────────────────────────────────
def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
vision_ocr_url: str | None, vision_ocr_model: str | None) -> None:
vision_ocr_url: str | None, vision_ocr_model: str | None,
vision_ocr_fallback_url: str | None = None,
vision_ocr_fallback_model: str | None = None) -> None:
"""Run the extraction pipeline; recorded as a restart-safe job."""
d = _LIB / doc_id
pdf = d / "source.pdf"
@ -127,6 +129,8 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
layout = mag.extract_pages(pdf, d, render_dpi=render_dpi, ocr_lang=ocr_lang,
vision_ocr_url=vision_ocr_url,
vision_ocr_model=vision_ocr_model,
vision_ocr_fallback_url=vision_ocr_fallback_url,
vision_ocr_fallback_model=vision_ocr_fallback_model,
progress=progress)
mag.save_layout(d, layout)
@ -183,6 +187,7 @@ async def magazine_ingest(doc_id: str, request: Request):
# approved and only re-detection is wanted.
settings = _load_settings()
vision_ocr_url = vision_ocr_model = None
vision_ocr_fallback_url = vision_ocr_fallback_model = None
want_vision = body.get("vision_ocr")
if want_vision is None:
want_vision = bool(settings.get("vision_ocr_enabled"))
@ -191,6 +196,12 @@ async def magazine_ingest(doc_id: str, request: Request):
vision_ocr_model = str(body.get("vision_ocr_model") or settings.get("vision_ocr_model") or "").strip()
if not (vision_ocr_url and vision_ocr_model):
vision_ocr_url = vision_ocr_model = None
vision_ocr_fallback_url = str(
body.get("vision_ocr_fallback_url") or settings.get("vision_ocr_fallback_url") or "").strip()
vision_ocr_fallback_model = str(
body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip()
if not (vision_ocr_fallback_url and vision_ocr_fallback_model):
vision_ocr_fallback_url = vision_ocr_fallback_model = None
job_id = uuid.uuid4().hex
background_job_create(job_id, "magazine-ingest", fingerprint=doc_id)
@ -199,7 +210,8 @@ async def magazine_ingest(doc_id: str, request: Request):
asyncio.create_task(
asyncio.to_thread(_ingest_sync, doc_id, job_id, ocr_lang, render_dpi,
vision_ocr_url, vision_ocr_model)
vision_ocr_url, vision_ocr_model,
vision_ocr_fallback_url, vision_ocr_fallback_model)
)
return {"job_id": job_id}

View File

@ -427,6 +427,134 @@
})();
})();
// ── Active OCR Engine (global picker, magazine ingestion) ───────────────
// Mirrors initActiveLlmPanel above: an OCR engine is the same shape of
// problem (an OpenAI-compatible endpoint plus a model name), just applied
// to vision_ocr_url/vision_ocr_model/vision_ocr_enabled instead of
// llm_url/llm_model. Reuses the same model-list endpoint, since it just
// proxies to whatever URL it's given rather than assuming it's the LLM one.
(function initActiveOcrPanel() {
const urlInp = $('ocr-active-url');
const modelSel = $('ocr-active-model');
const refreshBtn = $('ocr-active-refresh');
const statusEl = $('ocr-active-status');
const enabledCb = $('ocr-active-enabled');
if (!urlInp || !modelSel) return;
function setStatus(msg, cls) {
if (!statusEl) return;
statusEl.textContent = msg || '';
statusEl.className = 'llm-active-status' + (cls ? ' ' + cls : '');
}
async function fetchModels(url) {
url = (url || urlInp.value.trim());
if (!url) { modelSel.innerHTML = '<option value="">— set an endpoint —</option>'; setStatus(''); return; }
const want = (_appSettings && _appSettings.vision_ocr_model) || '';
setStatus('Loading models…');
if (refreshBtn) refreshBtn.disabled = true;
try {
const d = await fetch('/api/conversation/llm-models?url=' + encodeURIComponent(url)).then(r => r.json());
if (d.error) {
let msg = d.error;
try { const j = JSON.parse(d.error); msg = j.error?.message || j.detail || d.error; } catch (_) {}
if (msg.length > 50) msg = msg.substring(0, 47) + '...';
modelSel.innerHTML = '<option value="">Auth / Error: ' + escHtml(msg) + '</option>';
setStatus(msg, 'err');
return;
}
const models = d.models || [];
modelSel.innerHTML = models.length
? models.map(m => `<option value="${escHtml(m)}">${escHtml(m)}</option>`).join('')
: '<option value="">No models found</option>';
if (want && models.includes(want)) modelSel.value = want;
setStatus(models.length ? `${models.length} model${models.length === 1 ? '' : 's'}` : 'No models found',
models.length ? 'ok' : 'err');
} catch (e) {
modelSel.innerHTML = '<option value="">Fetch failed</option>';
setStatus('Could not reach endpoint', 'err');
} finally {
if (refreshBtn) refreshBtn.disabled = false;
}
}
function commitUrl() {
const url = urlInp.value.trim();
if (!url) return;
applyAndSaveSettings({ vision_ocr_url: url });
if (_appSettings) _appSettings.vision_ocr_url = url;
fetchModels(url);
}
urlInp.addEventListener('blur', commitUrl);
urlInp.addEventListener('keydown', e => { if (e.key === 'Enter') { e.preventDefault(); commitUrl(); } });
urlInp.addEventListener('change', commitUrl);
function refreshEndpointOptions(currentUrl) {
if (!urlInp || urlInp.tagName !== 'SELECT') return;
let html = '<option value="">— Select an endpoint —</option>';
const added = new Set();
document.querySelectorAll('#dc-grid-ocr .llm-local-card').forEach(card => {
const uInp = card.querySelector('.dc-url-inp');
if (!uInp) return;
const u = uInp.value.trim() || uInp.placeholder;
const name = card.querySelector('.llm-local-name')?.textContent.trim() || 'Local Engine';
if (u && !added.has(u)) { added.add(u); html += `<option value="${escHtml(u)}">${escHtml(name)}</option>`; }
});
const activeUrl = currentUrl || urlInp.value;
if (activeUrl && !added.has(activeUrl)) {
html += `<option value="${escHtml(activeUrl)}">Custom: ${escHtml(activeUrl)}</option>`;
}
urlInp.innerHTML = html;
if (activeUrl) urlInp.value = activeUrl;
}
urlInp.addEventListener('focus', () => refreshEndpointOptions());
modelSel.addEventListener('change', () => {
applyAndSaveSettings({ vision_ocr_model: modelSel.value });
if (_appSettings) _appSettings.vision_ocr_model = modelSel.value;
if (modelSel.value) setStatus('Active model: ' + modelSel.value, 'ok');
});
enabledCb?.addEventListener('change', () => {
applyAndSaveSettings({ vision_ocr_enabled: enabledCb.checked });
if (_appSettings) _appSettings.vision_ocr_enabled = enabledCb.checked;
});
refreshBtn?.addEventListener('click', () => fetchModels());
// "Use as OCR" card buttons (see DC_ROLE_USE.ocr in engines.js) sync this
// panel the same way "Use as LLM" syncs the Active Language Model one.
window.syncActiveOcrPanel = function (url) {
refreshEndpointOptions(url);
urlInp.value = url || '';
fetchModels(url);
};
(async function () {
let url = (_appSettings && _appSettings.vision_ocr_url) || '';
let enabled = (_appSettings && _appSettings.vision_ocr_enabled);
if (!url) {
try {
const s = await fetch('/api/settings').then(r => r.json());
url = s.vision_ocr_url || '';
enabled = s.vision_ocr_enabled;
if (!_appSettings) _appSettings = s;
else {
_appSettings.vision_ocr_url = s.vision_ocr_url || _appSettings.vision_ocr_url;
_appSettings.vision_ocr_model = s.vision_ocr_model || _appSettings.vision_ocr_model;
_appSettings.vision_ocr_enabled = s.vision_ocr_enabled;
}
} catch (_) {}
}
if (enabledCb) enabledCb.checked = enabled !== false;
setTimeout(() => {
refreshEndpointOptions(url);
urlInp.value = url || '';
if (url) fetchModels(url);
}, 100);
})();
})();
// ── Inline STT quick-test panel ─────────────────────────────────────────
(async function initSttTestPanel() {
const panel = $('stt-test-panel');

View File

@ -285,6 +285,7 @@ const DC_ROLE_USE = {
tts: { settingKey: 'tts_url', label: 'Use as TTS' },
stt: { settingKey: 'faster_whisper_url', label: 'Use as STT' },
llm: { settingKey: 'llm_url', label: 'Use as LLM' },
ocr: { settingKey: 'vision_ocr_url', label: 'Use as OCR' },
};
window.openAddEngineDialog = function(role) {
@ -797,6 +798,7 @@ function renderLocalContainers(containers) {
if (apiKey && settingKey === 'llm_url') patch.llm_api_key = apiKey;
await window.applyAndSaveSettings(patch);
if (settingKey === 'llm_url' && window.syncActiveLlmPanel) window.syncActiveLlmPanel(url, apiKey);
if (settingKey === 'vision_ocr_url' && window.syncActiveOcrPanel) window.syncActiveOcrPanel(url);
btn.classList.add('active');
toast(`✓ ${btn.textContent.trim()} — now available in the backend dropdown`, 'success');
}

View File

@ -20,6 +20,7 @@ const state = {
layout: null, // { page_count, pages[], blocks[] }
article: null, // the article being read or reviewed
sentences: [], // [{ text, blockId, page, bbox }]
images: [], // current article's figures, in reading order
view: 'library', // library | contents | review | read
pane: 'reflow', // reflow | pdf
playing: false,
@ -28,6 +29,10 @@ const state = {
audio: null,
voices: [],
podcast: null,
queue: [], // article ids for "play all" (empty = not queued)
queueIndex: -1,
driving: false, // large-control, minimal-interaction view
karaoke: true, // word-level highlighting (approximate, see estimateWordTimings)
};
MAG.magazine = state;
@ -387,14 +392,60 @@ function splitSentences(blocks) {
.map((s) => s.trim()).filter(Boolean);
(parts.length ? parts : [b.text]).forEach((text) => {
if (text && text.trim()) {
out.push({ text: text.trim(), blockId: b.id, page: b.page, bbox: b.bbox, type: b.type });
out.push({
text: text.trim(), blockId: b.id, page: b.page, bbox: b.bbox, type: b.type,
// Used to pick "the image reached so far" during playback (see
// currentImageIndex) -- the backend already computes this per
// block relative to the whole page, not just this article.
readingOrder: b.reading_order || 0,
words: text.trim().split(/\s+/).filter(Boolean),
});
}
});
});
return out;
}
async function magRead(articleId) {
// The current article's figures, in reading order, resolved from the full
// layout (fetched once) filtered to this article's own image ids. Kept
// separate from state.sentences because an image has no sentence of its
// own -- it is matched to "the most recent sentence at or before it".
function articleImages(article) {
const ids = new Set(article.images || []);
return (state.layout.images || [])
.filter((img) => ids.has(img.id))
.sort((a, b) => (a.reading_order || 0) - (b.reading_order || 0));
}
// The image to show once sentence `i` has been reached: the last image
// whose position in the page's reading order is at or before that
// sentence's own block. Falls back to the first image before any text has
// caught up to it, so a lead photo is not left blank at the very start.
function currentImageIndex(i) {
if (!state.images.length) return -1;
const order = (state.sentences[i] || {}).readingOrder || 0;
let found = -1;
for (let k = 0; k < state.images.length; k++) {
if ((state.images[k].reading_order || 0) <= order) found = k;
else break;
}
return found === -1 ? 0 : found;
}
function renderCurrentImage(i) {
const host = $('mag-read-image');
if (!host) return;
const idx = currentImageIndex(i);
if (idx < 0) { host.hidden = true; return; }
const img = state.images[idx];
if (state._shownImageId === img.id) { host.hidden = false; return; }
state._shownImageId = img.id;
host.hidden = false;
host.innerHTML = '<img src="/api/magazine/' + state.doc.id + '/images/' + encodeURIComponent(img.filename) +
'" alt="Figure from page ' + img.page + '" loading="lazy">';
}
async function magRead(articleId, opts) {
const a = state.articles.find((x) => x.id === articleId);
if (!a) return;
state.article = a;
@ -406,7 +457,10 @@ async function magRead(articleId) {
} catch (e) {
return say('Could not load article text: ' + e.message, 'err');
}
const saved = (state.doc.progress || {})[a.id];
state.images = articleImages(a);
state._shownImageId = null;
const forceStart = opts && opts.fromStart;
const saved = !forceStart && (state.doc.progress || {})[a.id];
state.idx = saved && saved.sentence < state.sentences.length ? saved.sentence : 0;
magShow('read');
renderArticle();
@ -426,8 +480,13 @@ function renderArticle() {
: s.type === 'byline' ? 'p class="mag-byline"' : s.type === 'caption'
? 'p class="mag-caption"' : 'span';
const open = tag.split(' ')[0];
// Each word gets its own span so playback can highlight it individually
// (see estimateWordTimings) -- sentence-level highlighting alone doesn't
// need this, but word-level does and this markup serves both.
const wordsHtml = s.words.map((w, j) =>
'<span class="mag-w" data-w="' + j + '">' + esc(w) + '</span>').join(' ');
return '<' + tag + ' class="mag-s' + (open === 'span' ? '' : '') + '" data-i="' + i + '" ' +
'onclick="magSeek(' + i + ')">' + esc(s.text) + '</' + open + '> ';
'onclick="magSeek(' + i + ')">' + wordsHtml + '</' + open + '> ';
}).join('');
highlight(state.idx);
}
@ -436,13 +495,54 @@ function highlight(i) {
document.querySelectorAll('#mag-article [data-i]').forEach((el) => {
el.classList.toggle('is-current', Number(el.dataset.i) === i);
});
clearWordHighlight();
const el = document.querySelector('#mag-article [data-i="' + i + '"]');
if (el && !prefersReducedMotion()) el.scrollIntoView({ block: 'center', behavior: 'smooth' });
else if (el) el.scrollIntoView({ block: 'center' });
const s = state.sentences[i];
if (s && state.pane === 'pdf') showPage(s.page, s.bbox);
renderCurrentImage(i);
const counter = $('mag-read-progress');
if (counter) counter.textContent = (i + 1) + ' / ' + state.sentences.length;
const qText = state.queue.length ? 'Article ' + (state.queueIndex + 1) + ' / ' + state.queue.length : '';
['mag-queue-progress', 'mag-queue-progress-2'].forEach((id) => {
const el = $(id);
if (el) el.textContent = qText;
});
}
function clearWordHighlight() {
document.querySelectorAll('#mag-article .mag-w.is-current').forEach((el) => el.classList.remove('is-current'));
}
// ── Word-level highlighting (approximate) ────────────────────────────────
// No forced-alignment or TTS timestamp source exists in this pipeline, so
// per-word timing is ESTIMATED, not measured: each word's share of the
// sentence's real (measured) audio duration is proportional to its own
// character length, with a small fixed share added per word so short words
// are not given an unrealistically instant duration. This is a common
// karaoke approximation, not true alignment -- it can drift on sentences
// with heavy punctuation-driven pauses.
function estimateWordTimings(words, totalSeconds) {
if (!words.length || !totalSeconds) return [];
const FIXED = 0.6; // arbitrary units, shifts weight toward evening out short words
const weights = words.map((w) => Math.max(1, w.replace(/[.,!?;:"'()«»„"]/g, '').length) + FIXED);
const total = weights.reduce((a, b) => a + b, 0);
let t = 0;
return weights.map((w) => {
const dur = totalSeconds * (w / total);
const start = t;
t += dur;
return { start, end: t };
});
}
function highlightWord(sentenceIdx, wordIdx) {
const sentenceEl = document.querySelector('#mag-article [data-i="' + sentenceIdx + '"]');
if (!sentenceEl) return;
sentenceEl.querySelectorAll('.mag-w').forEach((el) => {
el.classList.toggle('is-current', Number(el.dataset.w) === wordIdx);
});
}
function prefersReducedMotion() {
@ -508,46 +608,104 @@ async function magPlay() {
state.playing = true;
state.cancel = false;
syncPlayButton();
let finishedArticle = false;
try {
while (state.idx < state.sentences.length && !state.cancel) {
highlight(state.idx);
const text = state.sentences[state.idx].text;
const sentence = state.sentences[state.idx];
let blob;
try {
blob = await fetchTtsPreviewBlob(voice, text, 'mp3');
blob = await fetchTtsPreviewBlob(voice, sentence.text, 'mp3');
} catch (e) {
say('Synthesis failed: ' + e.message, 'err');
break;
}
if (state.cancel) break;
await playBlob(blob);
await playBlob(blob, sentence);
if (state.cancel) break;
state.idx += 1;
saveProgress();
}
finishedArticle = state.idx >= state.sentences.length && !state.cancel;
} finally {
state.playing = false;
state.cancel = false;
syncPlayButton();
if (state.idx >= state.sentences.length) {
}
if (finishedArticle) {
state.idx = state.sentences.length - 1;
if (state.queue.length && state.queueIndex < state.queue.length - 1) {
// Play-all: move straight to the next approved article rather than
// stopping -- the whole point of a queue is not touching the screen
// between articles.
state.queueIndex += 1;
await magRead(state.queue[state.queueIndex], { fromStart: true });
magPlay();
} else {
state.queue = [];
state.queueIndex = -1;
say('Article finished', 'ok');
}
}
}
function playBlob(blob) {
function playBlob(blob, sentence) {
return new Promise((resolve) => {
const url = URL.createObjectURL(blob);
const audio = new Audio(url);
state.audio = audio;
const done = () => { URL.revokeObjectURL(url); state.audio = null; resolve(); };
let timings = null;
const onTimeUpdate = () => {
if (!state.karaoke || !sentence || !sentence.words.length) return;
if (!timings) {
if (!audio.duration || !isFinite(audio.duration)) return;
timings = estimateWordTimings(sentence.words, audio.duration);
}
const t = audio.currentTime;
const wi = timings.findIndex((w) => t >= w.start && t < w.end);
if (wi !== -1) highlightWord(state.idx, wi);
};
if (state.karaoke) audio.addEventListener('timeupdate', onTimeUpdate);
const done = () => {
audio.removeEventListener('timeupdate', onTimeUpdate);
URL.revokeObjectURL(url);
state.audio = null;
resolve();
};
audio.onended = done;
audio.onerror = done;
audio.play().catch(done);
});
}
// ── Play all approved articles in sequence ──────────────────────────────
async function magPlayAll() {
const ids = state.articles.filter((a) => a.status === 'approved').map((a) => a.id);
if (!ids.length) return say('Approve at least one article first', 'err');
state.queue = ids;
state.queueIndex = 0;
await magRead(ids[0], { fromStart: true });
magPlay();
}
function magStopQueue() {
state.queue = [];
state.queueIndex = -1;
['mag-queue-progress', 'mag-queue-progress-2'].forEach((id) => {
const el = $(id);
if (el) el.textContent = '';
});
}
async function magQueueNext() {
if (!state.queue.length || state.queueIndex >= state.queue.length - 1) return;
const wasPlaying = state.playing;
if (wasPlaying) magPause();
state.queueIndex += 1;
await magRead(state.queue[state.queueIndex], { fromStart: true });
if (wasPlaying) magPlay();
}
function magPause() {
state.cancel = true;
if (state.audio) { try { state.audio.pause(); } catch (_) {} }
@ -557,11 +715,13 @@ function magPause() {
function syncPlayButton() {
const btn = $('mag-play');
if (!btn) return;
if (btn) {
btn.innerHTML = state.playing
? '<span class="mdi mdi-pause"></span> Pause'
: '<span class="mdi mdi-play"></span> Listen';
btn.setAttribute('aria-label', state.playing ? 'Pause narration' : 'Play narration');
}
if (state.driving) syncDrivingPlayButton();
}
let _saveTimer = null;
@ -707,10 +867,45 @@ function magShow(view) {
function magBackToContents() {
magPause();
magStopQueue();
state.driving = false;
magShow('contents');
magRefreshArticles();
}
// ── Driving mode ──────────────────────────────────────────────────────────
// Large, minimal-interaction controls for listening without looking at the
// screen (see Manual.md's planned driving-mode behaviour). Reuses whatever
// is already playing/queued rather than starting its own playback path --
// entering driving mode does not interrupt an article already in progress.
function magToggleDriving() {
state.driving = !state.driving;
const overlay = $('mag-driving');
if (overlay) overlay.hidden = !state.driving;
if (state.driving) renderDriving();
}
function renderDriving() {
const a = state.article;
if (!a) return;
const titleEl = $('mag-driving-title');
if (titleEl) titleEl.textContent = a.title || 'Untitled';
const byEl = $('mag-driving-by');
if (byEl) byEl.textContent = a.author ? 'By ' + a.author : '';
syncDrivingPlayButton();
const qEl = $('mag-driving-queue');
if (qEl) qEl.hidden = !state.queue.length;
}
function syncDrivingPlayButton() {
const btn = $('mag-driving-play');
if (!btn) return;
btn.innerHTML = state.playing
? '<span class="mdi mdi-pause"></span>'
: '<span class="mdi mdi-play"></span>';
btn.setAttribute('aria-label', state.playing ? 'Pause' : 'Play');
}
// ── Boot ─────────────────────────────────────────────────────────────────────
function magInit() {
@ -736,6 +931,10 @@ root.magShow = magShow;
root.magBackToContents = magBackToContents;
root.magGeneratePodcast = magGeneratePodcast;
root.magPlayPodcast = magPlayPodcast;
root.magPlayAll = magPlayAll;
root.magQueueNext = magQueueNext;
root.magStopQueue = magStopQueue;
root.magToggleDriving = magToggleDriving;
root.magInit = magInit;
MAG.magazineApi = { load: magLoadLibrary, refresh: magRefreshArticles, splitSentences, fmtPages };

View File

@ -1112,11 +1112,18 @@ function _wfUpdateHeaderNav(anyState, curIdx) {
const nextStep = curIdx >= 0 ? _wfNeighbor(curIdx, 1) : null;
const curStep = curIdx >= 0 ? WF_STEPS[curIdx] : null;
document.querySelectorAll('.section-head').forEach(head => {
// Studio has its own self-contained 4-phase nav (studio.js) — this
// stepper's Previous/Next buttons are hardcoded to the old s-reader/
// s-library/s-rehearser sections and don't know about Studio's phases,
// so injecting them here just breaks navigation when clicked.
if (head.closest('#s-caststudio')) {
// This stepper's Previous/Next buttons are hardcoded to the audiobook
// pipeline's three destinations (s-reader, s-library, s-rehearser) — an
// allowlist, not a blocklist, since every OTHER section (Studio has its
// own 4-phase nav; Engines, Magazines, Settings, etc. have nothing to do
// with this pipeline at all) would otherwise need to be named here one
// at a time as each one was noticed showing a "Step 1/8" bar and a
// "Cast Audiobook" button that make no sense on that page. Confirmed
// live: this exact bleed-through appeared on Engines (LLM/STT/TTS/OCR/
// Image Generation all share one .section-head) before this became an
// allowlist.
const onPipelinePage = !!head.closest('#s-reader, #s-library, #s-rehearser');
if (!onPipelinePage) {
const staleNav = head.querySelector('.wf-header-nav');
if (staleNav) staleNav.hidden = true;
const staleHint = head.nextElementSibling?.classList?.contains('wf-step-hint') ? head.nextElementSibling : null;

View File

@ -807,6 +807,42 @@
</div><!-- /image -->
<div class="s-engines-page" data-page="ocr">
<!-- ── Active OCR Engine (global picker) ───────────────────── -->
<div class="llm-active-panel card">
<div class="llm-active-head">
<span class="llm-active-icon"><span class="mdi mdi-star-four-points"></span></span>
<div>
<h4 class="llm-active-title">Active OCR Engine</h4>
<p class="llm-active-sub">Used by Magazine ingestion when a page needs OCR, or when its existing text scores poorly. Tesseract (bundled, no setup) is always the fallback if this is unreachable.</p>
</div>
</div>
<div class="llm-active-controls">
<label class="llm-active-lbl"><span class="mdi mdi-server-network"></span> Endpoint</label>
<select id="ocr-active-url" class="llm-active-url-inp" style="appearance: auto; cursor: pointer; padding-right: 25px;">
<option value="">— Select an endpoint —</option>
</select>
<label class="llm-active-lbl"><span class="mdi mdi-brain"></span> Model</label>
<select id="ocr-active-model" aria-label="Active OCR model"><option value="">— fetch models —</option></select>
<button id="ocr-active-refresh" class="btn-secondary" type="button" title="Fetch models"><span class="mdi mdi-refresh"></span></button>
<span id="ocr-active-status" class="llm-active-status"></span>
</div>
<label class="llm-active-lbl" style="margin-top:10px;display:flex;align-items:center;gap:6px;font-weight:normal">
<input type="checkbox" id="ocr-active-enabled"> Enable vision OCR escalation during ingestion
</label>
</div>
<div style="display:flex;justify-content:space-between;align-items:center;margin-top:20px;margin-bottom:12px">
<h4 class="llm-local-cat" style="margin:0;text-transform:uppercase">Local</h4>
<button class="btn-primary" type="button" onclick="openAddEngineDialog('ocr')"><span class="mdi mdi-plus"></span> Add Custom</button>
</div>
<div id="dc-grid-ocr" class="dc-grid" style="margin-bottom:14px"></div>
<div class="llm-info-note" style="margin-top:16px;">Two engines were confirmed live against a real scanned magazine: <strong>Baidu Unlimited-OCR</strong> gives the most accurate text and also labels running headers/footers/page numbers/titles directly (best default). <strong>KDL Frontier Parser nano</strong> answers faster but returns plain text with no layout coordinates or labels &mdash; a usable fallback, not a first choice.</div>
</div><!-- /ocr -->
<!-- ── Add Custom Engine dialog ─────────────────────────────── -->
<dialog id="add-engine-dlg" class="add-engine-dlg">
<div class="add-engine-form">

View File

@ -39,6 +39,9 @@
</button>
<h3 id="mag-doc-title">Magazine</h3>
<div class="mag-bar-actions">
<button type="button" class="btn-sm primary" onclick="magPlayAll()">
<span class="mdi mdi-playlist-play"></span> Play all articles
</button>
<button type="button" class="btn-sm" onclick="magIngest()">
<span class="mdi mdi-refresh"></span> Re-run detection
</button>
@ -70,6 +73,9 @@
<span id="mag-read-by" class="muted"></span>
<span id="mag-read-pages" class="muted"></span>
</div>
<button type="button" class="btn-sm" onclick="magToggleDriving()" title="Large controls for listening without looking at the screen">
<span class="mdi mdi-car-outline"></span> Driving mode
</button>
</div>
<div class="mag-controls">
@ -81,6 +87,11 @@
<span class="mdi mdi-play"></span> Listen
</button>
<span id="mag-read-progress" class="muted" aria-live="polite"></span>
<span id="mag-queue-progress" class="muted" aria-live="polite"></span>
<label class="mag-karaoke-toggle" title="Highlight each word approximately as it is spoken">
<input type="checkbox" id="mag-karaoke-cb" checked onchange="TTVC.magazine.karaoke = this.checked">
Word highlighting
</label>
<div class="mag-pane-tabs" role="group" aria-label="Reading view">
<button type="button" class="btn-sm active" data-mag-pane="reflow"
aria-pressed="true" onclick="magSetPane('reflow')">Article</button>
@ -90,6 +101,7 @@
</div>
<div id="mag-pane-reflow" class="mag-pane">
<div id="mag-read-image" class="mag-read-image" hidden></div>
<div id="mag-article" class="mag-article"></div>
</div>
@ -117,3 +129,28 @@
<div id="mag-podcast-script"></div>
</details>
</div>
<!-- ── Driving mode overlay ─────────────────────────────────────────── -->
<!-- Large controls, minimal text, no interaction required beyond play/
pause/skip -- reuses whatever is already playing rather than its
own playback path. -->
<div id="mag-driving" class="mag-driving" hidden role="dialog" aria-label="Driving mode">
<button type="button" class="mag-driving-exit" onclick="magToggleDriving()" aria-label="Exit driving mode">
<span class="mdi mdi-close"></span>
</button>
<div class="mag-driving-body">
<h2 id="mag-driving-title">Article</h2>
<p id="mag-driving-by" class="muted"></p>
<div id="mag-driving-queue" class="mag-driving-queue" hidden>
<span id="mag-queue-progress-2" class="muted"></span>
</div>
<div class="mag-driving-controls">
<button type="button" class="mag-driving-btn" onclick="magQueueNext()" aria-label="Next article" title="Next article (only in Play all)">
<span class="mdi mdi-skip-next"></span>
</button>
<button type="button" class="mag-driving-btn mag-driving-play" id="mag-driving-play" onclick="magPlay()" aria-label="Play">
<span class="mdi mdi-play"></span>
</button>
</div>
</div>
</div>

View File

@ -4816,6 +4816,17 @@ input[list="lang-codes"] {
/* The reading highlight has to survive both themes and colour-blindness, so it
carries a left rule as well as a tint rather than relying on hue alone. */
.mag-article [data-i].is-current { background: rgba(37,99,235,.16); box-shadow: -3px 0 0 0 var(--accent); }
/* Word-level highlighting is an ESTIMATE (see estimateWordTimings in
magazine.js) -- a distinct, lighter treatment from the sentence highlight
above keeps that difference legible rather than implying frame-accurate
timing. */
.mag-w.is-current { background: var(--accent); color: var(--on-accent); border-radius: 2px; }
.mag-karaoke-toggle { display: inline-flex; align-items: center; gap: 6px; font-size: 13px; color: var(--subtext); }
.mag-read-image { max-width: 68ch; margin: 0 auto 18px; }
.mag-read-image img { width: 100%; height: auto; max-height: 360px; object-fit: contain;
border-radius: var(--radius); border: 1px solid var(--border); background: var(--panel); display: block; }
.mag-page-wrap { position: relative; max-width: 900px; margin: 0 auto; }
.mag-page-wrap img { width: 100%; height: auto; display: block; border: 1px solid var(--border); border-radius: var(--radius); }
@ -4831,8 +4842,31 @@ input[list="lang-codes"] {
.mag-turns li { margin-bottom: 8px; border-radius: 3px; padding: 2px 4px; }
.mag-turns li.is-current { background: rgba(37,99,235,.16); box-shadow: -3px 0 0 0 var(--accent); }
/* ── Driving mode ──────────────────────────────────────────────────────────
Full-screen, minimal-interaction overlay: two oversized controls, large
type, nothing else competing for attention. Reuses whatever is already
playing/queued -- it is a view, not a separate playback path. */
.mag-driving { position: fixed; inset: 0; z-index: 500; background: var(--bg);
display: flex; align-items: center; justify-content: center; }
.mag-driving-exit { position: absolute; top: 20px; right: 20px; width: 52px; height: 52px;
border-radius: 50%; border: 1px solid var(--border); background: var(--surface); color: var(--text);
font-size: 22px; display: flex; align-items: center; justify-content: center; cursor: pointer; }
.mag-driving-exit:focus-visible { outline: 3px solid var(--accent); outline-offset: 2px; }
.mag-driving-body { text-align: center; max-width: 90vw; }
.mag-driving-body h2 { font-size: clamp(28px, 5vw, 48px); margin: 0 0 8px; }
.mag-driving-queue { margin-bottom: 20px; font-size: 18px; }
.mag-driving-controls { display: flex; align-items: center; justify-content: center; gap: 32px; margin-top: 32px; }
.mag-driving-btn { width: 96px; height: 96px; border-radius: 50%; border: none;
background: var(--panel); color: var(--text); font-size: 36px; cursor: pointer;
display: flex; align-items: center; justify-content: center; }
.mag-driving-btn.mag-driving-play { width: 140px; height: 140px; font-size: 56px;
background: var(--accent); color: var(--on-accent); }
.mag-driving-btn:focus-visible { outline: 3px solid var(--accent); outline-offset: 3px; }
.mag-driving-btn:active { transform: scale(0.96); }
@media (prefers-reduced-motion: reduce) {
.mag-page-box { transition: none; }
.mag-driving-btn:active { transform: none; }
}
@media (max-width: 860px) {
.mag-new-grid, .mag-review-head { grid-template-columns: 1fr; }
@ -4841,4 +4875,7 @@ input[list="lang-codes"] {
.mag-block { grid-template-columns: auto 1fr; }
.mag-block-type { display: none; }
.mag-article { font-size: 18px; }
.mag-driving-controls { gap: 20px; }
.mag-driving-btn { width: 76px; height: 76px; font-size: 28px; }
.mag-driving-btn.mag-driving-play { width: 110px; height: 110px; font-size: 44px; }
}

View File

@ -19,6 +19,7 @@ from core.magazine import (
ocr_page,
cluster_running_furniture,
looks_like_real_title,
Article,
BLOCK_AD,
BLOCK_BODY,
BLOCK_BYLINE,
@ -26,9 +27,11 @@ from core.magazine import (
BLOCK_DECK,
BLOCK_FURNITURE,
BLOCK_PULLQUOTE,
BLOCK_SIDEBAR,
BLOCK_TITLE,
Block,
article_text,
assign_images_to_articles,
assign_reading_order,
body_font_size,
build_articles,
@ -39,16 +42,19 @@ from core.magazine import (
estimate_listen_seconds,
find_continuation,
find_repeated_lines,
image_reading_position,
strip_continuation_markers,
group_articles,
looks_like_byline,
mark_advert_pages,
mark_pullquotes,
mark_sidebars,
ocr_quality_score,
parse_contents,
parse_vision_ocr_regions,
score_contents_page,
vision_ocr_page,
vision_ocr_page_chain,
)
PAGE_W, PAGE_H = 595.0, 842.0 # A4 in points
@ -831,5 +837,187 @@ class BuildArticlesTests(unittest.TestCase):
self.assertGreater(arts[0]["estimated_seconds"], 0)
class SidebarDetectionTests(unittest.TestCase):
"""BLOCK_SIDEBAR is geometry-only, like the rest of this module: a boxed
insert is reliably narrower than the page's regular column grid. This is
a heuristic, not a certainty -- review still shows every sidebar block
and lets it be ticked back in with one click."""
def test_a_narrow_third_column_is_marked_a_sidebar(self):
blocks = [
block("main1", 1, 50, 100, 300, 400, "Main body copy. " * 40),
block("side1", 1, 400, 100, 100, 400, "A boxed insert. " * 10),
]
assign_reading_order(blocks, PAGE_W)
mark_sidebars(blocks, PAGE_W)
self.assertEqual(next(b for b in blocks if b.id == "side1").type, BLOCK_SIDEBAR)
self.assertEqual(next(b for b in blocks if b.id == "main1").type, BLOCK_BODY)
def test_two_similarly_wide_columns_are_left_alone(self):
"""A normal two-column layout must not be mistaken for a sidebar."""
blocks = [
block("l", 1, 50, 100, 200, 400, "Left column. " * 30),
block("r", 1, 280, 100, 200, 400, "Right column. " * 30),
]
assign_reading_order(blocks, PAGE_W)
mark_sidebars(blocks, PAGE_W)
self.assertTrue(all(b.type == BLOCK_BODY for b in blocks))
def test_a_single_column_page_has_no_sidebar_candidates(self):
blocks = [block("a", 1, 50, 100, 480, 400, "Full width copy. " * 40)]
assign_reading_order(blocks, PAGE_W)
mark_sidebars(blocks, PAGE_W)
self.assertEqual(blocks[0].type, BLOCK_BODY)
def test_a_title_or_caption_in_the_narrow_column_is_untouched(self):
"""Only BLOCK_BODY is reclassified -- a caption already correctly
typed should not be relabelled just for sitting in a narrow column."""
blocks = [
block("main1", 1, 50, 100, 300, 400, "Main body copy. " * 40),
block("cap", 1, 400, 100, 100, 30, "A caption.", btype=BLOCK_CAPTION),
]
assign_reading_order(blocks, PAGE_W)
mark_sidebars(blocks, PAGE_W)
self.assertEqual(next(b for b in blocks if b.id == "cap").type, BLOCK_CAPTION)
def test_sidebar_text_is_excluded_from_narration_by_default(self):
"""End-to-end: a detected sidebar must not be swept into the
article's narrated blocks the way an advert page already is not."""
blocks = [
block("t", 18, 50, 60, 470, 50, "The Future of Batteries",
size=26.0, btype=BLOCK_TITLE),
block("main1", 18, 50, 150, 300, 400, "Main body copy. " * 40),
block("side1", 18, 400, 150, 100, 400, "Battery myths. " * 20),
]
assign_reading_order(blocks, PAGE_W)
mark_sidebars(blocks, PAGE_W)
arts = group_articles(blocks, [{"title": "The Future of Batteries",
"page": 18, "section": ""}], page_count=20)
self.assertNotIn("side1", arts[0].blocks)
self.assertIn("main1", arts[0].blocks)
class ImageAssignmentTests(unittest.TestCase):
"""Extracted figures are useless until linked to the article they
illustrate -- assign_images_to_articles is that link."""
def test_an_image_on_a_single_article_page_is_assigned_to_it(self):
art = Article(id="a1", title="T", pages=[5], blocks=["b1"])
blocks = [block("b1", 5, 50, 100, 300, 50, "Some text")]
images = [{"id": "img1", "page": 5, "bbox": [50, 200, 200, 150]}]
assign_images_to_articles(images, blocks, [art])
self.assertEqual(art.images, ["img1"])
def test_an_image_on_a_page_with_no_matching_article_is_skipped(self):
art = Article(id="a1", title="T", pages=[5], blocks=[])
images = [{"id": "img1", "page": 9, "bbox": [50, 200, 200, 150]}]
assign_images_to_articles(images, [], [art])
self.assertEqual(art.images, [])
def test_shared_page_image_goes_to_the_nearer_articles_blocks(self):
"""Two articles land on the same page (one ending, one starting) --
the figure must follow whichever article it actually sits beside."""
art_a = Article(id="a1", title="First", pages=[6], blocks=["ba"])
art_b = Article(id="a2", title="Second", pages=[6], blocks=["bb"])
blocks = [
block("ba", 6, 50, 50, 300, 100, "End of the first article."),
block("bb", 6, 50, 500, 300, 100, "Start of the second article."),
]
# The image sits right next to article B's block (y=520 vs y=500).
images = [{"id": "img1", "page": 6, "bbox": [50, 520, 200, 150]}]
assign_images_to_articles(images, blocks, [art_a, art_b])
self.assertEqual(art_a.images, [])
self.assertEqual(art_b.images, ["img1"])
class ImageReadingPositionTests(unittest.TestCase):
def test_image_takes_the_position_of_the_nearest_block_above_it(self):
blocks = [
block("b1", 1, 50, 50, 300, 100, "First paragraph."),
block("b2", 1, 50, 300, 300, 100, "Second paragraph."),
]
blocks[0].reading_order = 0
blocks[1].reading_order = 1
image = {"page": 1, "bbox": [50, 320, 200, 150]}
self.assertEqual(image_reading_position(image, blocks), 1)
def test_image_above_all_text_falls_back_to_the_nearest_block(self):
blocks = [block("b1", 1, 50, 300, 300, 100, "Paragraph.")]
blocks[0].reading_order = 5
image = {"page": 1, "bbox": [50, 10, 200, 150]}
self.assertEqual(image_reading_position(image, blocks), 5)
def test_no_blocks_on_the_page_returns_zero(self):
self.assertEqual(image_reading_position({"page": 9, "bbox": [0, 0, 1, 1]}, []), 0)
class WholePageVisionFallbackTests(unittest.TestCase):
"""KDL Frontier Parser nano, confirmed live, answers with plain text and
no coordinate markers at all -- unlike ocr-baidu-unlimited-ocr's labeled
regions. _whole_page_region is what lets that still produce something
usable rather than being discarded outright."""
def test_plain_text_with_no_markers_becomes_one_page_sized_region(self):
from core.magazine import _whole_page_region
with mock.patch("PIL.Image.open") as mopen:
mopen.return_value.__enter__.return_value.size = (897, 1049)
regions = _whole_page_region(
"Battery technology is entering a new phase. " * 5,
pathlib.Path("/fake.png"))
self.assertEqual(len(regions), 1)
self.assertEqual(regions[0]["label"], "text")
self.assertEqual(regions[0]["bbox"], [0.0, 0.0, 897.0, 1049.0])
def test_garbage_text_is_rejected_not_wrapped(self):
from core.magazine import _whole_page_region
self.assertIsNone(_whole_page_region(". | @)X", pathlib.Path("/fake.png")))
def test_too_short_text_is_rejected(self):
from core.magazine import _whole_page_region
self.assertIsNone(_whole_page_region("Hi.", pathlib.Path("/fake.png")))
def test_empty_text_is_rejected(self):
from core.magazine import _whole_page_region
self.assertIsNone(_whole_page_region("", pathlib.Path("/fake.png")))
class VisionEngineChainTests(unittest.TestCase):
"""The primary engine (labeled regions, best quality) is tried first; a
differently-shaped secondary is a real fallback, not just tesseract."""
def test_primary_success_short_circuits_the_fallback(self):
with mock.patch("core.magazine.vision_ocr_page") as mocked:
mocked.return_value = [{"label": "text", "bbox": [0, 0, 1, 1], "text": "x"}]
out = vision_ocr_page_chain(pathlib.Path("/fake.png"),
[("http://a", "m1"), ("http://b", "m2")])
self.assertEqual(mocked.call_count, 1)
self.assertIsNotNone(out)
def test_primary_failure_falls_through_to_the_secondary(self):
with mock.patch("core.magazine.vision_ocr_page") as mocked:
mocked.side_effect = [None, [{"label": "text", "bbox": [0, 0, 1, 1], "text": "x"}]]
out = vision_ocr_page_chain(pathlib.Path("/fake.png"),
[("http://a", "m1"), ("http://b", "m2")])
self.assertEqual(mocked.call_count, 2)
self.assertIsNotNone(out)
def test_every_engine_failing_returns_none(self):
with mock.patch("core.magazine.vision_ocr_page", return_value=None):
out = vision_ocr_page_chain(pathlib.Path("/fake.png"),
[("http://a", "m1"), ("http://b", "m2")])
self.assertIsNone(out)
def test_blank_engine_entries_are_skipped_without_calling_out(self):
with mock.patch("core.magazine.vision_ocr_page") as mocked:
mocked.return_value = [{"label": "text", "bbox": [0, 0, 1, 1], "text": "x"}]
vision_ocr_page_chain(pathlib.Path("/fake.png"),
[("", ""), ("http://b", "m2")])
self.assertEqual(mocked.call_count, 1)
self.assertEqual(mocked.call_args[0][1], "http://b")
def test_empty_engine_list_returns_none(self):
self.assertIsNone(vision_ocr_page_chain(pathlib.Path("/fake.png"), []))
if __name__ == "__main__":
unittest.main()