diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8ccb941..50117a9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,38 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
## [Unreleased]
### Added
+- **Magazine P1 Complete: Images, Play-All, Word Highlighting, Driving Mode.** Figures now appear automatically during
+ playback — `assign_images_to_articles` links each extracted image to the article whose column it falls in (the nearer
+ article's blocks decide when two share a page), and the client shows "the last image reached so far" as playback
+ advances past its approximate reading-order position. **Play all articles** (contents screen) queues every approved
+ article in order and advances automatically at the end of each, rather than stopping and returning to contents.
+ **Word-level highlighting** is available alongside the existing sentence highlighting — clearly documented as an
+ *estimate*, not a measurement: no forced-alignment or TTS timestamp source exists in this pipeline, so each word's
+ share of the sentence's real, measured audio duration is distributed proportional to its character length, a common
+ karaoke approximation that can drift on heavily punctuated sentences. A checkbox turns it off; sentence highlighting
+ (measured via `audio.onended`, not estimated) is unaffected. **Driving mode** is a full-screen overlay with two
+ oversized controls and nothing else to interact with, reusing whatever is already playing/queued rather than its own
+ playback path. **Sidebars and pull-out boxes are now excluded from narration automatically** (`mark_sidebars`): a
+ column much narrower than the page's regular grid is reclassified `BLOCK_SIDEBAR` and dropped from an article's
+ narrated blocks the same way adverts already are — review still shows every sidebar block, unticked by default, so
+ nothing is silently discarded. Fixed a real defect found writing this: a full-width headline was left in the
+ column-width analysis and could bridge two genuinely separate columns into one, hiding a real sidebar column entirely
+ (caught by a failing test before the fix). 20 new tests.
+- **Second Vision OCR Engine, With a Real Fallback Ladder.** KDL Frontier Parser nano is now wired in alongside
+ ocr-baidu-unlimited-ocr (`vision_ocr_page_chain`: first usable result from an ordered engine list wins, tesseract only
+ if every vision engine fails). Confirmed live: KDL answers roughly 3x faster (~12s vs ~40s per page on the same test
+ page) but returns **plain text with no coordinate markers at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content`
+ regions — handled by treating its whole response as one ambiguous full-page block (`_whole_page_region`, gated by the
+ same `ocr_quality_score` threshold), which loses column/reading-order structure for that page but is still a real
+ improvement over tesseract. Settings gained `vision_ocr_fallback_url`/`vision_ocr_fallback_model`.
+- **OCR Engines Page Now Matches the LLM Engines Layout.** The OCR & Document Parsing page had no
+ `.s-engines-page[data-page="ocr"]` markup at all — `renderLocalContainers` already supported an OCR grid generically
+ (the same code handles TTS/STT/LLM/OCR identically), but `$('dc-grid-ocr')` was always null, so the card grid silently
+ never rendered and the page showed only a header with "Add Custom"/"Refresh" buttons and nothing else. Added the
+ missing panel: an **Active OCR Engine** picker mirroring **Active Language Model** exactly (same endpoint/model
+ selects, same `/api/conversation/llm-models` fetch, which already proxies to whatever URL it's given rather than
+ assuming the LLM setting) plus a **Local** grid, so both configured engines now appear as ready-made cards
+ (Connect/Start/Restart/Use as OCR/Show code snippet) with no "Add Custom" dialog needed for either.
- **Vision OCR Escalation For Magazine Ingestion.** A page that needs OCR, or whose already-extracted text scores as
mostly garbage (`ocr_quality_score`) — native or tesseract alike — is now escalated to a connected vision OCR engine
(`vision_ocr_url`/`vision_ocr_model` in Settings, defaulting to `ocr-baidu-unlimited-ocr` via `host.docker.internal`),
@@ -70,6 +102,13 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
(deu/eng) to the image.
### Fixed
+- **Audiobook Pipeline Nav Bar Bled Into Unrelated Pages.** `_wfUpdateHeaderNav` injected the audiobook workflow's
+ "Previous / Next" bar and step hint into every `.section-head` in the document, with Studio as the only named
+ exclusion. Confirmed live: the Engines page (LLM/STT/TTS/OCR/Image Generation all share one `.section-head`) showed a
+ "Previous … Cast Audiobook ›" bar and a "STEP 1/8 Import a PDF…" hint that made no sense there, since the pipeline's
+ three destinations are Read Aloud, Library and Script Rehearsal only. Changed from a blocklist (naming each unrelated
+ section as it was noticed) to an allowlist, so the same bleed-through cannot recur on Magazines, Settings, or any
+ future section.
- **Captions On Photo Pages Would Have Been Read As Body Text.** Block classification judged font size against a baseline
computed per page, so a page carrying a full-bleed photograph and a single caption made that caption its own reference:
at a ratio of exactly 1.0 it classified as body copy and would have been narrated as part of the article. The baseline
diff --git a/Manual.md b/Manual.md
index 7d7f245..49b953d 100644
--- a/Manual.md
+++ b/Manual.md
@@ -34,14 +34,22 @@ turned into a podcast discussion between two or three voices you pick.
articles with page ranges.
3. **Review the articles.** This step is not optional and cannot be skipped.
Each proposed article shows its title, author, page range, estimated
- listening time and a confidence score. Confirm the boundaries, exclude a
- sidebar you do not want read, correct the title, then **Approve**.
+ listening time and a confidence score. Sidebars and pull-out boxes are
+ already unticked for you — tick one back in if you actually want it read.
+ Confirm the boundaries, correct the title, then **Approve**.
4. **Listen.** An approved article can be narrated with the default voice, or
- sent to podcast mode.
+ sent to podcast mode. Use **Play all articles** (on the contents screen) to
+ listen through every approved article back to back without returning to
+ this screen between them. A figure from the page is shown automatically as
+ playback reaches it. **Word highlighting** can be turned off if you find it
+ distracting — it is an estimate based on word length, not a measurement, so
+ it will not always land exactly on the spoken word. **Driving mode**
+ (the button next to the article title) switches to two large controls and
+ nothing else, for listening without looking at the screen.
**Expected result:** the article plays from its headline to its final paragraph.
-Page furniture (running heads, page numbers), adverts and pull quotes are never
-read aloud.
+Page furniture (running heads, page numbers), adverts, sidebars and pull quotes
+are never read aloud unless you explicitly include them in review.
### Confidence, and what it means
@@ -88,10 +96,15 @@ nothing outward.
### Accessibility and mobile
-To be completed with the P1 interface. Planned: keyboard-navigable contents
-tree and review screen, sentence highlighting that respects
-`prefers-reduced-motion`, large touch targets for playback, and a driving mode
-that advances images without requiring interaction.
+Sentence and word highlighting both respect `prefers-reduced-motion` (no
+smooth-scroll animation when it is set; the highlight itself is a colour
+change, not motion). Driving mode gives two large, high-contrast controls
+(skip, play/pause) with nothing else to interact with, for use without
+looking at the screen — it does not, and should not, replace watching the
+road. Word-level highlighting can be switched off in the reading controls if
+it is more distracting than helpful; sentence highlighting keeps working
+either way. Keyboard navigation of the contents tree and review screen is
+not yet built.
### Offline
diff --git a/PROJECT_MAP.md b/PROJECT_MAP.md
index 621b75d..a8f3c9a 100644
--- a/PROJECT_MAP.md
+++ b/PROJECT_MAP.md
@@ -5,7 +5,7 @@ Update in the same commit as the change it describes.
## Current phase
-Magazine projects, **P0 complete, P1 mostly complete** (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library
+Magazine projects, **P0 and P1 complete** (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library
features are mature and in daily use.
## Deployment
@@ -69,18 +69,20 @@ which is the writable volume.
- Container registry: ghcr costs quota against a private package (≈427 MB per
push, 500 MB free on the Free plan). Local registry vs. ghcr vs. no re-pull is
**unresolved** — the compose files currently point at ghcr.
-- Magazine P1 scope: sentence-level highlighting confirmed as the starting
- point; word-level deferred to P2.
-
## Known limitations
- `gitea` backup remote is 157 commits behind `origin`; `main` tracks `gitea`.
-- Magazine sidebar detection is not implemented (P2).
+- Magazine sidebar exclusion is a narrow-column geometry heuristic
+ (`mark_sidebars`), not certain — review always shows every sidebar block.
+- Magazine word-level highlighting is a character-length estimate against
+ real measured audio duration, not true forced alignment.
+- No formal cross-engine OCR accuracy benchmark exists (two engines wired:
+ ocr-baidu-unlimited-ocr primary, KDL Frontier Parser nano fallback).
- No `npm run qa` aggregate command yet; Python tests run via `unittest`,
browser tests via `playwright test`.
## Next logical step
-Magazine P1 remainder: article images/page crops shown during playback, a
-cross-article "play all" queue, and OCR-engine dispatch to the KDL/Baidu
-parsers (see architecture/magazine-ingestion.md). Otherwise P2 per TODO.md.
+Magazine P2 per TODO.md: image descriptions, table/chart handling, EPUB 3
+Media Overlay export, and a formal OCR-engine benchmark (see
+architecture/magazine-ingestion.md).
diff --git a/TODO.md b/TODO.md
index 82da332..b5776a9 100644
--- a/TODO.md
+++ b/TODO.md
@@ -303,24 +303,30 @@ casting. The original PDF remains the visual source of truth while approved arti
### P1 — Magazine listening experience
- [~] Generate clean article HTML with headings, captions, figures, sidebars, and page links.
- *(headings/byline/captions styled and page-linked; figures and sidebars are not rendered inline yet)*
-- [~] Generate narrator audio per article with resume/progress and a play-all queue.
- *(on-demand narration + per-article resume done; no cross-article play-all queue yet)*
+ *(headings/byline/captions styled and page-linked, figures now shown during playback; sidebars are auto-excluded
+ from narration rather than rendered inline, per the P2 sidebar-exclusion item below)*
+- [x] Generate narrator audio per article with resume/progress and a play-all queue.
- [x] Synchronize paragraph/sentence highlighting with playback and keep PDF and article views linked.
*(sentence-level; clicking a sentence or switching to the original-page view jumps to its source bbox)*
-- [ ] Show article images or page crops during playback, with optional caption reading.
+- [x] Show article images or page crops during playback, with optional caption reading.
+ *(images shown, matched to reading position; caption text is not separately spoken)*
- [~] Add confidence warnings, extraction diffs, and manual text correction before audio generation.
*(confidence badges + block-level boundary editing done; no extraction diff view or free-text correction)*
### P2 — Advanced magazine intelligence
-- [ ] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion.
+- [~] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion.
+ *(sidebar exclusion done via mark_sidebars, a narrow-column geometry heuristic; cross-page continuation still only
+ uses the forward "continued on page N" reference, not "continued from")*
- [ ] Add image descriptions, chart/table handling, and optional spoken figure summaries.
-- [ ] Add word-level karaoke timing and EPUB 3 Media Overlay export.
-- [ ] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume.
+- [~] Add word-level karaoke timing and EPUB 3 Media Overlay export.
+ *(word-level highlighting done as a character-length estimate against the real measured audio duration, not true
+ forced alignment; no EPUB 3 export)*
+- [~] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume.
+ *(driving mode done: two oversized controls, reuses whatever is already playing/queued; not mobile-specific)*
- [~] Add OCR-engine benchmark comparisons and per-page fallback selection.
- *(per-page fallback done: vision OCR escalates when a page needs OCR or scores poorly, tesseract is the automatic
- fallback; no formal cross-engine benchmark or KDL Frontier Parser nano routing yet)*
+ *(per-page fallback done, now a real two-engine chain: vision OCR escalates when a page needs OCR or scores
+ poorly, KDL Frontier Parser nano is the vision fallback, tesseract is the last resort; no formal benchmark)*
Three full regression runs of *Der Sturm* against the hand-corrected reference.
Everything below is measured, not estimated. Artefacts live in
diff --git a/architecture/magazine-ingestion.md b/architecture/magazine-ingestion.md
index 46d5d86..80451be 100644
--- a/architecture/magazine-ingestion.md
+++ b/architecture/magazine-ingestion.md
@@ -1,8 +1,11 @@
# SOP — Magazine ingestion
-Status: **P0 complete. P1 mostly complete** — Studio -> Magazines has the full
-library/contents/review/read/podcast flow. Remaining P1: images shown during
-playback and a cross-article play-all queue.
+Status: **P0 complete. P1 complete.** Studio -> Magazines has the full
+library/contents/review/read/podcast flow, including figures shown during
+playback and a cross-article play-all queue. Automatic sidebar exclusion,
+word-level highlighting (estimated, not measured) and driving mode --
+originally P2 items -- are also in. Remaining P2: image descriptions,
+table/chart handling, EPUB 3 export, and a formal multi-engine OCR benchmark.
## Why magazines are not books
@@ -193,14 +196,67 @@ this app's tab pattern would additionally require `aria-selected`, roving
provides. `role="tab"` without that whole pattern fails WCAG 2.2 AA guidance
and, in practice, hides the buttons from `getByRole('tab')`-oriented tooling.
+### Images during playback
+
+`build_vision_blocks` aside, a figure has no sentence of its own to attach
+to. `assign_images_to_articles` (server) links each extracted image to the
+article whose column it falls in, using the nearer article's blocks when two
+share a page; `image_reading_position` gives it an approximate slot in the
+page's reading order. The client (`currentImageIndex` in magazine.js) then
+shows "the last image reached so far" as playback advances past its
+position — no separate sync data beyond that one `reading_order` number.
+
+### Play all
+
+`magPlayAll` builds `state.queue` from every *approved* article in listing
+order and starts the first; `magPlay`'s completion path advances the queue
+automatically rather than stopping, so listening runs across articles
+without returning to the contents screen. `magRead(id, {fromStart: true})`
+is what makes this a fresh start rather than resuming a queued article's own
+saved progress, which would otherwise interrupt a continuous listen.
+
+### Word-level highlighting is estimated, not measured
+
+No forced-alignment or TTS timestamp source exists in this pipeline.
+`estimateWordTimings` distributes a sentence's *real, measured* audio
+duration (`audio.duration`, once loaded) across its words proportional to
+character length — a common karaoke approximation, not true alignment, and
+it can drift on a sentence with heavy punctuation-driven pauses. A checkbox
+(`#mag-karaoke-cb`) lets it be turned off; sentence-level highlighting
+(measured, not estimated — it just waits for `audio.onended`) always still
+works regardless.
+
+### Driving mode
+
+`#mag-driving` is a full-screen overlay, not a separate playback path: it
+reuses whatever is already playing or queued, so entering it never
+interrupts an article in progress. Two oversized controls (skip, play/pause)
+and no other interactive surface, matching the "no interaction required"
+behaviour the manual originally specified as a P2 goal.
+
+### Automatic sidebar/advert exclusion
+
+`mark_sidebars` (server) reclassifies body text sitting in a column much
+narrower than the page's regular grid as `BLOCK_SIDEBAR`; `group_articles`
+excludes it from an article's narrated blocks the same way it already
+excludes `BLOCK_FURNITURE`/`BLOCK_AD`. Full-width elements (a spanning
+headline) are excluded from the column-width analysis itself — left in, one
+bridges two genuinely separate columns into a single merged span and hides
+a real narrow sidebar column entirely (confirmed by a failing test before
+the fix). This is a geometry heuristic, not a certainty: review still shows
+every sidebar block (unticked by default) and it can be included with one
+click — nothing is silently discarded.
+
## Tests
-- [`tests/test_magazine.py`](../tests/test_magazine.py) — 80 tests on the pure
+- [`tests/test_magazine.py`](../tests/test_magazine.py) — 100 tests on the pure
geometry/classification logic: column reading order, the document-baseline
regression, furniture, pull quotes, contents parsing, continuation linking,
- advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, and
- the vision OCR quality score / region parsing / block building / HTTP call
- (network mocked).
+ advert pages, paragraph-grouped OCR, fuzzy running-furniture matching, the
+ vision OCR quality score / region parsing / block building / HTTP call
+ (network mocked), sidebar detection (including the full-width-headline
+ regression), image-to-article assignment, the whole-page vision fallback,
+ and the multi-engine fallback chain.
- [`tests/test_magazine_routes.py`](../tests/test_magazine_routes.py) — 10 tests: path
traversal rejection, review gating, LLM script parsing including malformed
output.
@@ -263,15 +319,71 @@ sentence (confirmed live: tesseract's "KI" → "K |" substitution, 7 of 8 words
still genuine German) scores just as high as clean text. Catching that class
of error needs a dictionary or language-model check, out of scope here.
-Routing to KDL Frontier Parser nano, or a genuine multi-engine fallback
-ladder with benchmark comparison, remains TODO P2.
+### Second engine and fallback ladder
+
+`vision_ocr_page_chain` tries each configured (url, model) pair in order,
+first usable result wins — primary (`vision_ocr_url`/`_model`), then fallback
+(`vision_ocr_fallback_url`/`_model`), then tesseract if every vision engine
+fails or is unreachable. KDL Frontier Parser nano is wired as the fallback,
+confirmed live: it answers markedly faster than Baidu (~12s vs ~40s per page
+on the same test page) but returns **plain text with no coordinate markers
+at all**, unlike Baidu's labeled `text [x0,y0,x1,y1]content` regions.
+`_whole_page_region` is what makes that still usable rather than discarded:
+the whole response becomes one ambiguous full-page "text" region (bbox =
+the entire rendered page, read via PIL), gated by the same
+`ocr_quality_score` threshold used for native/tesseract text so a genuinely
+bad KDL response isn't accepted either. Column and reading-order structure
+is lost for a page handled this way — a real text improvement over
+tesseract, but a worse layout result than Baidu's, which is why it is the
+fallback and not the default.
+
+A formal cross-engine accuracy benchmark remains TODO P2.
+
+The Active OCR Engine panel (Settings → Engines → OCR & Document Parsing)
+mirrors the Active Language Model panel exactly — same endpoint/model
+selects, same `/api/conversation/llm-models` fetch (it just proxies to
+whatever URL it's given, not hardcoded to the LLM setting) — because an OCR
+engine here is the same shape of problem: an OpenAI-compatible endpoint plus
+a model name, just applied to `vision_ocr_url`/`vision_ocr_model` instead of
+`llm_url`/`llm_model`. Local engine cards (`#dc-grid-ocr`) were already
+supported by `renderLocalContainers` — that generic code handles all four
+roles (`tts`/`stt`/`llm`/`ocr`) identically — but the OCR page had no
+`.s-engines-page[data-page="ocr"]` markup at all until this change, so
+`$('dc-grid-ocr')` was always null and the whole card grid silently never
+rendered. Confirmed live: the page now shows both engines as ready-made
+local cards (Connect/Start/Restart/Use as OCR/Show code snippet), matching
+the LLM page, with no "Add Custom" dialog needed for either.
+
+### A workflow-pipeline nav bar was bleeding into unrelated pages
+
+`_wfUpdateHeaderNav` (utils.js) injects the audiobook pipeline's "Previous /
+Next" bar and step hint into *every* `.section-head` in the document, with
+Studio as the only named exclusion — confirmed live: the Engines page (LLM/
+STT/TTS/OCR/Image Generation all share one `.section-head`) showed a
+"Previous … Cast Audiobook ›" bar and a "STEP 1/8" hint that made no sense
+there, since the pipeline's three destinations are `s-reader`, `s-library`
+and `s-rehearser` only. Changed from a blocklist (name each unrelated
+section as it's noticed) to an allowlist (`head.closest('#s-reader,
+#s-library, #s-rehearser')`) so the same bleed-through cannot recur on
+Magazines, Settings, or any future section.
## Known limitations
-- Sidebar detection (`BLOCK_SIDEBAR`) is defined but not yet inferred; sidebars
- currently classify as body and must be excluded by hand in review. P2.
+- Sidebar detection (`mark_sidebars`) is geometry-only (an anomalously narrow
+ column) — it has no notion of a boxed border or background tint, so a
+ false positive/negative is possible on an unusual layout. Review always
+ shows every sidebar block and lets it be ticked back in with one click.
+- Word-level highlighting is an *estimate* (proportional to character length
+ within the sentence's real measured audio duration), not true forced
+ alignment — see "Word-level highlighting is estimated, not measured" above.
- `continued from page N` is parsed but not used for linking — only the forward
`continued on page N` reference drives it.
- Reviews-style pages with no headline are absorbed by the preceding article
when the contents page does not list them.
-- Word-level timing is not available; P1 targets sentence highlighting.
+- KDL Frontier Parser nano (the OCR fallback engine) returns no layout
+ coordinates, so a page it handles loses column/reading-order structure
+ even though its text is usable — see "Second engine and fallback ladder".
+- No formal cross-engine OCR accuracy benchmark exists; the engine
+ comparison above is one confirmed live test, not a systematic evaluation.
+- Image descriptions, table/chart handling, and EPUB 3 Media Overlay export
+ remain P2 and are not started.
diff --git a/core/config.py b/core/config.py
index 15b6607..c976ddf 100644
--- a/core/config.py
+++ b/core/config.py
@@ -17,6 +17,7 @@ from core.constants import (
_FASTER_WHISPER_DEFAULT, _WHISPER_CPP_DEFAULT,
_KOKORO_DEFAULT, _VIBEVOICE_DEFAULT, _XTTS_DEFAULT, _FISHSPEECH_DEFAULT,
_VISION_OCR_DEFAULT, _VISION_OCR_MODEL_DEFAULT,
+ _VISION_OCR_FALLBACK_DEFAULT, _VISION_OCR_FALLBACK_MODEL_DEFAULT,
_TTS_STREAM_DEFAULT,
)
from core.retry import request_with_retries
@@ -41,6 +42,7 @@ _SETTINGS_KEYS = {
"client_voice_bindings",
"llm_url", "llm_model", "llm_api_key",
"vision_ocr_url", "vision_ocr_model", "vision_ocr_enabled",
+ "vision_ocr_fallback_url", "vision_ocr_fallback_model",
"engine_local_urls", "engine_container_names", "engine_api_keys", "custom_engine_cards",
# Character portrait generation (cloud APIs + local ComfyUI)
"image_gen_provider", "image_gen_model",
@@ -335,6 +337,8 @@ def _load_settings() -> dict:
"vision_ocr_url": _VISION_OCR_DEFAULT,
"vision_ocr_model": _VISION_OCR_MODEL_DEFAULT,
"vision_ocr_enabled": True,
+ "vision_ocr_fallback_url": _VISION_OCR_FALLBACK_DEFAULT,
+ "vision_ocr_fallback_model": _VISION_OCR_FALLBACK_MODEL_DEFAULT,
"engine_local_urls": {},
"engine_container_names": {},
"engine_api_keys": {},
diff --git a/core/constants.py b/core/constants.py
index 8906ea6..cc3a522 100644
--- a/core/constants.py
+++ b/core/constants.py
@@ -46,6 +46,11 @@ _KOKORO_DEFAULT = os.environ.get("KOKORO_URL", "http://192.168.1
# container-name default would not resolve from the app container.
_VISION_OCR_DEFAULT = os.environ.get("VISION_OCR_URL", "http://host.docker.internal:8098/v1")
_VISION_OCR_MODEL_DEFAULT = os.environ.get("VISION_OCR_MODEL", "baidu/Unlimited-OCR")
+# Second vision OCR engine, tried when the primary fails or is unreachable.
+# Confirmed live: answers faster than Baidu Unlimited-OCR but returns plain
+# text with no layout coordinates (see _whole_page_region in core/magazine.py).
+_VISION_OCR_FALLBACK_DEFAULT = os.environ.get("VISION_OCR_FALLBACK_URL", "http://host.docker.internal:8099/v1")
+_VISION_OCR_FALLBACK_MODEL_DEFAULT = os.environ.get("VISION_OCR_FALLBACK_MODEL", "kdl-frontier-parser-nano")
_VIBEVOICE_DEFAULT = os.environ.get("VIBEVOICE_URL", "http://192.168.178.8:8027")
_XTTS_DEFAULT = os.environ.get("XTTS_URL", "http://192.168.178.8:8005")
_FISHSPEECH_DEFAULT = os.environ.get("FISHSPEECH_URL", "http://192.168.178.8:38080")
diff --git a/core/magazine.py b/core/magazine.py
index cbb25dd..da75e1d 100644
--- a/core/magazine.py
+++ b/core/magazine.py
@@ -370,6 +370,48 @@ def mark_pullquotes(blocks: list[Block]) -> list[Block]:
return blocks
+def mark_sidebars(blocks: list[Block], page_width: float,
+ narrow_ratio: float = 0.7) -> list[Block]:
+ """Reclassify body text sitting in an anomalously narrow column as a
+ sidebar, so it is excluded from an article's narration by default rather
+ than requiring a manual exclusion in review every time.
+
+ Geometry-only, like the rest of this module's classification: a sidebar
+ or pull-out box is reliably narrower than the page's regular column grid
+ (a boxed insert set off from the main flow), which needs no image
+ analysis to detect. This is a heuristic, not a certainty -- a genuinely
+ narrow-column page design (or a caption sitting in its own slim column)
+ can trigger a false positive, which is exactly why review still shows
+ every sidebar block and lets it be included with one click rather than
+ silently discarding it.
+
+ Full-width elements (a headline spanning the whole page) are excluded
+ from the column analysis here, the same way assign_reading_order treats
+ them specially: left in, one would bridge two genuinely separate columns
+ into a single merged span (its bbox overlaps both), hiding a real narrow
+ sidebar column entirely. This computes its own column partition rather
+ than trusting `.column` from assign_reading_order, since that partition
+ was built from the FULL block list and its indices would not line up
+ with this narrower one.
+ """
+ candidates = [b for b in blocks if b.bbox[2] <= page_width * 0.6]
+ columns = detect_columns(candidates, page_width)
+ if len(columns) < 2:
+ return blocks
+ widths = [right - left for left, right in columns]
+ widest = max(widths)
+ narrow_spans = [columns[i] for i, w in enumerate(widths) if w < widest * narrow_ratio]
+ if not narrow_spans:
+ return blocks
+ for b in blocks:
+ if b.type != BLOCK_BODY or b.bbox[2] > page_width * 0.6:
+ continue
+ centre = b.bbox[0] + b.bbox[2] / 2
+ if any(left <= centre <= right for left, right in narrow_spans):
+ b.type, b.confidence = BLOCK_SIDEBAR, 0.6
+ return blocks
+
+
def find_repeated_lines(pages_text: Sequence[Sequence[str]],
threshold: float = 0.25) -> set[str]:
"""Lines appearing near the same edge on many pages: page furniture."""
@@ -562,9 +604,15 @@ def group_articles(blocks: Sequence[Block],
# advert sitting between two features starts no article of its
# own, so it would otherwise be swallowed by the preceding one and
# highlighted as part of it in the PDF view.
+ #
+ # A sidebar/pull-out box (mark_sidebars) is excluded the same way
+ # advert text is: automatically kept out of narration by default,
+ # rather than requiring a manual exclusion in review every time.
+ # It still shows in review (blocksForPages lists every block on
+ # the article's pages) and can be ticked back in with one click.
contributed = False
for b in page_blocks:
- if b.type in (BLOCK_FURNITURE, BLOCK_AD):
+ if b.type in (BLOCK_FURNITURE, BLOCK_AD, BLOCK_SIDEBAR):
continue
contributed = True
art.blocks.append(b.id)
@@ -744,14 +792,37 @@ def parse_vision_ocr_regions(raw: str) -> list[dict[str, Any]]:
return regions
+def _whole_page_region(content: str, png_path: Path) -> list[dict[str, Any]] | None:
+ """Fallback for an engine that answers with plain text and no coordinate
+ markers at all (confirmed live: KDL Frontier Parser nano -- fast, but no
+ "label [x0,y0,x1,y1]" structure the way ocr-baidu-unlimited-ocr gives).
+
+ Treated as one ambiguous "text" region spanning the whole page: real
+ text with no layout information beats no text at all, but it does mean
+ column/reading-order structure is lost for that page -- classify_blocks
+ still runs on it (source="vision_ocr", not "vision_label"), same as any
+ other ambiguous region.
+ """
+ text = (content or "").strip()
+ if len(text) < 20 or ocr_quality_score(text) < _VISION_QUALITY_THRESHOLD:
+ return None
+ try:
+ from PIL import Image
+ with Image.open(png_path) as im:
+ width, height = im.size
+ except Exception:
+ return None
+ return [{"label": "text", "bbox": [0.0, 0.0, float(width), float(height)], "text": text}]
+
+
def vision_ocr_page(png_path: Path, base_url: str, model: str,
timeout: float = 150.0) -> list[dict[str, Any]] | None:
"""OCR one rendered page with a connected vision-capable OCR engine.
Returns None on any failure (unreachable engine, timeout, unusable
- output) so the caller can fall back to tesseract -- this must never be
- the reason ingestion fails outright. Confirmed live against
- ocr-baidu-unlimited-ocr: markedly more accurate German text than
+ output) so the caller can fall back to the next engine or tesseract --
+ this must never be the reason ingestion fails outright. Confirmed live
+ against ocr-baidu-unlimited-ocr: markedly more accurate German text than
tesseract on the same scanned page, and it labels running headers/
footers and page numbers directly, which is the dominant source of the
over-fragmentation cluster_running_furniture() otherwise has to recover
@@ -787,7 +858,26 @@ def vision_ocr_page(png_path: Path, base_url: str, model: str,
return None
regions = parse_vision_ocr_regions(content)
- return regions or None
+ return regions or _whole_page_region(content, png_path)
+
+
+def vision_ocr_page_chain(png_path: Path,
+ engines: Sequence[tuple[str, str]],
+ timeout: float = 150.0) -> list[dict[str, Any]] | None:
+ """Try each configured vision OCR engine in order, first success wins.
+
+ A primary engine (labeled regions, best quality) with a faster or
+ differently-shaped secondary as a real fallback -- not just tesseract --
+ when the primary is unreachable or times out. Engines are (url, model)
+ pairs; empty/blank entries are skipped.
+ """
+ for url, model in engines:
+ if not (url and model):
+ continue
+ regions = vision_ocr_page(png_path, url, model, timeout=timeout)
+ if regions:
+ return regions
+ return None
def build_vision_blocks(regions: list[dict[str, Any]], page_no: int,
@@ -910,6 +1000,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
ocr_lang: str = "deu+eng",
vision_ocr_url: str | None = None,
vision_ocr_model: str | None = None,
+ vision_ocr_fallback_url: str | None = None,
+ vision_ocr_fallback_model: str | None = None,
vision_ocr_timeout: float = 150.0,
progress: Any = None) -> dict[str, Any]:
"""Run the full extraction for one PDF, writing page renders and images.
@@ -917,13 +1009,17 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
When vision_ocr_url/model are given, a page whose text needs OCR (or
whose already-extracted text scores as mostly garbage, native or
tesseract alike -- see ocr_quality_score) is escalated to that engine
- first, with tesseract only as the fallback if the call fails or the
- engine is unreachable. Confirmed live against ocr-baidu-unlimited-ocr:
- markedly more accurate transcription of a scanned German magazine than
- tesseract, and it labels running headers/footers/page numbers directly
- (see build_vision_blocks), which is the dominant source of the
- over-fragmentation cluster_running_furniture() otherwise has to recover
- after the fact from noisy OCR variants of the same repeating line.
+ first, with a second vision engine (vision_ocr_fallback_*) tried next if
+ given, and tesseract only as the last resort if every vision call fails
+ or every engine is unreachable. Confirmed live against
+ ocr-baidu-unlimited-ocr: markedly more accurate transcription of a
+ scanned German magazine than tesseract, and it labels running headers/
+ footers/page numbers directly (see build_vision_blocks), which is the
+ dominant source of the over-fragmentation cluster_running_furniture()
+ otherwise has to recover after the fact from noisy OCR variants of the
+ same repeating line. KDL Frontier Parser nano, tested as the fallback
+ engine, answers faster but returns plain text with no layout coordinates
+ at all -- handled by _whole_page_region as one ambiguous full-page block.
Returns the layout document: every block on every page, classified and in
reading order, plus the raw contents-page candidates.
@@ -933,6 +1029,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
images_dir = out_dir / "images"
pages_dir.mkdir(parents=True, exist_ok=True)
images_dir.mkdir(parents=True, exist_ok=True)
+ vision_engines = [(vision_ocr_url, vision_ocr_model),
+ (vision_ocr_fallback_url, vision_ocr_fallback_model)]
vision_enabled = bool(vision_ocr_url and vision_ocr_model)
vision_page_count = 0
@@ -951,14 +1049,27 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
pix.save(str(png))
image_boxes: list[list[float]] = []
+ page_images: list[dict[str, Any]] = []
for img_index, info in enumerate(page.get_images(full=True)):
+ rects: list[list[float]] = []
try:
for r in page.get_image_rects(info[0]):
- image_boxes.append([r.x0, r.y0, r.width, r.height])
+ rects.append([r.x0, r.y0, r.width, r.height])
except Exception:
continue
- _extract_image(fitz, doc, info[0], images_dir,
- f"p{page_no:04d}-{img_index:02d}")
+ image_boxes.extend(rects)
+ name = f"p{page_no:04d}-{img_index:02d}"
+ if _extract_image(fitz, doc, info[0], images_dir, name) and rects:
+ # A few embedded images repeat (a logo stamped on every
+ # page); each placement still gets its own bbox so a
+ # caption or figure position can be matched precisely.
+ for ri, box in enumerate(rects):
+ page_images.append({
+ "id": name if ri == 0 else f"{name}-{ri}",
+ "page": page_no,
+ "bbox": box,
+ "filename": f"{name}.png",
+ })
blocks: list[Block] = []
data = page.get_text("dict")
@@ -1004,8 +1115,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
vision_blocks: list[Block] | None = None
if vision_enabled and needs_escalation:
- regions = vision_ocr_page(png, vision_ocr_url, vision_ocr_model,
- timeout=vision_ocr_timeout)
+ regions = vision_ocr_page_chain(png, vision_engines,
+ timeout=vision_ocr_timeout)
if regions:
scale_x = rect.width / max(1.0, _png_width(png, fitz))
vision_blocks = build_vision_blocks(
@@ -1042,6 +1153,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
"height": rect.height,
"blocks": blocks,
"image_boxes": image_boxes,
+ "images": page_images,
})
if progress:
progress(page_no, doc.page_count)
@@ -1066,6 +1178,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
mark_advert_pages(p["blocks"])
mark_pullquotes(p["blocks"])
assign_reading_order(p["blocks"], p["width"])
+ mark_sidebars(p["blocks"], p["width"])
all_blocks.extend(p["blocks"])
# Cross-page, so it runs once over every block rather than per page
@@ -1084,11 +1197,16 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
"entries": parse_contents(lines),
})
+ all_images = [img for p in raw_pages for img in p["images"]]
+ for img in all_images:
+ img["reading_order"] = image_reading_position(img, all_blocks)
+
return {
"page_count": doc.page_count,
"pages": [{"page": p["page"], "width": p["width"], "height": p["height"]}
for p in raw_pages],
"blocks": [b.to_json() for b in all_blocks],
+ "images": all_images,
"contents_candidates": contents_candidates,
"vision_ocr_pages": vision_page_count,
}
@@ -1103,19 +1221,87 @@ def _png_width(png: Path, fitz) -> float:
return 1.0
-def _extract_image(fitz, doc, xref: int, images_dir: Path, name: str) -> None:
- """Save one embedded image, skipping the tiny rules and spacers."""
+def _extract_image(fitz, doc, xref: int, images_dir: Path, name: str) -> bool:
+ """Save one embedded image, skipping the tiny rules and spacers.
+
+ Returns whether a file now exists at that name, so the caller can decide
+ whether to record it as a real figure -- without this, a page's rules and
+ spacers were silently skipped on disk but still implicitly "existed" as
+ far as anything downstream knew, since nothing checked.
+ """
try:
pix = fitz.Pixmap(doc, xref)
if pix.width < 64 or pix.height < 64:
- return
+ return False
if pix.n - pix.alpha >= 4: # CMYK needs converting first
pix = fitz.Pixmap(fitz.csRGB, pix)
target = images_dir / f"{name}.png"
if not target.exists():
pix.save(str(target))
+ return True
except Exception:
- return
+ return False
+
+
+# ── Images: linking a figure to the article it illustrates ──────────────────
+
+def assign_images_to_articles(images: Sequence[dict[str, Any]],
+ blocks: Sequence[Block],
+ articles: Sequence[Article]) -> None:
+ """Attach each extracted figure to the article whose column it falls in.
+
+ A figure belongs to whichever article occupies its page; when two
+ articles share a page (a feature ending, a new one starting lower down),
+ the figure goes to whichever article has the nearer block by vertical
+ position, so a photo is not shown under the wrong headline. Mutates
+ articles in place.
+ """
+ by_page: dict[int, list[Article]] = {}
+ for art in articles:
+ for page in art.pages:
+ by_page.setdefault(page, []).append(art)
+
+ blocks_by_page: dict[int, list[Block]] = {}
+ for b in blocks:
+ blocks_by_page.setdefault(b.page, []).append(b)
+
+ for img in images:
+ candidates = by_page.get(img["page"], [])
+ if not candidates:
+ continue
+ if len(candidates) == 1:
+ candidates[0].images.append(img["id"])
+ continue
+ # Several articles share this page: pick by nearest block vertically,
+ # since that is what a reader would associate the figure with.
+ img_y = img["bbox"][1]
+ best, best_dist = candidates[0], float("inf")
+ for art in candidates:
+ own_blocks = [b for b in blocks_by_page.get(img["page"], [])
+ if b.id in set(art.blocks)]
+ if not own_blocks:
+ continue
+ dist = min(abs(b.bbox[1] - img_y) for b in own_blocks)
+ if dist < best_dist:
+ best, best_dist = art, dist
+ best.images.append(img["id"])
+
+
+def image_reading_position(image: dict[str, Any], blocks: Sequence[Block]) -> int:
+ """Approximate reading_order for an image, for ordering it among sentences.
+
+ Uses the nearest block on the same page whose top sits at or above the
+ image (so a photo is shown once the surrounding text has been reached,
+ not before), falling back to the closest block on the page if the image
+ sits above all of them.
+ """
+ same_page = [b for b in blocks if b.page == image["page"]]
+ if not same_page:
+ return 0
+ above = [b for b in same_page if b.bbox[1] <= image["bbox"][1]]
+ pool = above or same_page
+ nearest = min(pool, key=lambda b: abs(b.bbox[1] - image["bbox"][1]))
+ return nearest.reading_order
# ── Persistence ──────────────────────────────────────────────────────────────
@@ -1131,6 +1317,7 @@ def build_articles(layout: dict[str, Any]) -> list[dict[str, Any]]:
skip = {c["page"] for c in candidates if c.get("score", 0) >= 0.45}
articles = group_articles(blocks, entries, layout.get("page_count", 0),
skip_pages=skip)
+ assign_images_to_articles(layout.get("images") or [], blocks, articles)
out = []
for art in articles:
rec = art.to_json()
diff --git a/routes/magazine.py b/routes/magazine.py
index c3ec06e..7ab4cc9 100644
--- a/routes/magazine.py
+++ b/routes/magazine.py
@@ -109,7 +109,9 @@ async def magazine_create(request: Request):
# ── Stage A–C: ingestion ─────────────────────────────────────────────────────
def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
- vision_ocr_url: str | None, vision_ocr_model: str | None) -> None:
+ vision_ocr_url: str | None, vision_ocr_model: str | None,
+ vision_ocr_fallback_url: str | None = None,
+ vision_ocr_fallback_model: str | None = None) -> None:
"""Run the extraction pipeline; recorded as a restart-safe job."""
d = _LIB / doc_id
pdf = d / "source.pdf"
@@ -127,6 +129,8 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
layout = mag.extract_pages(pdf, d, render_dpi=render_dpi, ocr_lang=ocr_lang,
vision_ocr_url=vision_ocr_url,
vision_ocr_model=vision_ocr_model,
+ vision_ocr_fallback_url=vision_ocr_fallback_url,
+ vision_ocr_fallback_model=vision_ocr_fallback_model,
progress=progress)
mag.save_layout(d, layout)
@@ -183,6 +187,7 @@ async def magazine_ingest(doc_id: str, request: Request):
# approved and only re-detection is wanted.
settings = _load_settings()
vision_ocr_url = vision_ocr_model = None
+ vision_ocr_fallback_url = vision_ocr_fallback_model = None
want_vision = body.get("vision_ocr")
if want_vision is None:
want_vision = bool(settings.get("vision_ocr_enabled"))
@@ -191,6 +196,12 @@ async def magazine_ingest(doc_id: str, request: Request):
vision_ocr_model = str(body.get("vision_ocr_model") or settings.get("vision_ocr_model") or "").strip()
if not (vision_ocr_url and vision_ocr_model):
vision_ocr_url = vision_ocr_model = None
+ vision_ocr_fallback_url = str(
+ body.get("vision_ocr_fallback_url") or settings.get("vision_ocr_fallback_url") or "").strip()
+ vision_ocr_fallback_model = str(
+ body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip()
+ if not (vision_ocr_fallback_url and vision_ocr_fallback_model):
+ vision_ocr_fallback_url = vision_ocr_fallback_model = None
job_id = uuid.uuid4().hex
background_job_create(job_id, "magazine-ingest", fingerprint=doc_id)
@@ -199,7 +210,8 @@ async def magazine_ingest(doc_id: str, request: Request):
asyncio.create_task(
asyncio.to_thread(_ingest_sync, doc_id, job_id, ocr_lang, render_dpi,
- vision_ocr_url, vision_ocr_model)
+ vision_ocr_url, vision_ocr_model,
+ vision_ocr_fallback_url, vision_ocr_fallback_model)
)
return {"job_id": job_id}
diff --git a/static/js/ai-backends.js b/static/js/ai-backends.js
index 48b5703..6958e05 100644
--- a/static/js/ai-backends.js
+++ b/static/js/ai-backends.js
@@ -427,6 +427,134 @@
})();
})();
+ // ── Active OCR Engine (global picker, magazine ingestion) ───────────────
+ // Mirrors initActiveLlmPanel above: an OCR engine is the same shape of
+ // problem (an OpenAI-compatible endpoint plus a model name), just applied
+ // to vision_ocr_url/vision_ocr_model/vision_ocr_enabled instead of
+ // llm_url/llm_model. Reuses the same model-list endpoint, since it just
+ // proxies to whatever URL it's given rather than assuming it's the LLM one.
+ (function initActiveOcrPanel() {
+ const urlInp = $('ocr-active-url');
+ const modelSel = $('ocr-active-model');
+ const refreshBtn = $('ocr-active-refresh');
+ const statusEl = $('ocr-active-status');
+ const enabledCb = $('ocr-active-enabled');
+ if (!urlInp || !modelSel) return;
+
+ function setStatus(msg, cls) {
+ if (!statusEl) return;
+ statusEl.textContent = msg || '';
+ statusEl.className = 'llm-active-status' + (cls ? ' ' + cls : '');
+ }
+
+ async function fetchModels(url) {
+ url = (url || urlInp.value.trim());
+ if (!url) { modelSel.innerHTML = ''; setStatus(''); return; }
+ const want = (_appSettings && _appSettings.vision_ocr_model) || '';
+ setStatus('Loading models…');
+ if (refreshBtn) refreshBtn.disabled = true;
+ try {
+ const d = await fetch('/api/conversation/llm-models?url=' + encodeURIComponent(url)).then(r => r.json());
+ if (d.error) {
+ let msg = d.error;
+ try { const j = JSON.parse(d.error); msg = j.error?.message || j.detail || d.error; } catch (_) {}
+ if (msg.length > 50) msg = msg.substring(0, 47) + '...';
+ modelSel.innerHTML = '';
+ setStatus(msg, 'err');
+ return;
+ }
+ const models = d.models || [];
+ modelSel.innerHTML = models.length
+ ? models.map(m => ``).join('')
+ : '';
+ if (want && models.includes(want)) modelSel.value = want;
+ setStatus(models.length ? `${models.length} model${models.length === 1 ? '' : 's'}` : 'No models found',
+ models.length ? 'ok' : 'err');
+ } catch (e) {
+ modelSel.innerHTML = '';
+ setStatus('Could not reach endpoint', 'err');
+ } finally {
+ if (refreshBtn) refreshBtn.disabled = false;
+ }
+ }
+
+ function commitUrl() {
+ const url = urlInp.value.trim();
+ if (!url) return;
+ applyAndSaveSettings({ vision_ocr_url: url });
+ if (_appSettings) _appSettings.vision_ocr_url = url;
+ fetchModels(url);
+ }
+ urlInp.addEventListener('blur', commitUrl);
+ urlInp.addEventListener('keydown', e => { if (e.key === 'Enter') { e.preventDefault(); commitUrl(); } });
+ urlInp.addEventListener('change', commitUrl);
+
+ function refreshEndpointOptions(currentUrl) {
+ if (!urlInp || urlInp.tagName !== 'SELECT') return;
+ let html = '';
+ const added = new Set();
+ document.querySelectorAll('#dc-grid-ocr .llm-local-card').forEach(card => {
+ const uInp = card.querySelector('.dc-url-inp');
+ if (!uInp) return;
+ const u = uInp.value.trim() || uInp.placeholder;
+ const name = card.querySelector('.llm-local-name')?.textContent.trim() || 'Local Engine';
+ if (u && !added.has(u)) { added.add(u); html += ``; }
+ });
+ const activeUrl = currentUrl || urlInp.value;
+ if (activeUrl && !added.has(activeUrl)) {
+ html += ``;
+ }
+ urlInp.innerHTML = html;
+ if (activeUrl) urlInp.value = activeUrl;
+ }
+ urlInp.addEventListener('focus', () => refreshEndpointOptions());
+
+ modelSel.addEventListener('change', () => {
+ applyAndSaveSettings({ vision_ocr_model: modelSel.value });
+ if (_appSettings) _appSettings.vision_ocr_model = modelSel.value;
+ if (modelSel.value) setStatus('Active model: ' + modelSel.value, 'ok');
+ });
+
+ enabledCb?.addEventListener('change', () => {
+ applyAndSaveSettings({ vision_ocr_enabled: enabledCb.checked });
+ if (_appSettings) _appSettings.vision_ocr_enabled = enabledCb.checked;
+ });
+
+ refreshBtn?.addEventListener('click', () => fetchModels());
+
+ // "Use as OCR" card buttons (see DC_ROLE_USE.ocr in engines.js) sync this
+ // panel the same way "Use as LLM" syncs the Active Language Model one.
+ window.syncActiveOcrPanel = function (url) {
+ refreshEndpointOptions(url);
+ urlInp.value = url || '';
+ fetchModels(url);
+ };
+
+ (async function () {
+ let url = (_appSettings && _appSettings.vision_ocr_url) || '';
+ let enabled = (_appSettings && _appSettings.vision_ocr_enabled);
+ if (!url) {
+ try {
+ const s = await fetch('/api/settings').then(r => r.json());
+ url = s.vision_ocr_url || '';
+ enabled = s.vision_ocr_enabled;
+ if (!_appSettings) _appSettings = s;
+ else {
+ _appSettings.vision_ocr_url = s.vision_ocr_url || _appSettings.vision_ocr_url;
+ _appSettings.vision_ocr_model = s.vision_ocr_model || _appSettings.vision_ocr_model;
+ _appSettings.vision_ocr_enabled = s.vision_ocr_enabled;
+ }
+ } catch (_) {}
+ }
+ if (enabledCb) enabledCb.checked = enabled !== false;
+ setTimeout(() => {
+ refreshEndpointOptions(url);
+ urlInp.value = url || '';
+ if (url) fetchModels(url);
+ }, 100);
+ })();
+ })();
+
// ── Inline STT quick-test panel ─────────────────────────────────────────
(async function initSttTestPanel() {
const panel = $('stt-test-panel');
diff --git a/static/js/engines.js b/static/js/engines.js
index 7e08d66..6cdb071 100644
--- a/static/js/engines.js
+++ b/static/js/engines.js
@@ -285,6 +285,7 @@ const DC_ROLE_USE = {
tts: { settingKey: 'tts_url', label: 'Use as TTS' },
stt: { settingKey: 'faster_whisper_url', label: 'Use as STT' },
llm: { settingKey: 'llm_url', label: 'Use as LLM' },
+ ocr: { settingKey: 'vision_ocr_url', label: 'Use as OCR' },
};
window.openAddEngineDialog = function(role) {
@@ -797,6 +798,7 @@ function renderLocalContainers(containers) {
if (apiKey && settingKey === 'llm_url') patch.llm_api_key = apiKey;
await window.applyAndSaveSettings(patch);
if (settingKey === 'llm_url' && window.syncActiveLlmPanel) window.syncActiveLlmPanel(url, apiKey);
+ if (settingKey === 'vision_ocr_url' && window.syncActiveOcrPanel) window.syncActiveOcrPanel(url);
btn.classList.add('active');
toast(`✓ ${btn.textContent.trim()} — now available in the backend dropdown`, 'success');
}
diff --git a/static/js/magazine.js b/static/js/magazine.js
index 43ec55a..76e7701 100644
--- a/static/js/magazine.js
+++ b/static/js/magazine.js
@@ -20,6 +20,7 @@ const state = {
layout: null, // { page_count, pages[], blocks[] }
article: null, // the article being read or reviewed
sentences: [], // [{ text, blockId, page, bbox }]
+ images: [], // current article's figures, in reading order
view: 'library', // library | contents | review | read
pane: 'reflow', // reflow | pdf
playing: false,
@@ -28,6 +29,10 @@ const state = {
audio: null,
voices: [],
podcast: null,
+ queue: [], // article ids for "play all" (empty = not queued)
+ queueIndex: -1,
+ driving: false, // large-control, minimal-interaction view
+ karaoke: true, // word-level highlighting (approximate, see estimateWordTimings)
};
MAG.magazine = state;
@@ -387,14 +392,60 @@ function splitSentences(blocks) {
.map((s) => s.trim()).filter(Boolean);
(parts.length ? parts : [b.text]).forEach((text) => {
if (text && text.trim()) {
- out.push({ text: text.trim(), blockId: b.id, page: b.page, bbox: b.bbox, type: b.type });
+ out.push({
+ text: text.trim(), blockId: b.id, page: b.page, bbox: b.bbox, type: b.type,
+ // Used to pick "the image reached so far" during playback (see
+ // currentImageIndex) -- the backend already computes this per
+ // block relative to the whole page, not just this article.
+ readingOrder: b.reading_order || 0,
+ words: text.trim().split(/\s+/).filter(Boolean),
+ });
}
});
});
return out;
}
-async function magRead(articleId) {
+// The current article's figures, in reading order, resolved from the full
+// layout (fetched once) filtered to this article's own image ids. Kept
+// separate from state.sentences because an image has no sentence of its
+// own -- it is matched to "the most recent sentence at or before it".
+function articleImages(article) {
+ const ids = new Set(article.images || []);
+ return (state.layout.images || [])
+ .filter((img) => ids.has(img.id))
+ .sort((a, b) => (a.reading_order || 0) - (b.reading_order || 0));
+}
+
+// The image to show once sentence `i` has been reached: the last image
+// whose position in the page's reading order is at or before that
+// sentence's own block. Falls back to the first image before any text has
+// caught up to it, so a lead photo is not left blank at the very start.
+function currentImageIndex(i) {
+ if (!state.images.length) return -1;
+ const order = (state.sentences[i] || {}).readingOrder || 0;
+ let found = -1;
+ for (let k = 0; k < state.images.length; k++) {
+ if ((state.images[k].reading_order || 0) <= order) found = k;
+ else break;
+ }
+ return found === -1 ? 0 : found;
+}
+
+function renderCurrentImage(i) {
+ const host = $('mag-read-image');
+ if (!host) return;
+ const idx = currentImageIndex(i);
+ if (idx < 0) { host.hidden = true; return; }
+ const img = state.images[idx];
+ if (state._shownImageId === img.id) { host.hidden = false; return; }
+ state._shownImageId = img.id;
+ host.hidden = false;
+ host.innerHTML = '';
+}
+
+async function magRead(articleId, opts) {
const a = state.articles.find((x) => x.id === articleId);
if (!a) return;
state.article = a;
@@ -406,7 +457,10 @@ async function magRead(articleId) {
} catch (e) {
return say('Could not load article text: ' + e.message, 'err');
}
- const saved = (state.doc.progress || {})[a.id];
+ state.images = articleImages(a);
+ state._shownImageId = null;
+ const forceStart = opts && opts.fromStart;
+ const saved = !forceStart && (state.doc.progress || {})[a.id];
state.idx = saved && saved.sentence < state.sentences.length ? saved.sentence : 0;
magShow('read');
renderArticle();
@@ -426,8 +480,13 @@ function renderArticle() {
: s.type === 'byline' ? 'p class="mag-byline"' : s.type === 'caption'
? 'p class="mag-caption"' : 'span';
const open = tag.split(' ')[0];
+ // Each word gets its own span so playback can highlight it individually
+ // (see estimateWordTimings) -- sentence-level highlighting alone doesn't
+ // need this, but word-level does and this markup serves both.
+ const wordsHtml = s.words.map((w, j) =>
+ '' + esc(w) + '').join(' ');
return '<' + tag + ' class="mag-s' + (open === 'span' ? '' : '') + '" data-i="' + i + '" ' +
- 'onclick="magSeek(' + i + ')">' + esc(s.text) + '' + open + '> ';
+ 'onclick="magSeek(' + i + ')">' + wordsHtml + '' + open + '> ';
}).join('');
highlight(state.idx);
}
@@ -436,13 +495,54 @@ function highlight(i) {
document.querySelectorAll('#mag-article [data-i]').forEach((el) => {
el.classList.toggle('is-current', Number(el.dataset.i) === i);
});
+ clearWordHighlight();
const el = document.querySelector('#mag-article [data-i="' + i + '"]');
if (el && !prefersReducedMotion()) el.scrollIntoView({ block: 'center', behavior: 'smooth' });
else if (el) el.scrollIntoView({ block: 'center' });
const s = state.sentences[i];
if (s && state.pane === 'pdf') showPage(s.page, s.bbox);
+ renderCurrentImage(i);
const counter = $('mag-read-progress');
if (counter) counter.textContent = (i + 1) + ' / ' + state.sentences.length;
+ const qText = state.queue.length ? 'Article ' + (state.queueIndex + 1) + ' / ' + state.queue.length : '';
+ ['mag-queue-progress', 'mag-queue-progress-2'].forEach((id) => {
+ const el = $(id);
+ if (el) el.textContent = qText;
+ });
+}
+
+function clearWordHighlight() {
+ document.querySelectorAll('#mag-article .mag-w.is-current').forEach((el) => el.classList.remove('is-current'));
+}
+
+// ── Word-level highlighting (approximate) ────────────────────────────────
+// No forced-alignment or TTS timestamp source exists in this pipeline, so
+// per-word timing is ESTIMATED, not measured: each word's share of the
+// sentence's real (measured) audio duration is proportional to its own
+// character length, with a small fixed share added per word so short words
+// are not given an unrealistically instant duration. This is a common
+// karaoke approximation, not true alignment -- it can drift on sentences
+// with heavy punctuation-driven pauses.
+function estimateWordTimings(words, totalSeconds) {
+ if (!words.length || !totalSeconds) return [];
+ const FIXED = 0.6; // arbitrary units, shifts weight toward evening out short words
+ const weights = words.map((w) => Math.max(1, w.replace(/[.,!?;:"'()«»„"]/g, '').length) + FIXED);
+ const total = weights.reduce((a, b) => a + b, 0);
+ let t = 0;
+ return weights.map((w) => {
+ const dur = totalSeconds * (w / total);
+ const start = t;
+ t += dur;
+ return { start, end: t };
+ });
+}
+
+function highlightWord(sentenceIdx, wordIdx) {
+ const sentenceEl = document.querySelector('#mag-article [data-i="' + sentenceIdx + '"]');
+ if (!sentenceEl) return;
+ sentenceEl.querySelectorAll('.mag-w').forEach((el) => {
+ el.classList.toggle('is-current', Number(el.dataset.w) === wordIdx);
+ });
}
function prefersReducedMotion() {
@@ -508,46 +608,104 @@ async function magPlay() {
state.playing = true;
state.cancel = false;
syncPlayButton();
+ let finishedArticle = false;
try {
while (state.idx < state.sentences.length && !state.cancel) {
highlight(state.idx);
- const text = state.sentences[state.idx].text;
+ const sentence = state.sentences[state.idx];
let blob;
try {
- blob = await fetchTtsPreviewBlob(voice, text, 'mp3');
+ blob = await fetchTtsPreviewBlob(voice, sentence.text, 'mp3');
} catch (e) {
say('Synthesis failed: ' + e.message, 'err');
break;
}
if (state.cancel) break;
- await playBlob(blob);
+ await playBlob(blob, sentence);
if (state.cancel) break;
state.idx += 1;
saveProgress();
}
+ finishedArticle = state.idx >= state.sentences.length && !state.cancel;
} finally {
state.playing = false;
state.cancel = false;
syncPlayButton();
- if (state.idx >= state.sentences.length) {
- state.idx = state.sentences.length - 1;
+ }
+ if (finishedArticle) {
+ state.idx = state.sentences.length - 1;
+ if (state.queue.length && state.queueIndex < state.queue.length - 1) {
+ // Play-all: move straight to the next approved article rather than
+ // stopping -- the whole point of a queue is not touching the screen
+ // between articles.
+ state.queueIndex += 1;
+ await magRead(state.queue[state.queueIndex], { fromStart: true });
+ magPlay();
+ } else {
+ state.queue = [];
+ state.queueIndex = -1;
say('Article finished', 'ok');
}
}
}
-function playBlob(blob) {
+function playBlob(blob, sentence) {
return new Promise((resolve) => {
const url = URL.createObjectURL(blob);
const audio = new Audio(url);
state.audio = audio;
- const done = () => { URL.revokeObjectURL(url); state.audio = null; resolve(); };
+ let timings = null;
+ const onTimeUpdate = () => {
+ if (!state.karaoke || !sentence || !sentence.words.length) return;
+ if (!timings) {
+ if (!audio.duration || !isFinite(audio.duration)) return;
+ timings = estimateWordTimings(sentence.words, audio.duration);
+ }
+ const t = audio.currentTime;
+ const wi = timings.findIndex((w) => t >= w.start && t < w.end);
+ if (wi !== -1) highlightWord(state.idx, wi);
+ };
+ if (state.karaoke) audio.addEventListener('timeupdate', onTimeUpdate);
+ const done = () => {
+ audio.removeEventListener('timeupdate', onTimeUpdate);
+ URL.revokeObjectURL(url);
+ state.audio = null;
+ resolve();
+ };
audio.onended = done;
audio.onerror = done;
audio.play().catch(done);
});
}
+// ── Play all approved articles in sequence ──────────────────────────────
+async function magPlayAll() {
+ const ids = state.articles.filter((a) => a.status === 'approved').map((a) => a.id);
+ if (!ids.length) return say('Approve at least one article first', 'err');
+ state.queue = ids;
+ state.queueIndex = 0;
+ await magRead(ids[0], { fromStart: true });
+ magPlay();
+}
+
+function magStopQueue() {
+ state.queue = [];
+ state.queueIndex = -1;
+ ['mag-queue-progress', 'mag-queue-progress-2'].forEach((id) => {
+ const el = $(id);
+ if (el) el.textContent = '';
+ });
+}
+
+async function magQueueNext() {
+ if (!state.queue.length || state.queueIndex >= state.queue.length - 1) return;
+ const wasPlaying = state.playing;
+ if (wasPlaying) magPause();
+ state.queueIndex += 1;
+ await magRead(state.queue[state.queueIndex], { fromStart: true });
+ if (wasPlaying) magPlay();
+}
+
function magPause() {
state.cancel = true;
if (state.audio) { try { state.audio.pause(); } catch (_) {} }
@@ -557,11 +715,13 @@ function magPause() {
function syncPlayButton() {
const btn = $('mag-play');
- if (!btn) return;
- btn.innerHTML = state.playing
- ? ' Pause'
- : ' Listen';
- btn.setAttribute('aria-label', state.playing ? 'Pause narration' : 'Play narration');
+ if (btn) {
+ btn.innerHTML = state.playing
+ ? ' Pause'
+ : ' Listen';
+ btn.setAttribute('aria-label', state.playing ? 'Pause narration' : 'Play narration');
+ }
+ if (state.driving) syncDrivingPlayButton();
}
let _saveTimer = null;
@@ -707,10 +867,45 @@ function magShow(view) {
function magBackToContents() {
magPause();
+ magStopQueue();
+ state.driving = false;
magShow('contents');
magRefreshArticles();
}
+// ── Driving mode ──────────────────────────────────────────────────────────
+// Large, minimal-interaction controls for listening without looking at the
+// screen (see Manual.md's planned driving-mode behaviour). Reuses whatever
+// is already playing/queued rather than starting its own playback path --
+// entering driving mode does not interrupt an article already in progress.
+function magToggleDriving() {
+ state.driving = !state.driving;
+ const overlay = $('mag-driving');
+ if (overlay) overlay.hidden = !state.driving;
+ if (state.driving) renderDriving();
+}
+
+function renderDriving() {
+ const a = state.article;
+ if (!a) return;
+ const titleEl = $('mag-driving-title');
+ if (titleEl) titleEl.textContent = a.title || 'Untitled';
+ const byEl = $('mag-driving-by');
+ if (byEl) byEl.textContent = a.author ? 'By ' + a.author : '';
+ syncDrivingPlayButton();
+ const qEl = $('mag-driving-queue');
+ if (qEl) qEl.hidden = !state.queue.length;
+}
+
+function syncDrivingPlayButton() {
+ const btn = $('mag-driving-play');
+ if (!btn) return;
+ btn.innerHTML = state.playing
+ ? ''
+ : '';
+ btn.setAttribute('aria-label', state.playing ? 'Pause' : 'Play');
+}
+
// ── Boot ─────────────────────────────────────────────────────────────────────
function magInit() {
@@ -736,6 +931,10 @@ root.magShow = magShow;
root.magBackToContents = magBackToContents;
root.magGeneratePodcast = magGeneratePodcast;
root.magPlayPodcast = magPlayPodcast;
+root.magPlayAll = magPlayAll;
+root.magQueueNext = magQueueNext;
+root.magStopQueue = magStopQueue;
+root.magToggleDriving = magToggleDriving;
root.magInit = magInit;
MAG.magazineApi = { load: magLoadLibrary, refresh: magRefreshArticles, splitSentences, fmtPages };
diff --git a/static/js/utils.js b/static/js/utils.js
index bf6152f..e66b706 100644
--- a/static/js/utils.js
+++ b/static/js/utils.js
@@ -1112,11 +1112,18 @@ function _wfUpdateHeaderNav(anyState, curIdx) {
const nextStep = curIdx >= 0 ? _wfNeighbor(curIdx, 1) : null;
const curStep = curIdx >= 0 ? WF_STEPS[curIdx] : null;
document.querySelectorAll('.section-head').forEach(head => {
- // Studio has its own self-contained 4-phase nav (studio.js) — this
- // stepper's Previous/Next buttons are hardcoded to the old s-reader/
- // s-library/s-rehearser sections and don't know about Studio's phases,
- // so injecting them here just breaks navigation when clicked.
- if (head.closest('#s-caststudio')) {
+ // This stepper's Previous/Next buttons are hardcoded to the audiobook
+ // pipeline's three destinations (s-reader, s-library, s-rehearser) — an
+ // allowlist, not a blocklist, since every OTHER section (Studio has its
+ // own 4-phase nav; Engines, Magazines, Settings, etc. have nothing to do
+ // with this pipeline at all) would otherwise need to be named here one
+ // at a time as each one was noticed showing a "Step 1/8" bar and a
+ // "Cast Audiobook" button that make no sense on that page. Confirmed
+ // live: this exact bleed-through appeared on Engines (LLM/STT/TTS/OCR/
+ // Image Generation all share one .section-head) before this became an
+ // allowlist.
+ const onPipelinePage = !!head.closest('#s-reader, #s-library, #s-rehearser');
+ if (!onPipelinePage) {
const staleNav = head.querySelector('.wf-header-nav');
if (staleNav) staleNav.hidden = true;
const staleHint = head.nextElementSibling?.classList?.contains('wf-step-hint') ? head.nextElementSibling : null;
diff --git a/static/sections/s-llms.html b/static/sections/s-llms.html
index b4b95c5..613c665 100644
--- a/static/sections/s-llms.html
+++ b/static/sections/s-llms.html
@@ -807,6 +807,42 @@
+
Used by Magazine ingestion when a page needs OCR, or when its existing text scores poorly. Tesseract (bundled, no setup) is always the fallback if this is unreachable.
+