feat(magazine): add figure/chart/table description generation and spoken narration
Some checks are pending
Test and SonarQube / quality (push) Waiting to run

Adds a "Describe" action per figure in article review: sends the cropped
image to the configured vision engine (with fallback) for a type-aware
one- or two-sentence description -- a chart gets a prompt about what it
measures and its trend, a table about headings and what stands out, etc.
Editable before approval. A new "Read figure descriptions" toggle in the
reading controls speaks these where they occur; purely additive, off by
default, and a figure with no description behaves exactly as before.

New core/magazine.py describe_visual_region()/_chain() reuse the same
reliability guards as page OCR (repetition-loop detection, sentence-
boundary truncation). New routes: POST .../images/{id}/describe
(background job), PATCH .../images/{id} (manual edit), descriptions=1 on
the article-text endpoint.

Live-verified end to end against the ingested c't KI Wissen 2026
magazine. Also found and documented a real, config-dependent limitation:
the currently-configured primary vision engine is transcription-only and
returns empty content for any descriptive prompt (confirmed with three
different prompts) -- not a code defect, but description generation
needs a genuinely caption-capable engine reachable as the fallback.

251 tests passing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-09-14 17:39:57 +02:00
parent 1a2ea421c4
commit c81ae28da2
12 changed files with 733 additions and 15 deletions

View File

@ -5,6 +5,18 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
---
## [1.29.0] — 2026-09-14
### Added
- **Magazine figure descriptions.** In article review, "Describe" under a figure/chart/table/code block sends it to the configured vision engine (with fallback) for a one- or two-sentence description in plain language, type-aware (a chart gets a "what it measures and its trend" prompt, a table gets "headings and what stands out," etc.). Editable before approval. A new "Read figure descriptions" toggle in the reading controls speaks generated descriptions where they occur in the article; purely additive — off by default, and a figure with no generated description behaves exactly as before.
- New `core/magazine.py` functions `describe_visual_region()`/`describe_visual_region_chain()` (same reliability guards as page OCR: repetition-loop detection, sentence-boundary truncation), `POST /api/magazine/{doc}/images/{id}/describe` (background job) and `PATCH /api/magazine/{doc}/images/{id}` (manual edit) routes, `descriptions=1` query param on the article-text endpoint.
### Verified
- Live end-to-end against the ingested c't KI Wissen 2026 magazine (a real chart, article-008): ingest → Describe → persisted in layout.json → appears as a `figure_description` block in article text only when requested.
- Found and documented a real config-dependent limitation, not a code defect: the currently-configured primary vision engine (baidu/Unlimited-OCR) is transcription-only — it returns empty content for any descriptive prompt regardless of phrasing (confirmed with three different prompts) — so description generation currently depends on a caption-capable fallback engine being configured and running. See Manual.md's Errors table and PROJECT_MAP.md's Known limitations.
---
## [1.28.2] — 2026-09-14
### Fixed

View File

@ -63,13 +63,25 @@ turned into a podcast discussion between two or three voices you pick.
engine missed a region entirely, **"Draw it"** lets you click-drag a box
directly on the page image and add it as text (typed or OCRed) or as a
new figure. Confirm the boundaries, correct the title, then **Approve**.
Optionally, click **"Describe"** under a figure, chart, table or code block
to have the configured vision engine generate a one- or two-sentence
description for listeners who cannot see the image — e.g. what a chart
measures and its general trend, or a table's row/column headings and
what stands out. You can edit the generated text, or write your own from
scratch, before approving the article. Generating a description changes
nothing about existing narration by itself — it is only spoken if you
also turn on "Read figure descriptions" in step 4.
4. **Listen.** An approved article can be narrated with the default voice, or
sent to podcast mode. Use **Play all articles** (on the contents screen) to
listen through every approved article back to back without returning to
this screen between them. A figure from the page is shown automatically as
playback reaches it. **Word highlighting** can be turned off if you find it
distracting — it is an estimate based on word length, not a measurement, so
it will not always land exactly on the spoken word. **Driving mode**
it will not always land exactly on the spoken word. Turn on **"Read figure
descriptions"** in the reading controls to have any figure descriptions
generated during review spoken aloud where they occur in the article;
figures without a generated description are silently skipped, exactly as
before this option existed. **Driving mode**
(the button next to the article title) switches to two large controls and
nothing else, for listening without looking at the screen.
@ -113,14 +125,18 @@ unreviewed boundaries may include a neighbouring article or an advert.
| "Article has too little text" | Under 200 characters — usually a stub. | Check the boundaries in review. |
| A figure box sits over the wrong part of the page | The OCR engine grounded the figure loosely. | Press **"Fix figure boundaries"**; if the graphic is too pale to detect, draw the region by hand. |
| "Fix figure boundaries" reports nothing corrected | Every box already matched its page, or none could be placed with confidence. | Nothing is wrong; correct any remaining box by drawing it. |
| "No vision OCR engine configured" (Describe) | `vision_ocr_url`/`vision_ocr_model` are empty. | Set a vision engine in Settings. |
| "The vision engine returned nothing usable for this figure" | The configured engine (and fallback, if set) returned an empty or degenerate response for this image. Some vision models are transcription-only and cannot produce open-ended descriptions at all, however the prompt is phrased — confirmed with a pure-OCR model that answers its own "Text Recognition:" prompt but returns nothing for any descriptive one. | Configure a genuinely caption-capable vision model as the fallback (`vision_ocr_fallback_url`/`vision_ocr_fallback_model`), or write the description by hand. |
### Privacy
Ingestion, OCR and storage are entirely local to the container. Podcast mode is
the exception: **the article's text is sent to whatever LLM `llm_url` points
at.** If that is a hosted API rather than a model on your own network, the
article leaves your machine. Narration uses your local TTS stack and sends
nothing outward.
one exception: **the article's text is sent to whatever LLM `llm_url` points
at.** Generating a figure description is the other: **the cropped image is
sent to whatever `vision_ocr_url` (and, on failure, `vision_ocr_fallback_url`)
points at.** If either is a hosted API rather than a model on your own
network, that content leaves your machine. Narration uses your local TTS
stack and sends nothing outward.
### Accessibility and mobile

View File

@ -90,6 +90,13 @@ which is the writable volume.
real measured audio duration, not true forced alignment.
- No formal cross-engine OCR accuracy benchmark exists (two engines wired:
ocr-baidu-unlimited-ocr primary, KDL Frontier Parser nano fallback).
- Figure description generation (Magazine review, "Describe") needs a
genuinely caption-capable vision model — confirmed live that the primary
engine (baidu/Unlimited-OCR) is transcription-only and returns empty
content for any descriptive prompt, regardless of phrasing (three prompts
tried). Falls through to `vision_ocr_fallback_url`/`_model`, which worked
correctly once its container was started — that container
(`ocr-kdl-frontier-parser-nano`) is not running by default.
- Figure boundaries are re-derived from the page render (`snap_visual_bboxes`)
because vision OCR grounds figures loosely; it corrects roughly 100 of the
147 figure regions on `c't KI Wissen 2026` and deliberately declines the
@ -108,6 +115,7 @@ which is the writable volume.
## Next logical step
Magazine P2 per TODO.md: image descriptions, table/chart handling, EPUB 3
Media Overlay export, and a formal OCR-engine benchmark (see
architecture/magazine-ingestion.md).
Magazine P2 per TODO.md: image descriptions shipped; remaining — table/chart
handling depth, EPUB 3 Media Overlay export, and a formal OCR-engine
benchmark (see architecture/magazine-ingestion.md). Then Character system,
then Audiobook speaker attribution, per the user's stated priority order.

View File

@ -400,7 +400,14 @@ casting. The original PDF remains the visual source of truth while approved arti
- [~] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion.
*(sidebar exclusion done via mark_sidebars, a narrow-column geometry heuristic; cross-page continuation still only
uses the forward "continued on page N" reference, not "continued from")*
- [ ] Add image descriptions, chart/table handling, and optional spoken figure summaries.
- [x] Add image descriptions, chart/table handling, and optional spoken figure summaries.
*(per-figure "Describe" in review generates a type-aware description — chart/table/code/figure each get their
own prompt — via the configured vision engine with fallback; editable before approval; spoken only if "Read
figure descriptions" is enabled. Live-verified against a real chart: the configured primary vision engine
turned out to be a pure-OCR model that returns empty content for any non-transcription prompt regardless of
phrasing — confirmed with three different prompts — so description generation only works when a genuinely
caption-capable model is reachable as the fallback; documented as a config-dependent limitation in Manual.md,
not a code defect.)*
- [~] Add word-level karaoke timing and EPUB 3 Media Overlay export.
*(word-level highlighting done as a character-length estimate against the real measured audio duration, not true
forced alignment; no EPUB 3 export)*

View File

@ -1 +1 @@
1.28.2
1.29.0

View File

@ -1140,6 +1140,140 @@ def vision_ocr_page_chain(png_path: Path,
return None
# One line per visual type -- a chart needs "what trend", a table needs
# "what's listed", not the generic "what's shown" that suits a photo. Every
# variant still asks for one or two plain sentences: this is read aloud
# between the surrounding paragraphs, not a caption someone reads at their
# own pace, so it has to stay short regardless of how much detail the image
# could support.
_DESCRIBE_PROMPTS = {
BLOCK_CHART: (
"This image is a chart or graph from a magazine article. In one or "
"two plain sentences, for a listener who cannot see it, describe "
"what kind of chart it is, what it measures, and the general trend "
"or most notable values. Do not invent a number you cannot actually "
"read in the image. No markdown, no preamble -- the description "
"text only."
),
BLOCK_TABLE: (
"This image is a table from a magazine article. In one or two "
"plain sentences, for a listener who cannot see it, summarise what "
"it lists -- do not read out individual cells. No markdown, no "
"preamble -- the description text only."
),
BLOCK_CODE: (
"This image is a code listing or code box from a magazine article. "
"In one plain sentence, for a listener who cannot see it, describe "
"what it appears to show (the language and general purpose) -- do "
"not read the code line by line. No markdown, no preamble -- the "
"description text only."
),
BLOCK_FIGURE: (
"This image is a photo or illustration from a magazine article. In "
"one or two plain sentences, for a listener who cannot see it, "
"describe what it shows, including any text visible in the image "
"itself. No markdown, no preamble -- the description text only."
),
}
def describe_visual_region(image_path: Path, region_type: str, base_url: str,
model: str, timeout: float = 90.0) -> str | None:
"""Describe one already-cropped figure/chart/table/code image in plain
language, for optional inclusion in an article's spoken narration.
A deliberately different task from vision_ocr_page(): that prompt
explicitly forbids inventing a description ("Images, figures, and
charts may have empty transcription; do not invent descriptions") --
correct for transcription, wrong here, where a description is exactly
what is wanted. Reuses the same engine (already vision-capable and
already configured) with its own prompt instead, and the same
reliability guards -- a repeated-phrase decoding loop is exactly as
possible describing a busy chart as transcribing one.
Returns None on any failure (unreachable engine, timeout, empty or
degenerate response) so the caller can try a fallback engine or leave
the figure without a description -- never partway through one.
"""
try:
image_b64 = base64.b64encode(image_path.read_bytes()).decode("ascii")
except OSError:
return None
base_url = base_url.strip().rstrip("/")
if not urlsplit(base_url).path:
base_url += "/v1"
prompt = _DESCRIBE_PROMPTS.get(region_type, _DESCRIBE_PROMPTS[BLOCK_FIGURE])
unlimited_ocr = "unlimited-ocr" in model.lower()
payload = {
"model": model,
"messages": [{"role": "user", "content": [
{"type": "text", "text": prompt},
{"type": "image_url", "image_url": {
"url": f"data:image/png;base64,{image_b64}",
}},
]}],
"max_tokens": 300,
"temperature": 0.2,
}
if unlimited_ocr:
# Same anti-repetition settings as vision_ocr_page -- same model,
# same decoding-loop risk, regardless of the prompt asked of it.
payload.update({
"skip_special_tokens": False,
"ngram_size": 35,
"window_size": 128,
"images_config": {"image_mode": "gundam"},
})
try:
resp = request_with_retries(
"POST", f"{base_url.rstrip('/')}/chat/completions",
json=payload, timeout=timeout, attempts=2,
)
resp.raise_for_status()
content = resp.json()["choices"][0]["message"]["content"]
except requests.exceptions.RequestException as exc:
status = getattr(getattr(exc, "response", None), "status_code", None)
logging.getLogger(__name__).warning("Figure description request failed (%s%s)",
type(exc).__name__, f", HTTP {status}" if status else "")
return None
except (ValueError, KeyError, IndexError, TypeError):
logging.getLogger(__name__).warning("Figure description returned an invalid completion response")
return None
if not isinstance(content, str):
return None
text = content.strip().strip("`").strip()
if not text or len(text) < 8:
return None
if _looks_like_repetition_loop(text):
logging.getLogger(__name__).warning("Figure description returned a repeated phrase loop; discarding response")
return None
# A long response is either the model ignoring "one or two sentences" or
# a loop this pass's own repetition check didn't happen to catch --
# either way, narration wants a short aside, not a paragraph. Cut at a
# sentence boundary rather than mid-word.
limit = 600
if len(text) > limit:
cut = text.rfind(". ", 0, limit)
text = text[:cut + 1] if cut > limit // 2 else text[:limit].rstrip() + "…"
return text
def describe_visual_region_chain(image_path: Path, region_type: str,
engines: Sequence[tuple[str, str]],
timeout: float = 90.0) -> str | None:
"""Try each configured vision engine in order, first usable description
wins -- same shape as vision_ocr_page_chain."""
for url, model in engines:
if not (url and model):
continue
text = describe_visual_region(image_path, region_type, url, model, timeout=timeout)
if text:
return text
return None
def build_vision_blocks(regions: list[dict[str, Any]], page_no: int,
scale_x: float, id_prefix: str) -> list[Block]:
"""Turn one page's vision OCR regions into Blocks.
@ -2438,12 +2572,26 @@ def load_articles(doc_dir: Path) -> list[dict[str, Any]]:
return []
BLOCK_FIGURE_DESCRIPTION = "figure_description"
def article_text(layout: dict[str, Any], article: dict[str, Any],
include_captions: bool = False) -> list[dict[str, Any]]:
include_captions: bool = False,
include_descriptions: bool = False) -> list[dict[str, Any]]:
"""The narratable text of one article, in reading order.
Returned per block so the player can highlight the source region and jump
to the originating PDF page.
A figure/chart/table/code region has no text block of its own to
narrate -- see describe_visual_region() -- so an already-described
image belonging to this article is synthesised into a
BLOCK_FIGURE_DESCRIPTION entry at that image's own reading position when
`include_descriptions` is set. Optional and off by default: generating
a description costs a vision-model call the user may not have wanted
for every figure, and a description read aloud mid-article changes the
listening experience enough that it should be an explicit choice, not a
silent side effect of one being available.
"""
wanted = set(article.get("blocks") or [])
types = set(NARRATABLE) | ({BLOCK_CAPTION} if include_captions else set())
@ -2452,5 +2600,16 @@ def article_text(layout: dict[str, Any], article: dict[str, Any],
for b in layout.get("blocks", [])
if b["id"] in wanted and b["type"] in types
]
if include_descriptions:
wanted_images = set(article.get("images") or [])
for img in layout.get("images", []):
description = (img.get("description") or "").strip()
if img["id"] not in wanted_images or not description:
continue
blocks.append({
"id": f"desc-{img['id']}", "page": img["page"], "bbox": img["bbox"],
"type": BLOCK_FIGURE_DESCRIPTION, "text": description,
"reading_order": img.get("reading_order", 0),
})
blocks.sort(key=lambda b: (b["page"], b["reading_order"]))
return blocks

View File

@ -422,7 +422,7 @@ async def magazine_update_article(doc_id: str, article_id: str, request: Request
@router.get("/api/magazine/{doc_id}/articles/{article_id}/text")
async def magazine_article_text(doc_id: str, article_id: str,
captions: int = 0):
captions: int = 0, descriptions: int = 0):
"""The narratable text of one article, per block, in reading order."""
d, _ = _require_magazine(doc_id)
article_id = _article_id(article_id)
@ -431,7 +431,8 @@ async def magazine_article_text(doc_id: str, article_id: str,
if target is None:
raise HTTPException(404, "Article not found")
blocks = mag.article_text(mag.load_layout(d), target,
include_captions=bool(captions))
include_captions=bool(captions),
include_descriptions=bool(descriptions))
return {"article": target, "blocks": blocks}
@ -842,6 +843,113 @@ async def magazine_image(doc_id: str, name: str):
return FileResponse(path, media_type="image/png")
# ── Figure descriptions ───────────────────────────────────────────────────────
def _image_id(image_id: str) -> str:
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", image_id or ""):
raise HTTPException(400, "Bad image id")
return image_id
def _describe_image_sync(doc_id: str, job_id: str, image_id: str,
vision_ocr_url: str, vision_ocr_model: str,
vision_ocr_fallback_url: str | None,
vision_ocr_fallback_model: str | None) -> None:
"""Describe one figure/chart/table/code image, for optional inclusion in
the article's spoken narration (see article_text's include_descriptions).
Restart-safe for the same reason "OCR again" is: a vision-model call can
take tens of seconds.
"""
d = _LIB / doc_id
try:
layout = mag.load_layout(d)
if not layout:
raise FileNotFoundError("No layout yet -- run ingestion first")
image = next((i for i in layout.get("images", []) if i["id"] == image_id), None)
if image is None:
raise FileNotFoundError("Image not found")
png = d / "images" / str(image.get("filename") or f"{image_id}.png")
if not png.is_file():
raise FileNotFoundError("Figure image not found on disk")
background_job_update(job_id, stage="describe", detail="Describing figure")
engines = [(vision_ocr_url, vision_ocr_model),
(vision_ocr_fallback_url, vision_ocr_fallback_model)]
text = mag.describe_visual_region_chain(png, image.get("type", mag.BLOCK_FIGURE), engines)
if not text:
raise RuntimeError("The vision engine returned nothing usable for this figure")
image["description"] = text
mag.save_layout(d, layout)
background_job_update(job_id, stage="done", done=True,
detail="Figure described", result={"image_id": image_id, "description": text})
except Exception as exc: # noqa: BLE001 - surfaced to UI
background_job_update(job_id, stage="failed", done=True, error=str(exc)[:1000])
@router.post("/api/magazine/{doc_id}/images/{image_id}/describe")
async def magazine_describe_image(doc_id: str, image_id: str, request: Request):
"""Describe one figure/chart/table/code image with the configured vision
engine, in plain language for a listener who cannot see it. Purely
additive: the description is stored alongside the image and only ever
spoken if the reader explicitly turns on "Read figure descriptions" --
generating one changes nothing about existing narration on its own.
"""
d, _ = _require_magazine(doc_id)
image_id = _image_id(image_id)
layout = mag.load_layout(d)
if not layout or not any(i["id"] == image_id for i in layout.get("images", [])):
raise HTTPException(404, "Image not found")
body = {}
try:
body = await request.json()
except Exception:
pass
settings = _load_settings()
vision_ocr_url = str(body.get("vision_ocr_url") or settings.get("vision_ocr_url") or "").strip()
vision_ocr_model = str(body.get("vision_ocr_model") or settings.get("vision_ocr_model") or "").strip()
if not (vision_ocr_url and vision_ocr_model):
raise HTTPException(400, "No vision OCR engine configured -- set one in Settings")
vision_ocr_fallback_url = str(
body.get("vision_ocr_fallback_url") or settings.get("vision_ocr_fallback_url") or "").strip() or None
vision_ocr_fallback_model = str(
body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip() or None
job_id = uuid.uuid4().hex
background_job_create(job_id, "magazine-describe-image", fingerprint=f"{doc_id}:{image_id}")
asyncio.create_task(
asyncio.to_thread(_describe_image_sync, doc_id, job_id, image_id,
vision_ocr_url, vision_ocr_model,
vision_ocr_fallback_url, vision_ocr_fallback_model)
)
return {"job_id": job_id}
@router.patch("/api/magazine/{doc_id}/images/{image_id}")
async def magazine_update_image_description(doc_id: str, image_id: str, request: Request):
"""Set or clear a figure's description by hand -- trusted verbatim, the
same as any other text the user types directly (see add_manual_region).
Not for correcting a bad boundary, only the spoken description text.
"""
d, _ = _require_magazine(doc_id)
image_id = _image_id(image_id)
layout = mag.load_layout(d)
if not layout:
raise HTTPException(404, "Document not ingested yet")
image = next((i for i in layout.get("images", []) if i["id"] == image_id), None)
if image is None:
raise HTTPException(404, "Image not found")
try:
body = await request.json()
except Exception:
raise HTTPException(400, "Invalid JSON body")
if "description" not in body:
raise HTTPException(400, "Nothing to update -- expected a 'description' field")
description = str(body.get("description") or "").strip()[:2000]
image["description"] = description
mag.save_layout(d, layout)
return {"image_id": image_id, "description": description}
# ── Podcast mode ─────────────────────────────────────────────────────────────
_PODCAST_SYSTEM = (

View File

@ -822,6 +822,7 @@ function renderReview() {
'<span class="mag-block-text">' + (owner ? 'Belongs to Article ' + owner.index + ': ' + esc(owner.title)
: on ? 'Shown during playback for this article.' : 'Not linked to any article on this page.') + '</span>' +
'</div>' +
magDescribeMarkup(r.id, r.data.description) +
'</div>';
return;
}
@ -878,6 +879,79 @@ function magReviewToggleImage(id, checked) {
}
}
// ── Figure descriptions ──────────────────────────────────────────────────
// Purely additive: generating or editing a description never changes
// whether the image itself is shown or included (see magReviewToggleImage
// above) -- it only supplies the text "Read figure descriptions" narrates
// if the reader turns that on. Never spoken by itself.
function magDescribeMarkup(id, description) {
const safeId = esc(id);
const has = !!(description || '').trim();
return '<div class="mag-describe">' +
'<button type="button" class="btn-sm" onclick="magDescribeImage(\'' + safeId + '\')" id="mag-describe-btn-' + safeId + '">' +
'<span class="mdi mdi-text-to-speech" aria-hidden="true"></span> ' + (has ? 'Redo description' : 'Describe figure') +
'</button>' +
'<span class="muted" id="mag-describe-status-' + safeId + '" role="status"></span>' +
'<label class="mag-describe-label" for="mag-describe-text-' + safeId + '">Spoken description ' +
'<span class="muted">(only read aloud if "Read figure descriptions" is on)</span></label>' +
'<textarea class="mag-describe-text" id="mag-describe-text-' + safeId + '" rows="2" ' +
'placeholder="No description yet." onchange="magSaveImageDescription(\'' + safeId + '\', this)">' +
esc(description || '') + '</textarea>' +
'</div>';
}
async function magDescribeImage(id) {
const btn = $('mag-describe-btn-' + id);
const statusEl = $('mag-describe-status-' + id);
if (btn) btn.disabled = true;
if (statusEl) statusEl.textContent = 'Describing…';
let jobId;
try {
const r = await api('/api/magazine/' + state.doc.id + '/images/' + id + '/describe', { method: 'POST' });
jobId = r.job_id;
} catch (e) {
if (btn) btn.disabled = false;
if (statusEl) statusEl.textContent = '';
return say('Could not start: ' + e.message, 'err');
}
let result = null, failed = false;
for (;;) {
let job;
try { job = await api('/api/jobs/' + jobId); } catch (_) { break; }
if (job.done) {
if (job.error) { failed = true; say('Description failed: ' + job.error, 'err'); }
else { result = job.result; say('Figure described', 'ok'); }
break;
}
await new Promise((r) => setTimeout(r, 2000));
}
if (btn) { btn.disabled = false; btn.innerHTML = '<span class="mdi mdi-text-to-speech" aria-hidden="true"></span> Redo description'; }
if (statusEl) statusEl.textContent = '';
if (failed || !result) return;
const textarea = $('mag-describe-text-' + id);
if (textarea) textarea.value = result.description || '';
const layoutImg = (state.layout?.images || []).find((i) => i.id === id);
if (layoutImg) layoutImg.description = result.description || '';
}
async function magSaveImageDescription(id, textarea) {
const statusEl = $('mag-describe-status-' + id);
if (statusEl) statusEl.textContent = 'Saving…';
try {
await api('/api/magazine/' + state.doc.id + '/images/' + id, {
method: 'PATCH', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ description: textarea.value }),
});
const layoutImg = (state.layout?.images || []).find((i) => i.id === id);
if (layoutImg) layoutImg.description = textarea.value;
if (statusEl) statusEl.textContent = 'Saved';
setTimeout(() => { if (statusEl) statusEl.textContent = ''; }, 1500);
} catch (e) {
if (statusEl) statusEl.textContent = '';
say('Could not save description: ' + e.message, 'err');
}
}
function magReviewClearCurrentPage() {
if (state.reviewPage == null) return;
if (!_magReviewChecked) _magReviewChecked = new Set(state.article.blocks || []);
@ -1333,7 +1407,8 @@ async function magRead(articleId, opts) {
await ensureLayout();
try {
const data = await api('/api/magazine/' + state.doc.id + '/articles/' + a.id + '/text?captions=' +
($('mag-read-captions')?.checked ? 1 : 0));
($('mag-read-captions')?.checked ? 1 : 0) + '&descriptions=' +
($('mag-read-descriptions')?.checked ? 1 : 0));
state.sentences = splitSentences(data.blocks || []);
} catch (e) {
return say('Could not load article text: ' + e.message, 'err');
@ -1359,7 +1434,8 @@ function renderArticle() {
host.innerHTML = state.sentences.map((s, i) => {
const tag = s.type === 'title' ? 'h2' : s.type === 'deck' ? 'p class="mag-deck"'
: s.type === 'byline' ? 'p class="mag-byline"' : s.type === 'caption'
? 'p class="mag-caption"' : 'span';
? 'p class="mag-caption"' : s.type === 'figure_description'
? 'p class="mag-figure-description"' : 'span';
const open = tag.split(' ')[0];
// Each word gets its own span so playback can highlight it individually
// (see estimateWordTimings) -- sentence-level highlighting alone doesn't
@ -1852,6 +1928,8 @@ root.magReviewToggleBox = magReviewToggleBox;
root.magReocrPage = magReocrPage;
root.magRefineFigures = magRefineFigures;
root.magReviewToggleImage = magReviewToggleImage;
root.magDescribeImage = magDescribeImage;
root.magSaveImageDescription = magSaveImageDescription;
root.magReviewToggleDraw = magReviewToggleDraw;
root.magReviewCancelDrawPanel = magReviewCancelDrawPanel;
root.magReviewSubmitDraw = magReviewSubmitDraw;

View File

@ -129,6 +129,10 @@
<input type="checkbox" id="mag-karaoke-cb" checked onchange="TTVC.magazine.karaoke = this.checked">
Word highlighting
</label>
<label class="mag-karaoke-toggle" title="Speak each figure's description where it appears, if one has been generated in review. Takes effect next time this article is opened.">
<input type="checkbox" id="mag-read-descriptions">
Read figure descriptions
</label>
<div class="mag-pane-tabs" role="group" aria-label="Reading view">
<button type="button" class="btn-sm active" data-mag-pane="reflow"
aria-pressed="true" onclick="magSetPane('reflow')">Article</button>

View File

@ -4995,6 +4995,14 @@ input[list="lang-codes"] {
background: var(--mauve); color: #fff; font-size: 10px; font-weight: 700; }
.mag-block-image { cursor: default; grid-template-columns: auto auto 90px 1fr; }
.mag-block-text { font-size: 14px; line-height: 1.45; }
/* A 5th grid child in the 4-column .mag-block-image row -- span the whole
row below the thumbnail rather than being squeezed into column 1 alone. */
.mag-describe { grid-column: 1 / -1; display: flex; flex-wrap: wrap; align-items: center; gap: 8px;
margin-top: 4px; padding-top: 8px; border-top: 1px dashed var(--border); }
.mag-describe-label { flex-basis: 100%; font-size: 11px; color: var(--subtext); }
.mag-describe-text { flex: 1 1 100%; min-width: 0; min-height: 44px; resize: vertical;
background: var(--panel); border: 1px solid var(--border); color: var(--text);
border-radius: var(--radius); padding: 6px 8px; font-size: 13px; font-family: inherit; }
.mag-review-foot { display: flex; gap: 8px; justify-content: flex-end; margin-top: 16px;
position: sticky; bottom: 0; padding: 12px 0; background: var(--bg); }
@media (max-width: 980px) {
@ -5013,6 +5021,12 @@ input[list="lang-codes"] {
.mag-article .mag-deck { font-size: 19px; color: var(--subtext); margin: 0 0 12px; }
.mag-article .mag-byline { font-size: 14px; color: var(--subtext); margin: 0 0 20px; }
.mag-article .mag-caption { font-size: 14px; color: var(--subtext); font-style: italic; }
/* A synthesised aside, not part of the article's own written text -- set
off visually the same way a caption is, so a reader following along
silently (word-highlighting off) can tell it apart from the real prose. */
.mag-article .mag-figure-description { font-size: 15px; color: var(--subtext); font-style: italic;
padding-left: 10px; border-left: 2px solid var(--border); margin: 4px 0 14px; }
.mag-article .mag-figure-description::before { content: '\1F5BC '; font-style: normal; }
.mag-article [data-i] { cursor: pointer; border-radius: 3px; }
.mag-article [data-i]:hover { background: rgba(37,99,235,.07); }
.mag-article [data-i]:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }

View File

@ -23,6 +23,8 @@ from core.magazine import (
_looks_like_repetition_loop,
page_ink_grid,
BLOCK_FIGURE,
BLOCK_CHART,
BLOCK_TABLE,
looks_like_real_title,
Article,
BLOCK_AD,
@ -63,6 +65,9 @@ from core.magazine import (
score_contents_page,
vision_ocr_page,
vision_ocr_page_chain,
describe_visual_region,
describe_visual_region_chain,
BLOCK_FIGURE_DESCRIPTION,
)
PAGE_W, PAGE_H = 595.0, 842.0 # A4 in points
@ -863,6 +868,95 @@ class VisionOcrPageCallTests(unittest.TestCase):
self.assertIsNone(vision_ocr_page(pathlib.Path(__file__), "http://engine/v1", "qwen-vl"))
class DescribeVisualRegionTests(unittest.TestCase):
"""describe_visual_region() is a deliberately different task from
vision_ocr_page(): that prompt forbids inventing a description, this
one asks for exactly that -- but shares the same reliability guards."""
def _resp(self, status=200, json_body=None):
import requests
resp = mock.Mock(status_code=status)
resp.raise_for_status = mock.Mock(
side_effect=None if status < 400 else requests.exceptions.HTTPError(f"HTTP {status}"))
resp.json = mock.Mock(return_value=json_body or {})
return resp
def test_successful_call_returns_the_description(self):
body = {"choices": [{"message": {"content": "A bar chart showing sales rising over five years."}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)):
got = describe_visual_region(pathlib.Path(__file__), BLOCK_CHART, "http://engine/v1", "m")
self.assertEqual(got, "A bar chart showing sales rising over five years.")
def test_prompt_is_chosen_by_region_type(self):
body = {"choices": [{"message": {"content": "A description long enough to pass."}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)) as request:
describe_visual_region(pathlib.Path(__file__), BLOCK_TABLE, "http://engine/v1", "m")
prompt = request.call_args.kwargs["json"]["messages"][0]["content"][0]["text"]
self.assertIn("table", prompt.lower())
self.assertNotIn("chart or graph", prompt.lower())
def test_unknown_region_type_falls_back_to_the_figure_prompt(self):
body = {"choices": [{"message": {"content": "A description long enough to pass."}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)) as request:
describe_visual_region(pathlib.Path(__file__), "something-odd", "http://engine/v1", "m")
prompt = request.call_args.kwargs["json"]["messages"][0]["content"][0]["text"]
self.assertIn("photo or illustration", prompt.lower())
def test_a_repeated_phrase_loop_is_rejected(self):
body = {"choices": [{"message": {"content": "chart chart chart " * 20}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)):
self.assertIsNone(describe_visual_region(pathlib.Path(__file__), BLOCK_CHART, "http://engine/v1", "m"))
def test_too_short_a_response_is_rejected(self):
body = {"choices": [{"message": {"content": "Yes."}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)):
self.assertIsNone(describe_visual_region(pathlib.Path(__file__), BLOCK_FIGURE, "http://engine/v1", "m"))
def test_empty_response_is_rejected(self):
body = {"choices": [{"message": {"content": " "}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)):
self.assertIsNone(describe_visual_region(pathlib.Path(__file__), BLOCK_FIGURE, "http://engine/v1", "m"))
def test_a_long_response_is_cut_at_a_sentence_boundary(self):
# Genuinely varied sentences -- a repeated one would (correctly)
# trip the repetition-loop guard instead of exercising truncation.
sentences = [f"This is plausible sentence number {i} about the figure."
for i in range(20)]
body = {"choices": [{"message": {"content": " ".join(sentences)}}]}
with mock.patch("core.magazine.request_with_retries",
return_value=self._resp(json_body=body)):
got = describe_visual_region(pathlib.Path(__file__), BLOCK_FIGURE, "http://engine/v1", "m")
self.assertLessEqual(len(got), 601)
self.assertTrue(got.endswith("."))
def test_unreachable_engine_returns_none(self):
import requests
with mock.patch("core.magazine.request_with_retries",
side_effect=requests.exceptions.ConnectionError()):
self.assertIsNone(describe_visual_region(pathlib.Path(__file__), BLOCK_FIGURE, "http://engine/v1", "m"))
def test_missing_file_returns_none(self):
self.assertIsNone(describe_visual_region(pathlib.Path("/no/such/file.png"),
BLOCK_FIGURE, "http://engine/v1", "m"))
def test_chain_falls_back_to_the_second_engine(self):
with mock.patch("core.magazine.describe_visual_region",
side_effect=[None, "A usable description of the figure."]) as mocked:
got = describe_visual_region_chain(pathlib.Path(__file__), BLOCK_FIGURE,
[("http://a/v1", "m1"), ("http://b/v1", "m2")])
self.assertEqual(got, "A usable description of the figure.")
self.assertEqual(mocked.call_count, 2)
def test_chain_with_no_configured_engines_returns_none(self):
self.assertIsNone(describe_visual_region_chain(pathlib.Path(__file__), BLOCK_FIGURE, [("", ""), (None, None)]))
class UnlimitedOcrRegionTests(unittest.TestCase):
# The URL/error-path tests below remain deliberately adjacent to the
# native-marker tests; share the same mocked OpenAI response helper.
@ -979,6 +1073,60 @@ class ArticleTextTests(unittest.TestCase):
include_captions=True)]
self.assertEqual(with_caps, ["first", "second", "a caption"])
def test_described_image_is_narrated_at_its_reading_position(self):
layout = {
"blocks": [
{"id": "b1", "page": 1, "type": BLOCK_TITLE, "reading_order": 1,
"text": "Headline", "bbox": [0, 0, 1, 1]},
{"id": "b2", "page": 1, "type": BLOCK_BODY, "reading_order": 3,
"text": "Body after the figure", "bbox": [0, 0, 1, 1]},
],
"images": [
{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1], "reading_order": 2,
"description": "A photo of a cat."},
],
}
article = {"blocks": ["b1", "b2"], "images": ["img1"]}
got = article_text(layout, article, include_descriptions=True)
self.assertEqual([b["text"] for b in got],
["Headline", "A photo of a cat.", "Body after the figure"])
self.assertEqual(got[1]["type"], BLOCK_FIGURE_DESCRIPTION)
self.assertEqual(got[1]["page"], 1)
def test_descriptions_are_off_by_default(self):
layout = {
"blocks": [{"id": "b1", "page": 1, "type": BLOCK_BODY, "reading_order": 1,
"text": "Body", "bbox": [0, 0, 1, 1]}],
"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1],
"reading_order": 0, "description": "A photo."}],
}
article = {"blocks": ["b1"], "images": ["img1"]}
got = article_text(layout, article)
self.assertEqual([b["text"] for b in got], ["Body"])
def test_an_undescribed_image_is_silently_skipped(self):
"""No description yet (or an empty one) -- nothing to narrate, not
an empty sentence."""
layout = {
"blocks": [{"id": "b1", "page": 1, "type": BLOCK_BODY, "reading_order": 1,
"text": "Body", "bbox": [0, 0, 1, 1]}],
"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1], "reading_order": 0}],
}
article = {"blocks": ["b1"], "images": ["img1"]}
got = article_text(layout, article, include_descriptions=True)
self.assertEqual([b["text"] for b in got], ["Body"])
def test_a_described_image_belonging_to_another_article_is_excluded(self):
layout = {
"blocks": [{"id": "b1", "page": 1, "type": BLOCK_BODY, "reading_order": 1,
"text": "Body", "bbox": [0, 0, 1, 1]}],
"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1],
"reading_order": 0, "description": "Not this article's figure."}],
}
article = {"blocks": ["b1"], "images": []}
got = article_text(layout, article, include_descriptions=True)
self.assertEqual([b["text"] for b in got], ["Body"])
def test_listening_estimate_counts_only_narratable_words(self):
blocks = [
Block(id="b1", page=1, bbox=[0, 0, 1, 1], type=BLOCK_BODY,

View File

@ -291,3 +291,167 @@ class IngestPreviewHtmlTests(unittest.TestCase):
from routes.magazine import _render_ingest_preview_html
self.assertEqual(_render_ingest_preview_html([], None), "")
self.assertEqual(_render_ingest_preview_html([], []), "")
class DescribeImageJobTests(unittest.TestCase):
"""Figure descriptions are additive: generating one never touches the
article's own blocks, only the image record's own description field."""
def _layout(self):
return {
"images": [
{"id": "img1", "page": 1, "bbox": [0, 0, 10, 10], "filename": "img1.png", "type": "figure"},
],
}
def test_successful_description_is_saved(self):
from unittest.mock import patch
from routes.magazine import _describe_image_sync
layout = self._layout()
with patch('routes.magazine.mag.load_layout', return_value=layout), \
patch('routes.magazine.mag.describe_visual_region_chain',
return_value='A photo of a mountain.') as chain, \
patch('routes.magazine.mag.save_layout') as save, \
patch('routes.magazine.background_job_update') as update, \
patch('pathlib.Path.is_file', return_value=True):
_describe_image_sync('a' * 32, 'job', 'img1', 'http://e/v1', 'm', None, None)
save.assert_called_once()
self.assertEqual(layout['images'][0]['description'], 'A photo of a mountain.')
chain.assert_called_once()
self.assertEqual(chain.call_args.args[1], 'figure') # region type passed through
done = [c.kwargs for c in update.call_args_list if c.kwargs.get('done')]
self.assertEqual(done[-1]['stage'], 'done')
self.assertEqual(done[-1]['result'], {'image_id': 'img1', 'description': 'A photo of a mountain.'})
def test_no_layout_reports_failure(self):
from unittest.mock import patch
from routes.magazine import _describe_image_sync
with patch('routes.magazine.mag.load_layout', return_value=None), \
patch('routes.magazine.mag.save_layout') as save, \
patch('routes.magazine.background_job_update') as update:
_describe_image_sync('a' * 32, 'job', 'img1', 'http://e/v1', 'm', None, None)
save.assert_not_called()
done = [c.kwargs for c in update.call_args_list if c.kwargs.get('done')]
self.assertEqual(done[-1]['stage'], 'failed')
self.assertIn('No layout', done[-1]['error'])
def test_unknown_image_id_reports_failure(self):
from unittest.mock import patch
from routes.magazine import _describe_image_sync
with patch('routes.magazine.mag.load_layout', return_value=self._layout()), \
patch('routes.magazine.mag.save_layout') as save, \
patch('routes.magazine.background_job_update') as update:
_describe_image_sync('a' * 32, 'job', 'missing', 'http://e/v1', 'm', None, None)
save.assert_not_called()
done = [c.kwargs for c in update.call_args_list if c.kwargs.get('done')]
self.assertEqual(done[-1]['stage'], 'failed')
self.assertIn('not found', done[-1]['error'].lower())
def test_engine_returning_nothing_usable_is_a_failure_not_a_blank_save(self):
from unittest.mock import patch
from routes.magazine import _describe_image_sync
layout = self._layout()
with patch('routes.magazine.mag.load_layout', return_value=layout), \
patch('routes.magazine.mag.describe_visual_region_chain', return_value=None), \
patch('routes.magazine.mag.save_layout') as save, \
patch('routes.magazine.background_job_update') as update, \
patch('pathlib.Path.is_file', return_value=True):
_describe_image_sync('a' * 32, 'job', 'img1', 'http://e/v1', 'm', None, None)
save.assert_not_called()
self.assertNotIn('description', layout['images'][0])
done = [c.kwargs for c in update.call_args_list if c.kwargs.get('done')]
self.assertEqual(done[-1]['stage'], 'failed')
class DescribeImageRouteTests(unittest.IsolatedAsyncioTestCase):
async def test_unconfigured_vision_engine_is_rejected(self):
from unittest.mock import AsyncMock, patch
from routes.magazine import magazine_describe_image
layout = {"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1]}]}
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=layout), \
patch('routes.magazine._load_settings', return_value={}):
request = AsyncMock()
request.json.return_value = {}
with self.assertRaises(HTTPException) as err:
await magazine_describe_image('a' * 32, 'img1', request)
self.assertEqual(err.exception.status_code, 400)
async def test_unknown_image_id_is_a_404(self):
from unittest.mock import patch
from routes.magazine import magazine_describe_image
layout = {"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1]}]}
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=layout):
with self.assertRaises(HTTPException) as err:
await magazine_describe_image('a' * 32, 'missing', None)
self.assertEqual(err.exception.status_code, 404)
async def test_valid_request_starts_a_background_job(self):
from unittest.mock import AsyncMock, patch
from routes.magazine import magazine_describe_image
layout = {"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1]}]}
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=layout), \
patch('routes.magazine._load_settings',
return_value={'vision_ocr_url': 'http://e/v1', 'vision_ocr_model': 'm'}), \
patch('routes.magazine.background_job_create') as create, \
patch('routes.magazine.asyncio.to_thread', new=lambda *a, **k: None), \
patch('asyncio.create_task') as create_task:
request = AsyncMock()
request.json.return_value = {}
result = await magazine_describe_image('a' * 32, 'img1', request)
self.assertIn('job_id', result)
create.assert_called_once()
create_task.assert_called_once()
class UpdateImageDescriptionRouteTests(unittest.IsolatedAsyncioTestCase):
async def test_sets_the_description_verbatim(self):
from unittest.mock import AsyncMock, patch
from routes.magazine import magazine_update_image_description
layout = {"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1]}]}
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=layout), \
patch('routes.magazine.mag.save_layout') as save:
request = AsyncMock()
request.json.return_value = {'description': ' A hand-typed description. '}
result = await magazine_update_image_description('a' * 32, 'img1', request)
self.assertEqual(result['description'], 'A hand-typed description.')
self.assertEqual(layout['images'][0]['description'], 'A hand-typed description.')
save.assert_called_once()
async def test_unknown_image_id_is_a_404(self):
from unittest.mock import AsyncMock, patch
from routes.magazine import magazine_update_image_description
layout = {"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1]}]}
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=layout):
request = AsyncMock()
request.json.return_value = {'description': 'x'}
with self.assertRaises(HTTPException) as err:
await magazine_update_image_description('a' * 32, 'missing', request)
self.assertEqual(err.exception.status_code, 404)
async def test_missing_description_field_is_rejected(self):
from unittest.mock import AsyncMock, patch
from routes.magazine import magazine_update_image_description
layout = {"images": [{"id": "img1", "page": 1, "bbox": [0, 0, 1, 1]}]}
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=layout):
request = AsyncMock()
request.json.return_value = {}
with self.assertRaises(HTTPException) as err:
await magazine_update_image_description('a' * 32, 'img1', request)
self.assertEqual(err.exception.status_code, 400)
async def test_no_layout_is_a_404(self):
from unittest.mock import AsyncMock, patch
from routes.magazine import magazine_update_image_description
with patch('routes.magazine._require_magazine', return_value=(None, {})), \
patch('routes.magazine.mag.load_layout', return_value=None):
request = AsyncMock()
request.json.return_value = {'description': 'x'}
with self.assertRaises(HTTPException) as err:
await magazine_update_image_description('a' * 32, 'img1', request)
self.assertEqual(err.exception.status_code, 404)