mARTin-B78
c81ae28da2
Some checks are pending
Test and SonarQube / quality (push) Waiting to run
Adds a "Describe" action per figure in article review: sends the cropped
image to the configured vision engine (with fallback) for a type-aware
one- or two-sentence description -- a chart gets a prompt about what it
measures and its trend, a table about headings and what stands out, etc.
Editable before approval. A new "Read figure descriptions" toggle in the
reading controls speaks these where they occur; purely additive, off by
default, and a figure with no description behaves exactly as before.
New core/magazine.py describe_visual_region()/_chain() reuse the same
reliability guards as page OCR (repetition-loop detection, sentence-
boundary truncation). New routes: POST .../images/{id}/describe
(background job), PATCH .../images/{id} (manual edit), descriptions=1 on
the article-text endpoint.
Live-verified end to end against the ingested c't KI Wissen 2026
magazine. Also found and documented a real, config-dependent limitation:
the currently-configured primary vision engine is transcription-only and
returns empty content for any descriptive prompt (confirmed with three
different prompts) -- not a code defect, but description generation
needs a genuinely caption-capable engine reachable as the fallback.
251 tests passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
158 lines
9.2 KiB
Markdown
158 lines
9.2 KiB
Markdown
# User manual
|
|
|
|
Living manual. Every feature updates this file in the same commit as the code.
|
|
|
|
> Sections for the mature features (voice library, Read Aloud, audiobook
|
|
> casting, Rehearser) are still to be written — this file was created alongside
|
|
> the magazine feature and currently documents that area only.
|
|
|
|
---
|
|
|
|
## Magazines
|
|
|
|
### Purpose
|
|
|
|
Listen to a magazine or newspaper one **article** at a time, from its true
|
|
beginning to its true end, without adverts, sidebars or a neighbouring article
|
|
spliced into the middle. Each article can be narrated by a single voice, or
|
|
turned into a podcast discussion between two or three voices you pick.
|
|
|
|
### Prerequisites
|
|
|
|
- The magazine as a PDF (native text or scanned — both work).
|
|
- At least one voice in the library for narration.
|
|
- For podcast mode: an LLM configured in Settings (`llm_url`, `llm_model`), and
|
|
two or three voices chosen as hosts.
|
|
|
|
### Step by step
|
|
|
|
1. **Add the magazine.** Studio → *Add Magazine / Paper*. Give it a title and
|
|
issue, then upload the PDF.
|
|
2. **Ingest.** Ingestion runs in the background — you can close the browser. It
|
|
renders every page, extracts text (falling back to OCR on scanned pages),
|
|
classifies each region of the layout, finds the contents page, and proposes
|
|
articles with page ranges. While it runs, a live preview shows each page's
|
|
image next to its recognised text as soon as that page finishes -- a
|
|
detected headline is shown larger, a figure appears as its own picture
|
|
marker rather than as garbled on-image text, and page furniture is shown
|
|
dimmed -- so you can see ingestion actually working page by page instead
|
|
of watching a bare progress bar. This preview is a quick approximation,
|
|
not the final read: it runs before the layout is fully classified, so a
|
|
heading guess or an excluded marking here can still change once review
|
|
opens.
|
|
3. **Review the articles.** This step is not optional and cannot be skipped.
|
|
Each proposed article shows its title, author, page range, estimated
|
|
listening time and a confidence score. The review screen is split: the
|
|
scanned page on the left with a numbered, icon-labelled box per detected
|
|
region (title, byline, body text, caption, figure…), and the same regions
|
|
as text on the right — click either side to highlight the other. Sidebars
|
|
and pull-out boxes are already unticked for you — tick one back in if you
|
|
actually want it read. A region already claimed by a different article on
|
|
the same page is marked with that article's number, so it is never
|
|
mistaken for unclaimed text. If OCR quality is poor, **"OCR again"**
|
|
re-reads just the current page through the configured vision engine
|
|
(or **"OCR again (whole magazine)"** from the contents screen for every
|
|
page, which also resets any articles already reviewed). If a figure's
|
|
box is drawn in the wrong place or at the wrong size — a common miss,
|
|
because an OCR engine locates a photo far less precisely than it reads
|
|
type — **"Fix figure boundaries"** re-derives every figure box in the
|
|
issue from the scanned pages themselves. It calls no OCR engine, changes
|
|
no text and touches no approval, so it is safe to run at any point,
|
|
including on an issue you are halfway through reviewing; boxes it cannot
|
|
place confidently are left exactly as they were. If the layout
|
|
engine missed a region entirely, **"Draw it"** lets you click-drag a box
|
|
directly on the page image and add it as text (typed or OCRed) or as a
|
|
new figure. Confirm the boundaries, correct the title, then **Approve**.
|
|
Optionally, click **"Describe"** under a figure, chart, table or code block
|
|
to have the configured vision engine generate a one- or two-sentence
|
|
description for listeners who cannot see the image — e.g. what a chart
|
|
measures and its general trend, or a table's row/column headings and
|
|
what stands out. You can edit the generated text, or write your own from
|
|
scratch, before approving the article. Generating a description changes
|
|
nothing about existing narration by itself — it is only spoken if you
|
|
also turn on "Read figure descriptions" in step 4.
|
|
4. **Listen.** An approved article can be narrated with the default voice, or
|
|
sent to podcast mode. Use **Play all articles** (on the contents screen) to
|
|
listen through every approved article back to back without returning to
|
|
this screen between them. A figure from the page is shown automatically as
|
|
playback reaches it. **Word highlighting** can be turned off if you find it
|
|
distracting — it is an estimate based on word length, not a measurement, so
|
|
it will not always land exactly on the spoken word. Turn on **"Read figure
|
|
descriptions"** in the reading controls to have any figure descriptions
|
|
generated during review spoken aloud where they occur in the article;
|
|
figures without a generated description are silently skipped, exactly as
|
|
before this option existed. **Driving mode**
|
|
(the button next to the article title) switches to two large controls and
|
|
nothing else, for listening without looking at the screen.
|
|
|
|
**Expected result:** the article plays from its headline to its final paragraph.
|
|
Page furniture (running heads, page numbers), adverts, sidebars and pull quotes
|
|
are never read aloud unless you explicitly include them in review.
|
|
|
|
### Confidence, and what it means
|
|
|
|
| Shown | Meaning |
|
|
|---|---|
|
|
| ~90% | The contents page listed this article — title and start page are known. |
|
|
| ~55% | Detected from a headline alone; check the boundaries carefully. |
|
|
| Low | No body copy found. Usually an advert or a section divider — reject it. |
|
|
|
|
An article is never narrated until you approve it, whatever its confidence.
|
|
|
|
### Podcast mode
|
|
|
|
Choose two or three voices as hosts. The configured LLM writes a discussion of
|
|
that article for listeners who have not read it, and each host's turns are
|
|
synthesised with their voice.
|
|
|
|
The prompt forbids inventing facts, figures or quotes that are not in the
|
|
article; if the article reaches no conclusion the hosts say so. It is still a
|
|
generated discussion, not the author's words — treat it as a summary, and read
|
|
the original for anything that matters.
|
|
|
|
Podcast generation is refused for an article you have not approved, because
|
|
unreviewed boundaries may include a neighbouring article or an advert.
|
|
|
|
### Errors
|
|
|
|
| What you see | What happened | What to do |
|
|
|---|---|---|
|
|
| "Upload the PDF first" | The project has no `source.pdf`. | Upload it, then ingest. |
|
|
| Ingestion failed | The PDF is encrypted or corrupt. | Check it opens in a viewer. |
|
|
| "Approve the article before generating a podcast" | Review gate. | Approve it first. |
|
|
| "No LLM model configured" | `llm_model` is empty. | Set it in Settings. |
|
|
| "LLM did not return usable JSON" | The model ignored the format. | Retry, or pick a stronger model. |
|
|
| "Article has too little text" | Under 200 characters — usually a stub. | Check the boundaries in review. |
|
|
| A figure box sits over the wrong part of the page | The OCR engine grounded the figure loosely. | Press **"Fix figure boundaries"**; if the graphic is too pale to detect, draw the region by hand. |
|
|
| "Fix figure boundaries" reports nothing corrected | Every box already matched its page, or none could be placed with confidence. | Nothing is wrong; correct any remaining box by drawing it. |
|
|
| "No vision OCR engine configured" (Describe) | `vision_ocr_url`/`vision_ocr_model` are empty. | Set a vision engine in Settings. |
|
|
| "The vision engine returned nothing usable for this figure" | The configured engine (and fallback, if set) returned an empty or degenerate response for this image. Some vision models are transcription-only and cannot produce open-ended descriptions at all, however the prompt is phrased — confirmed with a pure-OCR model that answers its own "Text Recognition:" prompt but returns nothing for any descriptive one. | Configure a genuinely caption-capable vision model as the fallback (`vision_ocr_fallback_url`/`vision_ocr_fallback_model`), or write the description by hand. |
|
|
|
|
### Privacy
|
|
|
|
Ingestion, OCR and storage are entirely local to the container. Podcast mode is
|
|
one exception: **the article's text is sent to whatever LLM `llm_url` points
|
|
at.** Generating a figure description is the other: **the cropped image is
|
|
sent to whatever `vision_ocr_url` (and, on failure, `vision_ocr_fallback_url`)
|
|
points at.** If either is a hosted API rather than a model on your own
|
|
network, that content leaves your machine. Narration uses your local TTS
|
|
stack and sends nothing outward.
|
|
|
|
### Accessibility and mobile
|
|
|
|
Sentence and word highlighting both respect `prefers-reduced-motion` (no
|
|
smooth-scroll animation when it is set; the highlight itself is a colour
|
|
change, not motion). Driving mode gives two large, high-contrast controls
|
|
(skip, play/pause) with nothing else to interact with, for use without
|
|
looking at the screen — it does not, and should not, replace watching the
|
|
road. Word-level highlighting can be switched off in the reading controls if
|
|
it is more distracting than helpful; sentence highlighting keeps working
|
|
either way. Keyboard navigation of the contents tree and review screen is
|
|
not yet built.
|
|
|
|
### Offline
|
|
|
|
Ingested magazines are stored server-side, so any device on the network sees the
|
|
same library and reading progress. Playback needs the server; there is no
|
|
offline cache yet.
|