From 48af2b3ccbaa1d61a6229db72f69e8500b0361c5 Mon Sep 17 00:00:00 2001 From: mARTin-B78 Date: Sat, 3 Oct 2026 14:36:29 +0200 Subject: [PATCH] feat(magazine): detect continuation pages and suggest titles for untitled articles "Page N" entries come from review stand-ins created after Approve saved only the visible page's regions. New core/magazine_untitled.py classifies untitled entries (continuation, article head, advert, picture page, contents, junk), proposes merges into the previous article with confidence and evidence (optional LLM check in the ambiguous band), and suggests titles (deterministic, optional validated and cached LLM titles). Exposed through derived GET /articles fields, GET /untitled, quality report, undoable merge-into-previous / accept-title routes, POST /titles/suggest and scripts/magazine_untitled.py (dry run by default, --apply backs up first). Merges mark audio_stale; no audio is deleted; real titles are never overwritten. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + Manual.md | 20 + architecture/magazine-ingestion.md | 100 +++++ core/magazine_untitled.py | 613 +++++++++++++++++++++++++++++ routes/magazine.py | 162 +++++++- scripts/magazine_untitled.py | 167 ++++++++ tests/test_magazine_untitled.py | 461 ++++++++++++++++++++++ 7 files changed, 1522 insertions(+), 2 deletions(-) create mode 100644 core/magazine_untitled.py create mode 100644 scripts/magazine_untitled.py create mode 100644 tests/test_magazine_untitled.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b24dca..84fcb93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ## [Unreleased] ### Added +- **Untitled magazine entries ("Page 33") are explained, merged on request and given readable titles** (`core/magazine_untitled.py`, `scripts/magazine_untitled.py`). Root cause: the review screen creates a `Page N` stand-in for any page no article covers, and *Approve* without a toggled box saved only the visible page's regions, which cut multi-page features down to one page. On *c't KI Wissen 2026* this left 59 stand-ins, most of them continuation pages. A continuation detector scores each untitled entry against the article before it (lower-case/mid-sentence start, hyphenated break, the narration fusion rule, open ending, adjacent page, column width; a signed-off previous article or the page's own headline counts against). An optional local-LLM check runs only for ambiguous scores. Each proposal comes with a confidence and the evidence. Records whose title repeats the headline of the article they continue are proposed as well. Titles: a deterministic fallback (*Anzeige*, *Bildseite*, *Inhalt*, *Impressum*, *Editorial*, *Textfragment*, the page headline, or the first sentence cut to 8 words), plus optional validated and cached LLM titles in the magazine's language (`magazine_llm_titles`, `--no-llm`). Nothing changes silently. `GET /articles` gains derived `display_title`, `title_source`, `suggested_title`, `page_label`, `untitled_kind`, `merge_suggestion` and `audio_stale`, and `GET /untitled` plus `quality/report → untitled_articles` list everything. Changes happen only through the undoable `POST /articles/{id}/merge-into-previous` and `POST /articles/{id}/accept-title` (never over a real title), `POST /titles/suggest` (suggestion cache only), and the CLI's `--apply` (backup first). A merge keeps reading order, marks the article `audio_stale` (a fresh render clears it) and deletes no audio. See architecture/magazine-ingestion.md, "Untitled entries". - **Distinct, deterministic face profile per character** (`core/face_profile.py`). The cast of one book looked like one family because most sheets name no facial feature and the image model then draws its default face. Each character now gets a compact `face:` clause (face shape, jaw, nose, eyes, brows, mouth, cheekbones, skin, hair colour and style, facial hair, age signs, a mark) in the Subject of every character prompt: traits the sheet states (German or English) are kept, the rest comes from species/age/gender/culture-filtered option pools, chosen so that the characters of a book are spread apart (greedy max-distance in creation order; adding characters never changes an existing profile; relatives named in `relationships` share a few traits). Owner override: a `visual_face` text on the sheet. On the real books 02/03 the mean pairwise face-descriptor distance goes from 0.09/0.04 to 0.91 (44/68 characters had no face trait at all). See `architecture/character-visual-identity.md`. - **Dry run for faces:** `POST /api/character-face-preview {book}` lists every character's face profile (what came from the sheet, what was chosen) with a similarity matrix and before/after score; `POST /api/character-prompt-preview` also returns `face_clause`. Nothing is generated or saved. - **Character info is translated to English before image prompts are built** (`core/prompt_translation.py`, `_english_prompt_inputs`). Free-text sheet fields (species, appearance, clothing, age, props, story-moment quote) and the book setting are translated in one batched, low-temperature call through the app's existing LLM, validated per field (non-empty, no error echo, length, names preserved, not still German), and cached in the new `prompt_translation_cache` table keyed by source hash, language, model and prompt version. Records and the displayed German text never change; species/age/dwarf/period checks now run on English. If the LLM is down or output is rejected the original text is used, a warning is logged, `prompt_translation.used_untranslated_text` is reported and generation continues. `PROMPT_TRANSLATION=off` disables it. New read-only dry run `POST /api/character-prompt-preview`. See architecture/character-visual-identity.md. diff --git a/Manual.md b/Manual.md index 44a7f9a..2992716 100644 --- a/Manual.md +++ b/Manual.md @@ -648,6 +648,26 @@ sent. Websites that redistribute commercial magazines without a licence are deli **Mobile / accessibility:** the list is a labelled live region of ordinary buttons, keyboard reachable. **Offline / privacy:** the missed-text pass uses local OCR; the word re-read uses only the vision engine you configured. Nothing else leaves the machine. +### Untitled entries ("Page 33") and continuation pages + +**Purpose:** clean up contents/driving-mode entries that have no real title (`Page 33`, `Page 69` …). Most of them are continuation pages of the article before them that ended up as entries of their own. The others are advert pages, picture pages, or short text fragments. +**Why they exist:** the review screen creates such an entry when you open a page no article covers. A page loses its article when an article is approved on another page without any box being toggled: only the visible page's boxes were saved, so the other pages fell out of the article. +**What you get (nothing changes on its own):** +- Every entry now carries a **display title**. Real titles are shown unchanged. Placeholder titles show a suggestion instead: *Anzeige*, *Bildseite*, *Inhalt*, *Impressum*, *Editorial*, *Textfragment*, the page's own headline, or its first sentence (at most 8 words). The page number comes separately (`page_label`, e.g. "Seite 33") so the index can show it in smaller text. +- A **merge proposal** when an entry looks like the continuation of the previous article. Each proposal has a confidence (*high* ≥ 0.75, *medium* ≥ 0.5) and the evidence, for example "starts lower-case / mid-sentence", "previous text ends in a hyphenated word break", "adjacent page" or "previous article is signed off". +- `GET /api/magazine/{id}/untitled` lists all of them with counts. The quality report (`GET …/quality/report`) carries the same list as `untitled_articles`. +**Steps:** +1. Look at the proposals (`GET …/untitled`, or the dry run below). +2. Merge one: `POST /api/magazine/{id}/articles/{article}/merge-into-previous` with `{"into": ""}`. It can be undone with **Undo**. +3. Accept a title: `POST …/articles/{article}/accept-title` (optionally `{"title": "…"}`). An entry that already has a real title is refused, so titles you typed are never overwritten. +4. Optional, better titles from the local LLM: `POST …/titles/suggest` (background job). It only fills the suggestion cache (`title_suggestions.json`). Switch it off with the setting `magazine_llm_titles: false`. +5. Whole magazine from the command line: `python3 scripts/magazine_untitled.py ` is a dry run that changes nothing. Add `--apply` to write: a backup `articles.backup-untitled-