From 93c9d483274ee724143f16432a3d4c449f5803ad Mon Sep 17 00:00:00 2001 From: mARTin-B78 Date: Fri, 2 Oct 2026 09:02:48 +0200 Subject: [PATCH] chore: ignore the local TODO.md Co-Authored-By: Claude Haiku 4.5 --- .gitignore | 3 + TODO.md | 910 ----------------------------------------------------- 2 files changed, 3 insertions(+), 910 deletions(-) delete mode 100644 TODO.md diff --git a/.gitignore b/.gitignore index 0b01b45..38ad6b9 100644 --- a/.gitignore +++ b/.gitignore @@ -57,3 +57,6 @@ SONARQUBE-LLM-GUIDE.md # Personal test scans (real copyrighted magazine content, local only) Test-Magazin/ + +# local todo list +TODO.md diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 4cdccd0..0000000 --- a/TODO.md +++ /dev/null @@ -1,910 +0,0 @@ -# TTS Voice Creator Roadmap - -This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed. - -Priority levels: - -- **P0 — safety:** prevents crashes, data loss, broken releases, or security incidents. -- **P1 — high impact:** improves memory use, responsiveness, and maintainability. -- **P2 — quality:** improves diagnostics, polish, and long-term operations. - -## Prioritized execution backlog - -Effort estimates are for one developer familiar with this codebase: **S** = a few -hours, **M** = 1–2 days, **L** = several days, **XL** = multi-week or iterative. -Within each priority, tasks are ordered from smallest to largest effort. Complete -the first block before investing in architecture or polish. - -## Current status — 2026-09-04 - -All P0 safety, security, data-integrity, and release-protection work is complete. -P1 implementation is active: pagination, shared voice caching, bounded decoded-audio -caching, background jobs, canonical character identity, forms, and the frontend -namespace/manifest foundations and section-scoped lazy loading are working and -regression-tested. Remaining P1 work is intentionally open where it needs a real -architecture change: the larger Audiobook/Rehearser/style split. - -### Work order — set 2026-09-14 - -User-set priority for picking up feature work, in this order: - -1. **Magazine reader roadmap** (below) — contents-view redesign is blocked on a - layout decision from the user; the other open items (image/chart descriptions, - legal source browsing) are actionable without one. -2. **Character system** — portrait/concept-sheet identity mismatch, then - character-form UX polish. -3. **Audiobook speaker attribution** — resume from Book 1 re-measurement, then - the never-yet-run Book 2 generalization regression. - -Deployment and the deferred multi-user initiative below are explicitly parked, -not part of this order. - -### Magazine MP3 quality pass — set 2026-09-23 - -Work these in order; each item needs regression coverage against German and -English magazine text before it is marked complete. - -1. [x] **Remove printed line-break hyphens before narration.** Fuse words split - only because they crossed a printed line, while preserving real compounds - such as `KI-Modell`. -2. [x] **Expand German and English acronyms for TTS.** Keep a maintainable - pronunciation database and expand whole tokens such as `KI`, `AI`, `LLM` - and `API` in spoken text without changing the displayed article. -3. [x] **Conservatively restore OCR-missed German umlauts.** Use a spellchecker - only where an umlaut variant is known and unambiguous; never rewrite names - or unknown technical vocabulary on a guess. -4. [x] **Keep multi-page articles as one playlist item.** Prefer contents-page - anchors and explicit continuation markers, and prevent running heads or - internal headings from starting a new article. -5. [x] **Support two-voice printed interviews.** Detect repeated speaker labels, - use the selected narrator and optional second interview voice, and render - the turns as one article. - -### Magazine MP3 quality pass, part 2 — set 2026-09-28 - -Findings from the first whole-issue quality run. Every issue found must either be -fixed by an agent (vision re-read / LLM adjudication, never a blind rewrite) or be -marked on the page for a human. Order of work: - -1. [x] **Fuse one article's paragraphs into single narration blocks.** A paragraph - that continues into the next column or onto a later page is separate regions today, - so playback and the MP3 split the sentence at the seam. Join the fragments into one - block (keeping the source regions for highlighting) before sentence splitting. -2. [x] **Context-aware acronym converter.** Use the article's language (not a - per-fragment guess) and a per-article glossary: known acronyms come from the - pronunciation database, unknown ones are classified by the LLM from their sentence - (expansion / spelled letters / read as a word) and stay editable. -3. [x] **German and English spellchecker with fix-or-mark.** Per-block language, - tokens unknown in both dictionaries and the user's own dictionary are findings; - a vision re-read and an LLM adjudication fix what they can prove, everything left - is marked on the page with suggestions (accept / ignore / add to dictionary). - -### Magazine: verify, re-import and compare — set 2026-09-28 - -The current issue (c't KI-Wissen 2026) is the test bed. The point is to measure whether the newest -rules beat what was built up by hand, and to feed every difference back into the rules. Do these in order. - -- [x] **A — Finish verifying this magazine (automatic part).** 2026-09-28: full Auto-fix with the language - model (thinking off, ~4s/call) and vision re-read ran to completion: 94 misspellings fixed, 77 marked, - 52 blocks linked, 27 pages still need a human decision. Snapshot after this run is - `config/magazine_backups/bd871c7c22f0449984d3b0e3d603d6d6-after-autofix-20260928-1810`. - - **Healing re-run with both G fixes, 2026-09-29** (~17 min; the same "Auto-fix all pages" the user runs by - hand, triggered once to apply the two fixes retroactively): re-running `clean_up_regions` on the result - found **0** additional duplicates, confirming the 6 residual ones found before this run are gone. Also - surfaced (pre-existing behaviour, not part of either fix) one duplicate **article** pair sharing page 75 - (`article-020` "AVRunner im nächsten Desinfec't" / `article-021` "Wofür Softwareentwickler KI-Tools - nutzen", both approved) -- added to the pages needing review, since resolving two approved articles that - both claim the same page is a human decision. **Still open:** a human must work the 27 flagged pages - (same list as before, plus this new duplicate-article pair), approve/reject the not-reviewed articles, - and decide which of article-020/021 page 75 actually belongs to. -- [x] **B — Make a backup copy.** Taken 2026-09-28 after A's automatic pass (see snapshot above; layout, - articles, dictionary, quality report; 62 articles). Audio stays in the earlier 17:33 snapshot. -- [x] **C — Clone the PDF and re-import it.** Cloned as doc `345b2a841f104b03b3cdf61faceb66cc` from the same - `source.pdf`, ingested from nothing (no reuse of the reviewed layout). -- [x] **D — Process the new import to the maximum.** Full ingest (~2h27m) + Auto-fix with LLM and vision - (~35m) on the clone. Result: 60 pages need a decision (vs. 27 for the hand-reviewed magazine) -- expected, - since the backup carries months of manual box/order/approval fixes a from-scratch run cannot reproduce. -- [x] **E — Compare the new result with the backup.** Ran `scripts/compare_magazines.py`; report saved at - `architecture/evidence/compare-345b2a84-vs-backup-20260928.{md,json}`. Headline: text similarity 0.937, - box recall/precision 0.88/0.85, reading-order agreement 0.967, 51/56 articles matched. -- [x] **F — Show the differences and decide what is better.** Decided not needed: a side-by-side review UI - adds no value the markdown/JSON report from E doesn't already give for this workflow. -- [~] **G — Adjust the rules so the next magazine needs less work.** Two universal fixes landed 2026-09-28/29, - each traced from the worst-differing pages E flagged, not from one page in isolation: - 1. **Vision refusal hallucinations** (pages 82, 87, 124, 202): the vision OCR engine sometimes answers - with a sentence describing its own failure ("The image is too blurry to recognize...", "The Ground - Truth image displays...") instead of leaving an unreadable photo/screenshot region empty, tiled across - dozens of regions and narrated as real page text. Fixed in `core/magazine.py` - (`_drop_repeated_region_text`/`_looks_like_vision_refusal`): text repeated verbatim across 3+ regions - that also reads like a refusal is blanked before it becomes blocks. Verified live: re-OCR of page 82 - discarded 29 hallucinated regions, page 87 discarded 122. - 2. **Duplicate check runs too early** (pages 31, 42, 103, 151, 152, and 6 more on the already-reviewed - backup): the duplicate-box detector (`clean_up_regions`) only ran once, before the box-fitting/seam- - mending loop that can grow or merge a box afterward -- a box that becomes a duplicate only because of - that growth was left unmarked, so its text was narrated twice. Fixed by re-running the check after that - loop at ingest, and again after `recover_missing_text` in the whole-issue quality run - (`routes/magazine.py`). Verified live: re-running the check on the current layout alone found 13 such - duplicates in the fresh import and 6 in the already-reviewed backup (the bug predates this fix, so the - backup itself is not clean either -- its next Auto-fix/quality run will self-heal now). - Regression tests: `tests/test_magazine.py::VisionOcrRefusalHallucinationTests`, - `tests/test_magazine_routes.py::QualityRunJobTests::test_a_duplicate_created_by_recovering_missing_text_is_still_caught`, - `tests/test_magazine_routes.py::IngestSyncDuplicateRecheckTests`. - - **D→E re-run with both fixes, 2026-09-29** (re-running Auto-fix on the same clone; report at - `architecture/evidence/compare-345b2a84-vs-backup-20260929-after-fixes.{md,json}`): pages needing a - decision 60→53, overlapping pairs 188→14, clipped edges 127→96 -- real improvement. Narratable word - count also dropped slightly (68680→64946), consistent with duplicate text no longer being counted twice. - Caveat: this re-run's own "re-reading suspicious words" step calls the vision engine again on flagged - boxes, and that engine is not deterministic -- pages 82 and 87 got *worse* on this pass (re-hallucinated - differently) for reasons unrelated to either fix, which is why some individual pages regressed even as - the totals improved. A from-scratch re-ingest (true D) would be a cleaner test but costs ~2h27m; not run - again this pass. - - **Still open:** page 44's screenshot-in-article OCR (garbled but partly real transcription of a mockup - image) was looked at and deliberately left alone -- a known-word-ratio filter would also catch genuine - quoted text, and no reliable rule to tell them apart was found; page 3's biggest "difference" turned out - to be baseline swallowing real editorial text into a figure's alt text, i.e. new is better there, not a - regression to fix. A decision on whether to run the healed quality pass on the live production magazine - (6 residual duplicates found there, not yet corrected) is still open. - - **Ingest now runs Auto-fix itself, 2026-09-29.** A fresh import used to stop laid out but not worked, - leaving Auto-fix for a separate manual click; `_ingest_sync` now runs `_quality_run_sync` itself under - the same job (`routes/magazine.py`), so a fresh import opens already-fixed. An Auto-fix failure is - reported as an ingest failure without losing the ingest's own result. Regression tests: - `tests/test_magazine_routes.py::IngestSyncAutoQualityRunTests`. - - **Clean from-scratch re-ingest, 2026-09-29** (doc `64fe2b4def774c29a44c14a5c4d35571`, both fixes and the - auto-chained Auto-fix in place; ~2h39m combined; report at - `architecture/evidence/compare-64fe2b4d-vs-backup-20260929-fresh-reingest.{md,json}`): the ingest-time - ordering fix is proven in isolation -- re-running `clean_up_regions` on the finished layout found **0** - additional duplicates (13 were found this way on the previous, unfixed clone). The hallucination filter - also fired identically (29 + 122 regions discarded on pages 82/87, matching the first run). But the - whole-magazine comparison against the backup came back worse than the previous clone's (76 pages needing - a decision vs. 53, overlapping pairs 170 vs. 14) for two traced, non-regression reasons: (1) the vision - OCR engine is not deterministic across independent full-magazine passes -- already seen with pages 82/87 - specifically; (2) `scripts/compare_magazines.py` was comparing pages the backup has hand-marked as - full-page advertisements (skipped by its quality run) against the same pages freshly OCR'd in the new - import, which had never had that one-time manual step -- a review-state difference, not a pipeline one. - Fixed (2): the compare tool now excludes any page either side calls an advertisement from every - difference statistic, reporting the count separately instead (`pages_skipped_as_advertisement`; 18 on - this pair). Regression tests: `tests/test_compare_magazines.py` (advertisement handling). Re-run with the - fix: text similarity 0.837→0.870, box precision 0.62→0.66 -- a real but partial improvement, confirming - (1) still accounts for most of the remaining gap. Not fixable here: engine non-determinism is a property - of the external OCR service, not this pipeline; a fair A/B would need the same fixed OCR responses on - both sides, which the current tooling has no way to freeze. - -### Voice Studio (github.com/debpalash/voicestudio) — evaluation, container, comparison - -- [x] **1 — Study Voice Studio.** 2026-09-29: read the repo (README, LICENSE-NOTICE.md, feature-catalog.md, - engines/README.md, speech-platform.md, performance.md, mcp.md, docker.md) and the GitHub API record - (46.2k stars, 3,470+ commits, pushed today, license `AGPL-3.0-only` — quoted exactly from the API, not - the fetch tool's summary). AGPL-3.0: commercial use of the app is explicitly allowed; the Affero clause - only applies to a *modified* version offered to others over a network. No scraping beyond the repo's own - published files and its `/health` endpoint. -- [x] **2 — Run it as a Docker container.** Already running (user's own doing, confirmed live, not started - by this session): `http://192.168.178.8:3900/` on the same DGX Spark this app runs on -- - `GET /health` → `{"status":"ok","device":"cuda (NVIDIA GB10)","version":"0.5.6"}`. Real, non-hypothetical - risk noted in the comparison doc: it shares the DGX's one GPU with this app's Qwen3-TTS/vision-OCR/LLM - calls, so a heavy VoiceStudio job and a magazine Auto-fix run at the same time will contend for VRAM. -- [x] **3 — Detailed comparison with this software.** Written up at - `architecture/voicestudio-comparison.md`: side-by-side table, what each is ahead on, the shared-GPU risk, - and a narrow recommendation (not a replacement -- the magazine pipeline has no counterpart in VoiceStudio - to compare against; worth revisiting only as an alternate TTS backend for a second opinion on narration - quality, or its MCP server for a quick engine A/B). No integration started. - -### UI/UX and feature ideas from the VoiceStudio comparison — set 2026-09-29 - -From the deep-dive UI/UX comparison (`architecture/voicestudio-comparison.md`, "UI/UX and -feature-parity deep dive"). User picked these to build; telephony integrations (Twilio/ -Plivo/Telnyx) were in the same gap list but deliberately not picked -- leave that one alone. - -- [ ] **Projects menu: group Audiobooks, Magazines, Script Rehearsals and Video Dubbing - under one "Projects" nav section.** Currently `Reader` (audiobook), `Magazine` and - `Rehearser` are three separate top-level `nav-tree-head` sections in the sidebar - (`static/index.html`/served nav — see `data-nav-section="s-reader"` / - `"s-magazine"` / `"s-rehearser"`). Add a new "Projects" parent with these three as - submenu items, plus a fourth submenu item for the new Video Dubbing feature below (a - placeholder entry is fine until that feature exists, but say so in its UI rather than - linking to nothing silently). Update `Manual.md`'s navigation description and any - Playwright specs that assert the current nav structure. -- [ ] **Video Dubbing feature.** New capability, not currently in this app at all: upload a - video/audio file or paste a URL, transcribe it, translate to a target language (offer a - local/offline model, not only an LLM call -- see VoiceStudio's NLLB-200 option), then - re-synthesize speech timed to the original and mux it back into the video. This is a - multi-day feature (needs its own data model, a job pipeline, and a review UI, likely - similar in shape to the magazine pipeline's box/undo conventions) -- plan it properly - (goals, data model, API contract, error/offline behaviour) before implementing, per the - planning workflow, rather than bolting it onto an existing route file. -- [ ] **Live STT Dictation.** A global hotkey that starts listening, transcribes speech, and - inserts the resulting text into whatever field/app currently has focus -- distinct from - the existing "global hotkey daemon" in Connect Apps, which triggers TTS *playback*, not - speech-to-text insertion; do not conflate the two or reuse its name for this. Needs a - small native/OS-level component (focus capture, text insertion) similar in spirit to - VoiceStudio's Rust control sidecar -- decide the platform(s) to support first (this - household's own machines) rather than building a generic cross-platform solution - speculatively. -- [ ] **Onboarding / setup wizard.** Either (a) detect the local stack already configured - (Qwen3-TTS, vision OCR, the LLM router -- these already have URLs in Settings) and walk a - new install through pointing at them with a live check per engine, or (b) a single - one-click installer/setup flow that provisions everything needed, mirroring VoiceStudio's - tiered model packs (Fast/Balanced/Quality/Max, disk size shown before committing). Decide - which of (a)/(b) fits this app's actual deployment story (self-hosted, one household, engines - usually already running on the DGX) before building -- (a) is likely the better fit and - the smaller change; (b) only makes sense if this app starts bundling/installing its own - models rather than calling existing engine URLs. -- [ ] **Global command palette + cross-app "recent" history.** A `Ctrl`/`Cmd`+`K` palette to - jump to any section/action, and a unified "recent generations" view spanning Voice Clone/ - Design/Conversation/Magazine/Reader, not just each feature's own history (the magazine's - 20-step undo stack stays separate -- this is about *finding* a past generation across - features, not undoing one). Survey existing per-feature history/list endpoints first - (`/api/*/history`-shaped routes already exist for several features) so the palette reuses - them rather than duplicating state. -- [ ] **Curated voice-design preset gallery, integrated into the existing preset system.** - `s-design.html` already has save/load/delete for named presets (`design-preset-*`, - `routes` behind `design-preset-save`/`-load`/`-delete`) -- add a small built-in gallery of - named example personas (in VoiceStudio's style: "The Librarian", "The Anchor", …), each - with a one-line pitch and a pre-rendered demo clip, "Use this design →" filling the - description box from it. Ship a handful covering common needs (audiobook narrator, news - anchor, casual conversational, storyteller) rather than trying to match VoiceStudio's - exact catalog. Generate and store the demo clips once (not on every page load) and keep - them small (the existing 24 kHz mono WAV convention). -- [ ] **General UI/UX simplification pass -- reduce how overwhelming this app's settings/ - configuration screens are.** User's own observation comparing live: VoiceStudio's - integrations screen (`http://192.168.178.8:3900/#/integrations`) is "way cleaner" than - this app's Settings/Engines equivalent. Concrete evidence for the gap: `s-llms.html` - ("Engines") is 880 lines with 31+ labelled fields all visible on one page at once, versus - VoiceStudio's per-category model cards (TTS/ASR/Dictation/Diarisation/Translation/LLM) - that each expand on demand -- see the `models.png` screenshot referenced in - `architecture/voicestudio-comparison.md`. Direction: progressive disclosure (group into - collapsed cards/categories, expand one at a time) instead of one long flat form; fewer - fields visible per step; a simpler default view with advanced options tucked behind a - toggle, mirroring the "Fast/Balanced/Quality/Max" tiering idea from the onboarding-wizard - item above rather than exposing every knob unconditionally. Not limited to Engines -- - once a pattern is settled there, audit Settings' other pages and any other dense - single-page form (Payloads, the magazine quality-run's settings, etc.) against it. Needs - its own before/after review with the user (screenshots or a walkthrough) since "simpler" - is a judgement call, not a number to hit. - -### P0 — do first - -- [x] **S — Stop provider API keys from reaching frontend JavaScript or logs.** Provider keys are masked in settings - responses, probes resolve credentials server-side, and browser storage/log paths are clean. The separate external - caller key is available only through deliberate regeneration, uses `no-store`, and is not retained by the UI. -- [x] **S — Validate uploaded filenames and paths.** Upload endpoints reject traversal, path separators and control - characters before deriving extensions; ZIP imports reject absolute, traversal and malformed archive members. -- [x] **M — Add upload, request-size, temporary-file and concurrent-job limits.** Endpoint-specific upload/request - caps, bounded temporary work, cache eviction, and bounded TTS/STT/image job pools are implemented. -- [x] **M — Add automatic backups before bulk deletes, voice moves and recasts.** Make - recovery explicit and verify that backups can be restored. -- [x] **M — Strictly validate custom URLs and prevent SSRF.** HTTP downloads and provider/container probes share - scheme, hostname and private-destination validation with explicit local-network opt-in. -- [x] **M — Add authentication and rate limits for destructive/API endpoints.** External API authentication is - enabled by default and destructive/general mutation limits return bounded 429 responses. -- [x] **M — Validate character sheets and voice metadata with schemas.** Character/form and voice metadata are - bounded and type-checked at API boundaries. -- [x] **L — Add database migrations and prompt/generation history.** Preserve old record shapes and expose protected, - metadata-only generation history for auditing and follow-up recovery workflows. -- [x] **L — Harden container and Docker access.** The app runs non-root with dropped capabilities, - no-new-privileges and a read-only root filesystem; Compose Docker access is isolated behind a minimal container - proxy with only container inspection/control enabled. - -### P1 — next, highest user impact - -- [ ] **L — Magazine: model-based layout detection and reading order.** Evaluate replacing the hand-written - region and order rules with dedicated models, as in the approach from - [Building agentic document understanding with OCR and LLMs](https://damimartinez.github.io/building-agentic-document-understanding-with-ocr-and-llms/): - **PaddleOCR PP-DocLayout** (regions: text, title, figure, chart, table, footnote) and **LayoutReader** - (a LayoutLMv3 fine-tune that orders regions from text + boxes). Before building: licences (PaddleOCR - Apache-2.0; LayoutLMv3 weights CC BY-NC-SA 4.0 -- non-commercial only), GPU/RAM on the DGX (already near - its limit), and a measurement on the same 204-page issue with the existing continuation score (current - heuristics: 89 confirmed continuations; any model must beat that). A tagged PDF, when a publisher ships one, - already carries the reading order in its structure tree -- read that first when present - ([reading-order best practice](https://www.globalvision.co/artwork-best-practices/reading-order)). - -- [ ] **M — Magazine podcasts via Open Notebook (multi-speaker).** [Open Notebook](https://github.com/lfnovo/open-notebook) - has an advanced multi-speaker podcast generator ("Generate professional podcasts") that could make - the magazine podcast mode much better. It already runs on the LAN at `192.168.178.19` (Docker: API - `:5055`, UI `:8500`, SurrealDB `:8501`). Two options, decide after reading its API and licence: - (a) **hook it up** — send an approved article to Open Notebook as a source, ask it for a podcast - script/episode, and play or voice the result here (a new external integration: check licence, data - flow — article text leaves this app —, auth, rate/cost, and availability; keep it optional with the - current podcast mode as fallback); or (b) **copy the approach** — port its speaker/episode-profile - design into the existing podcast mode and voice it with this app's own TTS voices. Prefer (a) if its - API exposes script generation cleanly, since the voices would still come from here. - -- [x] **M — Magazine: let the reader decide the region order.** Done 2026-09-21 (type a position, or Order by clicking; hand-ordered pages are locked). Original note: On the boundary-review page, - give each region an explicit reading position: click the regions in the order they should be - read, or type a number into each one. The order drives playback and the OCR-results list. - Today the order is whatever the layout analysis produced, and a page with two text columns - comes out interleaved, so a listener hears half a sentence from the wrong column. -- [ ] **L — Magazine: overhaul region boxes and how they are handled in OCR and approval.** (Done 2026-09-21: type picker and delete per row; Clean up boxes — text in pictures, OCR noise, duplicates, run-on author lines. Done too: keyboard navigation, merge with next, split at cursor, per-row re-OCR and text editing. Done 2026-09-28: *Clean up boxes* now fits text boxes to their ink (grows over clipped edges, trims empty margins; hand-placed boxes untouched; on the measured issue only 5 of 658 boxes on pages 1-59 needed it). Still open: the 333 overlapping boxes with different text on the measured issue need a human.) - The boxes the layout analysis produces need a pass of their own, on top of the ordering work - above. Seen on real pages: a byline box that runs on into the first body line ("Von Hartmut - Gieselmann Ein guter Butler kennt die Vorlieben seines"), so author and paragraph both come - out wrong; boxes nested inside a figure and OCRed as separate headlines/captions ("Apple - Intelligence" on the slide in the photo); sliver boxes holding OCR noise ("mtavt;-:,!«"); - overlapping boxes that fight over the same lines; and running heads/footers picked up as - regions every time. Wanted: snap boxes to the real text block, split or merge where a box - crosses a paragraph or column edge, drop or fold in regions inside a figure, discard - degenerate slivers, and make the approval screen quick to correct — keyboard moves between - regions, merge/split in place, and a retype/re-OCR of one region without leaving the page. - -- [x] **M — Magazine: read text columns top-to-bottom, then left-to-right.** Done 2026-09-21 (column anchors from left edges, bands, pull-quotes last; 85→89 continuations on a 204-page issue; *Fix reading order* for existing issues). Original note: Group detected text - regions into columns first, order each column downwards, then take the columns left to right - (mirroring for RTL). Confirmed wrong on page 38 of issue 204: the left column's paragraphs and - the right column's are mixed into one sequence. Should land before the manual ordering above, - which is then only a correction tool rather than routine work. - -- [x] **S — Re-measure audiobook segmentation against `cast_run3.json`.** Run 3 has - 2,227 segments, 1,043 dialogue segments and 585 newline-fused segments, but zero - adjacent same-speaker dialogue pairs for `_audiobookMergeAdjacentDialogue` to merge. - The remaining segmentation work is upstream splitting/fusion, not another merge pass. -- [x] **S — Finish the remaining Playwright smoke checks.** Characters/Cast, profile - sorting, column visibility, voice ratings, and starting/cancelling a generation job are covered. -- [x] **M — Fix the Schwester Larissa unclosed-guillemet split.** Attribution normalization now repairs dialogue - split before mixed German guillemet closes (`»…‹` / `«…›`), with an epigraph regression test. -- [x] **M — Make profile portraits and character sheets use one canonical identity.** - Shared structured prompts now version consistent pairs as v3 and fingerprint canonical appearance fields; add - cross-record validation so mismatched profile/reference files are detected even when both are current-version. The - UI now flags missing, stale, changed, or explicitly inconsistent pair metadata; visual similarity itself remains a - human review concern, while generated/replaced assets are now detected using stored SHA-256 digests. -- [x] **M — Add bounded duplicate-generation protection.** Consistent portrait/reference, voice-design, and casting - jobs now deduplicate repeated requests with bounded, expiring `Idempotency-Key` contracts. -- [x] **M — Improve large-table performance.** Character productions and the flat voice library paginate at 50 - visible rows; low-memory mode also reduces rendered rows and image work. -- [x] **M — Add explicit cache invalidation and eviction.** Fish.audio search responses, persisted synthesized - audio, image-generation job mappings, and browser object URLs have bounded cleanup. -- [x] **M — Add constrained-device validation.** Chrome CPU throttling, a 500-character workload, browser - DOM/audio/heap/image-cache bounds, and a hard 4 GiB current-release container check are covered. The `1.24.21` - image served successfully at 52.54 MiB idle usage, 0.10% CPU, 6 PIDs, and `OOMKilled=false`; cgroup memory was - confirmed as 4,294,967,296 bytes and temporary/root filesystem usage was sampled. -- [x] **L — Add background jobs and bounded queues.** Image generation already has the full interactive job path, - and transcription, voice design, and casting now have optional ID/status/cancel/timeout paths while preserving - synchronous callers. Job metadata now persists in SQLite and marks interrupted work after restart; durable binary - result storage is bounded and authenticated; outbound TTS and LLM calls now share a bounded transient-failure retry policy, and `/api/jobs` - exposes sanitized recent job history. Completed voice-design audio and consistent-image outputs now use bounded, - atomic artifact files with authenticated download URLs; transcription text remains deliberately non-persistent. - Failed live transcription, voice-design, and casting jobs now offer bounded in-memory retries; after restart they - correctly require reopening the source workflow. -- [x] **L — Run the Book 2 generalization regression.** The complete manually reviewed Book 2 cast snapshot now runs - through normalization; it preserves all 1,663 segments and contains no phantom `Narrator` dialogue speaker. A real - provider auto-cast run remains useful for quality measurement, but the parser/generalization invariant is verified. -- [x] **L — Add character forms inside one character record.** `sheet.forms[]` stores bounded alternate appearances, - prompts, and per-form visual assets; the detail-page switcher preserves one character ID and voice, and selected - forms feed prompt generation and identity fingerprints. -- [x] **L — Modularize the frontend incrementally.** The shared manifest, compatibility namespace - bridge, section-scoped lazy loading, and responsibility-level Audiobook/Rehearser/style extraction are complete; - a full ES-module migration remains a separate follow-up. -- [x] **M — Add structured logs with request/job IDs.** API/MCP requests now receive sanitized correlation IDs and - duration logs; route-specific logs retain provider/job context without request bodies. -- [x] **L — Add linting and duplicate-function checks.** The manifest-driven frontend syntax/declaration gate now runs - locally and in the SonarQube quality workflow alongside the existing Python and bundle checks. -- [x] Expose extracted Rehearser and Audiobook parser APIs through the `window.TTVC` namespace bridge. - -### P2 — deployment - -- [x] **S — Deploy the app to `correspondence-nuc` (192.168.178.19), pointed at the DGX Spark's TTS engine over the - network.** Live since 2026-09-14, `v1.27.5`. NFS export set up on the DGX (`/home/sparky/Media/_Sounds/TTS_Voices` - → `192.168.178.19` only, `rw,no_root_squash`), mounted read-write on the NUC at `/mnt/dgx-tts-voices` and - persisted in `/etc/fstab` there — confirmed the NUC instance sees the same 344 voices the DGX does, not a - separate copy. Deployed by rsyncing the working tree from the DGX (excluding `.git`, `config/`, `logs/` and other - local-only paths) to `/home/sparky/Docker/tts-voice-creator-clone-and-design-2` on the NUC and building the image - there directly — the published `ghcr.io` image turned out to be both several versions stale and private with no - registry credentials on the NUC, not worth setting up for an image that would need re-pushing on every deploy - anyway when the source is right there to build from. - **One real gap found along the way:** `vision_ocr_url`/`vision_ocr_fallback_url` default to - `host.docker.internal`, which resolves to *whichever host the container is running on* — correct on the DGX - itself, silently wrong on the NUC (would have pointed at the NUC's own, nonexistent OCR engine). Pinned both to - the DGX's LAN IP explicitly via environment overrides in the NUC's compose file. - **Also needed:** the NUC's `config`/`logs` bind-mount directories were created root-owned by `docker compose` - (this session's NUC access is a root SSH key) while the container's own `app` user is uid 1000 — `chown -R - 1000:1000` on both before the app could open its SQLite state DB. Confirmed the accepted limitation from before - still holds as expected: no `docker-socket-proxy` on the NUC instance (nothing local for it to control, and - cross-host container restart never worked anyway), so a voice change still needs a manual restart on the DGX. - NUC compose file tracked here as `docker-compose.nuc.yml` (the plain `docker-compose.yml` in this repo stays the - DGX's own, per its `build:`/`dgx_net`/`docker-socket-proxy` shape) — deployed to the NUC as its - `docker-compose.yml`, since that host has no `.git` checkout of its own to diff against, only the rsynced tree. - A code change means: re-rsync the tree (excluding `.git`, `config/`, `logs/`), `docker compose build`, `docker - compose up -d` — no separate push/pull step, same as the DGX's own bind-mount-and-recreate pattern. - **Footgun confirmed 2026-09-27:** `docker-compose.yml` must ALSO be excluded from the rsync (`--exclude - docker-compose.yml`) — otherwise it silently overwrites the NUC's own compose file with the DGX's (wrong - `build`/`image`, `dgx_net`/`docker-socket-proxy` the NUC doesn't have), and `docker compose up -d` then fails with - `network dgx_net declared as external, but could not be found`. Recovery: `cp docker-compose.nuc.yml - docker-compose.yml` on the NUC, then `up -d --force-recreate`. - -## Deferred — public multi-user readiness (cross-app initiative) - -Not scheduled; blocked on a decision the user wants to make once, shared across -all self-hosted apps, not per-app. See `architecture/multi-tenant-and-identity.md` -for the full research findings and design sketch. Do not start without a fresh -planning pass — this is a multi-week, cross-repo effort. - -- [ ] Design and stand up a shared identity service across all self-hosted apps - (interconnected user profile, not per-app accounts), building on the existing - OurFlashcards GoTrue + Postgres + PostgREST stack on the NUC rather than a new - auth system per app. -- [ ] Once shared identity exists: add user_id/owner_id columns to every table in - core/database.py, move voices/output/reader_library/audio-cache to per-user - paths, replace the single global settings.json with per-user settings, replace - the single shared external_api_key with per-user API keys. - - [ ] Includes provider API keys (llm_api_key, groq_api_key, elevenlabs_api_key, - engine_api_keys, etc.) — today one value per provider for the whole install - (gitignored + 0600 settings.json, never echoed to the browser once saved). - Each user needs their own key per provider, scoped under their identity- - service user id; see architecture/multi-tenant-and-identity.md §1. -- [ ] Replace the global TTS/STT/image concurrency semaphores (routes/tts.py, - routes/stt.py, routes/conversation.py) with per-user fair-share admission - control. -- [ ] Required consent attestation + audit trail for voice cloning (decided: - required). Checkbox at voice upload time, logged with timestamp/account/IP; - add a user_id column to generation_history so misuse is traceable per account. -- [ ] Add a reverse-proxy/TLS edge layer (CORS, CSRF, CSP/HSTS) for internet - exposure — nothing in this repo handles this today. - -### P2 — after behavior and safety stabilize - -- [x] **M — Add structured logs with request/job IDs.** Report provider, endpoint, - duration and failure reason without logging prompts, keys or book contents. -- [x] **M — Separate provider health from configuration status.** TTS/STT discovery now returns independent - `configured`, `available`, and bounded `health` details; backend help text shows actionable connectivity errors. -- [x] **M — Add visible job history and retry controls.** Expose recent failures and - safe retry state to users. -- [x] **L — Add memory diagnostics.** Show browser heap where supported, rendered rows, - cache size, active jobs, server/container memory and recent operation sizes. -- [x] **L — Add linting and duplicate-function checks.** Run them in CI alongside the - existing compile, bundle and SonarQube checks. - -## P0 — Testing, data safety, and resource limits - -- [x] Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported). -- [x] Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides. -- [x] Limit upload sizes, temporary files, and concurrent jobs. Upload/request caps, bounded caches, and bounded - TTS/STT/image-generation pools are implemented; durable background-job conversion remains a separate P1 item. -- [x] Use atomic writes for `.meta.json` files. -- [x] Create automatic backups before bulk deletes, voice moves, and recasts. -- [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints. - -## P1 — Modular frontend architecture - -- [x] Split `audiobook.js`, `rehearser.js`, and `style.css` by responsibility; Rehearser casting, playback, training, - voice design, I/O, library, stage, and recording blocks plus Audiobook casting, preview, export, and feature styles - are isolated. Core Reader state and compatibility event wiring remain ordered in the legacy entry points. -- [x] Extract Rehearser stage display and cast-sidebar controls into `rehearser-stage.js`. -- [x] Replace broad globals with namespaces such as `window.TTVC.audiobook`; pure leaves, Audiobook/Reader/Rehearser - feature modules, Studio, and character-library persistence now use private scopes with explicit compatibility exports. - Voice-library loading, rendering, benchmarking, import workflows, character-sheet extraction, character-library - rendering/actions, voice-inspector selection, shared utility helpers, cast compatibility, i18n, and settings/log UI - are also scoped; legacy entry points remain only as explicit compatibility exports. -- [x] Generate the feature-module list for `loader.js` and `scripts/minify.mjs` from one shared manifest; - larger responsibility splits remain. -- [x] Add a dependency-free frontend syntax and top-level declaration collision gate (`npm run check:frontend`); - a full style/lint policy remains a follow-up. -- [x] Extract Rehearser voice matching and voice design; matching/import helpers now live with the voice-picker module, - while timing-sensitive DOM wiring remains in the cast renderer. -- [x] Extract Rehearser voice-picker presentation and preview state into `rehearser-voice-picker.js`. -- [x] Extract Rehearser cast view, sorting, filtering, and toolbar wiring into `rehearser-cast.js`. -- [x] Extract Rehearser training transcript comparison helpers into `rehearser-train.js`. -- [x] Extract Rehearser character development and backend-selection controls into `rehearser-voice-design.js`. -- [x] Extract Audiobook quote segmentation and conservative turn-taking into `audiobook-segmentation.js`. -- [x] Extract Rehearser library persistence and IndexedDB migration into `rehearser-library.js`. -- [x] Extract Audiobook parsing and casting; parsing, segmentation, cast normalization/serialization, live cast view, - LLM orchestration, preview, and export now live in focused modules. Core Reader state and compatibility wiring remain - in `audiobook.js`. -- [x] Extract Audiobook parsing primitives and speaker-heuristic constants into `audiobook-parse.js`. -- [x] Extract Audiobook cast serialization and page-stamping helpers into `audiobook-cast.js`. -- [x] Split the Performance/resource diagnostic styles into `static/css/resource-diagnostics.css`. -- [x] Split background-job history and retry-panel styles into `static/css/background-jobs.css`. -- [x] Split Voice Design form and sample-grid styles into `static/css/voice-design.css`. -- [x] Split `style.css` by feature; Rehearser rules now live in `static/css/rehearser.css` alongside the existing - diagnostics, background-job, voice-design, and Audiobook/Reader feature styles. -- [x] Decision — do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests. - -## P1 — Frontend performance and memory safety - -- [x] Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened. The shared manifest - keeps the foundation in the core bundle and loads Reader/Audiobook, Rehearser, Library/Characters, and Studio - groups once on first navigation; deep links load their group before activation. -- [x] Paginate large character and voice tables; render approximately 25–50 visible rows, with a constrained-device - regression covering a 250-character production. -- [x] Cache `/api/voices` with a shared 30-second session cache, one in-flight request, explicit refresh, and - invalidation after library metadata/folder mutations; the SQLite voice index remains authoritative. -- [x] Lazy-load profile images and concept art. -- [x] Update only affected table rows instead of rebuilding entire tables where the operation is row-local; benchmark, - normalization, rating, and voice-note updates now patch the existing visible rows. Structural changes still trigger - a full filtered rebuild so sorting, grouping, and pagination remain correct. -- [x] Add an LRU cache for decoded Rehearser audio and remove old entries automatically. Persisted line-audio is - already bounded; profile/concept and voice-library images now use a bounded viewport-aware browser cache that - removes old decoded sources while retaining restoration URLs. -- [x] Revoke discarded object URLs with `URL.revokeObjectURL(objectUrl)`; completed synthesized voice previews and - character/Rehearser local auditions now release their Blob URLs when playback ends or is replaced. -- [x] Add low-memory mode using `navigator.deviceMemory` where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less. -- [x] Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM. Images use - lazy decoding, and character/voice pages reduce from 50 to 25 rows when `navigator.deviceMemory <= 4`. - -## P1 — Backend stability and bounded concurrency - -- [x] Introduce background jobs for voice design, transcription, casting, and image generation. Synchronous callers - remain supported while each long-running path has an optional bounded background job. -- [x] Give every job an ID, progress, cancellation, timeout, retry, and idempotency key. Failed live jobs offer - bounded retries from stripped in-memory inputs; retry data is never returned or persisted, and restart correctly - requires reopening the source workflow. -- [x] Prevent duplicate generation when a user clicks twice for image, voice-design, and casting submissions. -- [x] Centralize provider health checks and error handling. TTS, STT, and container URL probes now share the bounded - `core.provider_health.probe_provider` mechanics and normalized connection/timeout/HTTP/JSON-shape errors; their - provider-specific endpoint lists and authentication policies remain at the route layer. -- [x] Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user. Consistent - portrait/reference rendering, preview/style/seed-finder synthesis, and the OpenAI-compatible speech proxy are - capped at two active jobs; STT transcription and benchmarks are capped at one active job. -- [x] Use bounded server queues for the implemented synchronous generation paths; durable background-job queues remain - a separate outstanding item above. - -## P0 — Security - -- [x] Never expose provider API keys to frontend JavaScript or logs. Provider keys are masked on read and absent from - browser storage; the external-client key is an intentional, explicit handoff for caller authentication and its - regeneration response is non-cacheable. -- [x] Validate uploaded filenames and paths. -- [x] Strictly validate custom URLs to prevent SSRF, with private destinations available only when explicitly allowed - for local provider/container operation. -- [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints. -- [x] Run containers as non-root with dropped capabilities and read-only filesystems where practical. -- [x] Isolate Docker socket access behind a minimal privileged service. - -## P0 — Data integrity and recovery - -- [x] Validate character sheets and voice metadata with schemas. -- [x] Add database migrations instead of silently changing record shapes. -- [x] Use atomic writes for `.meta.json` files. -- [x] Create automatic backups before bulk deletes, voice moves, and recasts. -- [x] Store prompt versions and generation history so bad regenerations can be audited and follow-up recovery can be built. - -## P0 — Testing - -### Automated tests - -- [x] Character parsing and phantom-speaker prevention, including malformed-response prose preservation. -- [x] Voice-matching constraints reject invented voice IDs and avoid unnecessary duplicate assignments. -- [x] Profile-completeness counting for empty, partial, and fully populated sheets. -- [x] Rating persistence. -- [x] STT endpoint normalization and backend-response shape handling. -- [x] ComfyUI workflow submission, prompt injection, polling, and image retrieval. -- [x] API timeout clamping and bounded upstream error text; live-provider failure drills remain environment-specific. - -### Playwright smoke suite - -- [x] Load the app. -- [x] Verify the production bundle loads. -- [x] Verify language switching and Rehearser parsing. -- [x] Verify clone source tabs switch correctly. -- [x] Open Characters / Cast. -- [x] Sort the profile column. -- [x] Hide and show columns. -- [x] Rate a voice. -- [x] Start and cancel a generation job. - -## P2 — Observability - -- [x] Add structured logs with request and job IDs. -- [x] Report provider, endpoint, duration, and error reason clearly. -- [x] Separate health status from “configured”; discovery APIs now preserve both states and expose sanitized probe errors. -- [x] Add visible job history and safe retry state; the authenticated `/api/jobs` metadata endpoint now feeds a - rendered panel with status, elapsed time, errors, cancellation, and an explicit reopen-source retry boundary. -- [x] Avoid logging prompts, API keys, or book contents by default. - -## P1 — Resource limits and constrained-device testing - -- [x] Add explicit Docker memory limits; the web container defaults to `mem_limit: 8g` and `mem_reservation: 2g`, configurable per host. -- [x] Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits. -- [x] Test with Chrome DevTools-equivalent CPU throttling; desktop Playwright uses Chromium CDP at 4× CPU and - asserts navigation plus bounded DOM/audio counts, with JS heap captured when Chromium exposes it. -- [x] Test the current image in a container limited to 4 GB RAM; the app served `/` with a 4 GiB cgroup limit, - ~50.6 MiB idle memory, 0.09% CPU, and no OOM kill. -- [x] Monitor browser-tab memory, JavaScript heap where supported, DOM-node count, audio elements, persisted jobs, - server RSS, configured-data disk, and temporary disk usage through the read-only Performance diagnostics panel. -- [x] Add a memory diagnostics page showing cache count/size, Docker/container memory, and largest recent operations. - The authenticated panel reports browser cache counters, cgroup memory current/max, server RSS, configured/temp - storage, persisted jobs, and sanitized operation durations; host-wide Docker metrics remain unavailable by design. - -## P1 — Speaker attribution: measured state and next steps - -**Status 2026-09-19 — reworked, see `architecture/audiobook-attribution.md`.** Span mode (deterministic -quote spans, the LLM only names speakers) replaced the segmentation-heavy engine: Book 1 every-word-right -92.3 % (was 87.0 %), narration/dialogue agreement 97.4 % (was 93.6 %), ~10 min instead of ~4 h; Books 2/3 -(never tuned on) 90.8 % / 89.9 %. The word-aligned scorer is `scripts/score_attribution.py`; the references -have label noise, so 1–2 points is not meaningful. Everything below the line is the historical record of -the classic engine's Runs 1–3. - -Still open (each needs a real decision or new evidence, not more prompt tuning): - -- [ ] A person known only by a role that means different people in different scenes ("der Fremde") is not - resolved to a character — needs scene-level identity tracking; the label is kept as its own cast entry - until the book itself ties it to a name. -- [ ] Look-alike main characters in one scene (Marcian/Zerwas/Lancorian) remain the biggest real error - class (~1/5 of spoken words). Tried and rejected: reasoning mode (25× slower, no gain), larger/smaller - windows, tag-based overrides, reasoning adjudication of disputes (made disputed lines worse), an explicit - scene-cast hint (names mentioned in the passage: +0.3 points, noise). What remains is mostly model - capacity plus reference noise; the useful lever left is human review guided by the agreed/disputed flags. - ---- - - -### Ordered follow-up queue - -These follow-ups are ordered by dependency and expected benefit. Estimates assume one developer familiar with this -codebase; benefit is relative user/reliability impact. - -- [x] **P1 / M — High benefit — Identity-anchor regression tests.** Add automated coverage proving avatar and concept-sheet - requests share canonical identity anchors and persist the prompt version used for each generated asset. -- [x] **P1 / S–M — Medium benefit — Preserve dialogue-only merging.** Keep the dialogue-only merge for recast/verify - outputs where adjacent pairs still occur, with regression coverage for the non-dialogue boundaries. -- [x] **P1 / L–XL — Very high benefit — Rework segmentation from Run 3.** The initial-cast final normalization now - merges narration only, preserving dialogue turn boundaries; Run 3 showed same-speaker dialogue joins were obscuring - upstream split/fusion errors. Recast/verify retains its separately tested dialogue-only merge. -- [x] **P1 / M — High benefit — Re-run Book 1.** Done 2026-09-19 with the span-mode engine: overall 92.2 % (was 87.0 %), type agreement 97.4 % (was 93.6 %), ≈11 min instead of ≈4 h. See `architecture/audiobook-attribution.md`. -- [x] **P1 / L–XL — Very high benefit — Run Book 2 generalization.** Done 2026-09-19: Book 2 90.8 % and Book 3 89.9 % overall (neither tuned on) vs 92.2 % on Book 1 — the approach generalises. Note the references carry label noise, so 1–2 points is not meaningful. -- [x] **P1 / L — Very high benefit — Unify visual identity.** Portrait and concept-sheet generation now use one - canonical identity description, stored prompt version, identity fingerprint, asset digests, and stale-asset review - path; shared-anchor and metadata persistence are covered by automated tests. -- [x] **P2 / M–L — Medium–high benefit — Enrich character-form UX.** Done 2026-09-19: forms panel (add/rename/delete/show, per-form stale flag, per-form and all-forms image generation), Physical/Clothing edits scoped to the active form. See `architecture/character-visual-identity.md`. - -## Magazine reader roadmap - -Magazines are a separate Studio document type: they use a narrator and layout-aware article extraction, not character -casting. The original PDF remains the visual source of truth while approved articles become linked, reflowable HTML. - -### P0 — Magazine foundation - -- [x] Add a Magazine project type beside Books and Theater Plays. -- [x] Add PDF/JPG/PNG import with page previews and safe local storage. *(2026-09-29: JPG/PNG page images become one scan-style PDF, see architecture/magazine-ingestion.md)* -- [x] Add OCR-engine selection from Engines > OCR & Document Parsing, with page-image input and structured output. -- [x] Detect contents/index pages and propose article titles, authors, and page ranges. -- [x] Build a review screen where users approve or correct article boundaries before TTS. - *(includes "Edit boxes" — user-requested 2026-09-14: drag an already-detected region to move or resize it, - or delete it outright, with an optional re-OCR of the corrected area for text regions. "Draw it" remains the - separate tool for a region detection missed entirely.)* -- [x] Preserve source coordinates for every extracted block, image, caption, and page reference. - -- [x] Generate per-article podcast scripts from a connected LLM with two or three user-picked host voices, - with a Studio panel to pick hosts, generate, and play the script back turn by turn. -- [x] **Make the ingestion "watching it work" panel more visually appealing** *(done: live region boxes drawn over the page image, colour legend, reduced-motion safe; commit 1edc941, covered by tests/magazine.spec.js)* — user-requested 2026-09-14. Today it's a - static side-by-side of the current page image and a plain text/label list (`#mag-ingest-preview`, - `static/sections/s-magazine.html`, populated by `routes/magazine.py`'s preview-HTML builder around line 158). - Wanted instead: draw the actual detected region boxes (image/table/code/figure/etc.) live, overlaid directly on - the page image as each region is classified — watching the layout engine work through the page, not just a - before/after. Likely an SVG or absolutely-positioned overlay using each block's `bbox` (already computed and - already used for the review screen's region boxes, `magReviewMarkup`-adjacent code in `static/js/magazine.js`) - drawn incrementally, one box per region as it's classified, rather than all boxes appearing at once at the end. - Purely cosmetic/decorative per the existing comment on that panel — must not become the only place the - essential progress info (percentage, page count, done/failed) lives; that stays in `#mag-ingest-status`. - -### P1 — Magazine listening experience - -- [x] Generate clean article HTML with headings, captions, figures, sidebars, and page links. - *(headings/byline/captions styled and page-linked, figures shown during playback, and (2026-09-29) sidebars shown in - place as labelled, never-narrated asides)* -- [x] Generate narrator audio per article with resume/progress and a play-all queue. -- [x] Synchronize paragraph/sentence highlighting with playback and keep PDF and article views linked. - *(sentence-level; clicking a sentence or switching to the original-page view jumps to its source bbox)* -- [x] Show article images or page crops during playback, with optional caption reading. - *(images shown, matched to reading position; caption text is not separately spoken)* -- [x] Add confidence warnings, extraction diffs, and manual text correction before audio generation. - *(confidence badges, boundary editing, per-region text editing, and (2026-09-29) an Edited mark with a word diff against the recognised text and one-click restore)* -- [ ] **Redesign the magazine contents view** (`#s-magazine`, the article list for one issue). Today it is a flat - list of title + page range + badge + Review button, which shows nothing of the issue itself: no page, no - opening text, no figure, no sign of which articles already have audio. Requested directions, none decided: - - **Three columns.** Page thumbnail · headline + opening text · the article's figures. Reads down the issue - page by page in column one, with the text and the pictures beside it. - - **Synthesis state per row.** A "synthesised / not synthesised" marker, the MP3's duration as mm:ss, and an - inline play button — so the list doubles as the playlist instead of having to enter an article to find out. - - **Flipbook.** The PDF as a page-turning spread (issuu.com as the reference), with the extracted content of - the visible pages in two columns underneath it. - Decide the shape first — the three ideas are not variants of one layout, and the middle one (synthesis state) - is worth having under any of them. Whatever is chosen has to keep working at phone width and stay navigable - by keyboard and screen reader; a flipbook in particular needs a non-gestural alternative. - -### P2 — Advanced magazine intelligence - -- [x] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion. - *(sidebar exclusion via mark_sidebars, a narrow-column geometry heuristic; cross-page continuation now follows both - "continued on page N" and "continued from page N", 2026-09-29, unit-tested, not yet seen on a real issue)* -- [x] Add image descriptions, chart/table handling, and optional spoken figure summaries. - *(per-figure "Describe" in review generates a type-aware description — chart/table/code/figure each get their - own prompt — via the configured vision engine with fallback; editable before approval; spoken only if "Read - figure descriptions" is enabled. Live-verified against a real chart: the configured primary vision engine - turned out to be a pure-OCR model that returns empty content for any non-transcription prompt regardless of - phrasing — confirmed with three different prompts — so description generation only works when a genuinely - caption-capable model is reachable as the fallback; documented as a config-dependent limitation in Manual.md, - not a code defect.)* -- [~] Add word-level karaoke timing and EPUB 3 Media Overlay export. - *(EPUB 3 Media Overlay export done 2026-09-30, EPUBCheck-clean, with exact sentence timings for newly rendered audio; word-level - highlighting is still a character-length estimate within each sentence, not forced alignment; the in-app MP3 player still - aligns its transcript by estimate and could use the new timing records)* -- [x] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume. - *(driving mode with large controls, queue auto-advance and resume at the saved position; 2026-09-30: Media Session for lock screen / headset / car controls and artwork, screen wake lock while playing)* -- [x] Add OCR-engine benchmark comparisons and per-page fallback selection. - *(per-page fallback is a real engine chain: vision OCR escalates when a page needs OCR or scores poorly, KDL nano is the vision fallback, tesseract the last resort; 2026-09-29: `scripts/bench_ocr.py` scores any engine against a native-text PDF's own text layer, first results for tesseract, Unlimited-OCR, docling and PaddleOCR in `architecture/evidence/ocr-bench-2026-09-29/`)* -- [ ] **Evaluate additional OCR/vision candidates against the two already wired (baidu/Unlimited-OCR, - KDL-Frontier-Parser-nano) and pick the best fit per task** — user-requested 2026-09-14. Goal: low RAM/VRAM - footprint with excellent speed and quality, for both page-text transcription and open-ended figure/chart - description (the latter is the harder requirement — confirmed live that baidu/Unlimited-OCR is - transcription-only and cannot do it at all, see `architecture/magazine-ingestion.md` and CHANGELOG 1.29.0). - Candidates to install as Docker containers alongside the existing engines, each added through the Engines tab - (`routes/docker.py`'s `_LOCAL_CONTAINER_DEFS`, role `ocr`) so they're selectable/connectable/benchmarkable like - any other engine, not hardcoded: - - [x] [docling](https://github.com/docling-project/docling) — document conversion/layout toolkit. *(2026-09-29: benchmarked on CPU, F1 0.867, 8.9 s/page, 2.4 GB; see architecture/evidence/ocr-bench-2026-09-29/)* - - [x] [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) — established general-purpose OCR. *(2026-09-29: benchmarked on CPU, F1 0.881, 36.5 s/page, 1.4 GB; not yet wired as an engine)* - - [ ] [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) — vision-language OCR model. *(not run 2026-09-29: only ~14 GB of the unified memory is free; needs a running model stopped for a test window)* - - [ ] *(not run 2026-09-29: ~30 GB does not fit beside the running sglang model)* Qwen3-VL-30B-A3B-Instruct-FP8 — general vision-language model; check whether it fits the RAM/VRAM - budget at 30B-A3B FP8 (likely the heaviest candidate — weigh against the "low footprint" goal - specifically) and whether it's meaningfully better at open-ended description than the nano-class - candidates above. - For each: measure page-transcription accuracy and figure/chart-description quality against the existing - c't KI Wissen 2026 test corpus, plus idle/active VRAM footprint and latency, before deciding what replaces or - joins the current primary/fallback pair. - -### P2 — Finding magazines to read - -- [~] **Browse and search online PDF magazine sources from inside the app** *(done 2026-09-29/30: Internet Archive + OPDS catalogues with in-app management, login via an environment variable, and result paging, see architecture/magazine-sources.md; still open: publisher free issues/feeds, open-access journals)* - without leaving for a browser and uploading by hand. Wanted: a catalogue of sources, a search across them, and - a one-click "ingest this issue". - **Blocked on a source decision, and that decision is the whole task.** The two sites named as examples - (`freemagazines.top`, `pdf-magazines.org`) redistribute current commercial titles — c't, BBC Wildlife, - AirForces Monthly — without the publishers' licence, so building a scraper against them is out: it is - copyright infringement regardless of who presses the button, and both would be trivially blocked or taken - down anyway. Sources that can be built against, in rough order of usefulness here: - - **OPDS catalogues** — the open standard for exactly this. Calibre-Web, Kavita and Komga all serve OPDS, so a - generic OPDS client also covers the user's own library on the NAS. Start here. - - **Internet Archive** (`archive.org` API) — a genuinely large public-domain and openly-licensed periodical - collection, with a documented search API and no scraping needed. - - **Publisher-run free issues and RSS/Atom feeds** — heise gives away sample c't issues, many trade titles - publish a free archive. Per-publisher work, but legitimate and high quality. - - **Open-access journals** (DOAJ, arXiv, PubMed Central) — if the appetite is reading rather than magazines - specifically, these are the largest legal corpus by far and all have real APIs. - Whatever is added, check licensing, rate limits and terms per source before wiring it (core rule 8), honour - `robots.txt`, and keep each source behind its own provider interface so one going away does not break the rest. - -Three full regression runs of *Der Sturm* against the hand-corrected reference. -Everything below is measured, not estimated. Artefacts live in -`config/regression_snapshots/der_sturm/` (`*_manual.json` are the human -reference — never overwrite them). - -| Run | Segments | Headline agreement | **Fully correct** | Wrong speaker | Unmatched | -|-----|---------:|-------------------:|------------------:|--------------:|----------:| -| 1 | 2143 | 72.6% | **54.2%** | 438 | 544 | -| 2 | 2273 | 77.7% | **52.8%** | 345 | 727 | -| 3 | 2227 | 78.7% | **52.2%** | 314 | 750 | -| human reference | 2110 | — | — | 0 | — | - -**Read the headline with care.** Agreement is computed only over rows whose text -matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute -correct segments went 1160 → 1200 → 1161. The rising headline is largely a -shrinking denominator; true end-to-end accuracy has drifted slightly *down*. - -### What genuinely improved -- Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did **not** - overcorrect (369 after cast collapsed to 19 after one recast pass). -- Wrong speaker names 438 → 345 → 314 — consistent, real. -- Verify pass now completes (~65 LLM calls) instead of never finishing. - -### The blocker: segmentation, not attribution -750 of ~1060 problem segments are text that doesn't match the human's at all: -roughly 400 split finer than the human, ~210 merged where the human split, ~70 -OCR/wording differences. Attribution work cannot fix these. - -- `fused` dialogue segments hit **212** (human: 6, run 2: 59). These are - `_audiobookMergeAdjacentDialogue` joins using `\n\n`; 137 of them match no - human segment. Splitting them back was tested and is **worse** (52.2% → 43.4%), - so the merge is not the primary cause — but merging blocks the human kept - separate is contributing. -- The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it - went to 750. **That prediction did not hold** and needs re-deriving on run 3. -- Fixed: attribution normalization repairs the Schwester Larissa epigraph when the LLM splits it before a mixed - guillemet close; the regression is covered by `tests/test_conversation.py`. - (unclosed-guillemet handling). - -### Next steps, highest value first -1. Attack segmentation directly — it is now the dominant error class. Run 3 has - no merge candidates, so inspect the earlier dialogue split/fusion stages and the - 585 already-fused segments instead of changing `_audiobookMergeAdjacentDialogue`. -2. Preserve the dialogue-only merge for recast/verify outputs where adjacent pairs - can still occur; do not claim it explains the run-3 mismatch. -3. Only then re-run book 1. Re-running before segmentation moves will re-measure - rules already known not to shift the number that matters. -4. **Book 2 generalization run never started.** Reference is backed up at - `config/regression_snapshots/book2/`. Its point is to test whether these rules - generalise or were overfit to book 1 — still unanswered, and the confidence - scorer in particular was tuned solely on book 1. - -### Character profiles -48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got -worse than run 2's 51. Generic role labels (`Fremder` 19 lines, `Vampir` 19, -`Inquisitor` 15, `Ork` 13, and the adverb `Unsicher`) still become cast members -with their own voices; the appositive resolver only fires where the book supplies -grammatical evidence (`Darrag der Schmied`), and pre-reveal aliases like -"der Fremde" deliberately withhold it. - -### Harness notes (cost real time; do not relearn) -- Casting auto-starts `audiobookAutoRepairCast` 800 ms after finishing, which - re-sets `_audiobook.running`. Guards now explain the refusal instead of - returning silently, but an automated driver must still wait for `running` to - clear and treat a sub-60s phase as a bailout, not a success. -- Stall detection must key on the passage counter (`#ab-cv-count`). `segments`, - `done` and `unknown` do not move during a verify pass — watching them killed a - healthy run twice. -- Character Sheets only saves after its passage loop ends; poll `/api/characters` - until records appear before closing the browser, or a 4.4 h run is discarded. - -## P1 — Character forms (implemented 2026-09-04) - -Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der -Nacht", "der Wolfsmensch") need more than one appearance per character. - -**Decided:** forms live INSIDE one character record, not as separate records — one -voice, one attribution target, so cast lines saying "Zerwas" always resolve. The -sheet gains a form switcher; each form carries its own `physical`, `image_prompt` -and portrait. **Detection is automatic**: the sheet prompt asks the LLM to identify -whether a character has distinct forms and describe each, so it generalises to any -book (werewolves, disguises, transformations) without per-book rules. - -Implemented as bounded `sheet.forms[]` data inside the existing character record. The extraction prompt asks for -distinct forms, the detail page switches the active appearance, selected forms feed prompt/fingerprint generation, -and generated/uploaded portrait/reference assets remain attached to that form. One character ID and one voice are -preserved for attribution. A future polish pass can add richer inline form editing and per-form bulk-generation UX. - -## P1 — Portrait / concept-sheet identity mismatch - -The avatar and the concept sheet can show two different people (Lancorian: dark-haired -man vs white-haired mage). Cause is understood: `image_prompt` has two sources — the -LLM writes it as one of the 53 character-sheet fields, while -`_build_character_portrait_prompt` builds it deterministically from the same structured -facts the concept sheet uses. The deterministic one keeps both images consistent; the -LLM-written one omits identity anchors (age, hair, build) so the model invents them. - -### Status — resolved 2026-09-19 - -Verified against the code and covered by tests (see `architecture/character-visual-identity.md`): - -- [x] One canonical identity description feeds both the avatar and the concept sheet - (`_character_visual_facts`; portrait and reference prompts are built deterministically, the - active appearance form is overlaid first). -- [x] `image_prompt` is no longer authored by the LLM during sheet extraction (server and browser - prompts); the deterministic prompt is generated right after extraction, before avatar-only - generation and before per-form generation. Hand-edited prompts are never overwritten. -- [x] Prompt version, identity fingerprint and asset digests are stored with every generated pair - — for the base appearance too (they used to be lost on reload) and per form. -- [x] Review indicator: per-character banner and `GET /api/characters/visual-pair-review`, now - form-aware; a pair is also flagged when one image was regenerated alone. -- [x] Tests: server/browser fingerprint parity, prompt anchors shared by both builders, review - endpoint (base + forms), and Playwright persistence tests that fail on the previous code. - -Still true, and deliberately not "fixed": the resolved prompt *text* is not stored per image -(only its version and the fingerprint of the facts it was built from). - -### Known regression from the 2026-09-02 batch -Regenerating the top-10 profiles from evidence windows (passages around each character's -own dialogue) produced weaker `physical` text than the app's full-book pass — Lancorian's -became behaviour ("peers through small windows, moves nimbly on spiral staircases") -rather than appearance, where the older sheet had "shoulder-length white hair, faded scar -across forehead". Prefer the app's own Character Sheets flow over evidence-window -shortcuts when appearance quality matters. - -- [ ] **Magazine: deleting a picture/chart/table region leaves its crop file and article - reference behind.** Found 2026-09-22 while cleaning up two invented figure regions by - hand. `delete_region()`/`DELETE .../regions/{block_id}` matches only the block's own id; - a picture/chart/table's derived crop has a *different* id (`p{page}-region-{block_id}`, - from `crop_visual_regions`), so deleting the block leaves that crop file on disk and its - id in any article's `images` list (the article would still try to show a picture of - content that's no longer recognised as one). Fix: when the deleted id is a block with a - crop derived from it, also remove the `p{page}-region-{id}` image entry, delete its file, - and strip that derived id (not just the block id) from every article's `images` list. - -- [ ] **Audiobook character image runs must be isolated per character.** Before generating - a new character's profile image or character sheet, clear/reset every generation field, - reference image and transient conditioning value inherited from the preceding character. - Profile and sheet images for the *same* character should continue to share identity, but - no appearance traits (for example dwarf stature) may leak into the next character. Add a - regression test that generates two deliberately dissimilar characters consecutively and - verifies that the second request contains only its own identity/features/references. - -- [ ] **Characters table on phones: the sticky name column covers the rating stars.** At iPhone width the star buttons in the Characters table cannot be tapped (Playwright: `td.lib-chars-tbl-name` intercepts the click). Found 2026-09-29; the phone-width case of `tests/characters.spec.js` "profile sorting, column visibility, and voice rating" is `test.fixme` until fixed.