637 lines
48 KiB
Markdown
637 lines
48 KiB
Markdown
# TTS Voice Creator Roadmap
|
||
|
||
This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed.
|
||
|
||
Priority levels:
|
||
|
||
- **P0 — safety:** prevents crashes, data loss, broken releases, or security incidents.
|
||
- **P1 — high impact:** improves memory use, responsiveness, and maintainability.
|
||
- **P2 — quality:** improves diagnostics, polish, and long-term operations.
|
||
|
||
## Prioritized execution backlog
|
||
|
||
Effort estimates are for one developer familiar with this codebase: **S** = a few
|
||
hours, **M** = 1–2 days, **L** = several days, **XL** = multi-week or iterative.
|
||
Within each priority, tasks are ordered from smallest to largest effort. Complete
|
||
the first block before investing in architecture or polish.
|
||
|
||
## Current status — 2026-09-04
|
||
|
||
All P0 safety, security, data-integrity, and release-protection work is complete.
|
||
P1 implementation is active: pagination, shared voice caching, bounded decoded-audio
|
||
caching, background jobs, canonical character identity, forms, and the frontend
|
||
namespace/manifest foundations and section-scoped lazy loading are working and
|
||
regression-tested. Remaining P1 work is intentionally open where it needs a real
|
||
architecture change: the larger Audiobook/Rehearser/style split.
|
||
|
||
### Work order — set 2026-09-14
|
||
|
||
User-set priority for picking up feature work, in this order:
|
||
|
||
1. **Magazine reader roadmap** (below) — contents-view redesign is blocked on a
|
||
layout decision from the user; the other open items (image/chart descriptions,
|
||
legal source browsing) are actionable without one.
|
||
2. **Character system** — portrait/concept-sheet identity mismatch, then
|
||
character-form UX polish.
|
||
3. **Audiobook speaker attribution** — resume from Book 1 re-measurement, then
|
||
the never-yet-run Book 2 generalization regression.
|
||
|
||
Deployment and the deferred multi-user initiative below are explicitly parked,
|
||
not part of this order.
|
||
|
||
### P0 — do first
|
||
|
||
- [x] **S — Stop provider API keys from reaching frontend JavaScript or logs.** Provider keys are masked in settings
|
||
responses, probes resolve credentials server-side, and browser storage/log paths are clean. The separate external
|
||
caller key is available only through deliberate regeneration, uses `no-store`, and is not retained by the UI.
|
||
- [x] **S — Validate uploaded filenames and paths.** Upload endpoints reject traversal, path separators and control
|
||
characters before deriving extensions; ZIP imports reject absolute, traversal and malformed archive members.
|
||
- [x] **M — Add upload, request-size, temporary-file and concurrent-job limits.** Endpoint-specific upload/request
|
||
caps, bounded temporary work, cache eviction, and bounded TTS/STT/image job pools are implemented.
|
||
- [x] **M — Add automatic backups before bulk deletes, voice moves and recasts.** Make
|
||
recovery explicit and verify that backups can be restored.
|
||
- [x] **M — Strictly validate custom URLs and prevent SSRF.** HTTP downloads and provider/container probes share
|
||
scheme, hostname and private-destination validation with explicit local-network opt-in.
|
||
- [x] **M — Add authentication and rate limits for destructive/API endpoints.** External API authentication is
|
||
enabled by default and destructive/general mutation limits return bounded 429 responses.
|
||
- [x] **M — Validate character sheets and voice metadata with schemas.** Character/form and voice metadata are
|
||
bounded and type-checked at API boundaries.
|
||
- [x] **L — Add database migrations and prompt/generation history.** Preserve old record shapes and expose protected,
|
||
metadata-only generation history for auditing and follow-up recovery workflows.
|
||
- [x] **L — Harden container and Docker access.** The app runs non-root with dropped capabilities,
|
||
no-new-privileges and a read-only root filesystem; Compose Docker access is isolated behind a minimal container
|
||
proxy with only container inspection/control enabled.
|
||
|
||
### P1 — next, highest user impact
|
||
|
||
- [ ] **M — Magazine: let the reader decide the region order.** On the boundary-review page,
|
||
give each region an explicit reading position: click the regions in the order they should be
|
||
read, or type a number into each one. The order drives playback and the OCR-results list.
|
||
Today the order is whatever the layout analysis produced, and a page with two text columns
|
||
comes out interleaved, so a listener hears half a sentence from the wrong column.
|
||
- [ ] **L — Magazine: overhaul region boxes and how they are handled in OCR and approval.**
|
||
The boxes the layout analysis produces need a pass of their own, on top of the ordering work
|
||
above. Seen on real pages: a byline box that runs on into the first body line ("Von Hartmut
|
||
Gieselmann Ein guter Butler kennt die Vorlieben seines"), so author and paragraph both come
|
||
out wrong; boxes nested inside a figure and OCRed as separate headlines/captions ("Apple
|
||
Intelligence" on the slide in the photo); sliver boxes holding OCR noise ("mtavt;-:,!«");
|
||
overlapping boxes that fight over the same lines; and running heads/footers picked up as
|
||
regions every time. Wanted: snap boxes to the real text block, split or merge where a box
|
||
crosses a paragraph or column edge, drop or fold in regions inside a figure, discard
|
||
degenerate slivers, and make the approval screen quick to correct — keyboard moves between
|
||
regions, merge/split in place, and a retype/re-OCR of one region without leaving the page.
|
||
|
||
- [ ] **M — Magazine: read text columns top-to-bottom, then left-to-right.** Group detected text
|
||
regions into columns first, order each column downwards, then take the columns left to right
|
||
(mirroring for RTL). Confirmed wrong on page 38 of issue 204: the left column's paragraphs and
|
||
the right column's are mixed into one sequence. Should land before the manual ordering above,
|
||
which is then only a correction tool rather than routine work.
|
||
|
||
- [x] **S — Re-measure audiobook segmentation against `cast_run3.json`.** Run 3 has
|
||
2,227 segments, 1,043 dialogue segments and 585 newline-fused segments, but zero
|
||
adjacent same-speaker dialogue pairs for `_audiobookMergeAdjacentDialogue` to merge.
|
||
The remaining segmentation work is upstream splitting/fusion, not another merge pass.
|
||
- [x] **S — Finish the remaining Playwright smoke checks.** Characters/Cast, profile
|
||
sorting, column visibility, voice ratings, and starting/cancelling a generation job are covered.
|
||
- [x] **M — Fix the Schwester Larissa unclosed-guillemet split.** Attribution normalization now repairs dialogue
|
||
split before mixed German guillemet closes (`»…‹` / `«…›`), with an epigraph regression test.
|
||
- [x] **M — Make profile portraits and character sheets use one canonical identity.**
|
||
Shared structured prompts now version consistent pairs as v3 and fingerprint canonical appearance fields; add
|
||
cross-record validation so mismatched profile/reference files are detected even when both are current-version. The
|
||
UI now flags missing, stale, changed, or explicitly inconsistent pair metadata; visual similarity itself remains a
|
||
human review concern, while generated/replaced assets are now detected using stored SHA-256 digests.
|
||
- [x] **M — Add bounded duplicate-generation protection.** Consistent portrait/reference, voice-design, and casting
|
||
jobs now deduplicate repeated requests with bounded, expiring `Idempotency-Key` contracts.
|
||
- [x] **M — Improve large-table performance.** Character productions and the flat voice library paginate at 50
|
||
visible rows; low-memory mode also reduces rendered rows and image work.
|
||
- [x] **M — Add explicit cache invalidation and eviction.** Fish.audio search responses, persisted synthesized
|
||
audio, image-generation job mappings, and browser object URLs have bounded cleanup.
|
||
- [x] **M — Add constrained-device validation.** Chrome CPU throttling, a 500-character workload, browser
|
||
DOM/audio/heap/image-cache bounds, and a hard 4 GiB current-release container check are covered. The `1.24.21`
|
||
image served successfully at 52.54 MiB idle usage, 0.10% CPU, 6 PIDs, and `OOMKilled=false`; cgroup memory was
|
||
confirmed as 4,294,967,296 bytes and temporary/root filesystem usage was sampled.
|
||
- [x] **L — Add background jobs and bounded queues.** Image generation already has the full interactive job path,
|
||
and transcription, voice design, and casting now have optional ID/status/cancel/timeout paths while preserving
|
||
synchronous callers. Job metadata now persists in SQLite and marks interrupted work after restart; durable binary
|
||
result storage is bounded and authenticated; outbound TTS and LLM calls now share a bounded transient-failure retry policy, and `/api/jobs`
|
||
exposes sanitized recent job history. Completed voice-design audio and consistent-image outputs now use bounded,
|
||
atomic artifact files with authenticated download URLs; transcription text remains deliberately non-persistent.
|
||
Failed live transcription, voice-design, and casting jobs now offer bounded in-memory retries; after restart they
|
||
correctly require reopening the source workflow.
|
||
- [x] **L — Run the Book 2 generalization regression.** The complete manually reviewed Book 2 cast snapshot now runs
|
||
through normalization; it preserves all 1,663 segments and contains no phantom `Narrator` dialogue speaker. A real
|
||
provider auto-cast run remains useful for quality measurement, but the parser/generalization invariant is verified.
|
||
- [x] **L — Add character forms inside one character record.** `sheet.forms[]` stores bounded alternate appearances,
|
||
prompts, and per-form visual assets; the detail-page switcher preserves one character ID and voice, and selected
|
||
forms feed prompt generation and identity fingerprints.
|
||
- [x] **L — Modularize the frontend incrementally.** The shared manifest, compatibility namespace
|
||
bridge, section-scoped lazy loading, and responsibility-level Audiobook/Rehearser/style extraction are complete;
|
||
a full ES-module migration remains a separate follow-up.
|
||
- [x] **M — Add structured logs with request/job IDs.** API/MCP requests now receive sanitized correlation IDs and
|
||
duration logs; route-specific logs retain provider/job context without request bodies.
|
||
- [x] **L — Add linting and duplicate-function checks.** The manifest-driven frontend syntax/declaration gate now runs
|
||
locally and in the SonarQube quality workflow alongside the existing Python and bundle checks.
|
||
- [x] Expose extracted Rehearser and Audiobook parser APIs through the `window.TTVC` namespace bridge.
|
||
|
||
### P2 — deployment
|
||
|
||
- [x] **S — Deploy the app to `correspondence-nuc` (192.168.178.19), pointed at the DGX Spark's TTS engine over the
|
||
network.** Live since 2026-09-14, `v1.27.5`. NFS export set up on the DGX (`/home/sparky/Media/_Sounds/TTS_Voices`
|
||
→ `192.168.178.19` only, `rw,no_root_squash`), mounted read-write on the NUC at `/mnt/dgx-tts-voices` and
|
||
persisted in `/etc/fstab` there — confirmed the NUC instance sees the same 344 voices the DGX does, not a
|
||
separate copy. Deployed by rsyncing the working tree from the DGX (excluding `.git`, `config/`, `logs/` and other
|
||
local-only paths) to `/home/sparky/Docker/tts-voice-creator-clone-and-design-2` on the NUC and building the image
|
||
there directly — the published `ghcr.io` image turned out to be both several versions stale and private with no
|
||
registry credentials on the NUC, not worth setting up for an image that would need re-pushing on every deploy
|
||
anyway when the source is right there to build from.
|
||
**One real gap found along the way:** `vision_ocr_url`/`vision_ocr_fallback_url` default to
|
||
`host.docker.internal`, which resolves to *whichever host the container is running on* — correct on the DGX
|
||
itself, silently wrong on the NUC (would have pointed at the NUC's own, nonexistent OCR engine). Pinned both to
|
||
the DGX's LAN IP explicitly via environment overrides in the NUC's compose file.
|
||
**Also needed:** the NUC's `config`/`logs` bind-mount directories were created root-owned by `docker compose`
|
||
(this session's NUC access is a root SSH key) while the container's own `app` user is uid 1000 — `chown -R
|
||
1000:1000` on both before the app could open its SQLite state DB. Confirmed the accepted limitation from before
|
||
still holds as expected: no `docker-socket-proxy` on the NUC instance (nothing local for it to control, and
|
||
cross-host container restart never worked anyway), so a voice change still needs a manual restart on the DGX.
|
||
NUC compose file tracked here as `docker-compose.nuc.yml` (the plain `docker-compose.yml` in this repo stays the
|
||
DGX's own, per its `build:`/`dgx_net`/`docker-socket-proxy` shape) — deployed to the NUC as its
|
||
`docker-compose.yml`, since that host has no `.git` checkout of its own to diff against, only the rsynced tree.
|
||
A code change means: re-rsync the tree (excluding `.git`, `config/`, `logs/`), `docker compose build`, `docker
|
||
compose up -d` — no separate push/pull step, same as the DGX's own bind-mount-and-recreate pattern.
|
||
|
||
## Deferred — public multi-user readiness (cross-app initiative)
|
||
|
||
Not scheduled; blocked on a decision the user wants to make once, shared across
|
||
all self-hosted apps, not per-app. See `architecture/multi-tenant-and-identity.md`
|
||
for the full research findings and design sketch. Do not start without a fresh
|
||
planning pass — this is a multi-week, cross-repo effort.
|
||
|
||
- [ ] Design and stand up a shared identity service across all self-hosted apps
|
||
(interconnected user profile, not per-app accounts), building on the existing
|
||
OurFlashcards GoTrue + Postgres + PostgREST stack on the NUC rather than a new
|
||
auth system per app.
|
||
- [ ] Once shared identity exists: add user_id/owner_id columns to every table in
|
||
core/database.py, move voices/output/reader_library/audio-cache to per-user
|
||
paths, replace the single global settings.json with per-user settings, replace
|
||
the single shared external_api_key with per-user API keys.
|
||
- [ ] Includes provider API keys (llm_api_key, groq_api_key, elevenlabs_api_key,
|
||
engine_api_keys, etc.) — today one value per provider for the whole install
|
||
(gitignored + 0600 settings.json, never echoed to the browser once saved).
|
||
Each user needs their own key per provider, scoped under their identity-
|
||
service user id; see architecture/multi-tenant-and-identity.md §1.
|
||
- [ ] Replace the global TTS/STT/image concurrency semaphores (routes/tts.py,
|
||
routes/stt.py, routes/conversation.py) with per-user fair-share admission
|
||
control.
|
||
- [ ] Required consent attestation + audit trail for voice cloning (decided:
|
||
required). Checkbox at voice upload time, logged with timestamp/account/IP;
|
||
add a user_id column to generation_history so misuse is traceable per account.
|
||
- [ ] Add a reverse-proxy/TLS edge layer (CORS, CSRF, CSP/HSTS) for internet
|
||
exposure — nothing in this repo handles this today.
|
||
|
||
### P2 — after behavior and safety stabilize
|
||
|
||
- [x] **M — Add structured logs with request/job IDs.** Report provider, endpoint,
|
||
duration and failure reason without logging prompts, keys or book contents.
|
||
- [x] **M — Separate provider health from configuration status.** TTS/STT discovery now returns independent
|
||
`configured`, `available`, and bounded `health` details; backend help text shows actionable connectivity errors.
|
||
- [x] **M — Add visible job history and retry controls.** Expose recent failures and
|
||
safe retry state to users.
|
||
- [x] **L — Add memory diagnostics.** Show browser heap where supported, rendered rows,
|
||
cache size, active jobs, server/container memory and recent operation sizes.
|
||
- [x] **L — Add linting and duplicate-function checks.** Run them in CI alongside the
|
||
existing compile, bundle and SonarQube checks.
|
||
|
||
## P0 — Testing, data safety, and resource limits
|
||
|
||
- [x] Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported).
|
||
- [x] Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides.
|
||
- [x] Limit upload sizes, temporary files, and concurrent jobs. Upload/request caps, bounded caches, and bounded
|
||
TTS/STT/image-generation pools are implemented; durable background-job conversion remains a separate P1 item.
|
||
- [x] Use atomic writes for `.meta.json` files.
|
||
- [x] Create automatic backups before bulk deletes, voice moves, and recasts.
|
||
- [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints.
|
||
|
||
## P1 — Modular frontend architecture
|
||
|
||
- [x] Split `audiobook.js`, `rehearser.js`, and `style.css` by responsibility; Rehearser casting, playback, training,
|
||
voice design, I/O, library, stage, and recording blocks plus Audiobook casting, preview, export, and feature styles
|
||
are isolated. Core Reader state and compatibility event wiring remain ordered in the legacy entry points.
|
||
- [x] Extract Rehearser stage display and cast-sidebar controls into `rehearser-stage.js`.
|
||
- [x] Replace broad globals with namespaces such as `window.TTVC.audiobook`; pure leaves, Audiobook/Reader/Rehearser
|
||
feature modules, Studio, and character-library persistence now use private scopes with explicit compatibility exports.
|
||
Voice-library loading, rendering, benchmarking, import workflows, character-sheet extraction, character-library
|
||
rendering/actions, voice-inspector selection, shared utility helpers, cast compatibility, i18n, and settings/log UI
|
||
are also scoped; legacy entry points remain only as explicit compatibility exports.
|
||
- [x] Generate the feature-module list for `loader.js` and `scripts/minify.mjs` from one shared manifest;
|
||
larger responsibility splits remain.
|
||
- [x] Add a dependency-free frontend syntax and top-level declaration collision gate (`npm run check:frontend`);
|
||
a full style/lint policy remains a follow-up.
|
||
- [x] Extract Rehearser voice matching and voice design; matching/import helpers now live with the voice-picker module,
|
||
while timing-sensitive DOM wiring remains in the cast renderer.
|
||
- [x] Extract Rehearser voice-picker presentation and preview state into `rehearser-voice-picker.js`.
|
||
- [x] Extract Rehearser cast view, sorting, filtering, and toolbar wiring into `rehearser-cast.js`.
|
||
- [x] Extract Rehearser training transcript comparison helpers into `rehearser-train.js`.
|
||
- [x] Extract Rehearser character development and backend-selection controls into `rehearser-voice-design.js`.
|
||
- [x] Extract Audiobook quote segmentation and conservative turn-taking into `audiobook-segmentation.js`.
|
||
- [x] Extract Rehearser library persistence and IndexedDB migration into `rehearser-library.js`.
|
||
- [x] Extract Audiobook parsing and casting; parsing, segmentation, cast normalization/serialization, live cast view,
|
||
LLM orchestration, preview, and export now live in focused modules. Core Reader state and compatibility wiring remain
|
||
in `audiobook.js`.
|
||
- [x] Extract Audiobook parsing primitives and speaker-heuristic constants into `audiobook-parse.js`.
|
||
- [x] Extract Audiobook cast serialization and page-stamping helpers into `audiobook-cast.js`.
|
||
- [x] Split the Performance/resource diagnostic styles into `static/css/resource-diagnostics.css`.
|
||
- [x] Split background-job history and retry-panel styles into `static/css/background-jobs.css`.
|
||
- [x] Split Voice Design form and sample-grid styles into `static/css/voice-design.css`.
|
||
- [x] Split `style.css` by feature; Rehearser rules now live in `static/css/rehearser.css` alongside the existing
|
||
diagnostics, background-job, voice-design, and Audiobook/Reader feature styles.
|
||
- [x] Decision — do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests.
|
||
|
||
## P1 — Frontend performance and memory safety
|
||
|
||
- [x] Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened. The shared manifest
|
||
keeps the foundation in the core bundle and loads Reader/Audiobook, Rehearser, Library/Characters, and Studio
|
||
groups once on first navigation; deep links load their group before activation.
|
||
- [x] Paginate large character and voice tables; render approximately 25–50 visible rows, with a constrained-device
|
||
regression covering a 250-character production.
|
||
- [x] Cache `/api/voices` with a shared 30-second session cache, one in-flight request, explicit refresh, and
|
||
invalidation after library metadata/folder mutations; the SQLite voice index remains authoritative.
|
||
- [x] Lazy-load profile images and concept art.
|
||
- [x] Update only affected table rows instead of rebuilding entire tables where the operation is row-local; benchmark,
|
||
normalization, rating, and voice-note updates now patch the existing visible rows. Structural changes still trigger
|
||
a full filtered rebuild so sorting, grouping, and pagination remain correct.
|
||
- [x] Add an LRU cache for decoded Rehearser audio and remove old entries automatically. Persisted line-audio is
|
||
already bounded; profile/concept and voice-library images now use a bounded viewport-aware browser cache that
|
||
removes old decoded sources while retaining restoration URLs.
|
||
- [x] Revoke discarded object URLs with `URL.revokeObjectURL(objectUrl)`; completed synthesized voice previews and
|
||
character/Rehearser local auditions now release their Blob URLs when playback ends or is replaced.
|
||
- [x] Add low-memory mode using `navigator.deviceMemory` where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less.
|
||
- [x] Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM. Images use
|
||
lazy decoding, and character/voice pages reduce from 50 to 25 rows when `navigator.deviceMemory <= 4`.
|
||
|
||
## P1 — Backend stability and bounded concurrency
|
||
|
||
- [x] Introduce background jobs for voice design, transcription, casting, and image generation. Synchronous callers
|
||
remain supported while each long-running path has an optional bounded background job.
|
||
- [x] Give every job an ID, progress, cancellation, timeout, retry, and idempotency key. Failed live jobs offer
|
||
bounded retries from stripped in-memory inputs; retry data is never returned or persisted, and restart correctly
|
||
requires reopening the source workflow.
|
||
- [x] Prevent duplicate generation when a user clicks twice for image, voice-design, and casting submissions.
|
||
- [x] Centralize provider health checks and error handling. TTS, STT, and container URL probes now share the bounded
|
||
`core.provider_health.probe_provider` mechanics and normalized connection/timeout/HTTP/JSON-shape errors; their
|
||
provider-specific endpoint lists and authentication policies remain at the route layer.
|
||
- [x] Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user. Consistent
|
||
portrait/reference rendering, preview/style/seed-finder synthesis, and the OpenAI-compatible speech proxy are
|
||
capped at two active jobs; STT transcription and benchmarks are capped at one active job.
|
||
- [x] Use bounded server queues for the implemented synchronous generation paths; durable background-job queues remain
|
||
a separate outstanding item above.
|
||
|
||
## P0 — Security
|
||
|
||
- [x] Never expose provider API keys to frontend JavaScript or logs. Provider keys are masked on read and absent from
|
||
browser storage; the external-client key is an intentional, explicit handoff for caller authentication and its
|
||
regeneration response is non-cacheable.
|
||
- [x] Validate uploaded filenames and paths.
|
||
- [x] Strictly validate custom URLs to prevent SSRF, with private destinations available only when explicitly allowed
|
||
for local provider/container operation.
|
||
- [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints.
|
||
- [x] Run containers as non-root with dropped capabilities and read-only filesystems where practical.
|
||
- [x] Isolate Docker socket access behind a minimal privileged service.
|
||
|
||
## P0 — Data integrity and recovery
|
||
|
||
- [x] Validate character sheets and voice metadata with schemas.
|
||
- [x] Add database migrations instead of silently changing record shapes.
|
||
- [x] Use atomic writes for `.meta.json` files.
|
||
- [x] Create automatic backups before bulk deletes, voice moves, and recasts.
|
||
- [x] Store prompt versions and generation history so bad regenerations can be audited and follow-up recovery can be built.
|
||
|
||
## P0 — Testing
|
||
|
||
### Automated tests
|
||
|
||
- [x] Character parsing and phantom-speaker prevention, including malformed-response prose preservation.
|
||
- [x] Voice-matching constraints reject invented voice IDs and avoid unnecessary duplicate assignments.
|
||
- [x] Profile-completeness counting for empty, partial, and fully populated sheets.
|
||
- [x] Rating persistence.
|
||
- [x] STT endpoint normalization and backend-response shape handling.
|
||
- [x] ComfyUI workflow submission, prompt injection, polling, and image retrieval.
|
||
- [x] API timeout clamping and bounded upstream error text; live-provider failure drills remain environment-specific.
|
||
|
||
### Playwright smoke suite
|
||
|
||
- [x] Load the app.
|
||
- [x] Verify the production bundle loads.
|
||
- [x] Verify language switching and Rehearser parsing.
|
||
- [x] Verify clone source tabs switch correctly.
|
||
- [x] Open Characters / Cast.
|
||
- [x] Sort the profile column.
|
||
- [x] Hide and show columns.
|
||
- [x] Rate a voice.
|
||
- [x] Start and cancel a generation job.
|
||
|
||
## P2 — Observability
|
||
|
||
- [x] Add structured logs with request and job IDs.
|
||
- [x] Report provider, endpoint, duration, and error reason clearly.
|
||
- [x] Separate health status from “configured”; discovery APIs now preserve both states and expose sanitized probe errors.
|
||
- [x] Add visible job history and safe retry state; the authenticated `/api/jobs` metadata endpoint now feeds a
|
||
rendered panel with status, elapsed time, errors, cancellation, and an explicit reopen-source retry boundary.
|
||
- [x] Avoid logging prompts, API keys, or book contents by default.
|
||
|
||
## P1 — Resource limits and constrained-device testing
|
||
|
||
- [x] Add explicit Docker memory limits; the web container defaults to `mem_limit: 8g` and `mem_reservation: 2g`, configurable per host.
|
||
- [x] Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits.
|
||
- [x] Test with Chrome DevTools-equivalent CPU throttling; desktop Playwright uses Chromium CDP at 4× CPU and
|
||
asserts navigation plus bounded DOM/audio counts, with JS heap captured when Chromium exposes it.
|
||
- [x] Test the current image in a container limited to 4 GB RAM; the app served `/` with a 4 GiB cgroup limit,
|
||
~50.6 MiB idle memory, 0.09% CPU, and no OOM kill.
|
||
- [x] Monitor browser-tab memory, JavaScript heap where supported, DOM-node count, audio elements, persisted jobs,
|
||
server RSS, configured-data disk, and temporary disk usage through the read-only Performance diagnostics panel.
|
||
- [x] Add a memory diagnostics page showing cache count/size, Docker/container memory, and largest recent operations.
|
||
The authenticated panel reports browser cache counters, cgroup memory current/max, server RSS, configured/temp
|
||
storage, persisted jobs, and sanitized operation durations; host-wide Docker metrics remain unavailable by design.
|
||
|
||
## P1 — Speaker attribution: measured state and next steps
|
||
|
||
**Status 2026-09-19 — reworked, see `architecture/audiobook-attribution.md`.** Span mode (deterministic
|
||
quote spans, the LLM only names speakers) replaced the segmentation-heavy engine: Book 1 every-word-right
|
||
92.3 % (was 87.0 %), narration/dialogue agreement 97.4 % (was 93.6 %), ~10 min instead of ~4 h; Books 2/3
|
||
(never tuned on) 90.8 % / 89.9 %. The word-aligned scorer is `scripts/score_attribution.py`; the references
|
||
have label noise, so 1–2 points is not meaningful. Everything below the line is the historical record of
|
||
the classic engine's Runs 1–3.
|
||
|
||
Still open (each needs a real decision or new evidence, not more prompt tuning):
|
||
|
||
- [ ] A person known only by a role that means different people in different scenes ("der Fremde") is not
|
||
resolved to a character — needs scene-level identity tracking; the label is kept as its own cast entry
|
||
until the book itself ties it to a name.
|
||
- [ ] Look-alike main characters in one scene (Marcian/Zerwas/Lancorian) remain the biggest real error
|
||
class (~1/5 of spoken words). Tried and rejected: reasoning mode (25× slower, no gain), larger/smaller
|
||
windows, tag-based overrides, reasoning adjudication of disputes (made disputed lines worse), an explicit
|
||
scene-cast hint (names mentioned in the passage: +0.3 points, noise). What remains is mostly model
|
||
capacity plus reference noise; the useful lever left is human review guided by the agreed/disputed flags.
|
||
|
||
---
|
||
|
||
|
||
### Ordered follow-up queue
|
||
|
||
These follow-ups are ordered by dependency and expected benefit. Estimates assume one developer familiar with this
|
||
codebase; benefit is relative user/reliability impact.
|
||
|
||
- [x] **P1 / M — High benefit — Identity-anchor regression tests.** Add automated coverage proving avatar and concept-sheet
|
||
requests share canonical identity anchors and persist the prompt version used for each generated asset.
|
||
- [x] **P1 / S–M — Medium benefit — Preserve dialogue-only merging.** Keep the dialogue-only merge for recast/verify
|
||
outputs where adjacent pairs still occur, with regression coverage for the non-dialogue boundaries.
|
||
- [x] **P1 / L–XL — Very high benefit — Rework segmentation from Run 3.** The initial-cast final normalization now
|
||
merges narration only, preserving dialogue turn boundaries; Run 3 showed same-speaker dialogue joins were obscuring
|
||
upstream split/fusion errors. Recast/verify retains its separately tested dialogue-only merge.
|
||
- [x] **P1 / M — High benefit — Re-run Book 1.** Done 2026-09-19 with the span-mode engine: overall 92.2 % (was 87.0 %), type agreement 97.4 % (was 93.6 %), ≈11 min instead of ≈4 h. See `architecture/audiobook-attribution.md`.
|
||
- [x] **P1 / L–XL — Very high benefit — Run Book 2 generalization.** Done 2026-09-19: Book 2 90.8 % and Book 3 89.9 % overall (neither tuned on) vs 92.2 % on Book 1 — the approach generalises. Note the references carry label noise, so 1–2 points is not meaningful.
|
||
- [x] **P1 / L — Very high benefit — Unify visual identity.** Portrait and concept-sheet generation now use one
|
||
canonical identity description, stored prompt version, identity fingerprint, asset digests, and stale-asset review
|
||
path; shared-anchor and metadata persistence are covered by automated tests.
|
||
- [x] **P2 / M–L — Medium–high benefit — Enrich character-form UX.** Done 2026-09-19: forms panel (add/rename/delete/show, per-form stale flag, per-form and all-forms image generation), Physical/Clothing edits scoped to the active form. See `architecture/character-visual-identity.md`.
|
||
|
||
## Magazine reader roadmap
|
||
|
||
Magazines are a separate Studio document type: they use a narrator and layout-aware article extraction, not character
|
||
casting. The original PDF remains the visual source of truth while approved articles become linked, reflowable HTML.
|
||
|
||
### P0 — Magazine foundation
|
||
|
||
- [x] Add a Magazine project type beside Books and Theater Plays.
|
||
- [~] Add PDF/JPG/PNG import with page previews and safe local storage. *(PDF + page renders done; JPG/PNG not done)*
|
||
- [x] Add OCR-engine selection from Engines > OCR & Document Parsing, with page-image input and structured output.
|
||
- [x] Detect contents/index pages and propose article titles, authors, and page ranges.
|
||
- [x] Build a review screen where users approve or correct article boundaries before TTS.
|
||
*(includes "Edit boxes" — user-requested 2026-09-14: drag an already-detected region to move or resize it,
|
||
or delete it outright, with an optional re-OCR of the corrected area for text regions. "Draw it" remains the
|
||
separate tool for a region detection missed entirely.)*
|
||
- [x] Preserve source coordinates for every extracted block, image, caption, and page reference.
|
||
|
||
- [x] Generate per-article podcast scripts from a connected LLM with two or three user-picked host voices,
|
||
with a Studio panel to pick hosts, generate, and play the script back turn by turn.
|
||
- [ ] **Make the ingestion "watching it work" panel more visually appealing** — user-requested 2026-09-14. Today it's a
|
||
static side-by-side of the current page image and a plain text/label list (`#mag-ingest-preview`,
|
||
`static/sections/s-magazine.html`, populated by `routes/magazine.py`'s preview-HTML builder around line 158).
|
||
Wanted instead: draw the actual detected region boxes (image/table/code/figure/etc.) live, overlaid directly on
|
||
the page image as each region is classified — watching the layout engine work through the page, not just a
|
||
before/after. Likely an SVG or absolutely-positioned overlay using each block's `bbox` (already computed and
|
||
already used for the review screen's region boxes, `magReviewMarkup`-adjacent code in `static/js/magazine.js`)
|
||
drawn incrementally, one box per region as it's classified, rather than all boxes appearing at once at the end.
|
||
Purely cosmetic/decorative per the existing comment on that panel — must not become the only place the
|
||
essential progress info (percentage, page count, done/failed) lives; that stays in `#mag-ingest-status`.
|
||
|
||
### P1 — Magazine listening experience
|
||
|
||
- [~] Generate clean article HTML with headings, captions, figures, sidebars, and page links.
|
||
*(headings/byline/captions styled and page-linked, figures now shown during playback; sidebars are auto-excluded
|
||
from narration rather than rendered inline, per the P2 sidebar-exclusion item below)*
|
||
- [x] Generate narrator audio per article with resume/progress and a play-all queue.
|
||
- [x] Synchronize paragraph/sentence highlighting with playback and keep PDF and article views linked.
|
||
*(sentence-level; clicking a sentence or switching to the original-page view jumps to its source bbox)*
|
||
- [x] Show article images or page crops during playback, with optional caption reading.
|
||
*(images shown, matched to reading position; caption text is not separately spoken)*
|
||
- [~] Add confidence warnings, extraction diffs, and manual text correction before audio generation.
|
||
*(confidence badges + block-level boundary editing done; no extraction diff view or free-text correction)*
|
||
- [ ] **Redesign the magazine contents view** (`#s-magazine`, the article list for one issue). Today it is a flat
|
||
list of title + page range + badge + Review button, which shows nothing of the issue itself: no page, no
|
||
opening text, no figure, no sign of which articles already have audio. Requested directions, none decided:
|
||
- **Three columns.** Page thumbnail · headline + opening text · the article's figures. Reads down the issue
|
||
page by page in column one, with the text and the pictures beside it.
|
||
- **Synthesis state per row.** A "synthesised / not synthesised" marker, the MP3's duration as mm:ss, and an
|
||
inline play button — so the list doubles as the playlist instead of having to enter an article to find out.
|
||
- **Flipbook.** The PDF as a page-turning spread (issuu.com as the reference), with the extracted content of
|
||
the visible pages in two columns underneath it.
|
||
Decide the shape first — the three ideas are not variants of one layout, and the middle one (synthesis state)
|
||
is worth having under any of them. Whatever is chosen has to keep working at phone width and stay navigable
|
||
by keyboard and screen reader; a flipbook in particular needs a non-gestural alternative.
|
||
|
||
### P2 — Advanced magazine intelligence
|
||
|
||
- [~] Improve cross-page continuation detection and automatic sidebar/advertisement exclusion.
|
||
*(sidebar exclusion done via mark_sidebars, a narrow-column geometry heuristic; cross-page continuation still only
|
||
uses the forward "continued on page N" reference, not "continued from")*
|
||
- [x] Add image descriptions, chart/table handling, and optional spoken figure summaries.
|
||
*(per-figure "Describe" in review generates a type-aware description — chart/table/code/figure each get their
|
||
own prompt — via the configured vision engine with fallback; editable before approval; spoken only if "Read
|
||
figure descriptions" is enabled. Live-verified against a real chart: the configured primary vision engine
|
||
turned out to be a pure-OCR model that returns empty content for any non-transcription prompt regardless of
|
||
phrasing — confirmed with three different prompts — so description generation only works when a genuinely
|
||
caption-capable model is reachable as the fallback; documented as a config-dependent limitation in Manual.md,
|
||
not a code defect.)*
|
||
- [~] Add word-level karaoke timing and EPUB 3 Media Overlay export.
|
||
*(word-level highlighting done as a character-length estimate against the real measured audio duration, not true
|
||
forced alignment; no EPUB 3 export)*
|
||
- [~] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume.
|
||
*(driving mode done: two oversized controls, reuses whatever is already playing/queued; not mobile-specific)*
|
||
- [~] Add OCR-engine benchmark comparisons and per-page fallback selection.
|
||
*(per-page fallback done, now a real two-engine chain: vision OCR escalates when a page needs OCR or scores
|
||
poorly, KDL Frontier Parser nano is the vision fallback, tesseract is the last resort; no formal benchmark)*
|
||
- [ ] **Evaluate additional OCR/vision candidates against the two already wired (baidu/Unlimited-OCR,
|
||
KDL-Frontier-Parser-nano) and pick the best fit per task** — user-requested 2026-09-14. Goal: low RAM/VRAM
|
||
footprint with excellent speed and quality, for both page-text transcription and open-ended figure/chart
|
||
description (the latter is the harder requirement — confirmed live that baidu/Unlimited-OCR is
|
||
transcription-only and cannot do it at all, see `architecture/magazine-ingestion.md` and CHANGELOG 1.29.0).
|
||
Candidates to install as Docker containers alongside the existing engines, each added through the Engines tab
|
||
(`routes/docker.py`'s `_LOCAL_CONTAINER_DEFS`, role `ocr`) so they're selectable/connectable/benchmarkable like
|
||
any other engine, not hardcoded:
|
||
- [ ] [docling](https://github.com/docling-project/docling) — document conversion/layout toolkit.
|
||
- [ ] [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) — established general-purpose OCR.
|
||
- [ ] [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) — vision-language OCR model.
|
||
- [ ] Qwen3-VL-30B-A3B-Instruct-FP8 — general vision-language model; check whether it fits the RAM/VRAM
|
||
budget at 30B-A3B FP8 (likely the heaviest candidate — weigh against the "low footprint" goal
|
||
specifically) and whether it's meaningfully better at open-ended description than the nano-class
|
||
candidates above.
|
||
For each: measure page-transcription accuracy and figure/chart-description quality against the existing
|
||
c't KI Wissen 2026 test corpus, plus idle/active VRAM footprint and latency, before deciding what replaces or
|
||
joins the current primary/fallback pair.
|
||
|
||
### P2 — Finding magazines to read
|
||
|
||
- [ ] **Browse and search online PDF magazine sources from inside the app**, so an issue can be found and pulled in
|
||
without leaving for a browser and uploading by hand. Wanted: a catalogue of sources, a search across them, and
|
||
a one-click "ingest this issue".
|
||
**Blocked on a source decision, and that decision is the whole task.** The two sites named as examples
|
||
(`freemagazines.top`, `pdf-magazines.org`) redistribute current commercial titles — c't, BBC Wildlife,
|
||
AirForces Monthly — without the publishers' licence, so building a scraper against them is out: it is
|
||
copyright infringement regardless of who presses the button, and both would be trivially blocked or taken
|
||
down anyway. Sources that can be built against, in rough order of usefulness here:
|
||
- **OPDS catalogues** — the open standard for exactly this. Calibre-Web, Kavita and Komga all serve OPDS, so a
|
||
generic OPDS client also covers the user's own library on the NAS. Start here.
|
||
- **Internet Archive** (`archive.org` API) — a genuinely large public-domain and openly-licensed periodical
|
||
collection, with a documented search API and no scraping needed.
|
||
- **Publisher-run free issues and RSS/Atom feeds** — heise gives away sample c't issues, many trade titles
|
||
publish a free archive. Per-publisher work, but legitimate and high quality.
|
||
- **Open-access journals** (DOAJ, arXiv, PubMed Central) — if the appetite is reading rather than magazines
|
||
specifically, these are the largest legal corpus by far and all have real APIs.
|
||
Whatever is added, check licensing, rate limits and terms per source before wiring it (core rule 8), honour
|
||
`robots.txt`, and keep each source behind its own provider interface so one going away does not break the rest.
|
||
|
||
Three full regression runs of *Der Sturm* against the hand-corrected reference.
|
||
Everything below is measured, not estimated. Artefacts live in
|
||
`config/regression_snapshots/der_sturm/` (`*_manual.json` are the human
|
||
reference — never overwrite them).
|
||
|
||
| Run | Segments | Headline agreement | **Fully correct** | Wrong speaker | Unmatched |
|
||
|-----|---------:|-------------------:|------------------:|--------------:|----------:|
|
||
| 1 | 2143 | 72.6% | **54.2%** | 438 | 544 |
|
||
| 2 | 2273 | 77.7% | **52.8%** | 345 | 727 |
|
||
| 3 | 2227 | 78.7% | **52.2%** | 314 | 750 |
|
||
| human reference | 2110 | — | — | 0 | — |
|
||
|
||
**Read the headline with care.** Agreement is computed only over rows whose text
|
||
matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute
|
||
correct segments went 1160 → 1200 → 1161. The rising headline is largely a
|
||
shrinking denominator; true end-to-end accuracy has drifted slightly *down*.
|
||
|
||
### What genuinely improved
|
||
- Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did **not**
|
||
overcorrect (369 after cast collapsed to 19 after one recast pass).
|
||
- Wrong speaker names 438 → 345 → 314 — consistent, real.
|
||
- Verify pass now completes (~65 LLM calls) instead of never finishing.
|
||
|
||
### The blocker: segmentation, not attribution
|
||
750 of ~1060 problem segments are text that doesn't match the human's at all:
|
||
roughly 400 split finer than the human, ~210 merged where the human split, ~70
|
||
OCR/wording differences. Attribution work cannot fix these.
|
||
|
||
- `fused` dialogue segments hit **212** (human: 6, run 2: 59). These are
|
||
`_audiobookMergeAdjacentDialogue` joins using `\n\n`; 137 of them match no
|
||
human segment. Splitting them back was tested and is **worse** (52.2% → 43.4%),
|
||
so the merge is not the primary cause — but merging blocks the human kept
|
||
separate is contributing.
|
||
- The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it
|
||
went to 750. **That prediction did not hold** and needs re-deriving on run 3.
|
||
- Fixed: attribution normalization repairs the Schwester Larissa epigraph when the LLM splits it before a mixed
|
||
guillemet close; the regression is covered by `tests/test_conversation.py`.
|
||
(unclosed-guillemet handling).
|
||
|
||
### Next steps, highest value first
|
||
1. Attack segmentation directly — it is now the dominant error class. Run 3 has
|
||
no merge candidates, so inspect the earlier dialogue split/fusion stages and the
|
||
585 already-fused segments instead of changing `_audiobookMergeAdjacentDialogue`.
|
||
2. Preserve the dialogue-only merge for recast/verify outputs where adjacent pairs
|
||
can still occur; do not claim it explains the run-3 mismatch.
|
||
3. Only then re-run book 1. Re-running before segmentation moves will re-measure
|
||
rules already known not to shift the number that matters.
|
||
4. **Book 2 generalization run never started.** Reference is backed up at
|
||
`config/regression_snapshots/book2/`. Its point is to test whether these rules
|
||
generalise or were overfit to book 1 — still unanswered, and the confidence
|
||
scorer in particular was tuned solely on book 1.
|
||
|
||
### Character profiles
|
||
48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got
|
||
worse than run 2's 51. Generic role labels (`Fremder` 19 lines, `Vampir` 19,
|
||
`Inquisitor` 15, `Ork` 13, and the adverb `Unsicher`) still become cast members
|
||
with their own voices; the appositive resolver only fires where the book supplies
|
||
grammatical evidence (`Darrag der Schmied`), and pre-reveal aliases like
|
||
"der Fremde" deliberately withhold it.
|
||
|
||
### Harness notes (cost real time; do not relearn)
|
||
- Casting auto-starts `audiobookAutoRepairCast` 800 ms after finishing, which
|
||
re-sets `_audiobook.running`. Guards now explain the refusal instead of
|
||
returning silently, but an automated driver must still wait for `running` to
|
||
clear and treat a sub-60s phase as a bailout, not a success.
|
||
- Stall detection must key on the passage counter (`#ab-cv-count`). `segments`,
|
||
`done` and `unknown` do not move during a verify pass — watching them killed a
|
||
healthy run twice.
|
||
- Character Sheets only saves after its passage loop ends; poll `/api/characters`
|
||
until records appear before closing the browser, or a 4.4 h run is discarded.
|
||
|
||
## P1 — Character forms (implemented 2026-09-04)
|
||
|
||
Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der
|
||
Nacht", "der Wolfsmensch") need more than one appearance per character.
|
||
|
||
**Decided:** forms live INSIDE one character record, not as separate records — one
|
||
voice, one attribution target, so cast lines saying "Zerwas" always resolve. The
|
||
sheet gains a form switcher; each form carries its own `physical`, `image_prompt`
|
||
and portrait. **Detection is automatic**: the sheet prompt asks the LLM to identify
|
||
whether a character has distinct forms and describe each, so it generalises to any
|
||
book (werewolves, disguises, transformations) without per-book rules.
|
||
|
||
Implemented as bounded `sheet.forms[]` data inside the existing character record. The extraction prompt asks for
|
||
distinct forms, the detail page switches the active appearance, selected forms feed prompt/fingerprint generation,
|
||
and generated/uploaded portrait/reference assets remain attached to that form. One character ID and one voice are
|
||
preserved for attribution. A future polish pass can add richer inline form editing and per-form bulk-generation UX.
|
||
|
||
## P1 — Portrait / concept-sheet identity mismatch
|
||
|
||
The avatar and the concept sheet can show two different people (Lancorian: dark-haired
|
||
man vs white-haired mage). Cause is understood: `image_prompt` has two sources — the
|
||
LLM writes it as one of the 53 character-sheet fields, while
|
||
`_build_character_portrait_prompt` builds it deterministically from the same structured
|
||
facts the concept sheet uses. The deterministic one keeps both images consistent; the
|
||
LLM-written one omits identity anchors (age, hair, build) so the model invents them.
|
||
|
||
### Status — resolved 2026-09-19
|
||
|
||
Verified against the code and covered by tests (see `architecture/character-visual-identity.md`):
|
||
|
||
- [x] One canonical identity description feeds both the avatar and the concept sheet
|
||
(`_character_visual_facts`; portrait and reference prompts are built deterministically, the
|
||
active appearance form is overlaid first).
|
||
- [x] `image_prompt` is no longer authored by the LLM during sheet extraction (server and browser
|
||
prompts); the deterministic prompt is generated right after extraction, before avatar-only
|
||
generation and before per-form generation. Hand-edited prompts are never overwritten.
|
||
- [x] Prompt version, identity fingerprint and asset digests are stored with every generated pair
|
||
— for the base appearance too (they used to be lost on reload) and per form.
|
||
- [x] Review indicator: per-character banner and `GET /api/characters/visual-pair-review`, now
|
||
form-aware; a pair is also flagged when one image was regenerated alone.
|
||
- [x] Tests: server/browser fingerprint parity, prompt anchors shared by both builders, review
|
||
endpoint (base + forms), and Playwright persistence tests that fail on the previous code.
|
||
|
||
Still true, and deliberately not "fixed": the resolved prompt *text* is not stored per image
|
||
(only its version and the fingerprint of the facts it was built from).
|
||
|
||
### Known regression from the 2026-09-02 batch
|
||
Regenerating the top-10 profiles from evidence windows (passages around each character's
|
||
own dialogue) produced weaker `physical` text than the app's full-book pass — Lancorian's
|
||
became behaviour ("peers through small windows, moves nimbly on spiral staircases")
|
||
rather than appearance, where the older sheet had "shoulder-length white hair, faded scar
|
||
across forehead". Prefer the app's own Character Sheets flow over evidence-window
|
||
shortcuts when appearance quality matters.
|