# TTS Voice Creator Roadmap This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed. Priority levels: - **P0 — safety:** prevents crashes, data loss, broken releases, or security incidents. - **P1 — high impact:** improves memory use, responsiveness, and maintainability. - **P2 — quality:** improves diagnostics, polish, and long-term operations. ## Prioritized execution backlog Effort estimates are for one developer familiar with this codebase: **S** = a few hours, **M** = 1–2 days, **L** = several days, **XL** = multi-week or iterative. Within each priority, tasks are ordered from smallest to largest effort. Complete the first block before investing in architecture or polish. ## Current status — 2026-09-04 All P0 safety, security, data-integrity, and release-protection work is complete. P1 implementation is active: pagination, shared voice caching, bounded decoded-audio caching, background jobs, canonical character identity, forms, and the frontend namespace/manifest foundations and section-scoped lazy loading are working and regression-tested. Remaining P1 work is intentionally open where it needs a real architecture change: the larger Audiobook/Rehearser/style split. ### P0 — do first - [x] **S — Stop provider API keys from reaching frontend JavaScript or logs.** Provider keys are masked in settings responses, probes resolve credentials server-side, and browser storage/log paths are clean. The separate external caller key is available only through deliberate regeneration, uses `no-store`, and is not retained by the UI. - [x] **S — Validate uploaded filenames and paths.** Upload endpoints reject traversal, path separators and control characters before deriving extensions; ZIP imports reject absolute, traversal and malformed archive members. - [x] **M — Add upload, request-size, temporary-file and concurrent-job limits.** Endpoint-specific upload/request caps, bounded temporary work, cache eviction, and bounded TTS/STT/image job pools are implemented. - [x] **M — Add automatic backups before bulk deletes, voice moves and recasts.** Make recovery explicit and verify that backups can be restored. - [x] **M — Strictly validate custom URLs and prevent SSRF.** HTTP downloads and provider/container probes share scheme, hostname and private-destination validation with explicit local-network opt-in. - [x] **M — Add authentication and rate limits for destructive/API endpoints.** External API authentication is enabled by default and destructive/general mutation limits return bounded 429 responses. - [x] **M — Validate character sheets and voice metadata with schemas.** Character/form and voice metadata are bounded and type-checked at API boundaries. - [x] **L — Add database migrations and prompt/generation history.** Preserve old record shapes and expose protected, metadata-only generation history for auditing and follow-up recovery workflows. - [x] **L — Harden container and Docker access.** The app runs non-root with dropped capabilities, no-new-privileges and a read-only root filesystem; Compose Docker access is isolated behind a minimal container proxy with only container inspection/control enabled. ### P1 — next, highest user impact - [x] **S — Re-measure audiobook segmentation against `cast_run3.json`.** Run 3 has 2,227 segments, 1,043 dialogue segments and 585 newline-fused segments, but zero adjacent same-speaker dialogue pairs for `_audiobookMergeAdjacentDialogue` to merge. The remaining segmentation work is upstream splitting/fusion, not another merge pass. - [x] **S — Finish the remaining Playwright smoke checks.** Characters/Cast, profile sorting, column visibility, voice ratings, and starting/cancelling a generation job are covered. - [x] **M — Fix the Schwester Larissa unclosed-guillemet split.** Attribution normalization now repairs dialogue split before mixed German guillemet closes (`»…‹` / `«…›`), with an epigraph regression test. - [x] **M — Make profile portraits and character sheets use one canonical identity.** Shared structured prompts now version consistent pairs as v3 and fingerprint canonical appearance fields; add cross-record validation so mismatched profile/reference files are detected even when both are current-version. The UI now flags missing, stale, changed, or explicitly inconsistent pair metadata; visual similarity itself remains a human review concern, while generated/replaced assets are now detected using stored SHA-256 digests. - [x] **M — Add bounded duplicate-generation protection.** Consistent portrait/reference, voice-design, and casting jobs now deduplicate repeated requests with bounded, expiring `Idempotency-Key` contracts. - [x] **M — Improve large-table performance.** Character productions and the flat voice library paginate at 50 visible rows; low-memory mode also reduces rendered rows and image work. - [x] **M — Add explicit cache invalidation and eviction.** Fish.audio search responses, persisted synthesized audio, image-generation job mappings, and browser object URLs have bounded cleanup. - [x] **M — Add constrained-device validation.** Chrome CPU throttling, a 500-character workload, browser DOM/audio/heap/image-cache bounds, and a hard 4 GiB current-release container check are covered. The `1.24.21` image served successfully at 52.54 MiB idle usage, 0.10% CPU, 6 PIDs, and `OOMKilled=false`; cgroup memory was confirmed as 4,294,967,296 bytes and temporary/root filesystem usage was sampled. - [x] **L — Add background jobs and bounded queues.** Image generation already has the full interactive job path, and transcription, voice design, and casting now have optional ID/status/cancel/timeout paths while preserving synchronous callers. Job metadata now persists in SQLite and marks interrupted work after restart; durable binary result storage is bounded and authenticated; outbound TTS and LLM calls now share a bounded transient-failure retry policy, and `/api/jobs` exposes sanitized recent job history. Completed voice-design audio and consistent-image outputs now use bounded, atomic artifact files with authenticated download URLs; transcription text remains deliberately non-persistent. Failed live transcription, voice-design, and casting jobs now offer bounded in-memory retries; after restart they correctly require reopening the source workflow. - [x] **L — Run the Book 2 generalization regression.** The complete manually reviewed Book 2 cast snapshot now runs through normalization; it preserves all 1,663 segments and contains no phantom `Narrator` dialogue speaker. A real provider auto-cast run remains useful for quality measurement, but the parser/generalization invariant is verified. - [x] **L — Add character forms inside one character record.** `sheet.forms[]` stores bounded alternate appearances, prompts, and per-form visual assets; the detail-page switcher preserves one character ID and voice, and selected forms feed prompt generation and identity fingerprints. - [x] **L — Modularize the frontend incrementally.** The shared manifest, compatibility namespace bridge, section-scoped lazy loading, and responsibility-level Audiobook/Rehearser/style extraction are complete; a full ES-module migration remains a separate follow-up. - [x] **M — Add structured logs with request/job IDs.** API/MCP requests now receive sanitized correlation IDs and duration logs; route-specific logs retain provider/job context without request bodies. - [x] **L — Add linting and duplicate-function checks.** The manifest-driven frontend syntax/declaration gate now runs locally and in the SonarQube quality workflow alongside the existing Python and bundle checks. - [x] Expose extracted Rehearser and Audiobook parser APIs through the `window.TTVC` namespace bridge. ### P2 — after behavior and safety stabilize - [x] **M — Add structured logs with request/job IDs.** Report provider, endpoint, duration and failure reason without logging prompts, keys or book contents. - [x] **M — Separate provider health from configuration status.** TTS/STT discovery now returns independent `configured`, `available`, and bounded `health` details; backend help text shows actionable connectivity errors. - [x] **M — Add visible job history and retry controls.** Expose recent failures and safe retry state to users. - [x] **L — Add memory diagnostics.** Show browser heap where supported, rendered rows, cache size, active jobs, server/container memory and recent operation sizes. - [x] **L — Add linting and duplicate-function checks.** Run them in CI alongside the existing compile, bundle and SonarQube checks. ## P0 — Testing, data safety, and resource limits - [x] Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported). - [x] Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides. - [x] Limit upload sizes, temporary files, and concurrent jobs. Upload/request caps, bounded caches, and bounded TTS/STT/image-generation pools are implemented; durable background-job conversion remains a separate P1 item. - [x] Use atomic writes for `.meta.json` files. - [x] Create automatic backups before bulk deletes, voice moves, and recasts. - [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints. ## P1 — Modular frontend architecture - [x] Split `audiobook.js`, `rehearser.js`, and `style.css` by responsibility; Rehearser casting, playback, training, voice design, I/O, library, stage, and recording blocks plus Audiobook casting, preview, export, and feature styles are isolated. Core Reader state and compatibility event wiring remain ordered in the legacy entry points. - [x] Extract Rehearser stage display and cast-sidebar controls into `rehearser-stage.js`. - [ ] Replace broad globals with namespaces such as `window.TTVC.audiobook`; pure leaves, Audiobook/Reader/Rehearser feature modules, Studio, and character-library persistence now use private scopes with explicit compatibility exports. Remaining utility and legacy library UI modules still require the same responsibility-by-responsibility migration. - [x] Generate the feature-module list for `loader.js` and `scripts/minify.mjs` from one shared manifest; larger responsibility splits remain. - [x] Add a dependency-free frontend syntax and top-level declaration collision gate (`npm run check:frontend`); a full style/lint policy remains a follow-up. - [x] Extract Rehearser voice matching and voice design; matching/import helpers now live with the voice-picker module, while timing-sensitive DOM wiring remains in the cast renderer. - [x] Extract Rehearser voice-picker presentation and preview state into `rehearser-voice-picker.js`. - [x] Extract Rehearser cast view, sorting, filtering, and toolbar wiring into `rehearser-cast.js`. - [x] Extract Rehearser training transcript comparison helpers into `rehearser-train.js`. - [x] Extract Rehearser character development and backend-selection controls into `rehearser-voice-design.js`. - [x] Extract Audiobook quote segmentation and conservative turn-taking into `audiobook-segmentation.js`. - [x] Extract Rehearser library persistence and IndexedDB migration into `rehearser-library.js`. - [x] Extract Audiobook parsing and casting; parsing, segmentation, cast normalization/serialization, live cast view, LLM orchestration, preview, and export now live in focused modules. Core Reader state and compatibility wiring remain in `audiobook.js`. - [x] Extract Audiobook parsing primitives and speaker-heuristic constants into `audiobook-parse.js`. - [x] Extract Audiobook cast serialization and page-stamping helpers into `audiobook-cast.js`. - [x] Split the Performance/resource diagnostic styles into `static/css/resource-diagnostics.css`. - [x] Split background-job history and retry-panel styles into `static/css/background-jobs.css`. - [x] Split Voice Design form and sample-grid styles into `static/css/voice-design.css`. - [x] Split `style.css` by feature; Rehearser rules now live in `static/css/rehearser.css` alongside the existing diagnostics, background-job, voice-design, and Audiobook/Reader feature styles. - [x] Decision — do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests. ## P1 — Frontend performance and memory safety - [x] Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened. The shared manifest keeps the foundation in the core bundle and loads Reader/Audiobook, Rehearser, Library/Characters, and Studio groups once on first navigation; deep links load their group before activation. - [x] Paginate large character and voice tables; render approximately 25–50 visible rows, with a constrained-device regression covering a 250-character production. - [x] Cache `/api/voices` with a shared 30-second session cache, one in-flight request, explicit refresh, and invalidation after library metadata/folder mutations; the SQLite voice index remains authoritative. - [x] Lazy-load profile images and concept art. - [x] Update only affected table rows instead of rebuilding entire tables where the operation is row-local; benchmark, normalization, rating, and voice-note updates now patch the existing visible rows. Structural changes still trigger a full filtered rebuild so sorting, grouping, and pagination remain correct. - [x] Add an LRU cache for decoded Rehearser audio and remove old entries automatically. Persisted line-audio is already bounded; profile/concept and voice-library images now use a bounded viewport-aware browser cache that removes old decoded sources while retaining restoration URLs. - [x] Revoke discarded object URLs with `URL.revokeObjectURL(objectUrl)`; completed synthesized voice previews and character/Rehearser local auditions now release their Blob URLs when playback ends or is replaced. - [x] Add low-memory mode using `navigator.deviceMemory` where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less. - [x] Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM. Images use lazy decoding, and character/voice pages reduce from 50 to 25 rows when `navigator.deviceMemory <= 4`. ## P1 — Backend stability and bounded concurrency - [x] Introduce background jobs for voice design, transcription, casting, and image generation. Synchronous callers remain supported while each long-running path has an optional bounded background job. - [x] Give every job an ID, progress, cancellation, timeout, retry, and idempotency key. Failed live jobs offer bounded retries from stripped in-memory inputs; retry data is never returned or persisted, and restart correctly requires reopening the source workflow. - [x] Prevent duplicate generation when a user clicks twice for image, voice-design, and casting submissions. - [x] Centralize provider health checks and error handling. TTS, STT, and container URL probes now share the bounded `core.provider_health.probe_provider` mechanics and normalized connection/timeout/HTTP/JSON-shape errors; their provider-specific endpoint lists and authentication policies remain at the route layer. - [x] Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user. Consistent portrait/reference rendering, preview/style/seed-finder synthesis, and the OpenAI-compatible speech proxy are capped at two active jobs; STT transcription and benchmarks are capped at one active job. - [x] Use bounded server queues for the implemented synchronous generation paths; durable background-job queues remain a separate outstanding item above. ## P0 — Security - [x] Never expose provider API keys to frontend JavaScript or logs. Provider keys are masked on read and absent from browser storage; the external-client key is an intentional, explicit handoff for caller authentication and its regeneration response is non-cacheable. - [x] Validate uploaded filenames and paths. - [x] Strictly validate custom URLs to prevent SSRF, with private destinations available only when explicitly allowed for local provider/container operation. - [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints. - [x] Run containers as non-root with dropped capabilities and read-only filesystems where practical. - [x] Isolate Docker socket access behind a minimal privileged service. ## P0 — Data integrity and recovery - [x] Validate character sheets and voice metadata with schemas. - [x] Add database migrations instead of silently changing record shapes. - [x] Use atomic writes for `.meta.json` files. - [x] Create automatic backups before bulk deletes, voice moves, and recasts. - [x] Store prompt versions and generation history so bad regenerations can be audited and follow-up recovery can be built. ## P0 — Testing ### Automated tests - [x] Character parsing and phantom-speaker prevention, including malformed-response prose preservation. - [x] Voice-matching constraints reject invented voice IDs and avoid unnecessary duplicate assignments. - [x] Profile-completeness counting for empty, partial, and fully populated sheets. - [x] Rating persistence. - [x] STT endpoint normalization and backend-response shape handling. - [x] ComfyUI workflow submission, prompt injection, polling, and image retrieval. - [x] API timeout clamping and bounded upstream error text; live-provider failure drills remain environment-specific. ### Playwright smoke suite - [x] Load the app. - [x] Verify the production bundle loads. - [x] Verify language switching and Rehearser parsing. - [x] Verify clone source tabs switch correctly. - [x] Open Characters / Cast. - [x] Sort the profile column. - [x] Hide and show columns. - [x] Rate a voice. - [x] Start and cancel a generation job. ## P2 — Observability - [x] Add structured logs with request and job IDs. - [x] Report provider, endpoint, duration, and error reason clearly. - [x] Separate health status from “configured”; discovery APIs now preserve both states and expose sanitized probe errors. - [x] Add visible job history and safe retry state; the authenticated `/api/jobs` metadata endpoint now feeds a rendered panel with status, elapsed time, errors, cancellation, and an explicit reopen-source retry boundary. - [x] Avoid logging prompts, API keys, or book contents by default. ## P1 — Resource limits and constrained-device testing - [x] Add explicit Docker memory limits; the web container defaults to `mem_limit: 8g` and `mem_reservation: 2g`, configurable per host. - [x] Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits. - [x] Test with Chrome DevTools-equivalent CPU throttling; desktop Playwright uses Chromium CDP at 4× CPU and asserts navigation plus bounded DOM/audio counts, with JS heap captured when Chromium exposes it. - [x] Test the current image in a container limited to 4 GB RAM; the app served `/` with a 4 GiB cgroup limit, ~50.6 MiB idle memory, 0.09% CPU, and no OOM kill. - [x] Monitor browser-tab memory, JavaScript heap where supported, DOM-node count, audio elements, persisted jobs, server RSS, configured-data disk, and temporary disk usage through the read-only Performance diagnostics panel. - [x] Add a memory diagnostics page showing cache count/size, Docker/container memory, and largest recent operations. The authenticated panel reports browser cache counters, cgroup memory current/max, server RSS, configured/temp storage, persisted jobs, and sanitized operation durations; host-wide Docker metrics remain unavailable by design. ## P1 — Speaker attribution: measured state and next steps (paused 2026-09-02) Three full regression runs of *Der Sturm* against the hand-corrected reference. Everything below is measured, not estimated. Artefacts live in `config/regression_snapshots/der_sturm/` (`*_manual.json` are the human reference — never overwrite them). | Run | Segments | Headline agreement | **Fully correct** | Wrong speaker | Unmatched | |-----|---------:|-------------------:|------------------:|--------------:|----------:| | 1 | 2143 | 72.6% | **54.2%** | 438 | 544 | | 2 | 2273 | 77.7% | **52.8%** | 345 | 727 | | 3 | 2227 | 78.7% | **52.2%** | 314 | 750 | | human reference | 2110 | — | — | 0 | — | **Read the headline with care.** Agreement is computed only over rows whose text matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute correct segments went 1160 → 1200 → 1161. The rising headline is largely a shrinking denominator; true end-to-end accuracy has drifted slightly *down*. ### What genuinely improved - Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did **not** overcorrect (369 after cast collapsed to 19 after one recast pass). - Wrong speaker names 438 → 345 → 314 — consistent, real. - Verify pass now completes (~65 LLM calls) instead of never finishing. ### The blocker: segmentation, not attribution 750 of ~1060 problem segments are text that doesn't match the human's at all: roughly 400 split finer than the human, ~210 merged where the human split, ~70 OCR/wording differences. Attribution work cannot fix these. - `fused` dialogue segments hit **212** (human: 6, run 2: 59). These are `_audiobookMergeAdjacentDialogue` joins using `\n\n`; 137 of them match no human segment. Splitting them back was tested and is **worse** (52.2% → 43.4%), so the merge is not the primary cause — but merging blocks the human kept separate is contributing. - The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it went to 750. **That prediction did not hold** and needs re-deriving on run 3. - Fixed: attribution normalization repairs the Schwester Larissa epigraph when the LLM splits it before a mixed guillemet close; the regression is covered by `tests/test_conversation.py`. (unclosed-guillemet handling). ### Next steps, highest value first 1. Attack segmentation directly — it is now the dominant error class. Run 3 has no merge candidates, so inspect the earlier dialogue split/fusion stages and the 585 already-fused segments instead of changing `_audiobookMergeAdjacentDialogue`. 2. Preserve the dialogue-only merge for recast/verify outputs where adjacent pairs can still occur; do not claim it explains the run-3 mismatch. 3. Only then re-run book 1. Re-running before segmentation moves will re-measure rules already known not to shift the number that matters. 4. **Book 2 generalization run never started.** Reference is backed up at `config/regression_snapshots/book2/`. Its point is to test whether these rules generalise or were overfit to book 1 — still unanswered, and the confidence scorer in particular was tuned solely on book 1. ### Character profiles 48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got worse than run 2's 51. Generic role labels (`Fremder` 19 lines, `Vampir` 19, `Inquisitor` 15, `Ork` 13, and the adverb `Unsicher`) still become cast members with their own voices; the appositive resolver only fires where the book supplies grammatical evidence (`Darrag der Schmied`), and pre-reveal aliases like "der Fremde" deliberately withhold it. ### Harness notes (cost real time; do not relearn) - Casting auto-starts `audiobookAutoRepairCast` 800 ms after finishing, which re-sets `_audiobook.running`. Guards now explain the refusal instead of returning silently, but an automated driver must still wait for `running` to clear and treat a sub-60s phase as a bailout, not a success. - Stall detection must key on the passage counter (`#ab-cv-count`). `segments`, `done` and `unknown` do not move during a verify pass — watching them killed a healthy run twice. - Character Sheets only saves after its passage loop ends; poll `/api/characters` until records appear before closing the browser, or a 4.4 h run is discarded. ## P1 — Character forms (implemented 2026-09-04) Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der Nacht", "der Wolfsmensch") need more than one appearance per character. **Decided:** forms live INSIDE one character record, not as separate records — one voice, one attribution target, so cast lines saying "Zerwas" always resolve. The sheet gains a form switcher; each form carries its own `physical`, `image_prompt` and portrait. **Detection is automatic**: the sheet prompt asks the LLM to identify whether a character has distinct forms and describe each, so it generalises to any book (werewolves, disguises, transformations) without per-book rules. Implemented as bounded `sheet.forms[]` data inside the existing character record. The extraction prompt asks for distinct forms, the detail page switches the active appearance, selected forms feed prompt/fingerprint generation, and generated/uploaded portrait/reference assets remain attached to that form. One character ID and one voice are preserved for attribution. A future polish pass can add richer inline form editing and per-form bulk-generation UX. ## P1 — Portrait / concept-sheet identity mismatch The avatar and the concept sheet can show two different people (Lancorian: dark-haired man vs white-haired mage). Cause is understood: `image_prompt` has two sources — the LLM writes it as one of the 53 character-sheet fields, while `_build_character_portrait_prompt` builds it deterministically from the same structured facts the concept sheet uses. The deterministic one keeps both images consistent; the LLM-written one omits identity anchors (age, hair, build) so the model invents them. ### Status — partially working **Working:** - Character records, structured profile fields and portrait images are saved and loaded independently without losing the stored image data. - The concept-sheet path already builds from structured character facts and is the more consistent identity source. - Portrait generation now uses baseline appearance guidance, explicit head-and-shoulders framing, neutral backgrounds and book-period grounding. - Existing scene/style/background contamination in saved portrait prompts is filtered in the portrait path. **Not working reliably:** - The profile/avatar portrait and the character concept sheet can still depict different hair colour, age, facial features, build or overall identity. - Character-sheet extraction can write an LLM-authored `image_prompt` that lacks the identity anchors used by the deterministic concept-sheet prompt. - Regenerating a portrait does not guarantee that it uses exactly the same identity description as the concept sheet. - Evidence-window profile regeneration can produce behaviour or actions in `physical` instead of stable appearance details. **Needs to be fixed:** - Make one canonical identity description the input for both avatar portraits and concept sheets: age range, hair, eyes, face, build, skin tone, distinctive marks and clothing era. - Remove `image_prompt` from the general sheet-extraction output, or overwrite it only after extraction using the canonical structured fields. - Store the resolved identity prompt/version with each generated image so old images can be identified as stale and regenerated deterministically. - Add a consistency check or review indicator when the avatar and concept-sheet images were generated from different prompt versions. - Add an automated test covering extraction → prompt generation → avatar and concept sheet generation, asserting that both image requests contain the same identity anchors. The app's own flow already calls `/api/character-generate-prompts` after extraction, so **this is mainly a process trap**: any script that regenerates sheets must call that endpoint too, or it leaves an LLM-authored prompt in place. Consider removing `image_prompt` from the sheet-extraction field list entirely so there is only one source. ### Known regression from the 2026-09-02 batch Regenerating the top-10 profiles from evidence windows (passages around each character's own dialogue) produced weaker `physical` text than the app's full-book pass — Lancorian's became behaviour ("peers through small windows, moves nimbly on spiral staircases") rather than appearance, where the older sheet had "shoulder-length white hair, faded scar across forehead". Prefer the app's own Character Sheets flow over evidence-window shortcuts when appearance quality matters.