tts-voice-creator-clone-and-design-2/TODO.md

TTS Voice Creator Roadmap

This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed.

Priority levels:

• P0 — safety: prevents crashes, data loss, broken releases, or security incidents.
• P1 — high impact: improves memory use, responsiveness, and maintainability.
• P2 — quality: improves diagnostics, polish, and long-term operations.

P0 — Testing, data safety, and resource limits

• Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported).
• Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides.
• Limit upload sizes, temporary files, and concurrent jobs.
• Use atomic writes for .meta.json files.
• Create automatic backups before bulk deletes, voice moves, and recasts.
• Add request-size limits, rate limits, and authentication for destructive/API endpoints.

P1 — Modular frontend architecture

• Split audiobook.js, rehearser.js, and style.css by responsibility.
• Replace broad globals with namespaces such as window.TTVC.audiobook.
• Generate loader.js and scripts/minify.mjs from one module manifest.
• Add linting and duplicate-function checks.
• Extract Rehearser voice matching and voice design.
• Extract Audiobook parsing and casting.
• Split style.css by feature.
• Do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests.

P1 — Frontend performance and memory safety

• Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened.
• Virtualize or paginate large character and voice tables; render approximately 30–50 visible rows.
• Cache /api/voices with explicit invalidation instead of repeated full scans.
• Lazy-load profile images and concept art.
• Update only affected table rows instead of rebuilding entire tables.
• Add an LRU cache for audio and images and remove old entries automatically.
• Revoke discarded object URLs with URL.revokeObjectURL(objectUrl); completed synthesized voice previews now release their Blob URLs when playback ends.
• Add low-memory mode using navigator.deviceMemory where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less.
• Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM.

P1 — Backend stability and bounded concurrency

• Introduce background jobs for voice design, transcription, casting, and image generation.
• Give every job an ID, progress, cancellation, timeout, retry, and idempotency key.
• Prevent duplicate generation when a user clicks twice.
• Centralize provider health checks and error handling.
• Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user.
• Use bounded server queues.

P0 — Security

• Never expose API keys to frontend JavaScript or logs.
• Validate uploaded filenames and paths.
• Strictly validate custom URLs to prevent SSRF.
• Add request-size limits, rate limits, and authentication for destructive/API endpoints.
• Run containers as non-root with dropped capabilities and read-only filesystems where practical.
• Isolate Docker socket access behind a minimal privileged service.

P0 — Data integrity and recovery

• Validate character sheets and voice metadata with schemas.
• Add database migrations instead of silently changing record shapes.
• Use atomic writes for .meta.json files.
• Create automatic backups before bulk deletes, voice moves, and recasts.
• Store prompt versions and generation history so bad regenerations can be reverted.

P0 — Testing

Automated tests

• Character parsing and phantom-speaker prevention.
• Voice-matching constraints.
• Profile-completeness counting.
• Rating persistence.
• STT endpoint normalization.
• ComfyUI workflow submission.
• API failure and timeout behavior.

Playwright smoke suite

• Load the app.
• Verify the production bundle loads.
• Verify language switching and Rehearser parsing.
• Verify clone source tabs switch correctly.
• Open Characters / Cast.
• Sort the profile column.
• Hide and show columns.
• Rate a voice.
• Start and cancel a generation job.

P2 — Observability

• Add structured logs with request and job IDs.
• Report provider, endpoint, duration, and error reason clearly.
• Separate health status from “configured”.
• Add visible job history and retry controls.
• Avoid logging prompts, API keys, or book contents by default.

P1 — Resource limits and constrained-device testing

• Add explicit Docker memory limits; the web container defaults to mem_limit: 8g and mem_reservation: 2g, configurable per host.
• Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits.
• Test with Chrome DevTools CPU/memory throttling.
• Test in a VM or container limited to 4 GB RAM.
• Monitor browser-tab memory, JavaScript heap, DOM-node count, active audio blobs, Docker memory, and temporary disk usage.
• Add a memory diagnostics page showing browser heap where supported, rendered rows, cache count/size, active jobs, server/container memory, and largest recent operations.

P1 — Speaker attribution: measured state and next steps (paused 2026-09-02)

Three full regression runs of Der Sturm against the hand-corrected reference. Everything below is measured, not estimated. Artefacts live in config/regression_snapshots/der_sturm/ (*_manual.json are the human reference — never overwrite them).

Run	Segments	Headline agreement	Fully correct	Wrong speaker	Unmatched
1	2143	72.6%	54.2%	438	544
2	2273	77.7%	52.8%	345	727
3	2227	78.7%	52.2%	314	750
human reference	2110	—	—	0	—

Read the headline with care. Agreement is computed only over rows whose text matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute correct segments went 1160 → 1200 → 1161. The rising headline is largely a shrinking denominator; true end-to-end accuracy has drifted slightly down.

What genuinely improved

• Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did not overcorrect (369 after cast collapsed to 19 after one recast pass).
• Wrong speaker names 438 → 345 → 314 — consistent, real.
• Verify pass now completes (~65 LLM calls) instead of never finishing.

The blocker: segmentation, not attribution

750 of ~1060 problem segments are text that doesn't match the human's at all: roughly 400 split finer than the human, ~210 merged where the human split, ~70 OCR/wording differences. Attribution work cannot fix these.

• fused dialogue segments hit 212 (human: 6, run 2: 59). These are _audiobookMergeAdjacentDialogue joins using \n\n; 137 of them match no human segment. Splitting them back was tested and is worse (52.2% → 43.4%), so the merge is not the primary cause — but merging blocks the human kept separate is contributing.
• The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it went to 750. That prediction did not hold and needs re-deriving on run 3.
• Known unfixed: the Schwester Larissa epigraph still splits mid-sentence (unclosed-guillemet handling).

Next steps, highest value first

1. Attack segmentation directly — it is now the dominant error class. Start by re-measuring the dialogue-merge rule against cast_run3.json rather than run 2, since its original justification no longer holds.
2. Decide the merge's fate on that evidence: keep, tune the join, or revert (_audiobookMergeAdjacentDialogue, audiobook.js). Note it deliberately ignores emotion tags, so a character block carries one emotion — reverting restores per-chunk emotion at the cost of segmentation.
3. Only then re-run book 1. Re-running before segmentation moves will re-measure rules already known not to shift the number that matters.
4. Book 2 generalization run never started. Reference is backed up at config/regression_snapshots/book2/. Its point is to test whether these rules generalise or were overfit to book 1 — still unanswered, and the confidence scorer in particular was tuned solely on book 1.

Character profiles

48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got worse than run 2's 51. Generic role labels (Fremder 19 lines, Vampir 19, Inquisitor 15, Ork 13, and the adverb Unsicher) still become cast members with their own voices; the appositive resolver only fires where the book supplies grammatical evidence (Darrag der Schmied), and pre-reveal aliases like "der Fremde" deliberately withhold it.

Harness notes (cost real time; do not relearn)

• Casting auto-starts audiobookAutoRepairCast 800 ms after finishing, which re-sets _audiobook.running. Guards now explain the refusal instead of returning silently, but an automated driver must still wait for running to clear and treat a sub-60s phase as a bailout, not a success.
• Stall detection must key on the passage counter (#ab-cv-count). segments, done and unknown do not move during a verify pass — watching them killed a healthy run twice.
• Character Sheets only saves after its passage loop ends; poll /api/characters until records appear before closing the browser, or a 4.4 h run is discarded.

P1 — Character forms (decided 2026-09-02, not yet built)

Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der Nacht", "der Wolfsmensch") need more than one appearance per character.

Decided: forms live INSIDE one character record, not as separate records — one voice, one attribution target, so cast lines saying "Zerwas" always resolve. The sheet gains a form switcher; each form carries its own physical, image_prompt and portrait. Detection is automatic: the sheet prompt asks the LLM to identify whether a character has distinct forms and describe each, so it generalises to any book (werewolves, disguises, transformations) without per-book rules.

Not started. Data model, sheet UI and prompt changes all still to do.

P1 — Portrait / concept-sheet identity mismatch

The avatar and the concept sheet can show two different people (Lancorian: dark-haired man vs white-haired mage). Cause is understood: image_prompt has two sources — the LLM writes it as one of the 53 character-sheet fields, while _build_character_portrait_prompt builds it deterministically from the same structured facts the concept sheet uses. The deterministic one keeps both images consistent; the LLM-written one omits identity anchors (age, hair, build) so the model invents them.

The app's own flow already calls /api/character-generate-prompts after extraction, so this is mainly a process trap: any script that regenerates sheets must call that endpoint too, or it leaves an LLM-authored prompt in place. Consider removing image_prompt from the sheet-extraction field list entirely so there is only one source.

Known regression from the 2026-09-02 batch

Regenerating the top-10 profiles from evidence windows (passages around each character's own dialogue) produced weaker physical text than the app's full-book pass — Lancorian's became behaviour ("peers through small windows, moves nimbly on spiral staircases") rather than appearance, where the older sheet had "shoulder-length white hair, faded scar across forehead". Prefer the app's own Character Sheets flow over evidence-window shortcuts when appearance quality matters.