tts-voice-creator-clone-and-design-2/TODO.md

TTS Voice Creator Roadmap

This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed.

Priority levels:

• P0 — safety: prevents crashes, data loss, broken releases, or security incidents.
• P1 — high impact: improves memory use, responsiveness, and maintainability.
• P2 — quality: improves diagnostics, polish, and long-term operations.

Prioritized execution backlog

Effort estimates are for one developer familiar with this codebase: S = a few hours, M = 1–2 days, L = several days, XL = multi-week or iterative. Within each priority, tasks are ordered from smallest to largest effort. Complete the first block before investing in architecture or polish.

P0 — do first

• S — Stop API keys from reaching frontend JavaScript or logs. Provider keys are now masked in settings responses and probes use server-side credentials; finish the external-client key handoff and audit remaining transient input/error paths.
• S — Validate uploaded filenames and paths. Reject traversal, unsafe names and writes outside the intended data directories.
• M — Add upload, request-size, temporary-file and concurrent-job limits. Apply endpoint-specific caps and clean up intermediates on failure.
• M — Add automatic backups before bulk deletes, voice moves and recasts. Make recovery explicit and verify that backups can be restored.
• M — Strictly validate custom URLs and prevent SSRF. Allow only intended schemes, hosts and network destinations for provider/container probes.
• M — Add authentication and rate limits for destructive/API endpoints. Protect settings, file operations, Docker controls and bulk mutations.
• M — Validate character sheets and voice metadata with schemas. Reject or repair malformed records at API boundaries.
• L — Add database migrations and prompt/generation history. Preserve old record shapes and allow bad regenerations to be reverted.
• L — Harden container and Docker access. The app now runs non-root with dropped capabilities, no-new-privileges and a read-only root filesystem; isolate the remaining Docker socket access behind a minimal privileged service.

P1 — next, highest user impact

• S — Re-measure audiobook segmentation against cast_run3.json. Run 3 has 2,227 segments, 1,043 dialogue segments and 585 newline-fused segments, but zero adjacent same-speaker dialogue pairs for _audiobookMergeAdjacentDialogue to merge. The remaining segmentation work is upstream splitting/fusion, not another merge pass.
• S — Finish the remaining Playwright smoke checks. Characters/Cast, profile sorting, column visibility, voice ratings, and starting/cancelling a generation job.
• M — Fix the Schwester Larissa unclosed-guillemet split. Add a regression test for the epigraph and verify the parser does not split mid-sentence.
• M — Make profile portraits and character sheets use one canonical identity. Share age, hair, eyes, face, build, skin tone, marks and era; version the prompt and detect stale or inconsistent images.
• M — Add bounded duplicate-generation protection. Consistent portrait/reference jobs now deduplicate repeated requests with Idempotency-Key; extend the same bounded contract to voice and casting jobs.
• M — Improve large-table performance. Virtualize/paginate character and voice tables, update only affected rows, and disable preloading on low-memory devices.
• M — Add explicit cache invalidation and eviction. Fish.audio search responses now have bounded eviction; add equivalent LRU cleanup for audio/image entries and object URLs.
• M — Add constrained-device validation. Test with Chrome CPU/memory throttling, a 4 GB VM/container, and measure browser, Docker and temporary-disk usage.
• L — Add background jobs and bounded queues. Give voice design, transcription, casting and image generation IDs, progress, cancellation, timeout, retry and idempotency support.
• L — Run the Book 2 generalization regression. Confirm speaker-attribution and confidence rules are not overfit to Der Sturm.
• L — Add character forms inside one character record. Support multiple appearances while preserving one voice and one attribution target.
• L — Modularize the frontend. Split Audiobook, Rehearser and styles, replace broad globals with namespaces, and consolidate the loader/module manifest.

P2 — after behavior and safety stabilize

• M — Add structured logs with request/job IDs. Report provider, endpoint, duration and failure reason without logging prompts, keys or book contents.
• M — Separate provider health from configuration status. Show actionable connectivity and authentication errors.
• M — Add visible job history and retry controls. Expose recent failures and safe retry state to users.
• L — Add memory diagnostics. Show browser heap where supported, rendered rows, cache size, active jobs, server/container memory and recent operation sizes.
• L — Add linting and duplicate-function checks. Run them in CI alongside the existing compile, bundle and SonarQube checks.

P0 — Testing, data safety, and resource limits

• Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported).
• Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides.
• Limit upload sizes, temporary files, and concurrent jobs.
• Use atomic writes for .meta.json files.
• Create automatic backups before bulk deletes, voice moves, and recasts.
• Add request-size limits, rate limits, and authentication for destructive/API endpoints.

P1 — Modular frontend architecture

• Split audiobook.js, rehearser.js, and style.css by responsibility.
• Replace broad globals with namespaces such as window.TTVC.audiobook.
• Generate loader.js and scripts/minify.mjs from one module manifest.
• Add linting and duplicate-function checks.
• Extract Rehearser voice matching and voice design.
• Extract Audiobook parsing and casting.
• Split style.css by feature.
• Do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests.

P1 — Frontend performance and memory safety

• Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened.
• Virtualize or paginate large character and voice tables; render approximately 30–50 visible rows.
• Cache /api/voices with explicit invalidation instead of repeated full scans.
• Lazy-load profile images and concept art.
• Update only affected table rows instead of rebuilding entire tables.
• Add an LRU cache for audio and images and remove old entries automatically.
• Revoke discarded object URLs with URL.revokeObjectURL(objectUrl); completed synthesized voice previews now release their Blob URLs when playback ends.
• Add low-memory mode using navigator.deviceMemory where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less.
• Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM.

P1 — Backend stability and bounded concurrency

• Introduce background jobs for voice design, transcription, casting, and image generation.
• Give every job an ID, progress, cancellation, timeout, retry, and idempotency key.
• Prevent duplicate generation when a user clicks twice.
• Centralize provider health checks and error handling.
• Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user. Consistent portrait/reference rendering is now capped at two active jobs; extend the cap to voice and STT paths.
• Use bounded server queues.

P0 — Security

• Never expose API keys to frontend JavaScript or logs. Provider keys are masked on read and absent from browser storage; the deliberate external-client key display and transient entry path still need a final design review.
• Validate uploaded filenames and paths.
• Strictly validate custom URLs to prevent SSRF.
• Add request-size limits, rate limits, and authentication for destructive/API endpoints.
• Run containers as non-root with dropped capabilities and read-only filesystems where practical.
• Isolate Docker socket access behind a minimal privileged service.

P0 — Data integrity and recovery

• Validate character sheets and voice metadata with schemas.
• Add database migrations instead of silently changing record shapes.
• Use atomic writes for .meta.json files.
• Create automatic backups before bulk deletes, voice moves, and recasts.
• Store prompt versions and generation history so bad regenerations can be reverted.

P0 — Testing

Automated tests

• Character parsing and phantom-speaker prevention.
• Voice-matching constraints.
• Profile-completeness counting.
• Rating persistence.
• STT endpoint normalization.
• ComfyUI workflow submission.
• API failure and timeout behavior.

Playwright smoke suite

• Load the app.
• Verify the production bundle loads.
• Verify language switching and Rehearser parsing.
• Verify clone source tabs switch correctly.
• Open Characters / Cast.
• Sort the profile column.
• Hide and show columns.
• Rate a voice.
• Start and cancel a generation job.

P2 — Observability

• Add structured logs with request and job IDs.
• Report provider, endpoint, duration, and error reason clearly.
• Separate health status from “configured”.
• Add visible job history and retry controls.
• Avoid logging prompts, API keys, or book contents by default.

P1 — Resource limits and constrained-device testing

• Add explicit Docker memory limits; the web container defaults to mem_limit: 8g and mem_reservation: 2g, configurable per host.
• Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits.
• Test with Chrome DevTools CPU/memory throttling.
• Test in a VM or container limited to 4 GB RAM.
• Monitor browser-tab memory, JavaScript heap, DOM-node count, active audio blobs, Docker memory, and temporary disk usage.
• Add a memory diagnostics page showing browser heap where supported, rendered rows, cache count/size, active jobs, server/container memory, and largest recent operations.

P1 — Speaker attribution: measured state and next steps (paused 2026-09-02)

Three full regression runs of Der Sturm against the hand-corrected reference. Everything below is measured, not estimated. Artefacts live in config/regression_snapshots/der_sturm/ (*_manual.json are the human reference — never overwrite them).

Run	Segments	Headline agreement	Fully correct	Wrong speaker	Unmatched
1	2143	72.6%	54.2%	438	544
2	2273	77.7%	52.8%	345	727
3	2227	78.7%	52.2%	314	750
human reference	2110	—	—	0	—

Read the headline with care. Agreement is computed only over rows whose text matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute correct segments went 1160 → 1200 → 1161. The rising headline is largely a shrinking denominator; true end-to-end accuracy has drifted slightly down.

What genuinely improved

• Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did not overcorrect (369 after cast collapsed to 19 after one recast pass).
• Wrong speaker names 438 → 345 → 314 — consistent, real.
• Verify pass now completes (~65 LLM calls) instead of never finishing.

The blocker: segmentation, not attribution

750 of ~1060 problem segments are text that doesn't match the human's at all: roughly 400 split finer than the human, ~210 merged where the human split, ~70 OCR/wording differences. Attribution work cannot fix these.

• fused dialogue segments hit 212 (human: 6, run 2: 59). These are _audiobookMergeAdjacentDialogue joins using \n\n; 137 of them match no human segment. Splitting them back was tested and is worse (52.2% → 43.4%), so the merge is not the primary cause — but merging blocks the human kept separate is contributing.
• The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it went to 750. That prediction did not hold and needs re-deriving on run 3.
• Known unfixed: the Schwester Larissa epigraph still splits mid-sentence (unclosed-guillemet handling).

Next steps, highest value first

1. Attack segmentation directly — it is now the dominant error class. Run 3 has no merge candidates, so inspect the earlier dialogue split/fusion stages and the 585 already-fused segments instead of changing _audiobookMergeAdjacentDialogue.
2. Preserve the dialogue-only merge for recast/verify outputs where adjacent pairs can still occur; do not claim it explains the run-3 mismatch.
3. Only then re-run book 1. Re-running before segmentation moves will re-measure rules already known not to shift the number that matters.
4. Book 2 generalization run never started. Reference is backed up at config/regression_snapshots/book2/. Its point is to test whether these rules generalise or were overfit to book 1 — still unanswered, and the confidence scorer in particular was tuned solely on book 1.

Character profiles

48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got worse than run 2's 51. Generic role labels (Fremder 19 lines, Vampir 19, Inquisitor 15, Ork 13, and the adverb Unsicher) still become cast members with their own voices; the appositive resolver only fires where the book supplies grammatical evidence (Darrag der Schmied), and pre-reveal aliases like "der Fremde" deliberately withhold it.

Harness notes (cost real time; do not relearn)

• Casting auto-starts audiobookAutoRepairCast 800 ms after finishing, which re-sets _audiobook.running. Guards now explain the refusal instead of returning silently, but an automated driver must still wait for running to clear and treat a sub-60s phase as a bailout, not a success.
• Stall detection must key on the passage counter (#ab-cv-count). segments, done and unknown do not move during a verify pass — watching them killed a healthy run twice.
• Character Sheets only saves after its passage loop ends; poll /api/characters until records appear before closing the browser, or a 4.4 h run is discarded.

P1 — Character forms (decided 2026-09-02, not yet built)

Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der Nacht", "der Wolfsmensch") need more than one appearance per character.

Decided: forms live INSIDE one character record, not as separate records — one voice, one attribution target, so cast lines saying "Zerwas" always resolve. The sheet gains a form switcher; each form carries its own physical, image_prompt and portrait. Detection is automatic: the sheet prompt asks the LLM to identify whether a character has distinct forms and describe each, so it generalises to any book (werewolves, disguises, transformations) without per-book rules.

Not started. Data model, sheet UI and prompt changes all still to do.

P1 — Portrait / concept-sheet identity mismatch

The avatar and the concept sheet can show two different people (Lancorian: dark-haired man vs white-haired mage). Cause is understood: image_prompt has two sources — the LLM writes it as one of the 53 character-sheet fields, while _build_character_portrait_prompt builds it deterministically from the same structured facts the concept sheet uses. The deterministic one keeps both images consistent; the LLM-written one omits identity anchors (age, hair, build) so the model invents them.

Status — partially working

Working:

• Character records, structured profile fields and portrait images are saved and loaded independently without losing the stored image data.
• The concept-sheet path already builds from structured character facts and is the more consistent identity source.
• Portrait generation now uses baseline appearance guidance, explicit head-and-shoulders framing, neutral backgrounds and book-period grounding.
• Existing scene/style/background contamination in saved portrait prompts is filtered in the portrait path.

Not working reliably:

• The profile/avatar portrait and the character concept sheet can still depict different hair colour, age, facial features, build or overall identity.
• Character-sheet extraction can write an LLM-authored image_prompt that lacks the identity anchors used by the deterministic concept-sheet prompt.
• Regenerating a portrait does not guarantee that it uses exactly the same identity description as the concept sheet.
• Evidence-window profile regeneration can produce behaviour or actions in physical instead of stable appearance details.

Needs to be fixed:

• Make one canonical identity description the input for both avatar portraits and concept sheets: age range, hair, eyes, face, build, skin tone, distinctive marks and clothing era.
• Remove image_prompt from the general sheet-extraction output, or overwrite it only after extraction using the canonical structured fields.
• Store the resolved identity prompt/version with each generated image so old images can be identified as stale and regenerated deterministically.
• Add a consistency check or review indicator when the avatar and concept-sheet images were generated from different prompt versions.
• Add an automated test covering extraction → prompt generation → avatar and concept sheet generation, asserting that both image requests contain the same identity anchors.

The app's own flow already calls /api/character-generate-prompts after extraction, so this is mainly a process trap: any script that regenerates sheets must call that endpoint too, or it leaves an LLM-authored prompt in place. Consider removing image_prompt from the sheet-extraction field list entirely so there is only one source.

Known regression from the 2026-09-02 batch

Regenerating the top-10 profiles from evidence windows (passages around each character's own dialogue) produced weaker physical text than the app's full-book pass — Lancorian's became behaviour ("peers through small windows, moves nimbly on spiral staircases") rather than appearance, where the older sheet had "shoulder-length white hair, faded scar across forehead". Prefer the app's own Character Sheets flow over evidence-window shortcuts when appearance quality matters.