tts-voice-creator-clone-and-design-2/TODO.md

TTS Voice Creator Roadmap

This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed.

Priority levels:

• P0 — safety: prevents crashes, data loss, broken releases, or security incidents.
• P1 — high impact: improves memory use, responsiveness, and maintainability.
• P2 — quality: improves diagnostics, polish, and long-term operations.

Prioritized execution backlog

Effort estimates are for one developer familiar with this codebase: S = a few hours, M = 1–2 days, L = several days, XL = multi-week or iterative. Within each priority, tasks are ordered from smallest to largest effort. Complete the first block before investing in architecture or polish.

P0 — do first

• S — Stop provider API keys from reaching frontend JavaScript or logs. Provider keys are masked in settings responses, probes resolve credentials server-side, and browser storage/log paths are clean. The separate external caller key is available only through deliberate regeneration, uses no-store, and is not retained by the UI.
• S — Validate uploaded filenames and paths. Upload endpoints reject traversal, path separators and control characters before deriving extensions; ZIP imports reject absolute, traversal and malformed archive members.
• M — Add upload, request-size, temporary-file and concurrent-job limits. Endpoint-specific upload/request caps, bounded temporary work, cache eviction, and bounded TTS/STT/image job pools are implemented.
• M — Add automatic backups before bulk deletes, voice moves and recasts. Make recovery explicit and verify that backups can be restored.
• M — Strictly validate custom URLs and prevent SSRF. HTTP downloads and provider/container probes share scheme, hostname and private-destination validation with explicit local-network opt-in.
• M — Add authentication and rate limits for destructive/API endpoints. External API authentication is enabled by default and destructive/general mutation limits return bounded 429 responses.
• M — Validate character sheets and voice metadata with schemas. Character/form and voice metadata are bounded and type-checked at API boundaries.
• L — Add database migrations and prompt/generation history. Preserve old record shapes and expose protected, metadata-only generation history for auditing and follow-up recovery workflows.
• L — Harden container and Docker access. The app runs non-root with dropped capabilities, no-new-privileges and a read-only root filesystem; Compose Docker access is isolated behind a minimal container proxy with only container inspection/control enabled.

P1 — next, highest user impact

• S — Re-measure audiobook segmentation against cast_run3.json. Run 3 has 2,227 segments, 1,043 dialogue segments and 585 newline-fused segments, but zero adjacent same-speaker dialogue pairs for _audiobookMergeAdjacentDialogue to merge. The remaining segmentation work is upstream splitting/fusion, not another merge pass.
• S — Finish the remaining Playwright smoke checks. Characters/Cast, profile sorting, column visibility, voice ratings, and starting/cancelling a generation job are covered.
• M — Fix the Schwester Larissa unclosed-guillemet split. Attribution normalization now repairs dialogue split before mixed German guillemet closes (»…‹ / «…›), with an epigraph regression test.
• M — Make profile portraits and character sheets use one canonical identity. Shared structured prompts now version consistent pairs as v3 and fingerprint canonical appearance fields; add cross-record validation so mismatched profile/reference files are detected even when both are current-version. The UI now flags missing, stale, changed, or explicitly inconsistent pair metadata; visual similarity itself still needs an image-level review workflow.
• M — Add bounded duplicate-generation protection. Consistent portrait/reference, voice-design, and casting jobs now deduplicate repeated requests with bounded, expiring Idempotency-Key contracts.
• M — Improve large-table performance. Character productions and the flat voice library paginate at 50 visible rows; low-memory mode also reduces rendered rows and image work.
• M — Add explicit cache invalidation and eviction. Fish.audio search responses, persisted synthesized audio, image-generation job mappings, and browser object URLs have bounded cleanup.
• M — Add constrained-device validation. Test with Chrome CPU/memory throttling, a 4 GB VM/container, and measure browser, Docker and temporary-disk usage. Automated browser emulation proves the 4 GB shared-pool cap, and the current 1.23.3 image served successfully under a hard 4 GiB cgroup limit without OOM; browser heap/DOM/audio and workload resource measurements remain.
• L — Add background jobs and bounded queues. Image generation already has the full interactive job path, and transcription, voice design, and casting now have optional ID/status/cancel/timeout paths while preserving synchronous callers. Job metadata now persists in SQLite and marks interrupted work after restart; durable binary result storage remains; outbound TTS and LLM calls now share a bounded transient-failure retry policy, and /api/jobs exposes sanitized recent job history.
• L — Run the Book 2 generalization regression. Added a neutral multi-speaker fixture and parser regression; a real Book 2 auto-cast/attribution output still needs to be captured and reviewed to confirm rules are not overfit to Der Sturm.
• L — Add character forms inside one character record. sheet.forms[] stores bounded alternate appearances, prompts, and per-form visual assets; the detail-page switcher preserves one character ID and voice, and selected forms feed prompt generation and identity fingerprints.
• L — Modularize the frontend. Split Audiobook, Rehearser and styles, replace broad globals with namespaces, and consolidate the loader/module manifest.

P2 — after behavior and safety stabilize

• M — Add structured logs with request/job IDs. Report provider, endpoint, duration and failure reason without logging prompts, keys or book contents.
• M — Separate provider health from configuration status. Show actionable connectivity and authentication errors.
• M — Add visible job history and retry controls. Expose recent failures and safe retry state to users.
• L — Add memory diagnostics. Show browser heap where supported, rendered rows, cache size, active jobs, server/container memory and recent operation sizes.
• L — Add linting and duplicate-function checks. Run them in CI alongside the existing compile, bundle and SonarQube checks.

P0 — Testing, data safety, and resource limits

• Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported).
• Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides.
• Limit upload sizes, temporary files, and concurrent jobs. Upload/request caps, bounded caches, and bounded TTS/STT/image-generation pools are implemented; durable background-job conversion remains a separate P1 item.
• Use atomic writes for .meta.json files.
• Create automatic backups before bulk deletes, voice moves, and recasts.
• Add request-size limits, rate limits, and authentication for destructive/API endpoints.

P1 — Modular frontend architecture

• Split audiobook.js, rehearser.js, and style.css by responsibility.
• Replace broad globals with namespaces such as window.TTVC.audiobook.
• Generate the feature-module list for loader.js and scripts/minify.mjs from one shared manifest; larger responsibility splits remain.
• Add a dependency-free frontend syntax and top-level declaration collision gate (npm run check:frontend); a full style/lint policy remains a follow-up.
• Extract Rehearser voice matching and voice design.
• Extract Audiobook parsing and casting.
• Split style.css by feature.
• Do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests.

P1 — Frontend performance and memory safety

• Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened.
• Paginate large character and voice tables; render approximately 25–50 visible rows, with a constrained-device regression covering a 250-character production.
• Cache /api/voices with explicit invalidation instead of repeated full scans.
• Lazy-load profile images and concept art.
• Update only affected table rows instead of rebuilding entire tables.
• Add an LRU cache for audio and images and remove old entries automatically. Persisted line-audio is now bounded; image-cache eviction remains.
• Revoke discarded object URLs with URL.revokeObjectURL(objectUrl); completed synthesized voice previews and character/Rehearser local auditions now release their Blob URLs when playback ends or is replaced.
• Add low-memory mode using navigator.deviceMemory where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less.
• Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM. Images use lazy decoding, and character/voice pages reduce from 50 to 25 rows when navigator.deviceMemory <= 4.

P1 — Backend stability and bounded concurrency

• Introduce background jobs for voice design, transcription, casting, and image generation. Image generation is complete and transcription has an optional background endpoint; voice design and casting remain synchronous.
• Give every job an ID, progress, cancellation, timeout, retry, and idempotency key. Consistent image jobs now provide all except retry; voice design and casting remain synchronous request paths.
• Prevent duplicate generation when a user clicks twice for image, voice-design, and casting submissions.
• Centralize provider health checks and error handling.
• Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user. Consistent portrait/reference rendering, preview/style/seed-finder synthesis, and the OpenAI-compatible speech proxy are capped at two active jobs; STT transcription and benchmarks are capped at one active job.
• Use bounded server queues for the implemented synchronous generation paths; durable background-job queues remain a separate outstanding item above.

P0 — Security

• Never expose provider API keys to frontend JavaScript or logs. Provider keys are masked on read and absent from browser storage; the external-client key is an intentional, explicit handoff for caller authentication and its regeneration response is non-cacheable.
• Validate uploaded filenames and paths.
• Strictly validate custom URLs to prevent SSRF, with private destinations available only when explicitly allowed for local provider/container operation.
• Add request-size limits, rate limits, and authentication for destructive/API endpoints.
• Run containers as non-root with dropped capabilities and read-only filesystems where practical.
• Isolate Docker socket access behind a minimal privileged service.

P0 — Data integrity and recovery

• Validate character sheets and voice metadata with schemas.
• Add database migrations instead of silently changing record shapes.
• Use atomic writes for .meta.json files.
• Create automatic backups before bulk deletes, voice moves, and recasts.
• Store prompt versions and generation history so bad regenerations can be audited and follow-up recovery can be built.

P0 — Testing

Automated tests

• Character parsing and phantom-speaker prevention, including malformed-response prose preservation.
• Voice-matching constraints reject invented voice IDs and avoid unnecessary duplicate assignments.
• Profile-completeness counting for empty, partial, and fully populated sheets.
• Rating persistence.
• STT endpoint normalization and backend-response shape handling.
• ComfyUI workflow submission, prompt injection, polling, and image retrieval.
• API timeout clamping and bounded upstream error text; live-provider failure drills remain environment-specific.

Playwright smoke suite

• Load the app.
• Verify the production bundle loads.
• Verify language switching and Rehearser parsing.
• Verify clone source tabs switch correctly.
• Open Characters / Cast.
• Sort the profile column.
• Hide and show columns.
• Rate a voice.
• Start and cancel a generation job.

P2 — Observability

• Add structured logs with request and job IDs.
• Report provider, endpoint, duration, and error reason clearly.
• Separate health status from “configured”.
• Add visible job history and retry controls; the authenticated /api/jobs metadata endpoint is available for the UI, but a rendered history panel and safe retry action remain.
• Avoid logging prompts, API keys, or book contents by default.

P1 — Resource limits and constrained-device testing

• Add explicit Docker memory limits; the web container defaults to mem_limit: 8g and mem_reservation: 2g, configurable per host.
• Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits.
• Test with Chrome DevTools CPU/memory throttling.
• Test the current image in a container limited to 4 GB RAM; the app served / with a 4 GiB cgroup limit, ~50.6 MiB idle memory, 0.09% CPU, and no OOM kill.
• Monitor browser-tab memory, JavaScript heap, DOM-node count, active audio blobs, Docker memory, and temporary disk usage.
• Add a memory diagnostics page showing browser heap where supported, rendered rows, cache count/size, active jobs, server/container memory, and largest recent operations.

P1 — Speaker attribution: measured state and next steps (paused 2026-09-02)

Three full regression runs of Der Sturm against the hand-corrected reference. Everything below is measured, not estimated. Artefacts live in config/regression_snapshots/der_sturm/ (*_manual.json are the human reference — never overwrite them).

Run	Segments	Headline agreement	Fully correct	Wrong speaker	Unmatched
1	2143	72.6%	54.2%	438	544
2	2273	77.7%	52.8%	345	727
3	2227	78.7%	52.2%	314	750
human reference	2110	—	—	0	—

Read the headline with care. Agreement is computed only over rows whose text matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute correct segments went 1160 → 1200 → 1161. The rising headline is largely a shrinking denominator; true end-to-end accuracy has drifted slightly down.

What genuinely improved

• Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did not overcorrect (369 after cast collapsed to 19 after one recast pass).
• Wrong speaker names 438 → 345 → 314 — consistent, real.
• Verify pass now completes (~65 LLM calls) instead of never finishing.

The blocker: segmentation, not attribution

750 of ~1060 problem segments are text that doesn't match the human's at all: roughly 400 split finer than the human, ~210 merged where the human split, ~70 OCR/wording differences. Attribution work cannot fix these.

• fused dialogue segments hit 212 (human: 6, run 2: 59). These are _audiobookMergeAdjacentDialogue joins using \n\n; 137 of them match no human segment. Splitting them back was tested and is worse (52.2% → 43.4%), so the merge is not the primary cause — but merging blocks the human kept separate is contributing.
• The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it went to 750. That prediction did not hold and needs re-deriving on run 3.
• Fixed: attribution normalization repairs the Schwester Larissa epigraph when the LLM splits it before a mixed guillemet close; the regression is covered by tests/test_conversation.py. (unclosed-guillemet handling).

Next steps, highest value first

1. Attack segmentation directly — it is now the dominant error class. Run 3 has no merge candidates, so inspect the earlier dialogue split/fusion stages and the 585 already-fused segments instead of changing _audiobookMergeAdjacentDialogue.
2. Preserve the dialogue-only merge for recast/verify outputs where adjacent pairs can still occur; do not claim it explains the run-3 mismatch.
3. Only then re-run book 1. Re-running before segmentation moves will re-measure rules already known not to shift the number that matters.
4. Book 2 generalization run never started. Reference is backed up at config/regression_snapshots/book2/. Its point is to test whether these rules generalise or were overfit to book 1 — still unanswered, and the confidence scorer in particular was tuned solely on book 1.

Character profiles

48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got worse than run 2's 51. Generic role labels (Fremder 19 lines, Vampir 19, Inquisitor 15, Ork 13, and the adverb Unsicher) still become cast members with their own voices; the appositive resolver only fires where the book supplies grammatical evidence (Darrag der Schmied), and pre-reveal aliases like "der Fremde" deliberately withhold it.

Harness notes (cost real time; do not relearn)

• Casting auto-starts audiobookAutoRepairCast 800 ms after finishing, which re-sets _audiobook.running. Guards now explain the refusal instead of returning silently, but an automated driver must still wait for running to clear and treat a sub-60s phase as a bailout, not a success.
• Stall detection must key on the passage counter (#ab-cv-count). segments, done and unknown do not move during a verify pass — watching them killed a healthy run twice.
• Character Sheets only saves after its passage loop ends; poll /api/characters until records appear before closing the browser, or a 4.4 h run is discarded.

P1 — Character forms (implemented 2026-09-04)

Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der Nacht", "der Wolfsmensch") need more than one appearance per character.

Decided: forms live INSIDE one character record, not as separate records — one voice, one attribution target, so cast lines saying "Zerwas" always resolve. The sheet gains a form switcher; each form carries its own physical, image_prompt and portrait. Detection is automatic: the sheet prompt asks the LLM to identify whether a character has distinct forms and describe each, so it generalises to any book (werewolves, disguises, transformations) without per-book rules.

Implemented as bounded sheet.forms[] data inside the existing character record. The extraction prompt asks for distinct forms, the detail page switches the active appearance, selected forms feed prompt/fingerprint generation, and generated/uploaded portrait/reference assets remain attached to that form. One character ID and one voice are preserved for attribution. A future polish pass can add richer inline form editing and per-form bulk-generation UX.

P1 — Portrait / concept-sheet identity mismatch

The avatar and the concept sheet can show two different people (Lancorian: dark-haired man vs white-haired mage). Cause is understood: image_prompt has two sources — the LLM writes it as one of the 53 character-sheet fields, while _build_character_portrait_prompt builds it deterministically from the same structured facts the concept sheet uses. The deterministic one keeps both images consistent; the LLM-written one omits identity anchors (age, hair, build) so the model invents them.

Status — partially working

Working:

• Character records, structured profile fields and portrait images are saved and loaded independently without losing the stored image data.
• The concept-sheet path already builds from structured character facts and is the more consistent identity source.
• Portrait generation now uses baseline appearance guidance, explicit head-and-shoulders framing, neutral backgrounds and book-period grounding.
• Existing scene/style/background contamination in saved portrait prompts is filtered in the portrait path.

Not working reliably:

• The profile/avatar portrait and the character concept sheet can still depict different hair colour, age, facial features, build or overall identity.
• Character-sheet extraction can write an LLM-authored image_prompt that lacks the identity anchors used by the deterministic concept-sheet prompt.
• Regenerating a portrait does not guarantee that it uses exactly the same identity description as the concept sheet.
• Evidence-window profile regeneration can produce behaviour or actions in physical instead of stable appearance details.

Needs to be fixed:

• Make one canonical identity description the input for both avatar portraits and concept sheets: age range, hair, eyes, face, build, skin tone, distinctive marks and clothing era.
• Remove image_prompt from the general sheet-extraction output, or overwrite it only after extraction using the canonical structured fields.
• Store the resolved identity prompt/version with each generated image so old images can be identified as stale and regenerated deterministically.
• Add a consistency check or review indicator when the avatar and concept-sheet images were generated from different prompt versions.
• Add an automated test covering extraction → prompt generation → avatar and concept sheet generation, asserting that both image requests contain the same identity anchors.

The app's own flow already calls /api/character-generate-prompts after extraction, so this is mainly a process trap: any script that regenerates sheets must call that endpoint too, or it leaves an LLM-authored prompt in place. Consider removing image_prompt from the sheet-extraction field list entirely so there is only one source.

Known regression from the 2026-09-02 batch

Regenerating the top-10 profiles from evidence windows (passages around each character's own dialogue) produced weaker physical text than the app's full-book pass — Lancorian's became behaviour ("peers through small windows, moves nimbly on spiral staircases") rather than appearance, where the older sheet had "shoulder-length white hair, faded scar across forehead". Prefer the app's own Character Sheets flow over evidence-window shortcuts when appearance quality matters.