tts-voice-creator-clone-and-design-2/TODO.md

# TTS Voice Creator Roadmap

This roadmap is ordered by priority. Refactors should be incremental and covered by tests before behavior is changed.

Priority levels:

- **P0 — safety:** prevents crashes, data loss, broken releases, or security incidents.
- **P1 — high impact:** improves memory use, responsiveness, and maintainability.
- **P2 — quality:** improves diagnostics, polish, and long-term operations.

## Prioritized execution backlog

Effort estimates are for one developer familiar with this codebase: **S** = a few
hours, **M** = 1–2 days, **L** = several days, **XL** = multi-week or iterative.
Within each priority, tasks are ordered from smallest to largest effort. Complete
the first block before investing in architecture or polish.

### P0 — do first

- [ ] **S — Stop API keys from reaching frontend JavaScript or logs.** Provider keys are now masked in settings
  responses and probes use server-side credentials; finish the external-client key handoff and audit remaining
  transient input/error paths.
- [x] **S — Validate uploaded filenames and paths.** Upload endpoints reject traversal, path separators and control
  characters before deriving extensions; ZIP imports reject absolute, traversal and malformed archive members.
- [ ] **M — Add upload, request-size, temporary-file and concurrent-job limits.** Apply
  endpoint-specific caps and clean up intermediates on failure.
- [x] **M — Add automatic backups before bulk deletes, voice moves and recasts.** Make
  recovery explicit and verify that backups can be restored.
- [ ] **M — Strictly validate custom URLs and prevent SSRF.** Allow only intended
  schemes, hosts and network destinations for provider/container probes.
- [ ] **M — Add authentication and rate limits for destructive/API endpoints.** Protect
  settings, file operations, Docker controls and bulk mutations.
- [ ] **M — Validate character sheets and voice metadata with schemas.** Reject or repair
  malformed records at API boundaries.
- [x] **L — Add database migrations and prompt/generation history.** Preserve old record shapes and expose protected,
  metadata-only generation history for auditing and follow-up recovery workflows.
- [x] **L — Harden container and Docker access.** The app runs non-root with dropped capabilities,
  no-new-privileges and a read-only root filesystem; Compose Docker access is isolated behind a minimal container
  proxy with only container inspection/control enabled.

### P1 — next, highest user impact

- [x] **S — Re-measure audiobook segmentation against `cast_run3.json`.** Run 3 has
  2,227 segments, 1,043 dialogue segments and 585 newline-fused segments, but zero
  adjacent same-speaker dialogue pairs for `_audiobookMergeAdjacentDialogue` to merge.
  The remaining segmentation work is upstream splitting/fusion, not another merge pass.
- [x] **S — Finish the remaining Playwright smoke checks.** Characters/Cast, profile
  sorting, column visibility, voice ratings, and starting/cancelling a generation job are covered.
- [x] **M — Fix the Schwester Larissa unclosed-guillemet split.** Attribution normalization now repairs dialogue
  split before mixed German guillemet closes (`»…‹` / `«…›`), with an epigraph regression test.
- [ ] **M — Make profile portraits and character sheets use one canonical identity.**
  Shared structured prompts now version consistent pairs as v3 and fingerprint canonical appearance fields; add
  cross-record validation so mismatched profile/reference files are detected even when both are current-version. The
  UI now flags missing, stale, changed, or explicitly inconsistent pair metadata; visual similarity itself still needs
  an image-level review workflow.
- [x] **M — Add bounded duplicate-generation protection.** Consistent portrait/reference, voice-design, and casting
  jobs now deduplicate repeated requests with bounded, expiring `Idempotency-Key` contracts.
- [ ] **M — Improve large-table performance.** Character productions and the flat voice library now paginate at 50
  visible rows; updating only affected rows and disabling preloading on low-memory devices remain.
- [ ] **M — Add explicit cache invalidation and eviction.** Fish.audio search responses and persisted synthesized
  line-audio clips now have bounded eviction; image entries and browser object URLs still need equivalent cleanup.
- [ ] **M — Add constrained-device validation.** Test with Chrome CPU/memory throttling,
  a 4 GB VM/container, and measure browser, Docker and temporary-disk usage.
- [ ] **L — Add background jobs and bounded queues.** Give voice design, transcription,
  casting and image generation IDs, progress, cancellation, timeout, retry and
  idempotency support.
- [ ] **L — Run the Book 2 generalization regression.** Confirm speaker-attribution and
  confidence rules are not overfit to *Der Sturm*.
- [ ] **L — Add character forms inside one character record.** Support multiple
  appearances while preserving one voice and one attribution target.
- [ ] **L — Modularize the frontend.** Split Audiobook, Rehearser and styles, replace
  broad globals with namespaces, and consolidate the loader/module manifest.

### P2 — after behavior and safety stabilize

- [ ] **M — Add structured logs with request/job IDs.** Report provider, endpoint,
  duration and failure reason without logging prompts, keys or book contents.
- [ ] **M — Separate provider health from configuration status.** Show actionable
  connectivity and authentication errors.
- [ ] **M — Add visible job history and retry controls.** Expose recent failures and
  safe retry state to users.
- [ ] **L — Add memory diagnostics.** Show browser heap where supported, rendered rows,
  cache size, active jobs, server/container memory and recent operation sizes.
- [ ] **L — Add linting and duplicate-function checks.** Run them in CI alongside the
  existing compile, bundle and SonarQube checks.

## P0 — Testing, data safety, and resource limits

- [x] Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported).
- [x] Add explicit Docker memory limits where appropriate; the web container defaults to an 8 GB limit and 2 GB reservation with environment overrides.
- [x] Limit upload sizes, temporary files, and concurrent jobs. Upload/request caps, bounded caches, and bounded
  TTS/STT/image-generation pools are implemented; durable background-job conversion remains a separate P1 item.
- [x] Use atomic writes for `.meta.json` files.
- [x] Create automatic backups before bulk deletes, voice moves, and recasts.
- [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints.

## P1 — Modular frontend architecture

- [ ] Split `audiobook.js`, `rehearser.js`, and `style.css` by responsibility.
- [ ] Replace broad globals with namespaces such as `window.TTVC.audiobook`.
- [ ] Generate `loader.js` and `scripts/minify.mjs` from one module manifest.
- [ ] Add linting and duplicate-function checks.
- [ ] Extract Rehearser voice matching and voice design.
- [ ] Extract Audiobook parsing and casting.
- [ ] Split `style.css` by feature.
- [ ] Do not migrate to TypeScript or rewrite the frontend until behavior is covered by tests.

## P1 — Frontend performance and memory safety

- [ ] Lazy-load Audiobook, Rehearser, Studio, and image-generation modules only when opened.
- [ ] Virtualize or paginate large character and voice tables; render approximately 30–50 visible rows.
- [ ] Cache `/api/voices` with explicit invalidation instead of repeated full scans.
- [x] Lazy-load profile images and concept art.
- [ ] Update only affected table rows instead of rebuilding entire tables.
- [ ] Add an LRU cache for audio and images and remove old entries automatically. Persisted line-audio is now bounded;
  image-cache eviction remains.
- [x] Revoke discarded object URLs with `URL.revokeObjectURL(objectUrl)`; completed synthesized voice previews and
  character/Rehearser local auditions now release their Blob URLs when playback ends or is replaced.
- [x] Add low-memory mode using `navigator.deviceMemory` where available; shared worker pools cap at two tasks on devices reporting approximately 4 GB RAM or less.
- [x] Disable image preloading and reduce table/cache sizes on devices with approximately 4 GB RAM. Images use
  lazy decoding, and character/voice pages reduce from 50 to 25 rows when `navigator.deviceMemory <= 4`.

## P1 — Backend stability and bounded concurrency

- [ ] Introduce background jobs for voice design, transcription, casting, and image generation.
- [ ] Give every job an ID, progress, cancellation, timeout, retry, and idempotency key. Consistent image jobs now
  provide all except retry; voice design and casting remain synchronous request paths.
- [x] Prevent duplicate generation when a user clicks twice for image, voice-design, and casting submissions.
- [ ] Centralize provider health checks and error handling.
- [x] Limit concurrency to 1–2 image generations, 1–2 voice generations, and 1 STT request per user. Consistent
  portrait/reference rendering, preview/style/seed-finder synthesis, and the OpenAI-compatible speech proxy are
  capped at two active jobs; STT transcription and benchmarks are capped at one active job.
- [x] Use bounded server queues for the implemented synchronous generation paths; durable background-job queues remain
  a separate outstanding item above.

## P0 — Security

- [ ] Never expose API keys to frontend JavaScript or logs. Provider keys are masked on read and absent from browser
  storage; the deliberate external-client key display and transient entry path still need a final design review.
- [x] Validate uploaded filenames and paths.
- [x] Strictly validate custom URLs to prevent SSRF, with private destinations available only when explicitly allowed
  for local provider/container operation.
- [x] Add request-size limits, rate limits, and authentication for destructive/API endpoints.
- [x] Run containers as non-root with dropped capabilities and read-only filesystems where practical.
- [x] Isolate Docker socket access behind a minimal privileged service.

## P0 — Data integrity and recovery

- [x] Validate character sheets and voice metadata with schemas.
- [x] Add database migrations instead of silently changing record shapes.
- [x] Use atomic writes for `.meta.json` files.
- [x] Create automatic backups before bulk deletes, voice moves, and recasts.
- [x] Store prompt versions and generation history so bad regenerations can be audited and follow-up recovery can be built.

## P0 — Testing

### Automated tests

- [ ] Character parsing and phantom-speaker prevention.
- [ ] Voice-matching constraints.
- [ ] Profile-completeness counting.
- [x] Rating persistence.
- [ ] STT endpoint normalization.
- [ ] ComfyUI workflow submission.
- [ ] API failure and timeout behavior.

### Playwright smoke suite

- [x] Load the app.
- [x] Verify the production bundle loads.
- [x] Verify language switching and Rehearser parsing.
- [x] Verify clone source tabs switch correctly.
- [x] Open Characters / Cast.
- [x] Sort the profile column.
- [x] Hide and show columns.
- [x] Rate a voice.
- [x] Start and cancel a generation job.

## P2 — Observability

- [ ] Add structured logs with request and job IDs.
- [ ] Report provider, endpoint, duration, and error reason clearly.
- [ ] Separate health status from “configured”.
- [ ] Add visible job history and retry controls.
- [ ] Avoid logging prompts, API keys, or book contents by default.

## P1 — Resource limits and constrained-device testing

- [x] Add explicit Docker memory limits; the web container defaults to `mem_limit: 8g` and `mem_reservation: 2g`, configurable per host.
- [x] Limit upload sizes and clean converted upload intermediates; continue expanding endpoint-specific temporary-file and job limits.
- [ ] Test with Chrome DevTools CPU/memory throttling.
- [ ] Test in a VM or container limited to 4 GB RAM.
- [ ] Monitor browser-tab memory, JavaScript heap, DOM-node count, active audio blobs, Docker memory, and temporary disk usage.
- [ ] Add a memory diagnostics page showing browser heap where supported, rendered rows, cache count/size, active jobs, server/container memory, and largest recent operations.

## P1 — Speaker attribution: measured state and next steps (paused 2026-09-02)

Three full regression runs of *Der Sturm* against the hand-corrected reference.
Everything below is measured, not estimated. Artefacts live in
`config/regression_snapshots/der_sturm/` (`*_manual.json` are the human
reference — never overwrite them).

| Run | Segments | Headline agreement | **Fully correct** | Wrong speaker | Unmatched |
|-----|---------:|-------------------:|------------------:|--------------:|----------:|
| 1   | 2143 | 72.6% | **54.2%** | 438 | 544 |
| 2   | 2273 | 77.7% | **52.8%** | 345 | 727 |
| 3   | 2227 | 78.7% | **52.2%** | 314 | 750 |
| human reference | 2110 | — | — | 0 | — |

**Read the headline with care.** Agreement is computed only over rows whose text
matches the human's, and that pool keeps shrinking (1598 → 1545 → 1475). Absolute
correct segments went 1160 → 1200 → 1161. The rising headline is largely a
shrinking denominator; true end-to-end accuracy has drifted slightly *down*.

### What genuinely improved
- Unknown speakers 121 → 5 → 8; the addressee/Unknown prompt rewrite did **not**
  overcorrect (369 after cast collapsed to 19 after one recast pass).
- Wrong speaker names 438 → 345 → 314 — consistent, real.
- Verify pass now completes (~65 LLM calls) instead of never finishing.

### The blocker: segmentation, not attribution
750 of ~1060 problem segments are text that doesn't match the human's at all:
roughly 400 split finer than the human, ~210 merged where the human split, ~70
OCR/wording differences. Attribution work cannot fix these.

- `fused` dialogue segments hit **212** (human: 6, run 2: 59). These are
  `_audiobookMergeAdjacentDialogue` joins using `\n\n`; 137 of them match no
  human segment. Splitting them back was tested and is **worse** (52.2% → 43.4%),
  so the merge is not the primary cause — but merging blocks the human kept
  separate is contributing.
- The merge was predicted (on run 2 data) to cut unmatched 727 → 588. On run 3 it
  went to 750. **That prediction did not hold** and needs re-deriving on run 3.
- Fixed: attribution normalization repairs the Schwester Larissa epigraph when the LLM splits it before a mixed
  guillemet close; the regression is covered by `tests/test_conversation.py`.
  (unclosed-guillemet handling).

### Next steps, highest value first
1. Attack segmentation directly — it is now the dominant error class. Run 3 has
   no merge candidates, so inspect the earlier dialogue split/fusion stages and the
   585 already-fused segments instead of changing `_audiobookMergeAdjacentDialogue`.
2. Preserve the dialogue-only merge for recast/verify outputs where adjacent pairs
   can still occur; do not claim it explains the run-3 mismatch.
3. Only then re-run book 1. Re-running before segmentation moves will re-measure
   rules already known not to shift the number that matters.
4. **Book 2 generalization run never started.** Reference is backed up at
   `config/regression_snapshots/book2/`. Its point is to test whether these rules
   generalise or were overfit to book 1 — still unanswered, and the confidence
   scorer in particular was tuned solely on book 1.

### Character profiles
48 human vs 64 auto (run 3), 35 matched, 13 missed. Auto over-generates and got
worse than run 2's 51. Generic role labels (`Fremder` 19 lines, `Vampir` 19,
`Inquisitor` 15, `Ork` 13, and the adverb `Unsicher`) still become cast members
with their own voices; the appositive resolver only fires where the book supplies
grammatical evidence (`Darrag der Schmied`), and pre-reveal aliases like
"der Fremde" deliberately withhold it.

### Harness notes (cost real time; do not relearn)
- Casting auto-starts `audiobookAutoRepairCast` 800 ms after finishing, which
  re-sets `_audiobook.running`. Guards now explain the refusal instead of
  returning silently, but an automated driver must still wait for `running` to
  clear and treat a sub-60s phase as a bailout, not a success.
- Stall detection must key on the passage counter (`#ab-cv-count`). `segments`,
  `done` and `unknown` do not move during a verify pass — watching them killed a
  healthy run twice.
- Character Sheets only saves after its passage loop ends; poll `/api/characters`
  until records appear before closing the browser, or a 4.4 h run is discarded.

## P1 — Character forms (decided 2026-09-02, not yet built)

Shape-shifting characters (Zerwas: human / vampire / wolf-demon, aliases "Fürst der
Nacht", "der Wolfsmensch") need more than one appearance per character.

**Decided:** forms live INSIDE one character record, not as separate records — one
voice, one attribution target, so cast lines saying "Zerwas" always resolve. The
sheet gains a form switcher; each form carries its own `physical`, `image_prompt`
and portrait. **Detection is automatic**: the sheet prompt asks the LLM to identify
whether a character has distinct forms and describe each, so it generalises to any
book (werewolves, disguises, transformations) without per-book rules.

Not started. Data model, sheet UI and prompt changes all still to do.

## P1 — Portrait / concept-sheet identity mismatch

The avatar and the concept sheet can show two different people (Lancorian: dark-haired
man vs white-haired mage). Cause is understood: `image_prompt` has two sources — the
LLM writes it as one of the 53 character-sheet fields, while
`_build_character_portrait_prompt` builds it deterministically from the same structured
facts the concept sheet uses. The deterministic one keeps both images consistent; the
LLM-written one omits identity anchors (age, hair, build) so the model invents them.

### Status — partially working

**Working:**

- Character records, structured profile fields and portrait images are saved and loaded
  independently without losing the stored image data.
- The concept-sheet path already builds from structured character facts and is the more
  consistent identity source.
- Portrait generation now uses baseline appearance guidance, explicit head-and-shoulders
  framing, neutral backgrounds and book-period grounding.
- Existing scene/style/background contamination in saved portrait prompts is filtered in
  the portrait path.

**Not working reliably:**

- The profile/avatar portrait and the character concept sheet can still depict different
  hair colour, age, facial features, build or overall identity.
- Character-sheet extraction can write an LLM-authored `image_prompt` that lacks the
  identity anchors used by the deterministic concept-sheet prompt.
- Regenerating a portrait does not guarantee that it uses exactly the same identity
  description as the concept sheet.
- Evidence-window profile regeneration can produce behaviour or actions in `physical`
  instead of stable appearance details.

**Needs to be fixed:**

- Make one canonical identity description the input for both avatar portraits and concept
  sheets: age range, hair, eyes, face, build, skin tone, distinctive marks and clothing
  era.
- Remove `image_prompt` from the general sheet-extraction output, or overwrite it only
  after extraction using the canonical structured fields.
- Store the resolved identity prompt/version with each generated image so old images can
  be identified as stale and regenerated deterministically.
- Add a consistency check or review indicator when the avatar and concept-sheet images
  were generated from different prompt versions.
- Add an automated test covering extraction → prompt generation → avatar and concept
  sheet generation, asserting that both image requests contain the same identity anchors.

The app's own flow already calls `/api/character-generate-prompts` after extraction, so
**this is mainly a process trap**: any script that regenerates sheets must call that
endpoint too, or it leaves an LLM-authored prompt in place. Consider removing
`image_prompt` from the sheet-extraction field list entirely so there is only one source.

### Known regression from the 2026-09-02 batch
Regenerating the top-10 profiles from evidence windows (passages around each character's
own dialogue) produced weaker `physical` text than the app's full-book pass — Lancorian's
became behaviour ("peers through small windows, moves nimbly on spiral staircases")
rather than appearance, where the older sheet had "shoulder-length white hair, faded scar
across forehead". Prefer the app's own Character Sheets flow over evidence-window
shortcuts when appearance quality matters.