tts-voice-creator-clone-and-design-2/PROJECT_MAP.md

Project map

Living index of where things are and what state they are in. Update in the same commit as the change it describes.

Current phase

Magazine projects, P0 and P1 complete (frontend reachable at Studio → Magazines). Audiobook/Rehearser/voice-library features are mature and in daily use. Latest release v1.59.0: global "All characters" view in the sidebar and per-book characters inside Studio, untitled-page cleanup with an exclude option, duplicate/clean tools. v1.58.0 added the driving-mode menu, auto-advance, article index with page labels, merge suggestions, distinct face profiles, English prompt translation, vision-response cache and the review Approve fix. v1.57.0 added driving mode with sentence/word-level sync via tts-aligner.

Deployment

Two self-hosted Docker instances, both port 7890:

• DGX Spark (192.168.178.8) — primary. docker-compose.yml builds locally and tags the result ghcr.io/martin-b78/tts-voice-creator-clone-and-design-2, but that registry copy is not kept current on every deploy (several versions behind as of 2026-09-14) — treat it as a snapshot, not a source of truth. Python and static files are bind-mounted, so a code change needs only a container recreate; rebuild only when Dockerfile or requirements.txt change.

• correspondence-nuc (192.168.178.19) — secondary, live since 2026-09-14. No TTS/OCR engines of its own; every generation call crosses the LAN to the DGX (docker-compose.nuc.yml, tracked here, pins vision_ocr_url/vision_ocr_fallback_url explicitly to the DGX's IP — their default resolves relative to whichever host the container runs on, silently wrong here otherwise). Voice library is the DGX's own directory, NFS-mounted read-write (/etc/exports on the DGX, /etc/fstab on the NUC) — never a separate copy that could drift. Deployed by rsyncing the working tree (excluding .git, config/, logs/) and building on the NUC itself, since the ghcr.io image is both stale and private with no registry login there. See TODO.md's "P2 — deployment" entry for the full setup and the one accepted limitation (voice-change auto-restart of the DGX's TTS containers doesn't reach across hosts — manual restart on the DGX, same as before this existed).

• tts-aligner (forced-aligner sidecar, aligner/) exists only in the DGX docker-compose.yml. The NUC compose file deliberately omits it: with ALIGNER_URL unset the app quietly falls back to estimated word timing, so the NUC works unchanged. Known limitation: no measured word times on the NUC.

• Rollback (either host): check out the previous tag (git checkout v1.58.0; on the NUC re-rsync that tree), then docker compose up -d --force-recreate (rebuild only if Dockerfile/requirements.txt changed). Stop the aligner with docker compose stop tts-aligner; audio/<id>.timings.json version 2 files stay readable by v1 readers.

Layout

Path	Role
server.py	FastAPI app, router registration, auth middleware
core/	domain logic — config, database, audio, voice, routing, magazine
routes/	HTTP boundary, one module per feature area
static/js/	frontend, one module per feature area (no build step for most)
tests/	test_*.py (unittest, run with python3 -m unittest tests.test_x), *.test.mjs (npm run test:unit, Node) and *.spec.js (Playwright — must mock every write, the app on :7890 is the real one)
scripts/	score_attribution.py (word-aligned cast scoring), release/minify/frontend-gate tooling
architecture/	SOPs for subsystems whose design is not obvious from code
workflows/	ComfyUI graphs for character portrait generation

Feature areas

Area	Server	Frontend	State
Voice library / clone / design	routes/library.py, core/voice.py	library.js	mature
Voice archetype gallery (40 own German-first design voices, cached previews)	core/voice_archetypes.py, routes/archetypes.py	voice-archetypes.js, s-design.html	new — see architecture/voice-archetypes.md
Read Aloud (books, PDFs)	routes/reader.py	reader.js	mature
Conversation metrics in the Latency column (TTFB charts STT/LLM/TTS, token usage)	routes/conversation.py (_usage_from_chunk, _post_llm_stream)	conversation.js, s-conversation.html	new 2026-10-06 — in-memory, last 50 turns
Audiobook casting (span mode: deterministic quote spans, LLM names speakers only; verify/recast)	routes/conversation.py, core/attribution_spans.py	audiobook*.js, audiobook-spans.js, audiobook-cast-spans.js	new 2026-09-19 — see architecture/audiobook-attribution.md
Characters (avatar/concept identity, appearance forms)	routes/characters.py, routes/conversation.py	library-characters.js, character-sheets.js	reworked 2026-09-19 — see architecture/character-visual-identity.md
Rehearser	routes/rehearsals_db.py	rehearser-*.js	mature
Studio (phase wizard)	—	studio.js	mature
TTS engines / routing	routes/tts.py, core/routing.py	engines.js	mature
STT	routes/stt.py	—	mature
Magazines (incl. paragraph fusion, acronym glossary, spellcheck: core/pronunciations.py, core/spelling.py)	core/magazine.py, routes/magazine.py	magazine.js, s-magazine.html	P0 + P1

Data model

Documents (books and magazines alike) live under <CONFIG_DIR>/reader_library/<doc_id>/, discriminated by meta.json → kind:

• kind: "pdf" / text — Read Aloud book. Text is extracted in the browser and not persisted; meta.json holds counts and a flat idx resume position.
• kind: "magazine" — adds layout.json, articles.json, pages/, images/, podcast/, dictionary.json (words kept as valid by the reviewer or the spelling agent), per-article glossary (acronym → spoken form, in articles.json), and a per-article progress map. See architecture/magazine-ingestion.md.

Settings, presets, routes and the SQLite database also live under CONFIG_DIR, which is the writable volume.

Environment

Variable	Purpose
CONFIG_DIR	writable state (settings, library, database)
VOICES_DIR / OUTPUT_DIR	voice WAV library
TTS_CONTAINER_NAMES	containers restarted after a voice change
DOCKER_HOST	points at the socket proxy; the raw socket is not mounted
ALIGNER_URL	forced-aligner sidecar for word timing (http://tts-aligner:8000; empty = off)
llm_url / llm_model / llm_api_key	settings.json, OpenAI-compatible LLM

QA: npm run qa -- --fast runs lint, unit, security, build checks (see architecture/qa.md); optional --sonar needs SONAR_HOST_URL + SONAR_TOKEN in the shell/CI secrets. Sonar triage: npm run sonar:triage (dry-run; architecture/sonar-triage.md).

Pending decisions

• Container registry: ghcr costs quota against a private package (≈427 MB per push, 500 MB free on the Free plan). Local registry vs. ghcr vs. no re-pull is unresolved — the compose files currently point at ghcr.

Known limitations

• Live data changes made on the DGX only (not on the NUC, which has its own DB/config): book KONTROLLE removed, 5 duplicate characters resolved, untitled-page cleanup applied (backups under config/backups/).
• 18 re-rendered/stale audio articles are pending the owner's voice choice.
• scripts/clean_character_records.py was applied for 5 pairs; the 17 remaining duplicate-sheet pairs need owner decisions.
• Audiobook attribution (span mode) is measured, not perfect: on three hand-corrected novels about 90-92 % of all words end up with the right voice and ~75-80 % of spoken words with the right speaker. The references themselves contain label noise (1-2 points is not meaningful). Open, real problems: a person known only by a role that means different people in different scenes ("der Fremde") is not resolved to a character; look-alike characters in one scene are the usual mix-up; very long monologues are inconsistent in the references.
• A speech is only found where the text marks it with a recognised quotation style (»…«, „…“, “…”, «…»); a book with none of them uses the older, slower whole-passage engine.
• Character image generation needs ComfyUI/an image provider; the deterministic prompts, forms and review logic are covered by tests without one, but a real pair render is not part of the suite.
• gitea backup remote is 157 commits behind origin; main tracks gitea.
• Magazine sidebar exclusion is a narrow-column geometry heuristic (mark_sidebars), not certain — review always shows every sidebar block.
• Magazine spelling relies on pyspellchecker word lists (installed in the container, not necessarily on a host Python): German has no morphological analysis, so unusual compounds beyond a two-part split are flagged and need Keep. Paragraph fusion is heuristic (paragraph_continues); a wrongly joined pair is fixed in review.
• Magazine word-level highlighting is a character-length estimate against real measured audio duration, not true forced alignment.
• No formal cross-engine OCR accuracy benchmark exists (two engines wired: ocr-baidu-unlimited-ocr primary, KDL Frontier Parser nano fallback).
• Figure description generation (Magazine review, "Describe") needs a genuinely caption-capable vision model — confirmed live that the primary engine (baidu/Unlimited-OCR) is transcription-only and returns empty content for any descriptive prompt, regardless of phrasing (three prompts tried). Falls through to vision_ocr_fallback_url/_model, which worked correctly once its container was started — that container (ocr-kdl-frontier-parser-nano) is not running by default.
• Figure boundaries are re-derived from the page render (snap_visual_bboxes) because vision OCR grounds figures loosely; it corrects roughly 100 of the 147 figure regions on c't KI Wissen 2026 and deliberately declines the rest. A pale chart or line drawing on white is below its ink threshold and keeps the engine's box — draw those by hand in review. The pass is idempotent (safe to re-run via "Fix figure boundaries" any number of times), including the one confirmed pair that cannot mathematically converge — two adjacent figures sharing a dark caption strip settle on the same answer every run rather than continuing to drift between two states.
• No npm run qa aggregate command yet; Python tests run via unittest, browser tests via playwright test.
• The app has no authentication for the browser UI and no per-user data isolation — single-tenant/LAN-trust only (one global settings/database/voice library, no accounts). See architecture/multi-tenant-and-identity.md and TODO.md's "Deferred — public multi-user readiness" section.

Next logical step

Magazine P2 per TODO.md: image descriptions shipped; remaining — table/chart handling depth, EPUB 3 Media Overlay export, and a formal OCR-engine benchmark (see architecture/magazine-ingestion.md). Then Character system, then Audiobook speaker attribution, per the user's stated priority order.