diff --git a/README.md b/README.md index 39e1568..0fdbc50 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,39 @@ CustomVoice uses the model's built-in speaker names. Define the speaker IDs you Then call the CustomVoice service on port `8022`. +## Consistent characters for long-form narration + +For an audiobook a character must sound identical across thousands of requests while still carrying emotion. Those two goals live in different services, so use each for what it is good at. + +**VoiceDesign has no seed.** `generate_voice_design()` accepts none, so a designed voice's identity is re-sampled from its prompt on every call. Low `temperature`/`top_p` narrows the spread but does not remove it. Treat VoiceDesign as a casting tool, not a narration engine. + +**VoiceClone is deterministic.** Every request seeds the RNG — from `seed` in `voices.json`, or a stable hash of the voice name when none is set — and the `.pt` prompt is a frozen recording of one specific reference. The same text yields the same audio across restarts. + +The workflow: + +1. **Cast** the character at port `8021` until a take sounds right. Generate the same line several times; the identity you hear wandering is what step 3 eliminates. +2. **Freeze** the take you liked: save ~10–20 s of clean audio as the character's reference WAV with an exact transcript as `ref_text`. The character is now a VoiceClone voice. +3. **Bake** the `.pt`. `extract_embeddings.py` writes a full ICL prompt (reference codec tokens plus transcript), which both pins the identity and puts you in the mode where instructions are followed reliably. +4. **Pin the seed.** Audition with `find_best_seed.py` and `/seed-samples/{voice}`, then lock the winner with `POST /voice-seed`. +5. **Narrate** at port `8020`, passing per-line `instruct` for emotion. + +Once a character's reference audio is chosen, treat it as immutable. Re-recording it invalidates the `.pt` (see v6.8) and shifts the identity, which means re-auditioning the seed and re-rendering everything already produced. + +### Directing a line + +`instruct` on a request is a *direction*, not a description. It is appended to the voice's own `instruct`, so keep the character in the voice and only the delivery in the request: + +```bash +curl http://localhost:8020/v1/audio/speech \ + -H "Content-Type: application/json" \ + -d '{"input":"You should not have come here.", + "voice":"DE_M_Zerwas", + "instruct":"quietly, through gritted teeth"}' \ + --output line_0417.wav +``` + +Write `"whispering, barely audible"`, not `"a gravelly old man, whispering"` — the second re-describes the character and invites drift. Cast on a calm, level reference: a reference read with strong emotion bakes that emotion into the identity and fights every later direction. Test each character's most extreme line early; if a direction breaks the identity, the reference is too short or too expressive. + ## Streaming backend The streaming service on port `8023` uses the same generated `config/voices.json` and active VoiceClone reference voices as port `8020`, but returns WAV chunks while generation is still running. Use it when time-to-first-audio matters more than waiting for the complete WAV response. @@ -213,7 +246,7 @@ The streaming service on port `8023` uses the same generated `config/voices.json | `response_format` | string | `wav` | `wav`, `pcm`, `mp3`, or `zip` (for timestamps) | | `speed` | float | 1.0 | Scales audio tempo via ffmpeg | | `language` | string | voice config | Per-request override for VoiceDesign/CustomVoice | -| `instruct` | string | voice config | Per-request style override for VoiceDesign/CustomVoice | +| `instruct` | string | voice config | Per-line delivery direction. Appended to the voice's own `instruct`, never replacing it. Supported by all three services | | `max_new_tokens` | int | server default | Per-request generation length override | WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response. @@ -344,6 +377,13 @@ The first request after container startup can be slower because CUDA graph captu ## Changelog +### v6.9 — 2026-07-29 +**Feature: Per-Line Emotional Direction on Cloned Voices** +- **Request-Level `instruct` for VoiceClone:** The clone server previously read `instruct` only from `voices.json`, so a character could have one fixed delivery for an entire book. `SpeechRequest` now accepts `instruct`, applied to both the streaming and non-streaming paths. +- **Identity-First Merging:** A per-request `instruct` is appended to the voice's own description rather than replacing it, so directing one line's emotion cannot re-roll the character. Matches the behaviour added to the VoiceDesign server in v6.8. +- **Why It Works Here:** Qwen3-TTS follows instructions reliably only in ICL mode, not with x-vector-only cloning. The `.pt` prompts written by `extract_embeddings.py` are full ICL prompts, so cloned voices qualify. +- **Documentation:** New "Consistent characters for long-form narration" section covering the cast-in-VoiceDesign, narrate-from-VoiceClone workflow for audiobooks. + ### v6.8 — 2026-07-29 **Fix: Stale Speaker Embeddings and Unregistered Voice Design Voices** - **Stale `.pt` Embeddings:** A speaker embedding is a cache baked from one specific pairing of reference audio and transcript. Re-recording a voice replaced those sources but left the old `.pt` in place, so the server kept cloning from an embedding whose audio tokens no longer matched its transcript — generation ignored the requested text and emitted unrelated filler. `generate_voices.py` now treats an embedding older than its reference audio or transcript as absent and regenerates it. diff --git a/patches/openai_server.patch b/patches/openai_server.patch index 90f7790..8c6440f 100644 --- a/patches/openai_server.patch +++ b/patches/openai_server.patch @@ -1,5 +1,5 @@ diff --git a/examples/openai_server.py b/examples/openai_server.py -index 2199e14..d10065f 100644 +index 2199e14..a65f81e 100644 --- a/examples/openai_server.py +++ b/examples/openai_server.py @@ -36,6 +36,7 @@ API usage: @@ -10,7 +10,7 @@ index 2199e14..d10065f 100644 import io import json import logging -@@ -66,9 +67,46 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API") +@@ -66,9 +67,66 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API") tts_model = None voices: dict = {} @@ -53,11 +53,31 @@ index 2199e14..d10065f 100644 + torch.manual_seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) ++ ++ ++def _merge_instruct(voice_cfg: dict, req_instruct: Optional[str]) -> Optional[str]: ++ """Combine a voice's identity instruct with a per-request delivery direction. ++ ++ A voice's instruct in voices.json describes WHO is speaking; an instruct ++ sent with the request describes HOW this one line is delivered. Replacing ++ the former with the latter would re-roll the character from a few words of ++ direction, so combine them, identity first — the same rule the VoiceDesign ++ server applies. ++ ++ Instructions are only followed reliably in ICL mode. The .pt prompts written ++ by extract_embeddings.py are full ICL prompts (ref codec tokens + ref_text), ++ not x-vector-only, so this path qualifies. ++ """ ++ base = str(voice_cfg.get("instruct") or "").strip() ++ line = str(req_instruct or "").strip() ++ if base and line and line != base: ++ return f"{base} {line}" ++ return line or base or None + # --------------------------------------------------------------------------- # Request / response models -@@ -79,8 +117,8 @@ class SpeechRequest(BaseModel): +@@ -79,8 +137,9 @@ class SpeechRequest(BaseModel): model: str = "tts-1" input: str voice: str = "alloy" @@ -65,10 +85,11 @@ index 2199e14..d10065f 100644 - speed: float = 1.0 # accepted but not yet applied + response_format: str = "wav" # wav | pcm | mp3 | zip + speed: float = 1.0 # scales audio tempo ++ instruct: Optional[str] = None # per-line delivery direction, see _merge_instruct # --------------------------------------------------------------------------- -@@ -145,6 +183,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes: +@@ -145,6 +204,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes: def resolve_voice(voice_name: str) -> dict: """Return voice config dict or fall back to default, else raise 400.""" @@ -90,7 +111,7 @@ index 2199e14..d10065f 100644 if voice_name in voices: return voices[voice_name] if default_voice and default_voice in voices: -@@ -168,7 +221,47 @@ def resolve_voice(voice_name: str) -> dict: +@@ -168,7 +242,47 @@ def resolve_voice(voice_name: str) -> dict: # --------------------------------------------------------------------------- @@ -135,11 +156,11 @@ index 2199e14..d10065f 100644 + return None + + -+async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float) -> AsyncGenerator[bytes, None]: ++async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float, instruct: Optional[str] = None) -> AsyncGenerator[bytes, None]: """ Run generate_voice_clone_streaming in a background thread and yield raw PCM bytes for each chunk as they arrive. -@@ -177,21 +270,63 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No +@@ -177,21 +291,63 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No _DONE = object() def producer(): @@ -179,7 +200,7 @@ index 2199e14..d10065f 100644 ref_text=voice_cfg.get("ref_text", ""), chunk_size=voice_cfg.get("chunk_size", 12), - non_streaming_mode=False, -+ instruct=voice_cfg.get("instruct"), ++ instruct=_merge_instruct(voice_cfg, instruct), + voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model), + non_streaming_mode=True, + temperature=voice_cfg.get("temperature", 0.8), @@ -207,7 +228,7 @@ index 2199e14..d10065f 100644 thread = threading.Thread(target=producer, daemon=True) thread.start() -@@ -203,7 +338,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No +@@ -203,7 +359,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No break if isinstance(item, Exception): raise item @@ -216,7 +237,7 @@ index 2199e14..d10065f 100644 # --------------------------------------------------------------------------- -@@ -220,7 +355,8 @@ async def health(): +@@ -220,7 +376,8 @@ async def health(): async def create_speech(req: SpeechRequest): if tts_model is None: raise HTTPException(status_code=503, detail="Model not loaded") @@ -226,7 +247,7 @@ index 2199e14..d10065f 100644 raise HTTPException(status_code=400, detail="'input' text is empty") voice_cfg = resolve_voice(req.voice) -@@ -230,36 +366,77 @@ async def create_speech(req: SpeechRequest): +@@ -230,36 +387,77 @@ async def create_speech(req: SpeechRequest): "wav": "audio/wav", "pcm": "audio/pcm", "mp3": "audio/mpeg", @@ -255,7 +276,7 @@ index 2199e14..d10065f 100644 - ref_audio=voice_cfg["ref_audio"], + ref_audio=voice_cfg.get("ref_audio"), ref_text=voice_cfg.get("ref_text", ""), -+ instruct=voice_cfg.get("instruct"), ++ instruct=_merge_instruct(voice_cfg, req.instruct), + voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model), + non_streaming_mode=True, + temperature=voice_cfg.get("temperature", 0.8), @@ -305,11 +326,11 @@ index 2199e14..d10065f 100644 if fmt == "wav": yield _wav_header(SAMPLE_RATE) # stream with unknown data length - async for raw_chunk in _stream_chunks(voice_cfg, req.input): -+ async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed): ++ async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed, req.instruct): yield raw_chunk return StreamingResponse(audio_stream(), media_type=content_type) -@@ -306,16 +483,20 @@ def _parse_args(): +@@ -306,16 +504,20 @@ def _parse_args(): p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)") p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)") p.add_argument("--device", default="cuda", help="Torch device (default: cuda)") @@ -331,7 +352,7 @@ index 2199e14..d10065f 100644 with open(args.voices) as f: voices = json.load(f) default_voice = next(iter(voices)) -@@ -344,6 +525,7 @@ def main(): +@@ -344,6 +546,7 @@ def main(): args.model, device=args.device, dtype=torch.bfloat16,