mARTin-B78/tts-dgx-spark-faster-qwen3-tts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v6.9: Per-line instruct on cloned voices

Browse Source 
The clone server passed instruct=voice_cfg.get("instruct") at both call sites,
so a voice's delivery was fixed in voices.json for its whole lifetime. For an
audiobook that means a character can be consistent or expressive, not both:
VoiceDesign takes a per-request instruct but has no seed, so its identity is
re-sampled on every call, while VoiceClone is deterministic but had no way to
direct a line.

SpeechRequest now accepts instruct, merged identity-first via _merge_instruct()
— the voice's own description followed by the line's direction — so directing
one line cannot re-roll the character. Same rule run_voicedesign_server.py got
in v6.8.

This is safe on the clone path specifically because extract_embeddings.py writes
full ICL prompts (ref codec tokens + ref_text), not x-vector-only ones;
model.py warns that instruct is unreliable in the latter mode.

The change lives in patches/openai_server.patch, regenerated against the
pristine upstream blob and verified to apply cleanly. README documents the
request field and adds a section on the cast-in-VoiceDesign,
narrate-from-VoiceClone workflow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-07-29 19:19:05 +02:00
parent 6198f15f64
commit 78e34ebddb
2 changed files with 77 additions and 16 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
README.md
View File

@ -186,6 +186,39 @@ CustomVoice uses the model's built-in speaker names. Define the speaker IDs you
Then call the CustomVoice service on port `8022`. Then call the CustomVoice service on port `8022`.
## Consistent characters for long-form narration
For an audiobook a character must sound identical across thousands of requests while still carrying emotion. Those two goals live in different services, so use each for what it is good at.
**VoiceDesign has no seed.** `generate_voice_design()` accepts none, so a designed voice's identity is re-sampled from its prompt on every call. Low `temperature`/`top_p` narrows the spread but does not remove it. Treat VoiceDesign as a casting tool, not a narration engine.
**VoiceClone is deterministic.** Every request seeds the RNG — from `seed` in `voices.json`, or a stable hash of the voice name when none is set — and the `.pt` prompt is a frozen recording of one specific reference. The same text yields the same audio across restarts.
The workflow:
1. **Cast** the character at port `8021` until a take sounds right. Generate the same line several times; the identity you hear wandering is what step 3 eliminates.
2. **Freeze** the take you liked: save ~10–20 s of clean audio as the character's reference WAV with an exact transcript as `ref_text`. The character is now a VoiceClone voice.
3. **Bake** the `.pt`. `extract_embeddings.py` writes a full ICL prompt (reference codec tokens plus transcript), which both pins the identity and puts you in the mode where instructions are followed reliably.
4. **Pin the seed.** Audition with `find_best_seed.py` and `/seed-samples/{voice}`, then lock the winner with `POST /voice-seed`.
5. **Narrate** at port `8020`, passing per-line `instruct` for emotion.
Once a character's reference audio is chosen, treat it as immutable. Re-recording it invalidates the `.pt` (see v6.8) and shifts the identity, which means re-auditioning the seed and re-rendering everything already produced.
### Directing a line
`instruct` on a request is a *direction*, not a description. It is appended to the voice's own `instruct`, so keep the character in the voice and only the delivery in the request:
```bash
curl http://localhost:8020/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"input":"You should not have come here.",
"voice":"DE_M_Zerwas",
"instruct":"quietly, through gritted teeth"}' \
--output line_0417.wav
```
Write `"whispering, barely audible"`, not `"a gravelly old man, whispering"` — the second re-describes the character and invites drift. Cast on a calm, level reference: a reference read with strong emotion bakes that emotion into the identity and fights every later direction. Test each character's most extreme line early; if a direction breaks the identity, the reference is too short or too expressive.
## Streaming backend ## Streaming backend
The streaming service on port `8023` uses the same generated `config/voices.json` and active VoiceClone reference voices as port `8020`, but returns WAV chunks while generation is still running. Use it when time-to-first-audio matters more than waiting for the complete WAV response. The streaming service on port `8023` uses the same generated `config/voices.json` and active VoiceClone reference voices as port `8020`, but returns WAV chunks while generation is still running. Use it when time-to-first-audio matters more than waiting for the complete WAV response.
@ -213,7 +246,7 @@ The streaming service on port `8023` uses the same generated `config/voices.json
| `response_format` | string | `wav` | `wav`, `pcm`, `mp3`, or `zip` (for timestamps) | | `response_format` | string | `wav` | `wav`, `pcm`, `mp3`, or `zip` (for timestamps) |
| `speed` | float | 1.0 | Scales audio tempo via ffmpeg | | `speed` | float | 1.0 | Scales audio tempo via ffmpeg |
| `language` | string | voice config | Per-request override for VoiceDesign/CustomVoice | | `language` | string | voice config | Per-request override for VoiceDesign/CustomVoice |
| `instruct` | string | voice config | Per-request style override for VoiceDesign/CustomVoice | | `instruct` | string | voice config | Per-line delivery direction. Appended to the voice's own `instruct`, never replacing it. Supported by all three services |
| `max_new_tokens` | int | server default | Per-request generation length override | | `max_new_tokens` | int | server default | Per-request generation length override |
WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response. WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response.
@ -344,6 +377,13 @@ The first request after container startup can be slower because CUDA graph captu
## Changelog ## Changelog
### v6.9 — 2026-07-29
**Feature: Per-Line Emotional Direction on Cloned Voices**
- **Request-Level `instruct` for VoiceClone:** The clone server previously read `instruct` only from `voices.json`, so a character could have one fixed delivery for an entire book. `SpeechRequest` now accepts `instruct`, applied to both the streaming and non-streaming paths.
- **Identity-First Merging:** A per-request `instruct` is appended to the voice's own description rather than replacing it, so directing one line's emotion cannot re-roll the character. Matches the behaviour added to the VoiceDesign server in v6.8.
- **Why It Works Here:** Qwen3-TTS follows instructions reliably only in ICL mode, not with x-vector-only cloning. The `.pt` prompts written by `extract_embeddings.py` are full ICL prompts, so cloned voices qualify.
- **Documentation:** New "Consistent characters for long-form narration" section covering the cast-in-VoiceDesign, narrate-from-VoiceClone workflow for audiobooks.
### v6.8 — 2026-07-29 ### v6.8 — 2026-07-29
**Fix: Stale Speaker Embeddings and Unregistered Voice Design Voices** **Fix: Stale Speaker Embeddings and Unregistered Voice Design Voices**
- **Stale `.pt` Embeddings:** A speaker embedding is a cache baked from one specific pairing of reference audio and transcript. Re-recording a voice replaced those sources but left the old `.pt` in place, so the server kept cloning from an embedding whose audio tokens no longer matched its transcript — generation ignored the requested text and emitted unrelated filler. `generate_voices.py` now treats an embedding older than its reference audio or transcript as absent and regenerates it. - **Stale `.pt` Embeddings:** A speaker embedding is a cache baked from one specific pairing of reference audio and transcript. Re-recording a voice replaced those sources but left the old `.pt` in place, so the server kept cloning from an embedding whose audio tokens no longer matched its transcript — generation ignored the requested text and emitted unrelated filler. `generate_voices.py` now treats an embedding older than its reference audio or transcript as absent and regenerates it.

51
patches/openai_server.patch
View File

@ -1,5 +1,5 @@
diff --git a/examples/openai_server.py b/examples/openai_server.py diff --git a/examples/openai_server.py b/examples/openai_server.py
index 2199e14..d10065f 100644 index 2199e14..a65f81e 100644
--- a/examples/openai_server.py --- a/examples/openai_server.py
+++ b/examples/openai_server.py +++ b/examples/openai_server.py
@@ -36,6 +36,7 @@ API usage: @@ -36,6 +36,7 @@ API usage:
@ -10,7 +10,7 @@ index 2199e14..d10065f 100644
import io import io
import json import json
import logging import logging
@@ -66,9 +67,46 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API") @@ -66,9 +67,66 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API")
tts_model = None tts_model = None
voices: dict = {} voices: dict = {}
@ -53,11 +53,31 @@ index 2199e14..d10065f 100644
+ torch.manual_seed(seed) + torch.manual_seed(seed)
+ if torch.cuda.is_available(): + if torch.cuda.is_available():
+ torch.cuda.manual_seed_all(seed) + torch.cuda.manual_seed_all(seed)
+
+
+def _merge_instruct(voice_cfg: dict, req_instruct: Optional[str]) -> Optional[str]:
+ """Combine a voice's identity instruct with a per-request delivery direction.
+
+ A voice's instruct in voices.json describes WHO is speaking; an instruct
+ sent with the request describes HOW this one line is delivered. Replacing
+ the former with the latter would re-roll the character from a few words of
+ direction, so combine them, identity first — the same rule the VoiceDesign
+ server applies.
+
+ Instructions are only followed reliably in ICL mode. The .pt prompts written
+ by extract_embeddings.py are full ICL prompts (ref codec tokens + ref_text),
+ not x-vector-only, so this path qualifies.
+ """
+ base = str(voice_cfg.get("instruct") or "").strip()
+ line = str(req_instruct or "").strip()
+ if base and line and line != base:
+ return f"{base} {line}"
+ return line or base or None
+ +
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Request / response models # Request / response models
@@ -79,8 +117,8 @@ class SpeechRequest(BaseModel): @@ -79,8 +137,9 @@ class SpeechRequest(BaseModel):
model: str = "tts-1" model: str = "tts-1"
input: str input: str
voice: str = "alloy" voice: str = "alloy"
@ -65,10 +85,11 @@ index 2199e14..d10065f 100644
- speed: float = 1.0 # accepted but not yet applied - speed: float = 1.0 # accepted but not yet applied
+ response_format: str = "wav" # wav | pcm | mp3 | zip + response_format: str = "wav" # wav | pcm | mp3 | zip
+ speed: float = 1.0 # scales audio tempo + speed: float = 1.0 # scales audio tempo
+ instruct: Optional[str] = None # per-line delivery direction, see _merge_instruct
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -145,6 +183,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes: @@ -145,6 +204,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
def resolve_voice(voice_name: str) -> dict: def resolve_voice(voice_name: str) -> dict:
"""Return voice config dict or fall back to default, else raise 400.""" """Return voice config dict or fall back to default, else raise 400."""
@ -90,7 +111,7 @@ index 2199e14..d10065f 100644
if voice_name in voices: if voice_name in voices:
return voices[voice_name] return voices[voice_name]
if default_voice and default_voice in voices: if default_voice and default_voice in voices:
@@ -168,7 +221,47 @@ def resolve_voice(voice_name: str) -> dict: @@ -168,7 +242,47 @@ def resolve_voice(voice_name: str) -> dict:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -135,11 +156,11 @@ index 2199e14..d10065f 100644
+ return None + return None
+ +
+ +
+async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float) -> AsyncGenerator[bytes, None]: +async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float, instruct: Optional[str] = None) -> AsyncGenerator[bytes, None]:
""" """
Run generate_voice_clone_streaming in a background thread and yield Run generate_voice_clone_streaming in a background thread and yield
raw PCM bytes for each chunk as they arrive. raw PCM bytes for each chunk as they arrive.
@@ -177,21 +270,63 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No @@ -177,21 +291,63 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
_DONE = object() _DONE = object()
def producer(): def producer():
@ -179,7 +200,7 @@ index 2199e14..d10065f 100644
ref_text=voice_cfg.get("ref_text", ""), ref_text=voice_cfg.get("ref_text", ""),
chunk_size=voice_cfg.get("chunk_size", 12), chunk_size=voice_cfg.get("chunk_size", 12),
- non_streaming_mode=False, - non_streaming_mode=False,
+ instruct=voice_cfg.get("instruct"), + instruct=_merge_instruct(voice_cfg, instruct),
+ voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model), + voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
+ non_streaming_mode=True, + non_streaming_mode=True,
+ temperature=voice_cfg.get("temperature", 0.8), + temperature=voice_cfg.get("temperature", 0.8),
@ -207,7 +228,7 @@ index 2199e14..d10065f 100644
thread = threading.Thread(target=producer, daemon=True) thread = threading.Thread(target=producer, daemon=True)
thread.start() thread.start()
@@ -203,7 +338,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No @@ -203,7 +359,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
break break
if isinstance(item, Exception): if isinstance(item, Exception):
raise item raise item
@ -216,7 +237,7 @@ index 2199e14..d10065f 100644
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -220,7 +355,8 @@ async def health(): @@ -220,7 +376,8 @@ async def health():
async def create_speech(req: SpeechRequest): async def create_speech(req: SpeechRequest):
if tts_model is None: if tts_model is None:
raise HTTPException(status_code=503, detail="Model not loaded") raise HTTPException(status_code=503, detail="Model not loaded")
@ -226,7 +247,7 @@ index 2199e14..d10065f 100644
raise HTTPException(status_code=400, detail="'input' text is empty") raise HTTPException(status_code=400, detail="'input' text is empty")
voice_cfg = resolve_voice(req.voice) voice_cfg = resolve_voice(req.voice)
@@ -230,36 +366,77 @@ async def create_speech(req: SpeechRequest): @@ -230,36 +387,77 @@ async def create_speech(req: SpeechRequest):
"wav": "audio/wav", "wav": "audio/wav",
"pcm": "audio/pcm", "pcm": "audio/pcm",
"mp3": "audio/mpeg", "mp3": "audio/mpeg",
@ -255,7 +276,7 @@ index 2199e14..d10065f 100644
- ref_audio=voice_cfg["ref_audio"], - ref_audio=voice_cfg["ref_audio"],
+ ref_audio=voice_cfg.get("ref_audio"), + ref_audio=voice_cfg.get("ref_audio"),
ref_text=voice_cfg.get("ref_text", ""), ref_text=voice_cfg.get("ref_text", ""),
+ instruct=voice_cfg.get("instruct"), + instruct=_merge_instruct(voice_cfg, req.instruct),
+ voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model), + voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model),
+ non_streaming_mode=True, + non_streaming_mode=True,
+ temperature=voice_cfg.get("temperature", 0.8), + temperature=voice_cfg.get("temperature", 0.8),
@ -305,11 +326,11 @@ index 2199e14..d10065f 100644
if fmt == "wav": if fmt == "wav":
yield _wav_header(SAMPLE_RATE) # stream with unknown data length yield _wav_header(SAMPLE_RATE) # stream with unknown data length
- async for raw_chunk in _stream_chunks(voice_cfg, req.input): - async for raw_chunk in _stream_chunks(voice_cfg, req.input):
+ async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed): + async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed, req.instruct):
yield raw_chunk yield raw_chunk
return StreamingResponse(audio_stream(), media_type=content_type) return StreamingResponse(audio_stream(), media_type=content_type)
@@ -306,16 +483,20 @@ def _parse_args(): @@ -306,16 +504,20 @@ def _parse_args():
p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)") p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)")
p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)") p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)")
p.add_argument("--device", default="cuda", help="Torch device (default: cuda)") p.add_argument("--device", default="cuda", help="Torch device (default: cuda)")
@ -331,7 +352,7 @@ index 2199e14..d10065f 100644
with open(args.voices) as f: with open(args.voices) as f:
voices = json.load(f) voices = json.load(f)
default_voice = next(iter(voices)) default_voice = next(iter(voices))
@@ -344,6 +525,7 @@ def main(): @@ -344,6 +546,7 @@ def main():
args.model, args.model,
device=args.device, device=args.device,
dtype=torch.bfloat16, dtype=torch.bfloat16,