fix: prevent voice drift and gender changes on long paragraphs (VoiceClone)

The VoiceClone server was using non_streaming_mode=False, a mode designed
for streaming LLM->TTS pipelines. In that mode only one text token enters
the model's KV cache during prefill; the rest feed via trailing_text_hiddens
at one step per codec frame. For a 54-word paragraph this provides only ~4s
of text guidance for ~18s of speech — 77% generated with no text conditioning.
Without text context the model free-runs and drifts, sometimes changing gender.

Fix: switch to non_streaming_mode=True (already the default for VoiceDesign
and CustomVoice) so the full text is in the prefill throughout generation.
Also lower default temperature 0.9->0.8 and add top_p=0.9 to reduce
accumulated sampling noise over long runs. Temperature, top_k, and top_p
are now configurable per voice in voices.json.

- patches/openai_server.patch: updated for new upstream HEAD; both streaming
  (WAV/PCM) and non-streaming (MP3) paths now use non_streaming_mode=True
- config/run_server.py: align warmup call to non_streaming_mode=True
- README.md: bump image tags v4->v5, add changelog section
- Version: v5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

README.md

@@ -8,16 +8,16 @@ This repo packages the DGX Spark fixes plus four OpenAI-compatible TTS backends:
 
 | Backend | Port | Image | Voice source |
 |---|---:|---|---|
-| VoiceClone | `8020` | `martinb78/faster-qwen3-tts-dgx-spark:v4` | Reference audio plus transcript |
-| VoiceDesign | `8021` | `martinb78/faster-qwen3-tts-dgx-spark:v4` | Text prompt describes the voice; no reference needed |
-| CustomVoice | `8022` | `martinb78/faster-qwen3-tts-dgx-spark:v4` | Separate CustomVoice model variant |
+| VoiceClone | `8020` | `martinb78/faster-qwen3-tts-dgx-spark:v5` | Reference audio plus transcript |
+| VoiceDesign | `8021` | `martinb78/faster-qwen3-tts-dgx-spark:v5` | Text prompt describes the voice; no reference needed |
+| CustomVoice | `8022` | `martinb78/faster-qwen3-tts-dgx-spark:v5` | Separate CustomVoice model variant |
 | Streaming | `8023` | `martinb78/qwen3-tts-streaming-dgx-spark:latest` | Same voices as `8020`, but streams WAV chunks while generating |
 
 All four backends expose the OpenAI `/v1/audio/speech` contract and work with **OpenWebUI**, **SillyTavern**, **llama-swap**, `curl`, or any OpenAI-compatible client.
 
 Both Docker images are published and publicly available:
 
-- `martinb78/faster-qwen3-tts-dgx-spark:v4` - used by VoiceClone, VoiceDesign, and CustomVoice.
+- `martinb78/faster-qwen3-tts-dgx-spark:v5` - used by VoiceClone, VoiceDesign, and CustomVoice.
 - `martinb78/qwen3-tts-streaming-dgx-spark:latest` - used by the streaming service.
 
 ## What this solves
@@ -336,6 +336,33 @@ The first request after container startup can be slower because CUDA graph captu
 - Docker plus NVIDIA Container Toolkit. Make sure you have configured the runtime: `sudo nvidia-ctk runtime configure --runtime=docker` and restarted the Docker daemon.
 - Local Qwen3-TTS model weights from Hugging Face.
 
+## Changelog
+
+### v5 — 2026-05-30
+**Fix: voice drifts and gender changes on long paragraphs (VoiceClone)**
+
+The VoiceClone server was using `non_streaming_mode=False`, a mode designed for streaming LLM→TTS pipelines where the text arrives token by token. In this mode only **one text token** enters the model's KV cache during prefill; the rest are fed one-per-codec-step via a `trailing_text_hiddens` tensor. For a typical 54-word paragraph that tensor holds ~49 steps (~4 seconds of guidance) while the actual speech takes ~18 seconds — leaving **77 % of the audio generated with no text conditioning at all**. The model free-runs for that portion and drifts away from the reference voice, sometimes changing gender entirely.
+
+The fix is to use `non_streaming_mode=True` (already the default for VoiceDesign and CustomVoice), which puts the full text in the prefill so the model can attend to it throughout generation. Temperature was also lowered from 0.9 to 0.8 and nucleus sampling (`top_p=0.9`) added to reduce accumulated stochasticity over long runs. All three parameters are now per-voice configurable in `voices.json`.
+
+Changes:
+- `patches/openai_server.patch` updated: VoiceClone streaming and MP3 paths now use `non_streaming_mode=True`
+- `config/run_server.py`: warmup call aligned to `non_streaming_mode=True`
+- Temperature default 0.9 → 0.8; `top_p=0.9` added; both overridable per voice via `voices.json`
+
+### v4 — 2026-05-24
+- Add async model loading and CUDA warmup for VoiceDesign and CustomVoice servers.
+- Replace test beep with real William and Natasha voice samples.
+
+### v3 — earlier
+- Add streaming TTS backend (port 8023).
+- Add CustomVoice server, benchmark tool, and VoiceDesign API improvements.
+- Add multi-source voice pipeline with VoiceDesign support.
+
+### v2 — earlier
+- Reduce latency: CUDA warmup, chunk_size=4, max-seq-len 2048.
+- Initial DGX Spark (GB10 / ARM64 / CUDA 13) packaging.
+
 ## Credits
 
 - [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) by Andres Marafioti.

config/run_server.py

@@ -54,6 +54,7 @@ def _do_warmup():
             ref_audio=ref_audio,
             ref_text=voice_cfg.get("ref_text", ""),
             chunk_size=12,
+            non_streaming_mode=True,
         ):
             pass
         logger.info("CUDA warmup complete — server ready.")

patches/openai_server.patch

@@ -1,20 +1,27 @@
 diff --git a/examples/openai_server.py b/examples/openai_server.py
-index 2199e14..d38a684 100644
+index 61047ea..2b0c8bb 100644
 --- a/examples/openai_server.py
 +++ b/examples/openai_server.py
-@@ -306,6 +306,7 @@ def _parse_args():
-     p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)")
-     p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)")
-     p.add_argument("--device", default="cuda", help="Torch device (default: cuda)")
-+    p.add_argument("--max-seq-len", type=int, default=4096, help="Max sequence length for CUDA graph static cache (default: 4096)")
-     return p.parse_args()
- 
- 
-@@ -344,6 +345,7 @@ def main():
-         args.model,
-         device=args.device,
-         dtype=torch.bfloat16,
-+        max_seq_len=args.max_seq_len,
+@@ -187,7 +187,10 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
+                     ref_text=voice_cfg.get("ref_text", ""),
+                     chunk_size=voice_cfg.get("chunk_size", 12),
+                     instruct=voice_cfg.get("instruct"),
+-                    non_streaming_mode=False,
++                    non_streaming_mode=True,
++                    temperature=voice_cfg.get("temperature", 0.8),
++                    top_k=voice_cfg.get("top_k", 50),
++                    top_p=voice_cfg.get("top_p", 0.9),
+                 ):
+                     q.put(chunk)
+         except Exception as exc:
+@@ -252,6 +255,10 @@ async def create_speech(req: SpeechRequest):
+                     ref_audio=voice_cfg["ref_audio"],
+                     ref_text=voice_cfg.get("ref_text", ""),
+                     instruct=voice_cfg.get("instruct"),
++                    non_streaming_mode=True,
++                    temperature=voice_cfg.get("temperature", 0.8),
++                    top_k=voice_cfg.get("top_k", 50),
++                    top_p=voice_cfg.get("top_p", 0.9),
                  )
-     SAMPLE_RATE = tts_model.sample_rate
-     logger.info("Model ready. Sample rate: %d Hz", SAMPLE_RATE)
+ 
+         audio_arrays, sr = await loop.run_in_executor(None, _generate)