docs: changelog for v6.15.1

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

.gitignore

@@ -55,3 +55,7 @@ venv/
 # Environment files
 .env
 .env.*
+
+# Local QA only (not published)
+qa.sh
+tests/

DOCKERHUB_STREAMING.md

@@ -30,10 +30,23 @@ docker run -d \
       --model /models/Qwen3-TTS
       --voices /config/voices.json
       --port 8000
-      --max-seq-len 4096
+      --max-seq-len 2048
   "
 ```
 
+Download the 1.7B Base weights before starting the container:
+
+```bash
+hf download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-Base
+```
+
+Streaming must use the same model size as the one voices.json was built for (1.7B-Base), otherwise requests fail with a 1024 vs 2048 tensor size mismatch.
+To run the 0.6B-Base model for lower latency next to a 1.7B service on the same `voices.json`, set `EMBED_TAG=0.6b` on the streaming service (and `EMBED_TAG=1.7b` on the 1.7B one) and mount `/voices` read-write; each model then keeps its own `<name>.<tag>.pt` beside the WAV.
+
+For an existing stack, change only the streaming service's model bind mount.
+`--model /models/Qwen3-TTS` loads that directory; `QWEN_TTS_MODEL` does not
+override this explicit argument. Use Base for reference-audio voice cloning.
+
 Check it's running:
 
 ```bash

Dockerfile

@@ -24,8 +24,8 @@ RUN git clone https://github.com/andimarafioti/faster-qwen3-tts.git /app && \
     cd /app && git checkout --detach ${FASTER_QWEN3_TTS_REF}
 
 # Apply DGX Spark patches (non_streaming_mode=True, per-voice temperature/top_k/top_p)
-COPY patches/openai_server.patch /tmp/
-RUN cd /app && git apply /tmp/openai_server.patch
+COPY patches/openai_server.patch patches/predictor_compile.patch /tmp/
+RUN cd /app && git apply /tmp/openai_server.patch && git apply /tmp/predictor_compile.patch
 
 # Create virtual environment (Ubuntu 24.04 enforces PEP 668)
 ENV VIRTUAL_ENV=/opt/venv

README.md

@@ -19,8 +19,8 @@ One Docker image covers all four backends, with two semantic tag aliases:
 
 | Tag | Use for |
 |---|---|
-| `:latest` / `:v6` / `:v6.12` | VoiceClone, VoiceDesign, CustomVoice |
-| `:latest-streaming` / `:v6-streaming` / `:v6.12-streaming` | Streaming VoiceClone |
+| `:latest` / `:v6` / `:v6.15.0` | VoiceClone, VoiceDesign, CustomVoice |
+| `:latest-streaming` / `:v6-streaming` / `:v6.15.0-streaming` | Streaming VoiceClone |
 
 Both tags point to the same image — the `-streaming` suffix is a semantic convention so compose files and version pins are unambiguous.
 
@@ -354,7 +354,10 @@ The first request after container startup can be slower because CUDA graph captu
 
 - The 1.7B Qwen3-TTS models use about 6 GB of GPU memory each in bfloat16.
 - The forum playbook shows the four API containers running together on DGX Spark with low visible memory pressure, but exact usage depends on model size, sequence length, and warmup state.
-- Use the 0.6B Qwen3-TTS variants if you want a lighter multi-service setup.
+- The 0.6B Qwen3-TTS variants are lighter, but their speaker embeddings (1024-dim) are incompatible with the 1.7B ones (2048-dim). Set `EMBED_TAG=0.6b` (and `EMBED_TAG=1.7b` on the 1.7B services) and mount `/voices` read-write: each voice then gets `<name>.<tag>.pt` beside its WAV and both sizes can share one `voices.json` (see `docker/docker-compose.portainer-0.6b.yml`).
+- Measured on GB10 with other GPU services running (streaming PCM, first request excluded): 0.6B-Base TTFA ~570 ms, RTF 0.53–0.60 (1.7–1.9× real-time); 1.7B-Base TTFA ~820 ms, RTF 0.78–0.84 (1.2–1.3× real-time). The first request after a start is much slower while graphs/embeddings are built.
+- Optional: `TTS_COMPILE_PREDICTOR=1` (image v6.15.0+) fuses the code predictor for ~10 % more throughput; see the v6.15.0 changelog entry.
+- Throughput is not limited by memory pressure or GPU clocks (re-measured with the other large GPU model stopped: identical). The streaming process runs at ~100 % of one CPU core at ~32 W GPU power, so per-step launch/Python overhead is the suspected limit (not profiled yet). Lowering latency in practice: use the 0.6B stream with `response_format: "pcm"` and send one or two sentences per request.
 - `--max-seq-len 2048` handles most sentence-style TTS requests. Long-form narration may need `4096`, with more memory required.
 - Pin services to different GPUs with `NVIDIA_VISIBLE_DEVICES=0`, `NVIDIA_VISIBLE_DEVICES=1`, and so on if your system has more than one GPU.
 
@@ -378,6 +381,61 @@ The first request after container startup can be slower because CUDA graph captu
 
 ## Changelog
 
+### v6.15.1 — 2026-10-12
+
+- **Docs:** German sample files (10 voices, 0.6B-Base streaming, same text) in `samples/german sample files/` with an `index.html` that has an audio player per voice (open locally or via GitHub Pages) and a link table, added for the German-pronunciation question in the forum.
+- **Docs:** image tag table now lists `v6.15.0`; the `v6.15.0` images and the `:v6`/`:latest` aliases are on Docker Hub.
+- **Image:** unchanged (`:v6.15.0`), no rebuild needed.
+- **Rollback:** `git checkout v6.15.0` (documentation only).
+
+### v6.15.0 — 2026-10-11
+
+- **Perf (opt-in):** `TTS_COMPILE_PREDICTOR=1` compiles the 15-step code-predictor loop with `torch.compile` (Inductor/Triton) before CUDA-graph capture. Profiling showed the predictor needs ~5350 kernels per frame (about 54 % of the decode time); fusing them cuts its replay from ~23 ms to ~17.5 ms. Measured through the API on GB10 with the 0.6B-Base streaming server (same voice and text, other TTS containers stopped): 1.6–1.7x -> 1.8x real-time on long text, first audio ~670 ms -> ~550 ms. Listening comparison of 5 voices: no audible difference. Costs about one extra minute at startup. Default is off (unchanged behaviour).
+- **Image:** rebuilt as `v6.15.0` / `v6.15.0-streaming` (adds `patches/predictor_compile.patch`) and pushed to Docker Hub together with the `:v6`, `:latest`, `:v6-streaming` and `:latest-streaming` aliases.
+- **Investigation notes:** memory pressure/swap, GPU clocks, CPU core type and `chunk_size` do not limit throughput; `float16` fails (inf/NaN logits, device-side assert); flash-attn is irrelevant for single-token decode. F5-TTS, EmoKnob and Breeze-TTS-2 were benchmarked against Qwen3-TTS: slower than real-time (Breeze 0.5x, EmoKnob 0.1x) or without streaming/German (F5 ~5x) and not usable quality-wise in our test.
+- **Rollback:** unset `TTS_COMPILE_PREDICTOR` (or use the `:v6.13.1` image).
+
+### v6.14.1 — 2026-10-11
+
+- **Docs:** measured GB10 benchmark numbers for 0.6B vs 1.7B, the finding that memory pressure and GPU clocks are not the bottleneck, and the corrected 0.6B guidance (`EMBED_TAG`) in "Performance and memory notes".
+- **Deploy note:** Portainer re-creates services from its stored stack file, so a `docker compose up` from the CLI is overwritten. Update the stack in the Portainer editor instead.
+- **Image:** unchanged (`:v6.13.1`), no rebuild needed.
+- **Rollback:** `git checkout v6.14.0` (documentation only).
+
+### v6.14.0 — 2026-10-11
+
+- **Feature:** `EMBED_TAG` environment variable (e.g. `1.7b`, `0.6b`). Speaker embeddings are stored beside the reference audio as `<name>.<tag>.pt`, so a 1.7B VoiceClone and a 0.6B Streaming backend can share one `voices.json` and one voice folder (mount `/voices` read-write). Existing `config/speakers/*.pt` files are adopted once for the `1.7b` tag; embeddings older than their WAV/TXT are regenerated.
+- **Feature:** `/voices` and `/v1/voices` discovery endpoints on all backends.
+- **Config:** new `docker/docker-compose.portainer-0.6b.yml` — 1.7B VoiceClone on :8020 plus 0.6B-Base Streaming on :8023 (`--max-seq-len 2048`). The 0.6B model needs its own embeddings (1024-dim); never share `.pt` files between model sizes.
+- **Image:** unchanged (`:v6.13.1`); the changes live in `/config` (bind mount), no rebuild needed.
+- **Rollback:** redeploy the previous compose file (`:v6.13.1` images); `EMBED_TAG` unset keeps the old `/config/speakers` behaviour.
+
+### v6.13.1 — 2026-10-04
+
+- **Fix:** Streaming must use the same model size as the one voices.json was built for (1.7B-Base), otherwise requests fail with a 1024 vs 2048 tensor size mismatch. Streaming compose examples now default to the 1.7B-Base model.
+- **Fix:** Out-of-range `chunk_size` values are clamped to 2–24 (e.g. -5/0/1 -> 2, 1000 -> 24) and still return 200; only non-integers return 422. Covered by an offline check of the patched `SpeechRequest` + `_clamp_chunk_size`.
+
+### v6.13 — 2026-10-04
+**Feature: Per-Request `chunk_size` for Streaming**
+- `/v1/audio/speech` accepts an optional integer `chunk_size` (codec frames per streamed chunk, 12 = 1 s), clamped to 2–24. It applies to the streaming wav/pcm path only; omitted keeps the voice's `voices.json` value (default unchanged).
+- Use case: a voice bot can request a small chunk for the first sentence of a reply (faster first audio) and the efficient default for the rest. Measured on GB10 (2.6 s sentence, shared GPU): chunk 4 → first audio 0.60 s, RTF 1.61, underrun 1.28 s; chunk 12 → 1.24 s, RTF 1.25, underrun 0.16 s.
+
+**Perf: Fixed-Window CUDA-Graph Codec Decode for Streaming**
+- **Root Cause:** upstream re-decoded the whole reference prompt (~210 frames) plus all generated frames for the first chunks of every request (~215 ms per decode, 3× per request), and decoded eagerly afterwards. The streaming path now always decodes the last 25 + `chunk_size` frames (reference codes are the left context at the start) through one captured CUDA graph per window size (~18–25 ms per chunk). Token generation is unchanged; the 25-frame context is the same one upstream uses after its first chunks, no extra boundary clicks were measured.
+- Measured on GB10 (voice `DE_Jarvis_2`, median of 3, no other GPU load): 8.4 s sentence chunk 12 → first audio 1.14 → 1.01 s, RTF 1.00 → 0.95, underrun 0.23 → 0.00 s; chunk 4 → first audio 0.58 → 0.44 s, RTF 1.15 → 0.99, underrun 1.02 → 0.00 s. With the LLM generating at the same time, chunk 4 underrun 5.86 → 3.90 s (the GPU is time-sliced, RTF stays 1.5–1.9).
+- Small chunks are now cheap: `chunk_size: 4` gives ~0.4 s first audio without stutter when the GPU is not shared.
+- Kill switch: env `TTS_GRAPH_DECODE=0` restores the upstream decode path.
+
+**Feature: `prebuffer_ms` for Streaming**
+- `/v1/audio/speech` accepts optional `prebuffer_ms` (0–10000): the server holds back that much audio before sending, so a client that plays on arrival rides out a slow start on a shared GPU. Default 0 (unchanged). Measured with LLM load, 2.6 s sentence, chunk 4: `prebuffer_ms: 1000` → underrun 1.08 → 0.00 s, first audio 1.45 → 2.44 s.
+
+**Fix: Client Disconnect Stops Generation**
+- When a streaming client hangs up (e.g. voice-bot barge-in), generation stops after the current chunk and frees the GPU lock. Previously the whole text was still generated and blocked the next request. Measured: next request's first audio 0.74 s after hanging up on a ~25 s text.
+- `config/run_server.py` warm-up now also captures the chunk-12 decode graph.
+
+**Recommended: `--max-seq-len 2048` for Streaming**
+- Run the streaming container with `--max-seq-len 2048` (saves ~7 ms per decode step versus larger values). All compose files in `docker/` already use it.
+
 ### v6.12 — 2026-10-03
 **Fix: Streaming (8023) Slower Than Realtime**
 - **Root Cause:** `generate_voices.py` wrote `chunk_size: 4` (~0.33 s audio) into every voice. Each streamed chunk runs a codec decode over 25 frames of left context plus a GPU sync, so a 4-frame chunk spent most of its time re-decoding context. Measured on GB10: RTF 1.13–1.49 with 1.0–1.9 s of playback underruns per 8.4 s sentence.

config/run_customvoice_server.py

@@ -321,6 +321,12 @@ async def list_audio_models():
     return _models_response
 
 
+@app.get("/voices")
+@app.get("/v1/voices")
+async def list_voices():
+    return _models_response
+
+
 @app.get("/speakers")
 async def get_speakers():
     return list(voices.keys())

config/run_server.py

@@ -32,6 +32,46 @@ import openai_server
 logger = logging.getLogger(__name__)
 
 
+# Speaker embeddings (.pt) are model-size specific (1.7B=2048-dim, 0.6B=1024-dim).
+# With EMBED_TAG set (e.g. "1.7b"/"0.6b") every voice's embedding lives next to its
+# reference audio as <name>.<tag>.pt, so several model sizes can share one voices.json.
+_EMBED_TAG = os.environ.get("EMBED_TAG", "").strip()
+
+
+def _retarget_embeddings(voice_cfg: dict) -> dict:
+    """Point voice_cfg at the per-model .pt beside the reference audio (idempotent)."""
+    ref_audio = voice_cfg.get("ref_audio")
+    if not _EMBED_TAG or not ref_audio or voice_cfg.get("_emb_tag") == _EMBED_TAG:
+        return voice_cfg
+    stem = os.path.splitext(ref_audio)[0]
+    pt = f"{stem}.{_EMBED_TAG}.pt"
+    legacy = voice_cfg.get("speaker_embeddings") or ""
+    try:
+        # ponytail: legacy /config/speakers/*.pt are 1.7B embeddings; adopt them once for the 1.7b tag only.
+        if _EMBED_TAG.lower() == "1.7b" and not os.path.exists(pt) and os.path.isfile(legacy):
+            import shutil
+            shutil.copy2(legacy, pt)
+        if os.path.exists(pt):
+            sources = [ref_audio, stem + ".txt", stem + ".reference.txt"]
+            newest = max(os.path.getmtime(f) for f in sources if os.path.exists(f))
+            if newest > os.path.getmtime(pt) + 1:  # reference changed -> embedding stale
+                os.remove(pt)
+    except OSError as exc:
+        logger.warning("Embedding retarget problem for %s: %s", pt, exc)
+    voice_cfg["speaker_embeddings"] = pt
+    voice_cfg["_emb_tag"] = _EMBED_TAG
+    return voice_cfg
+
+
+if _EMBED_TAG:
+    _orig_resolve_voice = openai_server.resolve_voice
+
+    def _resolve_voice_tagged(voice_name: str) -> dict:
+        return _retarget_embeddings(_orig_resolve_voice(voice_name))
+
+    openai_server.resolve_voice = _resolve_voice_tagged
+
+
 def _do_warmup():
     """Run one short generation to compile CUDA graphs before serving requests."""
     model = openai_server.tts_model
@@ -50,14 +90,19 @@ def _do_warmup():
 
     logger.info("Warming up CUDA graphs (first request will be fast)...")
     try:
-        for _ in model.generate_voice_clone_streaming(
+        kwargs = dict(
             text="Warmup.",
             language=voice_cfg.get("language", "Auto"),
             ref_audio=ref_audio,
             ref_text=voice_cfg.get("ref_text", ""),
-            chunk_size=12,
             non_streaming_mode=True,
-        ):
+        )
+        # Patched server: also captures the codec decode graph for chunk 12.
+        if getattr(openai_server, "_GRAPH_DECODE", False):
+            stream = openai_server._voice_clone_stream(12, **kwargs)
+        else:
+            stream = model.generate_voice_clone_streaming(chunk_size=12, **kwargs)
+        for _ in stream:
             pass
         logger.info("CUDA warmup complete — server ready.")
     except Exception as exc:
@@ -72,6 +117,7 @@ def _precompute_all_embeddings():
         return
         
     for voice_name, voice_cfg in list(voices.items()):
+        _retarget_embeddings(voice_cfg)
         # skip if already precomputed
         spk_emb_path = voice_cfg.get("speaker_embeddings") or voice_cfg.get("speaker embeddings")
         if spk_emb_path and os.path.isfile(spk_emb_path):
@@ -127,6 +173,11 @@ async def list_audio_voices():
 async def list_audio_models():
     return _models_response
 
+@openai_server.app.get('/voices')
+@openai_server.app.get('/v1/voices')
+async def list_voices():
+    return _models_response
+
 # SillyTavern speaker endpoint
 @openai_server.app.get('/speakers')
 async def get_speakers():

config/run_voicedesign_server.py

@@ -392,6 +392,12 @@ async def list_audio_models():
     _reload_voices_if_changed()
     return _models_response
 
+@app.get("/voices")
+@app.get("/v1/voices")
+async def list_voices():
+    _reload_voices_if_changed()
+    return _models_response
+
 @app.get("/speakers")
 async def get_speakers():
     _reload_voices_if_changed()

docker/docker-compose.portainer-0.6b.yml

@@ -0,0 +1,161 @@
+# ─────────────────────────────────────────────────────────────────────────────
+#  Qwen3-TTS GPU Stack
+#
+#  Ports:
+#    8020 -> Voice clone, OpenAI-compatible /v1/audio/speech 
+#    8021 -> VoiceDesign model for prompt-designed voices
+#    8022 -> CustomVoice model
+#    8023 -> Low-latency streaming voice clone endpoint 
+#
+#  Container Name	            RAM Usage	GPU VRAM Usage
+#  faster-qwen3-tts-voiceclone	~2.72 GiB	5327 MiB (~5.2 GiB)
+#  faster-qwen3-tts-customvoice	~2.88 GiB	4683 MiB (~4.6 GiB)
+#  faster-qwen3-tts-voicedesign	~2.90 GiB	4747 MiB (~4.6 GiB)
+#  faster-qwen3-tts-streaming	~2.54 GiB	5659 MiB (~5.5 GiB)
+#
+#  Voice files:
+#    The voice clone and streaming services scan active voices from:
+#    home/sparky/Media/_Sounds/TTS_Voices/active_voices
+#
+#  Portainer note:
+#    Set HF_TOKEN in the stack environment if the models ever need Hugging Face access.
+# ─────────────────────────────────────────────────────────────────────────────
+
+services:
+  # Standard voice cloning backend used by TTS Voice Creator and other OpenAI-compatible clients.
+  faster-qwen3-tts-voiceclone:
+    image: martinb78/faster-qwen3-tts-dgx-spark:latest
+    container_name: tts-faster-qwen3-tts-voiceclone
+    restart: unless-stopped
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=all
+      - HF_TOKEN=${HF_TOKEN}
+      - EMBED_TAG=1.7b
+    ports:
+      - "8020:8000"
+    volumes:
+      - /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-1.7B-Base:/models/Qwen3-TTS:ro
+      - /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
+      - /home/sparky/Media/_Sounds/TTS_Voices/active_voices:/voices:rw
+    command: >
+      /bin/bash -c "
+      python3 /config/generate_voices.py &&
+      python3 /config/run_server.py
+      --model /models/Qwen3-TTS
+      --voices /config/voices.json
+      --port 8000
+      --max-seq-len 4096
+      "
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    networks:
+      - dgx_net
+
+  # VoiceDesign backend: creates speech from a descriptive voice prompt instead of a reference clip.
+  faster-qwen3-tts-voicedesign:
+    image: martinb78/faster-qwen3-tts-dgx-spark:latest
+    container_name: tts-faster-qwen3-tts-voicedesign
+    restart: unless-stopped
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=all
+      - HF_TOKEN=${HF_TOKEN}
+    ports:
+      - "8021:8000"
+    volumes:
+      - /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-1.7B-VoiceDesign:/models/Qwen3-TTS-VoiceDesign:ro
+      - /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
+    command: >
+      /bin/bash -c "
+      python3 /config/run_voicedesign_server.py
+      --model /models/Qwen3-TTS-VoiceDesign
+      --voices /config/voicedesign_voices.json
+      --port 8000
+      --max-seq-len 2048
+      "
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    networks:
+      - dgx_net
+
+  # CustomVoice backend: keeps the separate CustomVoice model available on port 8022.
+  faster-qwen3-tts-customvoice:
+    image: martinb78/faster-qwen3-tts-dgx-spark:latest
+    container_name: tts-faster-qwen3-tts-customvoice
+    restart: unless-stopped
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=all
+      - HF_TOKEN=${HF_TOKEN}
+    ports:
+      - "8022:8000"
+    volumes:
+      - /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-1.7B-CustomVoice:/models/Qwen3-TTS-CustomVoice:ro
+      - /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
+    command: >
+      /bin/bash -c "
+      python3 /config/run_customvoice_server.py
+      --model /models/Qwen3-TTS-CustomVoice
+      --voices /config/customvoice_voices.json
+      --port 8000
+      --max-seq-len 2048
+      "
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    networks:
+      - dgx_net
+
+  # Streaming voice clone backend: same active voices as 8020, but streams WAV chunks while generating.
+  faster-qwen3-tts-streaming:
+    image: martinb78/faster-qwen3-tts-dgx-spark:latest-streaming
+    container_name: tts-faster-qwen3-tts-streaming
+    restart: unless-stopped
+    runtime: nvidia
+    environment:
+      - NVIDIA_VISIBLE_DEVICES=all
+      - HF_TOKEN=${HF_TOKEN}
+      - EMBED_TAG=0.6b
+    ports:
+      - "8023:8000"
+    volumes:
+      - /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-0.6B-Base:/models/Qwen3-TTS:ro
+      - /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
+      - /home/sparky/Media/_Sounds/TTS_Voices/active_voices:/voices:rw
+    command: >
+      /bin/bash -c "
+      python3 /config/generate_voices.py &&
+      python3 /config/run_server.py
+      --model /models/Qwen3-TTS
+      --voices /config/voices.json
+      --port 8000
+      --max-seq-len 2048
+      "
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    networks:
+      - dgx_net
+
+networks:
+  dgx_net:
+    external: true

docker/docker-compose.yml

@@ -139,7 +139,7 @@ services:
       --model /models/Qwen3-TTS
       --voices /config/voices.json
       --port 8000
-      --max-seq-len 4096
+      --max-seq-len 2048
       "
     deploy:
       resources:

patches/openai_server.patch

@@ -1,5 +1,5 @@
 diff --git a/examples/openai_server.py b/examples/openai_server.py
-index 2199e14..ca5eaf9 100644
+index 2199e14..3f3a4e6 100644
 --- a/examples/openai_server.py
 +++ b/examples/openai_server.py
 @@ -36,11 +36,13 @@ API usage:
@@ -84,7 +84,7 @@ index 2199e14..ca5eaf9 100644
  
  # ---------------------------------------------------------------------------
  # Request / response models
-@@ -79,8 +139,60 @@ class SpeechRequest(BaseModel):
+@@ -79,8 +139,68 @@ class SpeechRequest(BaseModel):
      model: str = "tts-1"
      input: str
      voice: str = "alloy"
@@ -93,6 +93,14 @@ index 2199e14..ca5eaf9 100644
 +    response_format: str = "wav"  # wav | pcm | mp3 | zip
 +    speed: float = 1.0           # scales audio tempo
 +    instruct: Optional[str] = None  # per-line delivery direction, see _merge_instruct
++    # Streaming only: codec frames per streamed chunk (12 = 1 s). Small values
++    # cut time-to-first-audio but cost throughput; clamped to 2..24.
++    # None keeps the voice's voices.json chunk_size.
++    chunk_size: Optional[int] = None
++    # Streaming only: hold back the first N ms of audio before sending, so a
++    # client that plays on arrival rides out a slower-than-realtime start
++    # (shared GPU) instead of stuttering. 0/None = send immediately. Max 10 s.
++    prebuffer_ms: Optional[int] = None
 +
 +
 +# ---------------------------------------------------------------------------
@@ -147,7 +155,7 @@ index 2199e14..ca5eaf9 100644
  
  
  # ---------------------------------------------------------------------------
-@@ -144,9 +256,39 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
+@@ -144,9 +264,39 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
  
  
  def resolve_voice(voice_name: str) -> dict:
@@ -188,7 +196,7 @@ index 2199e14..ca5eaf9 100644
      if default_voice and default_voice in voices:
          logger.warning(
              "Voice %r not configured; falling back to default voice %r",
-@@ -168,7 +310,47 @@ def resolve_voice(voice_name: str) -> dict:
+@@ -168,42 +318,214 @@ def resolve_voice(voice_name: str) -> dict:
  # ---------------------------------------------------------------------------
  
  
@@ -233,12 +241,81 @@ index 2199e14..ca5eaf9 100644
 +        return None
 +
 +
-+async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float, instruct: Optional[str] = None) -> AsyncGenerator[bytes, None]:
++def _clamp_chunk_size(n: int) -> int:
++    return max(2, min(24, n))
++
++
++# Fixed-window codec decode. Upstream re-decodes the whole reference prompt plus
++# every generated frame for the first chunks (~250 frames on long references)
++# and decodes eagerly (hundreds of small kernel launches). The 12 Hz codec
++# decoder is causal, so decoding the last (context + chunk) frames and keeping
++# the tail is exact; with the reference codes as left context the window is
++# always full, so one CUDA graph per window size covers every chunk.
++_DECODE_CONTEXT = 25  # frames, same as upstream chunked_decode / streaming
++_GRAPH_DECODE = os.environ.get("TTS_GRAPH_DECODE", "1") != "0"
++_decoder_graphs: dict = {}  # window frames -> (graph, static_in, static_out)
++
++
++def _decode_window(decoder, window: torch.Tensor) -> torch.Tensor:
++    """Decode [frames, Q] codes to a waveform tensor, graph-replayed when possible."""
++    codes = window.T.unsqueeze(0)
++    entry = _decoder_graphs.get(codes.shape[-1])
++    if entry is None:
++        static_in = codes.clone()
++        side = torch.cuda.Stream()
++        side.wait_stream(torch.cuda.current_stream())
++        with torch.cuda.stream(side):
++            for _ in range(2):
++                decoder(static_in)
++        torch.cuda.current_stream().wait_stream(side)
++        graph = torch.cuda.CUDAGraph()
++        with torch.cuda.graph(graph, capture_error_mode="thread_local"):
++            static_out = decoder(static_in)
++        entry = _decoder_graphs[codes.shape[-1]] = (graph, static_in, static_out)
++    graph, static_in, static_out = entry
++    static_in.copy_(codes)
++    graph.replay()
++    return static_out
++
++
++@torch.inference_mode()
++def _voice_clone_stream(chunk_size: int, **kwargs):
++    """generate_voice_clone_streaming with fixed-window (graphed) codec decode.
++
++    Yields float32 numpy chunks. Token generation is upstream's
++    fast_generate_streaming unchanged.
++    """
++    from faster_qwen3_tts.streaming import fast_generate_streaming
++
++    gen_keys = ("temperature", "top_k", "top_p")
++    gen = {k: kwargs.pop(k) for k in gen_keys if k in kwargs}
++    m, talker, config, tie, tam, tth, tpe, ref_codes = tts_model._prepare_generation(**kwargs)
++    decoder = m.speech_tokenizer.model.decoder
++    upsample = decoder.total_upsample
++    window = _DECODE_CONTEXT + chunk_size
++    hist = ref_codes.to(tie.device)[-window:] if ref_codes is not None else None
++    for codes, _timing in fast_generate_streaming(
++        talker=talker, talker_input_embeds=tie, attention_mask=tam,
++        trailing_text_hiddens=tth, tts_pad_embed=tpe, config=config,
++        predictor_graph=tts_model.predictor_graph, talker_graph=tts_model.talker_graph,
++        chunk_size=chunk_size, **gen,
++    ):
++        hist = codes if hist is None else torch.cat([hist, codes])[-window:]
++        if hist.shape[0] == window:
++            wav = _decode_window(decoder, hist)
++        else:  # x-vector voice, first chunks: short eager decode
++            wav = decoder(hist.T.unsqueeze(0))
++        yield wav[..., -codes.shape[0] * upsample:].flatten().float().cpu().numpy()
++
++
++async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float, instruct: Optional[str] = None, chunk_size: Optional[int] = None, prebuffer_ms: Optional[int] = None) -> AsyncGenerator[bytes, None]:
      """
      Run generate_voice_clone_streaming in a background thread and yield
      raw PCM bytes for each chunk as they arrive.
-@@ -177,21 +359,65 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
+     """
+     q: queue.Queue = queue.Queue()
      _DONE = object()
++    cancelled = threading.Event()
  
      def producer():
 +        process = None
@@ -280,19 +357,27 @@ index 2199e14..ca5eaf9 100644
 +                _seed_rng(voice_cfg.get("seed", _voice_seed(voice_name)))
 +                segments = _split_text_for_generation(text, _max_chars_per_segment())
 +                for seg in segments:
-+                    for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
++                    kwargs = dict(
 +                        text=seg,
 +                        language=voice_cfg.get("language", "Auto"),
 +                        ref_audio=voice_cfg.get("ref_audio"),
 +                        ref_text=voice_cfg.get("ref_text", ""),
-+                        chunk_size=voice_cfg.get("chunk_size", 12),
 +                        instruct=_merge_instruct(voice_cfg, instruct),
 +                        voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
 +                        non_streaming_mode=True,
 +                        temperature=voice_cfg.get("temperature", 0.8),
 +                        top_k=voice_cfg.get("top_k", 50),
 +                        top_p=voice_cfg.get("top_p", 0.9),
-+                    ):
++                    )
++                    cs = _clamp_chunk_size(chunk_size) if chunk_size is not None else voice_cfg.get("chunk_size", 12)
++                    if _GRAPH_DECODE:
++                        chunks = _voice_clone_stream(cs, **kwargs)
++                    else:
++                        chunks = (c for c, _sr, _t in tts_model.generate_voice_clone_streaming(chunk_size=cs, **kwargs))
++                    for chunk in chunks:
++                        if cancelled.is_set():  # client gone: free the GPU for the next request
++                            chunks.close()
++                            return
 +                        raw = _to_pcm16(chunk)
 +                        if process:
 +                            process.stdin.write(raw)
@@ -313,16 +398,38 @@ index 2199e14..ca5eaf9 100644
  
      thread = threading.Thread(target=producer, daemon=True)
      thread.start()
-@@ -203,7 +429,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
-             break
-         if isinstance(item, Exception):
-             raise item
+ 
+     loop = asyncio.get_event_loop()
+-    while True:
+-        item = await loop.run_in_executor(None, q.get)
+-        if item is _DONE:
+-            break
+-        if isinstance(item, Exception):
+-            raise item
 -        yield _to_pcm16(item)
-+        yield item
++    hold = max(0, min(10000, prebuffer_ms or 0)) * SAMPLE_RATE * 2 // 1000  # PCM16 bytes
++    pending = b""
++    try:
++        while True:
++            item = await loop.run_in_executor(None, q.get)
++            if item is _DONE:
++                break
++            if isinstance(item, Exception):
++                raise item
++            if hold:
++                pending += item
++                if len(pending) < hold:
++                    continue
++                item, pending, hold = pending, b"", 0
++            yield item
++        if pending:
++            yield pending
++    finally:
++        cancelled.set()
  
  
  # ---------------------------------------------------------------------------
-@@ -220,7 +446,8 @@ async def health():
+@@ -220,7 +542,8 @@ async def health():
  async def create_speech(req: SpeechRequest):
      if tts_model is None:
          raise HTTPException(status_code=503, detail="Model not loaded")
@@ -332,7 +439,7 @@ index 2199e14..ca5eaf9 100644
          raise HTTPException(status_code=400, detail="'input' text is empty")
  
      voice_cfg = resolve_voice(req.voice)
-@@ -230,36 +457,84 @@ async def create_speech(req: SpeechRequest):
+@@ -230,36 +553,84 @@ async def create_speech(req: SpeechRequest):
          "wav": "audio/wav",
          "pcm": "audio/pcm",
          "mp3": "audio/mpeg",
@@ -426,11 +533,11 @@ index 2199e14..ca5eaf9 100644
          if fmt == "wav":
              yield _wav_header(SAMPLE_RATE)  # stream with unknown data length
 -        async for raw_chunk in _stream_chunks(voice_cfg, req.input):
-+        async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed, req.instruct):
++        async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed, req.instruct, req.chunk_size, req.prebuffer_ms):
              yield raw_chunk
  
      return StreamingResponse(audio_stream(), media_type=content_type)
-@@ -306,16 +581,21 @@ def _parse_args():
+@@ -306,16 +677,21 @@ def _parse_args():
      p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)")
      p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)")
      p.add_argument("--device", default="cuda", help="Torch device (default: cuda)")
@@ -453,7 +560,7 @@ index 2199e14..ca5eaf9 100644
          with open(args.voices) as f:
              voices = json.load(f)
          default_voice = next(iter(voices))
-@@ -344,6 +624,7 @@ def main():
+@@ -344,6 +720,7 @@ def main():
          args.model,
          device=args.device,
          dtype=torch.bfloat16,

patches/predictor_compile.patch

@@ -0,0 +1,25 @@
+diff --git a/faster_qwen3_tts/predictor_graph.py b/faster_qwen3_tts/predictor_graph.py
+index 38a52ca..70851eb 100644
+--- a/faster_qwen3_tts/predictor_graph.py
++++ b/faster_qwen3_tts/predictor_graph.py
+@@ -13,6 +13,8 @@ Strategy:
+ - Unroll the full 15-step loop for deterministic shapes
+ - Capture the entire loop as a single CUDA graph
+ """
++import os
++
+ import torch
+ from transformers import StaticCache
+ from transformers.masking_utils import create_causal_mask, create_sliding_window_causal_mask
+@@ -175,6 +177,11 @@ class PredictorGraph:
+         self._init_cache_layers()
+         self._build_attention_masks()
+ 
++        # Opt-in: fuse the 15-step predictor loop with Inductor before graph capture.
++        # ~17% faster streaming on GB10 (0.6B); adds ~1 min to the first warmup.
++        if os.environ.get("TTS_COMPILE_PREDICTOR") == "1":
++            self._full_loop = torch.compile(self._full_loop, dynamic=False)
++
+         for _ in range(num_warmup):
+             self.static_cache.reset()
+             self._full_loop()

samples/german sample files/README.md

@@ -0,0 +1,26 @@
+# German sample files (0.6B-Base streaming)
+
+Ten voices from the active library, rendered with the 0.6B-Base streaming server
+(`v6.15.0`, `TTS_COMPILE_PREDICTOR=1`, `language: German`, default sampling).
+All files use the same text:
+
+> Guten Morgen! Das Wetter ist heute wunderbar, deshalb gehen wir am Rhein spazieren. Anschließend essen wir Käsekuchen und besprechen die Planung für das nächste Quartal, einschließlich Budget, Meeting und Zeitplan.
+
+Files are in `mp3/` (24 kHz mono, VBR ~130 kbit/s), numbered 01-10 with the voice ID.
+
+## Listen
+
+GitHub's directory view has no audio player. Open `index.html` in this folder from a local clone (it has a player per voice), or enable GitHub Pages for the repository and open the same page there. Direct links to the files:
+
+| # | Voice | File |
+|---|---|---|
+| 01 | DE_M_Gutfolg | [01_DE_M_Gutfolg.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/01_DE_M_Gutfolg.mp3) |
+| 02 | DE_Banker | [02_DE_Banker.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/02_DE_Banker.mp3) |
+| 03 | DE_N_Orkgeist | [03_DE_N_Orkgeist.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/03_DE_N_Orkgeist.mp3) |
+| 04 | DE_F_Amala | [04_DE_F_Amala.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/04_DE_F_Amala.mp3) |
+| 05 | DE_1_0 | [05_DE_1_0.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/05_DE_1_0.mp3) |
+| 06 | DE_F_19_German_Switzerland_Female_Leni | [06_DE_F_19_German_Switzerland_Female_Leni.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/06_DE_F_19_German_Switzerland_Female_Leni.mp3) |
+| 07 | DE_M_Rabe | [07_DE_M_Rabe.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/07_DE_M_Rabe.mp3) |
+| 08 | DE_N_Schwarzhaarige | [08_DE_N_Schwarzhaarige.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/08_DE_N_Schwarzhaarige.mp3) |
+| 09 | DE_N_Magier | [09_DE_N_Magier.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/09_DE_N_Magier.mp3) |
+| 10 | DE_M_Ork | [10_DE_M_Ork.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/10_DE_M_Ork.mp3) |

samples/german sample files/index.html

@@ -0,0 +1,95 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>German samples - Qwen3-TTS 0.6B streaming</title>
+<style>
+  :root { color-scheme: light dark; --bg:#fff; --fg:#1b1f24; --muted:#59636e; --line:#d1d9e0; }
+  @media (prefers-color-scheme: dark) { :root { --bg:#0d1117; --fg:#e6edf3; --muted:#9198a1; --line:#3d444d; } }
+  body { margin:0; padding:1.5rem 1rem; background:var(--bg); color:var(--fg); font:16px/1.5 system-ui,sans-serif; }
+  main { max-width:52rem; margin:0 auto; }
+  blockquote { margin:1rem 0; padding:.5rem 1rem; border-left:4px solid var(--line); color:var(--muted); }
+  .scroll { overflow-x:auto; }
+  table { border-collapse:collapse; width:100%; }
+  th, td { text-align:left; padding:.5rem; border-bottom:1px solid var(--line); vertical-align:middle; }
+  audio { width:100%; min-width:14rem; }
+  a { color:inherit; }
+  a:focus-visible, audio:focus-visible { outline:3px solid #0969da; outline-offset:2px; }
+</style>
+</head>
+<body>
+<main>
+  <h1>German samples: Qwen3-TTS 0.6B-Base streaming</h1>
+  <p>Ten voices from the active library, rendered with the 0.6B-Base streaming server (v6.15.0, <code>language: German</code>, default sampling). Every file uses the same text:</p>
+  <blockquote lang="de">Guten Morgen! Das Wetter ist heute wunderbar, deshalb gehen wir am Rhein spazieren. Anschließend essen wir Käsekuchen und besprechen die Planung für das nächste Quartal, einschließlich Budget, Meeting und Zeitplan.</blockquote>
+  <div class="scroll">
+    <table>
+      <thead><tr><th scope="col">#</th><th scope="col">Voice</th><th scope="col">Listen</th><th scope="col">File</th></tr></thead>
+      <tbody>
+      <tr>
+        <th scope="row">01</th>
+        <td>DE_M_Gutfolg</td>
+        <td><audio controls preload="none" src="mp3/01_DE_M_Gutfolg.mp3" aria-label="Sample 01: DE_M_Gutfolg"></audio></td>
+        <td><a href="mp3/01_DE_M_Gutfolg.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">02</th>
+        <td>DE_Banker</td>
+        <td><audio controls preload="none" src="mp3/02_DE_Banker.mp3" aria-label="Sample 02: DE_Banker"></audio></td>
+        <td><a href="mp3/02_DE_Banker.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">03</th>
+        <td>DE_N_Orkgeist</td>
+        <td><audio controls preload="none" src="mp3/03_DE_N_Orkgeist.mp3" aria-label="Sample 03: DE_N_Orkgeist"></audio></td>
+        <td><a href="mp3/03_DE_N_Orkgeist.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">04</th>
+        <td>DE_F_Amala</td>
+        <td><audio controls preload="none" src="mp3/04_DE_F_Amala.mp3" aria-label="Sample 04: DE_F_Amala"></audio></td>
+        <td><a href="mp3/04_DE_F_Amala.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">05</th>
+        <td>DE_1_0</td>
+        <td><audio controls preload="none" src="mp3/05_DE_1_0.mp3" aria-label="Sample 05: DE_1_0"></audio></td>
+        <td><a href="mp3/05_DE_1_0.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">06</th>
+        <td>DE_F_19_German_Switzerland_Female_Leni</td>
+        <td><audio controls preload="none" src="mp3/06_DE_F_19_German_Switzerland_Female_Leni.mp3" aria-label="Sample 06: DE_F_19_German_Switzerland_Female_Leni"></audio></td>
+        <td><a href="mp3/06_DE_F_19_German_Switzerland_Female_Leni.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">07</th>
+        <td>DE_M_Rabe</td>
+        <td><audio controls preload="none" src="mp3/07_DE_M_Rabe.mp3" aria-label="Sample 07: DE_M_Rabe"></audio></td>
+        <td><a href="mp3/07_DE_M_Rabe.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">08</th>
+        <td>DE_N_Schwarzhaarige</td>
+        <td><audio controls preload="none" src="mp3/08_DE_N_Schwarzhaarige.mp3" aria-label="Sample 08: DE_N_Schwarzhaarige"></audio></td>
+        <td><a href="mp3/08_DE_N_Schwarzhaarige.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">09</th>
+        <td>DE_N_Magier</td>
+        <td><audio controls preload="none" src="mp3/09_DE_N_Magier.mp3" aria-label="Sample 09: DE_N_Magier"></audio></td>
+        <td><a href="mp3/09_DE_N_Magier.mp3" download>mp3</a></td>
+      </tr>
+      <tr>
+        <th scope="row">10</th>
+        <td>DE_M_Ork</td>
+        <td><audio controls preload="none" src="mp3/10_DE_M_Ork.mp3" aria-label="Sample 10: DE_M_Ork"></audio></td>
+        <td><a href="mp3/10_DE_M_Ork.mp3" download>mp3</a></td>
+      </tr>
+      </tbody>
+    </table>
+  </div>
+</main>
+</body>
+</html>