Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 94738a2c00 | |||
| c40ec9f15a | |||
| 819a56888a | |||
| 187051c67f | |||
| 60f0e73b71 | |||
| 4049c44809 | |||
| ecd833cb89 | |||
| 119b92ea15 | |||
| 4afdd04953 | |||
| 575cebc454 | |||
| 37fb8c062e |
4
.gitignore
vendored
4
.gitignore
vendored
@ -55,3 +55,7 @@ venv/
|
||||
# Environment files
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Local QA only (not published)
|
||||
qa.sh
|
||||
tests/
|
||||
|
||||
@ -20,7 +20,7 @@ docker run -d \
|
||||
--name qwen3-tts-streaming \
|
||||
-p 8023:8000 \
|
||||
-e NVIDIA_VISIBLE_DEVICES=all \
|
||||
-v /path/to/Qwen3-TTS-12Hz-0.6B-Base:/models/Qwen3-TTS:ro \
|
||||
-v /path/to/Qwen3-TTS-12Hz-1.7B-Base:/models/Qwen3-TTS:ro \
|
||||
-v /path/to/faster-qwen3-tts/config:/config:rw \
|
||||
-v /path/to/active_voices:/voices:ro \
|
||||
martinb78/faster-qwen3-tts-dgx-spark:streaming \
|
||||
@ -34,12 +34,15 @@ docker run -d \
|
||||
"
|
||||
```
|
||||
|
||||
Download the 0.6B Base weights before starting the container:
|
||||
Download the 1.7B Base weights before starting the container:
|
||||
|
||||
```bash
|
||||
hf download Qwen/Qwen3-TTS-12Hz-0.6B-Base --local-dir /path/to/Qwen3-TTS-12Hz-0.6B-Base
|
||||
hf download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-Base
|
||||
```
|
||||
|
||||
Streaming must use the same model size as the one voices.json was built for (1.7B-Base), otherwise requests fail with a 1024 vs 2048 tensor size mismatch.
|
||||
To run the 0.6B-Base model for lower latency next to a 1.7B service on the same `voices.json`, set `EMBED_TAG=0.6b` on the streaming service (and `EMBED_TAG=1.7b` on the 1.7B one) and mount `/voices` read-write; each model then keeps its own `<name>.<tag>.pt` beside the WAV.
|
||||
|
||||
For an existing stack, change only the streaming service's model bind mount.
|
||||
`--model /models/Qwen3-TTS` loads that directory; `QWEN_TTS_MODEL` does not
|
||||
override this explicit argument. Use Base for reference-audio voice cloning.
|
||||
@ -107,7 +110,7 @@ Works with **OpenWebUI**, **SillyTavern**, **llama-swap**, and any OpenAI-compat
|
||||
- NVIDIA DGX Spark GB10 or another ARM64 system with CUDA 13
|
||||
- CUDA driver 580+
|
||||
- Docker + NVIDIA Container Toolkit
|
||||
- [Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) weights downloaded locally
|
||||
- [Qwen3-TTS-12Hz-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) weights downloaded locally
|
||||
|
||||
---
|
||||
|
||||
|
||||
@ -24,8 +24,8 @@ RUN git clone https://github.com/andimarafioti/faster-qwen3-tts.git /app && \
|
||||
cd /app && git checkout --detach ${FASTER_QWEN3_TTS_REF}
|
||||
|
||||
# Apply DGX Spark patches (non_streaming_mode=True, per-voice temperature/top_k/top_p)
|
||||
COPY patches/openai_server.patch /tmp/
|
||||
RUN cd /app && git apply /tmp/openai_server.patch
|
||||
COPY patches/openai_server.patch patches/predictor_compile.patch /tmp/
|
||||
RUN cd /app && git apply /tmp/openai_server.patch && git apply /tmp/predictor_compile.patch
|
||||
|
||||
# Create virtual environment (Ubuntu 24.04 enforces PEP 668)
|
||||
ENV VIRTUAL_ENV=/opt/venv
|
||||
|
||||
43
README.md
43
README.md
@ -19,8 +19,8 @@ One Docker image covers all four backends, with two semantic tag aliases:
|
||||
|
||||
| Tag | Use for |
|
||||
|---|---|
|
||||
| `:latest` / `:v6` / `:v6.13` | VoiceClone, VoiceDesign, CustomVoice |
|
||||
| `:latest-streaming` / `:v6-streaming` / `:v6.13-streaming` | Streaming VoiceClone |
|
||||
| `:latest` / `:v6` / `:v6.15.0` | VoiceClone, VoiceDesign, CustomVoice |
|
||||
| `:latest-streaming` / `:v6-streaming` / `:v6.15.0-streaming` | Streaming VoiceClone |
|
||||
|
||||
Both tags point to the same image — the `-streaming` suffix is a semantic convention so compose files and version pins are unambiguous.
|
||||
|
||||
@ -354,7 +354,10 @@ The first request after container startup can be slower because CUDA graph captu
|
||||
|
||||
- The 1.7B Qwen3-TTS models use about 6 GB of GPU memory each in bfloat16.
|
||||
- The forum playbook shows the four API containers running together on DGX Spark with low visible memory pressure, but exact usage depends on model size, sequence length, and warmup state.
|
||||
- Use the 0.6B Qwen3-TTS variants if you want a lighter multi-service setup.
|
||||
- The 0.6B Qwen3-TTS variants are lighter, but their speaker embeddings (1024-dim) are incompatible with the 1.7B ones (2048-dim). Set `EMBED_TAG=0.6b` (and `EMBED_TAG=1.7b` on the 1.7B services) and mount `/voices` read-write: each voice then gets `<name>.<tag>.pt` beside its WAV and both sizes can share one `voices.json` (see `docker/docker-compose.portainer-0.6b.yml`).
|
||||
- Measured on GB10 with other GPU services running (streaming PCM, first request excluded): 0.6B-Base TTFA ~570 ms, RTF 0.53–0.60 (1.7–1.9× real-time); 1.7B-Base TTFA ~820 ms, RTF 0.78–0.84 (1.2–1.3× real-time). The first request after a start is much slower while graphs/embeddings are built.
|
||||
- Optional: `TTS_COMPILE_PREDICTOR=1` (image v6.15.0+) fuses the code predictor for ~10 % more throughput; see the v6.15.0 changelog entry.
|
||||
- Throughput is not limited by memory pressure or GPU clocks (re-measured with the other large GPU model stopped: identical). The streaming process runs at ~100 % of one CPU core at ~32 W GPU power, so per-step launch/Python overhead is the suspected limit (not profiled yet). Lowering latency in practice: use the 0.6B stream with `response_format: "pcm"` and send one or two sentences per request.
|
||||
- `--max-seq-len 2048` handles most sentence-style TTS requests. Long-form narration may need `4096`, with more memory required.
|
||||
- Pin services to different GPUs with `NVIDIA_VISIBLE_DEVICES=0`, `NVIDIA_VISIBLE_DEVICES=1`, and so on if your system has more than one GPU.
|
||||
|
||||
@ -378,6 +381,40 @@ The first request after container startup can be slower because CUDA graph captu
|
||||
|
||||
## Changelog
|
||||
|
||||
### v6.15.1 — 2026-10-12
|
||||
|
||||
- **Docs:** German sample files (10 voices, 0.6B-Base streaming, same text) in `samples/german sample files/` with an `index.html` that has an audio player per voice (open locally or via GitHub Pages) and a link table, added for the German-pronunciation question in the forum.
|
||||
- **Docs:** image tag table now lists `v6.15.0`; the `v6.15.0` images and the `:v6`/`:latest` aliases are on Docker Hub.
|
||||
- **Image:** unchanged (`:v6.15.0`), no rebuild needed.
|
||||
- **Rollback:** `git checkout v6.15.0` (documentation only).
|
||||
|
||||
### v6.15.0 — 2026-10-11
|
||||
|
||||
- **Perf (opt-in):** `TTS_COMPILE_PREDICTOR=1` compiles the 15-step code-predictor loop with `torch.compile` (Inductor/Triton) before CUDA-graph capture. Profiling showed the predictor needs ~5350 kernels per frame (about 54 % of the decode time); fusing them cuts its replay from ~23 ms to ~17.5 ms. Measured through the API on GB10 with the 0.6B-Base streaming server (same voice and text, other TTS containers stopped): 1.6–1.7x -> 1.8x real-time on long text, first audio ~670 ms -> ~550 ms. Listening comparison of 5 voices: no audible difference. Costs about one extra minute at startup. Default is off (unchanged behaviour).
|
||||
- **Image:** rebuilt as `v6.15.0` / `v6.15.0-streaming` (adds `patches/predictor_compile.patch`) and pushed to Docker Hub together with the `:v6`, `:latest`, `:v6-streaming` and `:latest-streaming` aliases.
|
||||
- **Investigation notes:** memory pressure/swap, GPU clocks, CPU core type and `chunk_size` do not limit throughput; `float16` fails (inf/NaN logits, device-side assert); flash-attn is irrelevant for single-token decode. F5-TTS, EmoKnob and Breeze-TTS-2 were benchmarked against Qwen3-TTS: slower than real-time (Breeze 0.5x, EmoKnob 0.1x) or without streaming/German (F5 ~5x) and not usable quality-wise in our test.
|
||||
- **Rollback:** unset `TTS_COMPILE_PREDICTOR` (or use the `:v6.13.1` image).
|
||||
|
||||
### v6.14.1 — 2026-10-11
|
||||
|
||||
- **Docs:** measured GB10 benchmark numbers for 0.6B vs 1.7B, the finding that memory pressure and GPU clocks are not the bottleneck, and the corrected 0.6B guidance (`EMBED_TAG`) in "Performance and memory notes".
|
||||
- **Deploy note:** Portainer re-creates services from its stored stack file, so a `docker compose up` from the CLI is overwritten. Update the stack in the Portainer editor instead.
|
||||
- **Image:** unchanged (`:v6.13.1`), no rebuild needed.
|
||||
- **Rollback:** `git checkout v6.14.0` (documentation only).
|
||||
|
||||
### v6.14.0 — 2026-10-11
|
||||
|
||||
- **Feature:** `EMBED_TAG` environment variable (e.g. `1.7b`, `0.6b`). Speaker embeddings are stored beside the reference audio as `<name>.<tag>.pt`, so a 1.7B VoiceClone and a 0.6B Streaming backend can share one `voices.json` and one voice folder (mount `/voices` read-write). Existing `config/speakers/*.pt` files are adopted once for the `1.7b` tag; embeddings older than their WAV/TXT are regenerated.
|
||||
- **Feature:** `/voices` and `/v1/voices` discovery endpoints on all backends.
|
||||
- **Config:** new `docker/docker-compose.portainer-0.6b.yml` — 1.7B VoiceClone on :8020 plus 0.6B-Base Streaming on :8023 (`--max-seq-len 2048`). The 0.6B model needs its own embeddings (1024-dim); never share `.pt` files between model sizes.
|
||||
- **Image:** unchanged (`:v6.13.1`); the changes live in `/config` (bind mount), no rebuild needed.
|
||||
- **Rollback:** redeploy the previous compose file (`:v6.13.1` images); `EMBED_TAG` unset keeps the old `/config/speakers` behaviour.
|
||||
|
||||
### v6.13.1 — 2026-10-04
|
||||
|
||||
- **Fix:** Streaming must use the same model size as the one voices.json was built for (1.7B-Base), otherwise requests fail with a 1024 vs 2048 tensor size mismatch. Streaming compose examples now default to the 1.7B-Base model.
|
||||
- **Fix:** Out-of-range `chunk_size` values are clamped to 2–24 (e.g. -5/0/1 -> 2, 1000 -> 24) and still return 200; only non-integers return 422. Covered by an offline check of the patched `SpeechRequest` + `_clamp_chunk_size`.
|
||||
|
||||
### v6.13 — 2026-10-04
|
||||
**Feature: Per-Request `chunk_size` for Streaming**
|
||||
- `/v1/audio/speech` accepts an optional integer `chunk_size` (codec frames per streamed chunk, 12 = 1 s), clamped to 2–24. It applies to the streaming wav/pcm path only; omitted keeps the voice's `voices.json` value (default unchanged).
|
||||
|
||||
@ -321,6 +321,12 @@ async def list_audio_models():
|
||||
return _models_response
|
||||
|
||||
|
||||
@app.get("/voices")
|
||||
@app.get("/v1/voices")
|
||||
async def list_voices():
|
||||
return _models_response
|
||||
|
||||
|
||||
@app.get("/speakers")
|
||||
async def get_speakers():
|
||||
return list(voices.keys())
|
||||
|
||||
@ -32,6 +32,46 @@ import openai_server
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Speaker embeddings (.pt) are model-size specific (1.7B=2048-dim, 0.6B=1024-dim).
|
||||
# With EMBED_TAG set (e.g. "1.7b"/"0.6b") every voice's embedding lives next to its
|
||||
# reference audio as <name>.<tag>.pt, so several model sizes can share one voices.json.
|
||||
_EMBED_TAG = os.environ.get("EMBED_TAG", "").strip()
|
||||
|
||||
|
||||
def _retarget_embeddings(voice_cfg: dict) -> dict:
|
||||
"""Point voice_cfg at the per-model .pt beside the reference audio (idempotent)."""
|
||||
ref_audio = voice_cfg.get("ref_audio")
|
||||
if not _EMBED_TAG or not ref_audio or voice_cfg.get("_emb_tag") == _EMBED_TAG:
|
||||
return voice_cfg
|
||||
stem = os.path.splitext(ref_audio)[0]
|
||||
pt = f"{stem}.{_EMBED_TAG}.pt"
|
||||
legacy = voice_cfg.get("speaker_embeddings") or ""
|
||||
try:
|
||||
# ponytail: legacy /config/speakers/*.pt are 1.7B embeddings; adopt them once for the 1.7b tag only.
|
||||
if _EMBED_TAG.lower() == "1.7b" and not os.path.exists(pt) and os.path.isfile(legacy):
|
||||
import shutil
|
||||
shutil.copy2(legacy, pt)
|
||||
if os.path.exists(pt):
|
||||
sources = [ref_audio, stem + ".txt", stem + ".reference.txt"]
|
||||
newest = max(os.path.getmtime(f) for f in sources if os.path.exists(f))
|
||||
if newest > os.path.getmtime(pt) + 1: # reference changed -> embedding stale
|
||||
os.remove(pt)
|
||||
except OSError as exc:
|
||||
logger.warning("Embedding retarget problem for %s: %s", pt, exc)
|
||||
voice_cfg["speaker_embeddings"] = pt
|
||||
voice_cfg["_emb_tag"] = _EMBED_TAG
|
||||
return voice_cfg
|
||||
|
||||
|
||||
if _EMBED_TAG:
|
||||
_orig_resolve_voice = openai_server.resolve_voice
|
||||
|
||||
def _resolve_voice_tagged(voice_name: str) -> dict:
|
||||
return _retarget_embeddings(_orig_resolve_voice(voice_name))
|
||||
|
||||
openai_server.resolve_voice = _resolve_voice_tagged
|
||||
|
||||
|
||||
def _do_warmup():
|
||||
"""Run one short generation to compile CUDA graphs before serving requests."""
|
||||
model = openai_server.tts_model
|
||||
@ -77,6 +117,7 @@ def _precompute_all_embeddings():
|
||||
return
|
||||
|
||||
for voice_name, voice_cfg in list(voices.items()):
|
||||
_retarget_embeddings(voice_cfg)
|
||||
# skip if already precomputed
|
||||
spk_emb_path = voice_cfg.get("speaker_embeddings") or voice_cfg.get("speaker embeddings")
|
||||
if spk_emb_path and os.path.isfile(spk_emb_path):
|
||||
@ -132,6 +173,11 @@ async def list_audio_voices():
|
||||
async def list_audio_models():
|
||||
return _models_response
|
||||
|
||||
@openai_server.app.get('/voices')
|
||||
@openai_server.app.get('/v1/voices')
|
||||
async def list_voices():
|
||||
return _models_response
|
||||
|
||||
# SillyTavern speaker endpoint
|
||||
@openai_server.app.get('/speakers')
|
||||
async def get_speakers():
|
||||
|
||||
@ -392,6 +392,12 @@ async def list_audio_models():
|
||||
_reload_voices_if_changed()
|
||||
return _models_response
|
||||
|
||||
@app.get("/voices")
|
||||
@app.get("/v1/voices")
|
||||
async def list_voices():
|
||||
_reload_voices_if_changed()
|
||||
return _models_response
|
||||
|
||||
@app.get("/speakers")
|
||||
async def get_speakers():
|
||||
_reload_voices_if_changed()
|
||||
|
||||
161
docker/docker-compose.portainer-0.6b.yml
Normal file
161
docker/docker-compose.portainer-0.6b.yml
Normal file
@ -0,0 +1,161 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Qwen3-TTS GPU Stack
|
||||
#
|
||||
# Ports:
|
||||
# 8020 -> Voice clone, OpenAI-compatible /v1/audio/speech
|
||||
# 8021 -> VoiceDesign model for prompt-designed voices
|
||||
# 8022 -> CustomVoice model
|
||||
# 8023 -> Low-latency streaming voice clone endpoint
|
||||
#
|
||||
# Container Name RAM Usage GPU VRAM Usage
|
||||
# faster-qwen3-tts-voiceclone ~2.72 GiB 5327 MiB (~5.2 GiB)
|
||||
# faster-qwen3-tts-customvoice ~2.88 GiB 4683 MiB (~4.6 GiB)
|
||||
# faster-qwen3-tts-voicedesign ~2.90 GiB 4747 MiB (~4.6 GiB)
|
||||
# faster-qwen3-tts-streaming ~2.54 GiB 5659 MiB (~5.5 GiB)
|
||||
#
|
||||
# Voice files:
|
||||
# The voice clone and streaming services scan active voices from:
|
||||
# home/sparky/Media/_Sounds/TTS_Voices/active_voices
|
||||
#
|
||||
# Portainer note:
|
||||
# Set HF_TOKEN in the stack environment if the models ever need Hugging Face access.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
services:
|
||||
# Standard voice cloning backend used by TTS Voice Creator and other OpenAI-compatible clients.
|
||||
faster-qwen3-tts-voiceclone:
|
||||
image: martinb78/faster-qwen3-tts-dgx-spark:latest
|
||||
container_name: tts-faster-qwen3-tts-voiceclone
|
||||
restart: unless-stopped
|
||||
runtime: nvidia
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- HF_TOKEN=${HF_TOKEN}
|
||||
- EMBED_TAG=1.7b
|
||||
ports:
|
||||
- "8020:8000"
|
||||
volumes:
|
||||
- /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-1.7B-Base:/models/Qwen3-TTS:ro
|
||||
- /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
|
||||
- /home/sparky/Media/_Sounds/TTS_Voices/active_voices:/voices:rw
|
||||
command: >
|
||||
/bin/bash -c "
|
||||
python3 /config/generate_voices.py &&
|
||||
python3 /config/run_server.py
|
||||
--model /models/Qwen3-TTS
|
||||
--voices /config/voices.json
|
||||
--port 8000
|
||||
--max-seq-len 4096
|
||||
"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
networks:
|
||||
- dgx_net
|
||||
|
||||
# VoiceDesign backend: creates speech from a descriptive voice prompt instead of a reference clip.
|
||||
faster-qwen3-tts-voicedesign:
|
||||
image: martinb78/faster-qwen3-tts-dgx-spark:latest
|
||||
container_name: tts-faster-qwen3-tts-voicedesign
|
||||
restart: unless-stopped
|
||||
runtime: nvidia
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- HF_TOKEN=${HF_TOKEN}
|
||||
ports:
|
||||
- "8021:8000"
|
||||
volumes:
|
||||
- /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-1.7B-VoiceDesign:/models/Qwen3-TTS-VoiceDesign:ro
|
||||
- /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
|
||||
command: >
|
||||
/bin/bash -c "
|
||||
python3 /config/run_voicedesign_server.py
|
||||
--model /models/Qwen3-TTS-VoiceDesign
|
||||
--voices /config/voicedesign_voices.json
|
||||
--port 8000
|
||||
--max-seq-len 2048
|
||||
"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
networks:
|
||||
- dgx_net
|
||||
|
||||
# CustomVoice backend: keeps the separate CustomVoice model available on port 8022.
|
||||
faster-qwen3-tts-customvoice:
|
||||
image: martinb78/faster-qwen3-tts-dgx-spark:latest
|
||||
container_name: tts-faster-qwen3-tts-customvoice
|
||||
restart: unless-stopped
|
||||
runtime: nvidia
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- HF_TOKEN=${HF_TOKEN}
|
||||
ports:
|
||||
- "8022:8000"
|
||||
volumes:
|
||||
- /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-1.7B-CustomVoice:/models/Qwen3-TTS-CustomVoice:ro
|
||||
- /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
|
||||
command: >
|
||||
/bin/bash -c "
|
||||
python3 /config/run_customvoice_server.py
|
||||
--model /models/Qwen3-TTS-CustomVoice
|
||||
--voices /config/customvoice_voices.json
|
||||
--port 8000
|
||||
--max-seq-len 2048
|
||||
"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
networks:
|
||||
- dgx_net
|
||||
|
||||
# Streaming voice clone backend: same active voices as 8020, but streams WAV chunks while generating.
|
||||
faster-qwen3-tts-streaming:
|
||||
image: martinb78/faster-qwen3-tts-dgx-spark:latest-streaming
|
||||
container_name: tts-faster-qwen3-tts-streaming
|
||||
restart: unless-stopped
|
||||
runtime: nvidia
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- HF_TOKEN=${HF_TOKEN}
|
||||
- EMBED_TAG=0.6b
|
||||
ports:
|
||||
- "8023:8000"
|
||||
volumes:
|
||||
- /home/sparky/LLMs/vllm/Alibaba/Qwen3-TTS-12Hz-0.6B-Base:/models/Qwen3-TTS:ro
|
||||
- /home/sparky/Docker/tts-dgx-spark-faster-qwen3-tts/config:/config:rw
|
||||
- /home/sparky/Media/_Sounds/TTS_Voices/active_voices:/voices:rw
|
||||
command: >
|
||||
/bin/bash -c "
|
||||
python3 /config/generate_voices.py &&
|
||||
python3 /config/run_server.py
|
||||
--model /models/Qwen3-TTS
|
||||
--voices /config/voices.json
|
||||
--port 8000
|
||||
--max-seq-len 2048
|
||||
"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
networks:
|
||||
- dgx_net
|
||||
|
||||
networks:
|
||||
dgx_net:
|
||||
external: true
|
||||
@ -129,7 +129,7 @@ services:
|
||||
ports:
|
||||
- "8023:8000"
|
||||
volumes:
|
||||
- /path/to/Qwen3-TTS-12Hz-0.6B-Base:/models/Qwen3-TTS:ro
|
||||
- /path/to/Qwen3-TTS-12Hz-1.7B-Base:/models/Qwen3-TTS:ro
|
||||
- /path/to/faster-qwen3-tts/config:/config:rw
|
||||
- /path/to/active_voices:/voices:ro
|
||||
command: >
|
||||
|
||||
25
patches/predictor_compile.patch
Normal file
25
patches/predictor_compile.patch
Normal file
@ -0,0 +1,25 @@
|
||||
diff --git a/faster_qwen3_tts/predictor_graph.py b/faster_qwen3_tts/predictor_graph.py
|
||||
index 38a52ca..70851eb 100644
|
||||
--- a/faster_qwen3_tts/predictor_graph.py
|
||||
+++ b/faster_qwen3_tts/predictor_graph.py
|
||||
@@ -13,6 +13,8 @@ Strategy:
|
||||
- Unroll the full 15-step loop for deterministic shapes
|
||||
- Capture the entire loop as a single CUDA graph
|
||||
"""
|
||||
+import os
|
||||
+
|
||||
import torch
|
||||
from transformers import StaticCache
|
||||
from transformers.masking_utils import create_causal_mask, create_sliding_window_causal_mask
|
||||
@@ -175,6 +177,11 @@ class PredictorGraph:
|
||||
self._init_cache_layers()
|
||||
self._build_attention_masks()
|
||||
|
||||
+ # Opt-in: fuse the 15-step predictor loop with Inductor before graph capture.
|
||||
+ # ~17% faster streaming on GB10 (0.6B); adds ~1 min to the first warmup.
|
||||
+ if os.environ.get("TTS_COMPILE_PREDICTOR") == "1":
|
||||
+ self._full_loop = torch.compile(self._full_loop, dynamic=False)
|
||||
+
|
||||
for _ in range(num_warmup):
|
||||
self.static_cache.reset()
|
||||
self._full_loop()
|
||||
26
samples/german sample files/README.md
Normal file
26
samples/german sample files/README.md
Normal file
@ -0,0 +1,26 @@
|
||||
# German sample files (0.6B-Base streaming)
|
||||
|
||||
Ten voices from the active library, rendered with the 0.6B-Base streaming server
|
||||
(`v6.15.0`, `TTS_COMPILE_PREDICTOR=1`, `language: German`, default sampling).
|
||||
All files use the same text:
|
||||
|
||||
> Guten Morgen! Das Wetter ist heute wunderbar, deshalb gehen wir am Rhein spazieren. Anschließend essen wir Käsekuchen und besprechen die Planung für das nächste Quartal, einschließlich Budget, Meeting und Zeitplan.
|
||||
|
||||
Files are in `mp3/` (24 kHz mono, VBR ~130 kbit/s), numbered 01-10 with the voice ID.
|
||||
|
||||
## Listen
|
||||
|
||||
GitHub's directory view has no audio player. Open `index.html` in this folder from a local clone (it has a player per voice), or enable GitHub Pages for the repository and open the same page there. Direct links to the files:
|
||||
|
||||
| # | Voice | File |
|
||||
|---|---|---|
|
||||
| 01 | DE_M_Gutfolg | [01_DE_M_Gutfolg.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/01_DE_M_Gutfolg.mp3) |
|
||||
| 02 | DE_Banker | [02_DE_Banker.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/02_DE_Banker.mp3) |
|
||||
| 03 | DE_N_Orkgeist | [03_DE_N_Orkgeist.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/03_DE_N_Orkgeist.mp3) |
|
||||
| 04 | DE_F_Amala | [04_DE_F_Amala.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/04_DE_F_Amala.mp3) |
|
||||
| 05 | DE_1_0 | [05_DE_1_0.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/05_DE_1_0.mp3) |
|
||||
| 06 | DE_F_19_German_Switzerland_Female_Leni | [06_DE_F_19_German_Switzerland_Female_Leni.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/06_DE_F_19_German_Switzerland_Female_Leni.mp3) |
|
||||
| 07 | DE_M_Rabe | [07_DE_M_Rabe.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/07_DE_M_Rabe.mp3) |
|
||||
| 08 | DE_N_Schwarzhaarige | [08_DE_N_Schwarzhaarige.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/08_DE_N_Schwarzhaarige.mp3) |
|
||||
| 09 | DE_N_Magier | [09_DE_N_Magier.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/09_DE_N_Magier.mp3) |
|
||||
| 10 | DE_M_Ork | [10_DE_M_Ork.mp3](https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts/blob/main/samples/german%20sample%20files/mp3/10_DE_M_Ork.mp3) |
|
||||
95
samples/german sample files/index.html
Normal file
95
samples/german sample files/index.html
Normal file
@ -0,0 +1,95 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>German samples - Qwen3-TTS 0.6B streaming</title>
|
||||
<style>
|
||||
:root { color-scheme: light dark; --bg:#fff; --fg:#1b1f24; --muted:#59636e; --line:#d1d9e0; }
|
||||
@media (prefers-color-scheme: dark) { :root { --bg:#0d1117; --fg:#e6edf3; --muted:#9198a1; --line:#3d444d; } }
|
||||
body { margin:0; padding:1.5rem 1rem; background:var(--bg); color:var(--fg); font:16px/1.5 system-ui,sans-serif; }
|
||||
main { max-width:52rem; margin:0 auto; }
|
||||
blockquote { margin:1rem 0; padding:.5rem 1rem; border-left:4px solid var(--line); color:var(--muted); }
|
||||
.scroll { overflow-x:auto; }
|
||||
table { border-collapse:collapse; width:100%; }
|
||||
th, td { text-align:left; padding:.5rem; border-bottom:1px solid var(--line); vertical-align:middle; }
|
||||
audio { width:100%; min-width:14rem; }
|
||||
a { color:inherit; }
|
||||
a:focus-visible, audio:focus-visible { outline:3px solid #0969da; outline-offset:2px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>German samples: Qwen3-TTS 0.6B-Base streaming</h1>
|
||||
<p>Ten voices from the active library, rendered with the 0.6B-Base streaming server (v6.15.0, <code>language: German</code>, default sampling). Every file uses the same text:</p>
|
||||
<blockquote lang="de">Guten Morgen! Das Wetter ist heute wunderbar, deshalb gehen wir am Rhein spazieren. Anschließend essen wir Käsekuchen und besprechen die Planung für das nächste Quartal, einschließlich Budget, Meeting und Zeitplan.</blockquote>
|
||||
<div class="scroll">
|
||||
<table>
|
||||
<thead><tr><th scope="col">#</th><th scope="col">Voice</th><th scope="col">Listen</th><th scope="col">File</th></tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th scope="row">01</th>
|
||||
<td>DE_M_Gutfolg</td>
|
||||
<td><audio controls preload="none" src="mp3/01_DE_M_Gutfolg.mp3" aria-label="Sample 01: DE_M_Gutfolg"></audio></td>
|
||||
<td><a href="mp3/01_DE_M_Gutfolg.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">02</th>
|
||||
<td>DE_Banker</td>
|
||||
<td><audio controls preload="none" src="mp3/02_DE_Banker.mp3" aria-label="Sample 02: DE_Banker"></audio></td>
|
||||
<td><a href="mp3/02_DE_Banker.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">03</th>
|
||||
<td>DE_N_Orkgeist</td>
|
||||
<td><audio controls preload="none" src="mp3/03_DE_N_Orkgeist.mp3" aria-label="Sample 03: DE_N_Orkgeist"></audio></td>
|
||||
<td><a href="mp3/03_DE_N_Orkgeist.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">04</th>
|
||||
<td>DE_F_Amala</td>
|
||||
<td><audio controls preload="none" src="mp3/04_DE_F_Amala.mp3" aria-label="Sample 04: DE_F_Amala"></audio></td>
|
||||
<td><a href="mp3/04_DE_F_Amala.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">05</th>
|
||||
<td>DE_1_0</td>
|
||||
<td><audio controls preload="none" src="mp3/05_DE_1_0.mp3" aria-label="Sample 05: DE_1_0"></audio></td>
|
||||
<td><a href="mp3/05_DE_1_0.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">06</th>
|
||||
<td>DE_F_19_German_Switzerland_Female_Leni</td>
|
||||
<td><audio controls preload="none" src="mp3/06_DE_F_19_German_Switzerland_Female_Leni.mp3" aria-label="Sample 06: DE_F_19_German_Switzerland_Female_Leni"></audio></td>
|
||||
<td><a href="mp3/06_DE_F_19_German_Switzerland_Female_Leni.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">07</th>
|
||||
<td>DE_M_Rabe</td>
|
||||
<td><audio controls preload="none" src="mp3/07_DE_M_Rabe.mp3" aria-label="Sample 07: DE_M_Rabe"></audio></td>
|
||||
<td><a href="mp3/07_DE_M_Rabe.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">08</th>
|
||||
<td>DE_N_Schwarzhaarige</td>
|
||||
<td><audio controls preload="none" src="mp3/08_DE_N_Schwarzhaarige.mp3" aria-label="Sample 08: DE_N_Schwarzhaarige"></audio></td>
|
||||
<td><a href="mp3/08_DE_N_Schwarzhaarige.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">09</th>
|
||||
<td>DE_N_Magier</td>
|
||||
<td><audio controls preload="none" src="mp3/09_DE_N_Magier.mp3" aria-label="Sample 09: DE_N_Magier"></audio></td>
|
||||
<td><a href="mp3/09_DE_N_Magier.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">10</th>
|
||||
<td>DE_M_Ork</td>
|
||||
<td><audio controls preload="none" src="mp3/10_DE_M_Ork.mp3" aria-label="Sample 10: DE_M_Ork"></audio></td>
|
||||
<td><a href="mp3/10_DE_M_Ork.mp3" download>mp3</a></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
BIN
samples/german sample files/mp3/01_DE_M_Gutfolg.mp3
Normal file
BIN
samples/german sample files/mp3/01_DE_M_Gutfolg.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/02_DE_Banker.mp3
Normal file
BIN
samples/german sample files/mp3/02_DE_Banker.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/03_DE_N_Orkgeist.mp3
Normal file
BIN
samples/german sample files/mp3/03_DE_N_Orkgeist.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/04_DE_F_Amala.mp3
Normal file
BIN
samples/german sample files/mp3/04_DE_F_Amala.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/05_DE_1_0.mp3
Normal file
BIN
samples/german sample files/mp3/05_DE_1_0.mp3
Normal file
Binary file not shown.
Binary file not shown.
BIN
samples/german sample files/mp3/07_DE_M_Rabe.mp3
Normal file
BIN
samples/german sample files/mp3/07_DE_M_Rabe.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/08_DE_N_Schwarzhaarige.mp3
Normal file
BIN
samples/german sample files/mp3/08_DE_N_Schwarzhaarige.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/09_DE_N_Magier.mp3
Normal file
BIN
samples/german sample files/mp3/09_DE_N_Magier.mp3
Normal file
Binary file not shown.
BIN
samples/german sample files/mp3/10_DE_M_Ork.mp3
Normal file
BIN
samples/german sample files/mp3/10_DE_M_Ork.mp3
Normal file
Binary file not shown.
Loading…
Reference in New Issue
Block a user