tts-dgx-spark-faster-qwen3-tts/README.md

Faster-Qwen3-TTS for NVIDIA DGX Spark (GB10)

Run faster-qwen3-tts on the NVIDIA DGX Spark GB10 (ARM64 / SM 121 / CUDA 13) as a persistent, OpenAI-compatible TTS API.

This repo packages the DGX Spark fixes plus four OpenAI-compatible TTS backends:

Backend	Port	Image	Voice source
VoiceClone	8020	martinb78/faster-qwen3-tts-dgx-spark:v4	Reference audio plus transcript
VoiceDesign	8021	martinb78/faster-qwen3-tts-dgx-spark:v4	Text prompt describes the voice; no reference needed
CustomVoice	8022	martinb78/faster-qwen3-tts-dgx-spark:v4	Separate CustomVoice model variant
Streaming	8023	martinb78/qwen3-tts-streaming-dgx-spark:latest	Same voices as 8020, but streams WAV chunks while generating

All four backends expose the OpenAI /v1/audio/speech contract and work with OpenWebUI, SillyTavern, llama-swap, curl, or any OpenAI-compatible client.

Both Docker images are published and publicly available:

• martinb78/faster-qwen3-tts-dgx-spark:v4 - used by VoiceClone, VoiceDesign, and CustomVoice.
• martinb78/qwen3-tts-streaming-dgx-spark:latest - used by the streaming service.

What this solves

The DGX Spark GB10 has a unique ARM64 Grace CPU plus Blackwell GPU stack (SM 121 / CUDA 13). Standard ML containers often need small but important changes:

• torchaudio ARM64 wheels - resolved by using PyTorch's cu130 wheel index.
• Flash Attention on SM 121 - avoided; faster-qwen3-tts uses CUDA graphs instead.
• CUDA graph capture - configured for low-latency Qwen3-TTS inference.
• OpenAI compatibility - /v1/audio/speech, /v1/models, /v1/audio/voices, /v1/audio/models, and /speakers are available for common clients.

Quick start: VoiceClone only

Use the root docker-compose.yml when you only need voice cloning on port 8020.

docker pull martinb78/faster-qwen3-tts-dgx-spark:latest

mkdir -p models
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir ./models/Qwen3-TTS

cp .env.example .env
# Edit .env and set MODEL_PATH to your local Qwen3-TTS-12Hz-1.7B-Base directory.

# Add reference audio and transcripts to config/speakers/ first.
docker compose up -d

Build the image locally instead of pulling Docker Hub:

docker build -t faster-qwen3-tts-dgx-spark:latest .

If docker compose up reports that dgx_net is missing, create it once:

docker network create dgx_net

Check the server:

curl http://localhost:8020/health

Full stack: VoiceClone, VoiceDesign, CustomVoice, Streaming

Use config/docker-compose.yml when you want all four OpenAI-compatible backends side by side:

8020  ->  VoiceClone   (/v1/audio/speech, reference audio)
8021  ->  VoiceDesign  (text prompt describes the voice, no reference needed)
8022  ->  CustomVoice  (separate CustomVoice model variant)
8023  ->  Streaming    (same as 8020 but streams WAV chunks while generating)

1. Download the models you want to run:

huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-Base
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-VoiceDesign
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-CustomVoice

2. Edit config/docker-compose.yml and adjust the volume paths for your machine:

volumes:
  - /path/to/Qwen3-TTS-12Hz-1.7B-Base:/models/Qwen3-TTS:ro
  - /path/to/Qwen3-TTS-12Hz-1.7B-VoiceDesign:/models/Qwen3-TTS-VoiceDesign:ro
  - /path/to/Qwen3-TTS-12Hz-1.7B-CustomVoice:/models/Qwen3-TTS-CustomVoice:ro
  - /path/to/this/repo/config:/config:rw

3. Make sure the external Docker network exists, then start the stack:

docker network create dgx_net 2>/dev/null || true
cd config
docker compose up -d

4. Check the services:

curl http://localhost:8020/health   # VoiceClone
curl http://localhost:8021/health   # VoiceDesign
curl http://localhost:8022/health   # CustomVoice
curl http://localhost:8023/health   # Streaming VoiceClone

Adding VoiceClone voices

Place reference audio files in config/speakers/ using this naming convention:

EN_M_Speaker_Name.wav    # English, male
EN_F_Speaker_Name.wav    # English, female
DE_M_Speaker_Name.wav    # German, male

Reference audio should be 5-15 seconds long. Longer files can slow inference and reduce cloning quality.

For each audio file, create a matching transcript:

EN_M_Speaker_Name.reference.txt

Or use the auto-transcription script with a running Whisper-compatible ASR service:

python config/auto_transcribe.py --api-url http://localhost:8010/v1/audio/transcriptions

config/generate_voices.py runs on container startup and creates config/voices.json from your speaker files.

VoiceDesign voices

VoiceDesign does not need reference audio. Define reusable voice personalities in config/voicedesign_voices.json:

{
  "narrator": {
    "instruct": "Warm, confident narrator with a slight British accent",
    "language": "English"
  },
  "assistant_de": {
    "instruct": "Freundliche, klare Sprecherin, Hochdeutsch, professionell",
    "language": "German"
  }
}

Then call the VoiceDesign service on port 8021.

CustomVoice speakers

CustomVoice uses the model's built-in speaker names. Define the speaker IDs you want to expose in config/customvoice_voices.json:

{
  "Ryan": {
    "speaker": "Ryan",
    "language": "English",
    "instruct": ""
  },
  "Ono_Anna": {
    "speaker": "Ono_Anna",
    "language": "Japanese",
    "instruct": ""
  },
  "Sohee": {
    "speaker": "Sohee",
    "language": "Korean",
    "instruct": ""
  }
}

Then call the CustomVoice service on port 8022.

Streaming backend

The streaming service on port 8023 uses the same generated config/voices.json and active VoiceClone reference voices as port 8020, but returns WAV chunks while generation is still running. Use it when time-to-first-audio matters more than waiting for the complete WAV response.

API

Endpoints

Endpoint	Method	Description
/health	GET	Health check
/v1/audio/speech	POST	Generate speech in OpenAI-compatible format
/v1/models	GET	List available voice IDs
/v1/audio/voices	GET	OpenWebUI voice-list fallback
/v1/audio/models	GET	OpenWebUI model-list fallback
/speakers	GET	Speaker IDs for SillyTavern and simple clients

Speech request fields

Field	Type	Default	Notes
model	string	tts-1	Kept for OpenAI compatibility
input	string	required	Text to synthesize
voice	string	first configured voice	Voice ID from the selected service
response_format	string	wav	wav, pcm, or mp3
language	string	voice config	Per-request override for VoiceDesign/CustomVoice
instruct	string	voice config	Per-request style override for VoiceDesign/CustomVoice
max_new_tokens	int	server default	Per-request generation length override

WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response.

Examples

VoiceClone on port 8020:

curl http://localhost:8020/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1","input":"Hello world!","voice":"EN_M_Speaker_Name","response_format":"wav"}' \
  --output speech.wav

VoiceDesign on port 8021:

curl http://localhost:8021/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1","input":"Welcome to the show.","voice":"narrator"}' \
  --output speech.wav

Per-request VoiceDesign override:

curl http://localhost:8021/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "Herzlich willkommen.",
    "voice": "narrator",
    "language": "German",
    "instruct": "Speak slowly and warmly.",
    "max_new_tokens": 1024
  }' \
  --output speech_de.wav

CustomVoice on port 8022:

curl http://localhost:8022/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1","input":"This uses a built-in Qwen3-TTS speaker.","voice":"Ryan"}' \
  --output customvoice.wav

Streaming VoiceClone on port 8023:

curl http://localhost:8023/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1","input":"This starts playing as chunks arrive.","voice":"EN_M_Speaker_Name","response_format":"wav"}' \
  --output streaming.wav

Per-request fields win over the JSON voice config entry, so one configured voice can still be adjusted by callers for language, tone, or generation length.

Client configuration

OpenWebUI

In OpenWebUI Settings > Audio > Text-to-Speech:

Setting	Value
Engine	OpenAI
URL	http://your-host:8020/v1, http://your-host:8021/v1, http://your-host:8022/v1, or http://your-host:8023/v1
API Key	sk-dummy-key
TTS Model	tts-1
TTS Voice	Select from dropdown

llama-swap or other OpenAI-compatible clients

Point the client's OpenAI-compatible TTS base URL at the service you want:

http://your-host:8020/v1   # VoiceClone
http://your-host:8021/v1   # VoiceDesign
http://your-host:8022/v1   # CustomVoice
http://your-host:8023/v1   # Streaming VoiceClone

Benchmarking

Use config/benchmark_api.py to verify latency and real-time performance:

python config/benchmark_api.py --host localhost --port 8021 --runs 5

The benchmark reports:

Metric	Meaning
TTFA	Time to first audio byte; useful for interactive playback latency
RTF	Generation time divided by audio duration; lower is better
Speed	Audio duration divided by generation time; higher than 1.0x is faster than real time

The first request after container startup can be slower because CUDA graph capture runs once during warmup. Later requests should use the captured graph.

Performance and memory notes

• The 1.7B Qwen3-TTS models use about 6 GB of GPU memory each in bfloat16.
• The forum playbook shows the four API containers running together on DGX Spark with low visible memory pressure, but exact usage depends on model size, sequence length, and warmup state.
• Use the 0.6B Qwen3-TTS variants if you want a lighter multi-service setup.
• --max-seq-len 2048 handles most sentence-style TTS requests. Long-form narration may need 4096, with more memory required.
• Pin services to different GPUs with NVIDIA_VISIBLE_DEVICES=0, NVIDIA_VISIBLE_DEVICES=1, and so on if your system has more than one GPU.

Troubleshooting

Symptom	Likely cause	Fix
503 Model not loaded	Server still loading or warming up	Wait 30-60 seconds and check container logs
404 Voice not found	Voice ID is not in the JSON config	Check spelling or call /speakers
Very high TTFA	CUDA graph capture failed or fallback path is active	Check logs, reduce --max-seq-len, then restart
MP3 output error	MP3 dependencies are missing or ffmpeg is unavailable	Use wav/pcm or rebuild the image with MP3 support
OpenWebUI has no voices	Client cannot read the voice list	Confirm /v1/models and /v1/audio/voices are reachable from OpenWebUI

Hardware requirements

• NVIDIA DGX Spark GB10, or another ARM64 + NVIDIA GPU setup with CUDA 13 support.
• CUDA driver 580+ with CUDA 13.0 support.
• Docker plus NVIDIA Container Toolkit.
• Local Qwen3-TTS model weights from Hugging Face.

Credits

• faster-qwen3-tts by Andres Marafioti.
• Qwen3-TTS by the Alibaba Qwen team.
• DGX Spark compatibility, Docker, and OpenAI-compatible API packaging by mARTin-B78.
• NVIDIA Developer Forum playbook and source for the four-backend layout: Three times (VoiceClone | VoiceDesign | CustomVoice) - Faster-Qwen3-TTS for NVIDIA DGX Spark (GB10).

License

MIT (same as upstream faster-qwen3-tts).