mARTin-B78/tts-dgx-spark-faster-qwen3-tts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3 Commits 1 Branch 0 Tags 323 KiB
Python 97.5%
Dockerfile 2.5%
92e2381d0b
Go to file
mARTin-B78 92e2381d0b Reduce latency: CUDA warmup, chunk_size=4, max-seq-len 2048 
- run_server.py: warm up CUDA graphs at server startup via lifespan
  event so the first real request does not pay the 7-8s graph-
  compilation penalty; uses modern lifespan API instead of deprecated
  on_event
- generate_voices.py: set chunk_size=4 per voice so streaming clients
  receive first audio after ~333ms instead of ~1s
- docker-compose.yml: add --max-seq-len 2048 (halves static KV cache,
  reduces VRAM and graph-capture time)
- config/benchmark_api.py: new script to measure TTFA, RTF and speed
  against the live API endpoint

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 20:43:21 +02:00
config Reduce latency: CUDA warmup, chunk_size=4, max-seq-len 2048 2026-05-11 20:43:21 +02:00
patches Initial release: Faster-Qwen3-TTS for DGX Spark GB10 2026-04-14 00:06:05 +02:00
.env.example Initial release: Faster-Qwen3-TTS for DGX Spark GB10 2026-04-14 00:06:05 +02:00
.gitignore Initial release: Faster-Qwen3-TTS for DGX Spark GB10 2026-04-14 00:06:05 +02:00
docker-compose.yml Reduce latency: CUDA warmup, chunk_size=4, max-seq-len 2048 2026-05-11 20:43:21 +02:00
Dockerfile Initial release: Faster-Qwen3-TTS for DGX Spark GB10 2026-04-14 00:06:05 +02:00
LICENSE Initial release: Faster-Qwen3-TTS for DGX Spark GB10 2026-04-14 00:06:05 +02:00
README.md Initial release: Faster-Qwen3-TTS for DGX Spark GB10 2026-04-14 00:06:05 +02:00

README.md

Faster-Qwen3-TTS for NVIDIA DGX Spark (GB10)

Run faster-qwen3-tts on the NVIDIA DGX Spark GB10 (ARM64 / SM 121 / CUDA 13) as a Docker container with an OpenAI-compatible TTS API.

Integrates with OpenWebUI, SillyTavern, and any OpenAI TTS-compatible client.

What this solves

The DGX Spark GB10 has a unique combination of ARM64 (Grace CPU) + Blackwell GPU (SM 121) that causes issues with standard ML Docker images:

  • torchaudio ARM64 wheels - resolved by using PyTorch's cu130 wheel index
  • Flash Attention - won't compile on SM 121, but faster-qwen3-tts uses CUDA graphs instead (6-10x speedup)
  • CUDA graph capture - works on SM 121 with max_seq_len tuned for voice cloning workloads
  • OpenWebUI voice discovery - custom endpoints (/v1/models, /v1/audio/voices) for voice dropdown population

Quick Start

Option 1: Pull pre-built image (recommended)

# Pull the image
docker pull martinb78/faster-qwen3-tts-dgx-spark:latest

# Download the model
mkdir -p models
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir ./models/Qwen3-TTS

# Copy .env.example to .env and set MODEL_PATH
cp .env.example .env

# Add voice reference audio (5-15 second WAV/MP3 clips) to config/speakers/
# See "Adding Voices" below

# Start
docker compose up -d

Option 2: Build from source

docker build -t faster-qwen3-tts-dgx-spark:latest .

Adding Voices

Place reference audio files in config/speakers/ using this naming convention:

EN_M_Speaker_Name.wav    # English, Male
EN_F_Speaker_Name.wav    # English, Female
DE_M_Speaker_Name.wav    # German, Male

Important: Reference audio must be 5-15 seconds long. Longer files cause slow inference and poor voice cloning quality.

For each audio file, create a matching transcript:

EN_M_Speaker_Name.reference.txt

Or use the auto-transcription script (requires a running Whisper-compatible ASR service):

python config/auto_transcribe.py --api-url http://localhost:8010/v1/audio/transcriptions

The generate_voices.py script runs automatically on container startup and creates voices.json from your speaker files.

API Endpoints

Endpoint Method Description
/health GET Health check
/v1/audio/speech POST Generate speech (OpenAI-compatible)
/v1/models GET List available voices
/v1/audio/voices GET List voices (OpenWebUI fallback)
/v1/audio/models GET List models (OpenWebUI fallback)
/speakers GET List speaker IDs (SillyTavern)

Example

curl -X POST http://localhost:8020/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "tts-1", "input": "Hello world!", "voice": "speaker_name", "response_format": "wav"}' \
  --output speech.wav

OpenWebUI Configuration

In OpenWebUI Settings > Audio > Text-to-Speech:

Setting Value
Engine OpenAI
URL http://faster-qwen3-tts:8000/v1
API Key sk-dummy-key
TTS Model tts-1
TTS Voice Select from dropdown

Performance

On DGX Spark GB10 with the 1.7B model:

Input Audio Duration Generation Time RTF
Short sentence ~2s ~2.5s 0.8
Medium paragraph ~7s ~5.5s 0.77

First request is slower due to one-time CUDA graph warmup.

Hardware Requirements

  • NVIDIA DGX Spark GB10 (or any ARM64 + Blackwell GPU with CUDA 13)
  • ~6 GB GPU memory for the 1.7B model
  • CUDA driver 580+ with CUDA 13.0 support

Credits

License

MIT (same as upstream faster-qwen3-tts)