Update README.md
This commit is contained in:
mARTin
GitHub
parent
92e2381d0b
commit
b7b917db66
297
README.md
297
README.md
@ -1,127 +1,320 @@
|
||||
# Faster-Qwen3-TTS for NVIDIA DGX Spark (GB10)
|
||||
|
||||
Run [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) on the **NVIDIA DGX Spark GB10** (ARM64 / SM 121 / CUDA 13) as a Docker container with an OpenAI-compatible TTS API.
|
||||
Run [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) on the **NVIDIA DGX Spark GB10** (ARM64 / SM 121 / CUDA 13) as a persistent, OpenAI-compatible TTS API.
|
||||
|
||||
Integrates with **OpenWebUI**, **SillyTavern**, and any OpenAI TTS-compatible client.
|
||||
|
||||
|
||||
This repo packages the DGX Spark fixes plus API servers for three Qwen3-TTS modes:
|
||||
|
||||
| Mode | Port | Model | Voice source |
|
||||
|---|---:|---|---|
|
||||
| VoiceClone | `8020` | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Reference audio plus transcript |
|
||||
| VoiceDesign | `8021` | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | Plain-English voice instructions |
|
||||
| CustomVoice | `8022` | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Built-in speaker IDs |
|
||||
|
||||
All modes expose the OpenAI `/v1/audio/speech` contract and work with **OpenWebUI**, **SillyTavern**, **llama-swap**, `curl`, or any OpenAI-compatible client.
|
||||
|
||||
## What this solves
|
||||
|
||||
The DGX Spark GB10 has a unique combination of ARM64 (Grace CPU) + Blackwell GPU (SM 121) that causes issues with standard ML Docker images:
|
||||
The DGX Spark GB10 has a unique ARM64 Grace CPU plus Blackwell GPU stack (SM 121 / CUDA 13). Standard ML containers often need small but important changes:
|
||||
|
||||
- **torchaudio ARM64 wheels** - resolved by using PyTorch's `cu130` wheel index
|
||||
- **Flash Attention** - won't compile on SM 121, but faster-qwen3-tts uses CUDA graphs instead (6-10x speedup)
|
||||
- **CUDA graph capture** - works on SM 121 with max_seq_len tuned for voice cloning workloads
|
||||
- **OpenWebUI voice discovery** - custom endpoints (`/v1/models`, `/v1/audio/voices`) for voice dropdown population
|
||||
- **torchaudio ARM64 wheels** - resolved by using PyTorch's `cu130` wheel index.
|
||||
- **Flash Attention on SM 121** - avoided; faster-qwen3-tts uses CUDA graphs instead.
|
||||
- **CUDA graph capture** - configured for low-latency Qwen3-TTS inference.
|
||||
- **OpenAI compatibility** - `/v1/audio/speech`, `/v1/models`, `/v1/audio/voices`, `/v1/audio/models`, and `/speakers` are available for common clients.
|
||||
|
||||
## Quick Start
|
||||
## Quick start: VoiceClone only
|
||||
|
||||
### Option 1: Pull pre-built image (recommended)
|
||||
Use the root `docker-compose.yml` when you only need voice cloning on port `8020`.
|
||||
|
||||
```bash
|
||||
# Pull the image
|
||||
docker pull martinb78/faster-qwen3-tts-dgx-spark:latest
|
||||
|
||||
# Download the model
|
||||
mkdir -p models
|
||||
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir ./models/Qwen3-TTS
|
||||
|
||||
# Copy .env.example to .env and set MODEL_PATH
|
||||
cp .env.example .env
|
||||
# Edit .env and set MODEL_PATH to your local Qwen3-TTS-12Hz-1.7B-Base directory.
|
||||
|
||||
# Add voice reference audio (5-15 second WAV/MP3 clips) to config/speakers/
|
||||
# See "Adding Voices" below
|
||||
|
||||
# Start
|
||||
# Add reference audio and transcripts to config/speakers/ first.
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Option 2: Build from source
|
||||
Build the image locally instead of pulling Docker Hub:
|
||||
|
||||
```bash
|
||||
docker build -t faster-qwen3-tts-dgx-spark:latest .
|
||||
```
|
||||
|
||||
## Adding Voices
|
||||
If `docker compose up` reports that `dgx_net` is missing, create it once:
|
||||
|
||||
```bash
|
||||
docker network create dgx_net
|
||||
```
|
||||
|
||||
Check the server:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8020/health
|
||||
```
|
||||
|
||||
## Full stack: VoiceClone, VoiceDesign, CustomVoice
|
||||
|
||||
Use `config/docker-compose.yml` when you want all Qwen3-TTS modes side by side. The file also includes an optional low-latency streaming VoiceClone service on port `8023`; remove or comment that service if you only want the three main endpoints.
|
||||
|
||||
1. Download the models you want to run:
|
||||
|
||||
```bash
|
||||
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-Base
|
||||
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-VoiceDesign
|
||||
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --local-dir /path/to/Qwen3-TTS-12Hz-1.7B-CustomVoice
|
||||
```
|
||||
|
||||
2. Edit `config/docker-compose.yml` and adjust the volume paths for your machine:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- /path/to/Qwen3-TTS-12Hz-1.7B-Base:/models/Qwen3-TTS:ro
|
||||
- /path/to/Qwen3-TTS-12Hz-1.7B-VoiceDesign:/models/Qwen3-TTS-VoiceDesign:ro
|
||||
- /path/to/Qwen3-TTS-12Hz-1.7B-CustomVoice:/models/Qwen3-TTS-CustomVoice:ro
|
||||
- /path/to/this/repo/config:/config:rw
|
||||
```
|
||||
|
||||
3. Make sure the external Docker network exists, then start the stack:
|
||||
|
||||
```bash
|
||||
docker network create dgx_net 2>/dev/null || true
|
||||
cd config
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. Check the services:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8020/health # VoiceClone
|
||||
curl http://localhost:8021/health # VoiceDesign
|
||||
curl http://localhost:8022/health # CustomVoice
|
||||
```
|
||||
|
||||
## Adding VoiceClone voices
|
||||
|
||||
Place reference audio files in `config/speakers/` using this naming convention:
|
||||
|
||||
```
|
||||
EN_M_Speaker_Name.wav # English, Male
|
||||
EN_F_Speaker_Name.wav # English, Female
|
||||
DE_M_Speaker_Name.wav # German, Male
|
||||
```text
|
||||
EN_M_Speaker_Name.wav # English, male
|
||||
EN_F_Speaker_Name.wav # English, female
|
||||
DE_M_Speaker_Name.wav # German, male
|
||||
```
|
||||
|
||||
**Important:** Reference audio must be **5-15 seconds** long. Longer files cause slow inference and poor voice cloning quality.
|
||||
Reference audio should be **5-15 seconds** long. Longer files can slow inference and reduce cloning quality.
|
||||
|
||||
For each audio file, create a matching transcript:
|
||||
|
||||
```
|
||||
```text
|
||||
EN_M_Speaker_Name.reference.txt
|
||||
```
|
||||
|
||||
Or use the auto-transcription script (requires a running Whisper-compatible ASR service):
|
||||
Or use the auto-transcription script with a running Whisper-compatible ASR service:
|
||||
|
||||
```bash
|
||||
python config/auto_transcribe.py --api-url http://localhost:8010/v1/audio/transcriptions
|
||||
```
|
||||
|
||||
The `generate_voices.py` script runs automatically on container startup and creates `voices.json` from your speaker files.
|
||||
`config/generate_voices.py` runs on container startup and creates `config/voices.json` from your speaker files.
|
||||
|
||||
## API Endpoints
|
||||
## VoiceDesign voices
|
||||
|
||||
VoiceDesign does not need reference audio. Define reusable voice personalities in `config/voicedesign_voices.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"narrator": {
|
||||
"instruct": "Warm, confident narrator with a slight British accent",
|
||||
"language": "English"
|
||||
},
|
||||
"assistant_de": {
|
||||
"instruct": "Freundliche, klare Sprecherin, Hochdeutsch, professionell",
|
||||
"language": "German"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then call the VoiceDesign service on port `8021`.
|
||||
|
||||
## CustomVoice speakers
|
||||
|
||||
CustomVoice uses the model's built-in speaker names. Define the speaker IDs you want to expose in `config/customvoice_voices.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Ryan": {
|
||||
"speaker": "Ryan",
|
||||
"language": "English",
|
||||
"instruct": ""
|
||||
},
|
||||
"Ono_Anna": {
|
||||
"speaker": "Ono_Anna",
|
||||
"language": "Japanese",
|
||||
"instruct": ""
|
||||
},
|
||||
"Sohee": {
|
||||
"speaker": "Sohee",
|
||||
"language": "Korean",
|
||||
"instruct": ""
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then call the CustomVoice service on port `8022`.
|
||||
|
||||
## API
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|---|---|---|
|
||||
| `/health` | GET | Health check |
|
||||
| `/v1/audio/speech` | POST | Generate speech (OpenAI-compatible) |
|
||||
| `/v1/models` | GET | List available voices |
|
||||
| `/v1/audio/voices` | GET | List voices (OpenWebUI fallback) |
|
||||
| `/v1/audio/models` | GET | List models (OpenWebUI fallback) |
|
||||
| `/speakers` | GET | List speaker IDs (SillyTavern) |
|
||||
| `/v1/audio/speech` | POST | Generate speech in OpenAI-compatible format |
|
||||
| `/v1/models` | GET | List available voice IDs |
|
||||
| `/v1/audio/voices` | GET | OpenWebUI voice-list fallback |
|
||||
| `/v1/audio/models` | GET | OpenWebUI model-list fallback |
|
||||
| `/speakers` | GET | Speaker IDs for SillyTavern and simple clients |
|
||||
|
||||
### Example
|
||||
### Speech request fields
|
||||
|
||||
| Field | Type | Default | Notes |
|
||||
|---|---|---|---|
|
||||
| `model` | string | `tts-1` | Kept for OpenAI compatibility |
|
||||
| `input` | string | required | Text to synthesize |
|
||||
| `voice` | string | first configured voice | Voice ID from the selected service |
|
||||
| `response_format` | string | `wav` | `wav`, `pcm`, or `mp3` |
|
||||
| `language` | string | voice config | Per-request override for VoiceDesign/CustomVoice |
|
||||
| `instruct` | string | voice config | Per-request style override for VoiceDesign/CustomVoice |
|
||||
| `max_new_tokens` | int | server default | Per-request generation length override |
|
||||
|
||||
WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response.
|
||||
|
||||
### Examples
|
||||
|
||||
VoiceClone on port `8020`:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8020/v1/audio/speech \
|
||||
curl http://localhost:8020/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Hello world!", "voice": "speaker_name", "response_format": "wav"}' \
|
||||
-d '{"model":"tts-1","input":"Hello world!","voice":"EN_M_Speaker_Name","response_format":"wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
|
||||
## OpenWebUI Configuration
|
||||
VoiceDesign on port `8021`:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8021/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"tts-1","input":"Welcome to the show.","voice":"narrator"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
|
||||
Per-request VoiceDesign override:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8021/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "tts-1",
|
||||
"input": "Herzlich willkommen.",
|
||||
"voice": "narrator",
|
||||
"language": "German",
|
||||
"instruct": "Speak slowly and warmly.",
|
||||
"max_new_tokens": 1024
|
||||
}' \
|
||||
--output speech_de.wav
|
||||
```
|
||||
|
||||
CustomVoice on port `8022`:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8022/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"tts-1","input":"This uses a built-in Qwen3-TTS speaker.","voice":"Ryan"}' \
|
||||
--output customvoice.wav
|
||||
```
|
||||
|
||||
Per-request fields win over the JSON voice config entry, so one configured voice can still be adjusted by callers for language, tone, or generation length.
|
||||
|
||||
## Client configuration
|
||||
|
||||
### OpenWebUI
|
||||
|
||||
In OpenWebUI Settings > Audio > Text-to-Speech:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Engine | OpenAI |
|
||||
| URL | `http://faster-qwen3-tts:8000/v1` |
|
||||
| URL | `http://your-host:8020/v1`, `http://your-host:8021/v1`, or `http://your-host:8022/v1` |
|
||||
| API Key | `sk-dummy-key` |
|
||||
| TTS Model | `tts-1` |
|
||||
| TTS Voice | Select from dropdown |
|
||||
|
||||
## Performance
|
||||
### llama-swap or other OpenAI-compatible clients
|
||||
|
||||
On DGX Spark GB10 with the 1.7B model:
|
||||
Point the client's OpenAI-compatible TTS base URL at the service you want:
|
||||
|
||||
| Input | Audio Duration | Generation Time | RTF |
|
||||
|---|---|---|---|
|
||||
| Short sentence | ~2s | ~2.5s | 0.8 |
|
||||
| Medium paragraph | ~7s | ~5.5s | 0.77 |
|
||||
```text
|
||||
http://your-host:8020/v1 # VoiceClone
|
||||
http://your-host:8021/v1 # VoiceDesign
|
||||
http://your-host:8022/v1 # CustomVoice
|
||||
```
|
||||
|
||||
First request is slower due to one-time CUDA graph warmup.
|
||||
## Benchmarking
|
||||
|
||||
## Hardware Requirements
|
||||
Use `config/benchmark_api.py` to verify latency and real-time performance:
|
||||
|
||||
- NVIDIA DGX Spark GB10 (or any ARM64 + Blackwell GPU with CUDA 13)
|
||||
- ~6 GB GPU memory for the 1.7B model
|
||||
- CUDA driver 580+ with CUDA 13.0 support
|
||||
```bash
|
||||
python config/benchmark_api.py --host localhost --port 8021 --runs 5
|
||||
```
|
||||
|
||||
The benchmark reports:
|
||||
|
||||
| Metric | Meaning |
|
||||
|---|---|
|
||||
| TTFA | Time to first audio byte; useful for interactive playback latency |
|
||||
| RTF | Generation time divided by audio duration; lower is better |
|
||||
| Speed | Audio duration divided by generation time; higher than `1.0x` is faster than real time |
|
||||
|
||||
The first request after container startup can be slower because CUDA graph capture runs once during warmup. Later requests should use the captured graph.
|
||||
|
||||
## Performance and memory notes
|
||||
|
||||
- The 1.7B Qwen3-TTS models use about 6 GB of GPU memory each in bfloat16.
|
||||
- The forum playbook shows the three API containers running together on DGX Spark with low visible memory pressure, but exact usage depends on model size, sequence length, and warmup state.
|
||||
- Use the 0.6B Qwen3-TTS variants if you want a lighter multi-service setup.
|
||||
- `--max-seq-len 2048` handles most sentence-style TTS requests. Long-form narration may need `4096`, with more memory required.
|
||||
- Pin services to different GPUs with `NVIDIA_VISIBLE_DEVICES=0`, `NVIDIA_VISIBLE_DEVICES=1`, and so on if your system has more than one GPU.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `503 Model not loaded` | Server still loading or warming up | Wait 30-60 seconds and check container logs |
|
||||
| `404 Voice not found` | Voice ID is not in the JSON config | Check spelling or call `/speakers` |
|
||||
| Very high TTFA | CUDA graph capture failed or fallback path is active | Check logs, reduce `--max-seq-len`, then restart |
|
||||
| MP3 output error | MP3 dependencies are missing or ffmpeg is unavailable | Use `wav`/`pcm` or rebuild the image with MP3 support |
|
||||
| OpenWebUI has no voices | Client cannot read the voice list | Confirm `/v1/models` and `/v1/audio/voices` are reachable from OpenWebUI |
|
||||
|
||||
## Hardware requirements
|
||||
|
||||
- NVIDIA DGX Spark GB10, or another ARM64 + NVIDIA GPU setup with CUDA 13 support.
|
||||
- CUDA driver 580+ with CUDA 13.0 support.
|
||||
- Docker plus NVIDIA Container Toolkit.
|
||||
- Local Qwen3-TTS model weights from Hugging Face.
|
||||
|
||||
## Credits
|
||||
|
||||
- [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) by Andres Marafioti
|
||||
- [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) by Alibaba Qwen team
|
||||
- DGX Spark compatibility fixes by [mARTin-B78](https://github.com/mARTin-B78)
|
||||
- [faster-qwen3-tts](https://github.com/andimarafioti/faster-qwen3-tts) by Andres Marafioti.
|
||||
- [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) by the Alibaba Qwen team.
|
||||
- DGX Spark compatibility, Docker, and OpenAI-compatible API packaging by [mARTin-B78](https://github.com/mARTin-B78).
|
||||
- NVIDIA Developer Forum playbook: [Three times (VoiceClone | VoiceDesign | CustomVoice) - Faster-Qwen3-TTS for NVIDIA DGX Spark (GB10)](https://forums.developer.nvidia.com/t/three-times-voiceclone-voicedesign-customvoice-faster-qwen3-tts-for-nvidia-dgx-spark-gb10/370530).
|
||||
|
||||
## License
|
||||
|
||||
MIT (same as upstream faster-qwen3-tts)
|
||||
MIT (same as upstream faster-qwen3-tts).
|
||||
|
||||
Loading…
Reference in New Issue
Block a user