mARTin-B78/tts-dgx-spark-faster-qwen3-tts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: mARTin-B78:2ade7e1ca6ad6f88f892178e320779662851fc6e
Branches Tags
mARTin-B78:main
..
compare: mARTin-B78:f482d07ad01367d2affbf3ceb992a011a47c8e1c
Branches Tags
mARTin-B78:main

No commits in common. "2ade7e1ca6ad6f88f892178e320779662851fc6e" and "f482d07ad01367d2affbf3ceb992a011a47c8e1c" have entirely different histories.
2ade7e1ca6 ... f482d07ad0

14 changed files with 22 additions and 598 deletions
Show Stats Expand all files Collapse all files

3
.gitignore vendored
View File

@ -21,9 +21,6 @@ config/voices.json*
# Converted private audio (M4A → WAV, generated at runtime)
config/converted/
# Seed finder pre-generated WAV samples (batch output, not source files)
config/seed_samples/
# Editor backup files
*.py~
*.yml~

30
README.md
View File

@ -136,11 +136,8 @@ Or use the auto-transcription script with a running Whisper-compatible ASR servi
python config/auto_transcribe.py --api-url http://localhost:8010/v1/audio/transcriptions
```
`config/generate_voices.py` runs automatically in the background, continuously watching your `speakers` directory. Whenever you add a new `.wav` and `.txt` file, it instantly updates `config/voices.json`. The API server hot-reloads the changes, meaning **you never need to restart the container when adding new voices!**
`config/generate_voices.py` runs on container startup and creates `config/voices.json` from your speaker files.
When you start using a new voice for the first time, the server will automatically do the heavy lifting to extract the voice's acoustic fingerprint (a "speaker embedding") and save it as a `.pt` file in the `config/speakers/` directory. Even better, when the server starts up, it automatically precomputes missing `.pt` files in the background, so your first API requests will be lightning fast. Future requests will instantly load this `.pt` file instead of re-analyzing the audio, which dramatically speeds up Time To First Audio (TTFA).
> **Note:** The generation of the `.pt` embedding is completely deterministic. Running the extraction process twice on the same reference `.wav` and `.txt` will yield the exact same fingerprint, so the resulting voice will not vary between regenerations.
## VoiceDesign voices
VoiceDesign does not need reference audio. Define reusable voice personalities in `config/voicedesign_voices.json`:
@ -343,31 +340,6 @@ The first request after container startup can be slower because CUDA graph captu
## Changelog
### v6.6 — 2026-06-21
**Feature: Eager Background Precomputation of Speaker Embeddings**
- The server now automatically precomputes all missing `.pt` files in the background immediately after startup.
- This ensures all configured voices are pre-warmed and ready to deliver lightning-fast TTFA on the very first request without delaying server startup.
- The lazy-loading mechanism still remains active to instantly handle any new voices hot-reloaded while the server is running.
### v6.5.1 — 2026-06-21
**Documentation Update**
- Added documentation explicitly clarifying that `.pt` speaker embedding generation is fully deterministic and does not produce variable voice characteristics across restarts.
### v6.5 — 2026-06-20
**Feature: True Zero-Downtime Voice Hot-Reloading**
- The server now watches `voices.json` and hot-reloads it automatically when changes are detected.
- Added a background loop in `docker-compose.yml` that continuously runs `generate_voices.py` every 10 seconds.
- You can now drop new `.wav` and `.txt` files into your `speakers/` directory and they will be instantly available via the API without ever restarting the Docker container!
### v6.4 — 2026-06-20
**Feature: Fully Automated Speaker Embeddings (.pt files)**
- Integrated speaker embedding extraction directly into the API server (`openai_server.py`).
- When a new voice is requested for the first time, the server will automatically compute the speaker embedding and save it as a `.pt` file in `/config/speakers/`.
- Future requests for the same voice automatically use the `.pt` file instead of recalculating the prompt from the `.wav` or `.mp3` reference audio.
- This provides the massive TTFA speedup of precomputed embeddings without requiring any manual scripting or configuration.
### v6.3 — 2026-06-20
**Feature: Precomputed Speaker Embeddings (.pt files)**

198
config/find_best_seed.py
View File

@ -1,198 +0,0 @@
#!/usr/bin/env python3
"""
Find the best RNG seed for one or all voices by generating audio samples with
different seeds and saving them as numbered WAV files for comparison.
How it works:
1. For each voice × seed, temporarily sets that seed in voices.json
2. Calls the running server (hot-reload picks it up automatically)
3. Saves audio as <out-dir>/<voice-name>/seed_<N>.wav
4. Restores voices.json to its original state when done
Then listen to the files and pick the seed you prefer.
Add it to voices.json: "seed": <number>
Usage:
# All voices, seeds 1–15:
python find_best_seed.py --all-voices --range 1 15 --port 8020
# Single voice:
python find_best_seed.py --voice EN_F_NatashaNeural --range 1 20
python find_best_seed.py --voice DE_5_28 --seeds 1 7 42 100
"""
import argparse
import json
import os
import shutil
import sys
import time
import requests
VOICES_JSON = "/config/voices.json"
DE_TEXT = (
"Die 3.500 neuen High-End Geräte für das Server-Update benötigen eine "
"außergewöhnlich starke Kühlung und regelmäßige Maßnahmen, um die Performance "
"bei großer Last zu gewährleisten."
)
EN_TEXT = (
"The system administrator successfully configured the customized Docker stacks "
"and benchmarked the inference engines at exactly 8:45 AM."
)
MIXED_TEXT = DE_TEXT + " - " + EN_TEXT
def load_voices() -> dict:
with open(VOICES_JSON, "r", encoding="utf-8") as f:
return json.load(f)
def save_voices(voices: dict) -> None:
with open(VOICES_JSON, "w", encoding="utf-8") as f:
json.dump(voices, f, indent=2, ensure_ascii=False)
def generate(voice: str, text: str, host: str, port: int, timeout: int = 90) -> bytes:
url = f"http://{host}:{port}/v1/audio/speech"
resp = requests.post(
url,
json={"model": "tts-1", "input": text, "voice": voice, "response_format": "wav"},
timeout=timeout,
stream=False,
)
resp.raise_for_status()
return resp.content
def pick_text(voice_name: str, override: str | None) -> str:
if override:
return override
name_lower = voice_name.lower()
if name_lower.startswith("de_"):
return MIXED_TEXT
if name_lower.startswith("en_") or name_lower.startswith("gb_"):
return EN_TEXT
return MIXED_TEXT
def run_voice(voice_name: str, voices: dict, seeds: list[int],
text: str, host: str, port: int, out_dir: str) -> list[tuple]:
voice_dir = os.path.join(out_dir, voice_name)
os.makedirs(voice_dir, exist_ok=True)
results = []
for seed in seeds:
voices[voice_name]["seed"] = seed
save_voices(voices)
time.sleep(0.3) # let hot-reload detect mtime change
out_path = os.path.join(voice_dir, f"seed_{seed:05d}.wav")
print(f" seed {seed:5d} → ", end="", flush=True)
try:
t0 = time.time()
wav = generate(voice_name, text, host, port)
elapsed = time.time() - t0
with open(out_path, "wb") as f:
f.write(wav)
print(f"OK ({elapsed:.1f}s)")
results.append((seed, out_path, None))
except Exception as e:
print(f"FAILED: {e}")
results.append((seed, None, str(e)))
# Remove the temporary seed so the voice returns to its auto-seed
voices[voice_name].pop("seed", None)
save_voices(voices)
return results
def main():
p = argparse.ArgumentParser(description="Compare seeds for one or all voices")
target = p.add_mutually_exclusive_group(required=True)
target.add_argument("--voice", help="Single voice name")
target.add_argument("--all-voices", action="store_true", help="Run for every voice in voices.json")
p.add_argument("--seeds", type=int, nargs="+", help="Explicit list of seeds to try")
p.add_argument("--range", type=int, nargs=2, metavar=("START", "END"),
help="Try seeds START through END (inclusive). Default: 1–15")
p.add_argument("--text", default=None,
help="Override text (default: auto-picks DE/EN/mixed based on voice name)")
p.add_argument("--host", default="localhost")
p.add_argument("--port", type=int, default=8020)
p.add_argument("--out-dir", default="./seed_samples",
help="Root output directory (voice subdirs created inside)")
args = p.parse_args()
seeds = list(args.seeds or [])
if args.range:
seeds += list(range(args.range[0], args.range[1] + 1))
if not seeds:
seeds = list(range(1, 16)) # default 1–15
seeds = sorted(set(seeds))
voices = load_voices()
if args.voice:
voice_names = [args.voice]
if args.voice not in voices:
print(f"ERROR: voice {args.voice!r} not in voices.json", file=sys.stderr)
sys.exit(1)
else:
voice_names = list(voices.keys())
os.makedirs(args.out_dir, exist_ok=True)
# Single backup at the start; we restore on every error/exit
backup = VOICES_JSON + ".seed_backup"
shutil.copy2(VOICES_JSON, backup)
print(f"voices : {len(voice_names)}")
print(f"seeds : {seeds}")
print(f"output : {os.path.abspath(args.out_dir)}/")
print(f"total : ~{len(voice_names) * len(seeds)} requests\n")
summary: dict[str, list] = {}
try:
for i, voice_name in enumerate(voice_names, 1):
text = pick_text(voice_name, args.text)
print(f"[{i}/{len(voice_names)}] {voice_name}")
print(f" text: {text[:90]}{'...' if len(text) > 90 else ''}")
results = run_voice(voice_name, voices, seeds, text, args.host, args.port, args.out_dir)
summary[voice_name] = results
ok = sum(1 for _, p, _ in results if p)
print(f" → {ok}/{len(seeds)} OK\n")
except KeyboardInterrupt:
print("\n\nInterrupted — restoring voices.json...")
finally:
shutil.copy2(backup, VOICES_JSON)
os.remove(backup)
print("voices.json restored.")
# Write summary
summary_path = os.path.join(args.out_dir, "summary.txt")
with open(summary_path, "w") as f:
f.write(f"Seed samples — {len(voice_names)} voices, seeds {seeds}\n")
f.write("=" * 60 + "\n\n")
for voice_name, results in summary.items():
ok = [(s, path) for s, path, err in results if path]
failed = [(s, err) for s, path, err in results if not path]
f.write(f"{voice_name}:\n")
for s, path in ok:
f.write(f" seed {s:5d} {path}\n")
for s, err in failed:
f.write(f" seed {s:5d} FAILED: {err}\n")
f.write("\n")
total_ok = sum(1 for r in summary.values() for _, p, _ in r if p)
total = sum(len(r) for r in summary.values())
print(f"\nDone: {total_ok}/{total} samples generated.")
print(f"Summary written to {summary_path}")
print(f"\nListen to the WAV files, then add \"seed\": <number> to your chosen voices in voices.json")
if __name__ == "__main__":
main()

7
config/generate_voices.py
View File

@ -120,10 +120,7 @@ for scan_dir in SCAN_DIRS:
voices[voice_id] = entry
if voices != _existing:
with open(output_file, "w", encoding="utf-8") as f:
with open(output_file, "w", encoding="utf-8") as f:
json.dump(voices, f, indent=2, ensure_ascii=False)
print(f"Success! Generated voices.json with {len(voices)} mapped voices.")
else:
pass # No changes, do not update mtime
print(f"Success! Generated voices.json with {len(voices)} mapped voices.")

2
config/run_customvoice_server.py
View File

@ -252,8 +252,6 @@ def main():
parser.add_argument("--device", default="cuda")
parser.add_argument("--max-seq-len", type=int, default=2048)
args = parser.parse_args()
# Force a smaller max_seq_len to save VRAM and prevent OOM
args.max_seq_len = 1024
DEFAULT_MAX_NEW_TOKENS = args.max_seq_len
_load_model_kwargs = args

99
config/run_server.py
View File

@ -17,12 +17,10 @@ Startup:
import asyncio
import logging
import sys
import os
import json
import threading
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from fastapi import FastAPI
from fastapi.responses import JSONResponse
# Point Python to the app directory inside the container
@ -64,34 +62,10 @@ def _do_warmup():
logger.warning("Warmup failed (non-fatal): %s", exc)
def _precompute_all_embeddings():
"""Background task to precompute all missing embeddings to avoid lazy-load delay."""
model = openai_server.tts_model
voices = openai_server.voices
if not model or not voices:
return
for voice_name, voice_cfg in list(voices.items()):
# skip if already precomputed
spk_emb_path = voice_cfg.get("speaker_embeddings") or voice_cfg.get("speaker embeddings")
if spk_emb_path and os.path.isfile(spk_emb_path):
continue
logger.info("Background precomputing embedding for %r...", voice_name)
# We must lock the model to prevent concurrent generation with incoming requests
with openai_server._model_lock:
try:
# _load_voice_clone_prompt updates voice_cfg in place and saves the .pt
openai_server._load_voice_clone_prompt(voice_cfg, voice_name, model)
except Exception as e:
logger.error("Failed to background precompute for %r: %s", voice_name, e)
@asynccontextmanager
async def lifespan(app: FastAPI):
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, _do_warmup)
loop.run_in_executor(None, _precompute_all_embeddings)
yield
@ -133,76 +107,5 @@ async def get_speakers():
async def options_handler(path: str):
return JSONResponse(content={'status': 'ok'})
_voices_lock = threading.Lock()
@openai_server.app.post('/voice-seed')
async def set_voice_seed(request: Request):
"""Set (or clear) the seed for a voice in voices.json.
Body: {"voice": "EN_F_NatashaNeural", "seed": 7}
To remove a seed: {"voice": "EN_F_NatashaNeural", "seed": null}
"""
data = await request.json()
voice_name = data.get("voice")
seed = data.get("seed")
if not voice_name:
raise HTTPException(status_code=400, detail="'voice' field is required")
voices_path = '/config/voices.json'
with _voices_lock:
try:
with open(voices_path, 'r', encoding='utf-8') as f:
voices = json.load(f)
except FileNotFoundError:
raise HTTPException(status_code=404, detail="voices.json not found")
if voice_name not in voices:
raise HTTPException(status_code=404, detail=f"Voice {voice_name!r} not found")
if seed is None:
voices[voice_name].pop("seed", None)
else:
try:
voices[voice_name]["seed"] = int(seed)
except (TypeError, ValueError):
raise HTTPException(status_code=400, detail="'seed' must be an integer or null")
with open(voices_path, 'w', encoding='utf-8') as f:
json.dump(voices, f, indent=2, ensure_ascii=False)
return JSONResponse({"ok": True, "voice": voice_name, "seed": seed})
_SEED_SAMPLES_DIR = '/config/seed_samples'
@openai_server.app.get('/seed-samples/{voice_name}')
async def list_seed_samples(voice_name: str):
"""Return a sorted list of seed numbers for which a pre-generated WAV exists."""
import re as _re
voice_dir = os.path.join(_SEED_SAMPLES_DIR, voice_name)
if not os.path.isdir(voice_dir):
return JSONResponse({"seeds": []})
seeds = []
for fname in os.listdir(voice_dir):
m = _re.match(r'^seed_(\d+)\.wav$', fname)
if m:
seeds.append(int(m.group(1)))
seeds.sort()
return JSONResponse({"seeds": seeds})
@openai_server.app.get('/seed-sample/{voice_name}/{seed}')
async def get_seed_sample(voice_name: str, seed: int):
"""Serve a pre-generated seed WAV file."""
from fastapi.responses import FileResponse
path = os.path.join(_SEED_SAMPLES_DIR, voice_name, f'seed_{seed:05d}.wav')
if not os.path.isfile(path):
raise HTTPException(status_code=404, detail=f"No sample for seed {seed}")
return FileResponse(path, media_type='audio/wav')
if __name__ == '__main__':
openai_server.main()

106
config/seed_samples_progress.log
View File

@ -1,106 +0,0 @@
voices : 96
seeds : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
output : /config/seed_samples/
total : ~1440 requests
[1/96] EN_F_NatashaNeural
text: The system administrator successfully configured the customized Docker stacks and benchmar...
seed 1 → OK (7.5s)
seed 2 → OK (6.4s)
seed 3 → OK (6.6s)
seed 4 → OK (6.9s)
seed 5 → OK (6.2s)
seed 6 → OK (6.2s)
seed 7 → OK (5.7s)
seed 8 → OK (6.0s)
seed 9 → OK (9.1s)
seed 10 → OK (6.1s)
seed 11 → OK (5.9s)
seed 12 → OK (6.2s)
seed 13 → OK (6.3s)
seed 14 → OK (5.8s)
seed 15 → OK (6.1s)
→ 15/15 OK
[2/96] EN_M_WilliamNeural
text: The system administrator successfully configured the customized Docker stacks and benchmar...
seed 1 → OK (5.6s)
seed 2 → OK (5.3s)
seed 3 → OK (5.2s)
seed 4 → OK (5.3s)
seed 5 → OK (5.4s)
seed 6 → OK (5.8s)
seed 7 → OK (5.3s)
seed 8 → OK (5.2s)
seed 9 → OK (5.1s)
seed 10 → OK (5.3s)
seed 11 → OK (5.4s)
seed 12 → OK (5.3s)
seed 13 → OK (5.2s)
seed 14 → OK (5.2s)
seed 15 → OK (5.4s)
→ 15/15 OK
[3/96] DE_5_28
text: Die 3.500 neuen High-End Geräte für das Server-Update benötigen eine außergewöhnlich stark...
seed 1 → OK (12.6s)
seed 2 → OK (13.9s)
seed 3 → OK (12.6s)
seed 4 → OK (13.4s)
seed 5 → OK (12.9s)
seed 6 → OK (13.0s)
seed 7 → OK (12.4s)
seed 8 → OK (13.1s)
seed 9 → OK (13.2s)
seed 10 → OK (13.6s)
seed 11 → OK (12.5s)
seed 12 → OK (13.7s)
seed 13 → OK (13.3s)
seed 14 → OK (13.0s)
seed 15 → OK (14.5s)
→ 15/15 OK
[4/96] DE_Aliya_meine_Frau
text: Die 3.500 neuen High-End Geräte für das Server-Update benötigen eine außergewöhnlich stark...
seed 1 → OK (13.5s)
seed 2 → OK (12.9s)
seed 3 → OK (14.4s)
seed 4 → OK (13.6s)
seed 5 → OK (14.3s)
seed 6 → OK (12.7s)
seed 7 → OK (13.7s)
seed 8 → OK (14.1s)
seed 9 → OK (14.5s)
seed 10 → OK (13.3s)
seed 11 → OK (14.6s)
seed 12 → OK (13.8s)
seed 13 → OK (13.1s)
seed 14 → OK (13.7s)
seed 15 → OK (12.6s)
→ 15/15 OK
[5/96] DE_Anke_Harnack_U_Bahn_Hamburg_Stimme
text: Die 3.500 neuen High-End Geräte für das Server-Update benötigen eine außergewöhnlich stark...
seed 1 → OK (15.6s)
seed 2 → OK (14.4s)
seed 3 → OK (14.6s)
seed 4 → OK (15.8s)
seed 5 → OK (14.0s)
seed 6 → OK (29.5s)
seed 7 → OK (46.9s)
seed 8 → OK (75.1s)
seed 9 → FAILED: HTTPConnectionPool(host='localhost', port=8000): Read timed out.
seed 10 → OK (60.6s)
seed 11 → OK (14.7s)
seed 12 → OK (15.1s)
seed 13 → OK (15.0s)
seed 14 → OK (14.8s)
seed 15 → OK (14.7s)
→ 14/15 OK
[6/96] DE_Catrinja
text: Die 3.500 neuen High-End Geräte für das Server-Update benötigen eine außergewöhnlich stark...
seed 1 → OK (12.4s)
seed 2 → OK (12.7s)
seed 3 → OK (13.3s)
seed 4 →

2
docker/docker-compose.simple.yml
View File

@ -19,7 +19,7 @@ services:
- ../config/speakers:/voices:ro
command: >
/bin/bash -c "
(while true; do python3 /config/generate_voices.py; sleep 10; done) &
python3 /config/generate_voices.py &&
python3 /config/run_server.py
--model /models/Qwen3-TTS
--voices /config/voices.json

4
docker/docker-compose.yml
View File

@ -36,7 +36,7 @@ services:
- /path/to/active_voices:/voices:ro
command: >
/bin/bash -c "
(while true; do python3 /config/generate_voices.py; sleep 10; done) &
python3 /config/generate_voices.py &&
python3 /config/run_server.py
--model /models/Qwen3-TTS
--voices /config/voices.json
@ -134,7 +134,7 @@ services:
- /path/to/active_voices:/voices:ro
command: >
/bin/bash -c "
(while true; do python3 /config/generate_voices.py; sleep 10; done) &
python3 /config/generate_voices.py &&
python3 /config/run_server.py
--model /models/Qwen3-TTS
--voices /config/voices.json

163
patches/openai_server.patch
View File

@ -1,119 +1,25 @@
--- /tmp/upstream_openai_server.py 2026-06-21 14:29:34.858114787 +0200
+++ build/examples/openai_server.py 2026-06-21 14:26:33.557536313 +0200
@@ -36,6 +36,7 @@
"""
import argparse
import asyncio
+import hashlib
import io
import json
import logging
@@ -66,10 +67,29 @@
tts_model = None
voices: dict = {}
+voices_file_path: Optional[str] = None
+last_voices_mtime: float = 0.0
default_voice: Optional[str] = None
SAMPLE_RATE = 24000 # updated once the model loads
_model_lock = threading.Lock() # prevent concurrent GPU inference
+
+def _voice_seed(voice_name: str) -> int:
+ """Return a stable per-voice seed derived from the voice name.
+
+ Used as the default when no explicit 'seed' is set in voices.json.
+ MD5 is used only for its stable byte output — not for security.
+ """
+ return int(hashlib.md5(voice_name.encode()).hexdigest(), 16) % (2 ** 31)
+
+
+def _seed_rng(seed: int) -> None:
+ """Seed PyTorch CPU and CUDA RNGs for reproducible sampling."""
+ torch.manual_seed(seed)
+ if torch.cuda.is_available():
+ torch.cuda.manual_seed_all(seed)
+
+
# ---------------------------------------------------------------------------
# Request / response models
# ---------------------------------------------------------------------------
@@ -145,6 +165,21 @@
def resolve_voice(voice_name: str) -> dict:
"""Return voice config dict or fall back to default, else raise 400."""
+ global voices, last_voices_mtime
+ voice_name = voice_name.strip()
+
+ # Hot-reload voices.json if it was modified
+ if voices_file_path and os.path.exists(voices_file_path):
+ try:
+ current_mtime = os.path.getmtime(voices_file_path)
+ if current_mtime > last_voices_mtime:
+ with open(voices_file_path, "r", encoding="utf-8") as f:
+ voices = json.load(f)
+ last_voices_mtime = current_mtime
+ logger.info("Hot-reloaded %d voices from %s", len(voices), voices_file_path)
+ except Exception as e:
+ logger.warning("Failed to hot-reload voices.json: %s", e)
+
if voice_name in voices:
return voices[voice_name]
if default_voice and default_voice in voices:
@@ -168,7 +203,47 @@
diff --git a/examples/openai_server.py b/examples/openai_server.py
index 61047ea..2f95bbd 100644
--- a/examples/openai_server.py
+++ b/examples/openai_server.py
@@ -169,6 +169,16 @@ def resolve_voice(voice_name: str) -> dict:
# ---------------------------------------------------------------------------
-async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, None]:
+def _load_voice_clone_prompt(voice_cfg: dict, voice_name: str, tts_model):
+def _load_voice_clone_prompt(voice_cfg: dict):
+ spk_emb_path = voice_cfg.get("speaker_embeddings") or voice_cfg.get("speaker embeddings")
+ if spk_emb_path and os.path.isfile(spk_emb_path):
+ try:
+ return torch.load(spk_emb_path, map_location="cpu", weights_only=False)
+ except Exception as e:
+ logger.error("Failed to load speaker embeddings from %s: %s", spk_emb_path, e)
+
+ # Auto-generate if missing
+ ref_audio = voice_cfg.get("ref_audio")
+ if not ref_audio or not os.path.isfile(ref_audio):
+ return None
+
+ logger.info("Precomputing and saving speaker embedding for voice %r...", voice_name)
+ try:
+ ref_text = voice_cfg.get("ref_text", "")
+ # generate prompt using the model's built-in helper
+ prompt_items = tts_model.model.create_voice_clone_prompt(ref_audio, [ref_text])
+ vcp = tts_model.model._prompt_items_to_voice_clone_prompt(prompt_items)
+
+ # save it to the speakers directory
+ # If the file path is already in voice_cfg but doesn't exist, use that, otherwise generate a path
+ if spk_emb_path and not os.path.exists(spk_emb_path) and spk_emb_path.endswith('.pt'):
+ pt_path = spk_emb_path
+ else:
+ pt_path = f"/config/speakers/{voice_name}.pt"
+
+ os.makedirs(os.path.dirname(pt_path), exist_ok=True)
+ torch.save(vcp, pt_path)
+ logger.info("Saved speaker embedding to %s", pt_path)
+
+ # update in memory so future requests skip generating
+ voice_cfg["speaker_embeddings"] = pt_path
+
+ return vcp
+ except Exception as e:
+ logger.error("Failed to precompute speaker embedding: %s", e)
+ return None
+
+
+async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str) -> AsyncGenerator[bytes, None]:
async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, None]:
"""
Run generate_voice_clone_streaming in a background thread and yield
raw PCM bytes for each chunk as they arrive.
@@ -179,13 +254,19 @@
def producer():
try:
with _model_lock:
+ _seed_rng(voice_cfg.get("seed", _voice_seed(voice_name)))
@@ -183,11 +193,15 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
text=text,
language=voice_cfg.get("language", "Auto"),
@ -121,9 +27,9 @@
+ ref_audio=voice_cfg.get("ref_audio"),
ref_text=voice_cfg.get("ref_text", ""),
chunk_size=voice_cfg.get("chunk_size", 12),
instruct=voice_cfg.get("instruct"),
- non_streaming_mode=False,
+ instruct=voice_cfg.get("instruct"),
+ voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
+ voice_clone_prompt=_load_voice_clone_prompt(voice_cfg),
+ non_streaming_mode=True,
+ temperature=voice_cfg.get("temperature", 0.8),
+ top_k=voice_cfg.get("top_k", 50),
@ -131,19 +37,15 @@
):
q.put(chunk)
except Exception as exc:
@@ -244,11 +325,18 @@
def _generate():
with _model_lock:
+ _seed_rng(voice_cfg.get("seed", _voice_seed(req.voice)))
@@ -249,9 +263,14 @@ async def create_speech(req: SpeechRequest):
return tts_model.generate_voice_clone(
text=req.input,
language=voice_cfg.get("language", "Auto"),
- ref_audio=voice_cfg["ref_audio"],
+ ref_audio=voice_cfg.get("ref_audio"),
ref_text=voice_cfg.get("ref_text", ""),
+ instruct=voice_cfg.get("instruct"),
+ voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model),
instruct=voice_cfg.get("instruct"),
+ voice_clone_prompt=_load_voice_clone_prompt(voice_cfg),
+ non_streaming_mode=True,
+ temperature=voice_cfg.get("temperature", 0.8),
+ top_k=voice_cfg.get("top_k", 50),
@ -151,42 +53,3 @@
)
audio_arrays, sr = await loop.run_in_executor(None, _generate)
@@ -259,7 +347,7 @@
async def audio_stream():
if fmt == "wav":
yield _wav_header(SAMPLE_RATE) # stream with unknown data length
- async for raw_chunk in _stream_chunks(voice_cfg, req.input):
+ async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice):
yield raw_chunk
return StreamingResponse(audio_stream(), media_type=content_type)
@@ -306,16 +394,20 @@
p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)")
p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)")
p.add_argument("--device", default="cuda", help="Torch device (default: cuda)")
+ p.add_argument("--max-seq-len", type=int, default=4096, help="Max sequence length for CUDA graph static cache (default: 4096)")
return p.parse_args()
def main():
- global tts_model, voices, default_voice, SAMPLE_RATE
+ global tts_model, voices, voices_file_path, last_voices_mtime, default_voice, SAMPLE_RATE
args = _parse_args()
# Build voice registry
if args.voices:
+ voices_file_path = args.voices
+ if os.path.exists(args.voices):
+ last_voices_mtime = os.path.getmtime(args.voices)
with open(args.voices) as f:
voices = json.load(f)
default_voice = next(iter(voices))
@@ -344,6 +436,7 @@
args.model,
device=args.device,
dtype=torch.bfloat16,
+ max_seq_len=args.max_seq_len,
)
SAMPLE_RATE = tts_model.sample_rate
logger.info("Model ready. Sample rate: %d Hz", SAMPLE_RATE)

BIN
samples/speakers/DE_F_Sample.mp3
View File

Binary file not shown.

1
samples/speakers/DE_F_Sample.reference.txt
View File

@ -1 +0,0 @@
Dies ist ein deutsches Referenzsprachbeispiel. Das Wetter ist heute wunderschön, mit klarem blauen Himmel.

BIN
samples/speakers/EN_F_Sample.mp3
View File

Binary file not shown.

1
samples/speakers/EN_F_Sample.reference.txt
View File

@ -1 +0,0 @@
This is an English reference voice sample. The weather is beautiful today, with clear blue skies.