"""TTS Voice Creator - Clone and Design — FastAPI backend""" from __future__ import annotations import asyncio import base64 import glob import ipaddress import io import json import logging import os import re import shutil import socket import struct import subprocess import tempfile import time import uuid import wave from datetime import datetime, timezone from html import unescape from pathlib import Path from typing import AsyncGenerator from urllib.parse import quote, urljoin, urlparse, urlsplit import requests from fastapi import FastAPI, File, Form, HTTPException, Request, UploadFile from fastapi.responses import FileResponse, Response, StreamingResponse from fastapi.staticfiles import StaticFiles from pydub import AudioSegment logger = logging.getLogger("uvicorn.error") # ── Boot-time defaults ──────────────────────────────────────────────────────── _VOICES_DIR_DEFAULT = os.environ.get("VOICES_DIR", "/voices") _OUTPUT_DIR_DEFAULT = os.environ.get("OUTPUT_DIR", "/voices/active_voices") _WHISPER_DEFAULT = os.environ.get("WHISPER_URL", "http://host.docker.internal:8010") _TTS_DEFAULT = os.environ.get("TTS_URL", "http://host.docker.internal:8020") _TTS_STREAM_DEFAULT = os.environ.get("TTS_STREAM_URL", "http://host.docker.internal:8023") _CUSTOMVOICE_DEFAULT = os.environ.get("CUSTOMVOICE_URL", "http://host.docker.internal:8022") _VOICE_DESIGN_DEFAULT = os.environ.get("VOICE_DESIGN_URL", "http://host.docker.internal:8021") _NVIDIA_ROUTER_DEFAULT = os.environ.get("NVIDIA_SPEECH_ROUTER_URL", "http://host.docker.internal:8090") _NVIDIA_TTS_DEFAULT = os.environ.get("NVIDIA_MAGPIE_TTS_URL", "http://host.docker.internal:8091") _NVIDIA_ASR_DEFAULT = os.environ.get("NVIDIA_PARAKEET_ASR_URL", "http://host.docker.internal:8092") _NVIDIA_CLONE_DEFAULT = os.environ.get("NVIDIA_TTS_CLONE_URL", "http://host.docker.internal:8093") _NVIDIA_ZEROSHOT_DEFAULT = os.environ.get("NVIDIA_ZEROSHOT_TTS_URL", _NVIDIA_CLONE_DEFAULT) _NVIDIA_FLOW_DEFAULT = os.environ.get("NVIDIA_FLOW_TTS_URL", "http://host.docker.internal:8094") _FASTER_WHISPER_DEFAULT = os.environ.get("FASTER_WHISPER_URL", "http://host.docker.internal:8000") _WHISPER_CPP_DEFAULT = os.environ.get("WHISPER_CPP_URL", "http://host.docker.internal:8080") _GROQ_STT_ENDPOINT = "https://api.groq.com/openai/v1" _KOKORO_DEFAULT = os.environ.get("KOKORO_URL", "http://host.docker.internal:8880/v1") _VIBEVOICE_DEFAULT = os.environ.get("VIBEVOICE_URL", "http://192.168.178.8:8027") _XTTS_DEFAULT = os.environ.get("XTTS_URL", "http://host.docker.internal:8024") _TTS_CONTAINER = os.environ.get("TTS_CONTAINER_NAME", "faster-qwen3-tts") _TTS_CONTAINERS_RAW = os.environ.get("TTS_CONTAINER_NAMES", "") # comma-separated override _VOICE_DESIGN_MODEL = os.environ.get("VOICE_DESIGN_MODEL", "Qwen3-TTS-12Hz-1.7B-VoiceDesign") _VOICE_TARGET_DBFS = float(os.environ.get("VOICE_TARGET_DBFS", "-20.0")) _VOICE_PEAK_DBFS = float(os.environ.get("VOICE_PEAK_DBFS", "-1.0")) _MAX_UPLOAD_BYTES = int(os.environ.get("MAX_UPLOAD_MB", "1024")) * 1024 * 1024 _MAX_PICTURE_BYTES = int(os.environ.get("MAX_PICTURE_MB", "10")) * 1024 * 1024 _MAX_SOUND_BYTES = int(os.environ.get("MAX_SOUND_MB", "25")) * 1024 * 1024 _MAX_TTS_OUTPUT_SECONDS = float(os.environ.get("MAX_TTS_OUTPUT_SECONDS", "30")) _STT_REQUEST_TIMEOUT = float(os.environ.get("STT_REQUEST_TIMEOUT", os.environ.get("REQUEST_TIMEOUT", "900"))) _BENCHMARK_TEXT = os.environ.get("VOICE_BENCHMARK_TEXT", "This is a short realtime voice benchmark.") _BENCHMARK_SENTENCES = [ ("short", _BENCHMARK_TEXT), ("medium", "The quick brown fox jumps over the lazy dog near the river bank."), ( "long", "Artificial intelligence is transforming the way we interact with technology. " "From voice assistants to autonomous vehicles, machine learning models are becoming " "an integral part of everyday life.", ), ] _ALLOW_PRIVATE_DOWNLOADS = os.environ.get("ALLOW_PRIVATE_DOWNLOADS", "").lower() in {"1", "true", "yes"} def _default_config_dir() -> Path: env_dir = os.environ.get("CONFIG_DIR") if env_dir: return Path(env_dir) current = Path("/home/app/.config/tts-voice-creator") legacy = Path("/home/app/.config/voice-clone-factory") if legacy.exists(): legacy.mkdir(parents=True, exist_ok=True) if current.exists(): for name in ("settings.json", "voice_design_presets.json", "tts_routes.json"): src, dst = current / name, legacy / name if src.exists() and not dst.exists(): try: shutil.copy2(src, dst) except Exception as exc: logger.warning("Could not migrate config %s to mounted config dir: %s", name, exc) return legacy return current CONFIG_DIR = _default_config_dir() CONFIG_FILE = CONFIG_DIR / "settings.json" DESIGN_PRESETS_FILE = CONFIG_DIR / "voice_design_presets.json" TTS_ROUTES_FILE = CONFIG_DIR / "tts_routes.json" # ── Temp file registry ──────────────────────────────────────────────────────── TEMP_DIR = Path(tempfile.mkdtemp(prefix="vcf_")) _registry: dict[str, Path] = {} app = FastAPI(title="TTS Voice Creator - Clone and Design") # ── In-memory log buffer ─────────────────────────────────────────────────────── _LOG_BUFFER_MAX = 400 _log_buffer: list[dict] = [] class _BufferHandler(logging.Handler): def emit(self, record: logging.LogRecord) -> None: try: _log_buffer.insert(0, { "ts": datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(), "level": record.levelname, "name": record.name, "msg": record.getMessage(), }) del _log_buffer[_LOG_BUFFER_MAX:] except Exception: pass _buf_handler = _BufferHandler() _buf_handler.setLevel(logging.DEBUG) logging.getLogger().addHandler(_buf_handler) _ROUTING_LOG_MAX = int(os.environ.get("TTS_ROUTING_LOG_MAX", "120")) _routing_log: list[dict] = [] def _routing_log_add(**entry) -> None: item = { "ts": datetime.now(timezone.utc).isoformat(), **entry, } _routing_log.insert(0, item) del _routing_log[_ROUTING_LOG_MAX:] def _routing_log_request( request: Request, *, status: str, app: str, requested_voice: str, routed_voice: str, backend: str, route: dict | None, response_format: str, text: str, **extra, ) -> None: route = route or {} _routing_log_add( kind="proxy", status=status, app=app, requested_voice=requested_voice, routed_voice=routed_voice, backend=backend, language=str(route.get("detected_language", "")) or (_detect_text_language(text) if text else ""), matched=bool(route), route_id=str(route.get("id", "")), response_format=response_format, text_preview=text[:160], client=request.client.host if request.client else "", user_agent=str(request.headers.get("user-agent", ""))[:160], **extra, ) def _safe_child_path(root: Path, candidate: Path) -> Path: root_resolved = root.resolve() candidate_resolved = candidate.resolve() try: candidate_resolved.relative_to(root_resolved) except ValueError: raise HTTPException(403, "Access denied") return candidate_resolved def _validate_http_url(raw: str, *, allow_private: bool = True) -> str: raw = str(raw or "").strip() if not raw: raise HTTPException(400, "URL is required") parts = urlsplit(raw) if parts.scheme not in {"http", "https"}: raise HTTPException(400, "Only http:// and https:// URLs are allowed") if not parts.hostname: raise HTTPException(400, "URL must include a hostname") if parts.username or parts.password: raise HTTPException(400, "URLs with embedded credentials are not allowed") if allow_private: return raw try: infos = socket.getaddrinfo(parts.hostname, parts.port or (443 if parts.scheme == "https" else 80), type=socket.SOCK_STREAM) except socket.gaierror: raise HTTPException(400, "URL hostname could not be resolved") for info in infos: ip = ipaddress.ip_address(info[4][0]) if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast or ip.is_reserved: raise HTTPException(400, "Private, local, and reserved network URLs are not allowed for downloads") return raw def _copy_limited(src, dest, limit: int) -> int: total = 0 while True: chunk = src.read(1024 * 1024) if not chunk: break total += len(chunk) if total > limit: raise HTTPException(413, "Uploaded file is too large") dest.write(chunk) return total def _decode_chunked_bytes(data: bytes) -> bytes: result = bytearray() pos = 0 while pos < len(data): end = data.find(b"\r\n", pos) if end < 0: break try: size = int(data[pos:end].split(b";")[0].strip(), 16) except ValueError: break if size == 0: break pos = end + 2 result.extend(data[pos:pos + size]) pos += size + 2 return bytes(result) def _docker_get_json(path: str) -> tuple[int, dict | list | None]: sock_path = os.environ.get("DOCKER_SOCKET", "/var/run/docker.sock") if not Path(sock_path).exists(): raise RuntimeError(f"Docker socket not found: {sock_path}") request = ( f"GET {path} HTTP/1.1\r\n" "Host: docker\r\n" "Connection: close\r\n\r\n" ).encode("utf-8") chunks: list[bytes] = [] with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: sock.settimeout(10) sock.connect(sock_path) sock.sendall(request) while True: data = sock.recv(65536) if not data: break chunks.append(data) raw = b"".join(chunks) header_end = raw.find(b"\r\n\r\n") if header_end < 0: raise RuntimeError("Invalid Docker HTTP response") header_str = raw[:header_end].decode("utf-8", errors="replace") body_bytes = raw[header_end + 4:] status_line = header_str.splitlines()[0] m = re.match(r"HTTP/\S+\s+(\d+)", status_line) if not m: raise RuntimeError(f"Invalid Docker status line: {status_line!r}") code = int(m.group(1)) if "transfer-encoding: chunked" in header_str.lower(): body_bytes = _decode_chunked_bytes(body_bytes) if not body_bytes.strip(): return code, None return code, json.loads(body_bytes) def _docker_post(path: str) -> tuple[int, str]: sock_path = os.environ.get("DOCKER_SOCKET", "/var/run/docker.sock") if not Path(sock_path).exists(): raise RuntimeError(f"Docker socket not found: {sock_path}") request = ( f"POST {path} HTTP/1.1\r\n" "Host: docker\r\n" "Content-Length: 0\r\n" "Connection: close\r\n\r\n" ).encode("utf-8") chunks = [] with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: sock.settimeout(20) sock.connect(sock_path) sock.sendall(request) while True: data = sock.recv(8192) if not data: break chunks.append(data) raw = b"".join(chunks).decode("utf-8", errors="replace") status_line = raw.splitlines()[0] if raw else "" match = re.match(r"HTTP/\S+\s+(\d+)", status_line) if not match: raise RuntimeError("Invalid Docker API response") return int(match.group(1)), raw # ── Settings ────────────────────────────────────────────────────────────────── _SETTINGS_KEYS = { "whisper_url", "tts_url", "tts_stream_url", "tts_stream_mode", "tts_backend", "output_dir", "voices_scan_dir", "voice_design_url", "customvoice_url", "nvidia_router_url", "nvidia_tts_url", "nvidia_asr_url", "nvidia_clone_url", "nvidia_zeroshot_url", "nvidia_flow_url", "faster_whisper_url", "whisper_cpp_url", "groq_api_key", "kokoro_url", "vibevoice_url", "xtts_url", "whisper_api_key", "tts_api_key", "voice_design_api_key", "elevenlabs_api_key", "tts_stability_enabled", "tts_extra_params", "tts_extra_params_by_backend", # Captures settings "stt_language", "stt_preferred_backend", "auto_refine", "refine_model", "refine_fillers", "refine_repetitions", "refine_corrections", "refine_punctuation", "captures_default_voice", "client_voice_bindings", "llm_url", } _TTS_STABILITY_DEFAULT = {"temperature": 0.1, "top_p": 0.8, "seed": 0} _TTS_STABILITY_BY_BACKEND_DEFAULT = { "voice_clone": dict(_TTS_STABILITY_DEFAULT), "streaming": dict(_TTS_STABILITY_DEFAULT), "customvoice": dict(_TTS_STABILITY_DEFAULT), "voice_design": dict(_TTS_STABILITY_DEFAULT), "nvidia_magpie": {}, "nvidia_zeroshot": {}, "nvidia_flow": {}, "kokoro": {}, "vibevoice": {}, } _TTS_PAYLOAD_CORE_KEYS = {"model", "input", "voice", "response_format", "instruct", "language"} def _settings_bool(value, default: bool = True) -> bool: if value is None: return default if isinstance(value, bool): return value return str(value).strip().lower() not in {"0", "false", "off", "no"} def _clean_tts_param_dict(raw, fallback: dict | None = None) -> dict: if raw in (None, ""): raw = fallback or {} if isinstance(raw, str): try: raw = json.loads(raw) except Exception: raw = fallback or {} if not isinstance(raw, dict): return {} params = {} for key, value in raw.items(): if key in _TTS_PAYLOAD_CORE_KEYS or str(key).startswith("_"): continue if isinstance(value, (str, int, float, bool)) or value is None: params[str(key)] = value return params def _tts_extra_params(settings: dict, backend: str = "voice_clone") -> dict: if not _settings_bool(settings.get("tts_stability_enabled"), True): return {} backend = _clean_preview_backend(backend) if "_clean_preview_backend" in globals() else str(backend or "voice_clone") by_backend = settings.get("tts_extra_params_by_backend") if isinstance(by_backend, str): try: by_backend = json.loads(by_backend) except Exception: by_backend = None if isinstance(by_backend, dict): fallback = _TTS_STABILITY_BY_BACKEND_DEFAULT.get(backend, _TTS_STABILITY_DEFAULT) if backend in by_backend: return _clean_tts_param_dict(by_backend.get(backend), fallback) return _clean_tts_param_dict(fallback, fallback) return _clean_tts_param_dict(settings.get("tts_extra_params", _TTS_STABILITY_BY_BACKEND_DEFAULT.get(backend, _TTS_STABILITY_DEFAULT)), _TTS_STABILITY_BY_BACKEND_DEFAULT.get(backend, _TTS_STABILITY_DEFAULT)) def _apply_tts_extra_params(payload: dict, settings: dict, backend: str = "voice_clone") -> dict: params = _tts_extra_params(settings, backend) if params: payload.update(params) return payload def _strip_tts_extra_params(payload: dict) -> dict: return {k: v for k, v in payload.items() if k in _TTS_PAYLOAD_CORE_KEYS} def _post_tts_with_fallback(endpoint: str, payload: dict, headers: dict, **kwargs) -> requests.Response: resp = requests.post(endpoint, json=payload, headers=headers, **kwargs) fallback = _strip_tts_extra_params(payload) if resp.status_code in {400, 404, 415, 422} and fallback != payload: try: resp.close() except Exception: pass logger.warning("TTS backend rejected extra generation params; retrying without them") return requests.post(endpoint, json=fallback, headers=headers, **kwargs) return resp def _clean_preview_backend(value: str) -> str: key = re.sub(r"[^a-z0-9]+", "_", str(value or "voice_clone").lower()).strip("_") aliases = { "clone": "voice_clone", "base": "voice_clone", "voiceclone": "voice_clone", "voice_clone_base": "voice_clone", "stream": "streaming", "tts_streaming": "streaming", "custom": "customvoice", "custom_voice": "customvoice", "voice_design": "voice_design", "voicedesign": "voice_design", "design": "voice_design", "nvidia": "nvidia_magpie", "magpie": "nvidia_magpie", "nvidia_tts": "nvidia_magpie", "nvidia_magpie_tts": "nvidia_magpie", "nvidia_clone": "nvidia_zeroshot", "nvidia_zeroshot_tts": "nvidia_zeroshot", "magpie_zeroshot": "nvidia_zeroshot", "zeroshot": "nvidia_zeroshot", "zero_shot": "nvidia_zeroshot", "nvidia_flow_tts": "nvidia_flow", "magpie_flow": "nvidia_flow", "flow": "nvidia_flow", "kokoro_fastapi": "kokoro", "kokoro_tts": "kokoro", "kokoro_local": "kokoro", "vibevoice_service": "vibevoice", "vibe_voice": "vibevoice", "vibetts": "vibevoice", "xtts_v2": "xtts", "xtts2": "xtts", "coqui_xtts": "xtts", } key = aliases.get(key, key) return key if key in {"voice_clone", "streaming", "customvoice", "voice_design", "nvidia_magpie", "nvidia_zeroshot", "nvidia_flow", "kokoro", "vibevoice", "xtts"} else "voice_clone" def _preview_backend_base_url(settings: dict, backend: str) -> str: backend = _clean_preview_backend(backend) if backend == "streaming": return settings.get("tts_stream_url") or settings.get("tts_url") or _TTS_STREAM_DEFAULT if backend == "customvoice": return settings.get("customvoice_url") or _CUSTOMVOICE_DEFAULT if backend == "voice_design": return settings.get("voice_design_url") or _VOICE_DESIGN_DEFAULT if backend == "nvidia_magpie": return settings.get("nvidia_tts_url") or settings.get("nvidia_router_url") or _NVIDIA_TTS_DEFAULT if backend == "nvidia_zeroshot": return settings.get("nvidia_zeroshot_url") or settings.get("nvidia_clone_url") or settings.get("nvidia_router_url") or _NVIDIA_ZEROSHOT_DEFAULT if backend == "nvidia_flow": return settings.get("nvidia_flow_url") or settings.get("nvidia_clone_url") or settings.get("nvidia_router_url") or _NVIDIA_FLOW_DEFAULT if backend == "kokoro": return settings.get("kokoro_url") or _KOKORO_DEFAULT if backend == "vibevoice": return settings.get("vibevoice_url") or _VIBEVOICE_DEFAULT if backend == "xtts": return settings.get("xtts_url") or _XTTS_DEFAULT return settings.get("tts_url") or _TTS_DEFAULT def _normalize_settings(s: dict) -> dict: scan_dir = Path(s.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) output_dir = Path(s.get("output_dir", _OUTPUT_DIR_DEFAULT)) legacy_output_dirs = { Path("/voices/speakers"), scan_dir / "speakers", } if output_dir in legacy_output_dirs: s["output_dir"] = str(scan_dir / "active_voices") if s.get("tts_stream_mode") not in {"auto", "streaming", "buffered"}: s["tts_stream_mode"] = "auto" return s def _load_settings() -> dict: defaults = { "whisper_url": _WHISPER_DEFAULT, "tts_url": _TTS_DEFAULT, "tts_stream_url": _TTS_STREAM_DEFAULT, "tts_stream_mode": "auto", "tts_backend": "openai", "output_dir": _OUTPUT_DIR_DEFAULT, "voices_scan_dir": _VOICES_DIR_DEFAULT, "voice_design_url": _VOICE_DESIGN_DEFAULT, "customvoice_url": _CUSTOMVOICE_DEFAULT, "nvidia_router_url": _NVIDIA_ROUTER_DEFAULT, "nvidia_tts_url": _NVIDIA_TTS_DEFAULT, "nvidia_asr_url": _NVIDIA_ASR_DEFAULT, "nvidia_clone_url": _NVIDIA_CLONE_DEFAULT, "nvidia_zeroshot_url": _NVIDIA_ZEROSHOT_DEFAULT, "nvidia_flow_url": _NVIDIA_FLOW_DEFAULT, "faster_whisper_url": _FASTER_WHISPER_DEFAULT, "whisper_cpp_url": _WHISPER_CPP_DEFAULT, "groq_api_key": "", "kokoro_url": _KOKORO_DEFAULT, "vibevoice_url": _VIBEVOICE_DEFAULT, "xtts_url": _XTTS_DEFAULT, "whisper_api_key": "", "tts_api_key": "", "voice_design_api_key": "", "tts_stability_enabled": True, "tts_extra_params": _TTS_STABILITY_DEFAULT, "tts_extra_params_by_backend": _TTS_STABILITY_BY_BACKEND_DEFAULT, # Captures "stt_language": "", "stt_preferred_backend": "", "auto_refine": "off", "refine_model": "", "refine_fillers": True, "refine_repetitions": True, "refine_corrections": True, "refine_punctuation": True, "captures_default_voice": "", "client_voice_bindings": {}, "llm_url": "http://localhost:11434/v1", } if CONFIG_FILE.exists(): try: saved = json.loads(CONFIG_FILE.read_text()) defaults.update({k: v for k, v in saved.items() if k in _SETTINGS_KEYS}) except Exception: pass return _normalize_settings(defaults) def _save_settings(s: dict) -> None: CONFIG_DIR.mkdir(parents=True, exist_ok=True) CONFIG_FILE.write_text(json.dumps(s, indent=2)) @app.get("/api/settings") async def get_settings(): return _load_settings() @app.post("/api/settings") async def post_settings(request: Request): data = await request.json() s = _load_settings() s.update({k: v for k, v in data.items() if k in _SETTINGS_KEYS}) s = _normalize_settings(s) _save_settings(s) return {"ok": True} # ── TTS routing rules ───────────────────────────────────────────────────────── _ROUTE_LANGS = {"AUTO", "*", "EN", "DE", "FR", "ES", "IT", "PT", "NL", "PL"} _ROUTE_BACKENDS = {"voice_clone", "streaming", "voice_design", "nvidia_magpie", "nvidia_zeroshot", "nvidia_flow"} def _clean_route_token(value: str, default: str = "*") -> str: value = str(value or "").strip() if not value: return default value = re.sub(r"[^A-Za-z0-9_\-\.\* ]+", "_", value) return value[:80] or default def _clean_route_sound(value: str) -> str: value = str(value or "").strip() if not value: return "" value = re.sub(r"[^A-Za-z0-9_\-\.\* /]+", "_", value) return value[:240] def _clean_route_backend(value: str) -> str: key = re.sub(r"[^a-z0-9]+", "_", str(value or "voice_clone").lower()).strip("_") aliases = { "clone": "voice_clone", "base": "voice_clone", "tts": "voice_clone", "standard": "voice_clone", "voiceclone": "voice_clone", "voice_clone_base": "voice_clone", "stream": "streaming", "tts_streaming": "streaming", "voice_design": "voice_design", "voicedesign": "voice_design", "design": "voice_design", "nvidia": "nvidia_magpie", "magpie": "nvidia_magpie", "nvidia_tts": "nvidia_magpie", "nvidia_magpie_tts": "nvidia_magpie", "nvidia_clone": "nvidia_zeroshot", "nvidia_zeroshot_tts": "nvidia_zeroshot", "magpie_zeroshot": "nvidia_zeroshot", "zeroshot": "nvidia_zeroshot", "zero_shot": "nvidia_zeroshot", "nvidia_flow_tts": "nvidia_flow", "magpie_flow": "nvidia_flow", "flow": "nvidia_flow", } key = aliases.get(key, key) return key if key in _ROUTE_BACKENDS else "voice_clone" def _route_backend(route: dict | None, voice: str = "") -> str: if _resolve_virtual_voice(voice): return "voice_design" return _clean_route_backend((route or {}).get("backend", "voice_clone")) def _normalize_route(rule: dict, idx: int = 0) -> dict: lang = str(rule.get("language", "*") or "*").strip().upper() if lang not in _ROUTE_LANGS: lang = "*" output_voice = _clean_route_token(rule.get("output_voice", ""), "") return { "id": _clean_route_token(rule.get("id", f"route_{idx+1}"), f"route_{idx+1}"), "enabled": bool(rule.get("enabled", True)), "app": _clean_route_token(rule.get("app", "Open WebUI"), "Open WebUI"), "input_voice": _clean_route_token(rule.get("input_voice", "default"), "default"), "language": lang, "backend": _clean_route_backend(rule.get("backend", "voice_clone")), "output_voice": output_voice, "before_sound": _clean_route_sound(rule.get("before_sound", "")), "after_sound": _clean_route_sound(rule.get("after_sound", "")), } def _load_tts_routes() -> list[dict]: if not TTS_ROUTES_FILE.exists(): return [] try: raw = json.loads(TTS_ROUTES_FILE.read_text()) routes = raw.get("routes", raw) if isinstance(raw, dict) else raw if not isinstance(routes, list): return [] return [_normalize_route(r, i) for i, r in enumerate(routes) if isinstance(r, dict)] except Exception: return [] def _save_tts_routes(routes: list[dict]) -> None: TTS_ROUTES_FILE.parent.mkdir(parents=True, exist_ok=True) clean = [_normalize_route(r, i) for i, r in enumerate(routes)] TTS_ROUTES_FILE.write_text(json.dumps({"routes": clean}, indent=2)) def _app_key(value: str) -> str: return re.sub(r"[^a-z0-9]+", "", str(value or "").lower()) def _canonical_app_name(value: str) -> str: key = _app_key(value) if key in {"openwebui", "openwebuiapp"}: return "Open WebUI" if key == "sillytavern": return "SillyTavern" if key in {"homeassistant", "ha"}: return "Home Assistant" return str(value or "").strip() or "Open WebUI" def _request_app_name(request: Request) -> str: explicit = request.headers.get("x-tts-app") or request.headers.get("x-client-app") if explicit: return _canonical_app_name(explicit) header = ( request.headers.get("x-openwebui-app") or request.headers.get("x-openwebui-user-name") or request.headers.get("referer") or request.headers.get("origin") or request.headers.get("user-agent") or "" ) h = _app_key(header) if "openwebui" in h: return "Open WebUI" if "sillytavern" in h: return "SillyTavern" if "homeassistant" in h: return "Home Assistant" return "Open WebUI" def _detect_text_language(text: str) -> str: low = f" {text.lower()} " if re.search(r"[äöüß]", low) or re.search(r"\b(und|der|die|das|ich|nicht|mit|ist|ein|eine|auf|für)\b", low): return "DE" if re.search(r"[éèêàçùœ]", low) or re.search(r"\b(et|le|la|les|des|une|avec|pour|est|pas|que)\b", low): return "FR" if re.search(r"[áéíóúñ¿¡]", low) or re.search(r"\b(el|la|los|las|una|con|para|que|pero|está|hola)\b", low): return "ES" if re.search(r"\b(il|lo|gli|una|con|per|che|ciao|grazie|sono|della)\b", low): return "IT" if re.search(r"[ãõç]", low) or re.search(r"\b(com|para|uma|que|não|está|obrigado)\b", low): return "PT" if re.search(r"\b(het|een|niet|met|voor|zijn|maar|dank|goede)\b", low): return "NL" if re.search(r"[ąćęłńóśźż]", low) or re.search(r"\b(jest|nie|tak|dla|oraz|dzień|dziękuję)\b", low): return "PL" return "EN" def _route_specificity(rule: dict, app: str, voice: str, lang: str) -> tuple[int, int, int, int] | None: if not rule.get("enabled", True) or not rule.get("output_voice"): return None r_app = str(rule.get("app", "*")) r_voice = str(rule.get("input_voice", "*")) r_lang = str(rule.get("language", "*")).upper() app_ok = r_app == "*" or _app_key(r_app) == _app_key(app) voice_ok = r_voice == "*" or r_voice.lower() == voice.lower() lang_ok = r_lang in {"*", "AUTO"} or r_lang == lang if not (app_ok and voice_ok and lang_ok): return None return ( 1 if r_app != "*" else 0, 1 if r_voice != "*" else 0, 1 if r_lang not in {"*", "AUTO"} else 0, 0, ) def _resolve_tts_route(app: str, voice: str, text: str) -> tuple[str, dict | None]: lang = _detect_text_language(text) best: tuple[tuple[int, int, int, int], dict] | None = None for idx, rule in enumerate(_load_tts_routes()): spec = _route_specificity(rule, app, voice, lang) if spec is None: continue spec = (spec[0], spec[1], spec[2], -idx) if best is None or spec > best[0]: best = (spec, rule) if not best: return voice, None routed = dict(best[1]) routed["detected_language"] = lang routed["requested_voice"] = voice routed["app"] = app return str(routed["output_voice"]), routed @app.get("/api/tts-routes") async def get_tts_routes(): return {"routes": _load_tts_routes()} @app.post("/api/tts-routes") async def post_tts_routes(request: Request): data = await request.json() routes = data.get("routes", data) if isinstance(data, dict) else data if not isinstance(routes, list): raise HTTPException(400, "routes must be a list") _save_tts_routes(routes) return {"ok": True, "routes": _load_tts_routes()} @app.get("/api/tts-routing-log") async def get_tts_routing_log(limit: int = 60): limit = max(1, min(int(limit or 60), _ROUTING_LOG_MAX)) return {"items": _routing_log[:limit], "max": _ROUTING_LOG_MAX} @app.delete("/api/tts-routing-log") async def clear_tts_routing_log(): _routing_log.clear() return {"ok": True, "items": []} @app.get("/api/logs") async def get_server_logs(limit: int = 200): limit = max(1, min(int(limit or 200), _LOG_BUFFER_MAX)) return {"items": _log_buffer[:limit], "max": _LOG_BUFFER_MAX} @app.delete("/api/logs") async def clear_server_logs(): _log_buffer.clear() return {"ok": True} # ── VoiceDesign virtual voice presets ───────────────────────────────────────── _DEFAULT_DESIGN_PRESETS = { "EN_M_Young_Energetic": { "description": "Young adult male voice, clear English, bright and energetic, moderately high pitch, quick but controlled speaking rate, confident and friendly, suitable for tutorials or streaming.", "sample_text": "Hey everyone, welcome back. Today we are going to move quickly, keep it clear, and make this setup feel easy.", "language": "English", "gender": "M", }, "EN_F_Warm_Narrator": { "description": "Adult female English narrator, warm and smooth, medium pitch, calm pace, gentle emotion, clear articulation, suited for audiobooks and voice assistant responses.", "sample_text": "The room grew quiet as the morning light touched the window, and for a moment everything felt simple and kind.", "language": "English", "gender": "F", }, "DE_M_Elderly_Documentary": { "description": "Aeltere maennliche deutsche Stimme, tief und resonant, langsam und gelassen, klar artikuliert, ruhig und dokumentarisch, mit serioeser und vertrauensvoller Praesenz.", "sample_text": "Seit vielen Jahren beobachten wir diesen Ort, seine Geschichte und die Menschen, die ihn mit Leben fuellen.", "language": "German", "gender": "M", }, "DE_F_Young_Friendly": { "description": "Junge weibliche deutsche Stimme, hell und freundlich, natuerliche Sprechgeschwindigkeit, klare Aussprache, leicht optimistisch und nahbar, passend fuer Assistenten und kurze Erklaerungen.", "sample_text": "Hallo, schoen dass du da bist. Ich zeige dir kurz, wie alles funktioniert, Schritt fuer Schritt.", "language": "German", "gender": "F", }, "EN_N_Old_Wise_Assistant": { "description": "Older neutral English voice, gentle and wise, slightly low pitch, slow measured pace, soothing tone, very clear pronunciation, calm personality for guidance and reflective narration.", "sample_text": "Take a slow breath. We will look at the facts carefully, choose the next step, and keep moving.", "language": "English", "gender": "N", }, } def _slug_voice_design_name(name: str) -> str: slug = re.sub(r"[^A-Za-z0-9_.-]+", "_", name.strip()).strip("._-") return slug or "VoiceDesign" def _load_design_presets() -> dict: presets = dict(_DEFAULT_DESIGN_PRESETS) if DESIGN_PRESETS_FILE.exists(): try: saved = json.loads(DESIGN_PRESETS_FILE.read_text()) if isinstance(saved, dict): for name, preset in saved.items(): if isinstance(preset, dict): presets[_slug_voice_design_name(str(name))] = { "description": str(preset.get("description", "")), "sample_text": str(preset.get("sample_text", preset.get("text", ""))), "language": str(preset.get("language", "Auto")), "gender": str(preset.get("gender", "N")), } except Exception: pass return presets def _save_design_presets(presets: dict) -> None: cleaned = {} for name, preset in presets.items(): if not isinstance(preset, dict): continue key = _slug_voice_design_name(str(name)) cleaned[key] = { "description": str(preset.get("description", "")), "sample_text": str(preset.get("sample_text", preset.get("text", ""))), "language": str(preset.get("language", "Auto")), "gender": str(preset.get("gender", "N")), } CONFIG_DIR.mkdir(parents=True, exist_ok=True) DESIGN_PRESETS_FILE.write_text(json.dumps(cleaned, indent=2)) @app.get("/api/voice-design-presets") async def get_voice_design_presets(): return _load_design_presets() @app.post("/api/voice-design-presets") async def post_voice_design_presets(request: Request): data = await request.json() if not isinstance(data, dict): raise HTTPException(400, "Expected a preset object") _save_design_presets(data) return {"ok": True, "presets": _load_design_presets()} # ── Audio helpers ───────────────────────────────────────────────────────────── def _to_wav_24k(src: Path) -> Path: out = TEMP_DIR / f"{src.stem}_24k.wav" seg = AudioSegment.from_file(str(src)) seg = seg.set_frame_rate(24000).set_channels(1).set_sample_width(2) seg.export(str(out), format="wav") return out def _trim(src: Path, start_s: float, end_s: float) -> Path: seg = AudioSegment.from_file(str(src)) trimmed = seg[int(start_s * 1000):int(end_s * 1000)] trimmed, _ = _normalize_segment(trimmed) out = TEMP_DIR / f"{uuid.uuid4().hex}_trimmed.wav" trimmed.export(str(out), format="wav") return out def _duration(path: Path) -> float: if path.suffix.lower() == ".wav": try: with wave.open(str(path), "rb") as wf: frames = wf.getnframes() rate = wf.getframerate() if rate: return frames / float(rate) except Exception: pass try: seg = AudioSegment.from_file(str(path)) return len(seg) / 1000.0 except Exception: pass probe = subprocess.run( [ "ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "default=noprint_wrappers=1:nokey=1", str(path), ], capture_output=True, text=True, timeout=10, ) if probe.returncode != 0: raise RuntimeError(probe.stderr.strip() or "ffprobe failed") return float(probe.stdout.strip()) def _normalize_segment(seg: AudioSegment, target_dbfs: float = _VOICE_TARGET_DBFS, peak_dbfs: float = _VOICE_PEAK_DBFS) -> tuple[AudioSegment, dict]: before_dbfs = seg.dBFS if seg.dBFS != float("-inf") else None before_peak = seg.max_dBFS if seg.max_dBFS != float("-inf") else None if before_dbfs is None or before_peak is None: return seg, {"before_dbfs": before_dbfs, "after_dbfs": before_dbfs, "gain_db": 0.0, "peak_dbfs": before_peak} gain = target_dbfs - before_dbfs if before_peak + gain > peak_dbfs: gain = peak_dbfs - before_peak normalized = seg.apply_gain(gain) after_dbfs = normalized.dBFS if normalized.dBFS != float("-inf") else None after_peak = normalized.max_dBFS if normalized.max_dBFS != float("-inf") else None return normalized, { "before_dbfs": round(before_dbfs, 2), "after_dbfs": round(after_dbfs, 2) if after_dbfs is not None else None, "gain_db": round(gain, 2), "peak_dbfs": round(after_peak, 2) if after_peak is not None else None, } def _export_normalized_wav(src: Path, dest: Path, target_dbfs: float = _VOICE_TARGET_DBFS) -> dict: seg = AudioSegment.from_file(str(src)) seg = seg.set_frame_rate(24000).set_channels(1).set_sample_width(2) seg, info = _normalize_segment(seg, target_dbfs=target_dbfs) seg.export(str(dest), format="wav") return info def _loudness_info(path: Path) -> dict: seg = AudioSegment.from_file(str(path)) dbfs = seg.dBFS if seg.dBFS != float("-inf") else None peak = seg.max_dBFS if seg.max_dBFS != float("-inf") else None return { "dbfs": round(dbfs, 2) if dbfs is not None else None, "peak_dbfs": round(peak, 2) if peak is not None else None, "target_dbfs": _VOICE_TARGET_DBFS, } def _auto_trim_bounds(path: Path) -> dict: seg = AudioSegment.from_file(str(path)).set_channels(1) dur_ms = len(seg) if dur_ms <= 20_000: return { "start": 0.0, "end": dur_ms / 1000.0, "duration": dur_ms / 1000.0, "reason": "Audio is already short enough.", } chunk_ms = 250 chunks = [] overall_db = seg.dBFS if seg.dBFS != float("-inf") else -60.0 speech_floor = max(overall_db - 18.0, -45.0) for pos in range(0, dur_ms, chunk_ms): ch = seg[pos:pos + chunk_ms] db = ch.dBFS if ch.dBFS != float("-inf") else -80.0 max_db = ch.max_dBFS if ch.max_dBFS != float("-inf") else -80.0 chunks.append({"db": db, "speech": db >= speech_floor, "clipped": max_db > -1.0}) def score_window(start_ms: int, length_ms: int) -> tuple[float, dict]: first = max(0, start_ms // chunk_ms) last = min(len(chunks), (start_ms + length_ms + chunk_ms - 1) // chunk_ms) win = chunks[first:last] if not win: return -9999.0, {} speech_ratio = sum(1 for c in win if c["speech"]) / len(win) silence_ratio = 1.0 - speech_ratio clip_ratio = sum(1 for c in win if c["clipped"]) / len(win) speech_dbs = [c["db"] for c in win if c["speech"]] avg_db = sum(speech_dbs) / len(speech_dbs) if speech_dbs else -80.0 variance = sum((x - avg_db) ** 2 for x in speech_dbs) / len(speech_dbs) if speech_dbs else 100.0 loudness_penalty = abs(avg_db - (-20.0)) * 1.7 steadiness_penalty = min(18.0, variance ** 0.5 * 1.4) duration_s = length_ms / 1000.0 duration_penalty = abs(duration_s - 12.0) * 0.9 score = ( speech_ratio * 100.0 - silence_ratio * 55.0 - clip_ratio * 85.0 - loudness_penalty - steadiness_penalty - duration_penalty ) return score, { "speech_ratio": speech_ratio, "silence_ratio": silence_ratio, "clip_ratio": clip_ratio, "avg_db": avg_db, } best = None window_lengths = [8_000, 10_000, 12_000, 15_000, 18_000] for length_ms in window_lengths: if length_ms > dur_ms: continue for start_ms in range(0, dur_ms - length_ms + 1, 500): score, metrics = score_window(start_ms, length_ms) if best is None or score > best["score"]: best = {"start_ms": start_ms, "length_ms": length_ms, "score": score, "metrics": metrics} if best is None: end_ms = min(dur_ms, 12_000) return {"start": 0.0, "end": end_ms / 1000.0, "duration": end_ms / 1000.0, "reason": "Using the beginning because no stable speech window was found."} start_s = best["start_ms"] / 1000.0 end_s = (best["start_ms"] + best["length_ms"]) / 1000.0 m = best["metrics"] return { "start": round(start_s, 2), "end": round(end_s, 2), "duration": round(end_s - start_s, 2), "score": round(best["score"], 2), "reason": ( f"Selected {end_s - start_s:.1f}s with " f"{m.get('speech_ratio', 0) * 100:.0f}% speech, " f"{m.get('silence_ratio', 0) * 100:.0f}% silence, " f"avg {m.get('avg_db', -80):.1f} dBFS." ), } # ── Voice meta helpers ──────────────────────────────────────────────────────── _PICTURE_EXTS = [".jpg", ".jpeg", ".png", ".webp"] _SOUND_ASSET_DIRS = {"sound", "sounds", "sfx", "effects", "sound_effects", "route_sounds"} _SOUND_ASSET_PREFIXES = ("computerbeep", "beep", "ding", "chime", "notification") _AUDIO_EXTS = [".wav", ".mp3", ".m4a", ".flac", ".ogg", ".opus"] _UPLOAD_EXTS = _AUDIO_EXTS + [".mp4", ".mkv", ".webm", ".mov", ".avi"] _AUDIO_MIME = { ".wav": "audio/wav", ".mp3": "audio/mpeg", ".m4a": "audio/mp4", ".flac": "audio/flac", ".ogg": "audio/ogg", ".opus": "audio/ogg", } # Default flag (ISO country code) for each language code _LANG_FLAG_DEFAULT = { "EN": "GB", "DE": "DE", "ZH": "CN", "FR": "FR", "ES": "ES", "JA": "JP", "KO": "KR", "IT": "IT", "PT": "BR", "RU": "RU", "AR": "SA", "PL": "PL", "NL": "NL", "SV": "SE", "TR": "TR", "HI": "IN", } def _find_voice_audio(voice_id: str, scan_dir: Path) -> Path | None: for ext in _AUDIO_EXTS: for p in sorted(scan_dir.rglob(f"{voice_id}{ext}")): if not _is_internal_voice_file(p) and not _is_sound_asset_file(p): return p return None def _meta_path(wav: Path) -> Path: return wav.with_suffix(".meta.json") def _load_meta(wav: Path) -> dict: mp = _meta_path(wav) if mp.exists(): try: return json.loads(mp.read_text()) except Exception: pass # Auto-detect flag and gender from voice_id parts = wav.stem.split("_", 2) lang = parts[0].upper() if parts else "" gender = parts[1].upper() if len(parts) >= 2 and parts[1].upper() in ("F", "M", "N") else "" return { "note": "", "rating": 0, "flag": _LANG_FLAG_DEFAULT.get(lang, ""), "gender": gender, "enabled": True, } def _save_meta(wav: Path, meta: dict) -> None: _meta_path(wav).write_text(json.dumps(meta, indent=2)) def _picture_path(wav: Path) -> Path | None: for ext in _PICTURE_EXTS: p = wav.with_suffix(ext) if p.exists(): return p return None def _picture_mime(path: Path) -> str: return {".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".png": "image/png", ".webp": "image/webp"}.get(path.suffix.lower(), "image/jpeg") def _hidden_voices_dir(settings: dict) -> Path: return Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) / "hidden_voices" def _active_voices_dir(settings: dict) -> Path: return Path(settings.get("output_dir", _OUTPUT_DIR_DEFAULT)) def _voice_package_paths(audio: Path) -> list[Path]: parent = audio.parent paths = [] for sfx in _AUDIO_EXTS + [".reference.txt", ".meta.json"] + _PICTURE_EXTS: p = parent / f"{audio.stem}{sfx}" if p.exists(): paths.append(p) paths.extend( p for p in _backup_candidates(audio, _load_meta(audio)) if p.exists() and p.parent.resolve() == parent.resolve() ) return paths def _is_internal_voice_file(path: Path) -> bool: stem = path.stem.lower() name = path.name.lower() return ( stem.endswith(".original") or ".normalized.tmp" in stem or name.startswith(".") or name.endswith(".bak") ) def _is_sound_asset_file(path: Path) -> bool: stem = path.stem.lower() parent_names = {part.lower() for part in path.parts} return ( any(part in _SOUND_ASSET_DIRS for part in parent_names) or stem.startswith(_SOUND_ASSET_PREFIXES) ) def _voice_audio_files(root: Path): for ext in _AUDIO_EXTS: for p in root.rglob(f"*{ext}"): if not _is_internal_voice_file(p) and not _is_sound_asset_file(p): yield p def _backup_path(audio: Path) -> Path: return audio.with_name(f".{audio.stem}.original{audio.suffix}.bak") def _legacy_backup_path(audio: Path) -> Path: return audio.with_name(f"{audio.stem}.original{audio.suffix}") def _backup_candidates(audio: Path, meta: dict | None = None) -> list[Path]: candidates = [_backup_path(audio), _legacy_backup_path(audio)] if meta and meta.get("original_backup"): candidates.insert(0, Path(str(meta["original_backup"]))) seen: set[Path] = set() unique = [] for candidate in candidates: try: key = candidate.resolve() if candidate.exists() else candidate except Exception: key = candidate if key not in seen: seen.add(key) unique.append(candidate) return unique def _backup_audio_suffix(backup: Path, voice_id: str) -> str | None: name = backup.name for ext in _AUDIO_EXTS: if name == f".{voice_id}.original{ext}.bak" or name == f"{voice_id}.original{ext}": return ext return None def _remove_audio_variants(parent: Path, voice_id: str, keep: Path | None = None) -> None: keep_resolved = keep.resolve() if keep and keep.exists() else None for ext in _AUDIO_EXTS: p = parent / f"{voice_id}{ext}" if p.exists() and not _is_internal_voice_file(p) and (keep_resolved is None or p.resolve() != keep_resolved): p.unlink() def _remove_voice_package(audio: Path, keep: set[Path] | None = None) -> None: keep_resolved = {p.resolve() for p in (keep or set()) if p.exists()} for p in _voice_package_paths(audio): if p.exists() and p.resolve() not in keep_resolved: p.unlink() def _backup_original_voice(audio: Path) -> Path | None: if not audio.exists() or audio.suffix.lower() not in _AUDIO_EXTS: return None backup = _backup_path(audio) if not backup.exists(): legacy = _legacy_backup_path(audio) if legacy.exists(): shutil.move(str(legacy), str(backup)) else: shutil.copy2(str(audio), str(backup)) return backup def _voice_audio_from_request(data: dict, scan_dir: Path) -> Path | None: requested_path = data.get("path") if requested_path: p = _safe_child_path(scan_dir, Path(requested_path)) if not p.exists() or not p.is_file() or p.suffix.lower() not in _AUDIO_EXTS: raise HTTPException(404, "Voice file not found") return p voice_id = data.get("voice_id", "") if not voice_id: raise HTTPException(400, "voice_id or path is required") return _find_voice_audio(voice_id, scan_dir) def _move_voice_package(audio: Path, target_dir: Path) -> Path: if audio.parent.resolve() == target_dir.resolve(): return audio target_dir.mkdir(parents=True, exist_ok=True) paths = _voice_package_paths(audio) for src in paths: dest = target_dir / src.name if dest.exists() and dest.resolve() != src.resolve(): raise HTTPException(409, f"Target file already exists: {dest}") moved_audio = target_dir / audio.name for src in paths: dest = target_dir / src.name if dest.resolve() != src.resolve(): shutil.move(str(src), str(dest)) return moved_audio def _read_reference_text(audio: Path) -> tuple[bool, str]: ref = audio.with_suffix(".reference.txt") if not ref.exists(): return False, "" return True, ref.read_text(encoding="utf-8").strip() def _voice_entry(p: Path) -> dict: meta = _load_meta(p) parts = p.stem.split("_", 2) lang = parts[0].upper() if parts else "" try: duration = round(_duration(p), 2) except Exception: duration = None has_ref, transcript = _read_reference_text(p) loudness = meta.get("loudness", {}) benchmark = meta.get("benchmark", {}) if not meta.get("flag") and lang in _LANG_FLAG_DEFAULT: meta["flag"] = _LANG_FLAG_DEFAULT[lang] health = _voice_health(p, meta=meta, duration=duration, transcript=transcript) return { "id": p.stem, "path": str(p), "file_type": p.suffix.lower().lstrip("."), "duration": duration, "loudness": loudness, "benchmark": benchmark, "has_ref": has_ref, "transcript": transcript, "has_picture": _picture_path(p) is not None, "health": health, "lang": lang, **meta, } def _voice_health( p: Path, *, meta: dict | None = None, duration: float | None = None, transcript: str | None = None, ) -> dict: if transcript is None: _has_ref, transcript = _read_reference_text(p) if duration is None: try: duration = float(_duration(p)) except Exception: duration = 0.0 else: duration = float(duration) if meta is None: meta = _load_meta(p) words = re.findall(r"\b[\w'-]+\b", transcript, flags=re.UNICODE) word_count = len(words) words_per_sec = (word_count / duration) if duration > 0 else 0.0 warnings: list[str] = [] if not transcript: warnings.append("Missing reference transcript") if duration > 25: warnings.append("Reference audio is longer than the recommended 10-20 seconds") if duration > 0 and word_count and words_per_sec > 4.5: warnings.append("Reference transcript is too long for the audio; re-transcribe or shorten it") loudness = meta.get("loudness", {}) peak = loudness.get("peak_dbfs") if peak is not None and float(peak) > -0.1: warnings.append("Reference audio is clipping or too loud") return { "ok": not warnings, "warnings": warnings, "duration": round(duration, 2) if duration else None, "word_count": word_count, "words_per_sec": round(words_per_sec, 2) if words_per_sec else 0, "loudness": loudness, } def _benchmark_advice(audio: Path, elapsed_sec: float | None, audio_sec: float | None, clipped: bool = False, error: str = "") -> tuple[bool, list[str]]: advice: list[str] = [] try: ref_duration = float(_duration(audio)) except Exception: ref_duration = 0.0 name = audio.stem.lower() if ref_duration >= 40 or "privat_" in name or "privat-" in name: advice.append("Avoid for real-time assistants; long clone samples often benchmark slowly. Trim or remake as a 10-20 second voice.") elif ref_duration > 25: advice.append("Reference is longer than recommended. Trim to a clean 10-20 second sample.") if clipped: advice.append("Generated output hit the max-duration guard. Re-transcribe the reference exactly or remake this clone.") if elapsed_sec is not None and elapsed_sec > 12: advice.append("Slow synthesis. Prefer a shorter optimized voice for Open WebUI or Home Assistant.") if audio_sec and elapsed_sec: rtf = elapsed_sec / max(audio_sec, 0.01) if rtf > 2.0: advice.append("High real-time factor. Use a shorter reference, normalize volume, and remove silence/noise.") if error: advice.append("Benchmark failed. Check that Qwen3-TTS has rescanned this voice and that the reference files are valid.") realtime_ok = not error and not clipped and (elapsed_sec or 999) <= 8 and ref_duration <= 25 return realtime_ok, advice def _benchmark_summary(runs: list[dict]) -> dict: ok_runs = [r for r in runs if r.get("ok")] if not ok_runs: return {} def avg(key: str) -> float | None: vals = [float(r[key]) for r in ok_runs if r.get(key) is not None] return round(sum(vals) / len(vals), 3) if vals else None return { "avg_ttfa_ms": avg("ttfa_ms"), "avg_total_sec": avg("total_sec"), "avg_audio_sec": avg("audio_sec"), "avg_rtf": avg("rtf"), "avg_speed": avg("speed"), } def _benchmark_voice(audio: Path, settings: dict, sentences: list[tuple[str, str]]) -> dict: runs: list[dict] = [] for label, text in sentences: try: runs.append(_tts_benchmark_request(text, audio.stem, settings, label)) except Exception as e: runs.append({"ok": False, "label": label, "text": text, "error": str(e)}) ok_runs = [r for r in runs if r.get("ok")] summary = _benchmark_summary(runs) elapsed = summary.get("avg_total_sec") if summary else None audio_sec = summary.get("avg_audio_sec") if summary else None errors = [r.get("error", "Benchmark failed") for r in runs if not r.get("ok")] realtime_ok, advice = _benchmark_advice(audio, elapsed, audio_sec, error="; ".join(errors)) if summary.get("avg_rtf") is not None and summary["avg_rtf"] > 2: advice.append("RTF is above 2.0. This voice is likely too slow for real-time assistants.") return { "ok": bool(ok_runs) and not errors, "realtime_ok": realtime_ok and not errors, "elapsed_sec": round(elapsed, 2) if elapsed is not None else None, "audio_sec": round(audio_sec, 2) if audio_sec is not None else None, "rtf": round(summary.get("avg_rtf"), 2) if summary.get("avg_rtf") is not None else None, "speed": round(summary.get("avg_speed"), 2) if summary.get("avg_speed") is not None else None, "ttfa_ms": round(summary.get("avg_ttfa_ms"), 0) if summary.get("avg_ttfa_ms") is not None else None, "bytes": sum(int(r.get("bytes") or 0) for r in ok_runs), "clipped": False, "benchmarked_at": datetime.now(timezone.utc).isoformat(), "text": sentences[0][1] if len(sentences) == 1 else "short / medium / long", "runs": runs, "summary": summary, "advice": advice, **({"error": "; ".join(errors)} if errors else {}), } # ── Static files ────────────────────────────────────────────────────────────── STATIC_DIR = Path(__file__).parent / "static" STATIC_DIR.mkdir(exist_ok=True) @app.get("/") async def index(): return FileResponse( STATIC_DIR / "index.html", headers={"Cache-Control": "no-store, max-age=0"}, ) @app.get("/favicon.ico") async def favicon(): return Response(status_code=204) # ── Upload ──────────────────────────────────────────────────────────────────── @app.post("/api/upload") async def upload(file: UploadFile = File(...)): suffix = Path(file.filename or "audio").suffix.lower() or ".bin" if suffix not in _UPLOAD_EXTS: raise HTTPException(400, "Unsupported audio/video file type") dest = TEMP_DIR / f"{uuid.uuid4().hex}{suffix}" with dest.open("wb") as f: _copy_limited(file.file, f, _MAX_UPLOAD_BYTES) try: wav = _to_wav_24k(dest) except Exception as e: raise HTTPException(400, f"Audio conversion failed: {e}") fid = uuid.uuid4().hex _registry[fid] = wav return {"id": fid, "duration": _duration(wav), "filename": file.filename} @app.get("/api/route-sounds") async def list_route_sounds(): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) sounds = [] seen = set() for dirname in sorted(_SOUND_ASSET_DIRS): root = scan_dir / dirname if not root.exists() or not root.is_dir(): continue for path in sorted(root.rglob("*"), key=lambda p: str(p).lower()): if not path.is_file() or path.suffix.lower() not in _AUDIO_EXTS: continue try: safe = _safe_child_path(scan_dir, path) rel = str(safe.relative_to(scan_dir)) except Exception: continue if rel in seen: continue seen.add(rel) try: size = path.stat().st_size except Exception: size = None sounds.append({ "path": rel, "name": path.name, "folder": str(path.parent.relative_to(scan_dir)), "duration": None, "size": size, "type": path.suffix.lower().lstrip("."), }) return {"sounds": sounds} @app.get("/api/route-sounds/file/{sound_path:path}") async def route_sound_file(sound_path: str): try: path = _route_sound_path(_load_settings(), sound_path) except Exception: raise HTTPException(404, "Sound not found") if path is None: raise HTTPException(404, "Sound not found") mime = _AUDIO_MIME.get(path.suffix.lower(), "audio/wav") return FileResponse(str(path), media_type=mime, filename=path.name) @app.post("/api/route-sounds/upload") async def upload_route_sound(file: UploadFile = File(...), name: str = Form("")): suffix = Path(file.filename or "sound").suffix.lower() or ".bin" if suffix not in _AUDIO_EXTS: raise HTTPException(400, "Unsupported sound file type") raw_name = Path(name or file.filename or "sound").stem.strip() safe_name = re.sub(r"[^A-Za-z0-9_\-.]+", "_", raw_name).strip("._-")[:80] or f"sound_{uuid.uuid4().hex[:8]}" settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) sounds_dir = scan_dir / "sounds" sounds_dir.mkdir(parents=True, exist_ok=True) target = sounds_dir / f"{safe_name}.wav" tmp = TEMP_DIR / f"{uuid.uuid4().hex}{suffix}" with tmp.open("wb") as f: _copy_limited(file.file, f, _MAX_SOUND_BYTES) try: segment = AudioSegment.from_file(str(tmp)).set_channels(1).set_sample_width(2).set_frame_rate(24000) segment.export(str(target), format="wav") duration = len(segment) / 1000.0 except Exception as e: raise HTTPException(400, f"Sound conversion failed: {e}") finally: try: tmp.unlink(missing_ok=True) except Exception: pass return {"ok": True, "path": str(target.relative_to(scan_dir)), "filename": target.name, "duration": round(duration, 2)} def _source_import_url(value: str) -> str: raw = str(value or "").strip() validator = globals().get("_validate_download_url") if validator: return validator(raw) if not re.match(r"^https?://", raw, re.I): raise HTTPException(400, "Expected an http(s) URL") return raw def _source_download_limit(name: str, fallback_mb: int) -> int: value = globals().get(name) if isinstance(value, int): return value return int(os.environ.get(name.lstrip("_").replace("_BYTES", "_MB"), str(fallback_mb))) * 1024 * 1024 @app.post("/api/import-source-audio") async def import_source_audio(request: Request): data = await request.json() audio_url = _source_import_url(str(data.get("audio_url") or "")) filename = str(data.get("name") or Path(urlparse(audio_url).path).name or "source-audio").strip() suffix = Path(urlparse(audio_url).path).suffix.lower() if suffix not in _AUDIO_EXTS: suffix = ".bin" dest = TEMP_DIR / f"{uuid.uuid4().hex}{suffix}" try: with requests.get(audio_url, headers=_VOICE_SOURCE_HEADERS, timeout=30, stream=True) as r: r.raise_for_status() total = 0 with dest.open("wb") as f: for chunk in r.iter_content(1024 * 1024): if not chunk: continue total += len(chunk) if total > _source_download_limit("_MAX_UPLOAD_BYTES", 1024): raise HTTPException(413, "Downloaded audio is too large") f.write(chunk) wav = _to_wav_24k(dest) except HTTPException: raise except Exception as e: raise HTTPException(400, f"Source audio import failed: {e}") fid = uuid.uuid4().hex _registry[fid] = wav return {"id": fid, "duration": _duration(wav), "filename": filename or dest.name, "audio_url": audio_url} # ── Audio proxy (CORS bypass for external sources) ──────────────────────────── _PROXY_AUDIO_DOMAINS: set[str] = { "drive.usercontent.google.com", "drive.google.com", "aiartes.com", "freesound.org", "lanceblairvo.com", "raw.githubusercontent.com", "sample-files.com", } @app.get("/api/proxy-audio") async def proxy_audio(url: str): parsed = urlparse(url) if parsed.scheme not in ("http", "https"): raise HTTPException(400, "Only http/https URLs are supported") domain = parsed.netloc.lower().lstrip("www.") if not any(domain == d or domain.endswith("." + d) for d in _PROXY_AUDIO_DOMAINS): raise HTTPException(403, f"Domain not in audio proxy allowlist: {parsed.netloc}") try: resp = requests.get(url, headers=_VOICE_SOURCE_HEADERS, timeout=30, stream=True) resp.raise_for_status() except Exception as e: raise HTTPException(502, f"Proxy fetch failed: {e}") content_type = resp.headers.get("content-type", "audio/mpeg") def _stream(): for chunk in resp.iter_content(65536): if chunk: yield chunk return StreamingResponse( _stream(), media_type=content_type, headers={"Cache-Control": "public, max-age=3600", "Access-Control-Allow-Origin": "*"}, ) # ── Quick voice import (download + save directly to library) ────────────────── @app.post("/api/quick-import-voice") async def quick_import_voice(request: Request): data = await request.json() audio_url = _source_import_url(str(data.get("audio_url") or "")) voice_id = re.sub(r"[^A-Za-z0-9_\-\.]", "_", str(data.get("voice_id") or "").strip())[:80] if not voice_id: raise HTTPException(400, "voice_id is required") transcript = str(data.get("transcript") or "").strip() suffix = Path(urlparse(audio_url).path).suffix.lower() if suffix not in _AUDIO_EXTS: suffix = ".bin" dest = TEMP_DIR / f"{uuid.uuid4().hex}{suffix}" try: with requests.get(audio_url, headers=_VOICE_SOURCE_HEADERS, timeout=30, stream=True) as r: r.raise_for_status() total = 0 with dest.open("wb") as f: for chunk in r.iter_content(1024 * 1024): if not chunk: continue total += len(chunk) if total > _source_download_limit("_MAX_UPLOAD_BYTES", 1024): raise HTTPException(413, "Downloaded audio is too large") f.write(chunk) wav = _to_wav_24k(dest) except HTTPException: raise except Exception as e: raise HTTPException(400, f"Audio download failed: {e}") settings = _load_settings() out_dir = _active_voices_dir(settings) out_dir.mkdir(parents=True, exist_ok=True) final_id = voice_id if (out_dir / f"{final_id}.wav").exists(): for i in range(2, 1000): candidate = f"{voice_id}_{i}" if not (out_dir / f"{candidate}.wav").exists(): final_id = candidate break wav_dest = out_dir / f"{final_id}.wav" _remove_audio_variants(out_dir, final_id) loudness = _export_normalized_wav(wav, wav_dest) if transcript: (out_dir / f"{final_id}.reference.txt").write_text(transcript, encoding="utf-8") meta = _load_meta(wav_dest) meta["enabled"] = True meta["loudness"] = loudness _save_meta(wav_dest, meta) return {"voice_id": final_id, "loudness": loudness} # ── YouTube download (SSE) ──────────────────────────────────────────────────── @app.get("/api/download-yt") async def download_yt(url: str): url = _validate_http_url(url, allow_private=_ALLOW_PRIVATE_DOWNLOADS) out_path = TEMP_DIR / f"{uuid.uuid4().hex}.%(ext)s" async def event_stream() -> AsyncGenerator[str, None]: cmd = [ "yt-dlp", "--extract-audio", "--audio-format", "wav", "--audio-quality", "0", "--output", str(out_path), "--no-playlist", "--progress", "--newline", url, ] proc = await asyncio.create_subprocess_exec( *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, ) downloaded_path: Path | None = None assert proc.stdout is not None async for raw in proc.stdout: line = raw.decode(errors="replace").rstrip() if not line: continue if "Destination:" in line and "[ExtractAudio]" not in line: m = re.search(r"Destination:\s+(.+)$", line) if m: downloaded_path = Path(m.group(1).strip()) if line.startswith("[download]") or line.startswith("[ExtractAudio]"): pct_m = re.search(r"(\d+\.\d+)%", line) pct = pct_m.group(1) if pct_m else None yield f"data: {json.dumps({'msg': line, 'pct': pct})}\n\n" await proc.wait() if downloaded_path is None or not downloaded_path.exists(): matches = glob.glob(str(out_path).replace("%(ext)s", "*")) if matches: downloaded_path = Path(matches[0]) if downloaded_path is None or not downloaded_path.exists(): yield f"data: {json.dumps({'error': 'Download failed — no output file found'})}\n\n" return try: wav = _to_wav_24k(downloaded_path) except Exception as e: yield f"data: {json.dumps({'error': f'Conversion failed: {e}'})}\n\n" return fid = uuid.uuid4().hex _registry[fid] = wav yield f"data: {json.dumps({'done': True, 'id': fid, 'duration': _duration(wav)})}\n\n" return StreamingResponse(event_stream(), media_type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}) # ── Audio serving ───────────────────────────────────────────────────────────── @app.get("/api/audio/{fid}") async def serve_audio(fid: str): path = _registry.get(fid) if path is None or not path.exists(): raise HTTPException(404, "File not found") return FileResponse(str(path), media_type="audio/wav") # ── Process (trim + convert) ────────────────────────────────────────────────── @app.post("/api/auto-trim") async def auto_trim(request: Request): data = await request.json() fid: str = data["id"] src = _registry.get(fid) if src is None or not src.exists(): raise HTTPException(404, "Source file not found") try: return _auto_trim_bounds(src) except Exception as e: raise HTTPException(400, f"Auto trim failed: {e}") @app.post("/api/process") async def process(request: Request): data = await request.json() fid: str = data["id"] start = float(data.get("start", 0)) end: float | None = data.get("end") src = _registry.get(fid) if src is None or not src.exists(): raise HTTPException(404, "Source file not found") dur = _duration(src) if end is None or end <= start: end = dur trimmed = _trim(src, start, end) nid = uuid.uuid4().hex _registry[nid] = trimmed return {"id": nid, "duration": _duration(trimmed)} # ── Transcribe ──────────────────────────────────────────────────────────────── _STT_BACKEND_ALIASES = { "": "configured", "default": "configured", "whisper": "configured", "configured_whisper": "configured", "configured_stt": "configured", "parakeet": "nvidia_parakeet", "nvidia": "nvidia_parakeet", "nvidia_asr": "nvidia_parakeet", "nvidia_parakeet_asr": "nvidia_parakeet", "router": "nvidia_router", "speech_router": "nvidia_router", "nvidia_speech_router": "nvidia_router", "faster_whisper_server": "faster_whisper", "faster-whisper": "faster_whisper", "ctranslate2": "faster_whisper", "faster_w": "faster_whisper", "whisper-cpp": "whisper_cpp", "whisper_cpp_server": "whisper_cpp", "cpp": "whisper_cpp", "groq": "groq_whisper", "groq_stt": "groq_whisper", "groq-whisper": "groq_whisper", } _STT_BACKEND_METRICS: dict[str, dict] = { "configured": {"speed": "GPU / CPU", "latency": "1–5 s", "quality": "large-v3", "ram": "3 GB VRAM"}, "faster_whisper": {"speed": "~70× RT · GPU", "latency": "0.5–2 s", "quality": "large-v3", "ram": "1.5 GB VRAM"}, "whisper_cpp": {"speed": "~8–15× RT · CPU", "latency": "1–5 s", "quality": "large-v3 Q5", "ram": "~1 GB RAM"}, "groq_whisper": {"speed": "fastest cloud", "latency": "0.5–1 s", "quality": "Whisper Turbo", "ram": "cloud · 0"}, "nvidia_parakeet":{"speed": "~200× RT · GPU", "latency": "<0.3 s", "quality": "Parakeet-TDT", "ram": "~3 GB"}, "nvidia_router": {"speed": "GPU routed", "latency": "~0.5 s", "quality": "varies", "ram": "~11 GB"}, } _STT_VALID_BACKENDS = {"configured", "nvidia_parakeet", "nvidia_router", "faster_whisper", "whisper_cpp", "groq_whisper"} def _clean_stt_backend(value: str) -> str: key = re.sub(r"[^a-z0-9]+", "_", str(value or "configured").lower()).strip("_") key = _STT_BACKEND_ALIASES.get(key, key) return key if key in _STT_VALID_BACKENDS else "configured" def _stt_backend_url(settings: dict, backend: str) -> str: backend = _clean_stt_backend(backend) if backend == "nvidia_parakeet": return settings.get("nvidia_asr_url") or _NVIDIA_ASR_DEFAULT if backend == "nvidia_router": return settings.get("nvidia_router_url") or _NVIDIA_ROUTER_DEFAULT if backend == "faster_whisper": return settings.get("faster_whisper_url") or _FASTER_WHISPER_DEFAULT if backend == "whisper_cpp": return settings.get("whisper_cpp_url") or _WHISPER_CPP_DEFAULT if backend == "groq_whisper": return _GROQ_STT_ENDPOINT return settings.get("whisper_url") or _WHISPER_DEFAULT def _stt_backend_model(backend: str) -> str: backend = _clean_stt_backend(backend) if backend in {"nvidia_parakeet", "nvidia_router"}: return "whisper-1" if backend == "whisper_cpp": return "whisper-1" if backend == "groq_whisper": return "whisper-large-v3-turbo" return "large-v3" def _stt_backend_label(backend: str, url: str) -> str: labels = { "configured": "Configured Whisper/STT", "nvidia_parakeet": "NVIDIA Parakeet ASR", "nvidia_router": "NVIDIA Speech Router", "faster_whisper": "faster-whisper (CTranslate2 GPU)", "whisper_cpp": "whisper.cpp (CPU/CUDA)", "groq_whisper": "Groq Whisper (cloud · free)", } port = _backend_port_label(url) label = labels.get(backend, backend) return f"{port} {label}" if port else label def _stt_backend_api_key(settings: dict, backend: str) -> str: if backend == "groq_whisper": return settings.get("groq_api_key", "").strip() return settings.get("whisper_api_key", "").strip() def _stt_backend_health(url: str) -> tuple[bool, list[str]]: base = _validate_http_url(url, allow_private=True).rstrip("/") models: list[str] = [] ok = False try: r = requests.get(f"{base}/health", timeout=2) ok = r.status_code == 200 except Exception: pass try: r = requests.get(f"{base}/v1/models", timeout=3) if r.status_code == 200: ok = True payload = r.json() data = payload.get("data", []) if isinstance(payload, dict) else [] for item in data: if isinstance(item, dict) and item.get("id"): models.append(str(item["id"])) elif isinstance(item, str): models.append(item) except Exception: pass return ok, models @app.get("/api/stt-backends") async def stt_backends(): settings = _load_settings() items = [] seen_urls: set[tuple[str, str]] = set() ordered = ("configured", "faster_whisper", "whisper_cpp", "groq_whisper", "nvidia_parakeet", "nvidia_router") for backend in ordered: raw_url = _stt_backend_url(settings, backend) url = _validate_http_url(raw_url, allow_private=True).rstrip("/") key = (backend, url) if key in seen_urls: continue seen_urls.add(key) api_key = _stt_backend_api_key(settings, backend) if backend == "groq_whisper": ok = bool(api_key) models: list[str] = ["whisper-large-v3-turbo", "whisper-large-v3", "distil-whisper-large-v3-en"] else: ok, models = _stt_backend_health(url) items.append({ "id": backend, "label": _stt_backend_label(backend, url), "url": url, "port": _backend_port_label(url), "available": ok, "model": _stt_backend_model(backend), "models": models, "metrics": _STT_BACKEND_METRICS.get(backend, {}), }) return {"backends": items} def _transcription_text_from_response(resp: requests.Response) -> str: try: payload = resp.json() if isinstance(payload, str): return payload.strip() if isinstance(payload, dict): for key in ("text", "transcript", "transcription"): if payload.get(key) is not None: return str(payload[key]).strip() except Exception: pass return resp.text.strip() def _transcribe_audio(src: Path, settings: dict, backend: str = "configured") -> tuple[str, str]: backend = _clean_stt_backend(backend) stt_url = _validate_http_url(_stt_backend_url(settings, backend), allow_private=True).rstrip("/") stt_key = _stt_backend_api_key(settings, backend) hdrs = {"Authorization": f"Bearer {stt_key}"} if stt_key else {} model = _stt_backend_model(backend) with src.open("rb") as f: resp = requests.post( f"{stt_url}/v1/audio/transcriptions", files={"file": ("audio.wav", f, "audio/wav")}, data={"model": model, "response_format": "text"}, headers=hdrs, timeout=_STT_REQUEST_TIMEOUT, ) if resp.status_code in {400, 404, 422} and model != "whisper-1": with src.open("rb") as f: resp = requests.post( f"{stt_url}/v1/audio/transcriptions", files={"file": ("audio.wav", f, "audio/wav")}, data={"model": "whisper-1", "response_format": "text"}, headers=hdrs, timeout=60, ) resp.raise_for_status() return _transcription_text_from_response(resp), backend @app.post("/api/transcribe") async def transcribe(request: Request): data = await request.json() fid: str = data["id"] src = _registry.get(fid) if src is None or not src.exists(): raise HTTPException(404, "Audio not found") settings = _load_settings() backend = _clean_stt_backend(str(data.get("backend") or data.get("stt_backend") or "configured")) try: text, used_backend = await asyncio.to_thread(_transcribe_audio, src, settings, backend) return {"text": text, "backend": used_backend} except Exception as e: raise HTTPException(502, f"STT error ({backend}): {e}") @app.post("/api/transcribe-bytes") async def transcribe_bytes( file: UploadFile = File(...), backend: str = Form("configured"), ): """Accept raw audio upload and return transcription directly (used by hotkey daemon).""" suffix = Path(file.filename or "audio.wav").suffix.lower() or ".wav" if suffix not in _AUDIO_EXTS: raise HTTPException(400, "Unsupported audio type") tmp = TEMP_DIR / f"{uuid.uuid4().hex}_daemon{suffix}" wav_tmp = tmp try: with tmp.open("wb") as f: _copy_limited(file.file, f, _MAX_UPLOAD_BYTES) if suffix != ".wav": wav_tmp = _to_wav_24k(tmp) settings = _load_settings() stt_backend = _clean_stt_backend(backend) text, used_backend = await asyncio.to_thread(_transcribe_audio, wav_tmp, settings, stt_backend) return {"text": text, "backend": used_backend} except HTTPException: raise except Exception as e: raise HTTPException(502, f"STT error: {e}") finally: for p in {tmp, wav_tmp}: try: p.unlink(missing_ok=True) except Exception: pass # ── Save voice ──────────────────────────────────────────────────────────────── @app.post("/api/save") async def save_voice(request: Request): data = await request.json() fid: str = data["id"] voice_id: str = data["voice_id"].strip() transcript: str = data.get("transcript", "").strip() if not voice_id: raise HTTPException(400, "Voice ID is required") if not re.match(r"^[A-Za-z0-9_\-\.]+$", voice_id): raise HTTPException(400, "Voice ID may only contain A-Z, 0-9, _, -, .") src = _registry.get(fid) if src is None or not src.exists(): raise HTTPException(404, "Processed audio not found") settings = _load_settings() out_dir = _active_voices_dir(settings) out_dir.mkdir(parents=True, exist_ok=True) wav_dest = out_dir / f"{voice_id}.wav" txt_dest = out_dir / f"{voice_id}.reference.txt" _remove_audio_variants(out_dir, voice_id) loudness = _export_normalized_wav(src, wav_dest) txt_dest.write_text(transcript, encoding="utf-8") meta = _load_meta(wav_dest) meta["enabled"] = True meta["loudness"] = loudness _save_meta(wav_dest, meta) return {"voice_id": voice_id, "wav": str(wav_dest), "txt": str(txt_dest), "loudness": loudness} # ── Voice library ───────────────────────────────────────────────────────────── @app.post("/api/voice-load") @app.post("/api/voice/load") async def load_voice_for_edit(request: Request): data = await request.json() voice_id: str = data.get("voice_id", "") settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) audio = _voice_audio_from_request(data, scan_dir) if audio is None: raise HTTPException(404, f"Voice '{voice_id}' not found") try: wav = _to_wav_24k(audio) except Exception as e: raise HTTPException(400, f"Audio conversion failed: {e}") fid = uuid.uuid4().hex _registry[fid] = wav ref = audio.with_suffix(".reference.txt") return { "id": fid, "voice_id": voice_id or audio.stem, "path": str(audio), "duration": _duration(wav), "transcript": ref.read_text(encoding="utf-8").strip() if ref.exists() else "", "file_type": audio.suffix.lower().lstrip("."), } @app.post("/api/voice-replace") @app.post("/api/voice/replace") async def replace_voice_audio(request: Request): data = await request.json() fid: str = data["id"] voice_id: str = data["voice_id"].strip() transcript: str = data.get("transcript", "").strip() if not voice_id: raise HTTPException(400, "Voice ID is required") if not re.match(r"^[A-Za-z0-9_\-\.]+$", voice_id): raise HTTPException(400, "Voice ID may only contain A-Z, 0-9, _, -, .") src = _registry.get(fid) if src is None or not src.exists(): raise HTTPException(404, "Processed audio not found") settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) current = _voice_audio_from_request(data, scan_dir) if current is None: raise HTTPException(404, f"Voice '{voice_id}' not found") parent = current.parent old_id = current.stem meta = _load_meta(current) wav_dest = parent / f"{voice_id}.wav" txt_dest = parent / f"{voice_id}.reference.txt" if voice_id != old_id: old_files = {p.resolve() for p in _voice_package_paths(current)} suffixes = _AUDIO_EXTS + [".reference.txt", ".meta.json"] + _PICTURE_EXTS for sfx in suffixes: target = parent / f"{voice_id}{sfx}" if target.exists() and target.resolve() not in old_files: raise HTTPException(409, f"Voice '{voice_id}' already exists") backup = _backup_original_voice(current) if voice_id == old_id else None _remove_audio_variants(parent, voice_id, keep=wav_dest) loudness = _export_normalized_wav(src, wav_dest) txt_dest.write_text(transcript, encoding="utf-8") meta["loudness"] = loudness if backup: meta["original_backup"] = str(backup) meta["needs_tts_restart"] = True _save_meta(wav_dest, meta) if voice_id != old_id: keep = {wav_dest, txt_dest, _meta_path(wav_dest)} for ext in _PICTURE_EXTS: old_pic = parent / f"{old_id}{ext}" if old_pic.exists(): new_pic = parent / f"{voice_id}{ext}" shutil.copy2(str(old_pic), str(new_pic)) keep.add(new_pic) _remove_voice_package(current, keep) return {"voice_id": voice_id, "wav": str(wav_dest), "txt": str(txt_dest), "duration": _duration(wav_dest), "file_type": "wav", "loudness": loudness, "backup": str(backup) if backup else None} @app.get("/api/voices") async def list_voices(): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) active_dir = _active_voices_dir(settings) hidden_dir = _hidden_voices_dir(settings) voices = [] if scan_dir.exists(): seen: set[str] = set() all_audio = list(_voice_audio_files(scan_dir)) def voice_sort_key(path: Path): try: path.relative_to(active_dir) folder_rank = 0 except ValueError: try: path.relative_to(hidden_dir) folder_rank = 1 except ValueError: folder_rank = 2 return (path.stem.lower(), folder_rank, str(path).lower()) for p in sorted(all_audio, key=voice_sort_key): if p.stem in seen: continue # prefer first extension found (wav beats mp3 etc.) seen.add(p.stem) entry = _voice_entry(p) try: p.relative_to(hidden_dir) entry["enabled"] = False except ValueError: try: p.relative_to(active_dir) entry["enabled"] = True except ValueError: pass voices.append(entry) return voices # ── Voice meta update ───────────────────────────────────────────────────────── @app.post("/api/voice/meta") async def update_voice_meta(request: Request): data = await request.json() voice_id: str = data["voice_id"] settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice_id, scan_dir) if wav is None: raise HTTPException(404, f"Voice '{voice_id}' not found") meta = _load_meta(wav) if "enabled" in data: enabled = bool(data["enabled"]) wav = _move_voice_package(wav, _active_voices_dir(settings) if enabled else _hidden_voices_dir(settings)) meta = _load_meta(wav) meta["enabled"] = enabled for field in ("note", "rating", "flag", "gender", "loudness", "persona"): if field in data: meta[field] = data[field] if "transcript" in data: meta["needs_tts_restart"] = True _save_meta(wav, meta) if "transcript" in data: wav.with_suffix(".reference.txt").write_text(data.get("transcript", "").strip(), encoding="utf-8") return {"ok": True, "path": str(wav), "enabled": meta.get("enabled", True), "transcript": wav.with_suffix(".reference.txt").read_text(encoding="utf-8").strip() if wav.with_suffix(".reference.txt").exists() else ""} @app.post("/api/voices/sync-folders") async def sync_voice_folders(): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) active_dir = _active_voices_dir(settings) hidden_dir = _hidden_voices_dir(settings) moved = {"active": 0, "hidden": 0} conflicts = [] if not scan_dir.exists(): return {"moved": moved, "conflicts": conflicts} seen: set[str] = set() all_audio = list(_voice_audio_files(scan_dir)) for p in sorted(all_audio, key=lambda x: x.stem.lower()): if p.stem in seen or not p.exists(): continue seen.add(p.stem) meta = _load_meta(p) enabled = meta.get("enabled") is not False target_dir = active_dir if enabled else hidden_dir try: new_audio = _move_voice_package(p, target_dir) meta = _load_meta(new_audio) meta["enabled"] = enabled _save_meta(new_audio, meta) moved["active" if enabled else "hidden"] += int(new_audio != p) except HTTPException as e: conflicts.append({"voice_id": p.stem, "detail": e.detail}) return {"moved": moved, "conflicts": conflicts, "active_dir": str(active_dir), "hidden_dir": str(hidden_dir)} @app.post("/api/voices/calculate-db") async def calculate_voice_db(): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) results = [] errors = [] if not scan_dir.exists(): return {"calculated": 0, "errors": errors, "voices": results} seen: set[str] = set() all_audio = list(_voice_audio_files(scan_dir)) for audio in sorted(all_audio, key=lambda p: p.stem.lower()): if audio.stem in seen: continue seen.add(audio.stem) try: loudness = _loudness_info(audio) meta = _load_meta(audio) meta["loudness"] = loudness _save_meta(audio, meta) results.append({"voice_id": audio.stem, "path": str(audio), "loudness": loudness}) except Exception as e: errors.append({"voice_id": audio.stem, "detail": str(e)}) return {"calculated": len(results), "errors": errors, "voices": results, "target_dbfs": _VOICE_TARGET_DBFS, "peak_dbfs": _VOICE_PEAK_DBFS} @app.post("/api/voices/benchmark") async def benchmark_voices(request: Request): data = await request.json() active_only = bool(data.get("active_only", True)) limit = int(data.get("limit") or 0) sample_text = str(data.get("text") or "").strip() sentences = [("sample", sample_text)] if sample_text else list(_BENCHMARK_SENTENCES) settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) active_dir = _active_voices_dir(settings) results = [] errors = [] if not scan_dir.exists(): return { "benchmarked": 0, "errors": errors, "voices": results, "text": sample_text or "short / medium / long", "sentences": [{"label": label, "text": sent_text} for label, sent_text in sentences], "active_only": active_only, "max_tts_output_seconds": _MAX_TTS_OUTPUT_SECONDS, } target_voice = str(data.get("voice_id") or "").strip() seen: set[str] = set() if target_voice: audio = _find_voice_audio(target_voice, scan_dir) if audio is None: raise HTTPException(404, f"Voice '{target_voice}' not found") all_audio = [audio] else: all_audio = list(_voice_audio_files(active_dir if active_only and active_dir.exists() else scan_dir)) for audio in sorted(all_audio, key=lambda p: p.stem.lower()): if audio.stem in seen: continue seen.add(audio.stem) if limit and len(results) >= limit: break benchmark = await asyncio.to_thread(_benchmark_voice, audio, settings, sentences) try: meta = _load_meta(audio) meta["benchmark"] = benchmark _save_meta(audio, meta) except Exception as e: errors.append({"voice_id": audio.stem, "detail": f"Could not save benchmark: {e}"}) item = {"voice_id": audio.stem, "path": str(audio), "benchmark": benchmark} results.append(item) if not benchmark.get("ok", False): errors.append({"voice_id": audio.stem, "detail": benchmark.get("error", "Benchmark failed")}) return { "benchmarked": len(results), "errors": errors, "voices": results, "text": sample_text or "short / medium / long", "sentences": [{"label": label, "text": sent_text} for label, sent_text in sentences], "active_only": active_only, "max_tts_output_seconds": _MAX_TTS_OUTPUT_SECONDS, } @app.post("/api/voices/normalize-active") async def normalize_active_voices(): settings = _load_settings() active_dir = _active_voices_dir(settings) if not active_dir.exists(): return {"normalized": 0, "skipped": 0, "errors": [], "target_dbfs": _VOICE_TARGET_DBFS} normalized = [] skipped = [] errors = [] for wav in sorted((p for p in active_dir.rglob("*.wav") if not _is_internal_voice_file(p)), key=lambda p: p.stem.lower()): try: tmp = wav.with_suffix(".normalized.tmp.wav") loudness = _export_normalized_wav(wav, tmp) shutil.move(str(tmp), str(wav)) meta = _load_meta(wav) meta["enabled"] = True meta["loudness"] = loudness _save_meta(wav, meta) normalized.append({"voice_id": wav.stem, **loudness}) except Exception as e: errors.append({"voice_id": wav.stem, "detail": str(e)}) try: tmp = wav.with_suffix(".normalized.tmp.wav") if tmp.exists(): tmp.unlink() except Exception: pass for ext in [e for e in _AUDIO_EXTS if e != ".wav"]: skipped.extend(str(p) for p in active_dir.rglob(f"*{ext}") if not _is_internal_voice_file(p)) return { "normalized": len(normalized), "skipped": len(skipped), "errors": errors, "target_dbfs": _VOICE_TARGET_DBFS, "peak_dbfs": _VOICE_PEAK_DBFS, "voices": normalized, } @app.post("/api/voice/normalize") async def normalize_voice(request: Request): data = await request.json() target_dbfs = float(data.get("target_dbfs", _VOICE_TARGET_DBFS)) settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) audio = _voice_audio_from_request(data, scan_dir) if audio is None: raise HTTPException(404, "Voice not found") if audio.suffix.lower() != ".wav": raise HTTPException(400, "Only WAV voices can be normalized in place") tmp = audio.with_suffix(".normalized.tmp.wav") try: loudness = _export_normalized_wav(audio, tmp, target_dbfs=target_dbfs) shutil.move(str(tmp), str(audio)) meta = _load_meta(audio) meta["loudness"] = loudness meta["needs_tts_restart"] = True _save_meta(audio, meta) return {"ok": True, "voice_id": audio.stem, "path": str(audio), "duration": _duration(audio), "file_type": "wav", "loudness": loudness} except Exception as e: if tmp.exists(): tmp.unlink() raise HTTPException(400, f"Normalize failed: {e}") @app.post("/api/voice/undo") async def undo_voice_edit(request: Request): data = await request.json() settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) current = _voice_audio_from_request(data, scan_dir) if current is None: raise HTTPException(404, "Voice not found") meta = _load_meta(current) backup = next((p for p in _backup_candidates(current, meta) if p.exists()), None) if backup is None: raise HTTPException(404, "No original backup found") original_suffix = _backup_audio_suffix(backup, current.stem) or current.suffix.lower() restored = current.with_suffix(original_suffix) shutil.copy2(str(backup), str(restored)) if restored.resolve() != current.resolve(): _remove_audio_variants(current.parent, current.stem, keep=restored) try: meta["loudness"] = _loudness_info(restored) except Exception: meta.pop("loudness", None) meta["needs_tts_restart"] = True _save_meta(restored, meta) return { "ok": True, "voice_id": restored.stem, "path": str(restored), "duration": _duration(restored), "file_type": restored.suffix.lower().lstrip("."), "loudness": meta.get("loudness", {}), } # ── Voice rename ────────────────────────────────────────────────────────────── @app.post("/api/voice/rename") async def rename_voice(request: Request): data = await request.json() old_id: str = data["old_id"] new_id: str = data["new_id"].strip() if not re.match(r"^[A-Za-z0-9_\-\.]+$", new_id): raise HTTPException(400, "Invalid voice ID characters") settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(old_id, scan_dir) if wav is None: raise HTTPException(404, f"Voice '{old_id}' not found") parent = wav.parent if any((parent / f"{new_id}{ext}").exists() for ext in _AUDIO_EXTS): raise HTTPException(409, f"Voice '{new_id}' already exists") meta = _load_meta(wav) backup = next((p for p in _backup_candidates(wav, meta) if p.exists()), None) new_audio = parent / f"{new_id}{wav.suffix.lower()}" new_backup = _backup_path(new_audio) if backup and backup.exists() and backup.resolve() != new_backup.resolve() and new_backup.exists(): raise HTTPException(409, f"Backup already exists for '{new_id}'") suffixes = _AUDIO_EXTS + [".reference.txt", ".meta.json"] + _PICTURE_EXTS for sfx in suffixes: src = parent / f"{old_id}{sfx}" if src.exists(): src.rename(parent / f"{new_id}{sfx}") if backup and backup.exists(): if backup.resolve() != new_backup.resolve(): backup.rename(new_backup) new_meta = _load_meta(new_audio) new_meta["original_backup"] = str(new_backup) _save_meta(new_audio, new_meta) return {"new_id": new_id, "path": str(new_audio), "file_type": new_audio.suffix.lower().lstrip(".")} # ── Voice delete ───────────────────────────────────────────────────────────── @app.delete("/api/voice/{voice_id}") async def delete_voice(voice_id: str): if not re.match(r"^[A-Za-z0-9_\-\.]+$", voice_id): raise HTTPException(400, "Invalid voice ID") settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice_id, scan_dir) if wav is None: raise HTTPException(404, f"Voice '{voice_id}' not found") deleted = [] for f in _voice_package_paths(wav): if f.exists(): f.unlink() deleted.append(f.name) return {"deleted": deleted} # ── Voice picture upload ────────────────────────────────────────────────────── @app.post("/api/voice/picture") async def upload_picture(voice_id: str = Form(...), file: UploadFile = File(...)): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice_id, scan_dir) if wav is None: raise HTTPException(404, f"Voice '{voice_id}' not found") orig_suffix = Path(file.filename or "photo.jpg").suffix.lower() if orig_suffix not in _PICTURE_EXTS: orig_suffix = ".jpg" # Remove any existing picture first for ext in _PICTURE_EXTS: old = wav.with_suffix(ext) if old.exists(): old.unlink() dest = wav.with_suffix(orig_suffix) with dest.open("wb") as f: _copy_limited(file.file, f, _MAX_PICTURE_BYTES) return {"ok": True, "path": str(dest)} @app.post("/api/voice/picture-url") async def upload_picture_url(request: Request): data = await request.json() voice_id = str(data.get("voice_id") or "").strip() image_url = _source_import_url(str(data.get("image_url") or "")) if not voice_id or not re.match(r"^[A-Za-z0-9_\-\.]+$", voice_id): raise HTTPException(400, "Invalid voice ID") settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice_id, scan_dir) if wav is None: raise HTTPException(404, f"Voice '{voice_id}' not found") suffix = Path(urlparse(image_url).path).suffix.lower() if suffix not in _PICTURE_EXTS: suffix = ".jpg" dest = wav.with_suffix(suffix) try: with requests.get(image_url, headers=_VOICE_SOURCE_HEADERS, timeout=20, stream=True) as r: r.raise_for_status() content_type = (r.headers.get("content-type") or "").split(";", 1)[0].lower() if content_type and not content_type.startswith("image/"): raise HTTPException(400, "Image URL did not return an image") for ext in _PICTURE_EXTS: old = wav.with_suffix(ext) if old.exists(): old.unlink() total = 0 with dest.open("wb") as f: for chunk in r.iter_content(256 * 1024): if not chunk: continue total += len(chunk) if total > _source_download_limit("_MAX_PICTURE_BYTES", 10): raise HTTPException(413, "Image file is too large") f.write(chunk) except HTTPException: raise except Exception as e: raise HTTPException(400, f"Image import failed: {e}") return {"ok": True, "voice_id": voice_id, "path": str(dest)} # ── Voice picture serve ─────────────────────────────────────────────────────── @app.get("/api/voice/picture/{voice_id}") async def serve_picture(voice_id: str): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice_id, scan_dir) if wav is None: raise HTTPException(404, "Voice not found") pic = _picture_path(wav) if pic is None: raise HTTPException(404, "No picture") return FileResponse(str(pic), media_type=_picture_mime(pic)) # ── Serve voice WAV from library ────────────────────────────────────────────── @app.get("/api/voice-file") async def voice_file(path: str): settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) p = _safe_child_path(scan_dir, Path(path)) if not p.exists() or not p.is_file(): raise HTTPException(404, "File not found") if p.suffix.lower() not in _AUDIO_EXTS: raise HTTPException(404, "File not found") mime = _AUDIO_MIME.get(p.suffix.lower(), "audio/wav") return FileResponse( str(p), media_type=mime, headers={"Cache-Control": "no-store, max-age=0"}, ) # -- External voice source discovery ----------------------------------------- _VOICE_SOURCE_HEADERS = { "User-Agent": "TTS Voice Creator voice-source-scraper/1.0 (+local)", "Accept": "text/html,application/json,text/plain;q=0.9,*/*;q=0.8", } DEFAULT_VOICE_SOURCE_URLS = [ "https://aiartes.com/voiceai", "https://sample-files.com/downloads/audio/wav/voice-sample.wav", "https://freesound.org/people/Scott%20Simpson/", "https://lanceblairvo.com/raw-voiceover-samples/", "https://github.com/yaph/tts-samples/tree/main/mp3", "https://github.com/jim-schwoebel/voice_datasets", ] _DIRECT_AUDIO_RE = re.compile(r"\.(?:mp3|wav|ogg|flac|m4a|aac)(?:$|[?#])", re.I) _LANG_HINTS = { "english": "English", "german": "German", "deutsch": "German", "french": "French", "spanish": "Spanish", "italian": "Italian", "portuguese": "Portuguese", "dutch": "Dutch", "polish": "Polish", "russian": "Russian", "japanese": "Japanese", "korean": "Korean", "chinese": "Chinese", "arabic": "Arabic", "swedish": "Swedish", "turkish": "Turkish", "hindi": "Hindi", } def _source_get(url: str, timeout: int = 12) -> requests.Response: r = requests.get(url, headers=_VOICE_SOURCE_HEADERS, timeout=timeout) r.raise_for_status() return r def _plain_text(value: str) -> str: value = re.sub(r"<[^>]+>", " ", str(value or "")) value = unescape(value) return re.sub(r"\s+", " ", value).strip() def _voice_source_id(url: str) -> str: parsed = urlparse(url) base = (parsed.netloc + parsed.path).strip("/").lower() base = re.sub(r"[^a-z0-9]+", "-", base).strip("-") return base[:80] or "source" def _source_name_from_url(url: str) -> str: parsed = urlparse(url) host = parsed.netloc.replace("www.", "") tail = Path(parsed.path.rstrip("/")).stem.replace("-", " ").replace("_", " ").strip() return f"{host} / {tail}" if tail else host or url def _guess_language(*values: str) -> str: text = " ".join(str(v or "") for v in values).lower() for key, label in _LANG_HINTS.items(): if re.search(rf"\b{re.escape(key)}\b", text): return label code = re.search(r"(?:^|[^a-z])(en|de|fr|es|it|pt|nl|pl|ru|ja|ko|zh|ar|sv|tr|hi)(?:[^a-z]|$)", text) return {"en": "English", "de": "German", "fr": "French", "es": "Spanish", "it": "Italian", "pt": "Portuguese", "nl": "Dutch", "pl": "Polish", "ru": "Russian", "ja": "Japanese", "ko": "Korean", "zh": "Chinese", "ar": "Arabic", "sv": "Swedish", "tr": "Turkish", "hi": "Hindi"}.get(code.group(1), "Unknown") if code else "Unknown" def _guess_gender(*values: str) -> str: text = " ".join(str(v or "") for v in values).lower() if re.search(r"\b(female|woman|girl|fem|_f_|-f-)\b", text): return "Female" if re.search(r"\b(male|man|boy|masc|_m_|-m-)\b", text): return "Male" return "Unknown" def _source_item(source_id: str, source_name: str, name: str, kind: str, page_url: str, audio_url: str = "", image_url: str = "", category: str = "", description: str = "", file_type: str = "", language: str = "", gender: str = "") -> dict: if not file_type and audio_url: match = re.search(r"\.([A-Za-z0-9]+)(?:$|[?#])", audio_url) file_type = match.group(1).lower() if match else "audio" language = language or _guess_language(name, kind, category, description, page_url, audio_url) gender = gender or _guess_gender(name, kind, category, description, page_url, audio_url) return { "id": f"{source_id}:{uuid.uuid5(uuid.NAMESPACE_URL, page_url + audio_url + name + kind)}", "source_id": source_id, "source": source_name, "name": _plain_text(name)[:160], "kind": _plain_text(kind)[:80], "category": _plain_text(category)[:80], "description": _plain_text(description)[:420], "page_url": page_url, "audio_url": audio_url, "image_url": image_url, "file_type": file_type, "language": language, "gender": gender, "direct_audio": bool(audio_url), } def _source_result(source_id: str, name: str, homepage: str, description: str, items: list[dict]) -> dict: return { "id": source_id, "name": name, "homepage": homepage, "description": description, "items": items, "count": len(items), "direct_audio": sum(1 for item in items if item.get("direct_audio")), } def _direct_audio_source(url: str) -> dict: source_id = _voice_source_id(url) name = _source_name_from_url(url) item = _source_item( source_id, name, Path(urlparse(url).path).stem.replace("-", " ").replace("_", " ") or "Voice sample", "Direct audio file", url, audio_url=url, category="Direct audio", description="Direct audio URL from the editable source list.", ) return _source_result(source_id, name, url, "Single direct audio URL.", [item]) def _generic_audio_page_source(url: str) -> dict: source_id = _voice_source_id(url) source_name = _source_name_from_url(url) body = _source_get(url).text title = re.search(r"]*>(.*?)", body, re.I | re.S) page_title = _plain_text(title.group(1)) if title else source_name seen = set() items = [] for match in re.finditer(r"""(?:src|href)=["']([^"']+\.(?:mp3|wav|ogg|flac|m4a|aac)(?:[^"']*)?)["']""", body, re.I): audio_url = urljoin(url, unescape(match.group(1))) if audio_url in seen: continue seen.add(audio_url) stem = Path(urlparse(audio_url).path).stem.replace("-", " ").replace("_", " ").strip() or page_title items.append(_source_item( source_id, source_name, stem, "Audio sample", url, audio_url=audio_url, category="Page audio", description=f"Audio link found on {page_title}.", )) if not items: items.append(_source_item( source_id, source_name, page_title, "Source page", url, category="Source page", description="No direct audio file links were found. Open the source page for previews, licensing, and download details.", )) return _source_result(source_id, source_name, url, f"Audio links scraped from {page_title}.", items) def _scrape_aiartes_voiceai() -> dict: homepage = "https://aiartes.com/voiceai" body = _source_get(homepage).text items = [] for card in body.split('
')[1:]: title = re.search(r'
\s*([^<]+?)\s*
', card, re.S) if not title: continue image = re.search(r']+src="([^"]+)"', card, re.S) image_url = urljoin(homepage, image.group(1)) if image else "" pairs = re.findall( r'
\s*([^<]+?)\s*
\s*]*>\s* dict: return _direct_audio_source("https://sample-files.com/downloads/audio/wav/voice-sample.wav") def _scrape_freesound_scott_simpson() -> dict: return _generic_audio_page_source("https://freesound.org/people/Scott%20Simpson/") def _scrape_lanceblair_raw_samples() -> dict: return _generic_audio_page_source("https://lanceblairvo.com/raw-voiceover-samples/") def _scrape_yaph_tts_samples() -> dict: homepage = "https://github.com/yaph/tts-samples/tree/main/mp3" api = "https://api.github.com/repos/yaph/tts-samples/git/trees/main?recursive=1" data = _source_get(api).json() items = [] for entry in data.get("tree", []): path = entry.get("path", "") if entry.get("type") != "blob" or not path.lower().startswith("mp3/") or not path.lower().endswith(".mp3"): continue parts = path.split("/") language = parts[1] if len(parts) > 2 else "Unknown" stem = Path(path).stem.replace("_", " ").replace("-", " ").strip() or Path(path).stem raw_path = quote(path, safe="/") raw_url = f"https://raw.githubusercontent.com/yaph/tts-samples/main/{raw_path}" page_url = f"https://github.com/yaph/tts-samples/blob/main/{raw_path}" items.append(_source_item( "yaph-tts-samples", "yaph/tts-samples", stem, "MP3 sample", page_url, audio_url=raw_url, category=language, description=f"Language folder: {language}. Synthetic TTS sample MP3 from yaph/tts-samples.", file_type="mp3", language=language, )) items.sort(key=lambda x: (x.get("category", ""), x.get("name", ""))) return { "id": "yaph-tts-samples", "name": "yaph/tts-samples mp3", "homepage": homepage, "description": "GitHub-hosted TTS sample MP3 files grouped by language.", "items": items, "count": len(items), "direct_audio": len(items), } def _scrape_jim_voice_datasets() -> dict: homepage = "https://github.com/jim-schwoebel/voice_datasets" raw = "https://raw.githubusercontent.com/jim-schwoebel/voice_datasets/master/README.md" text = _source_get(raw).text speech = text start = text.find("### Speech datasets") end = text.find("### Audio events", start if start >= 0 else 0) if start >= 0: speech = text[start:end if end >= 0 else len(text)] items = [] for name, url, desc in re.findall(r"^\* \[([^\]]+)\]\(([^)]+)\)\s*-\s*(.+)$", speech, re.M): items.append(_source_item( "voice-datasets", "jim-schwoebel/voice_datasets", name, "Dataset link", urljoin(homepage, url.strip()), category="Speech dataset", description=desc, )) return { "id": "voice-datasets", "name": "jim-schwoebel/voice_datasets", "homepage": homepage, "description": "Curated speech and voice dataset links from the repository README.", "items": items, "count": len(items), "direct_audio": 0, } def _normalize_voice_source_urls(urls: list[str] | None = None) -> list[str]: raw_urls = urls or DEFAULT_VOICE_SOURCE_URLS normalized = [] seen = set() for value in raw_urls: url = str(value or "").strip() if not url or not re.match(r"^https?://", url, re.I): continue if url not in seen: seen.add(url) normalized.append(url) return normalized[:24] _GDRIVE_MIME_EXT = { "audio/mpeg": "mp3", "audio/mp3": "mp3", "audio/wav": "wav", "audio/x-wav": "wav", "audio/ogg": "ogg", "audio/vorbis": "ogg", "audio/flac": "flac", "audio/x-flac": "flac", "audio/aac": "aac", "audio/x-aac": "aac", "audio/x-m4a": "m4a", "audio/m4a": "m4a", "audio/mp4": "m4a", } def _scrape_google_drive_folder(url: str) -> dict: m = re.search(r"/folders/([A-Za-z0-9_-]+)", url) if not m: return _generic_audio_page_source(url) folder_id = m.group(1) source_id = _voice_source_id(url) source_name = "Google Drive" embed_url = f"https://drive.google.com/embeddedfolderview?id={folder_id}#list" try: body = _source_get(embed_url).text except Exception: return _generic_audio_page_source(url) items = [] seen: set[str] = set() for chunk in re.split(r"(?=
(.*?)
', chunk) mime_m = re.search(r"type/([a-zA-Z0-9/+\-]+)", chunk) if not eid or not title_m: continue file_id = eid.group(1) if file_id in seen: continue seen.add(file_id) filename = title_m.group(1).strip() mime = mime_m.group(1).lower() if mime_m else "" if not mime.startswith("audio"): continue ext = _GDRIVE_MIME_EXT.get(mime, Path(filename).suffix.lstrip(".").lower() or "mp3") gdrive_dl = f"https://drive.usercontent.google.com/download?id={file_id}&export=download&authuser=0" proxy_url = f"/api/proxy-audio?url={quote(gdrive_dl)}" stem = Path(filename).stem.replace("-", " ").replace("_", " ").strip() lang = _guess_language(filename) gender = _guess_gender(filename) item = _source_item( source_id, source_name, stem, "Google Drive audio", url, audio_url=proxy_url, file_type=ext, category="Google Drive", description=f"Audio file from Google Drive folder.", language=lang, gender=gender, ) item["import_url"] = gdrive_dl items.append(item) if not items: items.append(_source_item( source_id, source_name, "Google Drive folder", "Source page", url, category="Google Drive", description="No audio files found in this Google Drive folder. It may be private or empty.", )) return _source_result(source_id, source_name, url, f"{len(items)} audio files from Google Drive folder.", items) def _scrape_voice_source_url(url: str) -> dict: lower = url.lower() if "aiartes.com/voiceai" in lower: return _scrape_aiartes_voiceai() if "github.com/yaph/tts-samples" in lower: return _scrape_yaph_tts_samples() if "github.com/jim-schwoebel/voice_datasets" in lower: return _scrape_jim_voice_datasets() if "sample-files.com/downloads/audio/wav/voice-sample.wav" in lower: return _scrape_sample_files_voice_sample() if "freesound.org/people/scott%20simpson" in lower or "freesound.org/people/scott simpson" in lower: return _scrape_freesound_scott_simpson() if "lanceblairvo.com/raw-voiceover-samples" in lower: return _scrape_lanceblair_raw_samples() if "drive.google.com/drive/folders/" in lower or "drive.google.com/open?id=" in lower: return _scrape_google_drive_folder(url) if _DIRECT_AUDIO_RE.search(lower): return _direct_audio_source(url) return _generic_audio_page_source(url) def _voice_sources_payload(urls: list[str] | None = None) -> dict: sources = [] errors = [] source_urls = _normalize_voice_source_urls(urls) for url in source_urls: try: sources.append(_scrape_voice_source_url(url)) except Exception as e: errors.append({"source": url, "detail": str(e)}) return { "scraped_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), "source_urls": source_urls, "sources": sources, "total": sum(src.get("count", 0) for src in sources), "direct_audio": sum(src.get("direct_audio", 0) for src in sources), "errors": errors, } @app.get("/api/voice-sources") async def voice_sources(): return _voice_sources_payload(DEFAULT_VOICE_SOURCE_URLS) @app.post("/api/voice-sources") async def voice_sources_custom(request: Request): data = await request.json() return _voice_sources_payload(data.get("urls") if isinstance(data, dict) else None) # ── TTS voices from server ──────────────────────────────────────────────────── def _active_library_voice_options(settings: dict) -> list[dict]: active_dir = _active_voices_dir(settings) voices = [] seen = set() if not active_dir.exists(): return voices for audio in sorted(_voice_audio_files(active_dir), key=lambda p: p.stem.lower()): if audio.stem in seen: continue seen.add(audio.stem) meta = _load_meta(audio) if meta.get("enabled", True) is False: continue has_ref, transcript = _read_reference_text(audio) try: duration = round(_duration(audio), 2) except Exception: duration = None voices.append({ "id": audio.stem, "name": audio.stem, "duration": duration, "has_ref": has_ref, "has_transcript": bool(transcript), }) return voices _TTS_VOICE_ENDPOINTS = ("/v1/audio/voices", "/v1/audio/list_voices", "/v1/models", "/speakers", "/voices") def _voice_ids_from_payload(payload) -> list: if isinstance(payload, list): return payload if not isinstance(payload, dict): return [] for key in ("data", "voices", "speakers"): value = payload.get(key) if isinstance(value, list): if key == "data": return [m.get("id", m) if isinstance(m, dict) else m for m in value] return value grouped = [] for value in payload.values(): if isinstance(value, list): grouped.extend(value) elif isinstance(value, dict): nested = _voice_ids_from_payload(value) if nested: grouped.extend(nested) return grouped _KOKORO_BUILTIN_VOICES = [ "af", # Default American Female "af_bella", # Bella — American Female (warm) "af_nicole", # Nicole — American Female (clear) "af_sarah", # Sarah — American Female (expressive) "af_sky", # Sky — American Female (bright) "bf_emma", # Emma — British Female (refined) "bf_isabella",# Isabella — British Female (elegant) "am_adam", # Adam — American Male (deep) "am_michael", # Michael — American Male (smooth) "bm_george", # George — British Male (authoritative) "bm_lewis", # Lewis — British Male (natural) ] def _fetch_backend_voices(settings: dict, backend: str) -> list: backend = _clean_preview_backend(backend) if backend in {"nvidia_zeroshot", "nvidia_flow"}: return _active_library_voice_options(settings) tts_url = _validate_http_url(_preview_backend_base_url(settings, backend), allow_private=True).rstrip("/") key = (settings.get("voice_design_api_key") if backend == "voice_design" else settings.get("tts_api_key")) or "" tts_hdrs = {"Authorization": f"Bearer {key.strip()}"} if key.strip() else {} for ep in _TTS_VOICE_ENDPOINTS: try: r = requests.get(f"{tts_url}{ep}", headers=tts_hdrs, timeout=5) if r.status_code == 200: voices = _voice_ids_from_payload(r.json()) if voices: return voices except Exception: continue if backend == "kokoro": return _KOKORO_BUILTIN_VOICES return [] # ── Server-side directory browser ──────────────────────────────────────────── @app.get("/api/browse-dirs") async def browse_dirs(path: str = "/"): p = Path(path).resolve() if not p.is_dir(): raise HTTPException(404, "Not a directory") try: entries = sorted( [d.name for d in p.iterdir() if d.is_dir() and not d.name.startswith(".")], key=str.lower, ) except PermissionError: raise HTTPException(403, "Permission denied") parent = str(p.parent) if p.parent != p else None return {"path": str(p), "parent": parent, "dirs": entries} # ── ElevenLabs shared voice library proxy ──────────────────────────────────── @app.get("/api/elevenlabs/voices") async def elevenlabs_shared_voices(request: Request): settings = _load_settings() api_key = (settings.get("elevenlabs_api_key") or "").strip() _allowed = {"page_size", "page", "language", "gender", "age", "accent", "use_case", "category", "search", "featured"} params: dict = {k: v for k, v in request.query_params.items() if k in _allowed} params.setdefault("page_size", "24") headers = {"xi-api-key": api_key} if api_key else {} try: r = requests.get( "https://api.elevenlabs.io/v1/shared-voices", params=params, headers=headers, timeout=15, ) return r.json() except requests.exceptions.RequestException as exc: raise HTTPException(502, f"ElevenLabs API error: {exc}") @app.get("/api/tts-voices") async def tts_voices(backend: str = "voice_clone"): return _fetch_backend_voices(_load_settings(), backend) def _backend_port_label(url: str) -> str: try: parts = urlsplit(url) if parts.port: return str(parts.port) except Exception: pass return "" def _backend_display_name(backend: str, url: str) -> str: names = { "voice_clone": "Voice Clone/Base (WAV File)", "voice_design": "Voice Design", "customvoice": "CustomVoice", "streaming": "Streaming (WAV File)", "nvidia_magpie": "NVIDIA Magpie TTS", "nvidia_zeroshot":"NVIDIA Magpie Zeroshot Clone", "nvidia_flow": "NVIDIA Magpie Flow Clone", "kokoro": "Kokoro FastAPI (82M)", "vibevoice": "VibeVoice TTS", "xtts": "XTTS v2", } port = _backend_port_label(url) return f"{port} {names.get(backend, backend)}" if port else names.get(backend, backend) def _backend_capabilities(backend: str) -> dict: caps = { "voice_clone": { "purpose": "Clone a speaker from a short WAV/reference clip.", "identity": "Strongest match to saved WAV voices.", "style": "Weak per-request style; instruct may be ignored.", "best_for": "Known voices, multilingual cloning, benchmarks, and reliable speaker identity.", "uses_wav": True, "style_aware": False, "true_streaming": False, "speed": "~0.3× GPU", "latency": "1–3 s", "quality": "Premium clone", "ram": "~6 GB", }, "voice_design": { "purpose": "Create or reuse prompt-designed voices from natural-language descriptions.", "identity": "Prompt persona, not the selected WAV speaker unless you first export/clone it.", "style": "Strong style and emotion control through instruct text.", "best_for": "New characters, personas, dialogue, and designing reference WAVs to clone later.", "uses_wav": False, "style_aware": True, "true_streaming": False, "speed": "~0.4× GPU", "latency": "1–3 s", "quality": "Premium", "ram": "~5 GB", }, "customvoice": { "purpose": "Generate speech with the CustomVoice model voices.", "identity": "Uses CustomVoice speakers, not arbitrary active WAV voices unless trained/configured there.", "style": "Good per-request style and emotion control.", "best_for": "Controlled style with configured CustomVoice speakers.", "uses_wav": False, "style_aware": True, "true_streaming": False, "speed": "~0.3× GPU", "latency": "1–3 s", "quality": "Premium", "ram": "~5 GB", }, "streaming": { "purpose": "Low-latency playback from saved WAV/reference voices.", "identity": "Same WAV voice identity path as Base.", "style": "Weak per-request style in the current streaming server.", "best_for": "Long text, assistants, Open WebUI/SillyTavern playback that can start before completion.", "uses_wav": True, "style_aware": False, "true_streaming": True, "speed": "~0.1× GPU", "latency": "0.5–1 s", "quality": "Premium", "ram": "~5 GB", }, "nvidia_magpie": { "purpose": "Generate speech with NVIDIA Magpie fixed speaker voices.", "identity": "Uses Magpie speaker aliases such as sofia, aria, jason, leo, and john; it is not a WAV voice-cloning model.", "style": "Language and speaker are controlled by the backend voice config; per-request style text is usually ignored.", "best_for": "Fast local NVIDIA TTS voices and OpenAI-compatible assistant playback.", "uses_wav": False, "style_aware": False, "true_streaming": False, "speed": "~0.05× GPU", "latency": "0.3–0.8 s", "quality": "High", "ram": "~10 GB", }, "nvidia_zeroshot": { "purpose": "Clone a saved library voice through NVIDIA Magpie TTS Zeroshot NIM.", "identity": "Sends the selected WAV as audio_prompt; no prompt transcript is required.", "style": "Best with a clear 3-10 second prompt. Optional quality params can be configured in Settings.", "best_for": "Fast NVIDIA reference-audio cloning, streaming-class use cases, live agents, and games.", "uses_wav": True, "style_aware": False, "true_streaming": False, "speed": "~0.1× GPU", "latency": "0.5–1 s", "quality": "High clone", "ram": "~10 GB", }, "nvidia_flow": { "purpose": "Clone a saved library voice through NVIDIA Magpie TTS Flow NIM.", "identity": "Sends the selected WAV plus its exact saved reference transcript.", "style": "Offline high-fidelity clone path; prompt transcript must match the reference audio.", "best_for": "Studio-style dubbing, narration, and podcast-quality offline generation.", "uses_wav": True, "style_aware": False, "true_streaming": False, "speed": "~0.2× GPU", "latency": "1–2 s", "quality": "Studio", "ram": "~10 GB", }, "kokoro": { "purpose": "High-quality English TTS with Kokoro 82M model. OpenAI-compatible endpoint.", "identity": "Uses Kokoro built-in voices (af_bella, bf_emma, am_adam, …); no WAV cloning.", "style": "Voice selection via voice ID. Style instruction is not supported.", "best_for": "Fast, high-quality CPU TTS. Low RAM footprint. Easy local Docker setup.", "uses_wav": False, "style_aware": False, "true_streaming": False, "speed": "~0.1× CPU", "latency": "0.2–0.5 s", "quality": "High (82M)", "ram": "300 MB CPU", }, "vibevoice": { "purpose": "VibeVoice TTS service. Simple text-in, audio-out REST endpoint.", "identity": "Single built-in voice; no WAV cloning or voice selection.", "style": "Text only — no voice ID or style parameters.", "best_for": "Lightweight local TTS on port 8027. Minimal setup, fast response.", "uses_wav": False, "style_aware": False, "true_streaming": False, "speed": "fast", "latency": "0.2–1 s", "quality": "High", "ram": "~4 GB", }, "xtts": { "purpose": "XTTS v2 via xtts-api-server. OpenAI-compatible endpoint with speaker selection.", "identity": "Uses speakers registered in the XTTS server; not WAV voice cloning.", "style": "Speaker selected by voice ID. Style instruction not supported.", "best_for": "Local multi-speaker TTS with XTTS v2 model. Coqui/daswer123 docker setup.", "uses_wav": False, "style_aware": False, "true_streaming": False, "speed": "~0.3× GPU", "latency": "1–3 s", "quality": "High", "ram": "~5 GB", }, } return caps.get(_clean_preview_backend(backend), {}) def _backend_health(url: str) -> bool: base = url.rstrip("/") for ep in ("/health", "/v1/health", "/v1/audio/list_voices"): try: r = requests.get(f"{base}{ep}", timeout=2) if r.status_code == 200: return True except Exception: continue return False def _backend_available(backend: str, voices: list, health: bool) -> bool: if _clean_preview_backend(backend) in {"nvidia_zeroshot", "nvidia_flow", "kokoro", "vibevoice"}: return health return bool(voices) or health @app.get("/api/tts-backends") async def tts_backends(): settings = _load_settings() items = [] for backend in ("voice_clone", "voice_design", "customvoice", "streaming", "kokoro", "vibevoice", "xtts", "nvidia_magpie", "nvidia_zeroshot", "nvidia_flow"): url = _validate_http_url(_preview_backend_base_url(settings, backend), allow_private=True).rstrip("/") voices = _fetch_backend_voices(settings, backend) health = _backend_health(url) available = _backend_available(backend, voices, health) items.append({ "id": backend, "label": _backend_display_name(backend, url), "url": url, "port": _backend_port_label(url), "available": available, "voice_count": len(voices) if isinstance(voices, list) else 0, **_backend_capabilities(backend), }) return {"backends": items} def _clear_tts_restart_flags(settings: dict | None = None) -> int: cleared = 0 settings = settings or _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) if scan_dir.exists(): seen: set[str] = set() for audio in _voice_audio_files(scan_dir): if audio.stem in seen: continue seen.add(audio.stem) meta = _load_meta(audio) if meta.pop("needs_tts_restart", None) is not None: _save_meta(audio, meta) cleared += 1 return cleared def _tts_container_names() -> list[str]: """Return the list of TTS container names to restart, from env vars.""" multi = os.environ.get("TTS_CONTAINER_NAMES", _TTS_CONTAINERS_RAW).strip() if multi: return [c.strip() for c in multi.split(",") if c.strip()] single = os.environ.get("TTS_CONTAINER_NAME", _TTS_CONTAINER).strip() return [single] if single else [] @app.post("/api/tts/restart") async def restart_tts_container(): containers = _tts_container_names() if not containers: raise HTTPException(400, "No TTS container names configured (set TTS_CONTAINER_NAMES in docker-compose.yml)") results = [] errors = [] for container in containers: path = f"/containers/{quote(container, safe='')}/restart?t=10" try: code, raw = _docker_post(path) if code not in (204, 304): detail = raw.split("\r\n\r\n", 1)[-1].strip() or f"HTTP {code}" errors.append(f"{container}: {detail}") else: results.append(container) except PermissionError: raise HTTPException(502, "No permission to access /var/run/docker.sock — is the socket mounted in docker-compose.yml?") except Exception as e: errors.append(f"{container}: {e}") cleared = 0 try: cleared = _clear_tts_restart_flags() except Exception as e: logger.warning("Could not clear TTS restart flags: %s", e) if errors and not results: raise HTTPException(502, "; ".join(errors)) return { "ok": True, "restarted": results, "errors": errors, "cleared_restart_flags": cleared, } @app.get("/api/tts/restart-info") async def tts_restart_info(): containers = _tts_container_names() sock_ok = Path(os.environ.get("DOCKER_SOCKET", "/var/run/docker.sock")).exists() return {"containers": containers, "socket_available": sock_ok} # ── Local Docker container management ──────────────────────────────────────── _LOCAL_CONTAINER_DEFS: list[dict] = [ {"name": "faster-qwen3-tts-voiceclone", "label": "Qwen3 TTS · Voice Clone", "role": "tts", "port": 8020, "stack": "faster-qwen3-tts-dgx-spark-voice-design", "image": "martinb78/faster-qwen3-tts-dgx-spark:v4", "repo": "https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts", "description": "WAV voice cloning. Scans active_voices at startup — restart after adding or editing voices."}, {"name": "faster-qwen3-tts-voicedesign", "label": "Qwen3 TTS · Voice Design", "role": "tts", "port": 8021, "stack": "faster-qwen3-tts-dgx-spark-voice-design", "image": "martinb78/faster-qwen3-tts-dgx-spark:v4", "repo": "https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts", "description": "Instruction-based voice design. Describe a voice in words — no WAV needed."}, {"name": "faster-qwen3-tts-customvoice", "label": "Qwen3 TTS · Custom Voice", "role": "tts", "port": 8022, "stack": "faster-qwen3-tts-dgx-spark-voice-design", "image": "martinb78/faster-qwen3-tts-dgx-spark:v4", "repo": "https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts", "description": "Style control over configured premium speakers such as Ryan, Vivian, and Serena."}, {"name": "faster-qwen3-tts-streaming", "label": "Qwen3 TTS · Streaming", "role": "tts", "port": 8023, "stack": "faster-qwen3-tts-dgx-spark-voice-design", "image": "martinb78/qwen3-tts-streaming-dgx-spark:latest", "repo": "https://github.com/mARTin-B78/dgx-spark-faster-qwen3-tts", "description": "Low-latency progressive WAV streaming for voice clone voices."}, {"name": "parakeet-asr", "label": "NVIDIA Parakeet ASR", "role": "stt", "port": 8090, "stack": "nvidia-speech-gateway", "image": "parakeet-tdt-v3-spark:latest", "repo": "https://github.com/mARTin-B78/dgx-spark-parakeet-asr", "description": "NVIDIA Parakeet GPU-accelerated speech recognition on port 8090."}, {"name": "magpie-tts", "label": "NVIDIA Magpie TTS", "role": "tts", "port": 8091, "stack": "nvidia-speech-gateway", "image": "nvcr.io/nim/nvidia/magpie-tts-multilingual:latest","repo": "https://github.com/mARTin-B78/dgx-spark-parakeet-asr", "description": "NVIDIA neural TTS. GPU-accelerated, high-quality multilingual synthesis."}, {"name": "parakeet-rnnt-nim", "label": "NVIDIA Parakeet RNNT NIM", "role": "stt", "port": 8092, "stack": "nvidia-speech-gateway", "image": "nvcr.io/nim/nvidia/parakeet-1b-rnnt-multilingual:latest","repo": "https://github.com/mARTin-B78/dgx-spark-parakeet-asr", "description": "NVIDIA Parakeet RNNT NIM multilingual ASR on port 8092."}, ] def _container_status(name: str) -> dict: try: code, data = _docker_get_json(f"/containers/{quote(name, safe='')}/json") if code == 404: return {"status": "not_found"} if code == 200 and isinstance(data, dict): state = data.get("State", {}) return { "status": state.get("Status", "unknown"), "running": state.get("Running", False), "paused": state.get("Paused", False), "started_at": state.get("StartedAt", ""), "image": data.get("Config", {}).get("Image", ""), } return {"status": "error", "detail": f"HTTP {code}"} except Exception as e: return {"status": "error", "detail": str(e)} @app.get("/api/local-containers") async def get_local_containers(): sock_ok = Path(os.environ.get("DOCKER_SOCKET", "/var/run/docker.sock")).exists() results = [] for defn in _LOCAL_CONTAINER_DEFS: entry = {k: v for k, v in defn.items()} if sock_ok: entry.update(_container_status(defn["name"])) else: entry["status"] = "no_socket" results.append(entry) return {"containers": results, "socket_available": sock_ok} @app.post("/api/local-containers/{name}/start") async def start_local_container(name: str): if not any(c["name"] == name for c in _LOCAL_CONTAINER_DEFS): raise HTTPException(404, f"Unknown container: {name}") try: code, _ = _docker_post(f"/containers/{quote(name, safe='')}/start") except Exception as e: raise HTTPException(502, f"Docker start failed: {e}") if code not in (204, 304): raise HTTPException(502, f"Docker API returned HTTP {code}") return {"ok": True, "name": name, **_container_status(name)} @app.post("/api/local-containers/{name}/stop") async def stop_local_container(name: str): if not any(c["name"] == name for c in _LOCAL_CONTAINER_DEFS): raise HTTPException(404, f"Unknown container: {name}") try: code, _ = _docker_post(f"/containers/{quote(name, safe='')}/stop?t=10") except Exception as e: raise HTTPException(502, f"Docker stop failed: {e}") if code not in (204, 304): raise HTTPException(502, f"Docker API returned HTTP {code}") return {"ok": True, "name": name, **_container_status(name)} @app.post("/api/local-containers/{name}/restart") async def restart_local_container(name: str): if not any(c["name"] == name for c in _LOCAL_CONTAINER_DEFS): raise HTTPException(404, f"Unknown container: {name}") try: code, _ = _docker_post(f"/containers/{quote(name, safe='')}/restart?t=10") except Exception as e: raise HTTPException(502, f"Docker restart failed: {e}") if code not in (204, 304): raise HTTPException(502, f"Docker API returned HTTP {code}") return {"ok": True, "name": name, **_container_status(name)} @app.get("/api/probe-url") async def probe_url(url: str): """Server-side reachability check — avoids browser CORS restrictions.""" try: r = requests.get(url, timeout=5, allow_redirects=True, headers={"User-Agent": "TTS-Voice-Creator/probe"}) return {"ok": True, "status": r.status_code} except requests.exceptions.ConnectionError: return {"ok": False, "error": "Connection refused"} except requests.exceptions.Timeout: return {"ok": False, "error": "Timeout"} except Exception as e: return {"ok": False, "error": str(e)} @app.post("/api/tts/restart-flags/clear") async def clear_tts_restart_flags(): try: cleared = _clear_tts_restart_flags() except Exception as e: raise HTTPException(500, f"Could not clear TTS restart flags: {e}") return {"ok": True, "cleared_restart_flags": cleared} # ── TTS preview ─────────────────────────────────────────────────────────────── def _tts_request_config(text: str, voice: str, settings: dict, response_format: str = "wav", instruct: str = "", url_override: str = "", api_key_override: str | None = None, backend_override: str = "", extra_backend: str = "") -> tuple[str, dict, dict]: tts_url = _validate_http_url(url_override or settings.get("tts_url", _TTS_DEFAULT), allow_private=True).rstrip("/") backend = backend_override or settings.get("tts_backend", "openai") tts_key = (api_key_override if api_key_override is not None else settings.get("tts_api_key", "")).strip() tts_hdrs = {"Authorization": f"Bearer {tts_key}"} if tts_key else {} if backend == "localai": endpoint, payload = f"{tts_url}/tts", {"input": text, "model": voice, "response_format": response_format} elif backend == "pocket": endpoint, payload = f"{tts_url}/v1/audio/speech", {"input": text, "voice": voice, "response_format": response_format} else: endpoint, payload = f"{tts_url}/v1/audio/speech", {"model": "tts-1", "input": text, "voice": voice, "response_format": response_format} if instruct.strip(): payload["instruct"] = instruct.strip() _apply_tts_extra_params(payload, settings, extra_backend or backend_override or "voice_clone") return endpoint, payload, tts_hdrs def _tts_request_audio(text: str, voice: str, settings: dict, instruct: str = "", url_override: str = "", api_key_override: str | None = None, backend_override: str = "", extra_backend: str = "") -> tuple[bytes, str]: endpoint, payload, tts_hdrs = _tts_request_config(text, voice, settings, "wav", instruct, url_override, api_key_override, backend_override, extra_backend) resp = _post_tts_with_fallback(endpoint, payload, tts_hdrs, timeout=120) resp.raise_for_status() audio = resp.content if not audio: raise RuntimeError("backend returned empty audio") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return audio, media_type def _wav_data_offset(data: bytes) -> int | None: if len(data) < 12: return None if data[:4] != b"RIFF" or data[8:12] != b"WAVE": return 0 pos = 12 while pos + 8 <= len(data): chunk_sz = struct.unpack_from(" float | None: try: source_format = "wav" if audio[:4] == b"RIFF" or "wav" in media_type.lower() else None return len(AudioSegment.from_file(io.BytesIO(audio), format=source_format)) / 1000.0 except Exception: offset = _wav_data_offset(audio) if offset is not None and offset > 0 and len(audio) > offset: return (len(audio) - offset) / (24000 * 2) return None def _tts_benchmark_request(text: str, voice: str, settings: dict, label: str) -> dict: endpoint, payload, tts_hdrs = _tts_request_config(text, voice, settings, "wav") start = time.perf_counter() first_audio_at = None raw = bytearray() media_type = "audio/wav" with _post_tts_with_fallback(endpoint, payload, tts_hdrs, stream=True, timeout=180) as resp: resp.raise_for_status() media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" for chunk in resp.iter_content(chunk_size=512): if not chunk: continue raw.extend(chunk) if first_audio_at is None: if payload.get("response_format") == "wav" or "wav" in media_type.lower(): offset = _wav_data_offset(bytes(raw)) if offset is not None and len(raw) > offset: first_audio_at = time.perf_counter() else: first_audio_at = time.perf_counter() total = time.perf_counter() - start if not raw: raise RuntimeError("backend returned empty audio") audio_sec = _audio_duration_from_bytes(bytes(raw), media_type) rtf = total / audio_sec if audio_sec and audio_sec > 0 else None speed = audio_sec / total if audio_sec and total > 0 else None return { "ok": True, "label": label, "text": text, "ttfa_ms": round(((first_audio_at or time.perf_counter()) - start) * 1000, 1), "total_sec": round(total, 3), "audio_sec": round(audio_sec, 3) if audio_sec is not None else None, "rtf": round(rtf, 3) if rtf is not None else None, "speed": round(speed, 3) if speed is not None else None, "bytes": len(raw), } def _apply_voice_design_gender(instruct: str, gender: str) -> str: gender = (gender or "").strip().upper() if gender == "F": prefix = ( "MANDATORY SPEAKER IDENTITY: Female speaker / woman.\n" "gender: Female.\n" "Use a clearly feminine vocal timbre, light-to-medium resonance, and soprano or mezzo-soprano pitch range.\n" "Avoid male baritone, bass, chest-heavy, or masculine vocal qualities." ) elif gender == "M": prefix = ( "MANDATORY SPEAKER IDENTITY: Male speaker / man.\n" "gender: Male.\n" "Use a clearly masculine vocal timbre, medium-to-deep resonance, and tenor, baritone, or bass pitch range.\n" "Avoid feminine soprano or mezzo-soprano vocal qualities." ) else: return instruct return f"{prefix}\n\n{instruct.strip()}" def _voice_design_request_audio(instruct: str, text: str, language: str, settings: dict, gender: str = "") -> tuple[bytes, str]: vd_url = _validate_http_url(settings.get("voice_design_url") or _VOICE_DESIGN_DEFAULT, allow_private=True).rstrip("/") vd_key = (settings.get("voice_design_api_key") or settings.get("tts_api_key", "")).strip() vd_hdrs = {"Authorization": f"Bearer {vd_key}"} if vd_key else {} payload = { "model": _VOICE_DESIGN_MODEL, "input": text, "instruct": _apply_voice_design_gender(instruct, gender), "language": language, "response_format": "wav", } _apply_tts_extra_params(payload, settings, "voice_design") resp = _post_tts_with_fallback(f"{vd_url}/v1/audio/speech", payload, vd_hdrs, timeout=180) resp.raise_for_status() audio = resp.content if not audio: raise RuntimeError("backend returned empty audio") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return audio, media_type def _voice_design_voice_request_audio(voice: str, text: str, settings: dict, instruct: str = "", language: str = "Auto") -> tuple[bytes, str]: vd_url = _validate_http_url(settings.get("voice_design_url") or _VOICE_DESIGN_DEFAULT, allow_private=True).rstrip("/") vd_key = (settings.get("voice_design_api_key") or settings.get("tts_api_key", "")).strip() vd_hdrs = {"Authorization": f"Bearer {vd_key}"} if vd_key else {} payload = { "model": _VOICE_DESIGN_MODEL, "input": text, "voice": voice, "response_format": "wav", } if instruct.strip(): payload["instruct"] = instruct.strip() if language and language != "Auto": payload["language"] = language _apply_tts_extra_params(payload, settings, "voice_design") resp = _post_tts_with_fallback(f"{vd_url}/v1/audio/speech", payload, vd_hdrs, timeout=180) resp.raise_for_status() audio = resp.content if not audio: raise RuntimeError("backend returned empty audio") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return audio, media_type _NVIDIA_LANGUAGE_CODES = { "EN": "en-US", "DE": "de-DE", "ES": "es-ES", "FR": "fr-FR", "IT": "it-IT", "PT": "pt-PT", "NL": "nl-NL", "PL": "pl-PL", } def _nvidia_clone_language_code(voice: str, language: str = "") -> str: raw = str(language or "").strip() if raw and raw.lower() not in {"auto", "*"}: return raw prefix = str(voice or "").split("_", 1)[0].upper() return _NVIDIA_LANGUAGE_CODES.get(prefix, "en-US") def _form_value(value) -> str: if isinstance(value, bool): return "true" if value else "false" return str(value) def _nvidia_clone_request_audio(text: str, voice: str, settings: dict, mode: str = "zeroshot", reference_transcript: str = "", language: str = "") -> tuple[bytes, str]: mode = "flow" if str(mode).lower().endswith("flow") else "zeroshot" backend = "nvidia_flow" if mode == "flow" else "nvidia_zeroshot" base_url = _validate_http_url(_preview_backend_base_url(settings, backend), allow_private=True).rstrip("/") key = (settings.get("voice_design_api_key") or settings.get("tts_api_key", "")).strip() headers = {"Authorization": f"Bearer {key}"} if key else {} scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) audio_path = _find_voice_audio(voice, scan_dir) if audio_path is None: raise RuntimeError(f"reference voice not found in library: {voice}") prompt_wav = _to_wav_24k(audio_path) has_ref, saved_transcript = _read_reference_text(audio_path) prompt_transcript = str(reference_transcript or saved_transcript or "").strip() if mode == "flow" and not prompt_transcript: raise RuntimeError("NVIDIA Magpie Flow requires the selected voice to have an exact saved reference transcript") data = { "language": _nvidia_clone_language_code(voice, language), "text": text, } if mode == "flow": data["audio_prompt_transcript"] = prompt_transcript params = _tts_extra_params(settings, backend) for k, v in params.items(): if k in {"audio_prompt", "audio_prompt_transcript", "text", "language"}: continue data[k] = _form_value(v) endpoint = f"{base_url}/v1/audio/synthesize" with prompt_wav.open("rb") as f: files = {"audio_prompt": ("prompt.wav", f, "audio/wav")} resp = requests.post(endpoint, data=data, files=files, headers=headers, timeout=180) if resp.status_code in {400, 404, 415, 422} and params: try: resp.close() except Exception: pass fallback = {k: v for k, v in data.items() if k in {"language", "text", "audio_prompt_transcript"}} with prompt_wav.open("rb") as f: files = {"audio_prompt": ("prompt.wav", f, "audio/wav")} resp = requests.post(endpoint, data=fallback, files=files, headers=headers, timeout=180) resp.raise_for_status() audio = resp.content if not audio: raise RuntimeError("NVIDIA clone backend returned empty audio") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return audio, media_type def _vibevoice_request_audio(text: str, settings: dict) -> tuple[bytes, str]: url = _preview_backend_base_url(settings, "vibevoice").rstrip("/") resp = requests.post(f"{url}/tts", json={"text": text}, timeout=60) resp.raise_for_status() audio = resp.content if not audio: raise RuntimeError("VibeVoice backend returned empty audio") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return audio, media_type def _preview_request_audio(text: str, voice: str, settings: dict, instruct: str = "", backend: str = "voice_clone") -> tuple[bytes, str]: backend = _clean_preview_backend(backend) if backend == "streaming": resp = _open_tts_stream_response(text, voice, settings, instruct) audio, media_type = _read_tts_stream_response(resp) if not audio: raise RuntimeError("backend returned empty audio") return audio, media_type if backend == "customvoice": return _tts_request_audio( text, voice, settings, instruct, _preview_backend_base_url(settings, "customvoice"), settings.get("tts_api_key", ""), "openai", "customvoice", ) if backend == "voice_design": return _voice_design_voice_request_audio(voice, text, settings, instruct) if backend == "nvidia_magpie": return _tts_request_audio( text, voice, settings, instruct, _preview_backend_base_url(settings, "nvidia_magpie"), settings.get("tts_api_key", ""), "nvidia_magpie", "nvidia_magpie", ) if backend == "nvidia_zeroshot": return _nvidia_clone_request_audio(text, voice, settings, "zeroshot") if backend == "nvidia_flow": return _nvidia_clone_request_audio(text, voice, settings, "flow") if backend == "kokoro": return _tts_request_audio( text, voice, settings, instruct, url_override=_preview_backend_base_url(settings, "kokoro"), api_key_override=settings.get("tts_api_key", ""), backend_override="openai", extra_backend="kokoro", ) if backend == "vibevoice": return _vibevoice_request_audio(text, settings) if backend == "xtts": return _tts_request_audio( text, voice, settings, instruct, url_override=_preview_backend_base_url(settings, "xtts"), api_key_override=settings.get("tts_api_key", ""), backend_override="openai", extra_backend="xtts", ) return _tts_request_audio(text, voice, settings, instruct) def _requested_response_format(data: dict) -> str: requested = str(data.get("response_format") or data.get("format") or "wav").strip().lower() if requested in {"mp3", "mpeg"}: return "mp3" if requested in {"wav", "pcm"}: return "wav" return "wav" def _audio_ext_media(response_format: str) -> tuple[str, str]: if response_format == "mp3": return "mp3", "audio/mpeg" return "wav", "audio/wav" def _route_sound_path(settings: dict, value: str) -> Path | None: value = str(value or "").strip() if not value: return None scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) candidate = Path(value) if not candidate.is_absolute(): candidate = scan_dir / candidate path = _safe_child_path(scan_dir, candidate) if not path.exists() or not path.is_file() or path.suffix.lower() not in _AUDIO_EXTS: raise RuntimeError(f"Route sound not found or unsupported: {value}") return path def _sound_segment(path: Path) -> AudioSegment: return AudioSegment.from_file(str(path)).set_channels(1).set_sample_width(2).set_frame_rate(24000) def _apply_route_sounds(audio: bytes, media_type: str, route: dict | None, settings: dict) -> tuple[bytes, str, list[str]]: if not route: return audio, media_type, [] before = _route_sound_path(settings, str(route.get("before_sound", ""))) after = _route_sound_path(settings, str(route.get("after_sound", ""))) if not before and not after: return audio, media_type, [] source_format = "wav" if audio[:4] == b"RIFF" or "wav" in media_type.lower() else None speech = AudioSegment.from_file(io.BytesIO(audio), format=source_format) speech = speech.set_channels(1).set_sample_width(2).set_frame_rate(24000) combined = AudioSegment.empty() applied = [] if before: combined += _sound_segment(before) applied.append(f"before:{before.name}") combined += speech if after: combined += _sound_segment(after) applied.append(f"after:{after.name}") out = io.BytesIO() combined.export(out, format="wav") return out.getvalue(), "audio/wav", applied def _prepare_proxy_audio(audio: bytes, media_type: str, response_format: str) -> tuple[bytes, str, str, float | None, bool]: """Rewrite backend audio so clients receive playable headers and requested format.""" try: source_format = "wav" if audio[:4] == b"RIFF" or "wav" in media_type.lower() else None segment = AudioSegment.from_file(io.BytesIO(audio), format=source_format) except Exception: ext, wanted_media = _audio_ext_media(response_format) return audio, media_type or wanted_media, ext, None, False duration = len(segment) / 1000.0 clipped = False if _MAX_TTS_OUTPUT_SECONDS > 0 and duration > _MAX_TTS_OUTPUT_SECONDS: segment = segment[:int(_MAX_TTS_OUTPUT_SECONDS * 1000)] duration = len(segment) / 1000.0 clipped = True ext, wanted_media = _audio_ext_media(response_format) out = io.BytesIO() export_format = "mp3" if response_format == "mp3" else "wav" segment.export(out, format=export_format) return out.getvalue(), wanted_media, ext, duration, clipped def _parse_voice_design_dialogue(instruct: str, script: str) -> tuple[dict[str, str], list[tuple[str, str]]] | None: speakers: dict[str, str] = {} for raw in instruct.splitlines(): line = raw.strip() if not line: continue match = re.match(r'^"?([^":]+)"?\s*:\s*"?(.+?)"?$', line) if match: speakers[match.group(1).strip()] = match.group(2).strip() turns: list[tuple[str, str]] = [] for raw in script.splitlines(): line = raw.strip() if not line: continue match = re.match(r"^([^:]{1,40}):\s*(.+)$", line) if match: speaker = match.group(1).strip() text = match.group(2).strip() if speaker in speakers and text: turns.append((speaker, text)) if len(speakers) < 2 or len(turns) < 2: return None if len({speaker for speaker, _text in turns}) < 2: return None return speakers, turns def _infer_voice_design_gender(description: str) -> str: text = f" {description.lower()} " if re.search(r"\b(female|woman|girl|feminine|soprano|mezzo-soprano|mezzo)\b", text): return "F" if re.search(r"\b(male|man|boy|masculine|tenor|baritone|bass)\b", text): return "M" return "" def _audio_segment_from_wav(audio: bytes) -> AudioSegment: segment = AudioSegment.from_file(io.BytesIO(audio), format="wav") return segment.set_channels(1).set_sample_width(2).set_frame_rate(24000) def _wav_bytes_from_segment(segment: AudioSegment) -> bytes: out = io.BytesIO() segment.export(out, format="wav") return out.getvalue() def _voice_design_dialogue_request_audio( speakers: dict[str, str], turns: list[tuple[str, str]], language: str, settings: dict, ) -> tuple[bytes, str]: combined = AudioSegment.silent(duration=120, frame_rate=24000).set_channels(1).set_sample_width(2) pause = AudioSegment.silent(duration=180, frame_rate=24000).set_channels(1).set_sample_width(2) for speaker, text in turns: description = speakers[speaker] turn_instruct = f'Speaker "{speaker}".\n{description}' audio, _media_type = _voice_design_request_audio( turn_instruct, text, language, settings, gender=_infer_voice_design_gender(description), ) combined += _audio_segment_from_wav(audio) + pause return _wav_bytes_from_segment(combined), "audio/wav" @app.post("/api/tts-preview") async def tts_preview(request: Request): data = await request.json() text: str = data["text"] voice: str = data["voice"] response_format = _requested_response_format(data) instruct = str(data.get("instruct") or data.get("style_instruction") or "") backend = _clean_preview_backend(data.get("backend", "voice_clone")) settings = _load_settings() if data.get("apply_persona"): scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice, scan_dir) if wav: persona = _load_meta(wav).get("persona", "") if persona: llm_url = (settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/") try: text = await asyncio.to_thread(_rewrite_with_persona_sync, text, persona, llm_url) except Exception as e: raise HTTPException(502, f"Persona rewrite failed: {e}") try: audio, media_type = await asyncio.to_thread(_preview_request_audio, text, voice, settings, instruct, backend) except Exception as e: raise HTTPException(502, f"TTS error: {e}") audio, media_type, ext, _duration_sec, _clipped = _prepare_proxy_audio(audio, media_type, response_format) return Response( content=audio, media_type=media_type, headers={"Content-Disposition": f'inline; filename="{voice}_preview.{ext}"'}, ) @app.post("/api/tts-style-variation") async def tts_style_variation(request: Request): data = await request.json() source_voice = str(data.get("source_voice") or data.get("voice") or "").strip() new_voice_id = str(data.get("voice_id") or data.get("new_voice_id") or "").strip() text = str(data.get("text") or data.get("transcript") or "").strip() instruct = str(data.get("instruct") or data.get("style_instruction") or "").strip() backend = _clean_preview_backend(data.get("backend", "customvoice")) if not source_voice: raise HTTPException(400, "source_voice is required") if not new_voice_id: raise HTTPException(400, "new voice id is required") if not re.match(r"^[A-Za-z0-9_\-\.]+$", new_voice_id): raise HTTPException(400, "Voice ID may only contain A-Z, 0-9, _, -, .") if not text: raise HTTPException(400, "text/transcript is required") if not instruct: raise HTTPException(400, "style instruction is required") settings = _load_settings() try: audio, media_type = await asyncio.to_thread(_preview_request_audio, text, source_voice, settings, instruct, backend) audio, media_type, _ext, duration, _clipped = _prepare_proxy_audio(audio, media_type, "wav") except Exception as e: raise HTTPException(502, f"Style variation synthesis failed: {e}") tmp = TEMP_DIR / f"{uuid.uuid4().hex}_style_variation.wav" tmp.write_bytes(audio) out_dir = _active_voices_dir(settings) out_dir.mkdir(parents=True, exist_ok=True) wav_dest = out_dir / f"{new_voice_id}.wav" txt_dest = out_dir / f"{new_voice_id}.reference.txt" _remove_audio_variants(out_dir, new_voice_id) loudness = _export_normalized_wav(tmp, wav_dest) txt_dest.write_text(text, encoding="utf-8") meta = _load_meta(wav_dest) meta.update({ "enabled": True, "loudness": loudness, "source_voice": source_voice, "style_instruction": instruct, "style_backend": backend, "note": f"Style variation of {source_voice}: {instruct[:180]}", "needs_tts_restart": True, }) _save_meta(wav_dest, meta) return { "ok": True, "voice_id": new_voice_id, "source_voice": source_voice, "backend": backend, "wav": str(wav_dest), "txt": str(txt_dest), "duration": duration, "loudness": loudness, "needs_tts_restart": True, } _TTS_STREAM_SESSION_TTL = 15 * 60 _tts_stream_sessions: dict[str, dict] = {} def _purge_tts_stream_sessions() -> None: now = time.time() expired = [sid for sid, item in _tts_stream_sessions.items() if now - item.get("created", 0) > _TTS_STREAM_SESSION_TTL] for sid in expired: _tts_stream_sessions.pop(sid, None) def _tts_stream_request_config(text: str, voice: str, settings: dict, instruct: str = "") -> tuple[str, dict, dict]: stream_url = settings.get("tts_stream_url") or settings.get("tts_url") or _TTS_STREAM_DEFAULT tts_url = _validate_http_url(stream_url, allow_private=True).rstrip("/") tts_key = settings.get("tts_api_key", "").strip() tts_hdrs = {"Authorization": f"Bearer {tts_key}"} if tts_key else {} payload = {"model": "tts-1", "input": text, "voice": voice, "response_format": "wav"} if instruct.strip(): payload["instruct"] = instruct.strip() _apply_tts_extra_params(payload, settings, "streaming") return f"{tts_url}/v1/audio/speech", payload, tts_hdrs def _open_tts_stream_response(text: str, voice: str, settings: dict, instruct: str = "") -> requests.Response: endpoint, payload, tts_hdrs = _tts_stream_request_config(text, voice, settings, instruct) resp = _post_tts_with_fallback(endpoint, payload, tts_hdrs, stream=True, timeout=(10, 900)) try: resp.raise_for_status() except Exception as exc: detail = "" try: detail = resp.text[:500] except Exception: pass resp.close() raise RuntimeError(f"streaming backend error: {exc}{(': ' + detail) if detail else ''}") from exc return resp def _read_tts_stream_response(resp: requests.Response) -> tuple[bytes, str]: try: media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return b"".join(chunk for chunk in resp.iter_content(chunk_size=64 * 1024) if chunk), media_type finally: resp.close() def _iter_tts_stream_response(resp: requests.Response): try: for chunk in resp.iter_content(chunk_size=64 * 1024): if chunk: yield chunk finally: resp.close() @app.get("/api/tts-stream-health") async def tts_stream_health(): settings = _load_settings() try: stream_url = settings.get("tts_stream_url") or settings.get("tts_url") or _TTS_STREAM_DEFAULT base_url = _validate_http_url(stream_url, allow_private=True).rstrip("/") resp = await asyncio.to_thread(requests.get, f"{base_url}/health", timeout=3) return {"ok": resp.ok, "status_code": resp.status_code, "url": base_url} except Exception as e: return {"ok": False, "error": str(e)} @app.post("/api/tts-stream-session") async def tts_stream_session(request: Request): data = await request.json() text = str(data.get("text", "")).strip() voice = str(data.get("voice", "")).strip() if not voice: raise HTTPException(400, "voice is required") if not text: raise HTTPException(400, "text is required") _purge_tts_stream_sessions() sid = uuid.uuid4().hex instruct = str(data.get("instruct") or data.get("style_instruction") or "").strip() _tts_stream_sessions[sid] = {"created": time.time(), "text": text, "voice": voice, "instruct": instruct} return {"ok": True, "url": f"/api/tts-stream-session/{sid}"} @app.get("/api/tts-stream-session/{sid}") async def tts_stream_playback(sid: str): _purge_tts_stream_sessions() item = _tts_stream_sessions.pop(sid, None) if item is None: raise HTTPException(404, "stream session expired or not found") settings = _load_settings() try: resp = await asyncio.to_thread(_open_tts_stream_response, item["text"], item["voice"], settings, item.get("instruct", "")) except Exception as e: raise HTTPException(502, f"TTS stream error: {e}") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" return StreamingResponse( _iter_tts_stream_response(resp), media_type=media_type, headers={ "Cache-Control": "no-store", "X-Accel-Buffering": "no", "Content-Disposition": f"inline; filename=\"{item['voice']}_stream.wav\"", }, ) # ── Voice design ────────────────────────────────────────────────────────────── @app.post("/api/voice-design") async def voice_design(request: Request): data = await request.json() instruct: str = data.get("instruct", "").strip() sample_text: str = data.get("sample_text", "Hello! This is a voice design sample.").strip() language: str = data.get("language", "Auto") gender: str = data.get("gender", "") dialogue = bool(data.get("dialogue")) if not instruct: raise HTTPException(400, "instruct (voice description) is required") settings = _load_settings() try: if dialogue: parsed = _parse_voice_design_dialogue(instruct, sample_text) if not parsed: raise RuntimeError("dialogue mode needs speaker profiles and Speaker: text turns") audio, _media_type = await asyncio.to_thread( _voice_design_request_audio, instruct, sample_text, language, settings, "", ) else: audio, _media_type = await asyncio.to_thread( _voice_design_request_audio, instruct, sample_text, language, settings, gender, ) except Exception as e: raise HTTPException(502, f"Voice design error: {e}") audio, _media_type, _ext, _duration_sec, _clipped = _prepare_proxy_audio(audio, "audio/wav", "wav") tmp = TEMP_DIR / f"{uuid.uuid4().hex}_designed.wav" tmp.write_bytes(audio) fid = uuid.uuid4().hex _registry[fid] = tmp return {"id": fid, "duration": _duration(tmp)} # ── OpenAI-compatible proxy for exported and virtual VoiceDesign voices ─────── def _virtual_voice_id(name: str) -> str: return f"vd_{_slug_voice_design_name(name)}" def _resolve_virtual_voice(voice: str) -> tuple[str, dict] | None: if not voice.startswith("vd_"): return None wanted = _slug_voice_design_name(voice[3:]) presets = _load_design_presets() for name, preset in presets.items(): if _slug_voice_design_name(name) == wanted: return name, preset return None @app.get("/v1/models") async def openai_models_proxy(): now = 1686935002 data = [] seen = set() for name in sorted(_load_design_presets()): seen.add(_virtual_voice_id(name)) data.append({ "id": _virtual_voice_id(name), "object": "model", "created": now, "owned_by": "voice-design", }) for rule in _load_tts_routes(): alias = str(rule.get("input_voice", "")).strip() if rule.get("enabled", True) and alias and alias != "*" and alias not in seen: seen.add(alias) data.append({ "id": alias, "object": "model", "created": now, "owned_by": f"route:{rule.get('app', '*')}", }) try: voices = await tts_voices() for item in voices: if isinstance(item, str): voice_id = item elif isinstance(item, dict): voice_id = item.get("id") or item.get("voice") else: voice_id = str(item) if item is not None else "" if voice_id and voice_id not in seen: seen.add(str(voice_id)) data.append({ "id": str(voice_id), "object": "model", "created": now, "owned_by": "qwen", }) except Exception: pass return {"object": "list", "data": data} @app.get("/v1/audio/models") async def openai_audio_models_proxy(): return await openai_models_proxy() @app.get("/v1/audio/voices") async def openai_audio_voices_proxy(): models = await openai_models_proxy() return [m["id"] for m in models["data"]] @app.post("/api/tts-route-test") async def tts_route_test(request: Request): data = await request.json() text = str(data.get("input") or data.get("text") or "").strip() voice = str(data.get("voice") or "default").strip() app_name = _canonical_app_name(str(data.get("app") or data.get("client") or "Open WebUI").strip()) if not text: raise HTTPException(400, "input is required") routed_voice, route = _resolve_tts_route(app_name, voice, text) backend = _route_backend(route, routed_voice) settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) routed_audio = None if backend in {"voice_design", "nvidia_magpie"} else _find_voice_audio(routed_voice, scan_dir) sound_status = {} for key in ("before_sound", "after_sound"): raw = str((route or {}).get(key, "")) if raw: try: path = _route_sound_path(settings, raw) sound_status[key] = {"ok": True, "path": str(path)} except Exception as e: sound_status[key] = {"ok": False, "error": str(e), "path": raw} result = { "app": app_name, "requested_voice": voice, "routed_voice": routed_voice, "backend": backend, "detected_language": str(route.get("detected_language", _detect_text_language(text))) if route else _detect_text_language(text), "matched": bool(route), "route": route, "voice_health": _voice_health(routed_audio) if routed_audio else None, "sounds": sound_status, } _routing_log_add( kind="test", status="matched" if route else "no_match", app=app_name, requested_voice=voice, routed_voice=routed_voice, backend=backend, language=result["detected_language"], matched=bool(route), route_id=str((route or {}).get("id", "")), text_preview=text[:160], sounds=sound_status, ) return result @app.post("/v1/audio/speech") async def openai_speech_proxy(request: Request): data = await request.json() text = str(data.get("input") or data.get("text") or "").strip() voice = str(data.get("voice") or data.get("model") or "").strip() response_format = _requested_response_format(data) request_app = _canonical_app_name(str(data.get("app") or data.get("client") or "").strip()) if (data.get("app") or data.get("client")) else _request_app_name(request) original_voice = voice if not text: _routing_log_request( request, status="error", app=request_app, requested_voice=original_voice, routed_voice=voice, backend="", route=None, response_format=response_format, text=text, error="input is required", ) raise HTTPException(400, "input is required") if not voice: _routing_log_request( request, status="error", app=request_app, requested_voice=original_voice, routed_voice=voice, backend="", route=None, response_format=response_format, text=text, error="voice is required", ) raise HTTPException(400, "voice is required") settings = _load_settings() voice, route = _resolve_tts_route(request_app, voice, text) backend = _route_backend(route, voice) style_instruction = str(data.get("instruct") or data.get("style_instruction") or "") virtual = _resolve_virtual_voice(voice) route_has_sounds = bool((route or {}).get("before_sound") or (route or {}).get("after_sound")) if backend == "streaming" and not virtual and response_format == "wav" and not route_has_sounds: try: resp = await asyncio.to_thread(_open_tts_stream_response, text, voice, settings, style_instruction) except Exception as e: _routing_log_request( request, status="error", app=request_app, requested_voice=original_voice, routed_voice=voice, backend=backend, route=route, response_format=response_format, text=text, error=f"TTS stream proxy error: {e}", ) raise HTTPException(502, f"TTS stream proxy error: {e}") media_type = resp.headers.get("content-type", "audio/wav").split(";", 1)[0] or "audio/wav" _routing_log_request( request, status="streaming", app=request_app, requested_voice=original_voice, routed_voice=voice, backend=backend, route=route, response_format=response_format, text=text, media_type=media_type, sounds=[], ) return StreamingResponse( _iter_tts_stream_response(resp), media_type=media_type, headers={ "Cache-Control": "no-store", "X-Accel-Buffering": "no", "Content-Disposition": f'inline; filename="{voice}_speech.wav"', "X-TTS-Voice-Requested": original_voice, "X-TTS-Voice-Routed": voice, "X-TTS-Route-Backend": backend, "X-TTS-Route-Language": str(route.get("detected_language", "")) if route else "", "X-TTS-Response-Format": response_format, "X-TTS-Audio-Clipped": "false", "X-TTS-Route-Sounds": "", }, ) try: if virtual: _name, preset = virtual instruct = str(preset.get("description", "")).strip() if not instruct: raise RuntimeError(f"Virtual voice '{voice}' has no description") audio, media_type = await asyncio.to_thread( _voice_design_request_audio, instruct, text, str(preset.get("language", "Auto")), settings, str(preset.get("gender", "")), ) elif backend == "voice_design": audio, media_type = await asyncio.to_thread( _voice_design_voice_request_audio, voice, text, settings, style_instruction, str(data.get("language") or "Auto"), ) elif backend == "streaming": resp = await asyncio.to_thread(_open_tts_stream_response, text, voice, settings, style_instruction) audio, media_type = await asyncio.to_thread(_read_tts_stream_response, resp) if not audio: raise RuntimeError("backend returned empty audio") elif backend == "nvidia_magpie": audio, media_type = await asyncio.to_thread( _tts_request_audio, text, voice, settings, style_instruction, _preview_backend_base_url(settings, "nvidia_magpie"), settings.get("tts_api_key", ""), "nvidia_magpie", "nvidia_magpie", ) elif backend in {"nvidia_zeroshot", "nvidia_flow"}: audio, media_type = await asyncio.to_thread( _nvidia_clone_request_audio, text, voice, settings, "flow" if backend == "nvidia_flow" else "zeroshot", str(data.get("audio_prompt_transcript") or ""), str(data.get("language") or ""), ) else: audio, media_type = await asyncio.to_thread( _tts_request_audio, text, voice, settings, style_instruction, ) except Exception as e: _routing_log_request( request, status="error", app=request_app, requested_voice=original_voice, routed_voice=voice, backend=backend, route=route, response_format=response_format, text=text, error=f"TTS proxy error: {e}", ) raise HTTPException(502, f"TTS proxy error: {e}") try: audio, media_type, applied_sounds = _apply_route_sounds(audio, media_type, route, settings) except Exception as e: _routing_log_request( request, status="error", app=request_app, requested_voice=original_voice, routed_voice=voice, backend=backend, route=route, response_format=response_format, text=text, error=f"TTS route sound error: {e}", ) raise HTTPException(502, f"TTS route sound error: {e}") audio, media_type, ext, duration, clipped = _prepare_proxy_audio(audio, media_type, response_format) logger.info( "TTS proxy app=%s voice=%s routed=%s backend=%s lang=%s format=%s bytes=%s duration=%s clipped=%s sounds=%s", request_app, original_voice, voice, backend, str(route.get("detected_language", "")) if route else "", response_format, len(audio), f"{duration:.2f}" if duration is not None else "?", clipped, ",".join(applied_sounds) if applied_sounds else "-", ) _routing_log_request( request, status="ok" if route else "no_match", app=request_app, requested_voice=original_voice, routed_voice=voice, backend=backend, route=route, response_format=response_format, text=text, bytes=len(audio), duration=duration, clipped=clipped, media_type=media_type, sounds=applied_sounds, ) return Response( content=audio, media_type=media_type, headers={ "Content-Disposition": f'inline; filename="{voice}_speech.{ext}"', "X-TTS-Voice-Requested": original_voice, "X-TTS-Voice-Routed": voice, "X-TTS-Route-Backend": backend, "X-TTS-Route-Language": str(route.get("detected_language", "")) if route else "", "X-TTS-Response-Format": response_format, "X-TTS-Audio-Duration": f"{duration:.3f}" if duration is not None else "", "X-TTS-Audio-Clipped": "true" if clipped else "false", "X-TTS-Route-Sounds": ",".join(applied_sounds), }, ) @app.post("/v1") async def openai_speech_proxy_v1_shortcut(request: Request): return await openai_speech_proxy(request) # ── LLM text refinement ─────────────────────────────────────────────────────── @app.post("/api/refine-text") async def refine_text(request: Request): """Clean up raw STT transcription using a local OpenAI-compatible LLM.""" data = await request.json() text: str = (data.get("text") or "").strip() llm_url: str = (data.get("llm_url") or "http://localhost:11434/v1").rstrip("/") model: str = (data.get("model") or "").strip() toggles: dict = data.get("toggles") or {} if not text: raise HTTPException(400, "No text to refine") rules = [] if toggles.get("fillers", True): rules.append("Remove filler words (um, uh, like, you know, basically, literally, I mean, so, right, etc.)") if toggles.get("repetitions", True): rules.append("Remove repeated words and false starts (e.g. 'the the dog' → 'the dog', 'I was- I was going' → 'I was going')") if toggles.get("corrections", True): rules.append("Remove self-corrections and restarts, keeping only the final intended phrasing") if toggles.get("punctuation", True): rules.append("Fix punctuation, capitalisation, and sentence boundaries") if not rules: return {"text": text, "original": text} system = ( "You are a transcription cleanup assistant. " "Apply ONLY the following rules to the user's text. " "Return ONLY the cleaned text — no explanations, no quotes, no markdown:\n" + "\n".join(f"- {r}" for r in rules) ) payload: dict = { "messages": [ {"role": "system", "content": system}, {"role": "user", "content": text}, ], "temperature": 0.1, "max_tokens": 2048, } if model: payload["model"] = model try: resp = requests.post( f"{llm_url}/chat/completions", json=payload, headers={"Authorization": "Bearer no-key"}, timeout=60, ) resp.raise_for_status() refined = resp.json()["choices"][0]["message"]["content"].strip() if refined.startswith('"') and refined.endswith('"'): refined = refined[1:-1].strip() return {"text": refined, "original": text} except Exception as e: raise HTTPException(502, f"LLM refinement failed: {e}") # ── LLM persona rewrite ─────────────────────────────────────────────────────── def _rewrite_with_persona_sync(text: str, persona: str, llm_url: str, model: str = "") -> str: """Inline synchronous persona rewrite; raises RuntimeError on failure.""" system = ( f"Rephrase the user's text as if spoken by this character: {persona}\n" "Keep the same meaning but adapt vocabulary, tone, and style to the character. " "Return ONLY the rephrased text — no quotes, no explanation." ) payload: dict = { "messages": [ {"role": "system", "content": system}, {"role": "user", "content": text}, ], "temperature": 0.3, "max_tokens": 512, } if model: payload["model"] = model resp = requests.post( f"{llm_url.rstrip('/')}/chat/completions", json=payload, headers={"Authorization": "Bearer no-key"}, timeout=60, ) resp.raise_for_status() result = resp.json()["choices"][0]["message"]["content"].strip() if result.startswith('"') and result.endswith('"'): result = result[1:-1].strip() return result @app.post("/api/rewrite-with-persona") async def rewrite_with_persona(request: Request): """Rewrite user text in a voice persona's character using a local LLM.""" data = await request.json() text: str = (data.get("text") or "").strip() persona: str = (data.get("persona") or "").strip() llm_url: str = (data.get("llm_url") or "http://localhost:11434/v1").rstrip("/") model: str = (data.get("model") or "").strip() mode: str = (data.get("mode") or "rewrite").strip() if not persona: raise HTTPException(400, "No persona defined for this voice") if not text and mode != "compose": raise HTTPException(400, "No text provided") if mode == "compose": system = ( f"You are a voice assistant with this character: {persona}\n" "Write a single natural utterance in this character's voice about the topic given. " "Return ONLY the utterance — no quotes, no explanation." ) user_msg = text or "Introduce yourself briefly." temp = 0.9 else: system = ( f"Rephrase the user's text as if spoken by this character: {persona}\n" "Keep the same meaning but adapt vocabulary, tone, and style to the character. " "Return ONLY the rephrased text — no quotes, no explanation." ) user_msg = text temp = 0.3 payload: dict = { "messages": [ {"role": "system", "content": system}, {"role": "user", "content": user_msg}, ], "temperature": temp, "max_tokens": 512, } if model: payload["model"] = model try: resp = requests.post( f"{llm_url}/chat/completions", json=payload, headers={"Authorization": "Bearer no-key"}, timeout=60, ) resp.raise_for_status() result = resp.json()["choices"][0]["message"]["content"].strip() if result.startswith('"') and result.endswith('"'): result = result[1:-1].strip() return {"text": result, "original": text, "persona": persona} except Exception as e: raise HTTPException(502, f"LLM persona rewrite failed: {e}") # ── Audio effects ───────────────────────────────────────────────────────────── def _apply_audio_effects(audio_bytes: bytes, effects: list) -> bytes: try: from pedalboard import Pedalboard, Reverb, Chorus, Delay, Compressor, Gain, HighpassFilter, LowpassFilter, PitchShift # type: ignore import numpy as np # type: ignore except ImportError: raise RuntimeError("pedalboard is not installed — run: pip install pedalboard numpy") with io.BytesIO(audio_bytes) as buf: with wave.open(buf, "rb") as wf: n_channels = wf.getnchannels() sample_rate = wf.getframerate() n_frames = wf.getnframes() raw = wf.readframes(n_frames) sampwidth = wf.getsampwidth() import numpy as np # noqa: F811 dtype = {1: np.int8, 2: np.int16, 4: np.int32}.get(sampwidth, np.int16) samples = np.frombuffer(raw, dtype=dtype).astype(np.float32) / float(np.iinfo(dtype).max) samples = samples.reshape(1, -1) if n_channels == 1 else samples.reshape(-1, n_channels).T board = [] for fx in effects: t = fx.get("type", "") p = fx.get("params", {}) if t == "reverb": board.append(Reverb( room_size=float(p.get("room_size", 0.35)), damping=float(p.get("damping", 0.5)), wet_level=float(p.get("wet", 0.25)), dry_level=float(p.get("dry", 0.8)), )) elif t == "chorus": board.append(Chorus( rate_hz=float(p.get("rate_hz", 1.0)), depth=float(p.get("depth", 0.25)), mix=float(p.get("mix", 0.5)), )) elif t == "delay": board.append(Delay( delay_seconds=float(p.get("delay_s", 0.25)), feedback=float(p.get("feedback", 0.3)), mix=float(p.get("mix", 0.4)), )) elif t == "compressor": board.append(Compressor( threshold_db=float(p.get("threshold_db", -20.0)), ratio=float(p.get("ratio", 4.0)), attack_ms=float(p.get("attack_ms", 10.0)), release_ms=float(p.get("release_ms", 100.0)), )) elif t == "gain": board.append(Gain(gain_db=float(p.get("gain_db", 0.0)))) elif t == "highpass": board.append(HighpassFilter(cutoff_frequency_hz=float(p.get("cutoff_hz", 80.0)))) elif t == "lowpass": board.append(LowpassFilter(cutoff_frequency_hz=float(p.get("cutoff_hz", 8000.0)))) elif t == "pitch_shift": board.append(PitchShift(semitones=float(p.get("semitones", 0.0)))) if board: samples = Pedalboard(board)(samples, sample_rate) out = np.clip(samples, -1.0, 1.0) pcm = ((out[0] if out.shape[0] == 1 else out.T.reshape(-1)) * 32767).astype(np.int16).tobytes() buf_out = io.BytesIO() with wave.open(buf_out, "wb") as wf: wf.setnchannels(n_channels) wf.setsampwidth(2) wf.setframerate(sample_rate) wf.writeframes(pcm) return buf_out.getvalue() @app.post("/api/audio/effects") async def audio_effects(request: Request): """Apply an effects chain to a WAV file.""" form = await request.form() audio_file = form.get("audio") effects_json = str(form.get("effects") or "[]") if audio_file is None: raise HTTPException(400, "No audio file provided") audio_bytes = await audio_file.read() try: effects = json.loads(effects_json) except Exception: raise HTTPException(400, "Invalid effects JSON") try: result = await asyncio.to_thread(_apply_audio_effects, audio_bytes, effects) return Response(content=result, media_type="audio/wav") except RuntimeError as e: raise HTTPException(501, str(e)) except Exception as e: raise HTTPException(500, f"Effects processing failed: {e}") # ── Voices export / import ──────────────────────────────────────────────────── _EXPORT_SKIP_KEYS = {"groq_api_key", "whisper_api_key", "tts_api_key", "voice_design_api_key", "elevenlabs_api_key"} _IMPORT_ALLOWED_SUFFIXES = set(_AUDIO_EXTS + [".reference.txt", ".meta.json", ".jpg", ".jpeg", ".png", ".webp"]) @app.get("/api/voices/export") async def voices_export(): """Export all voices + non-sensitive settings as a ZIP archive.""" import zipfile settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) buf = io.BytesIO() with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf: safe_settings = {k: v for k, v in settings.items() if k not in _EXPORT_SKIP_KEYS} zf.writestr("settings.json", json.dumps(safe_settings, indent=2)) if scan_dir.exists(): for f in scan_dir.rglob("*"): if f.is_file(): try: zf.write(f, str(f.relative_to(scan_dir.parent))) except Exception: pass buf.seek(0) ts = datetime.now().strftime("%Y%m%d_%H%M%S") return Response( content=buf.read(), media_type="application/zip", headers={"Content-Disposition": f'attachment; filename="voices_export_{ts}.zip"'}, ) @app.post("/api/voices/import") async def voices_import(file: UploadFile = File(...)): """Import voices from a ZIP archive (skips settings.json and unsafe paths).""" import zipfile settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) scan_dir.mkdir(parents=True, exist_ok=True) content = await file.read() if len(content) > _MAX_UPLOAD_BYTES: raise HTTPException(413, "ZIP file too large") try: imported = 0 with zipfile.ZipFile(io.BytesIO(content)) as zf: for info in zf.infolist(): if info.is_dir(): continue parts = Path(info.filename).parts if any(p in ("..", "") for p in parts) or Path(info.filename).name == "settings.json": continue if Path(info.filename).suffix.lower() not in _IMPORT_ALLOWED_SUFFIXES: continue rel = parts[1:] if len(parts) > 1 else parts dest = scan_dir / Path(*rel) dest.parent.mkdir(parents=True, exist_ok=True) dest.write_bytes(zf.read(info)) imported += 1 except zipfile.BadZipFile: raise HTTPException(400, "Not a valid ZIP file") except Exception as e: raise HTTPException(500, f"Import failed: {e}") return {"ok": True, "imported": imported} # ── /speak REST endpoint ────────────────────────────────────────────────────── def _resolve_speak_voice(settings: dict, client_id: str, explicit_voice: str) -> str: if explicit_voice: return explicit_voice bindings: dict = settings.get("client_voice_bindings") or {} if client_id and client_id in bindings: return bindings[client_id] return settings.get("captures_default_voice") or "" @app.post("/speak") async def speak(request: Request): data = await request.json() text: str = str(data.get("text") or "").strip() if not text: raise HTTPException(400, "text is required") explicit_voice: str = str(data.get("voice") or data.get("profile_id") or data.get("profile") or "").strip() backend: str = _clean_preview_backend(str(data.get("backend") or "voice_clone")) apply_persona: bool = bool(data.get("apply_persona") or data.get("personality")) client_id: str = request.headers.get("X-Voice-Creator-Client-Id", "").strip() settings = _load_settings() voice = _resolve_speak_voice(settings, client_id, explicit_voice) if not voice: raise HTTPException(400, "No voice specified and no default voice configured") if apply_persona: scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice, scan_dir) if wav: persona = _load_meta(wav).get("persona", "") if persona: llm_url = (settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/") try: text = await asyncio.to_thread(_rewrite_with_persona_sync, text, persona, llm_url) except Exception as e: raise HTTPException(502, f"Persona rewrite failed: {e}") instruct = str(data.get("instruct") or "") try: audio, media_type = await asyncio.to_thread( _preview_request_audio, text, voice, settings, instruct, backend ) except Exception as e: raise HTTPException(502, f"TTS error: {e}") return Response(content=audio, media_type=media_type) @app.get("/speak/bindings") async def get_speak_bindings(): s = _load_settings() return {"bindings": s.get("client_voice_bindings") or {}} @app.put("/speak/bindings/{client_id}") async def put_speak_binding(client_id: str, request: Request): data = await request.json() voice = str(data.get("voice") or "").strip() if not voice: raise HTTPException(400, "voice is required") s = _load_settings() bindings: dict = dict(s.get("client_voice_bindings") or {}) bindings[client_id] = voice s["client_voice_bindings"] = bindings _save_settings(s) return {"ok": True, "client_id": client_id, "voice": voice} @app.delete("/speak/bindings/{client_id}") async def delete_speak_binding(client_id: str): s = _load_settings() bindings: dict = dict(s.get("client_voice_bindings") or {}) if client_id not in bindings: raise HTTPException(404, f"No binding for client '{client_id}'") del bindings[client_id] s["client_voice_bindings"] = bindings _save_settings(s) return {"ok": True, "client_id": client_id} # ── MCP JSON-RPC 2.0 server ─────────────────────────────────────────────────── _MCP_SERVER_INFO = {"name": "tts-voice-creator", "version": "1.0.0"} _MCP_TOOLS = [ { "name": "speak", "description": "Generate speech audio from text using a cloned voice. Returns a data URI with the WAV audio.", "inputSchema": { "type": "object", "properties": { "text": {"type": "string", "description": "Text to synthesize"}, "voice": {"type": "string", "description": "Voice ID (optional — uses per-client binding or default if omitted)"}, "apply_persona": {"type": "boolean", "description": "Rewrite text through the voice's persona before synthesis"}, }, "required": ["text"], }, }, { "name": "transcribe", "description": "Transcribe base64-encoded WAV audio to text.", "inputSchema": { "type": "object", "properties": { "audio_base64": {"type": "string", "description": "Base64-encoded audio bytes (WAV preferred)"}, }, "required": ["audio_base64"], }, }, { "name": "list_captures", "description": "List the 20 most recently generated audio files.", "inputSchema": {"type": "object", "properties": {}}, }, { "name": "list_profiles", "description": "List all available voice profiles with their language, persona, and enabled state.", "inputSchema": {"type": "object", "properties": {}}, }, ] async def _mcp_tool_speak(args: dict, client_id: str) -> dict: text = str(args.get("text") or "").strip() if not text: raise ValueError("text is required") explicit_voice = str(args.get("voice") or "").strip() apply_persona = bool(args.get("apply_persona", False)) settings = _load_settings() voice = _resolve_speak_voice(settings, client_id, explicit_voice) if not voice: raise ValueError("No voice specified and no default voice configured") if apply_persona: scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) wav = _find_voice_audio(voice, scan_dir) if wav: persona = _load_meta(wav).get("persona", "") if persona: llm_url = (settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/") text = await asyncio.to_thread(_rewrite_with_persona_sync, text, persona, llm_url) audio, _media_type = await asyncio.to_thread( _preview_request_audio, text, voice, settings, "", "voice_clone" ) audio_b64 = base64.b64encode(audio).decode() return { "content": [ {"type": "text", "text": f"Generated {len(audio)} bytes of speech audio for voice '{voice}'."}, {"type": "resource", "resource": {"uri": f"data:audio/wav;base64,{audio_b64}", "mimeType": "audio/wav"}}, ] } async def _mcp_tool_transcribe(args: dict) -> dict: raw = args.get("audio_base64") or "" try: audio_bytes = base64.b64decode(raw) except Exception: raise ValueError("audio_base64 is not valid base64") tmp = TEMP_DIR / f"{uuid.uuid4().hex}_mcp_transcribe.wav" tmp.write_bytes(audio_bytes) settings = _load_settings() try: text, used_backend = await asyncio.to_thread(_transcribe_audio, tmp, settings, "configured") finally: try: tmp.unlink(missing_ok=True) except Exception: pass return {"content": [{"type": "text", "text": text}], "backend": used_backend} async def _mcp_tool_list_captures() -> dict: settings = _load_settings() out_dir = _active_voices_dir(settings) if not out_dir.exists(): return {"content": [{"type": "text", "text": "[]"}]} files = sorted( (p for p in out_dir.rglob("*.wav") if not _is_internal_voice_file(p)), key=lambda p: p.stat().st_mtime, reverse=True, )[:20] result = [{"name": p.stem, "path": str(p), "mtime": p.stat().st_mtime} for p in files] return {"content": [{"type": "text", "text": json.dumps(result)}]} async def _mcp_tool_list_profiles() -> dict: settings = _load_settings() scan_dir = Path(settings.get("voices_scan_dir", _VOICES_DIR_DEFAULT)) profiles = [] seen: set[str] = set() for audio in sorted(_voice_audio_files(scan_dir), key=lambda p: p.stem.lower()): if audio.stem in seen: continue seen.add(audio.stem) meta = _load_meta(audio) parts = audio.stem.split("_", 2) lang = parts[0].upper() if parts else "" profiles.append({ "id": audio.stem, "name": audio.stem, "lang": lang, "persona": meta.get("persona", ""), "enabled": meta.get("enabled", True), }) return {"content": [{"type": "text", "text": json.dumps(profiles)}]} def _mcp_error_response(code: int, message: str, rpc_id) -> Response: body = {"jsonrpc": "2.0", "error": {"code": code, "message": message}, "id": rpc_id} return Response(content=json.dumps(body), media_type="application/json") @app.post("/mcp") async def mcp_jsonrpc(request: Request): try: body = await request.json() except Exception: return _mcp_error_response(-32700, "Parse error", None) rpc_id = body.get("id") method = body.get("method", "") params = body.get("params") or {} client_id = request.headers.get("X-Voice-Creator-Client-Id", "").strip() try: if method == "initialize": result = { "protocolVersion": "2024-11-05", "serverInfo": _MCP_SERVER_INFO, "capabilities": {"tools": {}}, } elif method == "notifications/initialized": return Response(status_code=204) elif method == "tools/list": result = {"tools": _MCP_TOOLS} elif method == "tools/call": tool_name = str(params.get("name") or "") tool_args = params.get("arguments") or {} if tool_name == "speak": result = await _mcp_tool_speak(tool_args, client_id) elif tool_name == "transcribe": result = await _mcp_tool_transcribe(tool_args) elif tool_name == "list_captures": result = await _mcp_tool_list_captures() elif tool_name == "list_profiles": result = await _mcp_tool_list_profiles() else: return _mcp_error_response(-32601, f"Unknown tool: {tool_name}", rpc_id) else: return _mcp_error_response(-32601, f"Method not found: {method}", rpc_id) except ValueError as e: return _mcp_error_response(-32602, str(e), rpc_id) except Exception as e: logger.exception("MCP tool error in method %s", method) return _mcp_error_response(-32000, f"Server error: {e}", rpc_id) return Response( content=json.dumps({"jsonrpc": "2.0", "result": result, "id": rpc_id}), media_type="application/json", ) @app.get("/mcp") async def mcp_sse(request: Request): """SSE keep-alive stream (satisfies MCP spec GET /mcp requirement).""" async def _keepalive() -> AsyncGenerator[str, None]: yield ": mcp-sse-ready\n\n" while True: await asyncio.sleep(15) yield ": ping\n\n" return StreamingResponse( _keepalive(), media_type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, ) # ── Conversation Playground ──────────────────────────────────────────────────── @app.get("/api/conversation/llm-models") async def conversation_llm_models(url: str = ""): """List models from a local LLM endpoint (Ollama / vLLM / LM Studio).""" settings = _load_settings() base = (url or settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/") try: base = _validate_http_url(base, allow_private=True) r = requests.get(f"{base}/models", timeout=5, headers={"Authorization": "Bearer no-key"}) if r.status_code == 200: payload = r.json() data = payload.get("data", []) if isinstance(payload, dict) else [] models = [ str(item["id"]) if isinstance(item, dict) and item.get("id") else str(item) for item in data if item ] return {"models": models, "url": base} except Exception: pass return {"models": [], "url": base} @app.post("/api/conversation/turn") async def conversation_turn( audio: UploadFile = File(...), stt_backend: str = Form("configured"), llm_url: str = Form(""), llm_model: str = Form(""), tts_backend: str = Form("voice_clone"), tts_voice: str = Form(""), system_prompt: str = Form("You are a helpful voice assistant. Keep replies short and conversational."), history: str = Form("[]"), ): """Stream a full conversation turn (STT → LLM → TTS) as Server-Sent Events.""" settings = _load_settings() eff_llm_url = (llm_url or settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/") suffix = Path(audio.filename or "audio.webm").suffix.lower() or ".webm" if suffix not in _UPLOAD_EXTS: suffix = ".webm" tmp = TEMP_DIR / f"{uuid.uuid4().hex}_conv{suffix}" try: with tmp.open("wb") as f: _copy_limited(audio.file, f, _MAX_UPLOAD_BYTES) wav_tmp = tmp if suffix == ".wav" else _to_wav_24k(tmp) except Exception as e: tmp.unlink(missing_ok=True) raise HTTPException(400, f"Audio upload failed: {e}") try: hist = json.loads(history) if history else [] if not isinstance(hist, list): hist = [] except Exception: hist = [] stt_be = _clean_stt_backend(stt_backend) tts_be = _clean_preview_backend(tts_backend) _tmp, _wav = tmp, wav_tmp async def generate(): t0 = time.monotonic() stt_ms = llm_ttft_ms = llm_total_ms = tts_ms = None transcript = llm_text = "" def sse(obj: dict) -> str: return f"data: {json.dumps(obj)}\n\n" # 1. STT try: t_stt = time.monotonic() transcript, _ = await asyncio.to_thread(_transcribe_audio, _wav, settings, stt_be) stt_ms = int((time.monotonic() - t_stt) * 1000) yield sse({"type": "transcript", "text": transcript, "stt_ms": stt_ms}) except Exception as e: yield sse({"type": "error", "stage": "stt", "message": str(e)}) return finally: for p in {_tmp, _wav}: try: p.unlink(missing_ok=True) except Exception: pass if not transcript.strip(): yield sse({"type": "error", "stage": "stt", "message": "No speech detected."}) return # 2. LLM stream messages = [{"role": "system", "content": system_prompt}] messages.extend(hist[-20:]) messages.append({"role": "user", "content": transcript}) llm_payload: dict = {"messages": messages, "stream": True, "temperature": 0.7, "max_tokens": 512} if llm_model: llm_payload["model"] = llm_model try: t_llm = time.monotonic() llm_resp = await asyncio.to_thread(lambda: requests.post( f"{eff_llm_url}/chat/completions", json=llm_payload, headers={"Authorization": "Bearer no-key"}, stream=True, timeout=120, )) llm_resp.raise_for_status() ttft_done = False for raw_line in llm_resp.iter_lines(): if not raw_line: continue line = raw_line.decode("utf-8") if isinstance(raw_line, bytes) else str(raw_line) if not line.startswith("data:"): continue chunk = line[5:].strip() if chunk == "[DONE]": break try: obj = json.loads(chunk) delta = ((obj.get("choices") or [{}])[0].get("delta") or {}).get("content") or "" if not delta and isinstance(obj.get("message"), dict): delta = obj["message"].get("content") or "" if delta: if not ttft_done: llm_ttft_ms = int((time.monotonic() - t_llm) * 1000) ttft_done = True llm_text += delta yield sse({"type": "token", "delta": delta}) except Exception: continue llm_total_ms = int((time.monotonic() - t_llm) * 1000) yield sse({"type": "llm_done", "text": llm_text, "llm_ttft_ms": llm_ttft_ms, "llm_total_ms": llm_total_ms}) except Exception as e: yield sse({"type": "error", "stage": "llm", "message": str(e)}) return if not llm_text.strip(): yield sse({"type": "error", "stage": "llm", "message": "LLM returned empty response."}) return # 3. TTS try: t_tts = time.monotonic() audio_bytes, mime = await asyncio.to_thread( _preview_request_audio, llm_text, tts_voice, settings, "", tts_be ) tts_ms = int((time.monotonic() - t_tts) * 1000) total_ms = int((time.monotonic() - t0) * 1000) yield sse({"type": "audio", "b64": base64.b64encode(audio_bytes).decode(), "mime": mime}) yield sse({"type": "stats", "stt_ms": stt_ms, "llm_ttft_ms": llm_ttft_ms, "llm_total_ms": llm_total_ms, "tts_ms": tts_ms, "total_ms": total_ms}) except Exception as e: yield sse({"type": "error", "stage": "tts", "message": str(e)}) return yield sse({"type": "done"}) return StreamingResponse(generate(), media_type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}) # ── Static ──────────────────────────────────────────────────────────────────── app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static") if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=7890, log_level="info")