# TTS Voice Creator — Clone & Design

A browser-based voice management studio running in Docker. Clone real voices, design new ones from text descriptions, manage your voice library, and connect any app or AI agent via REST or MCP.

---

## User Manual

For a step-by-step app tutorial, see [docs/USER_MANUAL.md](docs/USER_MANUAL.md).

---

## What it does

```
[ Drop file / YouTube URL / Record mic ]   or   [ Describe voice in text ]
                     ↓                                       ↓
      [ Interactive waveform trimmer ]          [ Qwen3-TTS VoiceDesign API ]
                     ↓                                       ↓
            [ 24 kHz · mono · 16-bit PCM WAV  +  reference.txt ]
                                   ↓
                     [ Named voice saved to disk ]
                         ↓                  ↓
              [ TTS preview / Library ]   [ /speak · /mcp · /v1 proxy ]
```

**Core workflows:**

- **Voice cloning** — record or import audio, trim a clean segment, auto-transcribe, normalize loudness, and save as a TTS reference voice.
- **Voice design** — describe a voice in plain language; Qwen3-TTS VoiceDesign synthesises one from scratch. Export to the clone library or use live as a virtual voice.
- **Character persona** — attach a personality description to any voice; a local LLM rewrites incoming text in that character's style before synthesis.
- **Speak API** — `POST /speak` for any script, app, or agent, with per-client voice bindings and optional persona rewrite.
- **Native MCP server** — built-in at `POST /mcp`; exposes `speak`, `transcribe`, `list_captures`, and `list_profiles` as MCP tools for Claude Code, Cursor, or any MCP-aware agent.
- **Global hotkey daemon** — run `hotkey_daemon.py` on the host for push-to-talk dictation that types transcribed text into any focused window.

---

## Requirements

| Requirement | Notes |
|---|---|
| Docker Engine + Docker Compose v2 | [docs.docker.com/get-docker](https://docs.docker.com/get-docker/) |
| A Docker network shared with your TTS/STT containers | `docker network create dgx_net` |
| A Whisper-compatible transcription server | See [Recommended backends](#recommended-backends) |
| A TTS server | See [Recommended backends](#recommended-backends) |
| ARM64 (aarch64) or x86-64 | Image based on `python:3.11-slim-bookworm` |
| Internet access during build | pip packages; WaveSurfer.js loaded from CDN at runtime |

---

## Installation & Setup

### 1 — Clone the repository

```bash
git clone https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2.git
cd tts-voice-creator-clone-and-design-2
```

### 2 — Create or reuse a Docker network

```bash
docker network create dgx_net   # skip if you already have one
```

### 3 — Configure `docker-compose.yml`

Set the host path for voices and the network name:

```yaml
volumes:
  - /your/voices/directory:/voices:rw
networks:
  dgx_net:
    external: true
```

### 4 — Build and start

```bash
docker compose up -d --build
```

### 5 — Open the UI

```
http://<host-ip>:7890
```

### 6 — Configure API endpoints

**Settings → Connections** — fill in at minimum:

| Field | Example |
|---|---|
| Voice Clone / Base URL | `http://host.docker.internal:8020` |
| Voice Design URL | `http://host.docker.internal:8021` |
| Whisper URL | `http://host.docker.internal:8010` |
| LLM base URL | `http://localhost:11434/v1` *(for persona rewrite)* |

Settings persist to a named Docker volume across restarts.

---

## Recommended Backends

### Speech-to-Text

| Option | Notes |
|---|---|
| **[faster-whisper-server](https://github.com/fedirz/faster-whisper-server)** | Best local option. GPU-accelerated, OpenAI-compatible. |
| **[Whisper.cpp server](https://github.com/ggerganov/whisper.cpp)** | CPU-only, no Python. |
| **[Groq Whisper API](https://console.groq.com/docs/speech-text)** | Fast cloud, generous free tier. |
| **NVIDIA Parakeet** | `http://host.docker.internal:8092` — 200× realtime on GPU. |

### Text-to-Speech

| Option | Backend ID | Notes |
|---|---|---|
| **[faster-qwen3-tts](https://github.com/mARTin-B78/TTS-Voice-Creator)** | `voice_clone` | Recommended. Docker wrapper for Qwen3-TTS with auto voice loading. |
| Qwen3-TTS Voice Design | `voice_design` | Prompt-based voice synthesis. Needs `Qwen3-TTS-12Hz-1.7B-VoiceDesign`. |
| CustomVoice | `customvoice` | Style-controlled synthesis with pre-configured speakers. |
| Streaming | `streaming` | Progressive low-latency WAV playback. |
| **[Kokoro FastAPI](https://github.com/remsky/Kokoro-FastAPI)** | `kokoro` | OpenAI-compatible. 82M model, ~300 MB RAM, CPU-friendly. |
| **VibeVoice** | `vibevoice` | Simple `POST /tts` service. Default: `http://192.168.178.8:8027`. |
| **NVIDIA Magpie TTS** | `nvidia_magpie` | Fixed-speaker GPU TTS. `http://host.docker.internal:8091`. |
| **NVIDIA Magpie Zeroshot** | `nvidia_zeroshot` | WAV-prompted zero-shot clone NIM. |
| **NVIDIA Magpie Flow** | `nvidia_flow` | WAV + transcript high-fidelity clone NIM. |
| **[ElevenLabs](https://elevenlabs.io/)** | — | Cloud; browse the shared library in the **Get Voices Online** tab. |

---

## Application Sections

### Voices — Clone library

Manage WAV reference voices. Crop and normalize audio, auto-transcribe, set metadata (language, gender, rating, note), and attach a **character persona** for LLM-based text rewriting.

| Feature | Details |
|---|---|
| Add voice | Drag-and-drop, URL/YouTube import, microphone recording |
| Waveform editor | Crop to clean segment, normalize loudness to target dBFS |
| Auto-transcribe | Sends audio to the configured STT backend |
| Character persona | Plain-language personality prompt; LLM rewrites text before synthesis |
| Active / hidden | Toggle moves voice between `active_voices` and `hidden_voices` |
| Sync / Calculate dB / Benchmark | Bulk tools for library health |

### Clone a Voice

Step-by-step wizard: import audio → trim → transcribe → save to library.

### Design a Voice

Describe a voice in natural language; Qwen3-TTS VoiceDesign generates it from scratch. Export to the clone library or use as a live `vd_...` virtual voice.

Includes curated sample prompts: acoustic attribute control, age control, timbre reuse, multi-speaker dialogue.

### Studio — Get Voices Online

Browse public voice clip sources, scrape and preview direct audio files, import from URLs, and browse the ElevenLabs shared voice library.

### Try It Out — TTS Playground

Generate speech from any backend and voice. Features:

- Backend and voice selector with live availability check
- Style instruction field (emotion, pacing, tone)
- **Apply character persona** toggle — LLM rewrites input text through the voice's personality before synthesis
- Chunked generation for long texts (split at sentence boundaries)
- Audio effects: reverb, compressor, chorus, pitch shift
- Generation history (last 20 clips, in-memory)
- Playlist with drag-to-reorder and merged WAV export

Also includes an **STT → TTS workspace**: record or upload speech → transcribe → LLM-refine → re-synthesize with any voice.

### Performance — Benchmark

Run latency and quality benchmarks across backends. Batch mode tests every active voice against a configurable sentence set. Results (RTSL, duration, per-voice) are shown in a sortable table and saved to voice metadata.

### App Routing

Route incoming OpenAI-compatible TTS requests by app name, voice, and detected language. Rules map generic voice names (`default`, `assistant`) to specific library voices. Before/after sounds and per-route backend selection are supported.

### Connect Your Apps

Ready-to-copy integration snippets for SillyTavern, Open WebUI, Home Assistant, generic curl, virtual VoiceDesign voices, streaming TTS, and the native MCP server.

### Settings

Organized into sub-pages: **Connections** (all backend URLs including VibeVoice, NVIDIA stack), **STT**, **Generation** (extra TTS params per backend), **LLM** (URL for persona rewrite and text refinement), **Voice** (scan directory, normalization targets), **Output**, **Captures**, **General**, **About** (MCP connection commands, `/speak` examples, hotkey daemon setup).

---

## Character Persona

Each voice can have a personality description stored in its `.meta.json` sidecar:

```json
{ "persona": "Gruff Scottish engineer, dry wit, technical vocabulary, measured pace." }
```

When persona rewrite is active, a local LLM at the configured LLM URL rephrases the input text in that character's voice before TTS synthesis. This runs through the same OpenAI-compatible endpoint used for text refinement.

**Activate persona rewrite:**
- In the playground: check **Apply character persona** before generating.
- Via API: pass `"apply_persona": true` in the request body to `/api/tts-preview` or `/speak`.
- Via MCP: pass `"apply_persona": true` to the `speak` tool.

Edit a voice's persona in the voice inspector (**Voices** tab → select voice → **Character persona** section).

---

## Speak API

`POST /speak` — generate audio from any script or app without needing routing rules.

```bash
# Bind a client ID to a voice (persisted)
curl -X PUT http://localhost:7890/speak/bindings/my-script \
  -H "Content-Type: application/json" \
  -d '{"voice": "EN_F_Anna"}'

# Generate speech — voice from binding, optional persona rewrite
curl -X POST http://localhost:7890/speak \
  -H "Content-Type: application/json" \
  -H "X-Voice-Creator-Client-Id: my-script" \
  -d '{"text": "Hello world", "apply_persona": true}' \
  --output speech.wav

# Or specify voice directly
curl -X POST http://localhost:7890/speak \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello world", "voice": "EN_F_Anna", "backend": "voice_clone"}' \
  --output speech.wav
```

**Voice resolution order:** explicit `voice` param → `X-Voice-Creator-Client-Id` binding → `captures_default_voice` in Settings.

**Binding endpoints:**

| Method | Path | Description |
|---|---|---|
| `GET` | `/speak/bindings` | List all per-client voice bindings |
| `PUT` | `/speak/bindings/{client_id}` | Set or update a binding |
| `DELETE` | `/speak/bindings/{client_id}` | Remove a binding |

---

## Native MCP Server

The app ships a built-in MCP server at `/mcp` (JSON-RPC 2.0, Streamable HTTP transport). No external Python script or extra packages required.

**Tools:**

| Tool | Arguments | Returns |
|---|---|---|
| `speak` | `text`, `voice?`, `apply_persona?` | Base64 WAV audio + confirmation text |
| `transcribe` | `audio_base64` | Transcribed text |
| `list_captures` | — | 20 most recent output WAV files |
| `list_profiles` | — | All voice profiles with id, lang, persona, enabled |

**Connect Claude Code:**

```bash
claude mcp add voice-creator \
  --transport http \
  --url http://localhost:7890/mcp \
  --header "X-Voice-Creator-Client-Id: claude-code"
```

**Any HTTP MCP client (Cursor, Windsurf, VS Code, etc.):**

```json
{
  "mcpServers": {
    "voice-creator": {
      "url": "http://localhost:7890/mcp",
      "headers": { "X-Voice-Creator-Client-Id": "my-agent" }
    }
  }
}
```

The `X-Voice-Creator-Client-Id` header binds this client to a voice via `PUT /speak/bindings/{id}`. Per-client bindings persist in Settings.

---

## Global Hotkey Daemon

`hotkey_daemon.py` is a standalone host-side script (runs outside Docker) for push-to-talk dictation. Hold **Ctrl+Shift+Space** to record, release to transcribe and type the result into any focused window.

```bash
# Install dependencies (host Python, not Docker)
pip install pynput sounddevice soundfile pyperclip requests

# Linux: install xdotool for direct key injection (falls back to clipboard paste)
sudo apt install xdotool

# Run
python hotkey_daemon.py --server http://localhost:7890
```

| Option | Default | Notes |
|---|---|---|
| `--server` | `http://localhost:7890` | Voice Creator URL |
| `--backend` | `configured` | STT backend to use |

**Platform notes:**
- **Linux X11:** uses `xdotool type` for direct injection; clipboard fallback on Wayland.
- **macOS:** clipboard paste via Cmd+V; grant Accessibility + Microphone in System Preferences.
- **Windows:** clipboard paste via Ctrl+V.

---

## OpenAI-Compatible Proxy

This app exposes a `/v1` proxy that any OpenAI-compatible client can use:

```
Base URL: http://YOUR_HOST:7890/v1
API key:  dummy   (any non-empty value)
Model:    tts-1
Voice:    EN_F_Anna   (or any active voice ID, or vd_<PresetName> for virtual voices)
```

Routing rules (App Routing tab) run on every request through this proxy.

---

## API Reference

### TTS endpoints

| Endpoint | Description |
|---|---|
| `POST /v1/audio/speech` | OpenAI-compatible TTS with routing |
| `POST /speak` | Direct TTS with per-client voice binding |
| `POST /api/tts-preview` | Preview any backend/voice combination |
| `POST /api/tts-preview` with `apply_persona:true` | Preview with LLM persona rewrite |
| `GET /v1/audio/voices` | List active voice IDs |
| `GET /v1/models` | List TTS models |

### STT endpoints

| Endpoint | Description |
|---|---|
| `POST /v1/audio/transcriptions` | OpenAI-compatible transcription |
| `POST /api/transcribe-bytes` | Multipart upload → transcription (used by hotkey daemon) |

### MCP endpoint

| Endpoint | Description |
|---|---|
| `POST /mcp` | JSON-RPC 2.0 handler (`initialize`, `tools/list`, `tools/call`) |
| `GET /mcp` | SSE keep-alive stream |

### Voice library

| Endpoint | Description |
|---|---|
| `GET /api/voices` | List all voices with metadata |
| `POST /api/save` | Save a voice from a temp file |
| `POST /api/update-voice-meta` | Update note, rating, flag, gender, persona |
| `POST /api/voices/calculate-db` | Normalize dBFS for all active voices |
| `POST /api/voices/benchmark` | Run latency/quality benchmarks |
| `GET /api/tts-backends` | List all backends with availability and capabilities |

### /speak bindings

| Endpoint | Description |
|---|---|
| `GET /speak/bindings` | List all per-client voice bindings |
| `PUT /speak/bindings/{id}` | Set a per-client voice binding |
| `DELETE /speak/bindings/{id}` | Remove a binding |

---

## Voice File Format

| Property | Value |
|---|---|
| Format | WAV (PCM, uncompressed) |
| Sample rate | 24 000 Hz |
| Channels | Mono |
| Bit depth | 16-bit |
| Duration | 5–20 seconds |
| Companion file | `<id>.reference.txt` — verbatim transcript |

The app converts all source audio automatically using `pydub` + `ffmpeg`.

### Sidecar files

| File | Contents |
|---|---|
| `<id>.reference.txt` | Reference transcript (required for cloning) |
| `<id>.meta.json` | Note, rating, flag, gender, enabled, **persona**, loudness, benchmark |
| `<id>.jpg` / `.png` / `.webp` | Profile picture shown in the library |

---

## Naming Convention

```
{LANG}_{GENDER}_{Name}.wav
```

Examples: `EN_F_Anna.wav`, `DE_M_Christoph.wav`, `ZH_F_Mei.wav`

Language codes: `EN DE ZH FR ES JA KO IT PT RU AR PL NL SV TR HI`  
Gender codes: `F` (female) · `M` (male) · `N` (neutral)

Custom IDs like `narrator_calm` are equally valid.

---

## Architecture

```
Browser (any device on your local network)
  └── port 7890 ──► FastAPI + uvicorn  (server.py)
                         ├── /v1          OpenAI-compatible TTS/STT proxy + routing
                         ├── /speak       Direct TTS with per-client bindings
                         ├── /mcp         Native MCP server (JSON-RPC 2.0)
                         ├── /api/...     Voice library, settings, benchmarks
                         ├── pydub+ffmpeg audio conversion & trimming
                         ├── yt-dlp       YouTube / URL audio extraction
                         ├── Whisper API  transcription (multiple backends)
                         ├── TTS backends voice_clone · voice_design · customvoice
                         │               streaming · kokoro · vibevoice
                         │               nvidia_magpie · nvidia_zeroshot · nvidia_flow
                         └── LLM API      persona rewrite, text refinement

Host machine (optional)
  └── hotkey_daemon.py   global Ctrl+Shift+Space → record → transcribe → type

Frontend (static HTML + Vanilla JS, no build step)
  ├── WaveSurfer.js 7    interactive waveform + region trimmer
  ├── MediaRecorder API  browser microphone recording
  └── MDI v7.4.47        icons
```

---

## CLI

`ttsvc_cli.py` talks to the running app over HTTP:

```bash
./ttsvc_cli.py settings show
./ttsvc_cli.py settings set --tts-url http://host.docker.internal:8020
./ttsvc_cli.py voices list --all
./ttsvc_cli.py voices enable EN_F_Anna
./ttsvc_cli.py routes openwebui-defaults --en EN_F_Anna --de DE_M_Max
./ttsvc_cli.py tts speak --voice EN_F_Anna --text "Hello world" --out test.wav
./ttsvc_cli.py create-voice --file sample.wav --voice-id EN_F_Anna --transcript "Exact words."
```

Use `--base-url http://HOST:7890` for remote instances.

---

## Guided Installer

```bash
python3 guided_installer.py
# or: start-guided-installer.sh  /  .command  /  .bat
```

Checks Docker, configures containers, generates Compose files, and runs deployment with visible logs.

---

## Volume Mounts

| Host path | Container path | Purpose |
|---|---|---|
| `${VOICE_HOST_DIR:-./voices}` | `/voices` | Voice collection (read-write) |
| Named volume `tts-voice-creator-clone-and-design-2` | `/home/app/.config/tts-voice-creator` | Settings and presets |
| `./static` | `/app/static` | Frontend (live-reload without rebuild) |
| `./server.py` | `/app/server.py` | Backend (live-reload without rebuild) |

---

## Troubleshooting

**"Cannot connect to TTS/STT API"** — ensure both containers share the same Docker network; use container hostnames, not `localhost`.

**Microphone not working** — browsers block `getUserMedia` on non-`localhost` origins without HTTPS. Use a reverse proxy with TLS for remote access.

**Persona rewrite not working** — check the LLM URL in **Settings → LLM server**. The endpoint must be OpenAI-compatible (e.g. Ollama at `http://localhost:11434/v1`).

**MCP tools not appearing** — run `claude mcp list` and confirm `voice-creator` is shown. Restart Claude Code after adding.

**Hotkey daemon not typing** — on Linux, install `xdotool` for X11; on Wayland it falls back to clipboard paste (Ctrl+V). On macOS, grant Accessibility permission.

**Voice not appearing after save** — restart the TTS container (`docker restart <name>`). Most engines only scan the voice directory on startup.

**Transcription garbled** — use `large-v3` Whisper model; manually correct the transcript before saving.