# TTS Voice Creator - Clone and Design

A browser-based app for managing Qwen3 TTS voice clones and Voice Design prompts. It runs in Docker, opens in any browser, and gives you one place to curate the voice library, crop/reference audio, normalize loudness, write reusable voice design prompts, and copy integration snippets for apps such as SillyTavern, Open WebUI, and Home Assistant.

---

## What it does

```
[ Drop file / YouTube URL / Record mic ]   or   [ Describe voice in text ]
                     ↓                                       ↓
      [ Interactive waveform trimmer ]          [ Qwen3-TTS VoiceDesign API ]
                     ↓                                       ↓
            [ 24 kHz · mono · 16-bit PCM WAV  +  reference.txt ]
                                   ↓
                     [ Named voice saved to disk ]
                                   ↓
                     [ TTS preview / Library management ]
```

Two workflows in one tool:

- **Voice cloning** — record or import a real voice, trim a clean segment, auto-transcribe, normalize loudness, and save it as a TTS reference voice.
- **Voice design** — describe a voice in plain language and let the Qwen3-TTS VoiceDesign model synthesize one from scratch. Designed voices can be exported to the clone library or used live as virtual voices through this app's `/v1` proxy.

---

## Requirements

| Requirement | Notes |
|---|---|
| Docker Engine + Docker Compose v2 | [docs.docker.com/get-docker](https://docs.docker.com/get-docker/) |
| A Docker network shared with your TTS/STT containers | `docker network create tts_net` |
| A Whisper-compatible transcription server | See [Recommended backends](#recommended-backends) |
| A TTS server | See [Recommended backends](#recommended-backends) |
| ARM64 (aarch64) or x86-64 | Image is based on `python:3.11-slim-bookworm` — works on both |
| Internet access during build | pip packages; WaveSurfer.js is loaded from CDN at runtime |

---

## Installation & Setup

### 1 — Clone the repository

```bash
git clone https://github.com/mARTin-B78/TTS-Voice-Creator.git tts-voice-creator-clone-and-design
cd tts-voice-creator-clone-and-design
```

### 2 — Create a Docker network (if you don't have one already)

All containers that need to talk to each other (TTS Voice Creator - Clone and Design, your TTS server, your Whisper server) should share a network:

```bash
docker network create tts_net
```

If you already have a shared network, note its name — you will use it in the next step.

### 3 — Configure volume paths

Edit `docker-compose.yml` (or `portainer-stack.yml` for Portainer) and set the host path where your voices are stored:

```yaml
volumes:
  - /your/voices/directory:/voices:rw
```

This directory is shared with the TTS server — voices saved here become immediately available after the TTS server is restarted.

Also update the network name if yours differs from the default:

```yaml
networks:
  tts_net:
    external: true
```

### 4 — Build the image

```bash
docker build -t tts-voice-creator:latest .
```

This step is required before the first run, and whenever you pull updates.

### 5a — Run with Docker Compose

```bash
docker compose up -d
```

### 5b — Deploy via Portainer

> Use **`portainer-stack.yml`**, not `docker-compose.yml`.  
> The Portainer file references the pre-built image and has no `build:` section, which avoids the "no such file: Dockerfile" error that occurs because Portainer stores stacks in its own internal directory.

1. Build the image on the host (step 4 above)
2. Portainer → **Stacks** → **Add stack**
3. Enter a stack name (e.g. `tts-voice-creator`)
4. Select **Web editor**
5. Paste the contents of [`portainer-stack.yml`](portainer-stack.yml)
6. Click **Deploy the stack**

### 6 — Open the UI

```
http://<host-ip>:7860
```

### 7 — Configure API endpoints

Click the **⚙ gear icon** (top right) and fill in:

| Field | Example | Notes |
|---|---|---|
| Whisper API URL | `http://whisper:8000` | Use container name if on the same network |
| Whisper API key | *(optional)* | Only needed for cloud APIs |
| TTS API URL | `http://tts-server:8000` | |
| TTS API key | *(optional)* | |
| TTS backend | `Qwen3-TTS / OpenAI` | Select to match your server |
| Voice Design URL | `http://host.docker.internal:8021` | Optional; required for prompt-based Voice Design. For Qwen-style dialogue/timbre reuse, point this at the 1.7B VoiceDesign backend. |
| Voice Design API key | *(leave empty)* | Falls back to TTS key if blank |
| Voice scan directory | `/voices` | Where the library tab looks for voices |
| Active voices directory | `/voices/active_voices` | Where newly saved and TTS-visible voices land |

Settings are persisted to a named Docker volume and survive container restarts.

---

## Recommended Backends

### Speech-to-Text (transcription)

The app sends audio to a Whisper-compatible REST endpoint (`POST /v1/audio/transcriptions`).

| Option | Type | Recommendation |
|---|---|---|
| **[faster-whisper-server](https://github.com/fedirz/faster-whisper-server)** | Local, free | Best local option. Runs `large-v3` efficiently on GPU or CPU. OpenAI-compatible API. |
| **[Whisper.cpp server](https://github.com/ggerganov/whisper.cpp)** | Local, free | Lightweight, works on any hardware with no Python. Good for CPU-only setups. |
| **[OpenAI Whisper API](https://platform.openai.com/docs/guides/speech-to-text)** | Cloud, paid | Highest accuracy, no local GPU needed. Set URL to `https://api.openai.com` and add your API key. |
| **[Groq Whisper API](https://console.groq.com/docs/speech-text)** | Cloud, free tier | Very fast inference, generous free tier. OpenAI-compatible endpoint. |

**Recommended for local use:** `faster-whisper-server` with `large-v3` on a GPU. Example Docker run:

```bash
docker run -d --gpus all --network tts_net --name whisper \
  -p 8000:8000 \
  fedirz/faster-whisper-server:latest-cuda
```

Then set the Whisper URL in settings to `http://whisper:8000`.

---

### Text-to-Speech

The app supports any server that exposes a compatible REST endpoint.

| Option | Type | Recommendation |
|---|---|---|
| **[Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS)** | Local, free | State-of-the-art multilingual TTS with voice cloning and voice design. Primary recommendation. |
| **[faster-qwen3-tts](https://github.com/mARTin-B78/TTS-Voice-Creator)** | Local wrapper | Docker wrapper for Qwen3-TTS with an OpenAI-compatible API and automatic voice loading from the configured active voices directory. |
| **[LocalAI](https://github.com/mudler/LocalAI)** | Local, free | Supports many TTS backends (Piper, Bark, XTTS, etc.) behind a unified API. Select "LocalAI" in settings. |
| **[Pocket-TTS](https://github.com/ai-joe-git/pocket-tts-server)** | Local, free | Lightweight XTTS-based server. Select "Pocket-TTS" in settings. |
| **[OpenAI TTS API](https://platform.openai.com/docs/guides/text-to-speech)** | Cloud, paid | High quality, no local hardware needed. Set URL to `https://api.openai.com` and add your API key. |
| **NVIDIA Magpie + Parakeet** | Local, free | OpenAI-compatible NVIDIA speech stack. Use Magpie Multilingual on `http://host.docker.internal:8091` for fixed-speaker TTS and Parakeet on `http://host.docker.internal:8092` for STT, or route both through `8090`. NVIDIA Magpie Zeroshot/Flow clone NIMs can be configured separately, usually on `http://host.docker.internal:8093`. |
| **[ElevenLabs](https://elevenlabs.io/docs/api-reference)** | Cloud, freemium | Best-in-class voice quality and cloning. OpenAI-compatible endpoint available. |

**Recommended for local use:** Qwen3-TTS is the best open-source multilingual TTS model as of 2025. It produces very natural speech, supports 16+ languages, and has built-in zero-shot voice cloning and voice design.

---

### Voice Cloning vs Voice Design

This tool supports two fundamentally different ways to create a new TTS voice:

#### Option A — Voice Cloning

You provide a real recording of a voice. The TTS model uses it as a reference to reproduce that speaker's characteristics.

**What makes a good reference recording:**

- **Duration:** 5–20 seconds (10–15 s is ideal)
- **Content:** Natural, expressive speech — not just counting or reciting the alphabet
- **Audio quality:** No background music, reverb, noise, or multiple speakers
- **Emotion:** Should match the emotion you want the TTS to reproduce (calm for a narrator, energetic for an announcer, etc.)
- **Format:** The app converts anything to 24 kHz mono 16-bit PCM WAV automatically
- **Transcript:** The reference text (what was spoken) is critical — Qwen3-TTS uses it for alignment

**Good sources for reference audio:**

- Record yourself or a voice actor directly in the app (mic tab)
- Extract a clean segment from a podcast, audiobook, or interview using the YouTube/URL downloader
- Any audio or video file: WAV, MP3, OGG, FLAC, M4A, MP4, MKV, WEBM

#### Option B — Voice Design

Describe the voice you want in plain language. The Qwen3-TTS VoiceDesign model synthesises a new voice from your description — no recording required.

**Requires:** A server running the `Qwen3-TTS-12Hz-1.7B-VoiceDesign` model or a compatible alias (typically a separate model from the main TTS model). The container environment variable `VOICE_DESIGN_MODEL` controls the model name sent to the VoiceDesign backend.

**How to write good voice descriptions:**

Describe: age, gender, pitch, pace, accent, emotion, and speaking style. Be specific.

```
# Good descriptions:
"A middle-aged British woman with a calm, authoritative tone and clear diction.
 Slightly formal, measured speech rate."

"Young male voice, mid-20s, enthusiastic and upbeat. American accent, slightly
 fast pace, warm and friendly."

"Elderly male narrator with a deep, resonant voice. Slow and deliberate,
 with gravitas. No discernible accent."

# Bad (too vague):
"A nice voice"
"Female voice"
```

---

## Application Tabs

### 1 Voice Clone

The first tab is the main voice library. It contains the old Source & Trim and
Name & Save workflow inside an `Add new voice` panel, so you no longer need to
move back and forth between separate tabs.

| Feature | Details |
|---|---|
| Add new voice | Opens drag and drop, YouTube/audio URL, microphone, waveform trim, transcript, and save controls |
| Library editing | Rename by editing the name directly; the app renames the WAV and sidecar files |
| Language | Formerly shown as Flag; language and region are stored in metadata |
| Type / Length / dB | Shows audio type, duration, and stored loudness values |
| Pencil optimizer | Opens waveform crop, transcript, dB controls, save crop, save volume, and undo crop |
| Active toggle | Moves complete voice packages between `active_voices` and `hidden_voices` |
| Copy active voices | Copies active voice IDs as a comma-separated list |

Cropping is reversible: the app keeps an original copy before replacing the
active voice file.

### 2 Voice Design

Voice Design is for prompt-based voices. Samples are listed in a collapsible
table. `Preview` generates audio from a sample, and `Use` fills the description,
sample text, and language fields.

| Feature | Details |
|---|---|
| Voice Design samples | Example prompts for acoustic attribute control, age control, gradual control, human-likeness, background information, and timbre reuse |
| Reference transcript | Shared transcript field used by samples and presets |
| Generate voice | Calls the VoiceDesign backend and plays the generated audio |
| Export to Voice Clone Library | Saves the generated WAV, transcript, name, language, and gender into the clone library |
| Prompt library | Stores reusable Voice Design presets |

### 3 Routing

The Routing tab lets this app act as an OpenAI-compatible TTS router. External
apps can send a simple voice name such as `default` to this app's `/v1` proxy,
and routing rules can map it to a real Qwen3-TTS voice by app and detected
language.

Example:

| App | Input voice | Language | Output voice |
|---|---|---|---|
| Open WebUI | `default` | `EN` | `EN_F_Anna` |
| Open WebUI | `default` | `DE` | `DE_M_Max` |

### 4 Use In Apps

This tab shows ready-to-copy examples for SillyTavern, Open WebUI, Home
Assistant, and OpenAI-compatible clients.

Exported clone voices should usually point external apps directly at the Qwen3
TTS server. Virtual VoiceDesign voices should point external apps at this app's
`/v1` proxy.

---

## Command Line Usage

Everything needed for headless operation is available through `ttsvc_cli.py`.
It talks to the running app over HTTP, so it works locally or against a remote
creator instance.

```bash
./ttsvc_cli.py --help
./ttsvc_cli.py settings show
./ttsvc_cli.py settings set --tts-url http://host.docker.internal:8020 --tts-backend openai
./ttsvc_cli.py voices list --all
./ttsvc_cli.py voices enable EN_F_Anna
./ttsvc_cli.py routes openwebui-defaults --en EN_F_Anna --de DE_M_Max
./ttsvc_cli.py routes list
./ttsvc_cli.py tts voices
./ttsvc_cli.py tts speak --voice default --text "Hallo, das ist ein Test." --out test.wav
./ttsvc_cli.py create-voice --file sample.wav --voice-id EN_F_NewVoice --transcript "Exact words spoken."
```

Use `--base-url http://HOST:7860` when the app is not running on localhost.
Each command and subcommand has its own `--help` output.

---

## Guided Installer

For non-command-line users, start the graphical installer:

```bash
python3 guided_installer.py
```

Or double-click one of these launchers:

- Linux: `start-guided-installer.sh`
- macOS: `start-guided-installer.command`
- Windows: `start-guided-installer.bat`

The installer checks Docker, Docker Compose, optional Portainer, this app, and
the Qwen3-TTS clone/design/custom containers. It explains what each component is
for, lets the user choose ports and model locations, generates a Docker Compose
file, and can run the selected Docker actions with visible logs.

Additional publishing assets:

- [Forum playbook](docs/forum-playbook.md)
- [Tutorial video script](docs/tutorial-video-script.md)

---

## API compatibility

### TTS endpoints

| Backend | Setting | Endpoint | Request body |
|---|---|---|---|
| Qwen3-TTS / OpenAI | `openai` | `POST /v1/audio/speech` | `{"model":"tts-1","input":"...","voice":"EN_F_Anna","response_format":"wav"}` |
| LocalAI | `localai` | `POST /tts` | `{"input":"...","model":"EN_F_Anna","response_format":"wav"}` |
| Pocket-TTS | `pocket` | `POST /v1/audio/speech` | `{"input":"...","voice":"EN_F_Anna","response_format":"wav"}` |
| NVIDIA Magpie Multilingual | `nvidia_magpie` | `POST /v1/audio/speech` | `{"model":"tts-1","input":"...","voice":"sofia","response_format":"wav"}` |
| NVIDIA Magpie Zeroshot | `nvidia_zeroshot` | `POST /v1/audio/synthesize` | Multipart form with `language`, `text`, and selected library WAV as `audio_prompt` |
| NVIDIA Magpie Flow | `nvidia_flow` | `POST /v1/audio/synthesize` | Multipart form with `language`, `text`, selected library WAV as `audio_prompt`, and saved reference transcript as `audio_prompt_transcript` |

### Voice Design endpoint

```
POST /v1/audio/speech
{
  "model":    "Qwen3-TTS-12Hz-1.7B-VoiceDesign",
  "input":    "sample text to speak",
  "instruct": "describe the voice here",
  "language": "Auto",
  "response_format": "wav"
}
```

For Qwen-style multi-speaker VoiceDesign, send the complete dialogue in `input`
and the speaker timbre map in `instruct`, for example:

```json
{
  "model": "Qwen3-TTS-12Hz-1.7B-VoiceDesign",
  "input": "Lucas:Hi there.\nMia:Hello back.",
  "instruct": "\"Lucas\": \"Male, 17 years old, tenor range\"\n\"Mia\": \"Female, 16 years old, mezzo-soprano range\"",
  "language": "English",
  "response_format": "wav"
}
```

### Whisper endpoint

```
POST /v1/audio/transcriptions
Content-Type: multipart/form-data

file=<audio.wav>
model=large-v3
response_format=text
```

For NVIDIA Parakeet, set the Whisper/STT URL to `http://host.docker.internal:8092` for the direct ASR container, or `http://host.docker.internal:8090` for the speech router. For NVIDIA voice cloning, set the NVIDIA Zeroshot/Flow NIM URL to the deployed Speech NIM endpoint, commonly `http://host.docker.internal:8093`; the app sends saved library WAVs as `audio_prompt`, and Flow also sends the saved `.reference.txt` transcript.

All endpoints accept an optional `Authorization: Bearer <key>` header, configurable in Settings.

---

## Timbre Control Support

There are two different modes:

- `faster-qwen3-tts` uses saved voice/reference audio IDs. It supports timbre reuse through cloned or exported voices, but it does not normally use prompt fields such as gender, age, pitch, texture, or emotion.
- `faster-qwen3-tts-voicedesign` is the prompt-based path. It can use acoustic descriptions and timbre-style prompts, including multi-speaker/timbre reuse style descriptions, but those voices are generated live from the prompt unless you export the result.

So: timbre descriptions belong in the Voice Design container. Stable reusable
voice IDs belong in the normal TTS container after export.

---

## Use In Apps

### Exported Clone Voices

For SillyTavern, Open WebUI, Home Assistant, or any OpenAI-compatible client:

```text
Base URL: http://YOUR_TTS_HOST:8020/v1
Model:    qwen
Voice:    one of the active voice IDs
```

After adding or changing voices, restart the TTS container so it rereads the
active voice folder.

### Virtual VoiceDesign Voices Without Export

This app exposes an OpenAI-compatible proxy:

```text
Base URL: http://YOUR_APP_HOST:7860/v1
Model:    qwen3-tts-voice-creator
Voice:    vd_<PresetName>
```

Virtual voices call the Voice Design backend live for each generation. This is
useful when you want to keep editing prompts without exporting WAV files yet. It
is slower than using exported clone voices and requires the VoiceDesign backend
URL to be configured.

Example request:

```bash
curl http://localhost:7860/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-tts-voice-creator",
    "voice": "vd_WarmNarrator",
    "input": "This line is generated through a saved Voice Design preset."
  }' \
  --output speech.wav
```

---

## Voice file format

Qwen3-TTS (and most modern neural TTS systems) expect reference audio in a specific format:

| Property | Value |
|---|---|
| Format | WAV (PCM, uncompressed) |
| Sample rate | **24 000 Hz** |
| Channels | **Mono** |
| Bit depth | **16-bit** |
| Duration | **5–20 seconds** |
| Companion file | `<voice_id>.reference.txt` — verbatim transcript of the spoken audio |

The app converts all source audio to this specification automatically using `pydub` + `ffmpeg`.

### Voice metadata sidecar files

Each saved voice can have optional sidecar files stored alongside the WAV:

| File | Contents |
|---|---|
| `<id>.reference.txt` | Transcript of the reference audio (required for voice cloning) |
| `<id>.meta.json` | Note, star rating, flag, gender, enabled state |
| `<id>.jpg` / `.png` / `.webp` | Profile picture shown in the library |

---

## Naming convention

Voice IDs have no forced format. The suggested convention is:

```
{LANG}_{GENDER}_{Name}.wav
```

Examples: `EN_F_Anna.wav`, `DE_M_Christoph.wav`, `ZH_F_Mei.wav`

Language codes: `EN DE ZH FR ES JA KO IT PT RU AR PL NL SV TR HI`  
Gender codes: `F` (female) · `M` (male) · `N` (neutral / non-binary)

Custom IDs like `narrator_calm` or `my-voice-v2` are equally valid — the naming helper in the UI is optional.

---

## Volume mounts

| Host path | Container path | Purpose |
|---|---|---|
| `${VOICE_HOST_DIR:-./voices}` | `/voices` | Shared voice collection (read-write) |
| Named volume `tts_voice_creator_config` | `/home/app/.config/tts-voice-creator` | Persisted settings and Voice Design presets |

The TTS server should mount and scan the `active_voices` subfolder. The `hidden_voices` subfolder is for voices kept in the library but removed from Qwen3-TTS model discovery.

---

## Reloading voices into your TTS server

Most local TTS servers load voices on startup. After saving a new voice, restart the TTS container:

```bash
docker restart <your-tts-container-name>
```

The default public Compose file does not mount the Docker socket. Restart the
TTS container from Docker, Portainer, or your host scripts.

---

## Public Repository Safety

The repository ignores private runtime data:

- `.env` files and local settings.
- `.claude/`, editor workspaces, logs, and caches.
- Voice folders and audio/video files.
- Per-voice transcript and metadata sidecars.

Do not commit real API keys, private voice samples, speaker folders, or
host-specific paths.

---

## Architecture

```
Browser (any device on your local network)
  └── port 7860 ──► FastAPI + uvicorn  (server.py)
                         ├── pydub + ffmpeg        audio conversion & trimming
                         ├── yt-dlp                YouTube / URL audio extraction
                         ├── Whisper API  ──────►  transcription
                         ├── TTS API      ──────►  preview & voice list
                         └── VoiceDesign API ───►  AI voice generation

Frontend (single HTML file, no build step)
  ├── WaveSurfer.js 7   interactive waveform + region trimmer
  └── MediaRecorder API browser microphone recording
```

---

## Troubleshooting

**WaveSurfer / UI doesn't load** — the container needs internet access to fetch WaveSurfer.js from `unpkg.com`. If your network is air-gapped, download the two JS files and serve them from `static/`.

**"Cannot connect to Whisper/TTS API"** — check that:
1. The API server container is running
2. Both containers are on the same Docker network
3. You're using the container name (not `localhost`) as the URL inside Docker — e.g. `http://whisper:8000`, not `http://localhost:8000`

**Microphone not working** — browsers block `getUserMedia` on non-`localhost` origins without HTTPS. Set up a reverse proxy with a TLS certificate (e.g. Nginx + Let's Encrypt or Caddy) if accessing from another device.

**Generated voice sounds wrong** — for cloning, check the reference audio quality: no background noise, single speaker, natural speech, and a correct transcript. For voice design, make the description more specific — add age, pitch, pace, and accent details.

**Voice not appearing in TTS server after save** — restart the TTS container. Most servers only scan the voices directory at startup.

**Transcription produces garbled text** — try a larger Whisper model (`large-v3` recommended), or manually type the transcript before saving.