tts-voice-creator-clone-and-design-2/docs/USER_MANUAL.md

# Voice Creator User Manual

This manual explains how to use **TTS Voice Creator - Clone, Design, Deploy** from the browser UI. It is written for day-to-day use: setting up engines, building a voice library, cloning and designing voices, rehearsing scripts, running a live voice agent, and connecting other apps.

## Table Of Contents

1. Quick start
2. How the app is organized
3. Engines and backend setup
4. Voice library
5. Clone a Voice
6. Design a Voice
7. Get Voices Online
8. Try It Out
9. Script Rehearser
10. Conversation Playground
11. Benchmark
12. App Routing
13. Connect Apps and APIs
14. Settings
15. Troubleshooting
16. Recommended workflows

## 1. Quick Start

1. Start the Docker stack.
2. Open the app in a browser, usually:

   ```text
   http://localhost:7890
   ```

   or, from another machine:

   ```text
   http://<server-ip>:7890
   ```

3. Go to **Setup -> Engines** and confirm your STT, TTS, and LLM containers are detected.
4. Go to **Setup -> Engines -> Text to Speech** and click **Use as TTS** or the backend-specific button for the TTS engine you want.
5. Go to **Speech to Text** or **Settings -> Connections** and set your STT endpoint.
6. Go to **Try It Out**, fetch voices, select a voice, type text, and synthesize.
7. Build your voice library with **Clone a Voice**, **Design a Voice**, or **Get Voices Online**.
8. Use **Script Rehearser** for scripts or **Conversation** for a live voice agent.

## 2. How The App Is Organized

The left sidebar is the main map of the app.

### Voices

- **My Voices**: Browse and manage all local voices.
- **All voices**: Every visible voice.
- **Cloned**: Voices created from reference audio.
- **Designed**: Voices created from a voice-design prompt.
- **Favorites**: Voices marked as favorites.
- **Hidden**: Voices hidden from normal pickers.
- **Library tools**: Bulk library maintenance.

### Voice Creation

- **Clone a Voice**: Create a voice from an audio file, recording, URL, or extracted clip.
- **Design a Voice**: Create a new voice from a text description.
- **Get Voices Online**: Search and import online voice clips.
- **Try It Out**: Test TTS, STT, effects, voice style, and persona rewriting.

### Script And Agent Workflows

- **Script Rehearser**: Import scripts, cast characters, assign voices, and rehearse scenes.
- **Conversation**: Talk to a live AI voice agent using STT -> LLM -> TTS.
- **Benchmark**: Measure speed and quality across voices and engines.

### Setup And Integration

- **Engines**: See local Docker engines and connect them to the app.
- **App Routing**: Route incoming TTS requests from other apps to voices/backends.
- **Connect Apps**: Copy API examples for other tools.
- **Settings**: Configure URLs, storage, keys, logs, backup, and defaults.

## 3. Engines And Backend Setup

The app is a control surface. Most speech generation and recognition happens in separate local or cloud engines.

### Engine Types

| Type | Used For | Examples |
| --- | --- | --- |
| Text to Speech | Generate spoken audio | Qwen3-TTS, Fish-Speech, XTTS, VibeVoice, NVIDIA Magpie |
| Speech to Text | Transcribe microphone or uploaded audio | faster-whisper, Parakeet, WhisperX, Nemotron ASR |
| Language Model | Rewrite text, cast characters, run conversation agent | LiteLLM, llama.cpp, Ollama, vLLM |

### Setup From Engines Page

1. Open **Setup -> Engines**.
2. Choose **Text to Speech**, **Speech to Text**, or **Language Models**.
3. Look at the cards at the top. These are detected local Docker containers.
4. If a card matches the engine you want, click its action button:
   - **Use as TTS**
   - **Use as Voice Design**
   - **Use as Custom Voice**
   - **Use as Fish-Speech**
   - **Use as VibeVoice**
5. If the engine is not detected, click **Add Custom** and enter the URL manually.
6. Use **Refresh** after starting or restarting containers.

### Common Local Ports

These are common local endpoints. Your stack may differ.

| Engine | Typical URL | Notes |
| --- | --- | --- |
| Qwen3 Voice Clone/Base | `http://host.docker.internal:8020` | Best default for cloned voices |
| Qwen3 Voice Design | `http://host.docker.internal:8021` | Required for design prompts |
| Qwen3 Custom Voice | `http://host.docker.internal:8022` | Custom voice/tone-style backend |
| Qwen3 Streaming | `http://host.docker.internal:8023` | Lower latency streaming |
| faster-whisper | `http://host.docker.internal:8010` | STT transcription |
| NVIDIA Parakeet ASR | `http://host.docker.internal:8090` or app card URL | Fast STT |
| NVIDIA Magpie TTS | `http://host.docker.internal:8091` | Fixed-speaker TTS |
| VibeVoice | `http://host.docker.internal:8027` | Simple TTS backend |
| XTTS v2 | `http://host.docker.internal:8005` | OpenAI-compatible voice clone endpoint |
| LiteLLM | `http://host.docker.internal:14000/v1` | OpenAI-compatible LLM gateway |
| llama-swap | `http://host.docker.internal:28080/v1` | LLM routing gateway |
| Ollama | `http://host.docker.internal:11434/v1` | Local LLM endpoint |

### Which Engines Make Sense Where?

| App Area | Good Engines |
| --- | --- |
| Clone a Voice | Qwen3 Voice Clone/Base, XTTS, Fish-Speech if configured |
| Design a Voice | Qwen3 Voice Design, Qwen3 Custom Voice |
| Try It Out | Any available TTS backend |
| Script Rehearser | Qwen3 Voice Clone/Base for cast voices; Voice Design for generating new cast voices; LLM for auto-casting |
| Conversation | Fast STT plus fast LLM plus fast/streaming TTS |
| Benchmark | Any backend you want to compare |

## 4. Voice Library

The voice library is the center of the app. A voice usually consists of:

- A reference audio file, usually WAV.
- A text transcript/reference prompt.
- Metadata such as language, gender, tags, rating, notes, and persona.
- Optional picture/avatar.

### Browsing Voices

1. Open **My Voices**.
2. Use sidebar filters such as **All voices**, **Cloned**, **Designed**, **Favorites**, and **Hidden**.
3. Use the search field to filter by name, tags, language, or metadata.
4. Click a voice to inspect and edit it.

### Voice Visibility

- Visible voices appear in normal pickers.
- Hidden voices are kept in the library but filtered out from most assignment/search fields.
- Use **Hidden** in the sidebar when you need to review or restore them.

### Metadata

For each voice you can usually manage:

- Display name
- Language
- Gender
- Tags
- Favorite flag
- Hidden/visible state
- Notes
- Character persona
- Reference text
- Picture/avatar

### Character Persona

A persona is a writing-style prompt attached to a voice. When enabled, the app sends your text through an LLM first so the generated speech sounds more like that character.

Example persona:

```text
Warm but dry-humored older engineer. Speaks clearly, uses practical metaphors, and avoids dramatic language.
```

Use persona rewrite when:

- You want an assistant to stay in character.
- A script line needs a more natural phrasing for a voice.
- You want a voice to preserve a speaking style across different apps.

Avoid persona rewrite when:

- You need exact text output.
- You are generating legal, medical, or quoted text.
- You are benchmarking raw TTS speed.

## 5. Clone A Voice

Use **Clone a Voice** when you have an audio sample and want to create a reusable voice.

### Good Source Audio

Best results come from audio that is:

- 5 to 30 seconds long.
- One speaker only.
- Clean and close-mic recorded.
- No music or loud background noise.
- Emotionally representative of the voice you want.
- Not heavily compressed or distorted.

### Workflow

1. Open **Clone a Voice**.
2. Choose a source:
   - Upload file
   - Record microphone
   - Paste URL
   - Import from another source if available
3. Use the waveform editor to trim to the cleanest section.
4. Transcribe the clip or enter the reference text manually.
5. Name the voice.
6. Set language, tags, and notes.
7. Click **Save to Voice Library**.
8. Go to **Try It Out** and test it.

### Tips

- Short clean clips beat long noisy clips.
- Trim out breaths, music, applause, and other speakers.
- Keep the transcript close to the actual words in the audio.
- Use consistent naming, for example `EN_F_Name_01` or `DE_M_Name_01`.

## 6. Design A Voice

Use **Design a Voice** when you want a new voice from a description rather than a recorded speaker.

### Workflow

1. Open **Design a Voice**.
2. Make sure the Voice Design backend is connected, usually Qwen3 Voice Design on port `8021`.
3. Write a voice prompt.
4. Generate a sample.
5. Adjust the prompt and regenerate until it fits.
6. Save the result to the voice library.

### Prompt Template

```text
A calm German male narrator in his late 40s, warm baritone, precise articulation, subtle smile, low breathiness, natural documentary pacing.
```

Useful prompt parts:

- Language/accent
- Age range
- Gender presentation
- Pitch/timbre
- Energy level
- Emotion
- Speaking pace
- Use case, such as narrator, villain, teacher, assistant

### Tone And Style Control

Some Qwen3-style backends support a style or instruction field per request. Use that for line-level emotion, such as:

```text
gentle reassurance, slower pace, quiet confidence
```

In Script Rehearser, tone can be set per line or character when the backend supports it.

## 7. Get Voices Online

Use **Get Voices Online** to find and import external voice clips.

General workflow:

1. Search or paste a source URL.
2. Preview the clip.
3. Import it into the library.
4. Clean up metadata.
5. Test it in **Try It Out**.

Always review licensing and consent before using a voice outside personal testing.

## 8. Try It Out

**Try It Out** is the fastest place to test voices and backends.

### Basic TTS Test

1. Select a TTS backend.
2. Click **Fetch voices**.
3. Pick a voice from the searchable picker.
4. Type text.
5. Optional: add style instruction.
6. Click synthesize/play.

### Searchable Voice Picker

The voice picker lets you:

- Search by voice name.
- See avatar/picture when available.
- Select quickly without scrolling through hundreds of voices.

If hidden voices appear where they should not, check the voice visibility in **My Voices -> Hidden**.

### Style Instruction

Use style instruction for delivery, not content.

Good:

```text
quiet, serious, slightly slower, emotionally restrained
```

Less good:

```text
change the sentence to be about pirates
```

### STT To TTS

Some Try It Out views let you:

1. Record or upload audio.
2. Transcribe it with STT.
3. Optionally clean/refine it with an LLM.
4. Re-synthesize it with another voice.

## 9. Script Rehearser

Script Rehearser is for importing a screenplay or dialogue, assigning voices to characters, and rehearsing selected parts.

### Main Phases

| Phase | Purpose |
| --- | --- |
| Library | Import, browse, or load scripts |
| Cast | Match characters to voices |
| Stage | Play, rehearse, select, hide, ignore, and edit lines |
| Summary | Review script/cast status |
| Import/Export | Move `.reh` files in and out |

### Import A Script

Options include:

- Paste text manually.
- Upload `.txt`, `.md`, `.pdf`, `.fdx`, or `.osf` when supported.
- Use **Browse scripts** to browse IMSDb scripts.
- Paste an IMSDb or Scripts.com URL.
- Import a `.reh` project.

### Browse Scripts

1. Open **Script Rehearser -> Library**.
2. Click **Browse scripts**.
3. Search titles in the modal.
4. Switch between thumbnail and list view.
5. Click a title to fetch it.
6. Click **Parse & cast** after it loads.

You can also paste a direct Scripts.com URL, for example:

```text
https://www.scripts.com/script/the_pianist_72/2
```

### Cast Characters

1. Parse the script.
2. Open **Cast**.
3. Choose a TTS backend.
4. Click **Fetch voices**.
5. Assign voices manually, or use LLM-assisted matching:
   - Match from local library
   - Match from online voices when available
   - Design new voices using Voice Design
6. Use the searchable voice picker to find voices quickly.
7. Add speaking-style notes for characters when needed.

### Stage Playback

The Stage lets you:

- Play the whole script.
- Play from a selected line.
- Click a character/avatar play button for an individual line.
- Skip descriptions.
- Pre-synthesize lines for faster playback.
- Re-synthesize stale lines when tone/style changed.
- Save/export the rehearsal project.

### Select Mode

Use **Select** in Stage when you want to work with only part of a script.

- Click **Select** to enter selection mode.
- Click lines to select or deselect them.
- Shift-click another line to select a range.
- Click **Set range** to make playback use that selected range.
- Use **Ignore**, **Un-ignore**, **Hide**, or **Delete** for bulk cleanup.
- Click **Done** when finished.

For actors, common navigation is usually by:

- Page number
- Scene heading
- Cue line
- Character entrance
- Act/scene number for stage plays

For this app, the cleanest practical workflow is: scene heading plus selected range. Example: "Rehearse from EXT. HARBOR - NIGHT through Jack's next line."

### Hide vs Ignore vs Delete

| Action | Meaning |
| --- | --- |
| Ignore | Keep visible, but skip during playback/synthesis |
| Hide | Remove from normal view, but keep recoverable |
| Delete | Permanently remove from the project |

## 10. Conversation Playground

Conversation Playground is a live voice-agent pipeline:

```text
Microphone -> STT -> LLM -> TTS -> Spoken answer
```

### Basic Setup

1. Pick an STT backend.
2. Enter or select an LLM endpoint.
3. Fetch/select an LLM model.
4. Pick a TTS backend.
5. Fetch/select a voice.
6. Set the system prompt.
7. Click the mic button to start live listening.

### Hot Mic Mode

The mic button is now an on/off toggle.

- Click once: mic is active and keeps listening.
- Click again: mic turns off.
- While active, silence does not submit empty turns.
- Speech interrupts LLM/TTS processing and starts a new recording.
- The app automatically enables Auto-stop, Hands-free, and Live agent for this mode.

### Live Agent Options

| Option | What It Does |
| --- | --- |
| Auto-stop | Sends speech after silence is detected |
| Hands-free | Re-arms the mic after the agent finishes |
| Live agent | Shorter silence and interruption delays |

### Interrupting The Agent

With hot mic or Live agent active, you can speak while the agent is:

- Generating LLM text
- Synthesizing speech
- Playing speech

The current turn is aborted and the app begins recording the new utterance.

### Good Conversation Settings

For fast live turn-taking:

- Use a fast STT backend such as Parakeet or faster-whisper on GPU.
- Use a small/fast LLM model for conversation.
- Use a low-latency TTS backend.
- Prefer streaming TTS if available.
- Keep the system prompt short.

## 11. Benchmark

Use **Benchmark** to compare voice/backend performance.

Typical use:

1. Select backend.
2. Select test text or sentence set.
3. Run one voice or a batch.
4. Compare latency, duration, and reliability.

Useful metrics:

- Time to first audio
- Total synthesis time
- Realtime factor
- Failures/timeouts
- Subjective quality notes

## 12. App Routing

App Routing lets other tools call one TTS endpoint while this app decides which voice/backend to use.

Use routing when:

- Multiple apps share the same TTS server.
- You want app-specific default voices.
- You want language-specific voices.
- You want before/after sounds for certain apps.

Typical workflow:

1. Open **App Routing**.
2. Add a client/app name.
3. Pick a default voice.
4. Choose backend if needed.
5. Add language or voice-name rules.
6. Test with the generated curl example.

## 13. Connect Apps And APIs

Open **Connect Apps** for ready-to-copy examples.

Common API patterns:

### Generate Speech

```bash
curl -X POST http://localhost:7890/speak \
  -H "Content-Type: application/json" \
  -d '{"text":"Hello from Voice Creator","voice":"EN_F_Example"}' \
  --output out.wav
```

### Bind A Client To A Voice

```bash
curl -X PUT http://localhost:7890/speak/bindings/my-app \
  -H "Content-Type: application/json" \
  -d '{"voice":"EN_F_Example"}'
```

### Use A Bound Voice

```bash
curl -X POST http://localhost:7890/speak \
  -H "Content-Type: application/json" \
  -H "X-Voice-Creator-Client-Id: my-app" \
  -d '{"text":"Use the voice bound to this app."}' \
  --output out.wav
```

### MCP

The app includes an MCP endpoint. Use **Connect Apps** or **Settings -> About** for current connection examples.

## 14. Settings

Settings are grouped by task.

| Area | Use It For |
| --- | --- |
| General | App defaults and behavior |
| Connections | Backend URLs for TTS, STT, LLM, NVIDIA, VibeVoice, etc. |
| Playback | Audio playback defaults |
| Captures | Saved recordings and generated clips |
| Payloads | Advanced request payload settings |
| Storage | Voice directories and file handling |
| API Keys | Cloud provider keys |
| Backup | Export/import app settings and library metadata |
| Logs | Inspect recent app logs |
| About | Version, examples, MCP info |

## 15. Troubleshooting

### Browser Microphone Does Not Work

Browsers require a secure context for microphone access.

Try one of these:

- Open the app at `http://localhost:7890` on the same machine.
- Use HTTPS.
- In Chrome, allow the local network origin as secure if needed.
- Check browser microphone permissions.

### Voice Picker Shows Old Data

- Click **Fetch voices** again.
- Hard refresh the page.
- If the app is installed as a PWA, close and reopen it.
- Check that the selected TTS backend is correct.

### Hidden Voices Appear

- Confirm the voice is actually marked hidden in **My Voices -> Hidden**.
- Fetch voices again.
- Hard refresh if the browser cached an older script.

### Script Browser Looks Wrong

- Switch between thumbnail and list view.
- Refresh the catalogue.
- Hard refresh the page if the modal layout looks stale.
- Paste the direct IMSDb/Scripts.com URL instead of browsing.

### Conversation Hears Nothing

- Confirm STT backend is running.
- Confirm browser mic permissions.
- Check **Speech to Text** engine status.
- Try the fake/manual text input to confirm LLM/TTS works.
- Turn hot mic off and on again.

### Conversation Sends Empty Turns

Hot mic mode should ignore silence. If empty turns still happen:

- Make sure the latest page is loaded with a hard refresh.
- Use **Auto-stop** and **Live agent** together.
- Increase microphone input level if real speech is too quiet.
- Reduce background noise.

### Agent Is Too Slow

Latency is the sum of STT, LLM, and TTS.

Improve it by:

- Using a faster STT engine.
- Using a smaller LLM model.
- Using streaming TTS.
- Keeping prompts short.
- Preloading models where possible.
- Avoiding huge conversation history.

### TTS Fails For A Voice

- Confirm the voice exists in the active voice library.
- Confirm the backend supports that voice type.
- Fetch voices again.
- Check logs in **Settings -> Logs**.
- Try another backend to isolate whether the issue is voice data or engine support.

### Service Worker Or Cache Issues

The app can be installed as a PWA and may cache UI files.

If UI changes do not appear:

1. Hard refresh the page.
2. Open DevTools and disable cache while reloading.
3. Unregister the service worker for the app origin.
4. Reopen the app.

## 16. Recommended Workflows

### Build A Clean Voice Library

1. Import or clone voices.
2. Normalize names and tags.
3. Mark favorites.
4. Hide experiments and failed voices.
5. Add persona prompts only to voices you want to use as characters.
6. Benchmark your best voices.

### Create A Script Rehearsal

1. Import or browse a script.
2. Parse and inspect characters.
3. Use Cast to assign voices.
4. Use LLM matching for a first pass.
5. Manually fix important roles.
6. Open Stage.
7. Select a scene/range.
8. Pre-synthesize the selected range.
9. Rehearse with playback or Train mode.
10. Save/export `.reh`.

### Build A Fast Voice Agent

1. Use Parakeet or faster-whisper for STT.
2. Use LiteLLM/llama-swap/Ollama with a small model.
3. Use a fast TTS backend.
4. Select a voice in Conversation.
5. Keep the system prompt short.
6. Turn on hot mic.
7. Interrupt naturally by speaking.

### Design Voices For A Cast

1. Parse script.
2. Open Cast.
3. Let the LLM describe character souls/briefs.
4. Use Voice Design for missing archetypes.
5. Save successful designs to the library.
6. Hide rejected experiments.
7. Assign final voices to the cast.

## Glossary

| Term | Meaning |
| --- | --- |
| Backend | A separate engine/server that performs TTS, STT, or LLM work |
| Voice clone | A reusable voice made from reference audio |
| Voice design | A voice generated from a text description |
| Persona | A character-writing prompt attached to a voice |
| STT | Speech to Text, transcription |
| TTS | Text to Speech, synthesis |
| LLM | Language model used for rewriting, matching, and conversation |
| Hot mic | Always-listening conversation mode toggled by the mic icon |
| `.reh` | Script Rehearser project export |