tts-voice-creator-clone-and-design-2/docs/USER_MANUAL.md

Voice Creator User Manual

This manual explains how to use TTS Voice Creator - Clone, Design, Deploy from the browser UI. It is written for day-to-day use: setting up engines, building a voice library, cloning and designing voices, rehearsing scripts, running a live voice agent, and connecting other apps.

Table Of Contents

1. Quick start
2. How the app is organized
3. Engines and backend setup
4. Voice library
5. Clone a Voice
6. Design a Voice
7. Get Voices Online
8. Try It Out
9. Script Rehearser
10. Conversation Playground
11. Benchmark
12. App Routing
13. Connect Apps and APIs
14. Settings
15. Troubleshooting
16. Recommended workflows

1. Quick Start

1. Start the Docker stack.

2. Open the app in a browser, usually:

   http://localhost:7890
   

   or, from another machine:

   http://<server-ip>:7890
   

3. Go to Setup -> Engines and confirm your STT, TTS, and LLM containers are detected.

4. Go to Setup -> Engines -> Text to Speech and click Use as TTS or the backend-specific button for the TTS engine you want.

5. Go to Speech to Text or Settings -> Connections and set your STT endpoint.

6. Go to Try It Out, fetch voices, select a voice, type text, and synthesize.

7. Build your voice library with Clone a Voice, Design a Voice, or Get Voices Online.

8. Use Script Rehearser for scripts or Conversation for a live voice agent.

2. How The App Is Organized

The left sidebar is the main map of the app.

Voices

• My Voices: Browse and manage all local voices.
• All voices: Every visible voice.
• Cloned: Voices created from reference audio.
• Designed: Voices created from a voice-design prompt.
• Favorites: Voices marked as favorites.
• Hidden: Voices hidden from normal pickers.
• Library tools: Bulk library maintenance.

Voice Creation

• Clone a Voice: Create a voice from an audio file, recording, URL, or extracted clip.
• Design a Voice: Create a new voice from a text description.
• Get Voices Online: Search and import online voice clips.
• Try It Out: Test TTS, STT, effects, voice style, and persona rewriting.

Script And Agent Workflows

• Script Rehearser: Import scripts, cast characters, assign voices, and rehearse scenes.
• Conversation: Talk to a live AI voice agent using STT -> LLM -> TTS.
• Benchmark: Measure speed and quality across voices and engines.

Setup And Integration

• Engines: See local Docker engines and connect them to the app.
• App Routing: Route incoming TTS requests from other apps to voices/backends.
• Connect Apps: Copy API examples for other tools.
• Settings: Configure URLs, storage, keys, logs, backup, and defaults.

3. Engines And Backend Setup

The app is a control surface. Most speech generation and recognition happens in separate local or cloud engines.

Engine Types

Type	Used For	Examples
Text to Speech	Generate spoken audio	Qwen3-TTS, Fish-Speech, XTTS, VibeVoice, NVIDIA Magpie
Speech to Text	Transcribe microphone or uploaded audio	faster-whisper, Parakeet, WhisperX, Nemotron ASR
Language Model	Rewrite text, cast characters, run conversation agent	LiteLLM, llama.cpp, Ollama, vLLM

Setup From Engines Page

1. Open Setup -> Engines.
2. Choose Text to Speech, Speech to Text, or Language Models.
3. Look at the cards at the top. These are detected local Docker containers.
4. If a card matches the engine you want, click its action button:
   • Use as TTS
   • Use as Voice Design
   • Use as Custom Voice
   • Use as Fish-Speech
   • Use as VibeVoice
5. If the engine is not detected, click Add Custom and enter the URL manually.
6. Use Refresh after starting or restarting containers.

Common Local Ports

These are common local endpoints. Your stack may differ.

Engine	Typical URL	Notes
Qwen3 Voice Clone/Base	http://host.docker.internal:8020	Best default for cloned voices
Qwen3 Voice Design	http://host.docker.internal:8021	Required for design prompts
Qwen3 Custom Voice	http://host.docker.internal:8022	Custom voice/tone-style backend
Qwen3 Streaming	http://host.docker.internal:8023	Lower latency streaming
faster-whisper	http://host.docker.internal:8010	STT transcription
NVIDIA Parakeet ASR	http://host.docker.internal:8090 or app card URL	Fast STT
NVIDIA Magpie TTS	http://host.docker.internal:8091	Fixed-speaker TTS
VibeVoice	http://host.docker.internal:8027	Simple TTS backend
XTTS v2	http://host.docker.internal:8005	OpenAI-compatible voice clone endpoint
LiteLLM	http://host.docker.internal:14000/v1	OpenAI-compatible LLM gateway
llama-swap	http://host.docker.internal:28080/v1	LLM routing gateway
Ollama	http://host.docker.internal:11434/v1	Local LLM endpoint

Which Engines Make Sense Where?

App Area	Good Engines
Clone a Voice	Qwen3 Voice Clone/Base, XTTS, Fish-Speech if configured
Design a Voice	Qwen3 Voice Design, Qwen3 Custom Voice
Try It Out	Any available TTS backend
Script Rehearser	Qwen3 Voice Clone/Base for cast voices; Voice Design for generating new cast voices; LLM for auto-casting
Conversation	Fast STT plus fast LLM plus fast/streaming TTS
Benchmark	Any backend you want to compare

4. Voice Library

The voice library is the center of the app. A voice usually consists of:

• A reference audio file, usually WAV.
• A text transcript/reference prompt.
• Metadata such as language, gender, tags, rating, notes, and persona.
• Optional picture/avatar.

Browsing Voices

1. Open My Voices.
2. Use sidebar filters such as All voices, Cloned, Designed, Favorites, and Hidden.
3. Use the search field to filter by name, tags, language, or metadata.
4. Click a voice to inspect and edit it.

Voice Visibility

• Visible voices appear in normal pickers.
• Hidden voices are kept in the library but filtered out from most assignment/search fields.
• Use Hidden in the sidebar when you need to review or restore them.

Metadata

For each voice you can usually manage:

• Display name
• Language
• Gender
• Tags
• Favorite flag
• Hidden/visible state
• Notes
• Character persona
• Reference text
• Picture/avatar

Character Persona

A persona is a writing-style prompt attached to a voice. When enabled, the app sends your text through an LLM first so the generated speech sounds more like that character.

Example persona:

Warm but dry-humored older engineer. Speaks clearly, uses practical metaphors, and avoids dramatic language.

Use persona rewrite when:

• You want an assistant to stay in character.
• A script line needs a more natural phrasing for a voice.
• You want a voice to preserve a speaking style across different apps.

Avoid persona rewrite when:

• You need exact text output.
• You are generating legal, medical, or quoted text.
• You are benchmarking raw TTS speed.

5. Clone A Voice

Use Clone a Voice when you have an audio sample and want to create a reusable voice.

Good Source Audio

Best results come from audio that is:

• 5 to 30 seconds long.
• One speaker only.
• Clean and close-mic recorded.
• No music or loud background noise.
• Emotionally representative of the voice you want.
• Not heavily compressed or distorted.

Workflow

1. Open Clone a Voice.
2. Choose a source:
   • Upload file
   • Record microphone
   • Paste URL
   • Import from another source if available
3. Use the waveform editor to trim to the cleanest section.
4. Transcribe the clip or enter the reference text manually.
5. Name the voice.
6. Set language, tags, and notes.
7. Click Save to Voice Library.
8. Go to Try It Out and test it.

Tips

• Short clean clips beat long noisy clips.
• Trim out breaths, music, applause, and other speakers.
• Keep the transcript close to the actual words in the audio.
• Use consistent naming, for example EN_F_Name_01 or DE_M_Name_01.

6. Design A Voice

Use Design a Voice when you want a new voice from a description rather than a recorded speaker.

Workflow

1. Open Design a Voice.
2. Make sure the Voice Design backend is connected, usually Qwen3 Voice Design on port 8021.
3. Write a voice prompt.
4. Generate a sample.
5. Adjust the prompt and regenerate until it fits.
6. Save the result to the voice library.

Prompt Template

A calm German male narrator in his late 40s, warm baritone, precise articulation, subtle smile, low breathiness, natural documentary pacing.

Useful prompt parts:

• Language/accent
• Age range
• Gender presentation
• Pitch/timbre
• Energy level
• Emotion
• Speaking pace
• Use case, such as narrator, villain, teacher, assistant

Tone And Style Control

Some Qwen3-style backends support a style or instruction field per request. Use that for line-level emotion, such as:

gentle reassurance, slower pace, quiet confidence

In Script Rehearser, tone can be set per line or character when the backend supports it.

7. Get Voices Online

Use Get Voices Online to find and import external voice clips.

General workflow:

1. Search or paste a source URL.
2. Preview the clip.
3. Import it into the library.
4. Clean up metadata.
5. Test it in Try It Out.

Always review licensing and consent before using a voice outside personal testing.

8. Try It Out

Try It Out is the fastest place to test voices and backends.

Basic TTS Test

1. Select a TTS backend.
2. Click Fetch voices.
3. Pick a voice from the searchable picker.
4. Type text.
5. Optional: add style instruction.
6. Click synthesize/play.

Searchable Voice Picker

The voice picker lets you:

• Search by voice name.
• See avatar/picture when available.
• Select quickly without scrolling through hundreds of voices.

If hidden voices appear where they should not, check the voice visibility in My Voices -> Hidden.

Style Instruction

Use style instruction for delivery, not content.

Good:

quiet, serious, slightly slower, emotionally restrained

Less good:

change the sentence to be about pirates

STT To TTS

Some Try It Out views let you:

1. Record or upload audio.
2. Transcribe it with STT.
3. Optionally clean/refine it with an LLM.
4. Re-synthesize it with another voice.

9. Script Rehearser

Script Rehearser is for importing a screenplay or dialogue, assigning voices to characters, and rehearsing selected parts.

Main Phases

Phase	Purpose
Library	Import, browse, or load scripts
Cast	Match characters to voices
Stage	Play, rehearse, select, hide, ignore, and edit lines
Summary	Review script/cast status
Import/Export	Move .reh files in and out

Import A Script

Options include:

• Paste text manually.
• Upload .txt, .md, .pdf, .fdx, or .osf when supported.
• Use Browse scripts to browse IMSDb scripts.
• Paste an IMSDb or Scripts.com URL.
• Import a .reh project.

Browse Scripts

1. Open Script Rehearser -> Library.
2. Click Browse scripts.
3. Search titles in the modal.
4. Switch between thumbnail and list view.
5. Click a title to fetch it.
6. Click Parse & cast after it loads.

You can also paste a direct Scripts.com URL, for example:

https://www.scripts.com/script/the_pianist_72/2

Cast Characters

1. Parse the script.
2. Open Cast.
3. Choose a TTS backend.
4. Click Fetch voices.
5. Assign voices manually, or use LLM-assisted matching:
   • Match from local library
   • Match from online voices when available
   • Design new voices using Voice Design
6. Use the searchable voice picker to find voices quickly.
7. Add speaking-style notes for characters when needed.

Stage Playback

The Stage lets you:

• Play the whole script.
• Play from a selected line.
• Click a character/avatar play button for an individual line.
• Skip descriptions.
• Pre-synthesize lines for faster playback.
• Re-synthesize stale lines when tone/style changed.
• Save/export the rehearsal project.

Select Mode

Use Select in Stage when you want to work with only part of a script.

• Click Select to enter selection mode.
• Click lines to select or deselect them.
• Shift-click another line to select a range.
• Click Set range to make playback use that selected range.
• Use Ignore, Un-ignore, Hide, or Delete for bulk cleanup.
• Click Done when finished.

For actors, common navigation is usually by:

• Page number
• Scene heading
• Cue line
• Character entrance
• Act/scene number for stage plays

For this app, the cleanest practical workflow is: scene heading plus selected range. Example: "Rehearse from EXT. HARBOR - NIGHT through Jack's next line."

Hide vs Ignore vs Delete

Action	Meaning
Ignore	Keep visible, but skip during playback/synthesis
Hide	Remove from normal view, but keep recoverable
Delete	Permanently remove from the project

10. Conversation Playground

Conversation Playground is a live voice-agent pipeline:

Microphone -> STT -> LLM -> TTS -> Spoken answer

Basic Setup

1. Pick an STT backend.
2. Enter or select an LLM endpoint.
3. Fetch/select an LLM model.
4. Pick a TTS backend.
5. Fetch/select a voice.
6. Set the system prompt.
7. Click the mic button to start live listening.

Hot Mic Mode

The mic button is now an on/off toggle.

• Click once: mic is active and keeps listening.
• Click again: mic turns off.
• While active, silence does not submit empty turns.
• Speech interrupts LLM/TTS processing and starts a new recording.
• The app automatically enables Auto-stop, Hands-free, and Live agent for this mode.

Live Agent Options

Option	What It Does
Auto-stop	Sends speech after silence is detected
Hands-free	Re-arms the mic after the agent finishes
Live agent	Shorter silence and interruption delays

Interrupting The Agent

With hot mic or Live agent active, you can speak while the agent is:

• Generating LLM text
• Synthesizing speech
• Playing speech

The current turn is aborted and the app begins recording the new utterance.

Good Conversation Settings

For fast live turn-taking:

• Use a fast STT backend such as Parakeet or faster-whisper on GPU.
• Use a small/fast LLM model for conversation.
• Use a low-latency TTS backend.
• Prefer streaming TTS if available.
• Keep the system prompt short.

11. Benchmark

Use Benchmark to compare voice/backend performance.

Typical use:

1. Select backend.
2. Select test text or sentence set.
3. Run one voice or a batch.
4. Compare latency, duration, and reliability.

Useful metrics:

• Time to first audio
• Total synthesis time
• Realtime factor
• Failures/timeouts
• Subjective quality notes

12. App Routing

App Routing lets other tools call one TTS endpoint while this app decides which voice/backend to use.

Use routing when:

• Multiple apps share the same TTS server.
• You want app-specific default voices.
• You want language-specific voices.
• You want before/after sounds for certain apps.

Typical workflow:

1. Open App Routing.
2. Add a client/app name.
3. Pick a default voice.
4. Choose backend if needed.
5. Add language or voice-name rules.
6. Test with the generated curl example.

13. Connect Apps And APIs

Open Connect Apps for ready-to-copy examples.

Common API patterns:

Generate Speech

curl -X POST http://localhost:7890/speak \
  -H "Content-Type: application/json" \
  -d '{"text":"Hello from Voice Creator","voice":"EN_F_Example"}' \
  --output out.wav

Bind A Client To A Voice

curl -X PUT http://localhost:7890/speak/bindings/my-app \
  -H "Content-Type: application/json" \
  -d '{"voice":"EN_F_Example"}'

Use A Bound Voice

curl -X POST http://localhost:7890/speak \
  -H "Content-Type: application/json" \
  -H "X-Voice-Creator-Client-Id: my-app" \
  -d '{"text":"Use the voice bound to this app."}' \
  --output out.wav

MCP

The app includes an MCP endpoint. Use Connect Apps or Settings -> About for current connection examples.

14. Settings

Settings are grouped by task.

Area	Use It For
General	App defaults and behavior
Connections	Backend URLs for TTS, STT, LLM, NVIDIA, VibeVoice, etc.
Playback	Audio playback defaults
Captures	Saved recordings and generated clips
Payloads	Advanced request payload settings
Storage	Voice directories and file handling
API Keys	Cloud provider keys
Backup	Export/import app settings and library metadata
Logs	Inspect recent app logs
About	Version, examples, MCP info

15. Troubleshooting

Browser Microphone Does Not Work

Browsers require a secure context for microphone access.

Try one of these:

• Open the app at http://localhost:7890 on the same machine.
• Use HTTPS.
• In Chrome, allow the local network origin as secure if needed.
• Check browser microphone permissions.

Voice Picker Shows Old Data

• Click Fetch voices again.
• Hard refresh the page.
• If the app is installed as a PWA, close and reopen it.
• Check that the selected TTS backend is correct.

Hidden Voices Appear

• Confirm the voice is actually marked hidden in My Voices -> Hidden.
• Fetch voices again.
• Hard refresh if the browser cached an older script.

Script Browser Looks Wrong

• Switch between thumbnail and list view.
• Refresh the catalogue.
• Hard refresh the page if the modal layout looks stale.
• Paste the direct IMSDb/Scripts.com URL instead of browsing.

Conversation Hears Nothing

• Confirm STT backend is running.
• Confirm browser mic permissions.
• Check Speech to Text engine status.
• Try the fake/manual text input to confirm LLM/TTS works.
• Turn hot mic off and on again.

Conversation Sends Empty Turns

Hot mic mode should ignore silence. If empty turns still happen:

• Make sure the latest page is loaded with a hard refresh.
• Use Auto-stop and Live agent together.
• Increase microphone input level if real speech is too quiet.
• Reduce background noise.

Agent Is Too Slow

Latency is the sum of STT, LLM, and TTS.

Improve it by:

• Using a faster STT engine.
• Using a smaller LLM model.
• Using streaming TTS.
• Keeping prompts short.
• Preloading models where possible.
• Avoiding huge conversation history.

TTS Fails For A Voice

• Confirm the voice exists in the active voice library.
• Confirm the backend supports that voice type.
• Fetch voices again.
• Check logs in Settings -> Logs.
• Try another backend to isolate whether the issue is voice data or engine support.

Service Worker Or Cache Issues

The app can be installed as a PWA and may cache UI files.

If UI changes do not appear:

1. Hard refresh the page.
2. Open DevTools and disable cache while reloading.
3. Unregister the service worker for the app origin.
4. Reopen the app.

16. Recommended Workflows

Build A Clean Voice Library

1. Import or clone voices.
2. Normalize names and tags.
3. Mark favorites.
4. Hide experiments and failed voices.
5. Add persona prompts only to voices you want to use as characters.
6. Benchmark your best voices.

Create A Script Rehearsal

1. Import or browse a script.
2. Parse and inspect characters.
3. Use Cast to assign voices.
4. Use LLM matching for a first pass.
5. Manually fix important roles.
6. Open Stage.
7. Select a scene/range.
8. Pre-synthesize the selected range.
9. Rehearse with playback or Train mode.
10. Save/export .reh.

Build A Fast Voice Agent

1. Use Parakeet or faster-whisper for STT.
2. Use LiteLLM/llama-swap/Ollama with a small model.
3. Use a fast TTS backend.
4. Select a voice in Conversation.
5. Keep the system prompt short.
6. Turn on hot mic.
7. Interrupt naturally by speaking.

Design Voices For A Cast

1. Parse script.
2. Open Cast.
3. Let the LLM describe character souls/briefs.
4. Use Voice Design for missing archetypes.
5. Save successful designs to the library.
6. Hide rejected experiments.
7. Assign final voices to the cast.

Glossary

Term	Meaning
Backend	A separate engine/server that performs TTS, STT, or LLM work
Voice clone	A reusable voice made from reference audio
Voice design	A voice generated from a text description
Persona	A character-writing prompt attached to a voice
STT	Speech to Text, transcription
TTS	Text to Speech, synthesis
LLM	Language model used for rewriting, matching, and conversation
Hot mic	Always-listening conversation mode toggled by the mic icon
.reh	Script Rehearser project export