- Deterministic hash-based character portraits and reference sheets - Implemented strict early Renaissance aesthetic controls and role-specific headwear rules - Added audiobook batch export scripts and progress tooling - Verified book profiles with a new 'confirmed' state tracking mechanism - Updated Pillow dependency for robust image handling - Configured .gitignore for audiobook and audio cache exports
283 lines
45 KiB
HTML
283 lines
45 KiB
HTML
<!doctype html><html><head><meta charset="utf-8"><title>Emotion samples - book03</title>
|
|
<style>
|
|
body{font-family:system-ui,sans-serif;background:#1e1e2e;color:#cdd6f4;padding:24px;max-width:900px;margin:0 auto}
|
|
h1{color:#f9e2af} h2{color:#f5c2e7;margin-top:40px;border-bottom:2px solid #45475a;padding-bottom:6px}
|
|
h3{color:#89b4fa;margin-top:28px;border-bottom:1px solid #313244;padding-bottom:4px}
|
|
table{width:100%;border-collapse:collapse;margin-top:8px}
|
|
td,th{padding:8px;border-bottom:1px solid #313244;text-align:left;vertical-align:middle}
|
|
th{color:#a6adc8;font-weight:600}
|
|
audio{width:280px;height:32px}
|
|
p.note{color:#a6adc8}
|
|
</style></head><body>
|
|
<h1>Emotion fix - sample comparison</h1>
|
|
<p class="note">Same lines, same voices, run through both backends after the fix. Fish-Speech gets a translated English [tag] baked into the text; Qwen3 Voice Design gets the native German instruct sentence, unchanged.</p>
|
|
|
|
<h2>1. Book-realistic emotions (as the LLM actually assigns them, e.g. "befehlend", "resigniert")</h2>
|
|
<h3>MARCIAN</h3><table><tr><th>Emotion</th><th>Fish-Speech</th><th>Qwen3 Voice Design</th></tr><tr><td>befehlend</td><td><audio controls src="marcian_1_befehlend_fish.mp3"></audio></td><td><audio controls src="marcian_1_befehlend_qwen3design.mp3"></audio></td></tr><tr><td>kalt und zynisch</td><td><audio controls src="marcian_2_kalt_und_zynisch_fish.mp3"></audio></td><td><audio controls src="marcian_2_kalt_und_zynisch_qwen3design.mp3"></audio></td></tr><tr><td>fragend</td><td><audio controls src="marcian_3_fragend_fish.mp3"></audio></td><td><audio controls src="marcian_3_fragend_qwen3design.mp3"></audio></td></tr><tr><td>entschlossen</td><td><audio controls src="marcian_4_entschlossen_fish.mp3"></audio></td><td><audio controls src="marcian_4_entschlossen_qwen3design.mp3"></audio></td></tr><tr><td>ernst</td><td><audio controls src="marcian_5_ernst_fish.mp3"></audio></td><td><audio controls src="marcian_5_ernst_qwen3design.mp3"></audio></td></tr></table><h3>LYSANDRA</h3><table><tr><th>Emotion</th><th>Fish-Speech</th><th>Qwen3 Voice Design</th></tr><tr><td>resigniert</td><td><audio controls src="lysandra_1_resigniert_fish.mp3"></audio></td><td><audio controls src="lysandra_1_resigniert_qwen3design.mp3"></audio></td></tr><tr><td>besorgt</td><td><audio controls src="lysandra_2_besorgt_fish.mp3"></audio></td><td><audio controls src="lysandra_2_besorgt_qwen3design.mp3"></audio></td></tr><tr><td>abweisend</td><td><audio controls src="lysandra_3_abweisend_fish.mp3"></audio></td><td><audio controls src="lysandra_3_abweisend_qwen3design.mp3"></audio></td></tr><tr><td>verächtlich</td><td><audio controls src="lysandra_4_verachtlich_fish.mp3"></audio></td><td><audio controls src="lysandra_4_verachtlich_qwen3design.mp3"></audio></td></tr><tr><td>schroff</td><td><audio controls src="lysandra_5_schroff_fish.mp3"></audio></td><td><audio controls src="lysandra_5_schroff_qwen3design.mp3"></audio></td></tr></table><h3>DARRAG</h3><table><tr><th>Emotion</th><th>Fish-Speech</th><th>Qwen3 Voice Design</th></tr><tr><td>befehlend</td><td><audio controls src="darrag_1_befehlend_fish.mp3"></audio></td><td><audio controls src="darrag_1_befehlend_qwen3design.mp3"></audio></td></tr><tr><td>entschlossen</td><td><audio controls src="darrag_2_entschlossen_fish.mp3"></audio></td><td><audio controls src="darrag_2_entschlossen_qwen3design.mp3"></audio></td></tr><tr><td>dringend</td><td><audio controls src="darrag_3_dringend_fish.mp3"></audio></td><td><audio controls src="darrag_3_dringend_qwen3design.mp3"></audio></td></tr><tr><td>drohend</td><td><audio controls src="darrag_4_drohend_fish.mp3"></audio></td><td><audio controls src="darrag_4_drohend_qwen3design.mp3"></audio></td></tr><tr><td>überrascht</td><td><audio controls src="darrag_5_uberrascht_fish.mp3"></audio></td><td><audio controls src="darrag_5_uberrascht_qwen3design.mp3"></audio></td></tr></table>
|
|
|
|
<h2>2. Basic emotions (happy / sad / angry / scared / excited)</h2>
|
|
<h3>MARCIAN</h3><table><tr><th>Emotion</th><th>Fish-Speech</th><th>Qwen3 Voice Design</th></tr><tr><td>happy (glücklich)</td><td><audio controls src="basic_marcian_1_happy_fish.mp3"></audio></td><td><audio controls src="basic_marcian_1_happy_qwen3design.mp3"></audio></td></tr><tr><td>sad (traurig)</td><td><audio controls src="basic_marcian_2_sad_fish.mp3"></audio></td><td><audio controls src="basic_marcian_2_sad_qwen3design.mp3"></audio></td></tr><tr><td>angry (wütend)</td><td><audio controls src="basic_marcian_3_angry_fish.mp3"></audio></td><td><audio controls src="basic_marcian_3_angry_qwen3design.mp3"></audio></td></tr><tr><td>scared (ängstlich)</td><td><audio controls src="basic_marcian_4_scared_fish.mp3"></audio></td><td><audio controls src="basic_marcian_4_scared_qwen3design.mp3"></audio></td></tr><tr><td>excited (aufgeregt)</td><td><audio controls src="basic_marcian_5_excited_fish.mp3"></audio></td><td><audio controls src="basic_marcian_5_excited_qwen3design.mp3"></audio></td></tr></table><h3>LYSANDRA</h3><table><tr><th>Emotion</th><th>Fish-Speech</th><th>Qwen3 Voice Design</th></tr><tr><td>happy (glücklich)</td><td><audio controls src="basic_lysandra_1_happy_fish.mp3"></audio></td><td><audio controls src="basic_lysandra_1_happy_qwen3design.mp3"></audio></td></tr><tr><td>sad (traurig)</td><td><audio controls src="basic_lysandra_2_sad_fish.mp3"></audio></td><td><audio controls src="basic_lysandra_2_sad_qwen3design.mp3"></audio></td></tr><tr><td>angry (wütend)</td><td><audio controls src="basic_lysandra_3_angry_fish.mp3"></audio></td><td><audio controls src="basic_lysandra_3_angry_qwen3design.mp3"></audio></td></tr><tr><td>scared (ängstlich)</td><td><audio controls src="basic_lysandra_4_scared_fish.mp3"></audio></td><td><audio controls src="basic_lysandra_4_scared_qwen3design.mp3"></audio></td></tr><tr><td>excited (aufgeregt)</td><td><audio controls src="basic_lysandra_5_excited_fish.mp3"></audio></td><td><audio controls src="basic_lysandra_5_excited_qwen3design.mp3"></audio></td></tr></table><h3>DARRAG</h3><table><tr><th>Emotion</th><th>Fish-Speech</th><th>Qwen3 Voice Design</th></tr><tr><td>happy (glücklich)</td><td><audio controls src="basic_darrag_1_happy_fish.mp3"></audio></td><td><audio controls src="basic_darrag_1_happy_qwen3design.mp3"></audio></td></tr><tr><td>sad (traurig)</td><td><audio controls src="basic_darrag_2_sad_fish.mp3"></audio></td><td><audio controls src="basic_darrag_2_sad_qwen3design.mp3"></audio></td></tr><tr><td>angry (wütend)</td><td><audio controls src="basic_darrag_3_angry_fish.mp3"></audio></td><td><audio controls src="basic_darrag_3_angry_qwen3design.mp3"></audio></td></tr><tr><td>scared (ängstlich)</td><td><audio controls src="basic_darrag_4_scared_fish.mp3"></audio></td><td><audio controls src="basic_darrag_4_scared_qwen3design.mp3"></audio></td></tr><tr><td>excited (aufgeregt)</td><td><audio controls src="basic_darrag_5_excited_fish.mp3"></audio></td><td><audio controls src="basic_darrag_5_excited_qwen3design.mp3"></audio></td></tr></table>
|
|
|
|
<h2>3. Controlled test — SAME exact line, only the tag/instruct changes</h2>
|
|
<p class="note">This isolates the emotion variable cleanly (sections 1 & 2 above used different lines per emotion). Measured acoustics (pitch/loudness) are shown next to each — Fish-Speech shows a large, consistent shift (sad/whisper much lower & quieter, angry/happy higher & louder); Qwen3 shows a strong shift for "happy" but almost none for "angry".</p>
|
|
<h3>Fish-Speech</h3>
|
|
<table><tr><th>Version</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness (RMS)</th></tr>
|
|
<tr><td>Neutral (no tag)</td><td><audio controls src="ctrl/neutral_notag.mp3"></audio></td><td>160</td><td>0.047</td></tr>
|
|
<tr><td>[angry]</td><td><audio controls src="ctrl/angry_tag.mp3"></audio></td><td>226</td><td>0.065</td></tr>
|
|
<tr><td>[happy]</td><td><audio controls src="ctrl/happy_tag.mp3"></audio></td><td>203</td><td>0.064</td></tr>
|
|
<tr><td>[sad]</td><td><audio controls src="ctrl/sad_tag.mp3"></audio></td><td>92</td><td>0.021</td></tr>
|
|
<tr><td>[whisper]</td><td><audio controls src="ctrl/whisper_tag.mp3"></audio></td><td>84</td><td>0.020</td></tr>
|
|
</table>
|
|
<h3>Qwen3 Voice Design</h3>
|
|
<table><tr><th>Version</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness (RMS)</th></tr>
|
|
<tr><td>Neutral</td><td><audio controls src="ctrl/neutral_qwen.mp3"></audio></td><td>158</td><td>0.057</td></tr>
|
|
<tr><td>Angry</td><td><audio controls src="ctrl/angry_qwen.mp3"></audio></td><td>158</td><td>0.065</td></tr>
|
|
<tr><td>Happy</td><td><audio controls src="ctrl/happy_qwen.mp3"></audio></td><td>310</td><td>0.072</td></tr>
|
|
</table>
|
|
|
|
<h2>4. English text/voice test (same character-role voice, English content)</h2>
|
|
<p class="note">Testing whether emotion differentiation is stronger in English than German. Pitch numbers below are measured (not felt) — Fish-Speech barely moves at all in English (72-80Hz across every emotion); Qwen3 Voice Design shows a genuinely clean, coherent gradient: sad < angry < shocked < happy. This is the clearest sign yet that <strong>Qwen3 Voice Design's emotional differentiation is much stronger in English than German</strong> — Fish-Speech stays weak regardless of language.</p>
|
|
<h3>Fish-Speech (English)</h3>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness</th></tr>
|
|
<tr><td>Neutral</td><td><audio controls src="ctrl6/en_fish_neutral.mp3"></audio></td><td>76</td><td>0.080</td></tr>
|
|
<tr><td>Whispering</td><td><audio controls src="ctrl6/en_fish_whispering.mp3"></audio></td><td>75</td><td>0.066</td></tr>
|
|
<tr><td>Sad</td><td><audio controls src="ctrl6/en_fish_sad.mp3"></audio></td><td>80</td><td>0.073</td></tr>
|
|
<tr><td>Angry</td><td><audio controls src="ctrl6/en_fish_angry.mp3"></audio></td><td>73</td><td>0.088</td></tr>
|
|
<tr><td>Happy</td><td><audio controls src="ctrl6/en_fish_happy.mp3"></audio></td><td>72</td><td>0.082</td></tr>
|
|
<tr><td>Shocked</td><td><audio controls src="ctrl6/en_fish_shocked.mp3"></audio></td><td>80</td><td>0.077</td></tr>
|
|
</table>
|
|
<h3>Qwen3 Voice Design (English)</h3>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness</th></tr>
|
|
<tr><td>Neutral</td><td><audio controls src="ctrl6/en_qwen_neutral.mp3"></audio></td><td>109</td><td>0.066</td></tr>
|
|
<tr><td>Whispering</td><td><audio controls src="ctrl6/en_qwen_whispering.mp3"></audio></td><td>(breathy, pitch untrackable)</td><td>0.040</td></tr>
|
|
<tr><td>Sad</td><td><audio controls src="ctrl6/en_qwen_sad.mp3"></audio></td><td>274</td><td>0.049</td></tr>
|
|
<tr><td>Angry</td><td><audio controls src="ctrl6/en_qwen_angry.mp3"></audio></td><td>297</td><td>0.090</td></tr>
|
|
<tr><td>Shocked</td><td><audio controls src="ctrl6/en_qwen_shocked.mp3"></audio></td><td>348</td><td>0.093</td></tr>
|
|
<tr><td>Happy</td><td><audio controls src="ctrl6/en_qwen_happy.mp3"></audio></td><td>398</td><td>0.091</td></tr>
|
|
</table>
|
|
|
|
<h2>5. German text + English instruction (Qwen3 Voice Design, Marcian's cloned voice)</h2>
|
|
<p class="note">Same German dialogue line, but the instruct sentence is written in English instead of German, to see if that captures some of section 4's improvement while keeping the actual book text in German. Angry stood out clearly (340Hz vs ~200Hz baseline); sad/happy/shocked stayed muddled together.</p>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness</th></tr>
|
|
<tr><td>Neutral</td><td><audio controls src="ctrl7/mix_neutral.mp3"></audio></td><td>223</td><td>0.062</td></tr>
|
|
<tr><td>Sad</td><td><audio controls src="ctrl7/mix_sad.mp3"></audio></td><td>201</td><td>0.072</td></tr>
|
|
<tr><td>Angry</td><td><audio controls src="ctrl7/mix_angry.mp3"></audio></td><td>340</td><td>0.097</td></tr>
|
|
<tr><td>Happy</td><td><audio controls src="ctrl7/mix_happy.mp3"></audio></td><td>222</td><td>0.074</td></tr>
|
|
<tr><td>Shocked</td><td><audio controls src="ctrl7/mix_shocked.mp3"></audio></td><td>175</td><td>0.068</td></tr>
|
|
</table>
|
|
|
|
<h2 style="color:#a6e3a1">6. ⚠ RE-TEST on the FIXED build (this is the one that matters)</h2>
|
|
<p class="note"><strong>Everything in sections 1–5 above was generated while a JavaScript bug was silently breaking the app.</strong>
|
|
A <code>typeof</code> check on a <code>const</code> in its temporal dead zone was throwing during startup, which stopped
|
|
the whole bundle from initializing past that point. Those earlier samples were measuring a half-working app, so please
|
|
disregard them. Below is the same controlled test (identical German line, identical voice, only the emotion varying)
|
|
re-run after the fix — plus emotion instructions are now written in English, which Qwen3 follows far more reliably.</p>
|
|
<h3>Qwen3 Voice Design — German text, English instruct</h3>
|
|
<p class="note">Now a clean, correctly-ordered gradient, and note the <strong>durations</strong> vary sensibly too: sad drags (5.76s), scared rushes (3.20s). That's real prosody, not just volume.</p>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness</th><th>Duration</th></tr><tr><td>Whisper</td><td><audio controls src="fixed/qwen_whisper.mp3"></audio></td><td>128</td><td>0.048</td><td>4.48s</td></tr><tr><td>Sad</td><td><audio controls src="fixed/qwen_sad.mp3"></audio></td><td>142</td><td>0.048</td><td>5.76s</td></tr><tr><td>Neutral</td><td><audio controls src="fixed/qwen_neutral.mp3"></audio></td><td>179</td><td>0.039</td><td>3.92s</td></tr><tr><td>Scared</td><td><audio controls src="fixed/qwen_scared.mp3"></audio></td><td>203</td><td>0.078</td><td>3.20s</td></tr><tr><td>Happy</td><td><audio controls src="fixed/qwen_happy.mp3"></audio></td><td>225</td><td>0.066</td><td>3.84s</td></tr><tr><td>Angry</td><td><audio controls src="fixed/qwen_angry.mp3"></audio></td><td>269</td><td>0.073</td><td>4.48s</td></tr></table>
|
|
<h3>Fish-Speech — inline [tag]</h3>
|
|
<p class="note">Still essentially two registers (quiet/low: whisper+sad — loud/high: everything else). Scared/happy/angry remain hard to tell apart.</p>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch (Hz)</th><th>Loudness</th><th>Duration</th></tr><tr><td>Whisper</td><td><audio controls src="fixed/fish_whisper.mp3"></audio></td><td>101</td><td>0.021</td><td>3.95s</td></tr><tr><td>Sad</td><td><audio controls src="fixed/fish_sad.mp3"></audio></td><td>91</td><td>0.020</td><td>4.46s</td></tr><tr><td>Neutral</td><td><audio controls src="fixed/fish_neutral.mp3"></audio></td><td>160</td><td>0.047</td><td>3.90s</td></tr><tr><td>Scared</td><td><audio controls src="fixed/fish_scared.mp3"></audio></td><td>223</td><td>0.064</td><td>4.23s</td></tr><tr><td>Happy</td><td><audio controls src="fixed/fish_happy.mp3"></audio></td><td>203</td><td>0.064</td><td>3.90s</td></tr><tr><td>Angry</td><td><audio controls src="fixed/fish_angry.mp3"></audio></td><td>226</td><td>0.065</td><td>4.50s</td></tr></table>
|
|
|
|
<h2 style="color:#f38ba8">7. ★ THE ACTUAL FIX — voice_clone backend (read this one)</h2>
|
|
<p class="note"><strong>Why everything above was misleading:</strong> Voice Design re-rolls the voice on every single call.
|
|
I generated the <em>identical</em> prompt 4 times and got 174, 232, 174 and 264 Hz — an 89 Hz spread with nothing
|
|
changed. That is the same size as the "emotion differences" I was reporting, so those numbers were mostly random noise.
|
|
It also explains what you heard: the male→female jumps, and "sad sounds happy". Voice Design designs a <em>new</em>
|
|
voice from the prompt; it is not a stable speaker, so it is the wrong tool for an audiobook.</p>
|
|
<p class="note"><strong>voice_clone is deterministic</strong> — the same input produced bit-identical audio twice over.
|
|
So on this backend, any difference you hear is genuinely the emotion, not a re-roll. Character identity holds
|
|
(measured timbre drift ≤ 0.011).</p>
|
|
<h3>A. Emotion instruct alone — <span style="color:#a6e3a1">recommended</span></h3>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch</th><th>Duration</th><th>Loudness</th></tr><tr><td>Whisper</td><td><audio controls src="clone/only_whisper.mp3"></audio></td><td>206</td><td>4.32s</td><td>0.072</td></tr><tr><td>Sad</td><td><audio controls src="clone/only_sad.mp3"></audio></td><td>170</td><td>4.16s</td><td>0.058</td></tr><tr><td>Neutral</td><td><audio controls src="clone/only_neutral.mp3"></audio></td><td>225</td><td>3.84s</td><td>0.071</td></tr><tr><td>Scared</td><td><audio controls src="clone/only_scared.mp3"></audio></td><td>239</td><td>4.16s</td><td>0.072</td></tr><tr><td>Happy</td><td><audio controls src="clone/only_happy.mp3"></audio></td><td>234</td><td>3.84s</td><td>0.070</td></tr><tr><td>Angry</td><td><audio controls src="clone/only_angry.mp3"></audio></td><td>233</td><td>3.92s</td><td>0.071</td></tr></table>
|
|
<h3>B. Same, but with the accent clause appended — <span style="color:#f38ba8">emotion collapses</span></h3>
|
|
<p class="note">Two style directives in one instruct compete and the trailing one wins: the range squashes to
|
|
222–250 Hz and sad all but disappears. I've now dropped the accent clause whenever a line carries an
|
|
emotion — a cloned voice already gets its accent from the reference WAV.</p>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch</th><th>Duration</th><th>Loudness</th></tr><tr><td>Whisper</td><td><audio controls src="clone/em_whisper.mp3"></audio></td><td>242</td><td>3.92s</td><td>0.062</td></tr><tr><td>Sad</td><td><audio controls src="clone/em_sad.mp3"></audio></td><td>232</td><td>4.08s</td><td>0.069</td></tr><tr><td>Neutral</td><td><audio controls src="clone/em_neutral.mp3"></audio></td><td>222</td><td>3.92s</td><td>0.066</td></tr><tr><td>Scared</td><td><audio controls src="clone/em_scared.mp3"></audio></td><td>250</td><td>3.92s</td><td>0.076</td></tr><tr><td>Happy</td><td><audio controls src="clone/em_happy.mp3"></audio></td><td>234</td><td>3.76s</td><td>0.066</td></tr><tr><td>Angry</td><td><audio controls src="clone/em_angry.mp3"></audio></td><td>231</td><td>3.92s</td><td>0.070</td></tr></table>
|
|
|
|
<h2 style="color:#fab387">8. Different characters — you were right about the shouting</h2>
|
|
<p class="note"><strong>Marcian's reference clip is the culprit for the shouting:</strong> it is only 3.1 seconds long and sits at
|
|
<strong>253 Hz</strong>, which is shouting range for a male voice, with the narrowest dynamic range of any voice checked.
|
|
Cloning reproduces that faithfully, so every Marcian line shouts no matter what emotion is requested.</p>
|
|
<p class="note"><strong>But swapping to calm voices exposes the deeper problem.</strong> On these two the emotional pitch range is
|
|
only 19–22 Hz — barely audible. Marcian's wider 69 Hz spread was mostly just his shouty baseline having room
|
|
to fall, not better emotional control. So voice_clone keeps identity rock-steady but delivers only weak emotion.</p>
|
|
<h3>DE_M_Darrag — calm 110 Hz reference</h3>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch</th><th>Duration</th></tr><tr><td>Whisper</td><td><audio controls src="voices/darrag_whisper.mp3"></audio></td><td>112</td><td>4.56s</td></tr><tr><td>Sad</td><td><audio controls src="voices/darrag_sad.mp3"></audio></td><td>109</td><td>4.32s</td></tr><tr><td>Neutral</td><td><audio controls src="voices/darrag_neutral.mp3"></audio></td><td>111</td><td>4.72s</td></tr><tr><td>Happy</td><td><audio controls src="voices/darrag_happy.mp3"></audio></td><td>110</td><td>4.40s</td></tr><tr><td>Angry</td><td><audio controls src="voices/darrag_angry.mp3"></audio></td><td>131</td><td>4.48s</td></tr></table>
|
|
<h3>Ernst Meinecke — professional narrator, 21s reference</h3>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch</th><th>Duration</th></tr><tr><td>Whisper</td><td><audio controls src="voices/ernst_whisper.mp3"></audio></td><td>145</td><td>5.68s</td></tr><tr><td>Sad</td><td><audio controls src="voices/ernst_sad.mp3"></audio></td><td>135</td><td>4.88s</td></tr><tr><td>Neutral</td><td><audio controls src="voices/ernst_neutral.mp3"></audio></td><td>154</td><td>5.12s</td></tr><tr><td>Happy</td><td><audio controls src="voices/ernst_happy.mp3"></audio></td><td>149</td><td>6.08s</td></tr><tr><td>Angry</td><td><audio controls src="voices/ernst_angry.mp3"></audio></td><td>151</td><td>5.28s</td></tr></table>
|
|
<p class="note" style="border-left:3px solid #fab387;padding-left:10px"><strong>Where this leaves us:</strong>
|
|
Fish-Speech → two registers only, everything shouts. Voice Design → strong changes but re-rolls the voice every
|
|
call, so identity drifts and results are random. voice_clone → perfectly stable identity but only ~20 Hz of emotion.
|
|
None of the three running backends does <em>both</em>. <strong>CustomVoice (port 8022) is down</strong> — per the Qwen3-TTS
|
|
docs that is the variant built specifically for timbre control plus emotional style, and it is the one option not yet tested.</p>
|
|
|
|
<h2 style="color:#89b4fa">9. You were right — it's the model, not the settings</h2>
|
|
<p class="note">I checked what each container actually loads, and your hypothesis was exactly correct:</p>
|
|
<table>
|
|
<tr><th>Backend</th><th>Model loaded</th><th>Instruction / emotion control?</th></tr>
|
|
<tr><td>voice_clone (8020)</td><td>Qwen3-TTS-12Hz-1.7B-<strong>Base</strong></td><td style="color:#f38ba8">No</td></tr>
|
|
<tr><td>streaming (8023)</td><td>same <strong>Base</strong> model</td><td style="color:#f38ba8">No</td></tr>
|
|
<tr><td>VoiceDesign (8021)</td><td>...-VoiceDesign</td><td style="color:#a6e3a1">Yes</td></tr>
|
|
<tr><td>CustomVoice (8022)</td><td>...-CustomVoice</td><td style="color:#a6e3a1">Yes</td></tr>
|
|
</table>
|
|
<p class="note">So my earlier "voice_clone responds to emotion" claim was wrong — that was a <em>Base</em> model being mildly
|
|
perturbed by instruction text it does not understand, which is why the range was only 19–22 Hz. I have also
|
|
confirmed on HuggingFace that <strong>no "Instruct" variant of Qwen3-TTS exists</strong> — the family is exactly these three.</p>
|
|
<h3>CustomVoice, started and tested (speaker Ryan, German text)</h3>
|
|
<p class="note">This is the strongest real emotion control measured so far: <strong>85 Hz range</strong>, with speaker identity
|
|
holding (timbre drift ≤0.008) and sensible pacing (angry fast 4.08s, scared slow 6.16s). Repeat runs confirm the effect is
|
|
genuine, not noise — sad lands at 111–136 Hz and angry at 159–210 Hz, ranges that never overlap.</p>
|
|
<table><tr><th>Emotion</th><th>Audio</th><th>Pitch</th><th>Duration</th><th>Loudness</th></tr><tr><td>Whisper</td><td><audio controls src="cv/ryan_whisper.mp3"></audio></td><td>121</td><td>5.76s</td><td>0.037</td></tr><tr><td>Sad</td><td><audio controls src="cv/ryan_sad.mp3"></audio></td><td>187</td><td>5.92s</td><td>0.043</td></tr><tr><td>Neutral</td><td><audio controls src="cv/ryan_neutral.mp3"></audio></td><td>134</td><td>5.12s</td><td>0.025</td></tr><tr><td>Happy</td><td><audio controls src="cv/ryan_happy.mp3"></audio></td><td>187</td><td>4.88s</td><td>0.066</td></tr><tr><td>Scared</td><td><audio controls src="cv/ryan_scared.mp3"></audio></td><td>163</td><td>6.16s</td><td>0.054</td></tr><tr><td>Angry</td><td><audio controls src="cv/ryan_angry.mp3"></audio></td><td>206</td><td>4.08s</td><td>0.053</td></tr></table>
|
|
<p class="note" style="border-left:3px solid #f38ba8;padding-left:10px"><strong>The catch:</strong> CustomVoice exposes only
|
|
<strong>9 fixed speakers</strong> (Ryan, Aiden, Vivian, Serena, Uncle_Fu, Dylan, Eric…) and its API has <em>no reference-audio
|
|
field at all</em> — it cannot clone. So it cannot voice the book's 57 cloned characters. Qwen3-TTS splits cloning and emotion
|
|
across two models that cannot be combined: Base clones but ignores instructions, CustomVoice obeys instructions but cannot clone,
|
|
VoiceDesign obeys instructions but invents a new voice on every call.</p>
|
|
|
|
<h2 style="color:#cba6f7">10. VoiceDesign vs CustomVoice — the consistency problem</h2>
|
|
<p class="note">Your plan (narrator on voice_clone, characters on VoiceDesign) is sound in structure —
|
|
<strong>the narrator on voice_clone is exactly right</strong>: identity is rock-steady and he needs no emotion.
|
|
But before committing 50 hours to VoiceDesign for the characters, listen to this.</p>
|
|
<p class="note">Each block below is the <strong>same character, same line, same emotion, called three times</strong>.
|
|
Nothing changes between runs. VoiceDesign has no seed parameter, so it cannot be pinned.</p>
|
|
<h3 style="color:#f38ba8">VoiceDesign — "happy" ×3 (identical requests)</h3>
|
|
<table><tr><th>#</th><th>Audio</th><th>Pitch</th></tr><tr><td>Run 1</td><td><audio controls src="ab/vd_happy_run0.mp3"></audio></td><td>163 Hz</td></tr><tr><td>Run 2</td><td><audio controls src="ab/vd_happy_run1.mp3"></audio></td><td>238 Hz</td></tr><tr><td>Run 3</td><td><audio controls src="ab/vd_happy_run2.mp3"></audio></td><td>178 Hz</td></tr><tr><td colspan="2" style="text-align:right"><em>spread across identical calls</em></td><td><strong>75 Hz</strong></td></tr></table>
|
|
<h3 style="color:#f38ba8">VoiceDesign — "sad" ×3 (identical requests)</h3>
|
|
<table><tr><th>#</th><th>Audio</th><th>Pitch</th></tr><tr><td>Run 1</td><td><audio controls src="ab/vd_sad_run0.mp3"></audio></td><td>190 Hz</td></tr><tr><td>Run 2</td><td><audio controls src="ab/vd_sad_run1.mp3"></audio></td><td>130 Hz</td></tr><tr><td>Run 3</td><td><audio controls src="ab/vd_sad_run2.mp3"></audio></td><td>214 Hz</td></tr><tr><td colspan="2" style="text-align:right"><em>spread across identical calls</em></td><td><strong>84 Hz</strong></td></tr></table>
|
|
<h3 style="color:#a6e3a1">CustomVoice — "sad" ×3 (identical requests)</h3>
|
|
<table><tr><th>#</th><th>Audio</th><th>Pitch</th></tr><tr><td>Run 1</td><td><audio controls src="ab/cv_sad_run0.mp3"></audio></td><td>125 Hz</td></tr><tr><td>Run 2</td><td><audio controls src="ab/cv_sad_run1.mp3"></audio></td><td>116 Hz</td></tr><tr><td>Run 3</td><td><audio controls src="ab/cv_sad_run2.mp3"></audio></td><td>119 Hz</td></tr><tr><td colspan="2" style="text-align:right"><em>spread across identical calls</em></td><td><strong>9 Hz</strong></td></tr></table>
|
|
<h3>CustomVoice — "happy" ×3 (identical requests)</h3>
|
|
<table><tr><th>#</th><th>Audio</th><th>Pitch</th></tr><tr><td>Run 1</td><td><audio controls src="ab/cv_happy_run0.mp3"></audio></td><td>158 Hz</td></tr><tr><td>Run 2</td><td><audio controls src="ab/cv_happy_run1.mp3"></audio></td><td>269 Hz</td></tr><tr><td>Run 3</td><td><audio controls src="ab/cv_happy_run2.mp3"></audio></td><td>178 Hz</td></tr><tr><td colspan="2" style="text-align:right"><em>spread across identical calls</em></td><td><strong>111 Hz</strong></td></tr></table>
|
|
<p class="note" style="border-left:3px solid #f38ba8;padding-left:10px">
|
|
<strong>Averaged over 3 runs each, VoiceDesign's emotions do not separate reliably:</strong> neutral 136 Hz,
|
|
sad 158, happy 231, angry 237 — but "happy" alone ranged 122–311 Hz across identical calls. The variation
|
|
<em>within</em> one emotion is larger than the difference <em>between</em> emotions, and sad landed above neutral
|
|
(backwards). Over ~2000 lines that means a character whose pitch wanders line to line.<br><br>
|
|
CustomVoice's "sad" by contrast is tight (116–125 Hz, 9 Hz spread) — genuinely reliable. Its weakness is
|
|
coverage: 9 fixed speakers, no cloning.</p>
|
|
|
|
<h2 style="color:#a6e3a1">11. Fish-Speech — I was wrong, and it may be the best option</h2>
|
|
<p class="note"><strong>Correction:</strong> Fish-Speech is <em>perfectly deterministic</em>. The same request twice produced
|
|
byte-identical audio (same MD5, 0 Hz spread) because the app pins a per-voice seed. That changes the picture completely:</p>
|
|
<table>
|
|
<tr><th>Engine</th><th>sad→angry separation</th><th>Run-to-run noise</th><th>Verdict</th></tr>
|
|
<tr><td><strong>Fish-Speech</strong></td><td><strong>135 Hz</strong></td><td><strong>0 Hz</strong></td><td style="color:#a6e3a1">signal, no noise</td></tr>
|
|
<tr><td>CustomVoice</td><td>51 Hz</td><td>26–51 Hz</td><td style="color:#f9e2af">signal ≈ noise</td></tr>
|
|
<tr><td>VoiceDesign</td><td>101 Hz</td><td>up to 189 Hz</td><td style="color:#f38ba8">noise > signal</td></tr>
|
|
</table>
|
|
<p class="note">And it is <strong>three</strong> registers, not two — I under-sold it earlier. Because output is deterministic,
|
|
this map is exact rather than a sample. Emotion tags reliably land in one of three intensity levels:</p>
|
|
<h3 style="color:#89b4fa">Subdued — quiet, low</h3><table><tr><th>Tag</th><th>Audio</th><th>Pitch</th><th>Dur</th><th>Loudness</th></tr><tr><td>sad</td><td><audio controls src="fishmap/sad.mp3"></audio></td><td>91 Hz</td><td>4.46s</td><td>0.0205</td></tr><tr><td>whispering</td><td><audio controls src="fishmap/whispering.mp3"></audio></td><td>101 Hz</td><td>3.95s</td><td>0.0211</td></tr><tr><td>calm</td><td><audio controls src="fishmap/calm.mp3"></audio></td><td>102 Hz</td><td>3.81s</td><td>0.0244</td></tr><tr><td>disappointed</td><td><audio controls src="fishmap/disappointed.mp3"></audio></td><td>105 Hz</td><td>3.99s</td><td>0.0231</td></tr><tr><td>curious</td><td><audio controls src="fishmap/curious.mp3"></audio></td><td>104 Hz</td><td>3.99s</td><td>0.0274</td></tr><tr><td>resigned</td><td><audio controls src="fishmap/resigned.mp3"></audio></td><td>115 Hz</td><td>4.27s</td><td>0.0353</td></tr></table><h3 style="color:#a6e3a1">Neutral / warm — mid</h3><table><tr><th>Tag</th><th>Audio</th><th>Pitch</th><th>Dur</th><th>Loudness</th></tr><tr><td>neutral</td><td><audio controls src="fishmap/neutral.mp3"></audio></td><td>160 Hz</td><td>3.9s</td><td>0.047</td></tr><tr><td>tender</td><td><audio controls src="fishmap/tender.mp3"></audio></td><td>168 Hz</td><td>3.44s</td><td>0.0413</td></tr><tr><td>sobbing</td><td><audio controls src="fishmap/sobbing.mp3"></audio></td><td>177 Hz</td><td>4.6s</td><td>0.052</td></tr></table><h3 style="color:#f38ba8">Energized — loud, high</h3><table><tr><th>Tag</th><th>Audio</th><th>Pitch</th><th>Dur</th><th>Loudness</th></tr><tr><td>confident</td><td><audio controls src="fishmap/confident.mp3"></audio></td><td>194 Hz</td><td>4.32s</td><td>0.0591</td></tr><tr><td>sarcastic</td><td><audio controls src="fishmap/sarcastic.mp3"></audio></td><td>200 Hz</td><td>4.04s</td><td>0.064</td></tr><tr><td>happy</td><td><audio controls src="fishmap/happy.mp3"></audio></td><td>203 Hz</td><td>3.9s</td><td>0.0642</td></tr><tr><td>scared</td><td><audio controls src="fishmap/scared.mp3"></audio></td><td>223 Hz</td><td>4.23s</td><td>0.0639</td></tr><tr><td>angry</td><td><audio controls src="fishmap/angry.mp3"></audio></td><td>226 Hz</td><td>4.5s</td><td>0.0649</td></tr><tr><td>excited</td><td><audio controls src="fishmap/excited.mp3"></audio></td><td>230 Hz</td><td>4.18s</td><td>0.0644</td></tr><tr><td>shouting</td><td><audio controls src="fishmap/shouting.mp3"></audio></td><td>234 Hz</td><td>4.04s</td><td>0.0649</td></tr></table>
|
|
<p class="note" style="border-left:3px solid #a6e3a1;padding-left:10px">
|
|
<strong>What this means for the book:</strong> Fish clones all 57 characters, holds every identity perfectly stable, and
|
|
gives three dependable intensity levels — subdued for sad/whispered/resigned lines, neutral for ordinary narration,
|
|
energized for angry/excited/frightened ones. What it will <em>not</em> do is distinguish angry from happy: both sit in the
|
|
energized band (226 vs 203 Hz). The emotional contour of a scene comes through; the specific colour does not.<br><br>
|
|
Determinism has a practical bonus too: a re-run reproduces identical audio, so an interrupted export resumes cleanly
|
|
instead of drifting.</p>
|
|
|
|
<h2 style="color:#f9e2af">12. English voice + English bracket tags (as requested) — and an important correction</h2>
|
|
<p class="note">Voice: <strong>EN_F_Melina</strong> (best dynamic range of the English references, 10.4s clip).
|
|
English text, English tags in square brackets before the line, per the S2 word-level-control article.
|
|
I tested both plain single-word tags and the article's richer free-form style (e.g. its own example
|
|
<code>[voice rough from crying, trying to sound normal]</code>, and pairing a physical reaction with an emotion).</p>
|
|
<h3>Simple single-word tags</h3>
|
|
<table><tr><th>Emotion</th><th>Tag</th><th>Audio</th><th>Pitch</th><th>Dur</th></tr><tr><td>neutral</td><td style="font-size:12px;color:#a6adc8"><em>(no tag)</em></td><td><audio controls src="en5/simple_neutral.mp3"></audio></td><td>187 Hz</td><td>4.23s</td></tr><tr><td>happy</td><td style="font-size:12px;color:#a6adc8">[happy]</td><td><audio controls src="en5/simple_happy.mp3"></audio></td><td>186 Hz</td><td>3.95s</td></tr><tr><td>sad</td><td style="font-size:12px;color:#a6adc8">[sad]</td><td><audio controls src="en5/simple_sad.mp3"></audio></td><td>186 Hz</td><td>4.18s</td></tr><tr><td>angry</td><td style="font-size:12px;color:#a6adc8">[angry]</td><td><audio controls src="en5/simple_angry.mp3"></audio></td><td>186 Hz</td><td>4.32s</td></tr><tr><td>scared</td><td style="font-size:12px;color:#a6adc8">[scared]</td><td><audio controls src="en5/simple_scared.mp3"></audio></td><td>197 Hz</td><td>3.95s</td></tr></table>
|
|
<h3>Rich descriptive tags (article's recommended style)</h3>
|
|
<table><tr><th>Emotion</th><th>Tag</th><th>Audio</th><th>Pitch</th><th>Dur</th></tr><tr><td>neutral</td><td style="font-size:12px;color:#a6adc8"><em>(no tag)</em></td><td><audio controls src="en5/rich_neutral.mp3"></audio></td><td>187 Hz</td><td>4.23s</td></tr><tr><td>happy</td><td style="font-size:12px;color:#a6adc8">[bright and delighted, barely containing a grin]</td><td><audio controls src="en5/rich_happy.mp3"></audio></td><td>185 Hz</td><td>4.04s</td></tr><tr><td>sad</td><td style="font-size:12px;color:#a6adc8">[voice rough from crying, trying to sound normal]</td><td><audio controls src="en5/rich_sad.mp3"></audio></td><td>188 Hz</td><td>4.27s</td></tr><tr><td>angry</td><td style="font-size:12px;color:#a6adc8">[furious, teeth clenched, spitting the words out]</td><td><audio controls src="en5/rich_angry.mp3"></audio></td><td>196 Hz</td><td>5.25s</td></tr><tr><td>scared</td><td style="font-size:12px;color:#a6adc8">[breathless with fear, voice trembling]</td><td><audio controls src="en5/rich_scared.mp3"></audio></td><td>183 Hz</td><td>3.95s</td></tr></table>
|
|
<p class="note" style="border-left:3px solid #f38ba8;padding-left:10px">
|
|
<strong>Correction to section 11 — I over-claimed.</strong> Both tag styles above land within
|
|
<strong>183–197 Hz</strong>, i.e. essentially no emotional differentiation, and the richer tags were no better
|
|
than the simple ones. Checking further, <em>every</em> other voice behaves the same way:<br><br>
|
|
<table>
|
|
<tr><th>Voice</th><th>sad</th><th>angry</th><th>range</th></tr>
|
|
<tr><td><strong>DE_M_Marcian</strong></td><td>91 Hz</td><td>226 Hz</td><td style="color:#a6e3a1"><strong>135 Hz</strong></td></tr>
|
|
<tr><td>DE_M_Darrag</td><td>86</td><td>94</td><td style="color:#f38ba8">8 Hz</td></tr>
|
|
<tr><td>DE_F_Lysandra</td><td>186</td><td>176</td><td style="color:#f38ba8">10 Hz</td></tr>
|
|
<tr><td>Ernst Meinecke (narrator)</td><td>109</td><td>111</td><td style="color:#f38ba8">2 Hz</td></tr>
|
|
<tr><td>EN_M_Alan_Rickman</td><td>72</td><td>80</td><td style="color:#f38ba8">8 Hz</td></tr>
|
|
<tr><td>EN_M_Albus_Dumbledore</td><td>189</td><td>184</td><td style="color:#f38ba8">5 Hz</td></tr>
|
|
</table><br>
|
|
<strong>Marcian is the single outlier</strong> — and section 11's whole "three registers" map was built on him alone.
|
|
His reference clip is an anomalous 3.1-second shout at 253 Hz, so a <code>[sad]</code> tag has enormous room to pull him
|
|
down. Normal, well-recorded references barely move at all. I also tested whether reference <em>length</em> was the cause by
|
|
truncating two voices to 3 seconds — it made no difference, so it is the unusual character of that particular clip,
|
|
not its duration.</p>
|
|
|
|
<h2 style="color:#a6e3a1">13. ★ How Fish-Speech S2 emotion ACTUALLY works</h2>
|
|
<p class="note">I investigated the mechanism instead of guessing. The finding explains every confusing result above:</p>
|
|
<p class="note" style="border-left:3px solid #a6e3a1;padding-left:10px;font-size:15px">
|
|
<strong>Fish-Speech emotion is driven by the REFERENCE CLIP, not by the text tags.</strong><br>
|
|
When you supply reference audio, S2 does in-context prosody imitation — it copies the reference's delivery, and that
|
|
overrides bracket tags almost entirely. Tags only work when there is no reference competing with them.</p>
|
|
<h3>Evidence A — the same tags, with and without a reference</h3>
|
|
<table><tr><th>Tag</th><th>No reference</th><th>With reference (cloning)</th></tr>
|
|
<tr><td>[sad]</td><td style="color:#a6e3a1">loudness 0.051 (−40%), 21% slower</td><td style="color:#f38ba8">0.096 (−11%)</td></tr>
|
|
<tr><td>[angry]</td><td style="color:#a6e3a1">0.094, dynamics 0.324, darker timbre</td><td style="color:#f38ba8">0.097, dynamics 0.245</td></tr>
|
|
<tr><td>[whispering]</td><td style="color:#a6e3a1">0.054 (−36%)</td><td style="color:#f38ba8">0.091 (−16%)</td></tr>
|
|
</table>
|
|
<h3>Evidence B — emotion transfers through the reference with NO tag at all</h3>
|
|
<p class="note">Identical plain sentence, no bracket tags anywhere. Only the reference clip changed:</p>
|
|
<table><tr><th>Reference used</th><th>Pitch</th><th>Loudness</th><th>Brightness</th></tr>
|
|
<tr><td>calm / neutral</td><td>183 Hz</td><td>0.075</td><td>2699</td></tr>
|
|
<tr><td><strong>scared</strong> reference</td><td>106 Hz</td><td><strong>0.022</strong> (5× quieter)</td><td>2380</td></tr>
|
|
<tr><td><strong>angry</strong> reference</td><td>121 Hz</td><td><strong>0.108</strong> (loud)</td><td>1778 (dark)</td></tr>
|
|
<tr><td><strong>happy</strong> reference</td><td>210 Hz</td><td>0.106</td><td>2177</td></tr>
|
|
</table>
|
|
<h3>The 5 samples you asked for — tags with no competing reference</h3>
|
|
<p class="note">These finally differentiate: <strong>10× loudness spread</strong>, pitch 110–222 Hz, and genuine
|
|
intonation differences (happy pstd 45 vs scared pstd 5). Caveat: with no reference the model also picks its own voice,
|
|
so identity is not controlled here (speaker distance 0.027 vs 0.011 for a true clone).</p>
|
|
<table><tr><th>Emotion</th><th>Tag used</th><th>Audio</th><th>Pitch</th><th>Dur</th><th>Loudness</th></tr><tr><td>neutral</td><td style="font-size:12px;color:#a6adc8"><em>(no tag)</em></td><td><audio controls src="works/neutral.mp3"></audio></td><td>182 Hz</td><td>3.58s</td><td>0.084</td></tr><tr><td>happy</td><td style="font-size:12px;color:#a6adc8">[cheerful and bright, smiling as she speaks]</td><td><audio controls src="works/happy.mp3"></audio></td><td>222 Hz</td><td>4.27s</td><td>0.0997</td></tr><tr><td>sad</td><td style="font-size:12px;color:#a6adc8">[voice rough from crying, trying to sound normal]</td><td><audio controls src="works/sad.mp3"></audio></td><td>155 Hz</td><td>3.2s</td><td>0.2098</td></tr><tr><td>angry</td><td style="font-size:12px;color:#a6adc8">[furious, teeth clenched, spitting the words out]</td><td><audio controls src="works/angry.mp3"></audio></td><td>129 Hz</td><td>3.72s</td><td>0.1037</td></tr><tr><td>scared</td><td style="font-size:12px;color:#a6adc8">[breathless with fear, voice trembling]</td><td><audio controls src="works/scared.mp3"></audio></td><td>110 Hz</td><td>4.13s</td><td>0.0208</td></tr></table>
|
|
<p class="note" style="border-left:3px solid #89b4fa;padding-left:10px">
|
|
<strong>What this means — and the way forward.</strong> The trade-off is inherent: a reference gives you the character's
|
|
identity but suppresses tag-based emotion; no reference gives you emotion but an uncontrolled voice. <em>But Evidence B points
|
|
at a real solution:</em> give each character <strong>several reference clips, one per emotional register</strong> (neutral,
|
|
subdued, intense) and pick the matching one per line. That uses the mechanism instead of fighting it — delivering
|
|
cloned identity <em>and</em> emotion, both fully deterministic.</p>
|
|
|
|
<h2 style="color:#f38ba8">14. ★ THE PROOF — same text, same tags, cloning on vs off</h2>
|
|
<p class="note">Your five-line example, generated twice through Fish Audio S2 Pro. Identical text, identical tags.
|
|
The <em>only</em> difference is whether a cloned reference voice was supplied.</p>
|
|
<h3 style="color:#f38ba8">A. WITH a cloned voice (DE_F_Einfaches_Maedchen) — what you heard</h3>
|
|
<audio controls src="multi/multi_tag.mp3" style="width:100%;max-width:520px"></audio>
|
|
<p class="note">Pitch across all five emotions: 241–268 Hz — a <strong>27 Hz</strong> spread. Every line lands in the
|
|
same place, which is exactly what you reported: "no difference between the lines".</p>
|
|
<h3 style="color:#a6e3a1">B. WITHOUT a reference voice — same text, same tags</h3>
|
|
<audio controls src="multi/no_ref.mp3" style="width:100%;max-width:520px"></audio>
|
|
<table><tr><th>Line</th><th>Pitch</th><th>Loudness</th></tr>
|
|
<tr><td>[happy]</td><td>293 Hz</td><td>0.194 (loudest)</td></tr>
|
|
<tr><td>[uncertain]</td><td>237 Hz</td><td>0.148</td></tr>
|
|
<tr><td>[sad]</td><td>184 Hz</td><td>0.125</td></tr>
|
|
<tr><td>[hopeful]</td><td>207 Hz</td><td>0.084</td></tr>
|
|
<tr><td>[determined]</td><td>170 Hz</td><td>0.061 (quietest)</td></tr></table>
|
|
<p class="note"><strong>123 Hz spread and 3.2× loudness range</strong> — four to five times the differentiation, in a
|
|
sensible order (happy highest and loudest, determined lowest and quietest). Because this is one single generation, the
|
|
voice stays consistent throughout; the earlier "5 different voices" problem only happened when each emotion was generated
|
|
as its own separate call.</p>
|
|
<p class="note" style="border-left:3px solid #f38ba8;padding-left:10px">
|
|
This is the trade-off in one page: <strong>Fish gives you the cloned character voice, or the emotion tags — never both.</strong>
|
|
The reference clip is prepended as an in-context example and the model imitates its delivery, which overrides the tags.</p>
|
|
|
|
<h2 style="color:#89b4fa">15. Controlling the voice WITHOUT a reference clip</h2>
|
|
<p class="note">If no reference clip is used, the voice can still be steered — by <strong>describing it in a tag</strong>.
|
|
S2-Pro accepts free-form descriptions, so gender, age and character are controllable directly:</p>
|
|
<table><tr><th>Description tag</th><th>Audio</th><th>Pitch</th></tr><tr><td><code>(no tag)</code></td><td><audio controls src="voicectl/no_tag.mp3"></audio></td><td>182 Hz</td></tr><tr><td><code>[deep male voice]</code></td><td><audio controls src="voicectl/deep_male.mp3"></audio></td><td>91 Hz</td></tr><tr><td><code>[young woman's voice]</code></td><td><audio controls src="voicectl/young_woman.mp3"></audio></td><td>191 Hz</td></tr><tr><td><code>[elderly man, frail and raspy]</code></td><td><audio controls src="voicectl/elderly_man.mp3"></audio></td><td>93 Hz</td></tr><tr><td><code>[small child, high voice]</code></td><td><audio controls src="voicectl/little_girl.mp3"></audio></td><td>202 Hz</td></tr><tr><td><code>[gruff old warrior, deep and gravelly]</code></td><td><audio controls src="voicectl/gruff_old_warrior.mp3"></audio></td><td>72 Hz</td></tr></table>
|
|
<p class="note">That is a 72–202 Hz range — clearly distinguishable men, women and children, all without any reference clip.</p>
|
|
<h3 style="color:#a6e3a1">Reproducible: same description + same seed = identical audio</h3>
|
|
<p class="note">Three runs of the same description and seed produced <strong>byte-identical files</strong> (matching MD5).
|
|
So a character can be pinned as "description + seed" and will sound the same every time — this fixes the
|
|
"5 different voices" problem from section 13.</p>
|
|
<h3 style="color:#f9e2af">The catch: emotion tags also shift the voice</h3>
|
|
<p class="note">Same character description throughout, only the emotion tag changes. Emotion clearly works
|
|
(174 Hz range, 6.4× loudness) — but the character drifts as well. "Identity drift" below compares timbre
|
|
against the neutral take; a true cloned voice stays at 0.001–0.011, and roughly 0.03+ starts to read as a different person.</p>
|
|
<table><tr><th>Tag</th><th>Audio</th><th>Pitch</th><th>Loudness</th><th>Identity drift</th></tr><tr><td><code>(no emotion tag)</code></td><td><audio controls src="chardesc/neutral.mp3"></audio></td><td>72 Hz</td><td>0.033</td><td>—</td></tr><tr><td><code>[happy]</code></td><td><audio controls src="chardesc/happy.mp3"></audio></td><td>174 Hz</td><td>0.113</td><td>0.033</td></tr><tr><td><code>[sad]</code></td><td><audio controls src="chardesc/sad.mp3"></audio></td><td>—</td><td>0.209</td><td>0.212</td></tr><tr><td><code>[angry]</code></td><td><audio controls src="chardesc/angry.mp3"></audio></td><td>123 Hz</td><td>0.201</td><td>0.044</td></tr><tr><td><code>[whisper]</code></td><td><audio controls src="chardesc/whisper.mp3"></audio></td><td>81 Hz</td><td>0.100</td><td>0.069</td></tr></table>
|
|
<p class="note" style="border-left:3px solid #89b4fa;padding-left:10px">
|
|
<strong>Please judge this by ear</strong> — do these five still sound like the same person, or like different speakers?
|
|
If they hold together well enough, this is a workable path for the audiobook: define each character as a description plus a
|
|
fixed seed, and emotion tags will actually function. If they do not, then cloned voices with neutral delivery remains the
|
|
honest option.</p>
|
|
</body></html> |