Emotion fix - sample comparison

Same lines, same voices, run through both backends after the fix. Fish-Speech gets a translated English [tag] baked into the text; Qwen3 Voice Design gets the native German instruct sentence, unchanged.

1. Book-realistic emotions (as the LLM actually assigns them, e.g. "befehlend", "resigniert")

MARCIAN

LYSANDRA

DARRAG

2. Basic emotions (happy / sad / angry / scared / excited)

MARCIAN

LYSANDRA

DARRAG

3. Controlled test — SAME exact line, only the tag/instruct changes

This isolates the emotion variable cleanly (sections 1 & 2 above used different lines per emotion). Measured acoustics (pitch/loudness) are shown next to each — Fish-Speech shows a large, consistent shift (sad/whisper much lower & quieter, angry/happy higher & louder); Qwen3 shows a strong shift for "happy" but almost none for "angry".

Fish-Speech

Qwen3 Voice Design

4. English text/voice test (same character-role voice, English content)

Testing whether emotion differentiation is stronger in English than German. Pitch numbers below are measured (not felt) — Fish-Speech barely moves at all in English (72-80Hz across every emotion); Qwen3 Voice Design shows a genuinely clean, coherent gradient: sad < angry < shocked < happy. This is the clearest sign yet that Qwen3 Voice Design's emotional differentiation is much stronger in English than German — Fish-Speech stays weak regardless of language.

Fish-Speech (English)

Qwen3 Voice Design (English)

5. German text + English instruction (Qwen3 Voice Design, Marcian's cloned voice)

Same German dialogue line, but the instruct sentence is written in English instead of German, to see if that captures some of section 4's improvement while keeping the actual book text in German. Angry stood out clearly (340Hz vs ~200Hz baseline); sad/happy/shocked stayed muddled together.

6. ⚠ RE-TEST on the FIXED build (this is the one that matters)

Everything in sections 1–5 above was generated while a JavaScript bug was silently breaking the app. A typeof check on a const in its temporal dead zone was throwing during startup, which stopped the whole bundle from initializing past that point. Those earlier samples were measuring a half-working app, so please disregard them. Below is the same controlled test (identical German line, identical voice, only the emotion varying) re-run after the fix — plus emotion instructions are now written in English, which Qwen3 follows far more reliably.

Qwen3 Voice Design — German text, English instruct

Now a clean, correctly-ordered gradient, and note the durations vary sensibly too: sad drags (5.76s), scared rushes (3.20s). That's real prosody, not just volume.

Fish-Speech — inline [tag]

Still essentially two registers (quiet/low: whisper+sad — loud/high: everything else). Scared/happy/angry remain hard to tell apart.

7. ★ THE ACTUAL FIX — voice_clone backend (read this one)

Why everything above was misleading: Voice Design re-rolls the voice on every single call. I generated the identical prompt 4 times and got 174, 232, 174 and 264 Hz — an 89 Hz spread with nothing changed. That is the same size as the "emotion differences" I was reporting, so those numbers were mostly random noise. It also explains what you heard: the male→female jumps, and "sad sounds happy". Voice Design designs a new voice from the prompt; it is not a stable speaker, so it is the wrong tool for an audiobook.

voice_clone is deterministic — the same input produced bit-identical audio twice over. So on this backend, any difference you hear is genuinely the emotion, not a re-roll. Character identity holds (measured timbre drift ≤ 0.011).

A. Emotion instruct alone — recommended

B. Same, but with the accent clause appended — emotion collapses

Two style directives in one instruct compete and the trailing one wins: the range squashes to 222–250 Hz and sad all but disappears. I've now dropped the accent clause whenever a line carries an emotion — a cloned voice already gets its accent from the reference WAV.

8. Different characters — you were right about the shouting

Marcian's reference clip is the culprit for the shouting: it is only 3.1 seconds long and sits at 253 Hz, which is shouting range for a male voice, with the narrowest dynamic range of any voice checked. Cloning reproduces that faithfully, so every Marcian line shouts no matter what emotion is requested.

But swapping to calm voices exposes the deeper problem. On these two the emotional pitch range is only 19–22 Hz — barely audible. Marcian's wider 69 Hz spread was mostly just his shouty baseline having room to fall, not better emotional control. So voice_clone keeps identity rock-steady but delivers only weak emotion.

DE_M_Darrag — calm 110 Hz reference

Ernst Meinecke — professional narrator, 21s reference

Where this leaves us: Fish-Speech → two registers only, everything shouts. Voice Design → strong changes but re-rolls the voice every call, so identity drifts and results are random. voice_clone → perfectly stable identity but only ~20 Hz of emotion. None of the three running backends does both. CustomVoice (port 8022) is down — per the Qwen3-TTS docs that is the variant built specifically for timbre control plus emotional style, and it is the one option not yet tested.

9. You were right — it's the model, not the settings

I checked what each container actually loads, and your hypothesis was exactly correct:

So my earlier "voice_clone responds to emotion" claim was wrong — that was a Base model being mildly perturbed by instruction text it does not understand, which is why the range was only 19–22 Hz. I have also confirmed on HuggingFace that no "Instruct" variant of Qwen3-TTS exists — the family is exactly these three.

CustomVoice, started and tested (speaker Ryan, German text)

This is the strongest real emotion control measured so far: 85 Hz range, with speaker identity holding (timbre drift ≤0.008) and sensible pacing (angry fast 4.08s, scared slow 6.16s). Repeat runs confirm the effect is genuine, not noise — sad lands at 111–136 Hz and angry at 159–210 Hz, ranges that never overlap.

The catch: CustomVoice exposes only 9 fixed speakers (Ryan, Aiden, Vivian, Serena, Uncle_Fu, Dylan, Eric…) and its API has no reference-audio field at all — it cannot clone. So it cannot voice the book's 57 cloned characters. Qwen3-TTS splits cloning and emotion across two models that cannot be combined: Base clones but ignores instructions, CustomVoice obeys instructions but cannot clone, VoiceDesign obeys instructions but invents a new voice on every call.

10. VoiceDesign vs CustomVoice — the consistency problem

Your plan (narrator on voice_clone, characters on VoiceDesign) is sound in structure — the narrator on voice_clone is exactly right: identity is rock-steady and he needs no emotion. But before committing 50 hours to VoiceDesign for the characters, listen to this.

Each block below is the same character, same line, same emotion, called three times. Nothing changes between runs. VoiceDesign has no seed parameter, so it cannot be pinned.

VoiceDesign — "happy" ×3 (identical requests)

VoiceDesign — "sad" ×3 (identical requests)

CustomVoice — "sad" ×3 (identical requests)

CustomVoice — "happy" ×3 (identical requests)

Averaged over 3 runs each, VoiceDesign's emotions do not separate reliably: neutral 136 Hz, sad 158, happy 231, angry 237 — but "happy" alone ranged 122–311 Hz across identical calls. The variation within one emotion is larger than the difference between emotions, and sad landed above neutral (backwards). Over ~2000 lines that means a character whose pitch wanders line to line.

CustomVoice's "sad" by contrast is tight (116–125 Hz, 9 Hz spread) — genuinely reliable. Its weakness is coverage: 9 fixed speakers, no cloning.

11. Fish-Speech — I was wrong, and it may be the best option

Correction: Fish-Speech is perfectly deterministic. The same request twice produced byte-identical audio (same MD5, 0 Hz spread) because the app pins a per-voice seed. That changes the picture completely:

And it is three registers, not two — I under-sold it earlier. Because output is deterministic, this map is exact rather than a sample. Emotion tags reliably land in one of three intensity levels:

Subdued — quiet, low

Neutral / warm — mid

Energized — loud, high

What this means for the book: Fish clones all 57 characters, holds every identity perfectly stable, and gives three dependable intensity levels — subdued for sad/whispered/resigned lines, neutral for ordinary narration, energized for angry/excited/frightened ones. What it will not do is distinguish angry from happy: both sit in the energized band (226 vs 203 Hz). The emotional contour of a scene comes through; the specific colour does not.

Determinism has a practical bonus too: a re-run reproduces identical audio, so an interrupted export resumes cleanly instead of drifting.

12. English voice + English bracket tags (as requested) — and an important correction

Voice: EN_F_Melina (best dynamic range of the English references, 10.4s clip). English text, English tags in square brackets before the line, per the S2 word-level-control article. I tested both plain single-word tags and the article's richer free-form style (e.g. its own example [voice rough from crying, trying to sound normal], and pairing a physical reaction with an emotion).

Simple single-word tags

Rich descriptive tags (article's recommended style)

Correction to section 11 — I over-claimed. Both tag styles above land within 183–197 Hz, i.e. essentially no emotional differentiation, and the richer tags were no better than the simple ones. Checking further, every other voice behaves the same way:

13. ★ How Fish-Speech S2 emotion ACTUALLY works

I investigated the mechanism instead of guessing. The finding explains every confusing result above:

Fish-Speech emotion is driven by the REFERENCE CLIP, not by the text tags.
When you supply reference audio, S2 does in-context prosody imitation — it copies the reference's delivery, and that overrides bracket tags almost entirely. Tags only work when there is no reference competing with them.

Evidence A — the same tags, with and without a reference

Evidence B — emotion transfers through the reference with NO tag at all

Identical plain sentence, no bracket tags anywhere. Only the reference clip changed:

The 5 samples you asked for — tags with no competing reference

These finally differentiate: 10× loudness spread, pitch 110–222 Hz, and genuine intonation differences (happy pstd 45 vs scared pstd 5). Caveat: with no reference the model also picks its own voice, so identity is not controlled here (speaker distance 0.027 vs 0.011 for a true clone).

What this means — and the way forward. The trade-off is inherent: a reference gives you the character's identity but suppresses tag-based emotion; no reference gives you emotion but an uncontrolled voice. But Evidence B points at a real solution: give each character several reference clips, one per emotional register (neutral, subdued, intense) and pick the matching one per line. That uses the mechanism instead of fighting it — delivering cloned identity and emotion, both fully deterministic.

14. ★ THE PROOF — same text, same tags, cloning on vs off

Your five-line example, generated twice through Fish Audio S2 Pro. Identical text, identical tags. The only difference is whether a cloned reference voice was supplied.

A. WITH a cloned voice (DE_F_Einfaches_Maedchen) — what you heard

Pitch across all five emotions: 241–268 Hz — a 27 Hz spread. Every line lands in the same place, which is exactly what you reported: "no difference between the lines".

B. WITHOUT a reference voice — same text, same tags

123 Hz spread and 3.2× loudness range — four to five times the differentiation, in a sensible order (happy highest and loudest, determined lowest and quietest). Because this is one single generation, the voice stays consistent throughout; the earlier "5 different voices" problem only happened when each emotion was generated as its own separate call.

This is the trade-off in one page: Fish gives you the cloned character voice, or the emotion tags — never both. The reference clip is prepended as an in-context example and the model imitates its delivery, which overrides the tags.

15. Controlling the voice WITHOUT a reference clip

If no reference clip is used, the voice can still be steered — by describing it in a tag. S2-Pro accepts free-form descriptions, so gender, age and character are controllable directly:

That is a 72–202 Hz range — clearly distinguishable men, women and children, all without any reference clip.

Reproducible: same description + same seed = identical audio

Three runs of the same description and seed produced byte-identical files (matching MD5). So a character can be pinned as "description + seed" and will sound the same every time — this fixes the "5 different voices" problem from section 13.

The catch: emotion tags also shift the voice

Same character description throughout, only the emotion tag changes. Emotion clearly works (174 Hz range, 6.4× loudness) — but the character drifts as well. "Identity drift" below compares timbre against the neutral take; a true cloned voice stays at 0.001–0.011, and roughly 0.03+ starts to read as a different person.

Please judge this by ear — do these five still sound like the same person, or like different speakers? If they hold together well enough, this is a workable path for the audiobook: define each character as a description plus a fixed seed, and emotion tags will actually function. If they do not, then cloned voices with neutral delivery remains the honest option.