README.md

@@ -186,39 +186,6 @@ CustomVoice uses the model's built-in speaker names. Define the speaker IDs you
 
 Then call the CustomVoice service on port `8022`.
 
-## Consistent characters for long-form narration
-
-For an audiobook a character must sound identical across thousands of requests while still carrying emotion. Those two goals live in different services, so use each for what it is good at.
-
-**VoiceDesign has no seed.** `generate_voice_design()` accepts none, so a designed voice's identity is re-sampled from its prompt on every call. Low `temperature`/`top_p` narrows the spread but does not remove it. Treat VoiceDesign as a casting tool, not a narration engine.
-
-**VoiceClone is deterministic.** Every request seeds the RNG — from `seed` in `voices.json`, or a stable hash of the voice name when none is set — and the `.pt` prompt is a frozen recording of one specific reference. The same text yields the same audio across restarts.
-
-The workflow:
-
-1. **Cast** the character at port `8021` until a take sounds right. Generate the same line several times; the identity you hear wandering is what step 3 eliminates.
-2. **Freeze** the take you liked: save ~10–20 s of clean audio as the character's reference WAV with an exact transcript as `ref_text`. The character is now a VoiceClone voice.
-3. **Bake** the `.pt`. `extract_embeddings.py` writes a full ICL prompt (reference codec tokens plus transcript), which both pins the identity and puts you in the mode where instructions are followed reliably.
-4. **Pin the seed.** Audition with `find_best_seed.py` and `/seed-samples/{voice}`, then lock the winner with `POST /voice-seed`.
-5. **Narrate** at port `8020`, passing per-line `instruct` for emotion.
-
-Once a character's reference audio is chosen, treat it as immutable. Re-recording it invalidates the `.pt` (see v6.8) and shifts the identity, which means re-auditioning the seed and re-rendering everything already produced.
-
-### Directing a line
-
-`instruct` on a request is a *direction*, not a description. It is appended to the voice's own `instruct`, so keep the character in the voice and only the delivery in the request:
-
-```bash
-curl http://localhost:8020/v1/audio/speech \
-  -H "Content-Type: application/json" \
-  -d '{"input":"You should not have come here.",
-       "voice":"DE_M_Zerwas",
-       "instruct":"quietly, through gritted teeth"}' \
-  --output line_0417.wav
-```
-
-Write `"whispering, barely audible"`, not `"a gravelly old man, whispering"` — the second re-describes the character and invites drift. Cast on a calm, level reference: a reference read with strong emotion bakes that emotion into the identity and fights every later direction. Test each character's most extreme line early; if a direction breaks the identity, the reference is too short or too expressive.
-
 ## Streaming backend
 
 The streaming service on port `8023` uses the same generated `config/voices.json` and active VoiceClone reference voices as port `8020`, but returns WAV chunks while generation is still running. Use it when time-to-first-audio matters more than waiting for the complete WAV response.
@@ -246,7 +213,7 @@ The streaming service on port `8023` uses the same generated `config/voices.json
 | `response_format` | string | `wav` | `wav`, `pcm`, `mp3`, or `zip` (for timestamps) |
 | `speed` | float | 1.0 | Scales audio tempo via ffmpeg |
 | `language` | string | voice config | Per-request override for VoiceDesign/CustomVoice |
-| `instruct` | string | voice config | Per-line delivery direction. Appended to the voice's own `instruct`, never replacing it. Supported by all three services |
+| `instruct` | string | voice config | Per-request style override for VoiceDesign/CustomVoice |
 | `max_new_tokens` | int | server default | Per-request generation length override |
 
 WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response.
@@ -377,19 +344,6 @@ The first request after container startup can be slower because CUDA graph captu
 
 ## Changelog
 
-### v6.10 — 2026-08-18
-**Fix: Long Text No Longer Truncated Mid-Sentence**
-- **Root Cause:** `/v1/audio/speech` sent the entire request text as one generate call, capped at `--max-seq-len` codec tokens. Text long enough to need more audio than that cap allowed got cut off wherever generation happened to be — mid-sentence, mid-word — with no error surfaced to the client.
-- **Automatic Segmentation:** Long input is now split into paragraph-sized segments (falling back to sentence boundaries for a paragraph that alone exceeds the safe budget) and generated back-to-back, stitched into one continuous stream or file. Applies to both the streaming (wav/pcm) and non-streaming (mp3/zip) paths.
-- **Why This Beats Raising `--max-seq-len`:** The CUDA graph static cache sized by `--max-seq-len` is reserved VRAM for the life of the container, whether or not any request needs it. Segmenting text keeps that cache small — `--max-seq-len` can be set to fit your VRAM budget instead of your longest expected input — while still supporting arbitrarily long text.
-
-### v6.9 — 2026-07-29
-**Feature: Per-Line Emotional Direction on Cloned Voices**
-- **Request-Level `instruct` for VoiceClone:** The clone server previously read `instruct` only from `voices.json`, so a character could have one fixed delivery for an entire book. `SpeechRequest` now accepts `instruct`, applied to both the streaming and non-streaming paths.
-- **Identity-First Merging:** A per-request `instruct` is appended to the voice's own description rather than replacing it, so directing one line's emotion cannot re-roll the character. Matches the behaviour added to the VoiceDesign server in v6.8.
-- **Why It Works Here:** Qwen3-TTS follows instructions reliably only in ICL mode, not with x-vector-only cloning. The `.pt` prompts written by `extract_embeddings.py` are full ICL prompts, so cloned voices qualify.
-- **Documentation:** New "Consistent characters for long-form narration" section covering the cast-in-VoiceDesign, narrate-from-VoiceClone workflow for audiobooks.
-
 ### v6.8 — 2026-07-29
 **Fix: Stale Speaker Embeddings and Unregistered Voice Design Voices**
 - **Stale `.pt` Embeddings:** A speaker embedding is a cache baked from one specific pairing of reference audio and transcript. Re-recording a voice replaced those sources but left the old `.pt` in place, so the server kept cloning from an embedding whose audio tokens no longer matched its transcript — generation ignored the requested text and emitted unrelated filler. `generate_voices.py` now treats an embedding older than its reference audio or transcript as absent and regenerates it.

patches/openai_server.patch

@@ -1,8 +1,8 @@
 diff --git a/examples/openai_server.py b/examples/openai_server.py
-index 2199e14..05151ce 100644
+index 2199e14..d10065f 100644
 --- a/examples/openai_server.py
 +++ b/examples/openai_server.py
-@@ -36,11 +36,13 @@ API usage:
+@@ -36,6 +36,7 @@ API usage:
  """
  import argparse
  import asyncio
@@ -10,13 +10,7 @@ index 2199e14..05151ce 100644
  import io
  import json
  import logging
- import os
- import queue
-+import re
- import struct
- import sys
- import threading
-@@ -66,9 +68,67 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API")
+@@ -66,9 +67,46 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API")
  
  tts_model = None
  voices: dict = {}
@@ -24,7 +18,6 @@ index 2199e14..05151ce 100644
 +last_voices_mtime: float = 0.0
  default_voice: Optional[str] = None
  SAMPLE_RATE = 24000  # updated once the model loads
-+MAX_SEQ_LEN = 4096  # CUDA graph static cache size, updated from --max-seq-len
  _model_lock = threading.Lock()  # prevent concurrent GPU inference
 +aligner_model = None
 +
@@ -60,31 +53,11 @@ index 2199e14..05151ce 100644
 +    torch.manual_seed(seed)
 +    if torch.cuda.is_available():
 +        torch.cuda.manual_seed_all(seed)
-+
-+
-+def _merge_instruct(voice_cfg: dict, req_instruct: Optional[str]) -> Optional[str]:
-+    """Combine a voice's identity instruct with a per-request delivery direction.
-+
-+    A voice's instruct in voices.json describes WHO is speaking; an instruct
-+    sent with the request describes HOW this one line is delivered. Replacing
-+    the former with the latter would re-roll the character from a few words of
-+    direction, so combine them, identity first — the same rule the VoiceDesign
-+    server applies.
-+
-+    Instructions are only followed reliably in ICL mode. The .pt prompts written
-+    by extract_embeddings.py are full ICL prompts (ref codec tokens + ref_text),
-+    not x-vector-only, so this path qualifies.
-+    """
-+    base = str(voice_cfg.get("instruct") or "").strip()
-+    line = str(req_instruct or "").strip()
-+    if base and line and line != base:
-+        return f"{base} {line}"
-+    return line or base or None
 +
  
  # ---------------------------------------------------------------------------
  # Request / response models
-@@ -79,8 +139,60 @@ class SpeechRequest(BaseModel):
+@@ -79,8 +117,8 @@ class SpeechRequest(BaseModel):
      model: str = "tts-1"
      input: str
      voice: str = "alloy"
@@ -92,62 +65,10 @@ index 2199e14..05151ce 100644
 -    speed: float = 1.0           # accepted but not yet applied
 +    response_format: str = "wav"  # wav | pcm | mp3 | zip
 +    speed: float = 1.0           # scales audio tempo
-+    instruct: Optional[str] = None  # per-line delivery direction, see _merge_instruct
-+
-+
-+# ---------------------------------------------------------------------------
-+# Text chunking for long input
-+#
-+# A single generate call is capped at MAX_SEQ_LEN codec tokens (the CUDA
-+# graph's static cache size). Rather than raising that cache to cover the
-+# longest text anyone might send — which burns VRAM permanently for every
-+# request — long input is split into paragraph/sentence-sized segments that
-+# each fit comfortably under the cap, generated back-to-back, and stitched
-+# into one continuous stream/file.
-+# ---------------------------------------------------------------------------
-+
-+_CODEC_HZ = 12          # talker codec frame rate (generated audio tokens/sec)
-+_CHARS_PER_SEC = 15.0   # conservative estimate of spoken characters/sec
-+_SEQ_LEN_SAFETY = 0.6   # stay well under the hard cap to leave room for slower speech
-+_SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+")
-+
-+
-+def _max_chars_per_segment() -> int:
-+    budget_seconds = (MAX_SEQ_LEN * _SEQ_LEN_SAFETY) / _CODEC_HZ
-+    return max(200, int(budget_seconds * _CHARS_PER_SEC))
-+
-+
-+def _split_text_for_generation(text: str, max_chars: int) -> list:
-+    """Split text into segments that each fit under max_chars.
-+
-+    Splits on paragraph breaks first (kept as separate segments so natural
-+    pauses land there), then falls back to sentence boundaries for any
-+    paragraph that alone exceeds the budget.
-+    """
-+    paragraphs = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
-+    if not paragraphs:
-+        return [text]
-+
-+    segments = []
-+    for para in paragraphs:
-+        if len(para) <= max_chars:
-+            segments.append(para)
-+            continue
-+        buf = ""
-+        for sent in _SENTENCE_SPLIT_RE.split(para):
-+            candidate = f"{buf} {sent}".strip() if buf else sent
-+            if len(candidate) > max_chars and buf:
-+                segments.append(buf)
-+                buf = sent
-+            else:
-+                buf = candidate
-+        if buf:
-+            segments.append(buf)
-+    return segments or [text]
  
  
  # ---------------------------------------------------------------------------
-@@ -145,6 +257,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
+@@ -145,6 +183,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
  
  def resolve_voice(voice_name: str) -> dict:
      """Return voice config dict or fall back to default, else raise 400."""
@@ -169,7 +90,7 @@ index 2199e14..05151ce 100644
      if voice_name in voices:
          return voices[voice_name]
      if default_voice and default_voice in voices:
-@@ -168,7 +295,47 @@ def resolve_voice(voice_name: str) -> dict:
+@@ -168,7 +221,47 @@ def resolve_voice(voice_name: str) -> dict:
  # ---------------------------------------------------------------------------
  
  
@@ -214,11 +135,11 @@ index 2199e14..05151ce 100644
 +        return None
 +
 +
-+async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float, instruct: Optional[str] = None) -> AsyncGenerator[bytes, None]:
++async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float) -> AsyncGenerator[bytes, None]:
      """
      Run generate_voice_clone_streaming in a background thread and yield
      raw PCM bytes for each chunk as they arrive.
-@@ -177,21 +344,65 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
+@@ -177,21 +270,63 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
      _DONE = object()
  
      def producer():
@@ -249,37 +170,29 @@ index 2199e14..05151ce 100644
 +
          try:
              with _model_lock:
--                for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
--                    text=text,
--                    language=voice_cfg.get("language", "Auto"),
--                    ref_audio=voice_cfg["ref_audio"],
--                    ref_text=voice_cfg.get("ref_text", ""),
--                    chunk_size=voice_cfg.get("chunk_size", 12),
--                    non_streaming_mode=False,
--                ):
--                    q.put(chunk)
 +                _seed_rng(voice_cfg.get("seed", _voice_seed(voice_name)))
-+                segments = _split_text_for_generation(text, _max_chars_per_segment())
-+                for seg in segments:
-+                    for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
-+                        text=seg,
-+                        language=voice_cfg.get("language", "Auto"),
-+                        ref_audio=voice_cfg.get("ref_audio"),
-+                        ref_text=voice_cfg.get("ref_text", ""),
-+                        chunk_size=voice_cfg.get("chunk_size", 12),
-+                        instruct=_merge_instruct(voice_cfg, instruct),
-+                        voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
-+                        non_streaming_mode=True,
-+                        temperature=voice_cfg.get("temperature", 0.8),
-+                        top_k=voice_cfg.get("top_k", 50),
-+                        top_p=voice_cfg.get("top_p", 0.9),
-+                    ):
-+                        raw = _to_pcm16(chunk)
-+                        if process:
-+                            process.stdin.write(raw)
-+                            process.stdin.flush()
-+                        else:
-+                            q.put(raw)
+                 for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
+                     text=text,
+                     language=voice_cfg.get("language", "Auto"),
+-                    ref_audio=voice_cfg["ref_audio"],
++                    ref_audio=voice_cfg.get("ref_audio"),
+                     ref_text=voice_cfg.get("ref_text", ""),
+                     chunk_size=voice_cfg.get("chunk_size", 12),
+-                    non_streaming_mode=False,
++                    instruct=voice_cfg.get("instruct"),
++                    voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
++                    non_streaming_mode=True,
++                    temperature=voice_cfg.get("temperature", 0.8),
++                    top_k=voice_cfg.get("top_k", 50),
++                    top_p=voice_cfg.get("top_p", 0.9),
+                 ):
+-                    q.put(chunk)
++                    raw = _to_pcm16(chunk)
++                    if process:
++                        process.stdin.write(raw)
++                        process.stdin.flush()
++                    else:
++                        q.put(raw)
          except Exception as exc:
              q.put(exc)
          finally:
@@ -294,7 +207,7 @@ index 2199e14..05151ce 100644
  
      thread = threading.Thread(target=producer, daemon=True)
      thread.start()
-@@ -203,7 +414,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
+@@ -203,7 +338,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
              break
          if isinstance(item, Exception):
              raise item
@@ -303,7 +216,7 @@ index 2199e14..05151ce 100644
  
  
  # ---------------------------------------------------------------------------
-@@ -220,7 +431,8 @@ async def health():
+@@ -220,7 +355,8 @@ async def health():
  async def create_speech(req: SpeechRequest):
      if tts_model is None:
          raise HTTPException(status_code=503, detail="Model not loaded")
@@ -313,7 +226,7 @@ index 2199e14..05151ce 100644
          raise HTTPException(status_code=400, detail="'input' text is empty")
  
      voice_cfg = resolve_voice(req.voice)
-@@ -230,36 +442,84 @@ async def create_speech(req: SpeechRequest):
+@@ -230,36 +366,77 @@ async def create_speech(req: SpeechRequest):
          "wav": "audio/wav",
          "pcm": "audio/pcm",
          "mp3": "audio/mpeg",
@@ -335,38 +248,23 @@ index 2199e14..05151ce 100644
  
          def _generate():
              with _model_lock:
--                return tts_model.generate_voice_clone(
--                    text=req.input,
--                    language=voice_cfg.get("language", "Auto"),
--                    ref_audio=voice_cfg["ref_audio"],
--                    ref_text=voice_cfg.get("ref_text", ""),
--                )
--
--        audio_arrays, sr = await loop.run_in_executor(None, _generate)
--        audio = audio_arrays[0] if audio_arrays else np.zeros(1, dtype=np.float32)
 +                _seed_rng(voice_cfg.get("seed", _voice_seed(req.voice)))
-+                segments = _split_text_for_generation(req.input, _max_chars_per_segment())
-+                pieces = []
-+                sr = SAMPLE_RATE
-+                for seg in segments:
-+                    audio_arrays, sr = tts_model.generate_voice_clone(
-+                        text=seg,
-+                        language=voice_cfg.get("language", "Auto"),
-+                        ref_audio=voice_cfg.get("ref_audio"),
-+                        ref_text=voice_cfg.get("ref_text", ""),
-+                        instruct=_merge_instruct(voice_cfg, req.instruct),
-+                        voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model),
-+                        non_streaming_mode=True,
-+                        temperature=voice_cfg.get("temperature", 0.8),
-+                        top_k=voice_cfg.get("top_k", 50),
-+                        top_p=voice_cfg.get("top_p", 0.9),
-+                    )
-+                    if audio_arrays:
-+                        pieces.append(audio_arrays[0])
-+                return pieces, sr
-+
-+        pieces, sr = await loop.run_in_executor(None, _generate)
-+        audio = np.concatenate(pieces) if pieces else np.zeros(1, dtype=np.float32)
+                 return tts_model.generate_voice_clone(
+                     text=req.input,
+                     language=voice_cfg.get("language", "Auto"),
+-                    ref_audio=voice_cfg["ref_audio"],
++                    ref_audio=voice_cfg.get("ref_audio"),
+                     ref_text=voice_cfg.get("ref_text", ""),
++                    instruct=voice_cfg.get("instruct"),
++                    voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model),
++                    non_streaming_mode=True,
++                    temperature=voice_cfg.get("temperature", 0.8),
++                    top_k=voice_cfg.get("top_k", 50),
++                    top_p=voice_cfg.get("top_p", 0.9),
+                 )
+ 
+         audio_arrays, sr = await loop.run_in_executor(None, _generate)
+         audio = audio_arrays[0] if audio_arrays else np.zeros(1, dtype=np.float32)
 +        
 +        if req.speed != 1.0:
 +            import subprocess
@@ -407,11 +305,11 @@ index 2199e14..05151ce 100644
          if fmt == "wav":
              yield _wav_header(SAMPLE_RATE)  # stream with unknown data length
 -        async for raw_chunk in _stream_chunks(voice_cfg, req.input):
-+        async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed, req.instruct):
++        async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed):
              yield raw_chunk
  
      return StreamingResponse(audio_stream(), media_type=content_type)
-@@ -306,16 +566,21 @@ def _parse_args():
+@@ -306,16 +483,20 @@ def _parse_args():
      p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)")
      p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)")
      p.add_argument("--device", default="cuda", help="Torch device (default: cuda)")
@@ -421,10 +319,9 @@ index 2199e14..05151ce 100644
  
  def main():
 -    global tts_model, voices, default_voice, SAMPLE_RATE
-+    global tts_model, voices, voices_file_path, last_voices_mtime, default_voice, SAMPLE_RATE, MAX_SEQ_LEN
++    global tts_model, voices, voices_file_path, last_voices_mtime, default_voice, SAMPLE_RATE
  
      args = _parse_args()
-+    MAX_SEQ_LEN = args.max_seq_len
  
      # Build voice registry
      if args.voices:
@@ -434,7 +331,7 @@ index 2199e14..05151ce 100644
          with open(args.voices) as f:
              voices = json.load(f)
          default_voice = next(iter(voices))
-@@ -344,6 +609,7 @@ def main():
+@@ -344,6 +525,7 @@ def main():
          args.model,
          device=args.device,
          dtype=torch.bfloat16,