v6.10: Auto-segment long text to fix mid-sentence truncation

A single generate call was capped at --max-seq-len codec tokens, so text
long enough to need more audio than that got cut off wherever generation
happened to be. Long input is now split into paragraph/sentence-sized
segments and generated back-to-back, so --max-seq-len can be sized for
VRAM budget instead of the longest expected input.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

README.md

@@ -186,6 +186,39 @@ CustomVoice uses the model's built-in speaker names. Define the speaker IDs you
 
 Then call the CustomVoice service on port `8022`.
 
+## Consistent characters for long-form narration
+
+For an audiobook a character must sound identical across thousands of requests while still carrying emotion. Those two goals live in different services, so use each for what it is good at.
+
+**VoiceDesign has no seed.** `generate_voice_design()` accepts none, so a designed voice's identity is re-sampled from its prompt on every call. Low `temperature`/`top_p` narrows the spread but does not remove it. Treat VoiceDesign as a casting tool, not a narration engine.
+
+**VoiceClone is deterministic.** Every request seeds the RNG — from `seed` in `voices.json`, or a stable hash of the voice name when none is set — and the `.pt` prompt is a frozen recording of one specific reference. The same text yields the same audio across restarts.
+
+The workflow:
+
+1. **Cast** the character at port `8021` until a take sounds right. Generate the same line several times; the identity you hear wandering is what step 3 eliminates.
+2. **Freeze** the take you liked: save ~10–20 s of clean audio as the character's reference WAV with an exact transcript as `ref_text`. The character is now a VoiceClone voice.
+3. **Bake** the `.pt`. `extract_embeddings.py` writes a full ICL prompt (reference codec tokens plus transcript), which both pins the identity and puts you in the mode where instructions are followed reliably.
+4. **Pin the seed.** Audition with `find_best_seed.py` and `/seed-samples/{voice}`, then lock the winner with `POST /voice-seed`.
+5. **Narrate** at port `8020`, passing per-line `instruct` for emotion.
+
+Once a character's reference audio is chosen, treat it as immutable. Re-recording it invalidates the `.pt` (see v6.8) and shifts the identity, which means re-auditioning the seed and re-rendering everything already produced.
+
+### Directing a line
+
+`instruct` on a request is a *direction*, not a description. It is appended to the voice's own `instruct`, so keep the character in the voice and only the delivery in the request:
+
+```bash
+curl http://localhost:8020/v1/audio/speech \
+  -H "Content-Type: application/json" \
+  -d '{"input":"You should not have come here.",
+       "voice":"DE_M_Zerwas",
+       "instruct":"quietly, through gritted teeth"}' \
+  --output line_0417.wav
+```
+
+Write `"whispering, barely audible"`, not `"a gravelly old man, whispering"` — the second re-describes the character and invites drift. Cast on a calm, level reference: a reference read with strong emotion bakes that emotion into the identity and fights every later direction. Test each character's most extreme line early; if a direction breaks the identity, the reference is too short or too expressive.
+
 ## Streaming backend
 
 The streaming service on port `8023` uses the same generated `config/voices.json` and active VoiceClone reference voices as port `8020`, but returns WAV chunks while generation is still running. Use it when time-to-first-audio matters more than waiting for the complete WAV response.
@@ -213,7 +246,7 @@ The streaming service on port `8023` uses the same generated `config/voices.json
 | `response_format` | string | `wav` | `wav`, `pcm`, `mp3`, or `zip` (for timestamps) |
 | `speed` | float | 1.0 | Scales audio tempo via ffmpeg |
 | `language` | string | voice config | Per-request override for VoiceDesign/CustomVoice |
-| `instruct` | string | voice config | Per-request style override for VoiceDesign/CustomVoice |
+| `instruct` | string | voice config | Per-line delivery direction. Appended to the voice's own `instruct`, never replacing it. Supported by all three services |
 | `max_new_tokens` | int | server default | Per-request generation length override |
 
 WAV and PCM are streamed as audio is generated. MP3 is encoded after generation and returned as a complete response.
@@ -344,6 +377,19 @@ The first request after container startup can be slower because CUDA graph captu
 
 ## Changelog
 
+### v6.10 — 2026-08-18
+**Fix: Long Text No Longer Truncated Mid-Sentence**
+- **Root Cause:** `/v1/audio/speech` sent the entire request text as one generate call, capped at `--max-seq-len` codec tokens. Text long enough to need more audio than that cap allowed got cut off wherever generation happened to be — mid-sentence, mid-word — with no error surfaced to the client.
+- **Automatic Segmentation:** Long input is now split into paragraph-sized segments (falling back to sentence boundaries for a paragraph that alone exceeds the safe budget) and generated back-to-back, stitched into one continuous stream or file. Applies to both the streaming (wav/pcm) and non-streaming (mp3/zip) paths.
+- **Why This Beats Raising `--max-seq-len`:** The CUDA graph static cache sized by `--max-seq-len` is reserved VRAM for the life of the container, whether or not any request needs it. Segmenting text keeps that cache small — `--max-seq-len` can be set to fit your VRAM budget instead of your longest expected input — while still supporting arbitrarily long text.
+
+### v6.9 — 2026-07-29
+**Feature: Per-Line Emotional Direction on Cloned Voices**
+- **Request-Level `instruct` for VoiceClone:** The clone server previously read `instruct` only from `voices.json`, so a character could have one fixed delivery for an entire book. `SpeechRequest` now accepts `instruct`, applied to both the streaming and non-streaming paths.
+- **Identity-First Merging:** A per-request `instruct` is appended to the voice's own description rather than replacing it, so directing one line's emotion cannot re-roll the character. Matches the behaviour added to the VoiceDesign server in v6.8.
+- **Why It Works Here:** Qwen3-TTS follows instructions reliably only in ICL mode, not with x-vector-only cloning. The `.pt` prompts written by `extract_embeddings.py` are full ICL prompts, so cloned voices qualify.
+- **Documentation:** New "Consistent characters for long-form narration" section covering the cast-in-VoiceDesign, narrate-from-VoiceClone workflow for audiobooks.
+
 ### v6.8 — 2026-07-29
 **Fix: Stale Speaker Embeddings and Unregistered Voice Design Voices**
 - **Stale `.pt` Embeddings:** A speaker embedding is a cache baked from one specific pairing of reference audio and transcript. Re-recording a voice replaced those sources but left the old `.pt` in place, so the server kept cloning from an embedding whose audio tokens no longer matched its transcript — generation ignored the requested text and emitted unrelated filler. `generate_voices.py` now treats an embedding older than its reference audio or transcript as absent and regenerates it.

patches/openai_server.patch

@@ -1,8 +1,8 @@
 diff --git a/examples/openai_server.py b/examples/openai_server.py
-index 2199e14..d10065f 100644
+index 2199e14..05151ce 100644
 --- a/examples/openai_server.py
 +++ b/examples/openai_server.py
-@@ -36,6 +36,7 @@ API usage:
+@@ -36,11 +36,13 @@ API usage:
  """
  import argparse
  import asyncio
@@ -10,7 +10,13 @@ index 2199e14..d10065f 100644
  import io
  import json
  import logging
-@@ -66,9 +67,46 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API")
+ import os
+ import queue
++import re
+ import struct
+ import sys
+ import threading
+@@ -66,9 +68,67 @@ app = FastAPI(title="faster-qwen3-tts OpenAI-compatible API")
  
  tts_model = None
  voices: dict = {}
@@ -18,6 +24,7 @@ index 2199e14..d10065f 100644
 +last_voices_mtime: float = 0.0
  default_voice: Optional[str] = None
  SAMPLE_RATE = 24000  # updated once the model loads
++MAX_SEQ_LEN = 4096  # CUDA graph static cache size, updated from --max-seq-len
  _model_lock = threading.Lock()  # prevent concurrent GPU inference
 +aligner_model = None
 +
@@ -53,11 +60,31 @@ index 2199e14..d10065f 100644
 +    torch.manual_seed(seed)
 +    if torch.cuda.is_available():
 +        torch.cuda.manual_seed_all(seed)
++
++
++def _merge_instruct(voice_cfg: dict, req_instruct: Optional[str]) -> Optional[str]:
++    """Combine a voice's identity instruct with a per-request delivery direction.
++
++    A voice's instruct in voices.json describes WHO is speaking; an instruct
++    sent with the request describes HOW this one line is delivered. Replacing
++    the former with the latter would re-roll the character from a few words of
++    direction, so combine them, identity first — the same rule the VoiceDesign
++    server applies.
++
++    Instructions are only followed reliably in ICL mode. The .pt prompts written
++    by extract_embeddings.py are full ICL prompts (ref codec tokens + ref_text),
++    not x-vector-only, so this path qualifies.
++    """
++    base = str(voice_cfg.get("instruct") or "").strip()
++    line = str(req_instruct or "").strip()
++    if base and line and line != base:
++        return f"{base} {line}"
++    return line or base or None
 +
  
  # ---------------------------------------------------------------------------
  # Request / response models
-@@ -79,8 +117,8 @@ class SpeechRequest(BaseModel):
+@@ -79,8 +139,60 @@ class SpeechRequest(BaseModel):
      model: str = "tts-1"
      input: str
      voice: str = "alloy"
@@ -65,10 +92,62 @@ index 2199e14..d10065f 100644
 -    speed: float = 1.0           # accepted but not yet applied
 +    response_format: str = "wav"  # wav | pcm | mp3 | zip
 +    speed: float = 1.0           # scales audio tempo
++    instruct: Optional[str] = None  # per-line delivery direction, see _merge_instruct
++
++
++# ---------------------------------------------------------------------------
++# Text chunking for long input
++#
++# A single generate call is capped at MAX_SEQ_LEN codec tokens (the CUDA
++# graph's static cache size). Rather than raising that cache to cover the
++# longest text anyone might send — which burns VRAM permanently for every
++# request — long input is split into paragraph/sentence-sized segments that
++# each fit comfortably under the cap, generated back-to-back, and stitched
++# into one continuous stream/file.
++# ---------------------------------------------------------------------------
++
++_CODEC_HZ = 12          # talker codec frame rate (generated audio tokens/sec)
++_CHARS_PER_SEC = 15.0   # conservative estimate of spoken characters/sec
++_SEQ_LEN_SAFETY = 0.6   # stay well under the hard cap to leave room for slower speech
++_SENTENCE_SPLIT_RE = re.compile(r"(?<=[.!?])\s+")
++
++
++def _max_chars_per_segment() -> int:
++    budget_seconds = (MAX_SEQ_LEN * _SEQ_LEN_SAFETY) / _CODEC_HZ
++    return max(200, int(budget_seconds * _CHARS_PER_SEC))
++
++
++def _split_text_for_generation(text: str, max_chars: int) -> list:
++    """Split text into segments that each fit under max_chars.
++
++    Splits on paragraph breaks first (kept as separate segments so natural
++    pauses land there), then falls back to sentence boundaries for any
++    paragraph that alone exceeds the budget.
++    """
++    paragraphs = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
++    if not paragraphs:
++        return [text]
++
++    segments = []
++    for para in paragraphs:
++        if len(para) <= max_chars:
++            segments.append(para)
++            continue
++        buf = ""
++        for sent in _SENTENCE_SPLIT_RE.split(para):
++            candidate = f"{buf} {sent}".strip() if buf else sent
++            if len(candidate) > max_chars and buf:
++                segments.append(buf)
++                buf = sent
++            else:
++                buf = candidate
++        if buf:
++            segments.append(buf)
++    return segments or [text]
  
  
  # ---------------------------------------------------------------------------
-@@ -145,6 +183,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
+@@ -145,6 +257,21 @@ def _to_mp3_bytes(pcm: np.ndarray, sample_rate: int) -> bytes:
  
  def resolve_voice(voice_name: str) -> dict:
      """Return voice config dict or fall back to default, else raise 400."""
@@ -90,7 +169,7 @@ index 2199e14..d10065f 100644
      if voice_name in voices:
          return voices[voice_name]
      if default_voice and default_voice in voices:
-@@ -168,7 +221,47 @@ def resolve_voice(voice_name: str) -> dict:
+@@ -168,7 +295,47 @@ def resolve_voice(voice_name: str) -> dict:
  # ---------------------------------------------------------------------------
  
  
@@ -135,11 +214,11 @@ index 2199e14..d10065f 100644
 +        return None
 +
 +
-+async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float) -> AsyncGenerator[bytes, None]:
++async def _stream_chunks(voice_cfg: dict, text: str, voice_name: str, speed: float, instruct: Optional[str] = None) -> AsyncGenerator[bytes, None]:
      """
      Run generate_voice_clone_streaming in a background thread and yield
      raw PCM bytes for each chunk as they arrive.
-@@ -177,21 +270,63 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
+@@ -177,21 +344,65 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
      _DONE = object()
  
      def producer():
@@ -170,29 +249,37 @@ index 2199e14..d10065f 100644
 +
          try:
              with _model_lock:
-+                _seed_rng(voice_cfg.get("seed", _voice_seed(voice_name)))
-                 for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
-                     text=text,
-                     language=voice_cfg.get("language", "Auto"),
+-                for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
+-                    text=text,
+-                    language=voice_cfg.get("language", "Auto"),
 -                    ref_audio=voice_cfg["ref_audio"],
-+                    ref_audio=voice_cfg.get("ref_audio"),
-                     ref_text=voice_cfg.get("ref_text", ""),
-                     chunk_size=voice_cfg.get("chunk_size", 12),
+-                    ref_text=voice_cfg.get("ref_text", ""),
+-                    chunk_size=voice_cfg.get("chunk_size", 12),
 -                    non_streaming_mode=False,
-+                    instruct=voice_cfg.get("instruct"),
-+                    voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
-+                    non_streaming_mode=True,
-+                    temperature=voice_cfg.get("temperature", 0.8),
-+                    top_k=voice_cfg.get("top_k", 50),
-+                    top_p=voice_cfg.get("top_p", 0.9),
-                 ):
+-                ):
 -                    q.put(chunk)
-+                    raw = _to_pcm16(chunk)
-+                    if process:
-+                        process.stdin.write(raw)
-+                        process.stdin.flush()
-+                    else:
-+                        q.put(raw)
++                _seed_rng(voice_cfg.get("seed", _voice_seed(voice_name)))
++                segments = _split_text_for_generation(text, _max_chars_per_segment())
++                for seg in segments:
++                    for chunk, _sr, _timing in tts_model.generate_voice_clone_streaming(
++                        text=seg,
++                        language=voice_cfg.get("language", "Auto"),
++                        ref_audio=voice_cfg.get("ref_audio"),
++                        ref_text=voice_cfg.get("ref_text", ""),
++                        chunk_size=voice_cfg.get("chunk_size", 12),
++                        instruct=_merge_instruct(voice_cfg, instruct),
++                        voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, voice_name, tts_model),
++                        non_streaming_mode=True,
++                        temperature=voice_cfg.get("temperature", 0.8),
++                        top_k=voice_cfg.get("top_k", 50),
++                        top_p=voice_cfg.get("top_p", 0.9),
++                    ):
++                        raw = _to_pcm16(chunk)
++                        if process:
++                            process.stdin.write(raw)
++                            process.stdin.flush()
++                        else:
++                            q.put(raw)
          except Exception as exc:
              q.put(exc)
          finally:
@@ -207,7 +294,7 @@ index 2199e14..d10065f 100644
  
      thread = threading.Thread(target=producer, daemon=True)
      thread.start()
-@@ -203,7 +338,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
+@@ -203,7 +414,7 @@ async def _stream_chunks(voice_cfg: dict, text: str) -> AsyncGenerator[bytes, No
              break
          if isinstance(item, Exception):
              raise item
@@ -216,7 +303,7 @@ index 2199e14..d10065f 100644
  
  
  # ---------------------------------------------------------------------------
-@@ -220,7 +355,8 @@ async def health():
+@@ -220,7 +431,8 @@ async def health():
  async def create_speech(req: SpeechRequest):
      if tts_model is None:
          raise HTTPException(status_code=503, detail="Model not loaded")
@@ -226,7 +313,7 @@ index 2199e14..d10065f 100644
          raise HTTPException(status_code=400, detail="'input' text is empty")
  
      voice_cfg = resolve_voice(req.voice)
-@@ -230,36 +366,77 @@ async def create_speech(req: SpeechRequest):
+@@ -230,36 +442,84 @@ async def create_speech(req: SpeechRequest):
          "wav": "audio/wav",
          "pcm": "audio/pcm",
          "mp3": "audio/mpeg",
@@ -248,23 +335,38 @@ index 2199e14..d10065f 100644
  
          def _generate():
              with _model_lock:
-+                _seed_rng(voice_cfg.get("seed", _voice_seed(req.voice)))
-                 return tts_model.generate_voice_clone(
-                     text=req.input,
-                     language=voice_cfg.get("language", "Auto"),
+-                return tts_model.generate_voice_clone(
+-                    text=req.input,
+-                    language=voice_cfg.get("language", "Auto"),
 -                    ref_audio=voice_cfg["ref_audio"],
-+                    ref_audio=voice_cfg.get("ref_audio"),
-                     ref_text=voice_cfg.get("ref_text", ""),
-+                    instruct=voice_cfg.get("instruct"),
-+                    voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model),
-+                    non_streaming_mode=True,
-+                    temperature=voice_cfg.get("temperature", 0.8),
-+                    top_k=voice_cfg.get("top_k", 50),
-+                    top_p=voice_cfg.get("top_p", 0.9),
-                 )
- 
-         audio_arrays, sr = await loop.run_in_executor(None, _generate)
-         audio = audio_arrays[0] if audio_arrays else np.zeros(1, dtype=np.float32)
+-                    ref_text=voice_cfg.get("ref_text", ""),
+-                )
+-
+-        audio_arrays, sr = await loop.run_in_executor(None, _generate)
+-        audio = audio_arrays[0] if audio_arrays else np.zeros(1, dtype=np.float32)
++                _seed_rng(voice_cfg.get("seed", _voice_seed(req.voice)))
++                segments = _split_text_for_generation(req.input, _max_chars_per_segment())
++                pieces = []
++                sr = SAMPLE_RATE
++                for seg in segments:
++                    audio_arrays, sr = tts_model.generate_voice_clone(
++                        text=seg,
++                        language=voice_cfg.get("language", "Auto"),
++                        ref_audio=voice_cfg.get("ref_audio"),
++                        ref_text=voice_cfg.get("ref_text", ""),
++                        instruct=_merge_instruct(voice_cfg, req.instruct),
++                        voice_clone_prompt=_load_voice_clone_prompt(voice_cfg, req.voice, tts_model),
++                        non_streaming_mode=True,
++                        temperature=voice_cfg.get("temperature", 0.8),
++                        top_k=voice_cfg.get("top_k", 50),
++                        top_p=voice_cfg.get("top_p", 0.9),
++                    )
++                    if audio_arrays:
++                        pieces.append(audio_arrays[0])
++                return pieces, sr
++
++        pieces, sr = await loop.run_in_executor(None, _generate)
++        audio = np.concatenate(pieces) if pieces else np.zeros(1, dtype=np.float32)
 +        
 +        if req.speed != 1.0:
 +            import subprocess
@@ -305,11 +407,11 @@ index 2199e14..d10065f 100644
          if fmt == "wav":
              yield _wav_header(SAMPLE_RATE)  # stream with unknown data length
 -        async for raw_chunk in _stream_chunks(voice_cfg, req.input):
-+        async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed):
++        async for raw_chunk in _stream_chunks(voice_cfg, req.input, req.voice, req.speed, req.instruct):
              yield raw_chunk
  
      return StreamingResponse(audio_stream(), media_type=content_type)
-@@ -306,16 +483,20 @@ def _parse_args():
+@@ -306,16 +566,21 @@ def _parse_args():
      p.add_argument("--host", default="0.0.0.0", help="Bind host (default: 0.0.0.0)")
      p.add_argument("--port", type=int, default=8000, help="Bind port (default: 8000)")
      p.add_argument("--device", default="cuda", help="Torch device (default: cuda)")
@@ -319,9 +421,10 @@ index 2199e14..d10065f 100644
  
  def main():
 -    global tts_model, voices, default_voice, SAMPLE_RATE
-+    global tts_model, voices, voices_file_path, last_voices_mtime, default_voice, SAMPLE_RATE
++    global tts_model, voices, voices_file_path, last_voices_mtime, default_voice, SAMPLE_RATE, MAX_SEQ_LEN
  
      args = _parse_args()
++    MAX_SEQ_LEN = args.max_seq_len
  
      # Build voice registry
      if args.voices:
@@ -331,7 +434,7 @@ index 2199e14..d10065f 100644
          with open(args.voices) as f:
              voices = json.load(f)
          default_voice = next(iter(voices))
-@@ -344,6 +525,7 @@ def main():
+@@ -344,6 +609,7 @@ def main():
          args.model,
          device=args.device,
          dtype=torch.bfloat16,