mARTin-B78/tts-voice-creator-clone-and-design-2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix streams permanently stuck holding the shared LLM lock (v1.14.2)

Browse Source 
Confirmed in production: a streaming attribution/character-sheets
request held _attribution_llm_lock for 11+ minutes, well past its
configured timeout, silently rejecting every subsequent passage as
"Attribution engine busy" and falling back to the non-streaming view -
looking exactly like live-thinking had stopped working, when actually
one earlier request never finished.

Root cause: requests' timeout= on a stream=True call only covers the
connect + first byte, not gaps between later body reads. If the LLM
backend goes silent mid-stream (connection left open, no more chunks),
the blocked socket recv() inside iter_lines() can hang indefinitely.
Since that's a native blocking call, not a Python-level yield point,
neither an in-loop wall-clock check nor GeneratorExit from a
disconnected client can interrupt it - both only take effect at the
next bytecode boundary, which never arrives while blocked in the C
extension.

Added _watchdog_close: a daemon thread that force-closes the upstream
connection if the wrapped block hasn't finished within the configured
timeout. Closing the socket from another thread makes the blocked
recv() raise, unblocking the generator so its normal except/finally
cleanup (including releasing the lock) actually runs. Verified against
the live LLM backend post-restart: stream completes normally with
real token-by-token deltas, and the lock is confirmed free immediately
after.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-07-06 20:57:49 +02:00
parent 6905219567
commit ce3ae79132
4 changed files with 105 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
CHANGELOG.md
View File

@ -9,6 +9,13 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
---
## [1.14.2] — 2026-07-06
### Fixed
- **Live streaming ("watch the LLM think") could get permanently stuck, silently breaking casting/character-sheets for the rest of the session** — confirmed in production: a stream held the shared attribution lock for 11+ minutes, causing every subsequent passage to be silently rejected as "busy" and fall back to the plain non-streaming view. This made it *look* like live-thinking had stopped working entirely, when actually one earlier passage never finished. Root cause: `requests`' `timeout=` on a `stream=True` call only guards the connection and first byte, not gaps between later body reads — if the LLM backend goes silent mid-stream, the blocked socket read can hang indefinitely, and since that's a native blocking call (not a Python-level `yield` point), neither a wall-clock check in the read loop nor the client disconnecting can interrupt it. Added a watchdog thread that force-closes the connection if a stream runs past its configured timeout, guaranteeing the shared lock always releases on schedule regardless of backend behavior.
---
## [1.14.1] — 2026-07-06
### Fixed

2
VERSION
View File

@ -1 +1 @@
1.14.1
1.14.2

61
routes/conversation.py
View File

@ -49,6 +49,42 @@ router = APIRouter()
# scenario this lock exists to prevent.
_attribution_llm_lock = threading.Lock()
@contextlib.contextmanager
def _watchdog_close(resp, timeout_seconds):
"""Force-close `resp`'s connection if the wrapped block hasn't finished
within timeout_seconds.
requests' own `timeout=` on a stream=True call only guards the connect
+ first byte — NOT the gaps between subsequent body reads. If the LLM
backend goes silent mid-stream (no more chunks, connection left open),
the blocking socket recv() inside iter_lines() can hang forever, and
since that's a native blocking call rather than a Python-level `yield`
point, neither a wall-clock check in the loop body nor GeneratorExit from
a disconnected client can interrupt it — both only take effect at the
next bytecode boundary, which never arrives. Observed in production: a
stuck stream held the shared attribution lock for 11+ minutes, silently
starving every other passage. Closing the connection from a separate
watchdog thread forces the blocked recv() to raise, unblocking the
generator so the normal except/finally cleanup (incl. releasing the
lock) actually runs.
"""
done = threading.Event()
def _kill():
if not done.wait(timeout_seconds):
try:
resp.close()
except Exception:
pass
t = threading.Thread(target=_kill, daemon=True)
t.start()
try:
yield
finally:
done.set()
# ── STT hallucination filter ──────────────────────────────────────────────────
# Whisper commonly hallucinates these phrases on silence/noise.
# Treat them as "no speech detected" rather than passing them to the LLM.
@ -816,7 +852,18 @@ async def character_sheets_stream(request: Request):
return
raw_parts = []
upstream.encoding = "utf-8"
# Two layers against a stuck stream: the wall-clock check catches
# a model that keeps trickling chunks past its budget, and
# _watchdog_close catches the connection going fully silent (which
# the in-loop check can't see, since a blocked socket read never
# reaches it — see _watchdog_close's docstring for why).
deadline = time.monotonic() + prep["timeout_seconds"]
with _watchdog_close(upstream, prep["timeout_seconds"]):
for line in upstream.iter_lines(decode_unicode=True):
if time.monotonic() > deadline:
print(f"[character-sheets/stream] wall-clock deadline hit after {prep['timeout_seconds']}s, aborting stream")
yield 'data: {"error": "LLM stream exceeded timeout"}\n\n'
return
if not line or not line.startswith("data:"):
continue
chunk = line[5:].strip()
@ -1372,7 +1419,21 @@ async def attribute_dialogue_stream(request: Request):
# UTF-8 when decode_unicode=True guesses the encoding — every umlaut
# then comes out mojibake'd ("Häfen" -> "Häfen"). Force UTF-8.
upstream.encoding = "utf-8"
# Two layers against a stuck stream: the wall-clock check catches
# a model that keeps trickling chunks past its budget, and
# _watchdog_close catches the connection going fully silent (which
# the in-loop check can't see, since a blocked socket read never
# reaches it — see _watchdog_close's docstring for why). Observed
# in production without the watchdog: a stuck stream held the
# shared lock for 11+ minutes, silently starving every other
# passage/request.
deadline = time.monotonic() + prep["timeout_seconds"]
with _watchdog_close(upstream, prep["timeout_seconds"]):
for line in upstream.iter_lines(decode_unicode=True):
if time.monotonic() > deadline:
print(f"[attribute-dialogue/stream] wall-clock deadline hit after {prep['timeout_seconds']}s, aborting stream")
yield 'data: {"error": "LLM stream exceeded timeout"}\n\n'
return
if not line or not line.startswith("data:"):
continue
chunk = line[5:].strip()

6
static/index.html
View File

@ -10,7 +10,7 @@
<meta name="format-detection" content="telephone=no">
<meta name="color-scheme" content="light dark">
<meta name="theme-color" content="#2563EB">
<meta name="app-version" content="1.14.1">
<meta name="app-version" content="1.14.2">
<link rel="manifest" href="/manifest.webmanifest">
<link rel="icon" href="/static/icon.svg" type="image/svg+xml">
<link rel="apple-touch-icon" href="/static/icon.svg">
@ -27,7 +27,7 @@
<!-- ── Core styles (local — no CDN dependency for first paint) ────────── -->
<link rel="stylesheet" href="/static/vendor/mdi/materialdesignicons.min.css">
<link rel="stylesheet" href="/static/style.css?v=1.14.1">
<link rel="stylesheet" href="/static/style.css?v=1.14.2">
<!-- ── Flag icons — non-blocking (loaded async, icons appear after JS) ── -->
@ -365,7 +365,7 @@ window.toggleNavTree = function(treeId, chevronId) {
</script>
<!-- loader.js: fetches sections → loads JS modules → removes skeleton -->
<script src="/static/loader.js?v=1.14.1"></script>
<script src="/static/loader.js?v=1.14.2"></script>
</body>
</html>