overlay: fix AT-SPI caret freeze; fuse routing match + live LLM into HUD; Release 1.6.0
Fix a desktop-session freeze (forced logout/reboot) caused by the overlay's AT-SPI caret tracker: it subscribed to the high-frequency object:text-caret-moved signal and made synchronous, blocking AT-SPI reads from inside the event handler, re-entering the a11y dispatcher and getting stormed by the app's own xdotool typing until GNOME stopped responding. Now track focus changes only and read the caret rectangle lazily, once, when the overlay shows — never on the hot path. Fuse voice-routing feedback into the overlay instead of a desktop notification: show the matched preset's emoji, name, and spoken keyword on a banner, narrate the phase (Transcribing -> Rewriting), and stream the LLM rewrite into the bubble token-by-token. Redundant per-dictation notifications are suppressed when the overlay is present (errors still notify); headless/overlay-off is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
6a9c1fc95a
commit
ae01472b77
@ -9,6 +9,31 @@ The version is defined in [`blitztext/__init__.py`](blitztext/__init__.py).
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [1.6.0] - 2026-06-07
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- **Session freeze when the overlay's caret tracking was active** (could lock up
|
||||||
|
the whole GNOME/X11 desktop, forcing a logout/reboot). The AT-SPI caret tracker
|
||||||
|
subscribed to the high-frequency `object:text-caret-moved` signal and made
|
||||||
|
**synchronous, blocking AT-SPI reads from inside the event handler** — which
|
||||||
|
re-enters the accessibility dispatcher and is stormed by the app's *own*
|
||||||
|
`xdotool` typing (one event per character), congesting the a11y bus until the
|
||||||
|
desktop stopped responding. It now tracks **focus changes only** and reads the
|
||||||
|
caret rectangle lazily (once, when the overlay shows), never from inside an
|
||||||
|
event dispatch.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- **Matched preset is fused into the overlay instead of a desktop notification**:
|
||||||
|
when voice routing picks a preset, the overlay shows its emoji icon, name, and
|
||||||
|
the spoken keyword on a banner, and narrates the phase ("Transcribing…" →
|
||||||
|
"Rewriting…"). With the overlay on, the redundant per-dictation notifications
|
||||||
|
are suppressed (errors still notify); headless/overlay-off keeps notifications.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- **Live LLM rewrite in the overlay**: rewrite presets now stream the model's
|
||||||
|
output into the bubble token-by-token, so you watch it write. The delivered
|
||||||
|
text is still the complete result, typed once the rewrite finishes.
|
||||||
|
|
||||||
## [1.5.1] - 2026-06-07
|
## [1.5.1] - 2026-06-07
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@ -6,4 +6,4 @@ counterpart to the macOS Blitztext menu bar app: it runs natively on the host
|
|||||||
(not in a container) so it can type into any application via xdotool.
|
(not in a container) so it can type into any application via xdotool.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__version__ = "1.5.1"
|
__version__ = "1.6.0"
|
||||||
|
|||||||
@ -6,8 +6,10 @@ through a chain of decreasing precision:
|
|||||||
|
|
||||||
1. AT-SPI caret — the real text insertion point, when the focused app exposes
|
1. AT-SPI caret — the real text insertion point, when the focused app exposes
|
||||||
it over accessibility (native GTK/Qt apps do; many terminals / Electron /
|
it over accessibility (native GTK/Qt apps do; many terminals / Electron /
|
||||||
web views do not). Tracked passively via the GLib main loop, so reads are
|
web views do not). We track only the *focused* object via a11y events (cheap,
|
||||||
instant and never block.
|
low-frequency) and read its caret rectangle lazily, once, when the overlay
|
||||||
|
shows — never from inside an event dispatch, since synchronous AT-SPI reads
|
||||||
|
on the hot path can wedge the accessibility bus and freeze the session.
|
||||||
2. Mouse pointer — `xdotool getmouselocation`. Always available on X11; a good
|
2. Mouse pointer — `xdotool getmouselocation`. Always available on X11; a good
|
||||||
proxy since the pointer is usually near where you're typing.
|
proxy since the pointer is usually near where you're typing.
|
||||||
3. Window / screen — top-centre of the target window, else screen bottom-centre.
|
3. Window / screen — top-centre of the target window, else screen bottom-centre.
|
||||||
@ -40,25 +42,39 @@ class Anchor:
|
|||||||
# Tier 1: AT-SPI caret tracking (best-effort, passive)
|
# Tier 1: AT-SPI caret tracking (best-effort, passive)
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
class _CaretTracker:
|
class _CaretTracker:
|
||||||
"""Passively follow the focused editable text's caret via AT-SPI events.
|
"""Remember the most recently focused text object; read its caret lazily.
|
||||||
|
|
||||||
We never poll the a11y tree on the hot path (that can be slow and can block
|
*Why so cautious.* AT-SPI queries (``get_character_extents`` and friends) are
|
||||||
on unresponsive apps). Instead we subscribe to caret-moved / focus events and
|
**synchronous, blocking D-Bus round-trips into the target application**. The
|
||||||
cache the last caret rectangle; `rect()` just returns the cached value if it
|
earlier design subscribed to the high-frequency ``object:text-caret-moved``
|
||||||
is fresh enough to still be meaningful.
|
signal and ran those blocking reads from *inside* the event handler. Two ways
|
||||||
|
that wedges a whole GNOME/X11 session:
|
||||||
|
|
||||||
|
• Calling a synchronous AT-SPI method from within an AT-SPI event dispatch
|
||||||
|
re-enters the a11y dispatcher and can deadlock the accessibility bus.
|
||||||
|
• ``text-caret-moved`` fires once *per character* — and delivering text is
|
||||||
|
exactly what this app does, typing via ``xdotool`` into the focused
|
||||||
|
field. So a single dictation became a storm of blocking round-trips on
|
||||||
|
the GTK main loop, congesting the a11y bus until the desktop froze.
|
||||||
|
|
||||||
|
So we now subscribe to **focus changes only** (rare, and never emitted by our
|
||||||
|
own synthetic typing), cache just the focused accessible, and do the one
|
||||||
|
blocking extents read **on demand** in :meth:`rect` — called once, when the
|
||||||
|
overlay shows, outside any event dispatch. Worst case is a slightly delayed
|
||||||
|
overlay placement, never a frozen session.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
STALE_SECONDS = 30.0 # ignore a cached caret older than this
|
STALE_SECONDS = 30.0 # ignore a focus older than this
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
self._ok = False
|
self._ok = False
|
||||||
self._listener = None
|
self._listener = None
|
||||||
self._rect: tuple[int, int, int, int] | None = None # x, y, w, h (screen)
|
self._focused = None # last focused accessible (read lazily)
|
||||||
self._stamp = 0.0
|
self._stamp = 0.0
|
||||||
self._Atspi = None
|
self._Atspi = None
|
||||||
|
|
||||||
def start(self) -> bool:
|
def start(self) -> bool:
|
||||||
"""Register AT-SPI listeners on the (already running) GLib main loop.
|
"""Register the AT-SPI focus listener on the running GLib main loop.
|
||||||
|
|
||||||
Safe to call when accessibility is disabled — it just returns False and
|
Safe to call when accessibility is disabled — it just returns False and
|
||||||
the anchor logic skips this tier from then on.
|
the anchor logic skips this tier from then on.
|
||||||
@ -74,10 +90,10 @@ class _CaretTracker:
|
|||||||
# init() is idempotent; returns 0/1. Connects to the a11y registry.
|
# init() is idempotent; returns 0/1. Connects to the a11y registry.
|
||||||
Atspi.init()
|
Atspi.init()
|
||||||
self._Atspi = Atspi
|
self._Atspi = Atspi
|
||||||
self._listener = Atspi.EventListener.new(self._on_event)
|
self._listener = Atspi.EventListener.new(self._on_focus)
|
||||||
# Caret moves give us the live position; focus changes let us grab the
|
# Focus changes only. Deliberately NOT "object:text-caret-moved": that
|
||||||
# caret of a freshly-focused field even before it moves.
|
# firehose (one event per typed character, including our own output)
|
||||||
self._listener.register("object:text-caret-moved")
|
# plus synchronous reads is what could freeze the session.
|
||||||
self._listener.register("object:state-changed:focused")
|
self._listener.register("object:state-changed:focused")
|
||||||
self._ok = True
|
self._ok = True
|
||||||
log("[overlay] AT-SPI caret tracking active")
|
log("[overlay] AT-SPI caret tracking active")
|
||||||
@ -90,26 +106,23 @@ class _CaretTracker:
|
|||||||
def stop(self) -> None:
|
def stop(self) -> None:
|
||||||
try:
|
try:
|
||||||
if self._listener is not None:
|
if self._listener is not None:
|
||||||
self._listener.deregister("object:text-caret-moved")
|
|
||||||
self._listener.deregister("object:state-changed:focused")
|
self._listener.deregister("object:state-changed:focused")
|
||||||
except Exception: # noqa: BLE001
|
except Exception: # noqa: BLE001
|
||||||
pass
|
pass
|
||||||
self._listener = None
|
self._listener = None
|
||||||
|
self._focused = None
|
||||||
self._ok = False
|
self._ok = False
|
||||||
|
|
||||||
def _on_event(self, event) -> None:
|
def _on_focus(self, event) -> None:
|
||||||
# Runs on the GLib main thread (same loop GTK uses). Keep it cheap and
|
# Runs on the GLib main thread (same loop GTK uses). Do the *minimum*:
|
||||||
# never raise — an exception here would bubble into the a11y dispatcher.
|
# stash the focused accessible and stamp it. Crucially, make NO synchronous
|
||||||
|
# AT-SPI calls here — that would re-enter the a11y dispatcher and risk
|
||||||
|
# deadlocking the bus. The blocking extents read happens later, in rect().
|
||||||
try:
|
try:
|
||||||
if event.type.startswith("object:state-changed:focused") and not event.detail1:
|
if not event.detail1:
|
||||||
return # a *de*focus event — nothing to read
|
return # a *de*focus event — nothing to track
|
||||||
source = event.source
|
self._focused = event.source
|
||||||
if source is None:
|
self._stamp = time.time()
|
||||||
return
|
|
||||||
rect = self._caret_rect(source)
|
|
||||||
if rect is not None:
|
|
||||||
self._rect = rect
|
|
||||||
self._stamp = time.time()
|
|
||||||
except Exception: # noqa: BLE001
|
except Exception: # noqa: BLE001
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@ -142,11 +155,17 @@ class _CaretTracker:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
def rect(self) -> tuple[int, int, int, int] | None:
|
def rect(self) -> tuple[int, int, int, int] | None:
|
||||||
if not self._ok or self._rect is None:
|
# Called once when the overlay shows (not on the a11y hot path), so the
|
||||||
|
# single blocking extents read here is safe: at worst it briefly delays
|
||||||
|
# the overlay, it cannot storm or re-enter the bus.
|
||||||
|
if not self._ok or self._focused is None:
|
||||||
return None
|
return None
|
||||||
if time.time() - self._stamp > self.STALE_SECONDS:
|
if time.time() - self._stamp > self.STALE_SECONDS:
|
||||||
return None
|
return None
|
||||||
return self._rect
|
try:
|
||||||
|
return self._caret_rect(self._focused)
|
||||||
|
except Exception: # noqa: BLE001 - focused app may be gone/unresponsive
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
@ -37,17 +37,25 @@ class Daemon:
|
|||||||
def __init__(self, cfg: Config, status_cb: StatusCallback | None = None,
|
def __init__(self, cfg: Config, status_cb: StatusCallback | None = None,
|
||||||
level_cb: Callable[[float], None] | None = None,
|
level_cb: Callable[[float], None] | None = None,
|
||||||
text_cb: Callable[[str], None] | None = None,
|
text_cb: Callable[[str], None] | None = None,
|
||||||
countdown_cb: Callable[[float | None, float], None] | None = None):
|
countdown_cb: Callable[[float | None, float], None] | None = None,
|
||||||
|
routing_cb: Callable[[str, str, str | None], None] | None = None):
|
||||||
self.cfg = cfg
|
self.cfg = cfg
|
||||||
self.status_cb = status_cb
|
self.status_cb = status_cb
|
||||||
# Optional UI feedback hooks for the on-screen overlay. The daemon stays
|
# Optional UI feedback hooks for the on-screen overlay. The daemon stays
|
||||||
# UI-agnostic: these are no-ops in headless mode. level_cb gets the live
|
# UI-agnostic: these are no-ops in headless mode. level_cb gets the live
|
||||||
# mic level (0..1); text_cb gets the running transcript while streaming;
|
# mic level (0..1); text_cb gets the running transcript while streaming
|
||||||
# countdown_cb(seconds_left, window) drives the silence auto-stop ring
|
# (and the live LLM rewrite); countdown_cb(seconds_left, window) drives the
|
||||||
# (seconds_left=None while you're speaking, so the ring clears).
|
# silence auto-stop ring (seconds_left=None while you're speaking, so the
|
||||||
|
# ring clears); routing_cb(icon, preset_name, keyword) fires when voice
|
||||||
|
# routing picks a preset, so the overlay can show it.
|
||||||
self.level_cb = level_cb
|
self.level_cb = level_cb
|
||||||
self.text_cb = text_cb
|
self.text_cb = text_cb
|
||||||
self.countdown_cb = countdown_cb
|
self.countdown_cb = countdown_cb
|
||||||
|
self.routing_cb = routing_cb
|
||||||
|
# An overlay consumes routing_cb, and it narrates every phase on-screen, so
|
||||||
|
# the redundant desktop notifications are fused into it (only errors still
|
||||||
|
# pop a bubble). Headless / overlay-off keeps the notifications.
|
||||||
|
self._overlay = routing_cb is not None
|
||||||
self._ov_meter = None
|
self._ov_meter = None
|
||||||
self._ov_text_final = ""
|
self._ov_text_final = ""
|
||||||
self._lock = threading.Lock()
|
self._lock = threading.Lock()
|
||||||
@ -99,14 +107,20 @@ class Daemon:
|
|||||||
notify(title, body, urgency=urgency, enabled=self.cfg.notify)
|
notify(title, body, urgency=urgency, enabled=self.cfg.notify)
|
||||||
|
|
||||||
def _dnotify(self, title: str, body: str = "", urgency: str = "normal") -> None:
|
def _dnotify(self, title: str, body: str = "", urgency: str = "normal") -> None:
|
||||||
"""Per-dictation notification — suppressed for hands-free (wakeword) sessions."""
|
"""Per-dictation notification — suppressed for hands-free (wakeword)
|
||||||
if not self._session_silent:
|
sessions, and (except for errors) when an overlay is narrating on-screen."""
|
||||||
self._notify(title, body, urgency=urgency)
|
if self._session_silent:
|
||||||
|
return
|
||||||
|
if self._overlay and urgency != "critical":
|
||||||
|
return # fused into the on-screen overlay instead of a desktop bubble
|
||||||
|
self._notify(title, body, urgency=urgency)
|
||||||
|
|
||||||
def _rnotify(self, title: str, body: str = "", urgency: str = "normal") -> None:
|
def _rnotify(self, title: str, body: str = "", urgency: str = "normal") -> None:
|
||||||
"""Routing feedback — which preset/keyword a voice command matched. Shown
|
"""Routing feedback — which preset/keyword a voice command matched. When an
|
||||||
even for hands-free sessions (it has its own toggle) so you can always see
|
overlay is present this is shown there (via routing_cb) instead of a
|
||||||
what you triggered. Only fires on a real match, so it never spams silence."""
|
notification; otherwise it pops a bubble. Only fires on a real match."""
|
||||||
|
if self._overlay:
|
||||||
|
return # shown on the overlay banner instead
|
||||||
notify(title, body, urgency=urgency, enabled=self.cfg.notify_routing)
|
notify(title, body, urgency=urgency, enabled=self.cfg.notify_routing)
|
||||||
|
|
||||||
def _emit(self, state: str, workflow: str | None = None, message: str = "") -> None:
|
def _emit(self, state: str, workflow: str | None = None, message: str = "") -> None:
|
||||||
@ -456,6 +470,13 @@ class Daemon:
|
|||||||
icon = (getattr(target, "icon", "") or "🎙") if target else "🎙"
|
icon = (getattr(target, "icon", "") or "🎙") if target else "🎙"
|
||||||
via = f"“{res.keyword}”" if res.keyword else "no keyword → default"
|
via = f"“{res.keyword}”" if res.keyword else "no keyword → default"
|
||||||
self._emit("busy", label, f"→ {label} ({via})")
|
self._emit("busy", label, f"→ {label} ({via})")
|
||||||
|
# Fuse the match onto the overlay (icon + preset + keyword); falls
|
||||||
|
# back to a desktop notification only when there's no overlay.
|
||||||
|
if self.routing_cb:
|
||||||
|
try:
|
||||||
|
self.routing_cb(icon, label, res.keyword)
|
||||||
|
except Exception: # noqa: BLE001 - UI must not break the engine
|
||||||
|
pass
|
||||||
self._rnotify(f"{icon} {label}", f"matched: {via}")
|
self._rnotify(f"{icon} {label}", f"matched: {via}")
|
||||||
log(f"→ routed to {label} (matched: {via})")
|
log(f"→ routed to {label} (matched: {via})")
|
||||||
else:
|
else:
|
||||||
@ -468,6 +489,17 @@ class Daemon:
|
|||||||
return
|
return
|
||||||
self._emit("busy", label, "Rewriting…")
|
self._emit("busy", label, "Rewriting…")
|
||||||
self._dnotify(f"⌛ {label}", "Rewriting…")
|
self._dnotify(f"⌛ {label}", "Rewriting…")
|
||||||
|
# Stream the rewrite into the overlay so you watch the model write
|
||||||
|
# (the bubble updates token-by-token). The delivered text is still
|
||||||
|
# the complete result, typed once the rewrite finishes.
|
||||||
|
on_token = None
|
||||||
|
if self.text_cb:
|
||||||
|
acc: list[str] = []
|
||||||
|
|
||||||
|
def on_token(delta: str, _acc=acc) -> None:
|
||||||
|
_acc.append(delta)
|
||||||
|
self.text_cb("".join(_acc))
|
||||||
|
|
||||||
try:
|
try:
|
||||||
text = llm.chat(
|
text = llm.chat(
|
||||||
self.cfg.active_llm,
|
self.cfg.active_llm,
|
||||||
@ -476,6 +508,7 @@ class Daemon:
|
|||||||
model=target.model or None,
|
model=target.model or None,
|
||||||
temperature=target.temperature,
|
temperature=target.temperature,
|
||||||
timeout=self.cfg.timeout,
|
timeout=self.cfg.timeout,
|
||||||
|
on_token=on_token,
|
||||||
)
|
)
|
||||||
except LLMError as exc:
|
except LLMError as exc:
|
||||||
self._emit("error", label, str(exc))
|
self._emit("error", label, str(exc))
|
||||||
|
|||||||
@ -108,6 +108,9 @@ class App:
|
|||||||
cfg, status_cb=self._status_cb,
|
cfg, status_cb=self._status_cb,
|
||||||
level_cb=self._on_level, text_cb=self._on_text,
|
level_cb=self._on_level, text_cb=self._on_text,
|
||||||
countdown_cb=self._on_countdown,
|
countdown_cb=self._on_countdown,
|
||||||
|
# Only consume routing on the overlay when there's an overlay to show
|
||||||
|
# it on; otherwise the daemon keeps the desktop notification.
|
||||||
|
routing_cb=self._on_routing if self.overlay is not None else None,
|
||||||
)
|
)
|
||||||
|
|
||||||
_install_css()
|
_install_css()
|
||||||
@ -276,6 +279,10 @@ class App:
|
|||||||
if self.overlay is not None:
|
if self.overlay is not None:
|
||||||
self.overlay.set_countdown(remaining, total)
|
self.overlay.set_countdown(remaining, total)
|
||||||
|
|
||||||
|
def _on_routing(self, icon: str, name: str, keyword: str | None) -> None:
|
||||||
|
if self.overlay is not None:
|
||||||
|
self.overlay.set_preset(icon, name, keyword)
|
||||||
|
|
||||||
def _overlay_status(self, state: str, message: str) -> None:
|
def _overlay_status(self, state: str, message: str) -> None:
|
||||||
"""Translate engine phases into overlay show/update/hide (GTK thread)."""
|
"""Translate engine phases into overlay show/update/hide (GTK thread)."""
|
||||||
ov = self.overlay
|
ov = self.overlay
|
||||||
@ -287,7 +294,10 @@ class App:
|
|||||||
ov.show(state, getattr(self.daemon, "_target_window", None))
|
ov.show(state, getattr(self.daemon, "_target_window", None))
|
||||||
elif state == "busy":
|
elif state == "busy":
|
||||||
self._ov_state = state
|
self._ov_state = state
|
||||||
ov.set_state("busy", message)
|
# The routing detail ("→ Nicer email (matched: …)") is shown on the
|
||||||
|
# preset banner, not as a phase chip — keep the chip a clean phase word.
|
||||||
|
phase = "Transcribing…" if message.startswith("→") else message
|
||||||
|
ov.set_state("busy", phase)
|
||||||
elif state == "done":
|
elif state == "done":
|
||||||
# Non-streaming: the 'done' message carries the final text. Streaming
|
# Non-streaming: the 'done' message carries the final text. Streaming
|
||||||
# already showed it live, so don't overwrite with "Streaming stopped".
|
# already showed it live, so don't overwrite with "Streaming stopped".
|
||||||
|
|||||||
@ -13,6 +13,7 @@ import os
|
|||||||
import urllib.error
|
import urllib.error
|
||||||
import urllib.request
|
import urllib.request
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
from .stt import reachable
|
from .stt import reachable
|
||||||
|
|
||||||
@ -48,18 +49,28 @@ def chat(
|
|||||||
model: str | None = None,
|
model: str | None = None,
|
||||||
temperature: float | None = None,
|
temperature: float | None = None,
|
||||||
timeout: int = 45,
|
timeout: int = 45,
|
||||||
|
on_token: Callable[[str], None] | None = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
|
"""Run a chat completion and return the full text.
|
||||||
|
|
||||||
|
When ``on_token`` is given, the request is streamed and each content delta is
|
||||||
|
handed to the callback as it arrives (so a UI can show the model writing in
|
||||||
|
real time). The callback is best-effort — it never affects the return value,
|
||||||
|
which is always the complete, stripped response.
|
||||||
|
"""
|
||||||
|
stream = on_token is not None
|
||||||
api_key = engine.api_key
|
api_key = engine.api_key
|
||||||
payload = json.dumps(
|
body_obj = {
|
||||||
{
|
"model": model or engine.model,
|
||||||
"model": model or engine.model,
|
"temperature": engine.temperature if temperature is None else temperature,
|
||||||
"temperature": engine.temperature if temperature is None else temperature,
|
"messages": [
|
||||||
"messages": [
|
{"role": "system", "content": system_prompt},
|
||||||
{"role": "system", "content": system_prompt},
|
{"role": "user", "content": user_text},
|
||||||
{"role": "user", "content": user_text},
|
],
|
||||||
],
|
}
|
||||||
}
|
if stream:
|
||||||
).encode("utf-8")
|
body_obj["stream"] = True
|
||||||
|
payload = json.dumps(body_obj).encode("utf-8")
|
||||||
|
|
||||||
headers = {"Content-Type": "application/json"}
|
headers = {"Content-Type": "application/json"}
|
||||||
if api_key:
|
if api_key:
|
||||||
@ -70,19 +81,46 @@ def chat(
|
|||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
body = json.loads(resp.read().decode("utf-8"))
|
if stream:
|
||||||
|
content = _read_stream(resp, on_token)
|
||||||
|
else:
|
||||||
|
body = json.loads(resp.read().decode("utf-8"))
|
||||||
|
content = body["choices"][0]["message"]["content"]
|
||||||
except urllib.error.HTTPError as exc:
|
except urllib.error.HTTPError as exc:
|
||||||
detail = exc.read().decode("utf-8", "replace")[:300]
|
detail = exc.read().decode("utf-8", "replace")[:300]
|
||||||
raise LLMError(f"HTTP {exc.code}: {detail}") from exc
|
raise LLMError(f"HTTP {exc.code}: {detail}") from exc
|
||||||
except urllib.error.URLError as exc:
|
except urllib.error.URLError as exc:
|
||||||
raise LLMError(f"Connection failed: {exc.reason}") from exc
|
raise LLMError(f"Connection failed: {exc.reason}") from exc
|
||||||
|
|
||||||
try:
|
|
||||||
content = body["choices"][0]["message"]["content"]
|
|
||||||
except (KeyError, IndexError, TypeError) as exc:
|
except (KeyError, IndexError, TypeError) as exc:
|
||||||
raise LLMError(f"Unexpected response: {str(body)[:300]}") from exc
|
raise LLMError(f"Unexpected response: {exc}") from exc
|
||||||
|
|
||||||
content = (content or "").strip()
|
content = (content or "").strip()
|
||||||
if not content:
|
if not content:
|
||||||
raise LLMError("Empty response from model.")
|
raise LLMError("Empty response from model.")
|
||||||
return content
|
return content
|
||||||
|
|
||||||
|
|
||||||
|
def _read_stream(resp, on_token: Callable[[str], None]) -> str:
|
||||||
|
"""Parse an OpenAI-style SSE stream, returning the accumulated content and
|
||||||
|
feeding each delta to ``on_token``. Tolerant of keep-alive blanks and the
|
||||||
|
trailing ``[DONE]`` sentinel."""
|
||||||
|
parts: list[str] = []
|
||||||
|
for raw in resp:
|
||||||
|
line = raw.decode("utf-8", "replace").strip()
|
||||||
|
if not line or not line.startswith("data:"):
|
||||||
|
continue
|
||||||
|
data = line[len("data:"):].strip()
|
||||||
|
if data == "[DONE]":
|
||||||
|
break
|
||||||
|
try:
|
||||||
|
obj = json.loads(data)
|
||||||
|
delta = obj["choices"][0]["delta"].get("content")
|
||||||
|
except (ValueError, KeyError, IndexError, TypeError):
|
||||||
|
continue
|
||||||
|
if delta:
|
||||||
|
parts.append(delta)
|
||||||
|
try:
|
||||||
|
on_token(delta)
|
||||||
|
except Exception: # noqa: BLE001 - UI hiccups must not break delivery
|
||||||
|
pass
|
||||||
|
return "".join(parts)
|
||||||
|
|||||||
@ -40,6 +40,7 @@ _TAIL_W = 20
|
|||||||
_TAIL_H = 11
|
_TAIL_H = 11
|
||||||
_GAP = 12 # clearance between the tail tip and the anchor
|
_GAP = 12 # clearance between the tail tip and the anchor
|
||||||
_BARS = 30 # waveform bar count
|
_BARS = 30 # waveform bar count
|
||||||
|
_PRESET_H = 22 # matched-preset header row (emoji + name + keyword)
|
||||||
_MIN_TEXT_H = 0
|
_MIN_TEXT_H = 0
|
||||||
_MAX_TEXT_H = 120
|
_MAX_TEXT_H = 120
|
||||||
|
|
||||||
@ -62,6 +63,12 @@ class Overlay:
|
|||||||
self._state = "recording"
|
self._state = "recording"
|
||||||
self._text = ""
|
self._text = ""
|
||||||
self._phase_label = "Listening…"
|
self._phase_label = "Listening…"
|
||||||
|
# Matched-preset banner (fused in from voice routing instead of a separate
|
||||||
|
# desktop notification): the preset's emoji, its name, and the spoken
|
||||||
|
# keyword that selected it.
|
||||||
|
self._preset_icon = ""
|
||||||
|
self._preset_name = ""
|
||||||
|
self._keyword = ""
|
||||||
self._anchor: caret.Anchor | None = None
|
self._anchor: caret.Anchor | None = None
|
||||||
self._tail_up = False # tail on top edge (bubble below anchor)?
|
self._tail_up = False # tail on top edge (bubble below anchor)?
|
||||||
self._tail_x = _WIDTH // 2 # tail tip, window-local x
|
self._tail_x = _WIDTH // 2 # tail tip, window-local x
|
||||||
@ -118,6 +125,11 @@ class Overlay:
|
|||||||
def set_text(self, text: str) -> None:
|
def set_text(self, text: str) -> None:
|
||||||
GLib.idle_add(self._set_text, text or "")
|
GLib.idle_add(self._set_text, text or "")
|
||||||
|
|
||||||
|
def set_preset(self, icon: str, name: str, keyword: str | None) -> None:
|
||||||
|
"""Show the matched voice-routing preset on the overlay (emoji + name +
|
||||||
|
the spoken keyword), in place of a separate desktop notification."""
|
||||||
|
GLib.idle_add(self._set_preset, icon or "", name or "", keyword or "")
|
||||||
|
|
||||||
def set_countdown(self, remaining: float | None, total: float) -> None:
|
def set_countdown(self, remaining: float | None, total: float) -> None:
|
||||||
"""Silence auto-stop progress: ``remaining`` seconds until it fires over
|
"""Silence auto-stop progress: ``remaining`` seconds until it fires over
|
||||||
a ``total``-second window, or ``None`` while you're still speaking."""
|
a ``total``-second window, or ``None`` while you're still speaking."""
|
||||||
@ -138,6 +150,9 @@ class Overlay:
|
|||||||
self._state = state
|
self._state = state
|
||||||
self._phase_label = _PHASES.get(state, ((1, 1, 1), ""))[1]
|
self._phase_label = _PHASES.get(state, ((1, 1, 1), ""))[1]
|
||||||
self._text = ""
|
self._text = ""
|
||||||
|
self._preset_icon = ""
|
||||||
|
self._preset_name = ""
|
||||||
|
self._keyword = ""
|
||||||
self._levels = deque([0.0] * _BARS, maxlen=_BARS)
|
self._levels = deque([0.0] * _BARS, maxlen=_BARS)
|
||||||
self._disp = [0.0] * _BARS
|
self._disp = [0.0] * _BARS
|
||||||
self._cd_deadline = None
|
self._cd_deadline = None
|
||||||
@ -163,6 +178,16 @@ class Overlay:
|
|||||||
self._area.queue_draw()
|
self._area.queue_draw()
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def _set_preset(self, icon: str, name: str, keyword: str) -> bool:
|
||||||
|
if (icon, name, keyword) == (self._preset_icon, self._preset_name, self._keyword):
|
||||||
|
return False
|
||||||
|
self._preset_icon = icon
|
||||||
|
self._preset_name = name
|
||||||
|
self._keyword = keyword
|
||||||
|
self._relayout()
|
||||||
|
self._area.queue_draw()
|
||||||
|
return False
|
||||||
|
|
||||||
def _set_countdown(self, remaining: float | None, total: float) -> bool:
|
def _set_countdown(self, remaining: float | None, total: float) -> bool:
|
||||||
if remaining is None:
|
if remaining is None:
|
||||||
self._cd_deadline = None
|
self._cd_deadline = None
|
||||||
@ -173,7 +198,13 @@ class Overlay:
|
|||||||
|
|
||||||
def _set_state(self, state: str, message: str) -> bool:
|
def _set_state(self, state: str, message: str) -> bool:
|
||||||
self._state = state
|
self._state = state
|
||||||
self._phase_label = _PHASES.get(state, ((1, 1, 1), message[:40]))[1] or message[:40]
|
if state == "busy":
|
||||||
|
# Honour a clean phase word from the caller ("Transcribing…",
|
||||||
|
# "Rewriting…") so the overlay narrates what's happening; fall back to
|
||||||
|
# the canned label otherwise.
|
||||||
|
self._phase_label = (message.strip() or _PHASES["busy"][1])[:28]
|
||||||
|
else:
|
||||||
|
self._phase_label = _PHASES.get(state, ((1, 1, 1), message[:40]))[1] or message[:40]
|
||||||
if state not in ("recording", "streaming"):
|
if state not in ("recording", "streaming"):
|
||||||
# The countdown only makes sense while listening; drop it as soon as
|
# The countdown only makes sense while listening; drop it as soon as
|
||||||
# we move on to transcribing / done / idle so the ring doesn't linger.
|
# we move on to transcribing / done / idle so the ring doesn't linger.
|
||||||
@ -255,8 +286,10 @@ class Overlay:
|
|||||||
|
|
||||||
def _relayout(self) -> None:
|
def _relayout(self) -> None:
|
||||||
text_h = self._text_height()
|
text_h = self._text_height()
|
||||||
|
preset_h = _PRESET_H if self._preset_name else 0
|
||||||
|
gap_preset = 8 if preset_h else 0
|
||||||
gap_text = 8 if text_h else 0
|
gap_text = 8 if text_h else 0
|
||||||
body_h = _HEADER_H + gap_text + text_h + 2 * _PAD
|
body_h = _HEADER_H + gap_preset + preset_h + gap_text + text_h + 2 * _PAD
|
||||||
self._height = body_h + _TAIL_H
|
self._height = body_h + _TAIL_H
|
||||||
|
|
||||||
geo = self._monitor_geo()
|
geo = self._monitor_geo()
|
||||||
@ -314,15 +347,56 @@ class Overlay:
|
|||||||
wf_w = w - _PAD - wf_x
|
wf_w = w - _PAD - wf_x
|
||||||
self._draw_wave(cr, wf_x, body_top + _PAD, wf_w, _HEADER_H)
|
self._draw_wave(cr, wf_x, body_top + _PAD, wf_w, _HEADER_H)
|
||||||
|
|
||||||
# Phase label (top-right, small) when there's room and no text yet.
|
# Phase label by the waveform (top-right) — only when there's no preset
|
||||||
if self._phase_label and not self._text:
|
# banner below (which carries the phase instead) and no text yet.
|
||||||
|
if self._phase_label and not self._text and not self._preset_name:
|
||||||
self._draw_label(cr, w - _PAD, body_top + _PAD + 12, self._phase_label)
|
self._draw_label(cr, w - _PAD, body_top + _PAD + 12, self._phase_label)
|
||||||
|
|
||||||
# Recognised text below the header row.
|
y = body_top + _PAD + _HEADER_H
|
||||||
|
# Matched-preset banner: emoji + name (left), live phase chip (right).
|
||||||
|
if self._preset_name:
|
||||||
|
y += 8
|
||||||
|
self._draw_preset(cr, _PAD, y, w - 2 * _PAD)
|
||||||
|
y += _PRESET_H
|
||||||
|
# Recognised / rewritten text below.
|
||||||
if self._text:
|
if self._text:
|
||||||
self._draw_text(cr, _PAD, body_top + _PAD + _HEADER_H + 8, w - 2 * _PAD)
|
y += 8
|
||||||
|
self._draw_text(cr, _PAD, y, w - 2 * _PAD)
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def _draw_preset(self, cr, x, y, w) -> None:
|
||||||
|
"""The matched voice-routing preset, fused onto the overlay in place of a
|
||||||
|
desktop notification: emoji + name (and the spoken keyword) on the left,
|
||||||
|
the current phase ("Transcribing…", "Rewriting…") on the right."""
|
||||||
|
cy = y + _PRESET_H / 2
|
||||||
|
cursor = x
|
||||||
|
if self._preset_icon:
|
||||||
|
ic = self._win.create_pango_layout(self._preset_icon)
|
||||||
|
ic.set_font_description(Pango.FontDescription("Sans 13"))
|
||||||
|
iw, ih = ic.get_pixel_size()
|
||||||
|
cr.set_source_rgba(1, 1, 1, 0.95)
|
||||||
|
cr.move_to(cursor, cy - ih / 2)
|
||||||
|
PangoCairo.show_layout(cr, ic)
|
||||||
|
cursor += iw + 7
|
||||||
|
name = GLib.markup_escape_text(self._preset_name)
|
||||||
|
if self._keyword:
|
||||||
|
kw = GLib.markup_escape_text(self._keyword)
|
||||||
|
markup = f'<b>{name}</b> <span alpha="55%">“{kw}”</span>'
|
||||||
|
else:
|
||||||
|
markup = f"<b>{name}</b>"
|
||||||
|
layout = self._win.create_pango_layout("")
|
||||||
|
layout.set_markup(markup, -1)
|
||||||
|
layout.set_font_description(Pango.FontDescription("Sans 10"))
|
||||||
|
layout.set_ellipsize(Pango.EllipsizeMode.END)
|
||||||
|
avail = (x + w) - cursor - 96 # leave room for the phase chip
|
||||||
|
layout.set_width(max(40, avail) * Pango.SCALE)
|
||||||
|
_nw, nh = layout.get_pixel_size()
|
||||||
|
cr.set_source_rgba(0.95, 0.96, 0.99, 0.98)
|
||||||
|
cr.move_to(cursor, cy - nh / 2)
|
||||||
|
PangoCairo.show_layout(cr, layout)
|
||||||
|
if self._phase_label:
|
||||||
|
self._draw_label(cr, x + w, cy, self._phase_label)
|
||||||
|
|
||||||
def _bubble_path(self, cr, x, y, w, h) -> None:
|
def _bubble_path(self, cr, x, y, w, h) -> None:
|
||||||
r = _RADIUS
|
r = _RADIUS
|
||||||
cr.new_sub_path()
|
cr.new_sub_path()
|
||||||
|
|||||||
@ -108,6 +108,41 @@ def test_matched_preset_announced_even_hands_free(monkeypatch):
|
|||||||
assert shown == [], "no announcement when 'Announce matched preset' is off"
|
assert shown == [], "no announcement when 'Announce matched preset' is off"
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_fuses_notifications(monkeypatch):
|
||||||
|
"""With an overlay (routing_cb wired), the on-screen bubble narrates every
|
||||||
|
phase, so informational desktop notifications are suppressed — but errors
|
||||||
|
still pop a bubble, and the routing match is shown on the overlay, not a
|
||||||
|
notification."""
|
||||||
|
calls = []
|
||||||
|
monkeypatch.setattr(daemon_mod, "notify", lambda *a, **k: calls.append((a, k)))
|
||||||
|
monkeypatch.setattr(daemon_mod, "detect_recorder", lambda pref="auto": "pw-record")
|
||||||
|
d = Daemon(Config(), routing_cb=lambda *a: None)
|
||||||
|
assert d._overlay is True
|
||||||
|
d._session_silent = False
|
||||||
|
|
||||||
|
d._dnotify("⌛ Nicer email", "Rewriting…") # informational
|
||||||
|
assert calls == [], "informational notifications fuse into the overlay"
|
||||||
|
|
||||||
|
d._dnotify("Error", "boom", urgency="critical") # errors still notify
|
||||||
|
assert len(calls) == 1, "errors must still pop a desktop bubble"
|
||||||
|
|
||||||
|
calls.clear()
|
||||||
|
d.cfg.notify_routing = True
|
||||||
|
d._rnotify("⚡ Nicer email", "matched: “nicer email”")
|
||||||
|
assert calls == [], "the match is shown on the overlay banner, not as a notification"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_overlay_keeps_notifications(monkeypatch):
|
||||||
|
"""Headless / overlay-off: notifications remain the only feedback."""
|
||||||
|
calls = []
|
||||||
|
monkeypatch.setattr(daemon_mod, "notify", lambda *a, **k: calls.append((a, k)))
|
||||||
|
d = _make_daemon(monkeypatch) # no routing_cb
|
||||||
|
assert d._overlay is False
|
||||||
|
d._session_silent = False
|
||||||
|
d._dnotify("⌛ Nicer email", "Rewriting…")
|
||||||
|
assert len(calls) == 1, "without an overlay, informational notifications still show"
|
||||||
|
|
||||||
|
|
||||||
def test_wakeword_while_busy_does_not_notify(monkeypatch):
|
def test_wakeword_while_busy_does_not_notify(monkeypatch):
|
||||||
"""The away-from-keyboard "Busy" storm: a detection arriving while the
|
"""The away-from-keyboard "Busy" storm: a detection arriving while the
|
||||||
previous clip is still being processed must be ignored silently."""
|
previous clip is still being processed must be ignored silently."""
|
||||||
|
|||||||
41
linux/tests/test_llm_stream.py
Normal file
41
linux/tests/test_llm_stream.py
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
"""Streaming the LLM rewrite into the overlay (so you watch the model write).
|
||||||
|
|
||||||
|
`llm.chat(..., on_token=cb)` must parse an OpenAI-style SSE stream, hand each
|
||||||
|
content delta to the callback, and still return the complete text.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
|
|
||||||
|
import blitztext.llm as llm
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_stream_accumulates_and_calls_back():
|
||||||
|
sse = (
|
||||||
|
'data: {"choices":[{"delta":{"content":"Hel"}}]}\n'
|
||||||
|
"\n" # keep-alive blank line
|
||||||
|
'data: {"choices":[{"delta":{"content":"lo"}}]}\n'
|
||||||
|
'data: {"choices":[{"delta":{}}]}\n' # role-only / empty delta
|
||||||
|
": comment line\n" # SSE comment, ignored
|
||||||
|
"data: [DONE]\n"
|
||||||
|
'data: {"choices":[{"delta":{"content":"X"}}]}\n' # after DONE -> ignored
|
||||||
|
)
|
||||||
|
tokens = []
|
||||||
|
out = llm._read_stream(io.BytesIO(sse.encode("utf-8")), tokens.append)
|
||||||
|
|
||||||
|
assert out == "Hello"
|
||||||
|
assert tokens == ["Hel", "lo"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_stream_survives_callback_errors():
|
||||||
|
sse = (
|
||||||
|
'data: {"choices":[{"delta":{"content":"a"}}]}\n'
|
||||||
|
'data: {"choices":[{"delta":{"content":"b"}}]}\n'
|
||||||
|
"data: [DONE]\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
def boom(_delta):
|
||||||
|
raise RuntimeError("UI exploded")
|
||||||
|
|
||||||
|
# A failing UI callback must not break accumulation / delivery.
|
||||||
|
out = llm._read_stream(io.BytesIO(sse.encode("utf-8")), boom)
|
||||||
|
assert out == "ab"
|
||||||
Loading…
Reference in New Issue
Block a user