Add realtime streaming and Linux app docs
This commit is contained in:
parent
4667144b72
commit
457835929e
@ -1,17 +1,22 @@
|
||||
# Contributing
|
||||
|
||||
Thanks for taking a look at Blitztext macOS Preview.
|
||||
Thanks for taking a look at Blitztext App Linux.
|
||||
|
||||
This repository is intentionally a preview. Contributions should make it easier to learn from, build, fork, or safely extend.
|
||||
This repository is intentionally a preview. Contributions should make it easier to learn from, build, fork, test, or safely extend.
|
||||
|
||||
## Inspiration
|
||||
|
||||
This Linux app is inspired by [cmagnussen/blitztext-app](https://github.com/cmagnussen/blitztext-app). Please keep that credit intact when changing project-facing docs.
|
||||
|
||||
## Good First Contributions
|
||||
|
||||
- improve build instructions
|
||||
- improve Linux setup instructions
|
||||
- add current Linux screenshots
|
||||
- fix confusing UI text
|
||||
- improve error messages
|
||||
- add tests around parsing or quality filters
|
||||
- document local model experiments
|
||||
- simplify setup
|
||||
- add tests around config parsing, routing, quality filters, and streaming URL handling
|
||||
- document known-good STT or LLM engine configs
|
||||
- simplify packaging and first-run setup
|
||||
|
||||
## Before Opening A Pull Request
|
||||
|
||||
@ -27,24 +32,33 @@ Keep changes small when possible. Avoid unrelated cleanup in the same PR.
|
||||
## Local Build
|
||||
|
||||
```bash
|
||||
./build.sh --debug
|
||||
cd linux
|
||||
./install.sh
|
||||
.venv/bin/python -m blitztext gui
|
||||
```
|
||||
|
||||
Package build:
|
||||
|
||||
```bash
|
||||
cd linux
|
||||
bash packaging/build-deb.sh
|
||||
```
|
||||
|
||||
## Security And Privacy
|
||||
|
||||
- Never commit API keys, tokens, private audio, or confidential transcripts.
|
||||
- Never commit API keys, tokens, private audio, confidential transcripts, or private endpoint URLs.
|
||||
- Avoid adding telemetry, hosted services, or external dependencies without a clear issue first.
|
||||
- Call out privacy-impacting changes in the pull request description.
|
||||
- Keep the preview honest: do not describe remote OpenAI workflows as offline or local.
|
||||
- Keep the preview honest: do not describe remote STT or rewrite workflows as offline or local.
|
||||
|
||||
## Project Boundaries
|
||||
|
||||
This preview currently does not include:
|
||||
|
||||
- other platforms
|
||||
- a hosted backend
|
||||
- packaged releases
|
||||
- bundled local model files
|
||||
- local text rewriting
|
||||
- production support
|
||||
- bundled STT model files
|
||||
- guaranteed Wayland support
|
||||
- local text rewriting unless the user configures a local OpenAI-compatible LLM endpoint
|
||||
|
||||
Those can be discussed in issues, but please keep PRs focused on the current macOS preview unless a maintainer agrees on a larger direction first.
|
||||
Those can be discussed in issues, but please keep PRs focused unless a maintainer agrees on a larger direction first.
|
||||
|
||||
174
README.md
174
README.md
@ -1,137 +1,151 @@
|
||||
# Blitztext App
|
||||
# Blitztext App Linux
|
||||
|
||||
Blitztext App is an experimental open-source macOS menubar app for turning speech into text.
|
||||
Blitztext App Linux is an experimental open-source Linux dictation app: focus any text field, press a hotkey, speak, and the text is typed back into the app you were using. It can also rewrite rough speech with an OpenAI-compatible LLM, or write live words through a Riva/NIM realtime STT server.
|
||||
|
||||
It is intentionally small and unfinished. The goal is to make a real workflow visible and hackable: press a hotkey, speak, get text back, optionally rewrite it, and paste it into the app you were using.
|
||||
This is a native host tool, not a browser app and not a hosted service. It is intentionally small, inspectable, and still rough around the edges.
|
||||
|
||||
This is a learning and experimentation project, not a polished product.
|
||||
> Preview status: Linux/X11, bring your own local models or endpoints, no hosted Blitztext backend, no warranty, no support guarantee.
|
||||
|
||||
> Preview status: bring your own OpenAI API key, no hosted backend, no warranty, no support guarantee.
|
||||
## Inspiration
|
||||
|
||||
Blitztext App Linux is inspired by [cmagnussen/blitztext-app](https://github.com/cmagnussen/blitztext-app), the original macOS menu-bar workflow for turning speech into text and cleaner writing. This Linux version keeps the spirit of that workflow while using Linux-native pieces: GTK, AppIndicator, global hotkeys, local `faster-whisper`, optional Riva/NIM realtime STT, and `xdotool` delivery.
|
||||
|
||||
## What It Does
|
||||
|
||||
- **Blitztext**: record speech and transcribe it.
|
||||
- **Blitztext+**: record speech, transcribe it, then turn the rough draft into cleaner writing.
|
||||
- **Blitztext $%&!**: turn frustrated speech into a calmer message.
|
||||
- **Blitztext :)**: add fitting emojis to dictated text.
|
||||
- **Dictate**: record speech and type the raw transcript into the focused field.
|
||||
- **Rewrite**: transcribe speech, send it to an OpenAI-compatible LLM, and type the improved result.
|
||||
- **Calm down / email / emoji workflows**: use configurable prompts for common writing transformations.
|
||||
- **Realtime STT streaming**: stream mic audio to a Riva/NIM realtime server, such as Nemotron ASR Streaming, and type stable words while you speak.
|
||||
- **Benchmark STT engines**: compare local and remote batch transcription engines against a reference clip.
|
||||
|
||||
## Important Preview Notes
|
||||
|
||||
- macOS only.
|
||||
- Bring your own OpenAI API key.
|
||||
- Linux first; tested on Ubuntu/GNOME with X11.
|
||||
- Auto-typing uses `xdotool`, so Wayland needs future `wtype`/`ydotool` support.
|
||||
- Batch transcription can run locally with `faster-whisper`.
|
||||
- Live streaming can use a local Riva/NIM realtime WebSocket endpoint.
|
||||
- Rewrite workflows call the OpenAI-compatible LLM endpoint you configure.
|
||||
- No hosted Blitztext backend is included or provided.
|
||||
- In online mode, audio and text are sent directly from the app to the OpenAI API.
|
||||
- Optional local transcription via WhisperKit/CoreML if you install a compatible model locally.
|
||||
- `./build.sh` creates a locally ad-hoc-signed development app. No notarized release binary is provided.
|
||||
- Not production ready.
|
||||
- Debian packaging exists for local installation, but this is still preview software.
|
||||
- No warranty and no support guarantee.
|
||||
|
||||
You are welcome to use, fork, adapt, and share this project under the license terms.
|
||||
|
||||
The intent is not to ship a one-click finished app. The intent is to make a real AI workflow understandable: clone it, build it, read the code, change it, break it, fix it, and suggest improvements. If you only want to download something and never look inside, this preview will probably feel rough. If you want to learn how a small native macOS AI app is put together, you are in the right place.
|
||||
The intent is not to ship a one-click finished product. The intent is to make a real AI workflow understandable: clone it, build it, read the code, change it, break it, fix it, and suggest improvements. If you only want to download something and never look inside, this preview will probably feel rough. If you want to learn how a small native Linux AI dictation tool is put together, you are in the right place.
|
||||
|
||||
## Screenshots
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="docs/screenshots/online-mode.png" alt="Blitztext online transcription mode" width="420"></td>
|
||||
<td><img src="docs/screenshots/local-mode.png" alt="Blitztext secure local transcription mode" width="420"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><img src="docs/screenshots/local-model-picker.png" alt="Blitztext local model picker" width="420"></td>
|
||||
<td><img src="docs/screenshots/settings-customize.png" alt="Blitztext settings and customization view" width="420"></td>
|
||||
</tr>
|
||||
</table>
|
||||
The old macOS screenshots were removed from this Linux-facing README so the docs do not misrepresent the app. Current Linux screenshots should show:
|
||||
|
||||
- the GTK control panel with workflow rows
|
||||
- Settings > Engines with local, OpenAI-compatible, and `riva_realtime` STT engines
|
||||
- Settings > Benchmark results
|
||||
- Settings > About with version, source, changelog, and license
|
||||
|
||||
The Linux icon assets live in [`linux/packaging`](linux/packaging/).
|
||||
|
||||
## Requirements
|
||||
|
||||
- macOS 14 or newer
|
||||
- Xcode 16 or newer (Swift 5.10), with Command Line Tools installed and selected for `xcodebuild`
|
||||
- [XcodeGen](https://github.com/yonaskolb/XcodeGen) to generate the Xcode project
|
||||
- For online transcription and rewriting: an OpenAI API key with access to:
|
||||
- `whisper-1` for transcription
|
||||
- `gpt-4o-mini` and optionally `gpt-4o` for rewriting
|
||||
- For local-only transcription: a WhisperKit CoreML model in:
|
||||
`~/Library/Application Support/Blitztext/models/whisperkit/`
|
||||
- Linux desktop with an **X11 session**.
|
||||
- Host tools: `xdotool`, `notify-send` from `libnotify-bin`, and one recorder: `pw-record`, `parecord`, or `arecord`.
|
||||
- Python 3.11+ when running from source.
|
||||
- Optional rewrite workflows: an OpenAI-compatible chat endpoint and API key if needed.
|
||||
- Optional realtime STT streaming: a Riva/NIM realtime server reachable through `/v1/realtime`.
|
||||
|
||||
The build also pulls one Swift Package dependency automatically:
|
||||
|
||||
- [`argmax-oss-swift`](https://github.com/argmaxinc/argmax-oss-swift) (WhisperKit) — used for local on-device transcription.
|
||||
|
||||
Install XcodeGen if needed:
|
||||
On Ubuntu/Debian:
|
||||
|
||||
```bash
|
||||
brew install xcodegen
|
||||
sudo apt install xdotool libnotify-bin pipewire-bin python3-gi
|
||||
```
|
||||
|
||||
## Build And Run
|
||||
## Install And Run
|
||||
|
||||
Recommended local package path:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/cmagnussen/blitztext-app.git
|
||||
cd blitztext-app
|
||||
./build.sh --run
|
||||
cd linux
|
||||
bash packaging/build-deb.sh
|
||||
sudo apt install ./dist/blitztext_*.deb
|
||||
blitztext tray
|
||||
```
|
||||
|
||||
For a local install into `/Applications`:
|
||||
Run from source:
|
||||
|
||||
```bash
|
||||
./build.sh --install --run
|
||||
cd linux
|
||||
./install.sh
|
||||
.venv/bin/python -m blitztext tray
|
||||
```
|
||||
|
||||
The generated `.app` is ad-hoc signed for local development only. Do not treat it as a trusted redistributable binary. A public binary release would need Developer ID signing and notarization.
|
||||
Other entry points:
|
||||
|
||||
On first launch, either paste your own OpenAI API key for online workflows or install a WhisperKit CoreML model for local transcription. Rewriting workflows still require OpenAI.
|
||||
```bash
|
||||
.venv/bin/python -m blitztext gui
|
||||
.venv/bin/python -m blitztext run
|
||||
.venv/bin/python -m blitztext config-path
|
||||
.venv/bin/python -m blitztext --version
|
||||
```
|
||||
|
||||
For fully local transcription, install a WhisperKit CoreML model and enable **Sicherer Lokaler Modus** in the app.
|
||||
|
||||
For a slower, more explicit walkthrough, see [docs/setup.md](docs/setup.md).
|
||||
|
||||
## Permissions
|
||||
|
||||
Blitztext asks for:
|
||||
|
||||
- **Microphone**: to record your voice.
|
||||
- **Accessibility**: to paste the result back into the app you were using.
|
||||
|
||||
If you do not grant Accessibility permission, you can still copy results manually.
|
||||
|
||||
Full Disk Access is not required. If auto-paste does not work even though transcription succeeds, open **System Settings -> Privacy & Security -> Accessibility**, enable Blitztext there, restart Blitztext, and try again with the cursor focused in a text field. If macOS shows multiple Blitztext entries, remove or disable the old ones and grant the permission to the app you just built or installed.
|
||||
For the full Linux guide, see [linux/README.md](linux/README.md). For setup details, see [docs/setup.md](docs/setup.md).
|
||||
|
||||
## Data Flow
|
||||
|
||||
The preview has no custom backend.
|
||||
|
||||
```text
|
||||
Online transcription: Your Mac -> OpenAI Audio Transcriptions API
|
||||
Text rewriting: Your Mac -> OpenAI Chat Completions API
|
||||
Local transcription: Your Mac -> WhisperKit/CoreML on device
|
||||
Batch transcription: Your Linux desktop -> local faster-whisper or your configured STT endpoint
|
||||
Realtime streaming: Your Linux desktop -> your configured Riva/NIM realtime endpoint
|
||||
Text rewriting: Your Linux desktop -> your configured OpenAI-compatible chat endpoint
|
||||
Text delivery: Blitztext -> xdotool -> focused X11 window
|
||||
```
|
||||
|
||||
The app stores your OpenAI API key in the user's macOS Keychain.
|
||||
|
||||
Read [docs/privacy.md](docs/privacy.md) before using the preview with sensitive content.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
BlitztextMac/
|
||||
App/ App lifecycle and paste handling
|
||||
Features/ Workflows, menu bar UI, settings
|
||||
Services/ Recording, OpenAI calls, hotkeys, local storage
|
||||
Views/ Shared SwiftUI views
|
||||
build.sh Local build script
|
||||
docs/ Setup, privacy, roadmap, preflight, landing page notes
|
||||
linux/
|
||||
blitztext/ Linux app package: GTK UI, daemon, STT/LLM engines, config
|
||||
packaging/ Debian package script, desktop entry, icon assets
|
||||
README.md Detailed Linux guide
|
||||
BlitztextMac/ Legacy/upstream macOS app code kept for reference
|
||||
build.sh Legacy macOS build script
|
||||
README.md Linux-first project overview
|
||||
docs/ Setup, privacy, roadmap, release notes, web brief
|
||||
```
|
||||
|
||||
## Local Models
|
||||
## Configuration
|
||||
|
||||
Local transcription is available as an experimental WhisperKit/CoreML path. The app does not bundle a model; choose one in the app, click install, and then switch on **Sicherer Lokaler Modus** from the menu bar or settings.
|
||||
Blitztext writes configuration to:
|
||||
|
||||
See [docs/local-models.md](docs/local-models.md).
|
||||
```text
|
||||
~/.config/blitztext/config.toml
|
||||
```
|
||||
|
||||
The Settings window can edit workflows, STT engines, LLM engines, input mode, microphone, language, benchmark clips, and About metadata. You can also edit the TOML directly.
|
||||
|
||||
## Realtime STT Streaming
|
||||
|
||||
For Nemotron ASR Streaming, add a realtime engine in **Settings > Engines** with `+ Stream`, save/restart, then create or edit a workflow with `mode = "stream"`.
|
||||
|
||||
```toml
|
||||
[[stt_engine]]
|
||||
name = "Nemotron ASR Streaming"
|
||||
type = "riva_realtime"
|
||||
url = "http://127.0.0.1:8006/v1"
|
||||
model = ""
|
||||
|
||||
[[workflow]]
|
||||
name = "STT Streaming"
|
||||
hotkey = "<ctrl>+<alt>+s"
|
||||
mode = "stream"
|
||||
```
|
||||
|
||||
The tested Nemotron ASR Streaming NIM exposes an English `en-US` model, so use `language = "en"` or `language = "en-US"` in `[general]` for that engine.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome, especially if they make the preview easier to build, understand, or fork.
|
||||
Contributions are welcome, especially if they make the preview easier to build, understand, test, or fork.
|
||||
|
||||
Please read [CONTRIBUTING.md](CONTRIBUTING.md) first.
|
||||
|
||||
@ -153,5 +167,5 @@ This is an experimental, non-commercial open-source project, provided as-is unde
|
||||
|
||||
The companion website (blitztext.de) is operated by Blackboat Internet GmbH:
|
||||
|
||||
- Impressum: https://www.blackboat.com/impressum
|
||||
- Datenschutz / Privacy: https://www.blackboat.com/datenschutz
|
||||
- Impressum: https://martin-bierschenk.de/impressum/
|
||||
- Datenschutz / Privacy: https://martin-bierschenk.de/datenschutz/
|
||||
|
||||
36
ROADMAP.md
36
ROADMAP.md
@ -4,28 +4,32 @@ This is a preview roadmap, not a promise.
|
||||
|
||||
## Current Scope
|
||||
|
||||
- macOS menubar app
|
||||
- local recording and hotkeys
|
||||
- direct OpenAI API calls with a user-provided API key
|
||||
- transcription, rewriting, calmer-message, and emoji workflows
|
||||
- no hosted backend
|
||||
- no other platforms
|
||||
- no packaged public release
|
||||
- Linux/X11 native dictation app
|
||||
- GTK control panel and AppIndicator tray
|
||||
- global hotkeys and modifier input mode
|
||||
- local batch transcription through `faster-whisper`
|
||||
- OpenAI-compatible batch STT endpoints
|
||||
- Riva/NIM realtime STT streaming through WebSocket
|
||||
- optional rewrite workflows through OpenAI-compatible chat endpoints
|
||||
- xdotool typing or clipboard paste into the focused window
|
||||
- Debian package for local Ubuntu/Debian installs
|
||||
- no hosted Blitztext backend
|
||||
|
||||
## Next Useful Work
|
||||
|
||||
- Make first-run setup clearer.
|
||||
- Improve credential setup, validation, and recovery UX.
|
||||
- Add a small automated test layer around prompt construction and text quality filters.
|
||||
- Add provider boundaries so OpenAI and future local transcription can be swapped more cleanly.
|
||||
- Prototype local transcription with WhisperKit or whisper.cpp.
|
||||
- Reduce the Accessibility blast radius, ideally by moving synthetic paste into a smaller helper with narrower responsibilities.
|
||||
- Add stronger supply-chain checks around downloaded local speech models.
|
||||
- Add signed and notarized release builds when the project is ready for non-developer users.
|
||||
- Capture and add current Linux screenshots for the control panel, settings, benchmark, and About tab.
|
||||
- Improve first-run setup for STT engines, microphones, and output mode.
|
||||
- Add a guided realtime STT streaming test that does not type into the active window.
|
||||
- Add automated tests around config parsing, routing, quality filters, URL handling, and streaming protocol message generation.
|
||||
- Improve Wayland support with `wtype` or `ydotool` as alternatives to `xdotool`.
|
||||
- Add safer clipboard handling and clearer output-mode diagnostics.
|
||||
- Add a lightweight release checklist for `.deb` builds and source installs.
|
||||
- Document known-good local LLM and STT server configurations.
|
||||
|
||||
## Not In Scope Yet
|
||||
|
||||
- Production support.
|
||||
- Accounts, sync, teams, or hosted infrastructure.
|
||||
- Claims that the app is offline or privacy-complete.
|
||||
- Claims that the app is offline or privacy-complete by default.
|
||||
- App Store distribution.
|
||||
- A polished one-click consumer release.
|
||||
|
||||
19
SECURITY.md
19
SECURITY.md
@ -1,6 +1,6 @@
|
||||
# Security Policy
|
||||
|
||||
Blitztext macOS Preview is experimental software.
|
||||
Blitztext App Linux is experimental software.
|
||||
|
||||
It is provided as-is, without warranty, support guarantees, or production-readiness claims.
|
||||
|
||||
@ -12,11 +12,9 @@ Only the current `main` branch is considered for security fixes.
|
||||
|
||||
Please do not open a public issue with sensitive security details.
|
||||
|
||||
Use GitHub private vulnerability reporting for this repository. Maintainers should enable it before making the repository public.
|
||||
Use GitHub private vulnerability reporting for this repository. If private vulnerability reporting is not available yet, open a minimal public issue titled `Security contact request` without technical details.
|
||||
|
||||
If private vulnerability reporting is not available yet, open a minimal public issue titled `Security contact request` without technical details.
|
||||
|
||||
Do not include OpenAI API keys, access tokens, private recordings, or confidential transcripts in a report.
|
||||
Do not include API keys, access tokens, private recordings, confidential transcripts, screenshots with sensitive text, or private endpoint URLs in a report.
|
||||
|
||||
Include:
|
||||
|
||||
@ -27,10 +25,11 @@ Include:
|
||||
|
||||
## Security Notes
|
||||
|
||||
- The app sends audio and text directly to OpenAI when you use the remote workflows.
|
||||
- Your OpenAI API key is stored in the user's macOS Keychain.
|
||||
- Temporary audio files may exist briefly during processing.
|
||||
- Accessibility permission allows the app to paste text into the current app.
|
||||
- The app currently runs **without** the macOS App Sandbox. This is a deliberate trade-off for the preview: the menubar workflow needs Accessibility-based paste into arbitrary frontmost apps, system-wide hotkeys, and Application Support paths for local WhisperKit models, all of which are awkward or impossible inside a strict sandbox. Hardened Runtime is enabled, and the entitlements are limited to microphone input and outbound network access. Reintroducing the sandbox is on the roadmap once these flows are reworked.
|
||||
- Blitztext types into the focused X11 window through `xdotool` or uses the clipboard, depending on output mode.
|
||||
- Global hotkeys and synthetic typing are powerful desktop interactions; review the code before using it with sensitive workflows.
|
||||
- Batch transcription may create temporary audio files during processing; the app attempts to delete them when the workflow ends or is cancelled.
|
||||
- Remote STT, realtime STT, and rewrite workflows send data to the endpoints you configure.
|
||||
- The app does not store API keys directly; config entries name environment variables such as `OPENAI_API_KEY`.
|
||||
- The app does not include a hosted backend.
|
||||
|
||||
Do not use this preview for confidential or regulated data without your own review.
|
||||
|
||||
14
SUPPORT.md
14
SUPPORT.md
@ -1,13 +1,14 @@
|
||||
# Support
|
||||
|
||||
Blitztext App is an experimental preview. There is no service-level agreement, paid support channel, or guarantee that issues will be fixed.
|
||||
Blitztext App Linux is an experimental preview. There is no service-level agreement, paid support channel, or guarantee that issues will be fixed.
|
||||
|
||||
## Before Asking For Help
|
||||
|
||||
- Make sure you can build the app with `./build.sh --debug`.
|
||||
- Check that your OpenAI API key is entered in the app settings.
|
||||
- Confirm that macOS microphone permission is granted.
|
||||
- Grant Accessibility permission if you expect automatic paste into other apps.
|
||||
- Make sure you can install from `linux/install.sh` or build the `.deb` with `linux/packaging/build-deb.sh`.
|
||||
- Confirm you are running an X11 session if you expect automatic typing through `xdotool`.
|
||||
- Check that your microphone works and that the Settings input meter moves.
|
||||
- Check that your selected STT engine is the right type: `local`, `openai`, or `riva_realtime`.
|
||||
- For rewrite workflows, confirm your OpenAI-compatible LLM endpoint and API key environment variable.
|
||||
- Read [docs/privacy.md](docs/privacy.md) before testing with sensitive content.
|
||||
|
||||
## Where To Ask
|
||||
@ -16,8 +17,9 @@ Use GitHub Issues for reproducible bugs and focused feature ideas.
|
||||
|
||||
Please do not post:
|
||||
|
||||
- OpenAI API keys
|
||||
- API keys
|
||||
- access tokens
|
||||
- private endpoint URLs
|
||||
- private audio recordings
|
||||
- confidential transcripts
|
||||
- screenshots that show sensitive content
|
||||
|
||||
@ -2,6 +2,8 @@
|
||||
|
||||
The source code in this repository is licensed under the MIT License.
|
||||
|
||||
The project name, app name, logos, icons, and visual identity are not granted as trademarks or brand assets by the MIT License.
|
||||
Project names, logos, app icons, screenshots, and visual identity are not automatically granted as trademarks or brand assets by the MIT License.
|
||||
|
||||
You may fork the code under the license terms. If you publish a fork as a separate app or service, use your own name, icon, and branding unless you have explicit permission.
|
||||
|
||||
This project credits [cmagnussen/blitztext-app](https://github.com/cmagnussen/blitztext-app) as inspiration. That credit is not a trademark grant from the original project.
|
||||
|
||||
@ -29,5 +29,5 @@ Protect `main`:
|
||||
|
||||
- Keep Issues enabled for bugs and focused requests.
|
||||
- Enable Discussions only if you want a lower-friction place for questions.
|
||||
- Set repository topics after the project is public.
|
||||
- Set repository topics after the project is public: `linux`, `dictation`, `speech-to-text`, `gtk`, `x11`, `faster-whisper`, `riva`, `nim`, `openai-compatible`.
|
||||
- Review the GitHub community profile before sharing the repo widely.
|
||||
|
||||
@ -1,22 +1,22 @@
|
||||
# Landing Page Brief
|
||||
|
||||
Domain: `blitztext.app`
|
||||
Domain: `blitztext.de`
|
||||
|
||||
Goal: a very small landing page for an experimental open-source macOS preview.
|
||||
Goal: a small, honest landing page for an experimental open-source Linux dictation preview.
|
||||
|
||||
## Hero
|
||||
|
||||
Headline:
|
||||
|
||||
> Blitztext macOS Preview
|
||||
> Blitztext App Linux
|
||||
|
||||
Subline:
|
||||
|
||||
> Speak your thoughts. Turn them into text, cleaner writing, or calmer messages.
|
||||
> Speak into any focused Linux text field. Get text, cleaner writing, or live streaming transcripts.
|
||||
|
||||
Body:
|
||||
|
||||
> An experimental open-source macOS menubar app. Not finished, not hosted, not plug-and-play. Built to learn from, fork, and improve.
|
||||
> An experimental open-source Linux/X11 dictation app inspired by the original macOS Blitztext workflow. Not finished, not hosted, not plug-and-play. Built to learn from, fork, and improve.
|
||||
|
||||
Primary CTA:
|
||||
|
||||
@ -28,55 +28,61 @@ Secondary CTA:
|
||||
|
||||
Small line:
|
||||
|
||||
> Bring your own OpenAI API key. Optional local transcription. No hosted Blitztext backend.
|
||||
> Local faster-whisper, optional Riva/NIM realtime STT, optional OpenAI-compatible rewriting. No hosted Blitztext backend.
|
||||
|
||||
## Sections
|
||||
|
||||
1. What it does
|
||||
- Dictate
|
||||
- Improve
|
||||
- Calm down
|
||||
- Dictate into the focused app
|
||||
- Improve or rewrite rough speech
|
||||
- Calm down messages
|
||||
- Add emojis
|
||||
- Stream live STT through Riva/NIM
|
||||
|
||||
2. How it works
|
||||
- Build the app locally
|
||||
- Paste your own OpenAI API key
|
||||
- Install the Linux app or run from source
|
||||
- Configure a local or remote STT engine
|
||||
- Press a hotkey and speak
|
||||
- Get text back on the clipboard
|
||||
- Blitztext types into the focused X11 window
|
||||
|
||||
3. Open-source preview
|
||||
- macOS-only
|
||||
- Linux/X11 first
|
||||
- MIT License
|
||||
- inspired by `cmagnussen/blitztext-app`
|
||||
- experimental
|
||||
- no warranty
|
||||
- optional local transcription with user-installed WhisperKit models
|
||||
- no hosted backend
|
||||
|
||||
4. Privacy, plainly
|
||||
- online workflows send audio and text to OpenAI
|
||||
- secure local mode keeps transcription on device
|
||||
- local batch STT can stay on device
|
||||
- realtime STT goes to the Riva/NIM endpoint you configure
|
||||
- rewriting goes to the OpenAI-compatible endpoint you configure
|
||||
- no public Blitztext backend
|
||||
- rewriting still uses OpenAI
|
||||
- do your own review before sensitive use
|
||||
|
||||
5. Roadmap
|
||||
- easier setup
|
||||
- signed releases
|
||||
- local transcription experiments
|
||||
- clearer setup and security docs
|
||||
- community issues
|
||||
- clearer setup
|
||||
- current Linux screenshots
|
||||
- Wayland support exploration
|
||||
- better streaming diagnostics
|
||||
- basic tests and release checks
|
||||
|
||||
## Do Not Promise
|
||||
|
||||
- offline use
|
||||
- production readiness
|
||||
- hosted service
|
||||
- free usage without API costs
|
||||
- no data leaves the device
|
||||
- free usage without API or infrastructure costs
|
||||
- no data leaves the device by default
|
||||
- guaranteed support
|
||||
- other platforms
|
||||
- bundled local models
|
||||
- local rewriting
|
||||
- macOS support in the Linux app
|
||||
- bundled STT models
|
||||
- local rewriting unless the user configures a local LLM endpoint
|
||||
|
||||
## Visual Direction
|
||||
|
||||
Use a real macOS screenshot or short demo GIF. Keep the page calm, sparse, and honest. Avoid fake metrics, oversized SaaS claims, and corporate origin story.
|
||||
Use real Linux screenshots or a short demo GIF: GTK control panel, Settings > Engines, Benchmark, and About. Keep the page calm, sparse, and honest. Avoid fake metrics, oversized SaaS claims, and corporate origin story.
|
||||
|
||||
## Legal Links
|
||||
|
||||
- Impressum: https://martin-bierschenk.de/impressum/
|
||||
- Datenschutz / Privacy: https://martin-bierschenk.de/datenschutz/
|
||||
|
||||
@ -1,72 +1,78 @@
|
||||
# Local Models
|
||||
# Local And Realtime Speech Models
|
||||
|
||||
Blitztext can run transcription locally with WhisperKit/CoreML. The app does not bundle a speech model, but it can download the selected compatible model from Hugging Face into the local cache.
|
||||
Blitztext App Linux supports two local-first speech paths:
|
||||
|
||||
## Recommended First Model
|
||||
- in-process `faster-whisper` for batch transcription
|
||||
- external Riva/NIM realtime servers for live streaming transcription
|
||||
|
||||
Use Whisper Small for the first local test. It is multilingual, supports German, and is much lighter than the large variants.
|
||||
The app does not bundle speech models. You choose the model in Settings or in `~/.config/blitztext/config.toml`.
|
||||
|
||||
- [argmaxinc/whisperkit-coreml: openai_whisper-small_216MB](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-small_216MB)
|
||||
## Local Batch Transcription
|
||||
|
||||
Local cache path:
|
||||
The default local engine uses `faster-whisper`.
|
||||
|
||||
Recommended first model:
|
||||
|
||||
```toml
|
||||
[whisper]
|
||||
model = "small"
|
||||
device = "auto"
|
||||
compute_type = "auto"
|
||||
```
|
||||
|
||||
Useful model sizes:
|
||||
|
||||
- `tiny`: fastest, lowest quality
|
||||
- `base`: small and responsive
|
||||
- `small`: good first default for dictation
|
||||
- `medium`: better quality, slower
|
||||
- `large-v3`: highest quality, much heavier
|
||||
|
||||
You can also use a local model path supported by `faster-whisper`.
|
||||
|
||||
## Realtime Riva/NIM Transcription
|
||||
|
||||
For live words while speaking, use a `riva_realtime` STT engine. The tested Nemotron ASR Streaming NIM exposes a WebSocket endpoint through `/v1/realtime` and reports this model:
|
||||
|
||||
```text
|
||||
~/Library/Application Support/Blitztext/models/whisperkit/openai_whisper-small_216MB
|
||||
cache-aware-parakeet-rnnt-en-US-asr-streaming-sortformer
|
||||
```
|
||||
|
||||
## Other Compatible Models
|
||||
Recommended engine config:
|
||||
|
||||
You can also install larger WhisperKit CoreML models into the same cache directory:
|
||||
|
||||
- [openai_whisper-large-v3-v20240930_626MB](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-large-v3-v20240930_626MB)
|
||||
- [openai_whisper-large-v3-v20240930_turbo_632MB](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-large-v3-v20240930_turbo_632MB)
|
||||
|
||||
The app detects installed model folders that contain `AudioEncoder.mlmodelc`, `MelSpectrogram.mlmodelc`, and `TextDecoder.mlmodelc`.
|
||||
|
||||
## Install From The App
|
||||
|
||||
Open Blitztext, go to **Settings > Anpassen**, choose a local model, and click **Installieren**. You can also switch on **Sicherer Lokaler Modus** from the main popover; if the selected model is missing, Blitztext starts the download and installs it into the local cache.
|
||||
|
||||
After the model is installed, the Blitztext transcription workflow can run in local mode. The rewriting workflows still use OpenAI, so they are paused while secure local mode is active.
|
||||
|
||||
## Optional Manual Install
|
||||
|
||||
If you prefer the CLI path, install the Hugging Face CLI so the `hf` command is available:
|
||||
|
||||
```bash
|
||||
python3 -m pip install --upgrade "huggingface_hub[cli]"
|
||||
```toml
|
||||
[[stt_engine]]
|
||||
name = "Nemotron ASR Streaming"
|
||||
type = "riva_realtime"
|
||||
url = "http://127.0.0.1:8006/v1"
|
||||
model = ""
|
||||
```
|
||||
|
||||
Create the local model cache:
|
||||
Use `model = ""` to keep the server default. For the tested Nemotron container, set the general language to English:
|
||||
|
||||
```bash
|
||||
mkdir -p "$HOME/Library/Application Support/Blitztext/models/whisperkit"
|
||||
```toml
|
||||
[general]
|
||||
language = "en"
|
||||
```
|
||||
|
||||
Download the recommended first model:
|
||||
## Batch NIMs And Other STT Servers
|
||||
|
||||
```bash
|
||||
hf download argmaxinc/whisperkit-coreml \
|
||||
--include 'openai_whisper-small_216MB/*' \
|
||||
--local-dir "$HOME/Library/Application Support/Blitztext/models/whisperkit" \
|
||||
--max-workers 4
|
||||
Use `type = "openai"` only for servers that implement batch `/v1/audio/transcriptions` correctly.
|
||||
|
||||
Example:
|
||||
|
||||
```toml
|
||||
[[stt_engine]]
|
||||
name = "Parakeet batch ASR"
|
||||
type = "openai"
|
||||
url = "http://127.0.0.1:8090/v1"
|
||||
model = "parakeet-tdt-0.6b-v3"
|
||||
```
|
||||
|
||||
Expected folder layout:
|
||||
|
||||
```text
|
||||
~/Library/Application Support/Blitztext/models/whisperkit/
|
||||
openai_whisper-small_216MB/
|
||||
AudioEncoder.mlmodelc/
|
||||
MelSpectrogram.mlmodelc/
|
||||
TextDecoder.mlmodelc/
|
||||
```
|
||||
|
||||
If the folder is nested differently, the app will not detect the model.
|
||||
Streaming-only NIMs may still show `/v1/audio/transcriptions` in Swagger, but return `bad model` or `No Offline ASR models found`. Use `riva_realtime` for those.
|
||||
|
||||
## Notes
|
||||
|
||||
- First use can be slower because the model has to load and prewarm.
|
||||
- Local transcription avoids sending audio to OpenAI for the Blitztext workflow.
|
||||
- The app currently supports local transcription only, not local rewriting.
|
||||
- Models are downloaded on demand so the repository and app package stay small and auditable.
|
||||
- First local Whisper use can be slower because the model has to load or download.
|
||||
- Realtime streaming needs `sounddevice` and `websockets`; both are in `linux/requirements.txt`.
|
||||
- The benchmark tab is for batch engines. Streaming engines are live-only and are not benchmarked with WAV uploads.
|
||||
|
||||
@ -1,15 +1,18 @@
|
||||
# Open Source Preflight
|
||||
|
||||
Use this checklist before making the repository public.
|
||||
Use this checklist before making `blitztext-app-linux` public or cutting a public preview release.
|
||||
|
||||
## P0 Before Public
|
||||
|
||||
- Run a local build with `./build.sh --debug`.
|
||||
- Run a local source install from `linux/install.sh`.
|
||||
- Run `linux/.venv/bin/python -m py_compile` across the app package.
|
||||
- Build a `.deb` with `linux/packaging/build-deb.sh` and install it on a clean Ubuntu/Debian test machine.
|
||||
- Run a secret scan across the working tree and commit history.
|
||||
- Confirm there are no private URLs, hosted backend credentials, internal docs, or old project references.
|
||||
- Keep the repository private until another maintainer has reviewed the first public commit.
|
||||
- Confirm the root `LICENSE`, `README.md`, `SECURITY.md`, `CONTRIBUTING.md`, and `SUPPORT.md` are present.
|
||||
- Make the preview status explicit: experimental, bring your own OpenAI API key, no hosted backend, no warranty.
|
||||
- Confirm there are no private URLs, hosted backend credentials, internal docs, private recordings, or confidential transcripts.
|
||||
- Confirm old macOS-only claims have been replaced with Linux/X11 wording.
|
||||
- Confirm the root `LICENSE`, `README.md`, `SECURITY.md`, `CONTRIBUTING.md`, `SUPPORT.md`, and `TRADEMARKS.md` are present.
|
||||
- Keep the preview status explicit: experimental, no hosted backend, no warranty, no support guarantee.
|
||||
- Credit the inspiration project: `cmagnussen/blitztext-app`.
|
||||
- Enable GitHub private vulnerability reporting, secret scanning, and push protection before switching the repo public.
|
||||
- Enable Dependabot alerts.
|
||||
- Protect `main` with pull requests, at least one review, and required CI checks.
|
||||
@ -17,15 +20,15 @@ Use this checklist before making the repository public.
|
||||
|
||||
## P1 Soon After Public
|
||||
|
||||
- Enable private vulnerability reporting.
|
||||
- Decide whether Issues alone are enough or whether Discussions should be enabled for questions.
|
||||
- Add repository topics such as `macos`, `swift`, `menubar`, `speech-to-text`, and `openai`.
|
||||
- Add a lightweight release process only after the build is signed and notarized.
|
||||
- Add basic tests once provider boundaries are extracted.
|
||||
- Add repository topics such as `linux`, `dictation`, `speech-to-text`, `gtk`, `x11`, `faster-whisper`, `riva`, `nim`, and `openai-compatible`.
|
||||
- Add current Linux screenshots to `docs/screenshots/`.
|
||||
- Add a small test layer for config parsing, workflow routing, and streaming URL handling.
|
||||
- Add release notes for `.deb` artifacts.
|
||||
|
||||
## P2 Later
|
||||
|
||||
- Add CODEOWNERS if multiple maintainers become active.
|
||||
- Add local model cleanup after the in-app download/install flow.
|
||||
- Consider CodeQL once the repo has enough surface area to justify scheduled scans.
|
||||
- Add signed and notarized release artifacts for non-developer testers.
|
||||
- Add Wayland support notes or implementation.
|
||||
- Add signed release artifacts if the project becomes useful beyond developer previews.
|
||||
|
||||
@ -1,36 +1,46 @@
|
||||
# Privacy Notes
|
||||
|
||||
Blitztext macOS Preview does not include a hosted backend.
|
||||
Blitztext App Linux does not include a hosted backend.
|
||||
|
||||
When you use the online workflows, your Mac sends data directly to OpenAI:
|
||||
Data goes only to the engines and endpoints you configure:
|
||||
|
||||
- audio recordings for transcription
|
||||
- transcribed or typed text for rewriting
|
||||
- custom terms and prompt context if you configured them
|
||||
- local `faster-whisper` for local batch transcription
|
||||
- your configured OpenAI-compatible STT endpoint for remote batch transcription
|
||||
- your configured Riva/NIM realtime endpoint for streaming transcription
|
||||
- your configured OpenAI-compatible chat endpoint for rewriting workflows
|
||||
|
||||
When **Sicherer Lokaler Modus** is enabled and a WhisperKit/CoreML model is installed, transcription runs on your Mac and does not send audio to OpenAI. Rewriting workflows still require OpenAI and are paused while secure local mode is active.
|
||||
|
||||
You are responsible for your OpenAI account, API usage, costs, and data handling.
|
||||
You are responsible for API access, billing, endpoint security, and data handling for any remote or local service you connect.
|
||||
|
||||
## Local Data
|
||||
|
||||
The app stores:
|
||||
|
||||
- your OpenAI API key in the user's macOS Keychain
|
||||
- workflow settings in local app support storage
|
||||
- optional WhisperKit/CoreML model folders in local app support storage
|
||||
- temporary audio files while a transcription is being processed; the app attempts to delete each recording when the workflow ends or is cancelled
|
||||
- workflow, hotkey, engine, model, microphone, and UI settings in `~/.config/blitztext/config.toml`
|
||||
- temporary audio files while a batch transcription is being processed; the app attempts to delete each recording when the workflow ends or is cancelled
|
||||
- local Python dependencies in the source venv or bundled package venv
|
||||
|
||||
Workflow output may also be placed on your clipboard so it can be pasted into another app. Auto-paste marks the clipboard entry as concealed for compatible clipboard managers, but the generated text intentionally remains on the clipboard as a fallback if automatic paste is blocked. Clipboard managers, macOS, or other apps may still observe clipboard contents while they are present.
|
||||
Blitztext does not store API keys itself. Instead, config entries name environment variables such as `OPENAI_API_KEY`; you provide those variables in your shell, service, or desktop environment.
|
||||
|
||||
The app uses the system TLS trust store for OpenAI and Hugging Face requests. It does not currently pin certificates. A user-installed or managed root certificate can therefore affect HTTPS trust decisions on that Mac.
|
||||
Workflow output may be typed directly into the focused X11 window or placed on the clipboard, depending on the configured output mode. Clipboard managers and other apps may observe clipboard contents while they are present.
|
||||
|
||||
Settings such as custom prompts, custom terms, and context are stored in local app support storage as plain JSON. Do not put secrets into those fields.
|
||||
## Network Data Flow
|
||||
|
||||
```text
|
||||
Local batch STT: microphone -> temporary WAV -> local faster-whisper
|
||||
Remote batch STT: microphone -> temporary WAV -> configured /audio/transcriptions endpoint
|
||||
Realtime STT: microphone PCM chunks -> configured Riva/NIM realtime WebSocket endpoint
|
||||
Rewrite workflows: transcript text -> configured OpenAI-compatible chat endpoint
|
||||
Delivery: generated text -> xdotool / clipboard -> focused app
|
||||
```
|
||||
|
||||
The app uses your system trust store for HTTPS connections made by Python libraries. It does not pin certificates.
|
||||
|
||||
## Offline Scope
|
||||
|
||||
Only transcription can run locally. Any workflow that rewrites, improves, or transforms text still uses OpenAI.
|
||||
Batch transcription can run locally with `faster-whisper`. Realtime transcription can be local if your Riva/NIM server is local. Rewriting is local only if you configure a local OpenAI-compatible LLM endpoint.
|
||||
|
||||
Do not describe a workflow as fully offline unless every configured endpoint is local and you have verified the network path.
|
||||
|
||||
## Sensitive Content
|
||||
|
||||
Do not use this preview with confidential, regulated, or highly sensitive content unless you have reviewed the code, your OpenAI settings, and your legal/privacy requirements.
|
||||
Do not use this preview with confidential, regulated, or highly sensitive content unless you have reviewed the code, your local services, your remote provider settings, and your legal/privacy requirements.
|
||||
|
||||
137
docs/setup.md
137
docs/setup.md
@ -1,80 +1,115 @@
|
||||
# Setup
|
||||
|
||||
This guide is for people who want to build and inspect the preview themselves.
|
||||
This guide is for people who want to build and inspect Blitztext App Linux themselves.
|
||||
|
||||
## 1. Requirements
|
||||
|
||||
- macOS 14 or newer
|
||||
- Full Xcode, with Command Line Tools installed
|
||||
- XcodeGen
|
||||
- Homebrew, if you want to install XcodeGen with `brew install xcodegen`
|
||||
- Optional for online workflows: an OpenAI API key
|
||||
- Optional for secure local transcription: a local WhisperKit/CoreML model
|
||||
- Linux desktop with an X11 session
|
||||
- Python 3.11+
|
||||
- `xdotool`
|
||||
- `notify-send` from `libnotify-bin`
|
||||
- one recorder: `pw-record`, `parecord`, or `arecord`
|
||||
- GTK/PyGObject for the tray and settings UI (`python3-gi` on Ubuntu/Debian)
|
||||
- Optional rewrite workflows: an OpenAI-compatible chat endpoint and API key if needed
|
||||
- Optional realtime STT streaming: a Riva/NIM realtime server such as Nemotron ASR Streaming
|
||||
|
||||
Install XcodeGen manually if needed:
|
||||
On Ubuntu/Debian:
|
||||
|
||||
```bash
|
||||
brew install xcodegen
|
||||
sudo apt install xdotool libnotify-bin pipewire-bin python3-gi
|
||||
```
|
||||
|
||||
## 2. Clone And Build
|
||||
## 2. Clone And Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/cmagnussen/blitztext-app.git
|
||||
cd blitztext-app
|
||||
./build.sh --debug
|
||||
git clone https://github.com/mARTin-B78/blitztext-app-linux.git
|
||||
cd blitztext-app-linux/linux
|
||||
./install.sh
|
||||
```
|
||||
|
||||
To launch after building:
|
||||
If your local repository still uses the older `blitztext-app` name, the commands are the same once you `cd linux`.
|
||||
|
||||
## 3. Run
|
||||
|
||||
```bash
|
||||
./build.sh --run
|
||||
.venv/bin/python -m blitztext tray
|
||||
```
|
||||
|
||||
## 3. Configure OpenAI For Online Workflows
|
||||
Alternatives:
|
||||
|
||||
Open the app settings and paste your own OpenAI API key if you want online transcription or rewriting workflows.
|
||||
|
||||
The preview currently uses:
|
||||
|
||||
- `whisper-1` for transcription
|
||||
- `gpt-4o-mini` for lightweight rewriting
|
||||
- `gpt-4o` for the calmer-message workflow
|
||||
|
||||
You are responsible for API access, billing, and data handling in your own OpenAI account.
|
||||
|
||||
Never commit your API key into this repository, issues, logs, or screenshots.
|
||||
|
||||
You can skip this step if you only want to test local transcription with a local WhisperKit model.
|
||||
|
||||
## 4. Optional Local Transcription
|
||||
|
||||
To use secure local transcription, choose a compatible WhisperKit CoreML model in the app and click **Installieren**. Blitztext stores models in:
|
||||
|
||||
```text
|
||||
~/Library/Application Support/Blitztext/models/whisperkit/
|
||||
```bash
|
||||
.venv/bin/python -m blitztext gui
|
||||
.venv/bin/python -m blitztext run
|
||||
.venv/bin/python -m blitztext config-path
|
||||
```
|
||||
|
||||
Recommended first model: `openai_whisper-small_216MB`.
|
||||
## 4. Debian Package
|
||||
|
||||
See [local-models.md](local-models.md) for the exact command, model links, and expected folder layout.
|
||||
```bash
|
||||
cd linux
|
||||
bash packaging/build-deb.sh
|
||||
sudo apt install ./dist/blitztext_*.deb
|
||||
blitztext tray
|
||||
```
|
||||
|
||||
## 5. macOS Permissions
|
||||
The package installs Blitztext under `/opt/blitztext`, adds a desktop entry, and bundles the Python dependencies from `requirements.txt`.
|
||||
|
||||
The app needs Microphone permission to record audio.
|
||||
## 5. Configure STT Engines
|
||||
|
||||
For automatic paste into the previous app, grant Accessibility permission in macOS System Settings. Without it, you can still copy and paste manually.
|
||||
Open **Settings > Engines**.
|
||||
|
||||
Blitztext does not need Full Disk Access. Auto-paste uses the Accessibility permission because the app simulates Cmd+V after putting the result on the clipboard.
|
||||
Common options:
|
||||
|
||||
- `local`: in-process `faster-whisper`
|
||||
- `openai`: OpenAI-compatible batch `/audio/transcriptions` endpoint
|
||||
- `riva_realtime`: Riva/NIM realtime WebSocket transcription for `mode = "stream"`
|
||||
|
||||
For Nemotron ASR Streaming:
|
||||
|
||||
```toml
|
||||
[[stt_engine]]
|
||||
name = "Nemotron ASR Streaming"
|
||||
type = "riva_realtime"
|
||||
url = "http://127.0.0.1:8006/v1"
|
||||
model = ""
|
||||
```
|
||||
|
||||
Then create or edit a workflow with:
|
||||
|
||||
```toml
|
||||
mode = "stream"
|
||||
```
|
||||
|
||||
## 6. Configure Rewrite Workflows
|
||||
|
||||
Rewrite workflows use an OpenAI-compatible chat endpoint. You can point them at OpenAI, LiteLLM, llama-swap, vLLM, LM Studio, or another compatible server.
|
||||
|
||||
For OpenAI:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
Then set the LLM engine in **Settings > Engines** or edit `~/.config/blitztext/config.toml`.
|
||||
|
||||
Never commit API keys into this repository, issues, logs, or screenshots.
|
||||
|
||||
## 7. Permissions And Desktop Session
|
||||
|
||||
Blitztext needs microphone access through your Linux audio stack and uses `xdotool` to type into the currently focused X11 window.
|
||||
|
||||
If text delivery does not work:
|
||||
|
||||
- confirm you are on X11, not Wayland
|
||||
- check that `xdotool getactivewindow` works in a terminal
|
||||
- focus a normal text field before triggering a workflow
|
||||
- try `output = "paste"` or `output = "type"` in config
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- If `xcodebuild` reports that the active developer directory is only Command Line Tools, run `sudo xcode-select -s /Applications/Xcode.app/Contents/Developer`.
|
||||
- If the build cannot find XcodeGen, install it explicitly with `brew install xcodegen`.
|
||||
- If online transcription fails immediately, check whether the API key is present and valid.
|
||||
- If secure local mode is disabled, check whether a WhisperKit model is installed in the expected folder.
|
||||
- If transcription works but paste does not, this is not an OpenAI billing issue. Check **Privacy & Security -> Accessibility**, restart Blitztext after changing the permission, and make sure the cursor is focused in a text field before starting the workflow.
|
||||
- If macOS shows multiple Blitztext entries under Accessibility, remove or disable stale entries, run the app from the final location (`/Applications` if you used `./build.sh --install`), then grant the permission again.
|
||||
- If the target app blocks synthetic paste or the target app was not detected, the result still stays on the clipboard so you can press Cmd+V manually.
|
||||
- If audio is missing, check Microphone permission and macOS input settings.
|
||||
- If you see OpenAI errors, verify model access and account billing.
|
||||
- If the tray does not start, confirm `python3-gi` is visible to the venv. `install.sh` uses `--system-site-packages` for this reason.
|
||||
- If local Whisper is slow on arm64, use a smaller model such as `small`, `base`, or `tiny`.
|
||||
- If realtime streaming connects but produces poor text, confirm the server language. The tested Nemotron ASR Streaming NIM is `en-US`.
|
||||
- If a batch STT endpoint returns `bad model`, check whether it is actually a streaming-only NIM. Use `riva_realtime` for realtime services and `openai` only for batch-compatible services.
|
||||
- If audio is missing, check the selected microphone in **Settings > General** and watch the input level meter.
|
||||
- If rewriting fails, verify your LLM endpoint, model name, API key environment variable, and account billing if using a cloud provider.
|
||||
|
||||
@ -9,6 +9,13 @@ The version is defined in [`blitztext/__init__.py`](blitztext/__init__.py).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Realtime STT streaming mode**: new `mode = "stream"` workflow support and a
|
||||
`riva_realtime` STT engine for Riva/NIM WebSocket transcription, including a
|
||||
Settings shortcut for Nemotron ASR Streaming on `http://127.0.0.1:8006/v1`.
|
||||
- **Settings About tab** with the app version, source link, changelog, and
|
||||
license text.
|
||||
|
||||
## [1.1.0] - 2026-06-04
|
||||
|
||||
### Added
|
||||
@ -74,7 +81,7 @@ into that field.
|
||||
AppIndicator typelibs and GNOME `ubuntu-appindicators` extension are already
|
||||
present on the target host).
|
||||
|
||||
[Unreleased]: https://github.com/mARTin-B78/blitztext-app/compare/v1.1.0...HEAD
|
||||
[1.1.0]: https://github.com/mARTin-B78/blitztext-app/compare/v1.0.1...v1.1.0
|
||||
[1.0.1]: https://github.com/mARTin-B78/blitztext-app/compare/v1.0.0...v1.0.1
|
||||
[1.0.0]: https://github.com/mARTin-B78/blitztext-app/releases/tag/v1.0.0
|
||||
[Unreleased]: https://github.com/mARTin-B78/blitztext-app-linux/compare/v1.1.0...HEAD
|
||||
[1.1.0]: https://github.com/mARTin-B78/blitztext-app-linux/compare/v1.0.1...v1.1.0
|
||||
[1.0.1]: https://github.com/mARTin-B78/blitztext-app-linux/compare/v1.0.0...v1.0.1
|
||||
[1.0.0]: https://github.com/mARTin-B78/blitztext-app-linux/releases/tag/v1.0.0
|
||||
|
||||
@ -8,8 +8,13 @@ This runs **on the host** (not in a container), so it can type into *any*
|
||||
application — the Linux equivalent of the macOS app's Accessibility-based
|
||||
auto-paste. (A sandboxed Docker/browser version can't do that; an earlier
|
||||
experiment along those lines was moved out to
|
||||
`~/Docker/correspondence/blitztext`.) Transcription is **local** via
|
||||
[faster-whisper]; only the optional rewrite step calls out to an LLM.
|
||||
`~/Docker/correspondence/blitztext`.) Batch transcription is **local** via
|
||||
[faster-whisper]; live streaming can use a local Riva/NIM realtime server. Only
|
||||
the optional rewrite step calls out to an LLM.
|
||||
|
||||
## Inspiration
|
||||
|
||||
Blitztext App Linux is inspired by [cmagnussen/blitztext-app](https://github.com/cmagnussen/blitztext-app), the original macOS menu-bar workflow for turning speech into text and cleaner writing. This Linux version keeps the workflow but uses Linux-native pieces: GTK, AppIndicator, global hotkeys, `faster-whisper`, optional Riva/NIM realtime STT, and `xdotool`.
|
||||
|
||||
## How it works
|
||||
|
||||
@ -20,10 +25,13 @@ hotkey ──▶ record mic (pw-record/arecord) ──▶ faster-whisper (local)
|
||||
│ └── mode "rewrite": LLM (OpenAI-compatible)
|
||||
▼
|
||||
xdotool types it into the focused window
|
||||
|
||||
mode "stream" ──▶ mic PCM chunks ──▶ Riva/NIM realtime WebSocket ──▶ live xdotool typing
|
||||
```
|
||||
|
||||
Each hotkey **toggles**: press to start recording, press again to stop — then it
|
||||
transcribes, optionally rewrites, and types the result where your cursor is.
|
||||
Each normal hotkey **toggles**: press to start recording, press again to stop —
|
||||
then it transcribes, optionally rewrites, and types the result where your cursor
|
||||
is. Streaming workflows type stable words live while you speak.
|
||||
|
||||
## Requirements
|
||||
|
||||
@ -34,6 +42,8 @@ transcribes, optionally rewrites, and types the result where your cursor is.
|
||||
sudo apt install xdotool libnotify-bin pipewire-bin
|
||||
```
|
||||
- Python 3.11+.
|
||||
- Optional realtime STT streaming: a Riva/NIM realtime server such as Nemotron
|
||||
ASR Streaming, reachable through `/v1/realtime`.
|
||||
|
||||
## Install
|
||||
|
||||
@ -60,8 +70,8 @@ cd linux
|
||||
./install.sh
|
||||
```
|
||||
|
||||
This creates `.venv`, installs `faster-whisper` + `pynput`, and writes the
|
||||
default config to `~/.config/blitztext/config.toml`.
|
||||
This creates `.venv`, installs the Python dependencies from `requirements.txt`,
|
||||
and writes the default config to `~/.config/blitztext/config.toml`.
|
||||
|
||||
> For the **tray** from source, the venv must be built on a Python that can see
|
||||
> the system `python3-gi` — `install.sh` uses `python3 -m venv
|
||||
@ -70,7 +80,7 @@ default config to `~/.config/blitztext/config.toml`.
|
||||
|
||||
## Run
|
||||
|
||||
Three front-ends, same engine (local Whisper + global hotkeys + xdotool typing):
|
||||
Three front-ends, same engine layer (STT engines + global hotkeys + xdotool typing):
|
||||
|
||||
```bash
|
||||
# optional: only needed for the "rewrite" workflows
|
||||
@ -81,6 +91,29 @@ export OPENAI_API_KEY=sk-...
|
||||
.venv/bin/python -m blitztext run # headless, hotkeys only
|
||||
```
|
||||
|
||||
### Realtime STT streaming
|
||||
|
||||
For Nemotron ASR Streaming, add a realtime engine in **Settings > Engines** with
|
||||
`+ Stream`, save/restart, then create or edit a workflow with `mode = "stream"`.
|
||||
The default realtime URL is:
|
||||
|
||||
```toml
|
||||
[[stt_engine]]
|
||||
name = "Nemotron ASR Streaming"
|
||||
type = "riva_realtime"
|
||||
url = "http://127.0.0.1:8006/v1"
|
||||
model = ""
|
||||
|
||||
[[workflow]]
|
||||
name = "STT Streaming"
|
||||
hotkey = "<ctrl>+<alt>+s"
|
||||
mode = "stream"
|
||||
```
|
||||
|
||||
The current Nemotron ASR Streaming model exposed by the tested NIM is English
|
||||
`en-US`, so use `language = "en"` or `language = "en-US"` in `[general]` for
|
||||
that engine.
|
||||
|
||||
### System tray (recommended)
|
||||
|
||||
The tray is the closest match to the macOS menu-bar app: a status icon with a
|
||||
@ -165,3 +198,18 @@ python -m blitztext config-path # print config location
|
||||
```
|
||||
|
||||
[faster-whisper]: https://github.com/SYSTRAN/faster-whisper
|
||||
|
||||
## License
|
||||
|
||||
Code is released under the MIT License. See [../LICENSE](../LICENSE).
|
||||
|
||||
Project names, logos, and app icons are not automatically granted as trademarks or brand assets. See [../TRADEMARKS.md](../TRADEMARKS.md).
|
||||
|
||||
## Legal / Impressum & Datenschutz
|
||||
|
||||
This is an experimental, non-commercial open-source project, provided as-is under the MIT License without warranty or support. Nothing is sold here and no installation or operation is performed on your behalf.
|
||||
|
||||
The companion website (blitztext.de) is operated by Blackboat Internet GmbH:
|
||||
|
||||
- Impressum: https://martin-bierschenk.de/impressum/
|
||||
- Datenschutz / Privacy: https://martin-bierschenk.de/datenschutz/
|
||||
|
||||
@ -18,7 +18,7 @@ CONFIG_PATH = CONFIG_DIR / "config.toml"
|
||||
class Workflow:
|
||||
name: str
|
||||
hotkey: str
|
||||
mode: str # "transcribe" | "rewrite"
|
||||
mode: str # "transcribe" | "rewrite" | "stream"
|
||||
prompt: str = ""
|
||||
# Spoken trigger phrases for voice routing (matched at start/end of speech).
|
||||
keywords: list[str] = field(default_factory=list)
|
||||
@ -366,7 +366,8 @@ threshold = 0.82 # 0..1 fuzzy-match strictness (higher = stricter)
|
||||
# Speech-to-text engines (presets). The active one is used for transcription.
|
||||
# type = "local" -> in-process faster-whisper (uses [whisper] above)
|
||||
# type = "openai" -> remote OpenAI-compatible /audio/transcriptions server
|
||||
# (faster-whisper-server, Groq, WhisperX, NIMs, ...)
|
||||
# (faster-whisper-server, Groq, WhisperX, batch ASR NIMs, ...)
|
||||
# type = "riva_realtime" -> Riva/NIM realtime WebSocket STT, for mode="stream"
|
||||
# ----------------------------------------------------------------------------
|
||||
[stt]
|
||||
active = "Local faster-whisper"
|
||||
@ -382,6 +383,13 @@ type = "local"
|
||||
# model = "Systran/faster-whisper-base"
|
||||
# api_key_env = "" # e.g. GROQ_API_KEY for a cloud endpoint
|
||||
|
||||
# [[stt_engine]]
|
||||
# name = "Nemotron ASR Streaming"
|
||||
# type = "riva_realtime"
|
||||
# url = "http://localhost:8006/v1"
|
||||
# model = "" # blank = use the realtime server default
|
||||
# api_key_env = ""
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# LLM engines (presets) for the rewrite step. Any OpenAI-compatible chat API
|
||||
# (OpenAI, vLLM, llama-swap, Ollama /v1, LM Studio, Groq, OpenRouter, ...).
|
||||
@ -408,6 +416,7 @@ temperature = 0.3
|
||||
# ----------------------------------------------------------------------------
|
||||
# Workflows / presets. mode = "transcribe" types the raw transcript. mode =
|
||||
# "rewrite" sends it through the LLM with `prompt` as the system prompt.
|
||||
# "stream" writes live words via a riva_realtime STT engine.
|
||||
# keywords = spoken trigger phrases for voice routing (start or end of speech)
|
||||
# hotkey = optional direct global hotkey ("" = none; voice routing is primary)
|
||||
# A workflow may override the [rewrite] defaults with its own model/temperature.
|
||||
@ -420,6 +429,13 @@ description = "Speak, get plain text."
|
||||
hotkey = ""
|
||||
mode = "transcribe"
|
||||
|
||||
# [[workflow]]
|
||||
# name = "STT Streaming"
|
||||
# icon = "⚡"
|
||||
# description = "Live words while you speak."
|
||||
# hotkey = "<ctrl>+<alt>+s"
|
||||
# mode = "stream"
|
||||
|
||||
[[workflow]]
|
||||
name = "Nicer email"
|
||||
icon = "✉"
|
||||
|
||||
@ -17,12 +17,13 @@ from .llm import LLMError
|
||||
from .logbuffer import log
|
||||
from .notify import notify
|
||||
from .paste import active_window_id, deliver
|
||||
from .streaming import RivaRealtimeStreamer
|
||||
from .recorder import Recording, detect_recorder
|
||||
from .routing import route
|
||||
from .transcribe import Transcriber
|
||||
|
||||
# status_cb(state, workflow_name, message)
|
||||
# state in {"loading", "idle", "recording", "busy", "done", "error"}
|
||||
# state in {"loading", "idle", "recording", "streaming", "busy", "done", "error"}
|
||||
StatusCallback = Callable[[str, str | None, str], None]
|
||||
|
||||
|
||||
@ -32,6 +33,8 @@ class Daemon:
|
||||
self.status_cb = status_cb
|
||||
self._lock = threading.Lock()
|
||||
self._recording: Recording | None = None
|
||||
self._streaming: RivaRealtimeStreamer | None = None
|
||||
self._stream_segment_text = ""
|
||||
self._active_workflow: Workflow | None = None
|
||||
self._target_window: str | None = None
|
||||
self._busy = False
|
||||
@ -83,28 +86,78 @@ class Daemon:
|
||||
|
||||
@property
|
||||
def is_recording(self) -> bool:
|
||||
return self._recording is not None
|
||||
return self._recording is not None or self._streaming is not None
|
||||
|
||||
# -- recording control ----------------------------------------------------
|
||||
def start_dictation(self, workflow: Workflow | None = None) -> None:
|
||||
wf = workflow or self._route_workflow
|
||||
streamer: RivaRealtimeStreamer | None = None
|
||||
with self._lock:
|
||||
if not self.ready or self._busy or self._recording is not None:
|
||||
if not self.ready or self._busy or self.is_recording:
|
||||
return
|
||||
self._target_window = active_window_id()
|
||||
self._recording = Recording(self.recorder_name, self.cfg.mic)
|
||||
self._active_workflow = wf
|
||||
if wf.mode == "stream":
|
||||
engine = self.cfg.active_stt
|
||||
if not engine.is_streaming:
|
||||
self._active_workflow = None
|
||||
self._emit("error", wf.name, "Active STT engine is not realtime streaming")
|
||||
self._notify("Streaming unavailable", "Select a riva_realtime STT engine.", "critical")
|
||||
return
|
||||
self._stream_segment_text = ""
|
||||
streamer = RivaRealtimeStreamer(
|
||||
engine,
|
||||
device=self.cfg.mic,
|
||||
language=self._stream_language(),
|
||||
on_text=self._on_stream_text,
|
||||
on_status=lambda msg: log(f"stream: {msg}"),
|
||||
on_error=lambda exc, label=wf.name: self._on_stream_error(label, exc),
|
||||
)
|
||||
self._streaming = streamer
|
||||
else:
|
||||
self._recording = Recording(self.recorder_name, self.cfg.mic)
|
||||
|
||||
if streamer is not None:
|
||||
self._emit("streaming", wf.name, "Live transcript…")
|
||||
self._notify(f"● {wf.name}", "Live transcript…")
|
||||
try:
|
||||
streamer.start()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
with self._lock:
|
||||
if self._streaming is streamer:
|
||||
self._streaming = None
|
||||
self._active_workflow = None
|
||||
self._emit("error", wf.name, str(exc))
|
||||
self._notify("Streaming failed", str(exc), "critical")
|
||||
return
|
||||
|
||||
self._emit("recording", wf.name, "Recording…")
|
||||
self._notify(f"● {wf.name}", "Recording…")
|
||||
|
||||
def finish_dictation(self, send_enter: bool = False) -> None:
|
||||
with self._lock:
|
||||
if self._recording is None:
|
||||
return
|
||||
rec, wf, win = self._recording, self._active_workflow, self._target_window
|
||||
self._recording = None
|
||||
self._active_workflow = None
|
||||
self._busy = True
|
||||
if self._streaming is not None:
|
||||
streamer, wf, win = self._streaming, self._active_workflow, self._target_window
|
||||
self._streaming = None
|
||||
self._active_workflow = None
|
||||
self._stream_segment_text = ""
|
||||
else:
|
||||
streamer = None
|
||||
if self._recording is None:
|
||||
return
|
||||
rec, wf, win = self._recording, self._active_workflow, self._target_window
|
||||
self._recording = None
|
||||
self._active_workflow = None
|
||||
self._busy = True
|
||||
if streamer is not None:
|
||||
streamer.stop()
|
||||
if send_enter:
|
||||
from .paste import press_enter
|
||||
press_enter(win)
|
||||
self._emit("done", wf.name if wf else None, "Streaming stopped")
|
||||
self._emit("idle", None, "Ready")
|
||||
return
|
||||
|
||||
audio_path = rec.stop()
|
||||
threading.Thread(
|
||||
target=self._process, args=(audio_path, wf, win, send_enter), daemon=True
|
||||
@ -112,11 +165,24 @@ class Daemon:
|
||||
|
||||
def cancel_dictation(self) -> None:
|
||||
with self._lock:
|
||||
if self._recording is None:
|
||||
return
|
||||
rec = self._recording
|
||||
self._recording = None
|
||||
self._active_workflow = None
|
||||
if self._streaming is not None:
|
||||
streamer = self._streaming
|
||||
self._streaming = None
|
||||
self._active_workflow = None
|
||||
self._stream_segment_text = ""
|
||||
rec = None
|
||||
else:
|
||||
streamer = None
|
||||
if self._recording is None:
|
||||
return
|
||||
rec = self._recording
|
||||
self._recording = None
|
||||
self._active_workflow = None
|
||||
if streamer is not None:
|
||||
streamer.stop()
|
||||
self._emit("idle", None, "Cancelled")
|
||||
self._notify("Cancelled", "Streaming stopped.", "low")
|
||||
return
|
||||
rec.discard()
|
||||
self._emit("idle", None, "Cancelled")
|
||||
self._notify("Cancelled", "Recording discarded.", "low")
|
||||
@ -129,11 +195,56 @@ class Daemon:
|
||||
if self._busy:
|
||||
self._notify("Busy", "Still processing the last clip…", "low")
|
||||
return
|
||||
if self._recording is None:
|
||||
if not self.is_recording:
|
||||
self.start_dictation(workflow)
|
||||
else:
|
||||
self.finish_dictation(send_enter=False)
|
||||
|
||||
# -- live streaming -------------------------------------------------------
|
||||
def _stream_language(self) -> str:
|
||||
lang = (self.cfg.language or "").strip()
|
||||
if lang.lower() == "en":
|
||||
return "en-US"
|
||||
if lang.lower().startswith("en-"):
|
||||
return lang
|
||||
return ""
|
||||
|
||||
def _stable_stream_text(self, text: str, final: bool) -> str:
|
||||
text = quality.clean(text, strip_trailing_punctuation=False)
|
||||
if final:
|
||||
return text
|
||||
cut = max(text.rfind(" "), text.rfind("\n"), text.rfind("\t"))
|
||||
return text[:cut + 1] if cut >= 0 else ""
|
||||
|
||||
def _on_stream_text(self, text: str, final: bool) -> None:
|
||||
stable = self._stable_stream_text(text, final)
|
||||
if not stable:
|
||||
return
|
||||
with self._lock:
|
||||
win = self._target_window
|
||||
current = self._stream_segment_text
|
||||
if not stable.startswith(current):
|
||||
if not final:
|
||||
return
|
||||
suffix = ""
|
||||
else:
|
||||
suffix = stable[len(current):]
|
||||
if suffix:
|
||||
deliver(suffix, mode="type", window_id=win, type_delay_ms=self.cfg.type_delay_ms)
|
||||
if final and (stable or current) and not stable.endswith((" ", "\n", "\t")):
|
||||
deliver(" ", mode="type", window_id=win, type_delay_ms=self.cfg.type_delay_ms)
|
||||
with self._lock:
|
||||
self._stream_segment_text = "" if final else stable
|
||||
|
||||
def _on_stream_error(self, label: str, exc: Exception) -> None:
|
||||
with self._lock:
|
||||
self._streaming = None
|
||||
self._active_workflow = None
|
||||
self._stream_segment_text = ""
|
||||
self._emit("error", label, str(exc))
|
||||
self._notify("Streaming failed", str(exc), "critical")
|
||||
log(f"ERROR ({label} streaming): {exc}")
|
||||
|
||||
# -- worker ---------------------------------------------------------------
|
||||
def _process(self, audio_path, workflow: Workflow, window_id, send_enter: bool = False) -> None:
|
||||
label = workflow.name
|
||||
|
||||
@ -801,8 +801,8 @@ class SettingsDialog:
|
||||
changelog = _read_first(paths["changelog"])
|
||||
license_text = _read_first(paths["license"])
|
||||
|
||||
title = Gtk.Label(label="Blitztext", xalign=0.0)
|
||||
title.set_markup("<b>Blitztext</b>")
|
||||
title = Gtk.Label(label="Blitztext App Linux", xalign=0.0)
|
||||
title.set_markup("<b>Blitztext App Linux</b>")
|
||||
page.pack_start(title, False, False, 0)
|
||||
|
||||
version = Gtk.Label(label=f"Version {__version__}", xalign=0.0)
|
||||
@ -810,8 +810,8 @@ class SettingsDialog:
|
||||
page.pack_start(version, False, False, 2)
|
||||
|
||||
source = Gtk.LinkButton.new_with_label(
|
||||
"https://github.com/mARTin-B78/blitztext-app",
|
||||
"Source: github.com/mARTin-B78/blitztext-app",
|
||||
"https://github.com/mARTin-B78/blitztext-app-linux",
|
||||
"Source: github.com/mARTin-B78/blitztext-app-linux",
|
||||
)
|
||||
source.set_halign(Gtk.Align.START)
|
||||
page.pack_start(source, False, False, 4)
|
||||
|
||||
@ -215,9 +215,9 @@ class App:
|
||||
|
||||
def _apply_status(self, state: str, workflow: str | None, message: str) -> bool:
|
||||
colors = {"loading": "#ff9f0a", "idle": "#34c759", "recording": "#ff3b30",
|
||||
"busy": "#ff9f0a", "done": "#34c759", "error": "#ff3b30"}
|
||||
"streaming": "#ff3b30", "busy": "#ff9f0a", "done": "#34c759", "error": "#ff3b30"}
|
||||
labels = {"loading": "Loading…", "idle": "Ready", "recording": "Recording",
|
||||
"busy": "Working…", "done": "Ready", "error": message[:40] or "Error"}
|
||||
"streaming": "Live", "busy": "Working…", "done": "Ready", "error": message[:40] or "Error"}
|
||||
self._set_dot(colors.get(state, "#7b818b"))
|
||||
self.status_lbl.set_text(labels.get(state, message))
|
||||
|
||||
@ -231,12 +231,12 @@ class App:
|
||||
if add:
|
||||
ctx.add_class(add)
|
||||
|
||||
if state == "recording":
|
||||
if state in ("recording", "streaming"):
|
||||
self._active = workflow
|
||||
for nm, r in self._rows.items():
|
||||
if nm == workflow:
|
||||
cls(nm, "recording", "dim")
|
||||
r["pill"].set_text("● Stop")
|
||||
r["pill"].set_text("● Live" if state == "streaming" else "● Stop")
|
||||
r["pill"].get_style_context().add_class("rec")
|
||||
else:
|
||||
cls(nm, "dim", "recording")
|
||||
|
||||
268
linux/blitztext/streaming.py
Normal file
268
linux/blitztext/streaming.py
Normal file
@ -0,0 +1,268 @@
|
||||
"""Realtime STT client for Riva/NIM WebSocket transcription.
|
||||
|
||||
The NIM HTTP docs expose session creation, but the live audio path is a
|
||||
WebSocket at /v1/realtime?intent=transcription. Audio is sent as base64 PCM16
|
||||
chunks and transcript events arrive as interim deltas and completed segments.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable
|
||||
|
||||
from .stt import STTEngine
|
||||
|
||||
TextCallback = Callable[[str, bool], None]
|
||||
StatusCallback = Callable[[str], None]
|
||||
ErrorCallback = Callable[[Exception], None]
|
||||
|
||||
|
||||
class StreamingSTTError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class RealtimeURLs:
|
||||
session_url: str
|
||||
websocket_url: str
|
||||
|
||||
|
||||
def realtime_urls(base_url: str) -> RealtimeURLs:
|
||||
"""Build the HTTP session URL and WebSocket URL from a user base URL."""
|
||||
raw = (base_url or "").strip().rstrip("/")
|
||||
if not raw:
|
||||
raise StreamingSTTError("Missing realtime STT URL.")
|
||||
if "://" not in raw:
|
||||
raw = "http://" + raw
|
||||
|
||||
parsed = urllib.parse.urlparse(raw)
|
||||
path = parsed.path.rstrip("/")
|
||||
if path.endswith("/realtime"):
|
||||
realtime_path = path
|
||||
elif path.endswith("/v1"):
|
||||
realtime_path = path + "/realtime"
|
||||
else:
|
||||
realtime_path = path + "/v1/realtime"
|
||||
|
||||
http_scheme = "https" if parsed.scheme == "https" else "http"
|
||||
ws_scheme = "wss" if http_scheme == "https" else "ws"
|
||||
netloc = parsed.netloc
|
||||
session = urllib.parse.urlunparse((http_scheme, netloc, realtime_path + "/transcription_sessions", "", "", ""))
|
||||
ws = urllib.parse.urlunparse((ws_scheme, netloc, realtime_path, "", "intent=transcription", ""))
|
||||
return RealtimeURLs(session, ws)
|
||||
|
||||
|
||||
class RivaRealtimeStreamer:
|
||||
"""Owns one live microphone -> Riva WebSocket transcription session."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
engine: STTEngine,
|
||||
*,
|
||||
device: str = "",
|
||||
language: str = "",
|
||||
sample_rate: int = 16000,
|
||||
chunk_frames: int = 1600,
|
||||
on_text: TextCallback | None = None,
|
||||
on_status: StatusCallback | None = None,
|
||||
on_error: ErrorCallback | None = None,
|
||||
):
|
||||
self.engine = engine
|
||||
self.device = device
|
||||
self.language = language.strip()
|
||||
self.sample_rate = sample_rate
|
||||
self.chunk_frames = chunk_frames
|
||||
self.on_text = on_text
|
||||
self.on_status = on_status
|
||||
self.on_error = on_error
|
||||
self._stop = threading.Event()
|
||||
self._ready = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
self._error: Exception | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if self._thread is not None:
|
||||
return
|
||||
self._thread = threading.Thread(target=self._run_thread, daemon=True)
|
||||
self._thread.start()
|
||||
self._ready.wait(timeout=5.0)
|
||||
if self._error:
|
||||
raise self._error
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=1.0)
|
||||
|
||||
def _run_thread(self) -> None:
|
||||
try:
|
||||
asyncio.run(self._run())
|
||||
except Exception as exc: # noqa: BLE001 - surfaced to daemon/UI
|
||||
self._error = exc
|
||||
self._ready.set()
|
||||
if self.on_error:
|
||||
self.on_error(exc)
|
||||
|
||||
async def _run(self) -> None:
|
||||
try:
|
||||
import sounddevice as sd
|
||||
import websockets
|
||||
except ModuleNotFoundError as exc:
|
||||
raise StreamingSTTError(
|
||||
"Realtime streaming needs the Python packages 'websockets' and 'sounddevice'."
|
||||
) from exc
|
||||
|
||||
urls = realtime_urls(self.engine.url)
|
||||
session = self._create_session(urls.session_url)
|
||||
headers = self._auth_headers()
|
||||
websocket = await self._connect_websocket(websockets, urls.websocket_url, headers)
|
||||
audio_q: queue.Queue[bytes] = queue.Queue(maxsize=60)
|
||||
|
||||
def audio_cb(indata, _frames, _time_info, status):
|
||||
if status and self.on_status:
|
||||
self.on_status(str(status))
|
||||
try:
|
||||
audio_q.put_nowait(bytes(indata))
|
||||
except queue.Full:
|
||||
pass
|
||||
|
||||
try:
|
||||
await self._initialize_websocket(websocket, session)
|
||||
self._ready.set()
|
||||
if self.on_status:
|
||||
self.on_status("Streaming")
|
||||
|
||||
with sd.RawInputStream(
|
||||
samplerate=self.sample_rate,
|
||||
channels=1,
|
||||
dtype="int16",
|
||||
blocksize=self.chunk_frames,
|
||||
device=self._resolve_device(sd),
|
||||
callback=audio_cb,
|
||||
):
|
||||
send_task = asyncio.create_task(self._send_audio(websocket, audio_q))
|
||||
recv_task = asyncio.create_task(self._receive_text(websocket))
|
||||
await send_task
|
||||
try:
|
||||
await asyncio.wait_for(recv_task, timeout=8.0)
|
||||
except asyncio.TimeoutError:
|
||||
recv_task.cancel()
|
||||
finally:
|
||||
await websocket.close()
|
||||
|
||||
def _auth_headers(self) -> dict[str, str]:
|
||||
if not self.engine.api_key_env:
|
||||
return {}
|
||||
key = os.environ.get(self.engine.api_key_env)
|
||||
return {"Authorization": f"Bearer {key}"} if key else {}
|
||||
|
||||
def _create_session(self, url: str) -> dict:
|
||||
body = b"{}"
|
||||
headers = {"Content-Type": "application/json", **self._auth_headers()}
|
||||
req = urllib.request.Request(url, data=body, headers=headers, method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=10) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:300]
|
||||
raise StreamingSTTError(f"HTTP {exc.code} from {url}: {detail}") from exc
|
||||
except (urllib.error.URLError, json.JSONDecodeError, OSError) as exc:
|
||||
raise StreamingSTTError(f"Cannot initialize realtime STT session at {url}: {exc}") from exc
|
||||
|
||||
async def _connect_websocket(self, websockets, url: str, headers: dict[str, str]):
|
||||
try:
|
||||
return await websockets.connect(url, additional_headers=headers or None)
|
||||
except TypeError:
|
||||
return await websockets.connect(url, extra_headers=headers or None)
|
||||
|
||||
async def _initialize_websocket(self, websocket, session: dict) -> None:
|
||||
first = json.loads(await asyncio.wait_for(websocket.recv(), timeout=5.0))
|
||||
if first.get("type") != "conversation.created":
|
||||
raise StreamingSTTError(f"Unexpected realtime greeting: {first}")
|
||||
|
||||
updated = deepcopy(session)
|
||||
updated["input_audio_format"] = "pcm16"
|
||||
updated.setdefault("input_audio_params", {})
|
||||
updated["input_audio_params"]["sample_rate_hz"] = self.sample_rate
|
||||
updated["input_audio_params"]["num_channels"] = 1
|
||||
updated.setdefault("recognition_config", {})
|
||||
updated["recognition_config"]["max_alternatives"] = 1
|
||||
if self.language:
|
||||
updated.setdefault("input_audio_transcription", {})
|
||||
updated["input_audio_transcription"]["language"] = self.language
|
||||
if self.engine.model:
|
||||
updated.setdefault("input_audio_transcription", {})
|
||||
updated["input_audio_transcription"]["model"] = self.engine.model
|
||||
|
||||
await websocket.send(json.dumps({"type": "transcription_session.update", "session": updated}))
|
||||
response = json.loads(await asyncio.wait_for(websocket.recv(), timeout=5.0))
|
||||
if response.get("type") != "transcription_session.updated":
|
||||
raise StreamingSTTError(f"Realtime session update failed: {response}")
|
||||
|
||||
async def _send_audio(self, websocket, audio_q: queue.Queue[bytes]) -> None:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
chunk = await asyncio.to_thread(audio_q.get, True, 0.2)
|
||||
except queue.Empty:
|
||||
continue
|
||||
await self._send_chunk(websocket, chunk)
|
||||
|
||||
drain_until = time.monotonic() + 0.3
|
||||
while time.monotonic() < drain_until:
|
||||
try:
|
||||
chunk = audio_q.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
await self._send_chunk(websocket, chunk)
|
||||
await websocket.send(json.dumps({"type": "input_audio_buffer.done"}))
|
||||
|
||||
async def _send_chunk(self, websocket, chunk: bytes) -> None:
|
||||
audio = base64.b64encode(chunk).decode("ascii")
|
||||
await websocket.send(json.dumps({"type": "input_audio_buffer.append", "audio": audio}))
|
||||
await websocket.send(json.dumps({"type": "input_audio_buffer.commit"}))
|
||||
|
||||
async def _receive_text(self, websocket) -> None:
|
||||
while True:
|
||||
try:
|
||||
raw = await asyncio.wait_for(websocket.recv(), timeout=1.0)
|
||||
except asyncio.TimeoutError:
|
||||
continue
|
||||
event = json.loads(raw)
|
||||
kind = event.get("type", "")
|
||||
if kind == "conversation.item.input_audio_transcription.delta":
|
||||
text = (event.get("delta") or "").strip()
|
||||
if text and self.on_text:
|
||||
self.on_text(text, False)
|
||||
elif kind == "conversation.item.input_audio_transcription.completed":
|
||||
text = (event.get("transcript") or "").strip()
|
||||
if text and self.on_text:
|
||||
self.on_text(text, True)
|
||||
if event.get("is_last_result"):
|
||||
return
|
||||
elif kind == "conversation.item.input_audio_transcription.failed":
|
||||
raise StreamingSTTError(event.get("error", {}).get("message", "Transcription failed."))
|
||||
elif kind == "error":
|
||||
err = event.get("error", {})
|
||||
raise StreamingSTTError(err.get("message") or str(err) or "Realtime STT error.")
|
||||
|
||||
def _resolve_device(self, sd):
|
||||
if not self.device:
|
||||
return None
|
||||
try:
|
||||
for i, d in enumerate(sd.query_devices()):
|
||||
if d["max_input_channels"] > 0 and self.device in d["name"]:
|
||||
return i
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return None
|
||||
@ -23,7 +23,7 @@ from urllib.parse import urlparse
|
||||
@dataclass
|
||||
class STTEngine:
|
||||
name: str
|
||||
type: str = "local" # "local" | "openai"
|
||||
type: str = "local" # "local" | "openai" | "riva_realtime"
|
||||
url: str = "" # base URL incl. /v1 for remote, e.g. http://localhost:8010/v1
|
||||
model: str = "" # remote model id, or local whisper size override
|
||||
api_key_env: str = "" # env var holding a bearer key (optional)
|
||||
@ -32,6 +32,10 @@ class STTEngine:
|
||||
def is_local(self) -> bool:
|
||||
return self.type == "local"
|
||||
|
||||
@property
|
||||
def is_streaming(self) -> bool:
|
||||
return self.type == "riva_realtime"
|
||||
|
||||
|
||||
class STTError(RuntimeError):
|
||||
pass
|
||||
@ -107,6 +111,8 @@ def transcribe(
|
||||
if local_transcriber is None:
|
||||
raise STTError("Local engine selected but the model isn't loaded.")
|
||||
return local_transcriber.transcribe(audio_path, language=language, hotwords=hotwords)
|
||||
if engine.is_streaming:
|
||||
raise STTError("Streaming STT engines are live-only. Use a workflow with mode = \"stream\".")
|
||||
return _transcribe_remote(engine, audio_path, language=language, prompt=hotwords, timeout=timeout)
|
||||
|
||||
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
[Desktop Entry]
|
||||
Type=Application
|
||||
Name=Blitztext
|
||||
Name=Blitztext Linux
|
||||
GenericName=Voice Dictation
|
||||
Comment=Speak and have the text typed into any field
|
||||
Comment=Linux voice dictation with local, remote, and realtime STT engines
|
||||
Exec=blitztext tray
|
||||
Icon=blitztext
|
||||
Terminal=false
|
||||
Categories=Utility;AudioVideo;Accessibility;
|
||||
Keywords=dictation;speech;voice;transcription;whisper;stt;
|
||||
Keywords=dictation;speech;voice;transcription;whisper;stt;riva;nim;
|
||||
StartupNotify=false
|
||||
|
||||
@ -1,8 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build a self-contained .deb for Blitztext (Ubuntu/Debian, arm64).
|
||||
#
|
||||
# Bundles a relocatable venv with the Python deps (faster-whisper, pynput,
|
||||
# tomli-w) under /opt/blitztext so installation needs no pip/network. System
|
||||
# Bundles a relocatable venv with the Python deps from requirements.txt under
|
||||
# /opt/blitztext so installation needs no pip/network. System
|
||||
# integration (python3-gi, xdotool, libnotify-bin, a recorder) is declared via
|
||||
# Depends so the Software app / apt pull them in.
|
||||
set -euo pipefail
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
|
||||
Upstream-Name: blitztext
|
||||
Source: https://github.com/mARTin-B78/blitztext-app
|
||||
Upstream-Name: blitztext-app-linux
|
||||
Source: https://github.com/mARTin-B78/blitztext-app-linux
|
||||
|
||||
Files: *
|
||||
Copyright: 2026 Martin Bierschenk
|
||||
|
||||
@ -2,3 +2,4 @@ faster-whisper>=1.0
|
||||
pynput>=1.7
|
||||
tomli-w>=1.0
|
||||
sounddevice>=0.4
|
||||
websockets>=12.0
|
||||
|
||||
Loading…
Reference in New Issue
Block a user