🤖

AI Backends

Connect local or cloud services for speech recognition, synthesis, and text generation.

Local AI Services

Run everything on your own hardware — no API key, no rate limits, no data leaves your machine.

100% Local

Active Docker Stack

Start, stop, or install TTS and ASR containers running on this machine.

Checking container status…

Language Models (LLM)

🦙 Ollama OpenAI-compat

Easiest local LLM setup. Pull any model with one command. Runs Llama, Mistral, Qwen, Phi, Gemma and many more.

URL
Quick start 
curl https://ollama.ai/install.sh | sh
ollama pull llama3.3
ollama.ai ↗
⚡ vLLM Detected in stack

High-throughput GPU inference. Already in your Docker stack. Serve any HuggingFace model with PagedAttention.

URL
Add a model to your stack 
--model Qwen/Qwen3-8B \
--served-model-name qwen3-8b
docs.vllm.ai ↗
🎛️ LM Studio OpenAI-compat

GUI app for running quantized models. Built-in model browser, chat UI, and a local server with OpenAI API.

URL
Enable in LM Studio
Developer tab → Start server → Port 1234
Check "Enable CORS" for browser access
lmstudio.ai ↗
🔷 llama.cpp OpenAI-compat

Lightweight GGUF model server. Minimal RAM usage. Runs on CPU or GPU. Great for embedding in Docker.

URL
Docker one-liner 
docker run -p 8080:8080 ghcr.io/ggml-org/llama.cpp:server \
  -hf QuantFactory/Meta-Llama-3-8B-GGUF \
  -hff Meta-Llama-3-8B.Q4_K_M.gguf
github.com/ggml-org/llama.cpp ↗

Speech Recognition (STT)

🎙️ faster-whisper-server OpenAI-compat
⚡ ~70× RT GPU ⏰ 0.5–2 s ⭐ large-v3 💾 1.5 GB VRAM

Drop-in local replacement for the Whisper API. GPU-accelerated via CTranslate2. OpenAI-compatible endpoint.

URL
docker-compose snippet 
services:
  whisper:
    image: fedirz/faster-whisper-server:latest-cuda
    ports: ["8000:8000"]
    environment:
      - WHISPER__MODEL=large-v3
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
github.com/fedirz/faster-whisper-server ↗
🔉 whisper.cpp HTTP server
⚡ ~8–15× RT CPU ⏰ 1–5 s ⭐ large-v3 Q5 💾 ~1 GB RAM

Minimal C++ Whisper with a built-in HTTP server. CPU or Metal/CUDA. Low memory, fast on consumer hardware.

URL
Build & run 
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp && cmake -B build && cmake --build build -j
./build/bin/whisper-server \
  -m models/ggml-large-v3.bin --port 8080
github.com/ggml-org/whisper.cpp ↗

Text-to-Speech (local TTS)

🔊 Piper TTS Fast · offline
⚡ ~1× CPU realtime ⏰ <50 ms ⭐ Good (VITS) 💾 ~50 MB RAM

Lightning-fast offline TTS. Runs on CPU in real time. 50+ language voices available. Uses Wyoming protocol (port 10200) — not directly OpenAI-compatible.

URL
Docker 
docker run -p 10200:10200 \
  rhasspy/wyoming-piper \
  --voice en_US-lessac-medium
github.com/rhasspy/piper ↗
🌸 Kokoro FastAPI OpenAI-compat TTS
⚡ ~0.1× CPU RTF ⏰ ~200 ms ⭐ High (82M) 💾 300 MB CPU

Kokoro-82M running behind an OpenAI-compatible TTS endpoint. 11 built-in voices (af_bella, bf_emma, am_adam…). Drop-in replacement for OpenAI’s TTS API.

URL
Docker 
docker run -p 8880:8880 \
  ghcr.io/remsky/kokoro-fastapi-cpu:v0.2.2
# GPU:
docker run -p 8880:8880 --gpus all \
  ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.2
github.com/remsky/Kokoro-FastAPI ↗
🗣️ XTTS v2 Voice cloning
⚡ ~0.5× GPU RTF ⏰ 1–3 s ⭐ High (WAV clone) 💾 3–4 GB VRAM

Coqui XTTS — multilingual voice cloning from a 6-second sample. 17 languages. Compatible with this app’s voice library.

URL
Docker 
docker run -p 8020:80 --gpus all \
  -v /voices:/voices \
  daswer123/xtts-api-server:latest
xtts-api-server ↗
💡
Adding a local service to this app

All OpenAI-compatible services work the same way: open App Routing in the sidebar, add a new backend entry with the local URL, leave the API key field empty, and set it as the default backend for the voice type you want.