diff --git a/CHANGELOG.md b/CHANGELOG.md index 55e0656..7b526de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi - **Share an article or a magazine as a link that opens straight in driving mode.** A *Share* button in the magazine toolbar (whole magazine), one in the playlist header and one in driving mode (current article) build `/#s-magazine?doc=&article=&mode=drive`. They use the browser's share sheet where there is one (it needs https, so on a plain-http LAN address they copy the link instead, with a polite aria-live toast, or show a box with the link if even copying is refused). Opening such a link, also on a first visit and also pasted into an already-open tab, opens the magazine and then driving mode for that article with audio, pictures and text; when the browser blocks autoplay a large *Tap to play* button appears. The link carries ids only (no key or password) and does not change who may reach the app; the ids are validated against a strict pattern and never put into markup, and anything malformed is ignored. Tested with Playwright (desktop deep link, blocked autoplay, hostile parameters, share sheet at 375 px, copy fallback). - **Filter the page grid by status.** Each legend entry is now a toggle button with the number of pages in that status; choose one or several to show only pages that have any of them. It combines with *Show only pages that need me*, the summary says how many pages are shown, *Show all pages* clears everything, a status no page has is greyed out, and the buttons announce their on/off state and work from the keyboard. +### Added +- **Word-level timing with a forced-aligner sidecar.** A new, separate `tts-aligner` container (`aligner/`, torchaudio MMS_FA, CPU by default, optional `--profile gpu`; weights cached in the `aligner-models` volume, no published port, non-root, read-only filesystem, 3 GB limit) answers `POST /align` and `POST /align/batch` with `[{word, start, end}]`. The app (image unchanged, no torch) measures each sentence's words in a background job the first time an article is played (or via `POST .../articles/{id}/align-words`, progress via `GET`), writes `words: [[start_ms, end_ms], ...]` per segment into `audio/.timings.json` (now version 2; readers of version 1 keep working) and `/timings` returns `words` per sentence. The player marks the word from those times in Compact, Lyrics (new per-word marking) and the playlist, and falls back to the previous length-based estimate when no word times exist or the sidecar is down. Measured on one article: after-pause word starts are within 29 ms on average (max 170 ms) of the audio's energy onsets. + ### Fixed - **Sentence timings now also work for interview renders.** Every recorded audio file carries `"interview": true` (set whenever a second voice is configured, even without speaker labels), and the timings endpoint and EPUB export discarded all such records, so the player and the EPUB fell back to estimated sentence positions for every current article. The record is now matched to the text again, for interview renders by the label-stripped length the renderer recorded, so `/timings` returns exact (or anchored) sentence times and speech onsets for these articles too. Records for text edited since the render are still ignored. diff --git a/Manual.md b/Manual.md index a0c877b..f47d3a5 100644 --- a/Manual.md +++ b/Manual.md @@ -291,7 +291,8 @@ turned into a podcast discussion between two or three voices you pick. **Hands-free:** while a track plays, the lock screen, notification shade, Bluetooth headset and car display show the article title, magazine and first-page artwork and offer play/pause, previous/next article and ±15 s (and seeking, where the device offers it). In driving mode the screen is kept awake while audio plays and allowed to sleep when it is paused or you leave. Both are optional browser features; where one is missing or refused, playback itself is unaffected. Opening driving mode resumes the first unfinished article at the position you left it. **Layout and views:** driving mode is a top bar, a content area and a controls dock. The dock (position slider, time and playback speed, then previous, back 15 s, play/pause, forward 15 s, next) is always completely visible at the bottom and is never covered or pushed off screen, on phones from 320x568, tablets and in landscape. The top bar has *Share*, a *menu* button that shows the current view, *Index* and *Exit*. The menu opens a flyout: choose **Picture** (the picture fills the area), **Lyrics** (the whole text) or **Compact** (picture plus three lines, the default; in landscape side by side), and switch *Show text*, *Show pictures* and *Minimize text and picture* on or off. Keyboard: Enter or Arrow Down opens the menu, arrows/Home/End move, Enter chooses, Escape or a tap outside closes it and returns focus to the button. **Index:** the *Index* button opens a list of all articles of the magazine with their length (articles without audio are dimmed and marked *no audio*). The article that is playing is highlighted; tap it or press Enter on another to jump there while driving mode keeps playing. Escape or the close button closes the list (Tab stays inside it while open); the controls below stay visible and usable by touch. Lengths come from the already generated audio, nothing is recomputed. **Speed** is the button after the time next to the slider. **Picture zoom:** double-tap or double-click, pinch, mouse wheel, the + and - keys, or the zoom in / zoom out / reset buttons on the picture; drag (or arrow keys) to move a zoomed picture; Escape or the reset button returns to full view. **Lyrics:** the spoken sentence is large and kept in the middle, earlier and later ones are dimmed; tap or press Enter on a sentence to play from it; scrolling by hand stops the automatic following and shows *Back to current*. With reduced motion switched on in your device the text jumps instead of gliding. **Minimize** (in the menu) reduces the area to the title and the current line so the dock keeps minimal text; choose it again to expand. Your view and the minimized state are remembered in this browser only. Titles wrap to two lines (with an ellipsis beyond that), picture captions to two lines. **Three lines of text:** driving mode shows the previous sentence (small), the current one (large, the spoken word marked) and the next (small) in a box of fixed height, so nothing moves around it. Tap or press Enter on the previous or next line to jump to that sentence. The text is deliberately not announced by screen readers sentence by sentence; the play/pause button and the position slider stay the way to control it. With reduced motion switched on in your device, lines change without animation. On a phone (375 px wide) the sentences wrap and a very long current sentence scrolls inside its box. Turn the text off with *Show text* in the menu to see only the picture. - **Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. The highlight lights up a sentence just before its first word is heard (each sentence starts with a short silence, which is not counted). Older audio, interview-style audio and audio whose text was edited afterwards has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync. + **Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. The highlight lights up a sentence just before its first word is heard (each sentence starts with a short silence, which is not counted). Audio from before sentence times were recorded, and audio whose text was edited afterwards, has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync. Interview-voice audio is covered like any other. + **Word by word:** the first time an article is played, the app asks the forced-aligner service (the separate `tts-aligner` container, see below) to measure where each word starts and ends and stores that beside the audio. Playback never waits for it: until it is ready (usually well under a minute for a magazine article) the marked word is estimated from word length as before, and it switches to the measured word on its own. The marked word is the last one that has started, so it stays lit through a pause; the *Lyrics* view now marks the spoken word too (as do *Compact* and the playlist text). If the service is not running, nothing changes for you and the app tries again later (at most every 10 minutes per article). Prerequisites: `docker compose up -d tts-aligner` (first start downloads about 1.2 GB of model weights into the `aligner-models` volume; after that it needs no internet). `ALIGNER_URL` empty turns word timing off. Privacy: text and audio go only to your own container on the internal Docker network. Accessibility: the marking is visual only and adds no announcements; with reduced motion nothing animates. API: `POST /api/magazine/{doc}/articles/{id}/align-words` starts it explicitly, `GET` shows progress. **Share a link:** the *Share* button in the toolbar (the whole magazine), the share button in the playlist header and the one at the top left of driving mode (the current article) share a link of the form `/#s-magazine?doc=&article=
&mode=drive`. Where the browser has a share sheet (phones, and any browser on https) it opens; otherwise the link is copied and a confirmation is announced (on a plain-http address a box with the link is shown to copy by hand). Opening the link goes straight to that article in driving mode with audio, pictures and text; if the browser refuses to start audio without a tap, a large *Tap to play* button appears. A link for the whole magazine opens at its first unfinished article. **Privacy and access:** the link contains only ids, no password or key. It is just an address: whoever opens it needs to reach this app the same way as for any other page (this app has no user accounts; access is controlled by your network or reverse proxy), and an unknown or removed article simply shows a message. Only articles that already have audio can be shared. A link whose parameters are not in the expected form is ignored. **Expected result:** the article plays from its headline to its final paragraph. diff --git a/PROJECT_MAP.md b/PROJECT_MAP.md index e0f1250..9ef5061 100644 --- a/PROJECT_MAP.md +++ b/PROJECT_MAP.md @@ -77,6 +77,7 @@ which is the writable volume. | `VOICES_DIR` / `OUTPUT_DIR` | voice WAV library | | `TTS_CONTAINER_NAMES` | containers restarted after a voice change | | `DOCKER_HOST` | points at the socket proxy; the raw socket is not mounted | +| `ALIGNER_URL` | forced-aligner sidecar for word timing (`http://tts-aligner:8000`; empty = off) | | `llm_url` / `llm_model` / `llm_api_key` | settings.json, OpenAI-compatible LLM | ## Pending decisions diff --git a/aligner/.dockerignore b/aligner/.dockerignore new file mode 100644 index 0000000..676f6fb --- /dev/null +++ b/aligner/.dockerignore @@ -0,0 +1,3 @@ +tests +__pycache__ +*.pyc diff --git a/aligner/Dockerfile b/aligner/Dockerfile new file mode 100644 index 0000000..e367fb2 --- /dev/null +++ b/aligner/Dockerfile @@ -0,0 +1,19 @@ +# Forced-aligner sidecar: word-level timing with torchaudio MMS_FA. Not part of the app image. +# CPU (default): docker compose up -d --build tts-aligner +# GPU (optional): docker compose --profile gpu up -d --build tts-aligner-gpu +FROM python:3.12-slim-bookworm +ARG TORCH_INDEX=https://download.pytorch.org/whl/cpu +ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1 TORCH_HOME=/models/torch HF_HOME=/models/hf +RUN apt-get update && apt-get install -y --no-install-recommends libsndfile1 curl \ + && rm -rf /var/lib/apt/lists/* +COPY requirements.txt /tmp/requirements.txt +RUN pip install --index-url "$TORCH_INDEX" torch==2.10.0 torchaudio==2.10.0 \ + && pip install -r /tmp/requirements.txt +RUN useradd --uid 10001 --no-create-home --shell /usr/sbin/nologin aligner && mkdir -p /models && chown aligner /models +WORKDIR /srv +COPY app.py . +USER aligner +EXPOSE 8000 +HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \ + CMD curl -fsS http://127.0.0.1:8000/health || exit 1 +CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1", "--no-access-log"] diff --git a/aligner/app.py b/aligner/app.py new file mode 100644 index 0000000..2dd1975 --- /dev/null +++ b/aligner/app.py @@ -0,0 +1,230 @@ +"""Forced aligner sidecar: where does each word of a known text start and end in its audio? + +POST /align multipart: audio (wav/mp3/flac file), text, language, [start, end] seconds +POST /align/batch multipart: audio, spans (JSON [{start,end,text}]), language -- many sentences, one upload +GET /health + +Times are seconds in the uploaded audio. Words are the whitespace-separated tokens of the text, in order; a +token with no letters (digits, punctuation) cannot be aligned and gets the gap between its neighbours +(aligned=false). Model: torchaudio MMS_FA (weights are fetched once into TORCH_HOME, a volume, never baked in). +No secrets, no auth: the service is reachable only on the internal Docker network. +""" +from __future__ import annotations + +import asyncio +import io +import json +import os +import re +import threading +import time +import unicodedata +from typing import Any + +import numpy as np +from fastapi import FastAPI, File, Form, HTTPException, UploadFile + +SAMPLE_RATE = 16000 +MAX_AUDIO_BYTES = int(os.environ.get("ALIGNER_MAX_AUDIO_BYTES", 40 * 1024 * 1024)) +MAX_SPAN_SECONDS = float(os.environ.get("ALIGNER_MAX_SPAN_SECONDS", 120)) +MAX_AUDIO_SECONDS = float(os.environ.get("ALIGNER_MAX_AUDIO_SECONDS", 900)) +MAX_SPANS = int(os.environ.get("ALIGNER_MAX_SPANS", 400)) +MAX_TEXT_CHARS = int(os.environ.get("ALIGNER_MAX_TEXT_CHARS", 20000)) +REQUEST_TIMEOUT = float(os.environ.get("ALIGNER_REQUEST_TIMEOUT", 110)) +QUEUE_WAIT = float(os.environ.get("ALIGNER_QUEUE_WAIT", 30)) +DEVICE = os.environ.get("ALIGNER_DEVICE", "cpu") +_LANG_RE = re.compile(r"^[A-Za-z]{2,3}(-[A-Za-z0-9]{2,8})?$") + +app = FastAPI(title="aligner", docs_url=None, redoc_url=None, openapi_url=None) +_gate = asyncio.Semaphore(1) # one alignment at a time: the model is the scarce resource +_state: dict[str, Any] = {"bundle": None, "model": None, "tok": None, "aligner": None, "error": None} +_load_lock = threading.Lock() + + +# ── text ─────────────────────────────────────────────────────────────────────── + +def normalize_word(word: str) -> str: + """Romanize one token to the a-z' alphabet of the MMS_FA model (umlauts and accents folded).""" + w = unicodedata.normalize("NFKD", word.lower().replace("ß", "ss").replace("æ", "ae").replace("œ", "oe")) + w = "".join(c for c in w if not unicodedata.combining(c)) + return re.sub(r"[^a-z']", "", w) + + +def tokenize(text: str) -> list[str]: + return text.split() + + +# ── model ────────────────────────────────────────────────────────────────────── + +def _load() -> None: + with _load_lock: + if _state["model"] is not None: + return + import torch # imported lazily: /health works before the model is loaded + import torchaudio + torch.set_num_threads(int(os.environ.get("ALIGNER_THREADS", "4"))) + bundle = torchaudio.pipelines.MMS_FA + model = bundle.get_model(with_star=False).to(DEVICE).eval() + _state.update(bundle=bundle, model=model, tok=bundle.get_tokenizer(), aligner=bundle.get_aligner(), error=None) + + +def align_samples(samples: np.ndarray, words: list[str]) -> list[tuple[float, float, bool]]: + """Per whitespace token (start, end, aligned) in seconds relative to `samples` (16 kHz mono float32).""" + import torch + norm = [normalize_word(w) for w in words] + keep = [i for i, n in enumerate(norm) if n] + out: list[list[Any]] = [[None, None, False] for _ in words] + if keep and len(samples) >= 400: + wav = torch.from_numpy(samples)[None].to(DEVICE) + with torch.inference_mode(): + emission, _ = _state["model"](wav) + spans = _state["aligner"](emission[0], _state["tok"]([norm[i] for i in keep])) + frame = len(samples) / emission.shape[1] / SAMPLE_RATE + for i, sp in zip(keep, spans): + out[i] = [sp[0].start * frame, sp[-1].end * frame, True] + # tokens without letters: the gap between their neighbours + total = len(samples) / SAMPLE_RATE + for i, o in enumerate(out): + if o[2]: + continue + prev_end = next((out[j][1] for j in range(i - 1, -1, -1) if out[j][2]), 0.0) + next_start = next((out[j][0] for j in range(i + 1, len(out)) if out[j][2]), total) + o[0], o[1] = prev_end, max(prev_end, next_start) + return [(float(a), float(b), bool(c)) for a, b, c in out] + + +# ── audio ────────────────────────────────────────────────────────────────────── + +def decode(data: bytes) -> np.ndarray: + """Any libsndfile format (wav, flac, mp3) to 16 kHz mono float32.""" + import soundfile as sf + try: + with sf.SoundFile(io.BytesIO(data)) as f: + if f.frames / max(1, f.samplerate) > MAX_AUDIO_SECONDS: + raise HTTPException(413, f"audio longer than {MAX_AUDIO_SECONDS:.0f} s") + x = f.read(dtype="float32", always_2d=True) + sr = f.samplerate + except HTTPException: + raise + except Exception: + raise HTTPException(415, "audio is not a readable wav/mp3/flac file") + x = x.mean(axis=1) + if sr != SAMPLE_RATE: + import torch + import torchaudio.functional as F + x = F.resample(torch.from_numpy(x), sr, SAMPLE_RATE).numpy() + return np.ascontiguousarray(x, dtype=np.float32) + + +def _check_language(language: str) -> None: + if not _LANG_RE.match(language or ""): + raise HTTPException(422, "language must be an ISO code such as 'de' or 'en'") + + +def _check_text(text: str, strict: bool = True) -> list[str]: + if not isinstance(text, str) or len(text) > MAX_TEXT_CHARS: + raise HTTPException(422, f"text must be a string of at most {MAX_TEXT_CHARS} characters") + words = tokenize(text) + if not words: + raise HTTPException(422, "text is empty") + if strict and not any(normalize_word(w) for w in words): + raise HTTPException(422, "text has no letters the model can align (latin alphabets only)") + return words + + +def _slice(x: np.ndarray, start: float, end: float) -> tuple[np.ndarray, float]: + total = len(x) / SAMPLE_RATE + if not (0 <= start < end <= total + 0.05): + raise HTTPException(422, f"span {start}-{end} s is outside the audio (0-{total:.2f} s)") + if end - start > MAX_SPAN_SECONDS: + raise HTTPException(422, f"span longer than {MAX_SPAN_SECONDS:.0f} s") + return x[int(start * SAMPLE_RATE):int(min(end, total) * SAMPLE_RATE)], start + + +def _run(x: np.ndarray, spans: list[tuple[float, float, str]], deadline: float) -> list[list[dict[str, Any]]]: + _load() + result = [] + for start, end, text in spans: + if time.monotonic() > deadline: + raise HTTPException(504, "alignment took longer than the request limit") + words = tokenize(text) + piece, offset = _slice(x, start, end) + times = align_samples(piece, words) + result.append([{"word": w, "start": round(offset + a, 3), "end": round(offset + b, 3), "aligned": ok} + for w, (a, b, ok) in zip(words, times)]) + return result + + +async def _guarded(x: np.ndarray, spans: list[tuple[float, float, str]]) -> list[list[dict[str, Any]]]: + try: + await asyncio.wait_for(_gate.acquire(), QUEUE_WAIT) + except asyncio.TimeoutError: + raise HTTPException(503, "aligner busy, retry shortly", headers={"Retry-After": "10"}) + try: + deadline = time.monotonic() + REQUEST_TIMEOUT + return await asyncio.wait_for(asyncio.to_thread(_run, x, spans, deadline), REQUEST_TIMEOUT + 5) + except asyncio.TimeoutError: + raise HTTPException(504, "alignment took longer than the request limit") + except HTTPException: + raise + except Exception as exc: # model download/load failure and the like + _state["error"] = type(exc).__name__ + raise HTTPException(503, f"aligner model unavailable ({type(exc).__name__})") + finally: + _gate.release() + + +async def _read_audio(audio: UploadFile) -> np.ndarray: + data = await audio.read(MAX_AUDIO_BYTES + 1) + if len(data) > MAX_AUDIO_BYTES: + raise HTTPException(413, "audio file too large") + return await asyncio.to_thread(decode, data) + + +@app.on_event("startup") +async def _preload() -> None: + """Load (and on first run download) the model in the background so the first request is not slow.""" + def work() -> None: + try: + _load() + except Exception as exc: # reported by /health, retried on the first request + _state["error"] = type(exc).__name__ + threading.Thread(target=work, name="preload", daemon=True).start() + + +# ── routes ───────────────────────────────────────────────────────────────────── + +@app.get("/health") +async def health(): + return {"status": "ok", "model_loaded": _state["model"] is not None, "device": DEVICE, + "last_error": _state["error"]} + + +@app.post("/align") +async def align(audio: UploadFile = File(...), text: str = Form(...), language: str = Form("de"), + start: float | None = Form(None), end: float | None = Form(None)): + _check_language(language) + _check_text(text) + x = await _read_audio(audio) + s = 0.0 if start is None else start + e = len(x) / SAMPLE_RATE if end is None else end + words = (await _guarded(x, [(s, e, text)]))[0] + return {"language": language, "words": words} + + +@app.post("/align/batch") +async def align_batch(audio: UploadFile = File(...), spans: str = Form(...), language: str = Form("de")): + _check_language(language) + try: + raw = json.loads(spans) + parsed = [(float(r["start"]), float(r["end"]), str(r["text"])) for r in raw] + except (ValueError, KeyError, TypeError): + raise HTTPException(422, "spans must be a JSON list of {start, end, text}") + if not parsed or len(parsed) > MAX_SPANS: + raise HTTPException(422, f"between 1 and {MAX_SPANS} spans are allowed") + for _, _, t in parsed: + _check_text(t, strict=False) # a sentence of digits only just gets its gap back + if sum(len(t) for _, _, t in parsed) > MAX_TEXT_CHARS * 4: + raise HTTPException(422, "too much text") + x = await _read_audio(audio) + return {"language": language, "spans": await _guarded(x, parsed)} diff --git a/aligner/requirements.txt b/aligner/requirements.txt new file mode 100644 index 0000000..17c94e1 --- /dev/null +++ b/aligner/requirements.txt @@ -0,0 +1,6 @@ +# torch / torchaudio are installed separately in the Dockerfile (CPU or CUDA wheel index) +fastapi==0.115.6 +uvicorn==0.32.1 +python-multipart==0.0.20 +soundfile==0.13.1 +numpy==2.2.6 diff --git a/aligner/tests/test_app.py b/aligner/tests/test_app.py new file mode 100644 index 0000000..6ba67cf --- /dev/null +++ b/aligner/tests/test_app.py @@ -0,0 +1,84 @@ +"""Validation and text-handling tests for the aligner; they never load the model.""" +import io +import json +import os +import sys +import unittest +import wave + +import numpy as np + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) +from fastapi.testclient import TestClient # noqa: E402 + +import app as aligner # noqa: E402 + + +def wav_bytes(seconds=1.0, sr=16000): + buf = io.BytesIO() + with wave.open(buf, "wb") as w: + w.setnchannels(1); w.setsampwidth(2); w.setframerate(sr) + w.writeframes((np.zeros(int(seconds * sr))).astype(" dict[str, Any] | None: data = json.loads(mp3_path.with_suffix(".timings.json").read_text("utf-8")) except (OSError, ValueError): return None - return data if isinstance(data, dict) and data.get("version") == 1 and isinstance(data.get("segments"), list) else None + return data if isinstance(data, dict) and data.get("version") in (1, 2) and isinstance(data.get("segments"), list) else None def tag_article_mp3(path: Path, article: dict[str, Any], lyrics: str, *, diff --git a/core/word_alignment.py b/core/word_alignment.py new file mode 100644 index 0000000..595cb03 --- /dev/null +++ b/core/word_alignment.py @@ -0,0 +1,233 @@ +"""Word-level timing of a narrated article, measured by the forced-aligner sidecar (aligner/). + +The MP3 is rendered sentence by sentence, so `audio/.timings.json` already says where every sentence sits. +This module asks the sidecar where each *word* of a sentence starts and ends inside that span and records the +answer as `words: [[start_ms, end_ms], ...]` per segment (the file becomes version 2; version 1 readers simply +ignore the key). Never blocks playback: it runs in a background thread, retries with backoff, and when the +sidecar is down it gives up quietly -- the player then interpolates words inside each sentence as before. +""" +from __future__ import annotations + +import io +import json +import logging +import os +import threading +import time +from pathlib import Path +from typing import Any, Callable, Sequence + +import requests + +from core import epub_export + +log = logging.getLogger(__name__) + +CHUNK_SECONDS = 45.0 # audio per sidecar request: bounded time per call and real progress +CHUNK_MAX_SPANS = 25 +ATTEMPTS = 3 +CONNECT_TIMEOUT = 3.0 +READ_TIMEOUT = 150.0 +FAILURE_COOLDOWN = 600.0 # after a failed run, do not retry on every timings request +WORDS_VERSION = 2 + +_jobs: dict[tuple[str, str], dict[str, Any]] = {} +_lock = threading.Lock() + + +def aligner_url() -> str: + return os.environ.get("ALIGNER_URL", "").strip().rstrip("/") + + +# ── planning ─────────────────────────────────────────────────────────────────── + +def alignment_jobs(units: Sequence[dict[str, Any]], blocks: Sequence[dict[str, Any]], + interview: bool) -> list[dict[str, Any]]: + """Group sentence units into alignment jobs: [{units: [idx...], start, end, text, lead}] (ms). + + A block whose sentences all have their own recorded utterance gets one job per sentence (tightest span). + A block whose spans were only anchored (counts disagreed) is aligned as one span and split afterwards. + `lead` counts leading tokens that were never spoken (an interview "Speaker:" label) and get no time. + """ + from core.magazine import _INTERVIEW_LABEL_RE + by_block: dict[int, list[int]] = {} + for idx, u in enumerate(units): + by_block.setdefault(u["block"], []).append(idx) + jobs: list[dict[str, Any]] = [] + for block, idxs in by_block.items(): + lead = 0 + if interview: + m = _INTERVIEW_LABEL_RE.match(units[idxs[0]]["text"]) + if m: + lead = len(units[idxs[0]]["text"].split()) - len(m.group(2).split()) + exact = all(units[i].get("nth") is not None for i in idxs) + groups = [[i] for i in idxs] if exact else [idxs] + for g in groups: + text = " ".join(units[i]["text"] for i in g) + first_lead = lead if g[0] == idxs[0] else 0 + spoken = " ".join(text.split()[first_lead:]) + if not spoken: + continue + jobs.append({"units": g, "start": units[g[0]]["start"], "end": units[g[-1]]["end"], + "text": spoken, "lead": first_lead}) + return jobs + + +def _chunks(jobs: Sequence[dict[str, Any]]) -> list[list[dict[str, Any]]]: + out: list[list[dict[str, Any]]] = [] + cur: list[dict[str, Any]] = [] + for j in jobs: + if cur and (len(cur) >= CHUNK_MAX_SPANS or (j["end"] - cur[0]["start"]) / 1000 > CHUNK_SECONDS): + out.append(cur) + cur = [] + cur.append(j) + if cur: + out.append(cur) + return out + + +# ── sidecar client ───────────────────────────────────────────────────────────── + +class AlignerError(RuntimeError): + """The sidecar could not be used (down, busy too long, rejected the request).""" + + +def _post_chunk(url: str, wav: bytes, spans: list[dict[str, Any]], language: str, + sleep: Callable[[float], None] = time.sleep) -> list[list[dict[str, Any]]]: + last: Exception | None = None + for attempt in range(ATTEMPTS): + try: + r = requests.post(f"{url}/align/batch", files={"audio": ("chunk.wav", wav, "audio/wav")}, + data={"spans": json.dumps(spans), "language": language}, + timeout=(CONNECT_TIMEOUT, READ_TIMEOUT)) + if r.status_code == 200: + return r.json()["spans"] + if 400 <= r.status_code < 500 and r.status_code != 429: + raise AlignerError(f"aligner rejected the request ({r.status_code})") + last = AlignerError(f"aligner answered {r.status_code}") + except requests.RequestException as exc: + last = AlignerError(f"aligner unreachable ({type(exc).__name__})") + except (ValueError, KeyError) as exc: + last = AlignerError(f"aligner sent an unreadable answer ({type(exc).__name__})") + if attempt < ATTEMPTS - 1: + sleep(2 ** attempt) + raise last or AlignerError("aligner failed") + + +def _wav_slice(audio: Any, start_ms: int, end_ms: int) -> bytes: + buf = io.BytesIO() + audio[start_ms:end_ms].set_frame_rate(16000).set_channels(1).export(buf, format="wav") + return buf.getvalue() + + +# ── the job ──────────────────────────────────────────────────────────────────── + +def align_words(mp3: Path, units: Sequence[dict[str, Any]], jobs: Sequence[dict[str, Any]], language: str, *, + url: str, progress: Callable[[int, int], None] | None = None, + post: Callable[..., Any] = _post_chunk) -> dict[int, list[list[int]]]: + """Ask the sidecar for the words of every job; returns {unit index: [[start_ms, end_ms], ...]} for the + units it could time (a unit's list has one pair per whitespace token, label tokens first at zero length).""" + from pydub import AudioSegment + audio = AudioSegment.from_file(mp3) + chunks = _chunks(jobs) + result: dict[int, list[list[int]]] = {} + done = 0 + for chunk in chunks: + lo = max(0, int(chunk[0]["start"]) - 100) + hi = min(len(audio), int(chunk[-1]["end"]) + 100) + spans = [{"start": round((j["start"] - lo) / 1000, 3), "end": round(min(j["end"], hi) / 1000 - lo / 1000, 3), + "text": j["text"]} for j in chunk] + answer = post(url, _wav_slice(audio, lo, hi), spans, language) + for j, words in zip(chunk, answer): + pairs = [[int(round(lo + w["start"] * 1000)), int(round(lo + w["end"] * 1000))] for w in words] + pairs = [[j["start"], j["start"]]] * j["lead"] + pairs + at = 0 + for i in j["units"]: + n = len(units[i]["text"].split()) + result[i] = pairs[at:at + n] + at += n + done += 1 + if progress: + progress(done, len(jobs)) + return result + + +def build_segment_words(units: Sequence[dict[str, Any]], per_unit: dict[int, list[list[int]]], + n_segments: int) -> dict[int, list[list[int]]]: + """Per record segment, the words of all its sentences in order -- only for segments fully covered.""" + out: dict[int, list[list[int]]] = {} + bad: set[int] = set() + for idx, u in enumerate(units): + k = u["seg"] + words = per_unit.get(idx) + if words is None or len(words) != len(u["text"].split()): + bad.add(k) + continue + out.setdefault(k, []).extend(words) + return {k: v for k, v in out.items() if k not in bad and 0 <= k < n_segments} + + +def write_words(mp3: Path, segment_words: dict[int, list[list[int]]]) -> bool: + """Add `words` to the record (atomic replace; everything else in the file is kept as it was).""" + path = mp3.with_suffix(".timings.json") + try: + rec = json.loads(path.read_text("utf-8")) + segs = rec["segments"] + for k, words in segment_words.items(): + segs[k]["words"] = words + rec["version"] = WORDS_VERSION + rec["words_engine"] = "mms_fa" + tmp = path.with_suffix(".tmp") + tmp.write_text(json.dumps(rec), "utf-8") + tmp.replace(path) + return True + except (OSError, ValueError, KeyError, IndexError): + log.warning("Could not write word timings for %s", mp3.name) + return False + + +# ── background job registry ──────────────────────────────────────────────────── + +def status(doc_id: str, article_id: str) -> dict[str, Any]: + with _lock: + return dict(_jobs.get((doc_id, article_id)) or {"state": "idle", "done": 0, "total": 0}) + + +def should_autostart(doc_id: str, article_id: str) -> bool: + if not aligner_url(): + return False + st = status(doc_id, article_id) + if st["state"] == "running": + return False + # Whatever the last outcome, do not re-run on every request: a run that finished without covering every + # sentence (or after the audio was re-rendered) is retried after the cooldown. + return time.time() - st.get("finished", 0) >= FAILURE_COOLDOWN + + +def start(doc_id: str, article_id: str, run: Callable[[Callable[[int, int], None]], dict[str, Any]]) -> bool: + """Run `run(progress)` in a daemon thread unless this article is already being aligned.""" + key = (doc_id, article_id) + with _lock: + if (_jobs.get(key) or {}).get("state") == "running": + return False + _jobs[key] = {"state": "running", "done": 0, "total": 0, "started": time.time()} + + def progress(done: int, total: int) -> None: + with _lock: + _jobs[key].update(done=done, total=total) + + def target() -> None: + try: + info = run(progress) + end = {"state": "done", **info} + except AlignerError as exc: + log.warning("Word alignment for %s skipped: %s", article_id, exc) + end = {"state": "failed", "error": str(exc)} + except Exception as exc: # never let a worker thread die noisily + log.exception("Word alignment for %s failed", article_id) + end = {"state": "failed", "error": type(exc).__name__} + with _lock: + _jobs[key].update(end, finished=time.time()) + + threading.Thread(target=target, name=f"align-{article_id}", daemon=True).start() + return True diff --git a/docker-compose.yml b/docker-compose.yml index 7fdb4d6..e8be8b8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -83,6 +83,8 @@ services: - TTS_CONTAINER_NAMES=tts-faster-qwen3-tts-voiceclone,tts-faster-qwen3-tts-streaming - VOICE_DESIGN_MODEL=Qwen3-TTS-12Hz-1.7B-VoiceDesign - DOCKER_HOST=tcp://docker-socket-proxy:2375 + # Word-level forced aligner sidecar; empty disables word timing (sentence timing keeps working) + - ALIGNER_URL=${ALIGNER_URL-http://tts-aligner:8000} extra_hosts: - "host.docker.internal:host-gateway" @@ -90,6 +92,86 @@ services: networks: - dgx_net + # Word-level forced aligner (torchaudio MMS_FA). Separate image so the app stays torch-free. Reachable only on + # the Compose network (no published port); the app calls http://tts-aligner:8000. Weights (~1.2 GB) are + # downloaded once into the named volume; after that the service needs no outbound access. + tts-aligner: + build: + context: ./aligner + image: tts-aligner:local + container_name: tts-aligner + restart: unless-stopped + read_only: true + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + tmpfs: + - /tmp:size=256m,mode=1777 + mem_limit: "${ALIGNER_MEM_LIMIT:-3g}" + cpus: "${ALIGNER_CPUS:-4}" + environment: + - ALIGNER_DEVICE=cpu + - ALIGNER_THREADS=4 + expose: + - "8000" + volumes: + - aligner-models:/models:rw + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 60s + networks: + - dgx_net + + # Optional GPU variant: `docker compose --profile gpu up -d --build tts-aligner-gpu` (stop tts-aligner first; + # both answer to the name tts-aligner). Off by default because the DGX GPU is often full. + tts-aligner-gpu: + profiles: ["gpu"] + build: + context: ./aligner + args: + TORCH_INDEX: https://download.pytorch.org/whl/cu130 + image: tts-aligner:gpu + container_name: tts-aligner-gpu + restart: unless-stopped + read_only: true + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + tmpfs: + - /tmp:size=256m,mode=1777 + mem_limit: "${ALIGNER_MEM_LIMIT:-3g}" + environment: + - ALIGNER_DEVICE=cuda + expose: + - "8000" + volumes: + - aligner-models:/models:rw + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: 1 + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 60s + networks: + dgx_net: + aliases: + - tts-aligner + networks: dgx_net: external: true + +volumes: + aligner-models: diff --git a/routes/magazine.py b/routes/magazine.py index 76bcbe5..dc783d1 100644 --- a/routes/magazine.py +++ b/routes/magazine.py @@ -35,6 +35,7 @@ from fastapi import APIRouter, File, HTTPException, Request, UploadFile from fastapi.responses import FileResponse, Response from core import magazine as mag +from core import word_alignment from core.config import _load_settings from core.constants import CONFIG_DIR from core.database import ( @@ -754,11 +755,86 @@ async def magazine_article_timings(doc_id: str, article_id: str): k = 0 seen.add(u["block"]) flat.append(on[k] if k is not None and k < len(on) else 0) - return {"available": True, "quality": quality, "sentences": [ + sentences = [ {"text": u["text"], "block": u["block"], "block_id": blocks[u["block"]].get("id"), "start": round(u["start"] / 1000, 3), "end": round(u["end"] / 1000, 3), "speech": round((u["start"] + min(max(0, o), max(0, u["end"] - u["start"] - 10))) / 1000, 3)} - for u, o in zip(units, flat)]} + for u, o in zip(units, flat)] + words_state = _attach_words(sentences, units, segs) + if words_state != "ready" and word_alignment.should_autostart(doc_id, article_id): + # Lazily, in the background: playback never waits for it; the next request carries the words. + _start_word_alignment(d, doc_id, article_id) + words_state = "running" + return {"available": True, "quality": quality, "words": words_state, + "words_progress": word_alignment.status(doc_id, article_id), "sentences": sentences} + + +def _attach_words(sentences: list[dict[str, Any]], units: list[dict[str, Any]], segs: list[dict[str, Any]]) -> str: + """Add `words: [[start, end], ...]` (seconds, one pair per whitespace token) to every sentence whose + segment has recorded word times that fit its text. Returns ready | partial | none.""" + have = 0 + cursor: dict[int, int] = {} + for s, u in zip(sentences, units): + k = u.get("seg") + w = segs[k].get("words") if k is not None and 0 <= k < len(segs) else None + n = len(u["text"].split()) + at = cursor.get(k, 0) + cursor[k] = at + n + if isinstance(w, list) and len(w) >= at + n and all(isinstance(p, list) and len(p) == 2 for p in w[at:at + n]): + s["words"] = [[round(a / 1000, 3), round(b / 1000, 3)] for a, b in w[at:at + n]] + have += 1 + if not have: + return "none" + return "ready" if have == len(sentences) else "partial" + + +def _start_word_alignment(d: Path, doc_id: str, article_id: str) -> bool: + url = word_alignment.aligner_url() + if not url: + return False + + def run(progress): + from core import epub_export as ee + layout = mag.load_layout(d) + target = next((a for a in mag.load_articles(d) if a.get("id") == article_id), None) + mp3 = d / "audio" / f"{article_id}.mp3" + timings = mag.read_audio_timings(mp3) if mp3.is_file() else None + if target is None or not timings: + return {"words": 0} + blocks = mag.article_text(layout, target, fuse=True, glossary=target.get("glossary")) + units, quality = ee.plan_sentences(blocks, float(timings.get("duration_ms") or 0), timings, + mag.spoken_chars_for(blocks, timings)) + if quality == "estimated": + return {"words": 0} + jobs = word_alignment.alignment_jobs(units, blocks, bool(timings.get("interview"))) + language = mag.detect_article_language(b["text"] for b in blocks) + per_unit = word_alignment.align_words(mp3, units, jobs, language, url=url, progress=progress) + seg_words = word_alignment.build_segment_words(units, per_unit, len(timings["segments"])) + word_alignment.write_words(mp3, seg_words) + return {"words": sum(len(v) for v in seg_words.values())} + + return word_alignment.start(doc_id, article_id, run) + + +@router.get("/api/magazine/{doc_id}/articles/{article_id}/align-words") +async def magazine_align_words_status(doc_id: str, article_id: str): + """Progress of the word-timing job: {state: idle|running|done|failed, done, total, error?, enabled}.""" + _require_magazine(doc_id) + article_id = _article_id(article_id) + return {**word_alignment.status(doc_id, article_id), "enabled": bool(word_alignment.aligner_url())} + + +@router.post("/api/magazine/{doc_id}/articles/{article_id}/align-words", status_code=202) +async def magazine_align_words(doc_id: str, article_id: str): + """Measure word times for this article's audio with the aligner sidecar (background job, idempotent).""" + d, _ = _require_magazine(doc_id) + article_id = _article_id(article_id) + if not word_alignment.aligner_url(): + raise HTTPException(503, "Word alignment is not configured (ALIGNER_URL is empty)") + if not (d / "audio" / f"{article_id}.timings.json").is_file(): + raise HTTPException(404, "This article has no sentence timings yet -- render its audio first") + started = _start_word_alignment(d, doc_id, article_id) + return {**word_alignment.status(doc_id, article_id), "started": started} @router.post("/api/magazine/{doc_id}/articles/auto-link") diff --git a/static/js/magazine.js b/static/js/magazine.js index a98d72c..5f62c84 100644 --- a/static/js/magazine.js +++ b/static/js/magazine.js @@ -5141,8 +5141,11 @@ function playlistTimeline(duration) { if (s.type === 'caption') return { start: cursor, end: cursor, speech: cursor, skip: true }; const t = exact[n++]; cursor = t.end; - const speech = Number.isFinite(t.speech) ? Math.min(t.end, Math.max(t.start, t.speech)) : t.start; - return { start: t.start, end: t.end, speech, skip: false }; + let speech = Number.isFinite(t.speech) ? Math.min(t.end, Math.max(t.start, t.speech)) : t.start; + // Measured word times (forced aligner) when they fit this sentence's words; else the length-based estimate. + const wt = t.words && t.words.length === s.words.length ? t.words : null; + if (wt) speech = Math.min(t.end, Math.max(t.start, wt[0][0])); + return { start: t.start, end: t.end, speech, skip: false, wordTimes: wt }; }); } else { const weights = sentences.map((s) => s.type === 'caption' ? 0 : Math.max(6, s.text.replace(/\s+/g, ' ').length)); @@ -5191,6 +5194,32 @@ function playlistLocal(span, time) { } // Which word is being said, by the share of the sentence's characters spoken so far. +// Measured word times for one sentence: [[start, end], ...] in seconds, or null when absent or malformed. +function magCleanWordTimes(w) { + if (!Array.isArray(w) || !w.length) return null; + const out = w.map((p) => (Array.isArray(p) && p.length === 2 ? [Number(p[0]), Number(p[1])] : null)); + return out.every((p) => p && Number.isFinite(p[0]) && Number.isFinite(p[1])) ? out : null; +} + +// The word being said at `time`: with measured word times the last word that has started (so a pause keeps +// the word just spoken lit, and the lead-in before the first word lights none); otherwise the share of the +// sentence's characters spoken so far (playlistWordAt). +function playlistWordIndex(index, timeline, time) { + const span = timeline[index]; + const sentence = state.playlistSentences[index]; + if (span && span.wordTimes) { + const wt = span.wordTimes; + if (time < wt[0][0] - 0.02) return -1; + let lo = 0, hi = wt.length - 1, found = 0; + while (lo <= hi) { + const mid = (lo + hi) >> 1; + if (wt[mid][0] <= time + 0.02) { found = mid; lo = mid + 1; } else hi = mid - 1; + } + return found; + } + return playlistWordAt(sentence && sentence.words, playlistLocal(span, time)); +} + function playlistWordAt(words, local) { if (!words || !words.length) return -1; const lens = words.map((w) => String(w).length + 1); @@ -5253,7 +5282,7 @@ function magPlaylistSync() { const index = playlistIndexAt(timeline, time); const span = timeline[index]; const local = playlistLocal(span, time); - const wordIdx = playlistWordAt(state.playlistSentences[index]?.words, local); + const wordIdx = playlistWordIndex(index, timeline, time); const follow = $('mag-playlist-karaoke')?.checked !== false; document.querySelectorAll('#mag-playlist-transcript [data-playlist-sentence]').forEach((el) => { el.classList.toggle('is-current', follow && Number(el.dataset.playlistSentence) === index); @@ -5284,7 +5313,7 @@ function magPlaylistFrame() { const time = audio.currentTime; const timeline = playlistTimeline(playlistDuration(a)); const index = playlistIndexAt(timeline, time); - const wordIdx = playlistWordAt(state.playlistSentences[index]?.words, playlistLocal(timeline[index], time)); + const wordIdx = playlistWordIndex(index, timeline, time); if (index !== state.syncIndex || wordIdx !== state.syncWord || Math.abs(time - (state.syncTime || 0)) >= 0.25) magPlaylistSync(); } @@ -5302,6 +5331,26 @@ function magSyncLoopStart() { state.syncRaf = requestAnimationFrame(tick); } +// The server measures word times in the background the first time an article is played; pick them up when +// they are ready without interrupting playback (the timeline cache key changes, so it rebuilds on the next frame). +async function magPollWordTimes(a, token, attempt) { + if (attempt > 40) return; + await new Promise((r) => setTimeout(r, attempt < 3 ? 4000 : 15000)); + if (token !== state.playlistLoadToken || state.playlistCurrent !== a.id) return; + const t = await api('/api/magazine/' + state.doc.id + '/articles/' + a.id + '/timings').catch(() => null); + if (token !== state.playlistLoadToken || !t || !t.available) return; + if (t.words === 'ready' || t.words === 'partial') { + const exact = state.playlistExact; + if (exact && exact.length === t.sentences.length) { + t.sentences.forEach((u, i) => { exact[i].words = magCleanWordTimes(u.words); }); + state.playlistTimelineKey = null; + state.wordTimesLoaded = a.id; + } + } + const st = t.words_progress && t.words_progress.state; + if (t.words !== 'ready' && (t.words === 'running' || st === 'running' || t.words === 'partial' && st !== 'done')) magPollWordTimes(a, token, attempt + 1); +} + async function loadPlaylistCompanion(a) { const token = ++state.playlistLoadToken; state.playlistSentences = []; @@ -5320,12 +5369,14 @@ async function loadPlaylistCompanion(a) { if (token !== state.playlistLoadToken || state.playlistCurrent !== a.id) return; state.playlistSentences = splitSentences(data.blocks || []); state.playlistExact = timings && timings.available && Array.isArray(timings.sentences) - ? timings.sentences.map((u) => ({ start: Number(u.start), end: Number(u.end), speech: u.speech == null ? NaN : Number(u.speech) })) + ? timings.sentences.map((u) => ({ start: Number(u.start), end: Number(u.end), speech: u.speech == null ? NaN : Number(u.speech), + words: magCleanWordTimes(u.words) })) .filter((u) => Number.isFinite(u.start) && Number.isFinite(u.end)) : null; state.playlistTimelineKey = null; state.playlistImages = articleImages(a); renderPlaylistCompanion(); + if (timings && timings.available && (timings.words === 'running' || timings.words === 'partial')) magPollWordTimes(a, token, 0); } catch (_) { if (token === state.playlistLoadToken) renderPlaylistCompanion(); } @@ -5890,6 +5941,7 @@ function magDrivingSync(audio, duration, pct, sentenceIndex, local, timeline) { const time = $('mag-driving-time'); if (time) time.textContent = magFmtClock(audio.currentTime) + ' / ' + magFmtClock(duration); const sentences = state.playlistSentences || []; + const wi = timeline ? playlistWordIndex(sentenceIndex, timeline, audio.currentTime) : playlistWordAt(sentences[sentenceIndex]?.words, local || 0); const lyrics = $('mag-driving-lyrics'); if (lyrics) { lyrics.hidden = !prefs.text || !sentences.length; @@ -5932,7 +5984,6 @@ function magDrivingSync(audio, duration, pct, sentenceIndex, local, timeline) { const cur = lyrics.querySelector('.mag-driving-line.is-current'); lyrics.querySelectorAll('.mag-dw.is-current').forEach((w) => w.classList.remove('is-current')); const words = cur ? cur.querySelectorAll('.mag-dw') : []; - const wi = playlistWordAt(sentences[sentenceIndex]?.words, local || 0); if (words[wi]) words[wi].classList.add('is-current'); } } @@ -5952,7 +6003,7 @@ function magDrivingSync(audio, duration, pct, sentenceIndex, local, timeline) { const noimg = $('mag-driving-noimg'); if (noimg) noimg.hidden = !!show || !prefs.images; } - magDrivingSyncFull(sentences, sentenceIndex); + magDrivingSyncFull(sentences, sentenceIndex, wi); } // ── Views, whole-text "lyrics", picture zoom ────────────────────────────────────────────────────── @@ -6012,7 +6063,7 @@ function magDrivingBackToCurrent() { magDrivingScrollToCurrent(true); } -function magDrivingSyncFull(sentences, index) { +function magDrivingSyncFull(sentences, index, wi) { const box = $('mag-driving-full'); const prefs = magDrivingPrefs(); if (!box || prefs.view !== 'lyrics' || !prefs.text) return; @@ -6020,7 +6071,8 @@ function magDrivingSyncFull(sentences, index) { if (_drvFull.key !== key) { _drvFull.key = key; _drvFull.index = -1; _drvFull.follow = true; box.innerHTML = sentences.map((sent, i) => - '

' + esc(sent.text || '') + '

').join(''); + '

' + + (sent.words || [sent.text || '']).map((w) => '' + esc(w) + '').join(' ') + '

').join(''); box.scrollTop = 0; } if (!_drvFull.wired) { @@ -6050,10 +6102,21 @@ function magDrivingSyncFull(sentences, index) { const cur = lines[index]; if (cur) { cur.classList.add('is-current'); cur.setAttribute('aria-current', 'true'); if (!box.contains(document.activeElement) || document.activeElement === box) cur.tabIndex = 0; } _drvFull.index = index; + _drvFull.word = -1; if (_drvFull.follow) { _drvFull.programmatic = Date.now(); magDrivingScrollToCurrent(true); } } else if (_drvFull.follow && !_drvFull.settled) { _drvFull.settled = true; magDrivingScrollToCurrent(false); } + // Per-word highlight inside the current line (touches the DOM only when the word changes). + if (_drvFull.word !== wi) { + const line = box.children[index]; + if (line) { + line.querySelectorAll('.mag-fw.is-current').forEach((w) => w.classList.remove('is-current')); + const w = wi >= 0 ? line.querySelectorAll('.mag-fw')[wi] : null; + if (w) w.classList.add('is-current'); + } + _drvFull.word = wi; + } } // Picture zoom: buttons, wheel, keys, double tap/click, pinch and drag-to-pan. State lives in _drvZoom. diff --git a/static/style.css b/static/style.css index 3db3d62..7c370e2 100644 --- a/static/style.css +++ b/static/style.css @@ -5482,6 +5482,7 @@ input[list="lang-codes"] { } @keyframes mag-line-in { from { opacity: 0; transform: translateY(10px); } to { opacity: 1; transform: none; } } @keyframes mag-line-fade { from { opacity: 0; } to { opacity: .6; } } +.mag-fw.is-current { background: var(--accent); color: var(--on-accent, #fff); border-radius: 4px; padding: 0 3px; } .mag-dw.is-current { background: var(--accent); color: var(--on-accent, #fff); border-radius: 4px; padding: 0 3px; } .mag-driving-lyrics { position: relative; } diff --git a/tests/magazine.spec.js b/tests/magazine.spec.js index 0f502cd..9be88f0 100644 --- a/tests/magazine.spec.js +++ b/tests/magazine.spec.js @@ -3154,3 +3154,105 @@ test.describe('Driving mode layout and views', () => { } }); }); + + +// ── Word-level highlight from measured word times (forced aligner), with fallback to interpolation ────────── +test.describe('Driving mode: measured word times drive the word highlight', () => { + const W_START = [0.7, 1.2, 1.6, 2.3], W_END = [1.1, 1.5, 2.2, 2.9]; // seconds into each 3 s sentence + const EFF = [0, 3.55, 6.55, 9.55]; + const N_WORDS = [4, 4, 3, 4]; // words in 'Alpha one is here.' ... 'Delta four ends it.' + const expectedWord = (t) => { + const i = EFF.reduce((n, e, k) => (e <= t ? k : n), 0), rel = t - 3 * i; + if (rel < W_START[0] - 0.02) return [i, -1]; + return [i, W_START.slice(0, N_WORDS[i]).reduce((n, s, k) => (s <= rel + 0.02 ? k : n), 0)]; + }; + + async function openWords(page, { withWords = true, speed } = {}) { + await mockBackend(page, { articleStatus: 'approved', hasAudio: true }); + const blocks = [{ id: 'b1', page: 3, bbox: [0, 0, 1, 1], type: 'body', + text: 'Alpha one is here. Bravo two is next. Charlie three follows. Delta four ends it.', reading_order: 0 }]; + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/text*`, route => route.fulfill({ json: { article: ARTICLE_APPROVED, blocks } })); + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/timings`, route => route.fulfill({ json: { available: true, quality: 'exact', + words: withWords ? 'ready' : 'none', + sentences: [0, 3, 6, 9].map((s, i) => ({ text: 'S' + i, block: 0, start: s, end: s + 3, speech: s + 0.7, + ...(withWords ? { words: W_START.slice(0, N_WORDS[i]).map((w, k) => [s + w, s + W_END[k]]) } : {}) })) } })); + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/audio`, route => { + const wav = silentWav(12000), range = /bytes=(\d+)-(\d*)/.exec(route.request().headers().range || ''); + if (!range) return route.fulfill({ body: wav, contentType: 'audio/wav', headers: { 'Accept-Ranges': 'bytes' } }); + const from = Number(range[1]), to = range[2] ? Number(range[2]) : wav.length - 1; + return route.fulfill({ status: 206, body: wav.subarray(from, to + 1), contentType: 'audio/wav', + headers: { 'Accept-Ranges': 'bytes', 'Content-Range': `bytes ${from}-${to}/${wav.length}` } }); + }); + await page.goto('/', { waitUntil: 'domcontentloaded' }); + await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 }); + await navTo(page, 's-magazine'); + await page.locator('#mag-library-list .mag-card').first().click(); + await page.getByRole('button', { name: 'Playlist' }).click(); + await page.locator('.mag-playlist-track-play').click(); + await expect(page.locator('#mag-playlist-transcript [data-playlist-sentence]')).toHaveCount(4); + await page.evaluate(() => window.magOpenDriving()); + await expect(page.locator('#mag-driving')).toBeVisible(); + await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').duration)).toBeGreaterThan(11); + if (speed) await page.evaluate((v) => { const a = document.getElementById('mag-playlist-audio'); a.defaultPlaybackRate = v; a.playbackRate = v; }, speed); + } + const lit = (page) => page.evaluate(() => { + const idx = (sel, wsel, attr) => { + const line = document.querySelector(sel); if (!line) return null; + const ws = [...line.querySelectorAll(wsel)]; return [Number(line.dataset[attr]), ws.findIndex((w) => w.classList.contains('is-current'))]; + }; + return { t: document.getElementById('mag-playlist-audio').currentTime, + compact: idx('#mag-driving-lyrics .mag-driving-line.is-current', '.mag-dw', 'i'), + lyrics: idx('#mag-driving-full .mag-driving-fline.is-current', '.mag-fw', 'i'), + list: idx('#mag-playlist-transcript .is-current', '.mag-w', 'playlistSentence') }; + }); + const seekPaused = (page, t) => page.evaluate((x) => { const a = document.getElementById('mag-playlist-audio'); a.pause(); a.currentTime = x; }, t); + + test('word follows the measured times at several positions and seeks, in compact, lyrics and playlist views', async ({ page }) => { + await openWords(page); + await page.evaluate(() => window.magDrivingSetView('compact')); + for (const t of [0.3, 0.9, 1.35, 2.5, 3.8, 4.3, 5.0, 3.0, 8.4, 11.2, 0.8, 7.0]) { + await seekPaused(page, t); + const want = expectedWord(t); + await expect.poll(async () => (await lit(page)).compact, { message: 'compact at ' + t }).toEqual(want); + await expect.poll(async () => (await lit(page)).list, { message: 'playlist at ' + t }).toEqual(want); + } + await page.evaluate(() => window.magDrivingSetView('lyrics')); + for (const t of [1.0, 2.0, 4.0, 9.0, 6.0]) { + await seekPaused(page, t); + await expect.poll(async () => (await lit(page)).lyrics, { message: 'lyrics at ' + t }).toEqual(expectedWord(t)); + } + }); + + test('measured words differ from the length estimate: the long-pause word stays lit until the next starts', async ({ page }) => { + await openWords(page); + await seekPaused(page, 1.52 + 0.0); // word 1 ended at 1.5; next starts at 1.6: word 1 stays lit + await expect.poll(async () => (await lit(page)).list).toEqual([0, 1]); + await seekPaused(page, 1.7); + await expect.poll(async () => (await lit(page)).list).toEqual([0, 2]); + }); + + for (const speed of [1.5, 2]) { + test(`playing at ${speed}x the lit word matches the measured table`, async ({ page }) => { + await openWords(page, { speed }); + await page.evaluate(() => { const a = document.getElementById('mag-playlist-audio'); a.currentTime = 0.4; a.play(); }); + const wall0 = Date.now(); let checked = 0; + while (Date.now() - wall0 < 5000) { + const s = await lit(page); + const near = [...EFF, ...W_START.flatMap((w) => [0, 3, 6, 9].map((b) => b + w))].some((e) => Math.abs(s.t - e) < 0.07 * speed + 0.04); + if (!near && s.compact) { expect(s.compact, 'at ' + s.t).toEqual(expectedWord(s.t)); checked++; } + if (s.t > 11.5) break; + await page.waitForTimeout(50); + } + expect(checked).toBeGreaterThan(4); + }); + } + + test('without measured words the highlight falls back to the interpolation and still moves', async ({ page }) => { + await openWords(page, { withWords: false }); + await page.evaluate(() => window.magDrivingSetView('compact')); + await seekPaused(page, 1.0); + await expect.poll(async () => (await lit(page)).compact?.[1]).toBeGreaterThanOrEqual(0); + await seekPaused(page, 2.9); + await expect.poll(async () => (await lit(page)).compact?.[1]).toBeGreaterThanOrEqual(2); + }); +}); diff --git a/tests/test_word_alignment.py b/tests/test_word_alignment.py new file mode 100644 index 0000000..d803f74 --- /dev/null +++ b/tests/test_word_alignment.py @@ -0,0 +1,228 @@ +"""Word-timing job: planning, sidecar client, v2 record format and graceful fallback (sidecar mocked).""" +from __future__ import annotations + +import json +import os +import shutil +import tempfile +import time +import unittest +from pathlib import Path +from unittest import mock + +_CONFIG = tempfile.mkdtemp(prefix="ttvc-wordalign-test-") +os.environ.setdefault("CONFIG_DIR", _CONFIG) + +import requests + +from core import epub_export, word_alignment as wa + + +def units_for(record, blocks): + return epub_export.plan_sentences(blocks, record["duration_ms"], record, None) + + +BLOCKS = [{"id": "b1", "text": "Erster Satz hier. Zweiter Satz."}, {"id": "b2", "text": "Dritter Satz."}] +RECORD = {"version": 1, "interview": False, "duration_ms": 6000, "segments": [ + {"chars": len(BLOCKS[0]["text"]), "times": [[0, 2000], [2000, 3000]]}, + {"chars": len(BLOCKS[1]["text"]), "times": [[3000, 6000]]}]} + + +def fake_post_factory(calls): + def post(url, wav, spans, language, **_): + calls.append((url, spans, language)) + # one word per token, evenly spread over the span + out = [] + for sp in spans: + toks = sp["text"].split() + step = (sp["end"] - sp["start"]) / len(toks) + out.append([{"word": t, "start": round(sp["start"] + i * step, 3), + "end": round(sp["start"] + (i + 1) * step, 3), "aligned": True} for i, t in enumerate(toks)]) + return out + return post + + +class PlanningTests(unittest.TestCase): + def test_exact_blocks_get_one_job_per_sentence(self): + units, q = units_for(RECORD, BLOCKS) + self.assertEqual(q, "exact") + jobs = wa.alignment_jobs(units, BLOCKS, False) + self.assertEqual([(j["start"], j["end"], j["text"]) for j in jobs], + [(0, 2000, "Erster Satz hier."), (2000, 3000, "Zweiter Satz."), (3000, 6000, "Dritter Satz.")]) + + def test_anchored_block_is_aligned_as_one_span(self): + rec = json.loads(json.dumps(RECORD)) + rec["segments"][0]["times"] = [[0, 3000]] # one utterance for two sentences + units, q = units_for(rec, BLOCKS) + self.assertEqual(q, "anchored") + jobs = wa.alignment_jobs(units, BLOCKS, False) + self.assertEqual(jobs[0]["text"], "Erster Satz hier. Zweiter Satz.") + self.assertEqual(jobs[0]["units"], [0, 1]) + + def test_speaker_label_is_not_sent_and_gets_a_zero_length_slot(self): + blocks = [{"id": "b1", "text": "Frage Eins: Wie geht es? Gut."}] + rec = {"version": 1, "interview": True, "duration_ms": 4000, "segments": [ + {"chars": len("Wie geht es? Gut."), "times": [[0, 2000], [2000, 4000]]}]} + units, q = epub_export.plan_sentences(blocks, 4000, rec, [len("Wie geht es? Gut.")]) + self.assertEqual(q, "exact") + jobs = wa.alignment_jobs(units, blocks, True) + self.assertEqual(jobs[0]["text"], "Wie geht es?") + self.assertEqual(jobs[0]["lead"], 2) + + +class JobTests(unittest.TestCase): + def setUp(self): + self.d = Path(tempfile.mkdtemp(prefix="ttvc-wa-")) + self.addCleanup(shutil.rmtree, self.d, True) + from pydub import AudioSegment + from pydub.generators import Sine + self.mp3 = self.d / "a1.mp3" + Sine(300).to_audio_segment(duration=6000).export(self.mp3, format="mp3") + self.rec_path = self.d / "a1.timings.json" + self.rec_path.write_text(json.dumps(RECORD)) + self.units, _ = units_for(RECORD, BLOCKS) + self.jobs = wa.alignment_jobs(self.units, BLOCKS, False) + + def test_words_are_written_as_version_2_and_old_fields_survive(self): + calls = [] + got = wa.align_words(self.mp3, self.units, self.jobs, "de", url="http://x", post=fake_post_factory(calls)) + seg_words = wa.build_segment_words(self.units, got, 2) + self.assertTrue(wa.write_words(self.mp3, seg_words)) + rec = json.loads(self.rec_path.read_text()) + self.assertEqual(rec["version"], 2) + self.assertEqual(rec["segments"][0]["times"], RECORD["segments"][0]["times"]) # v1 data untouched + self.assertEqual(len(rec["segments"][0]["words"]), 5) # 3 + 2 tokens + self.assertEqual(len(rec["segments"][1]["words"]), 2) + first, last = rec["segments"][0]["words"][0], rec["segments"][0]["words"][-1] + self.assertAlmostEqual(first[0], 0, delta=150) + self.assertAlmostEqual(last[1], 3000, delta=150) + # the v1 reader still accepts it + from core import magazine as mag + self.assertIsNotNone(mag.read_audio_timings(self.mp3)) + self.assertEqual(calls[0][2], "de") + + def test_chunks_are_bounded_and_progress_reported(self): + many = [{"units": [0], "start": i * 10000, "end": i * 10000 + 9000, "text": "a", "lead": 0} for i in range(7)] + self.assertTrue(all(len(c) <= 4 for c in wa._chunks(many))) # 45 s per request + seen = [] + wa.align_words(self.mp3, self.units, self.jobs, "de", url="http://x", + post=fake_post_factory([]), progress=lambda a, b: seen.append((a, b))) + self.assertEqual(seen[-1], (3, 3)) + + def test_incomplete_segment_gets_no_words(self): + got = {0: [[0, 1]] * 3} # sentence 1 missing for segment 0 + self.assertEqual(wa.build_segment_words(self.units, got, 2), {}) + + +class ClientFallbackTests(unittest.TestCase): + def test_retries_with_backoff_then_raises_when_sidecar_is_down(self): + sleeps = [] + with mock.patch.object(wa.requests, "post", side_effect=requests.ConnectionError("down")) as p: + with self.assertRaises(wa.AlignerError): + wa._post_chunk("http://x", b"w", [{"start": 0, "end": 1, "text": "a"}], "de", sleep=sleeps.append) + self.assertEqual(p.call_count, wa.ATTEMPTS) + self.assertEqual(sleeps, [1, 2]) + + def test_client_error_is_not_retried(self): + resp = mock.Mock(status_code=422) + with mock.patch.object(wa.requests, "post", return_value=resp) as p: + with self.assertRaises(wa.AlignerError): + wa._post_chunk("http://x", b"w", [], "de", sleep=lambda s: None) + self.assertEqual(p.call_count, 1) + + def test_busy_then_ok(self): + ok = mock.Mock(status_code=200); ok.json.return_value = {"spans": [[]]} + with mock.patch.object(wa.requests, "post", side_effect=[mock.Mock(status_code=503), ok]): + self.assertEqual(wa._post_chunk("http://x", b"w", [], "de", sleep=lambda s: None), [[]]) + + def test_failed_job_is_recorded_and_not_restarted_within_cooldown(self): + def run(progress): + raise wa.AlignerError("aligner unreachable") + with mock.patch.dict(os.environ, {"ALIGNER_URL": "http://x"}): + self.assertTrue(wa.should_autostart("docF", "a1")) + self.assertTrue(wa.start("docF", "a1", run)) + for _ in range(100): + if wa.status("docF", "a1")["state"] != "running": + break + time.sleep(0.02) + st = wa.status("docF", "a1") + self.assertEqual(st["state"], "failed") + self.assertFalse(wa.should_autostart("docF", "a1")) + + def test_disabled_without_url(self): + with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}): + self.assertFalse(wa.should_autostart("docX", "a1")) + + +class RouteTests(unittest.TestCase): + """GET /timings carries words when present and never waits for the job.""" + + def setUp(self): + from fastapi import FastAPI + from fastapi.testclient import TestClient + import routes.magazine as routes + self.routes = routes + self.lib = Path(tempfile.mkdtemp(prefix="ttvc-lib-")) + self.addCleanup(shutil.rmtree, self.lib, True) + p = mock.patch.object(routes, "_LIB", self.lib); p.start(); self.addCleanup(p.stop) + app = FastAPI(); app.include_router(routes.router) + self.client = TestClient(app) + self.doc = self.client.post("/api/magazine/docs", json={"title": "T"}).json()["id"] + d = self.d = self.lib / self.doc + blocks = [{"id": "b1", "page": 1, "bbox": [0, 0, 10, 10], "type": "body", "text": BLOCKS[0]["text"], "reading_order": 1}, + {"id": "b2", "page": 1, "bbox": [0, 40, 10, 10], "type": "body", "text": BLOCKS[1]["text"], "reading_order": 2}] + (d / "layout.json").write_text(json.dumps({"pages": [{"page": 1, "width": 600, "height": 800}], "blocks": blocks, "images": []})) + (d / "articles.json").write_text(json.dumps({"articles": [ + {"id": "a1", "title": "T", "pages": [1], "blocks": ["b1", "b2"], "images": [], "status": "approved"}]})) + (d / "audio").mkdir(exist_ok=True) + (d / "audio" / "a1.mp3").write_bytes(b"x") + self.rec = d / "audio" / "a1.timings.json" + self.rec.write_text(json.dumps(dict(RECORD, interview=True))) + self.url = f"/api/magazine/{self.doc}/articles/a1/timings" + + def test_without_sidecar_the_answer_is_unchanged_and_instant(self): + with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}): + body = self.client.get(self.url).json() + self.assertTrue(body["available"]) + self.assertEqual(body["words"], "none") + self.assertNotIn("words", body["sentences"][0]) + + def test_words_are_returned_per_sentence(self): + rec = json.loads(self.rec.read_text()); rec["version"] = 2 + rec["segments"][0]["words"] = [[0, 400], [500, 900], [1000, 1900], [2100, 2500], [2600, 2900]] + rec["segments"][1]["words"] = [[3100, 3500], [3600, 5900]] + self.rec.write_text(json.dumps(rec)) + body = self.client.get(self.url).json() + self.assertEqual(body["words"], "ready") + self.assertEqual(body["sentences"][0]["words"], [[0.0, 0.4], [0.5, 0.9], [1.0, 1.9]]) + self.assertEqual(body["sentences"][1]["words"], [[2.1, 2.5], [2.6, 2.9]]) + self.assertEqual(body["sentences"][2]["words"][1], [3.6, 5.9]) + + def test_mismatching_words_are_ignored(self): + rec = json.loads(self.rec.read_text()); rec["segments"][0]["words"] = [[0, 1]] + self.rec.write_text(json.dumps(rec)) + with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}): + body = self.client.get(self.url).json() + self.assertNotIn("words", body["sentences"][0]) + + def test_first_request_starts_the_job_in_background(self): + started = [] + with mock.patch.dict(os.environ, {"ALIGNER_URL": "http://x"}), \ + mock.patch.object(self.routes, "_start_word_alignment", lambda *a: started.append(a) or True): + body = self.client.get(self.url).json() + self.assertEqual(body["words"], "running") + self.assertEqual(len(started), 1) + self.assertTrue(body["available"]) + + def test_explicit_endpoint_requires_configuration_and_timings(self): + url = f"/api/magazine/{self.doc}/articles/a1/align-words" + with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}): + self.assertEqual(self.client.post(url).status_code, 503) + with mock.patch.dict(os.environ, {"ALIGNER_URL": "http://x"}): + self.rec.unlink() + self.assertEqual(self.client.post(url).status_code, 404) + self.assertIn("state", self.client.get(url).json()) + + +if __name__ == "__main__": + unittest.main()