feat(magazine): word-level timing via forced-aligner sidecar with player word highlight
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
788f186b9c
commit
9e5ea609ce
@ -11,6 +11,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
||||
- **Share an article or a magazine as a link that opens straight in driving mode.** A *Share* button in the magazine toolbar (whole magazine), one in the playlist header and one in driving mode (current article) build `/#s-magazine?doc=<id>&article=<id>&mode=drive`. They use the browser's share sheet where there is one (it needs https, so on a plain-http LAN address they copy the link instead, with a polite aria-live toast, or show a box with the link if even copying is refused). Opening such a link, also on a first visit and also pasted into an already-open tab, opens the magazine and then driving mode for that article with audio, pictures and text; when the browser blocks autoplay a large *Tap to play* button appears. The link carries ids only (no key or password) and does not change who may reach the app; the ids are validated against a strict pattern and never put into markup, and anything malformed is ignored. Tested with Playwright (desktop deep link, blocked autoplay, hostile parameters, share sheet at 375 px, copy fallback).
|
||||
- **Filter the page grid by status.** Each legend entry is now a toggle button with the number of pages in that status; choose one or several to show only pages that have any of them. It combines with *Show only pages that need me*, the summary says how many pages are shown, *Show all pages* clears everything, a status no page has is greyed out, and the buttons announce their on/off state and work from the keyboard.
|
||||
|
||||
### Added
|
||||
- **Word-level timing with a forced-aligner sidecar.** A new, separate `tts-aligner` container (`aligner/`, torchaudio MMS_FA, CPU by default, optional `--profile gpu`; weights cached in the `aligner-models` volume, no published port, non-root, read-only filesystem, 3 GB limit) answers `POST /align` and `POST /align/batch` with `[{word, start, end}]`. The app (image unchanged, no torch) measures each sentence's words in a background job the first time an article is played (or via `POST .../articles/{id}/align-words`, progress via `GET`), writes `words: [[start_ms, end_ms], ...]` per segment into `audio/<id>.timings.json` (now version 2; readers of version 1 keep working) and `/timings` returns `words` per sentence. The player marks the word from those times in Compact, Lyrics (new per-word marking) and the playlist, and falls back to the previous length-based estimate when no word times exist or the sidecar is down. Measured on one article: after-pause word starts are within 29 ms on average (max 170 ms) of the audio's energy onsets.
|
||||
|
||||
### Fixed
|
||||
- **Sentence timings now also work for interview renders.** Every recorded audio file carries `"interview": true` (set whenever a second voice is configured, even without speaker labels), and the timings endpoint and EPUB export discarded all such records, so the player and the EPUB fell back to estimated sentence positions for every current article. The record is now matched to the text again, for interview renders by the label-stripped length the renderer recorded, so `/timings` returns exact (or anchored) sentence times and speech onsets for these articles too. Records for text edited since the render are still ignored.
|
||||
|
||||
|
||||
@ -291,7 +291,8 @@ turned into a podcast discussion between two or three voices you pick.
|
||||
**Hands-free:** while a track plays, the lock screen, notification shade, Bluetooth headset and car display show the article title, magazine and first-page artwork and offer play/pause, previous/next article and ±15 s (and seeking, where the device offers it). In driving mode the screen is kept awake while audio plays and allowed to sleep when it is paused or you leave. Both are optional browser features; where one is missing or refused, playback itself is unaffected. Opening driving mode resumes the first unfinished article at the position you left it.
|
||||
**Layout and views:** driving mode is a top bar, a content area and a controls dock. The dock (position slider, time and playback speed, then previous, back 15 s, play/pause, forward 15 s, next) is always completely visible at the bottom and is never covered or pushed off screen, on phones from 320x568, tablets and in landscape. The top bar has *Share*, a *menu* button that shows the current view, *Index* and *Exit*. The menu opens a flyout: choose **Picture** (the picture fills the area), **Lyrics** (the whole text) or **Compact** (picture plus three lines, the default; in landscape side by side), and switch *Show text*, *Show pictures* and *Minimize text and picture* on or off. Keyboard: Enter or Arrow Down opens the menu, arrows/Home/End move, Enter chooses, Escape or a tap outside closes it and returns focus to the button. **Index:** the *Index* button opens a list of all articles of the magazine with their length (articles without audio are dimmed and marked *no audio*). The article that is playing is highlighted; tap it or press Enter on another to jump there while driving mode keeps playing. Escape or the close button closes the list (Tab stays inside it while open); the controls below stay visible and usable by touch. Lengths come from the already generated audio, nothing is recomputed. **Speed** is the button after the time next to the slider. **Picture zoom:** double-tap or double-click, pinch, mouse wheel, the + and - keys, or the zoom in / zoom out / reset buttons on the picture; drag (or arrow keys) to move a zoomed picture; Escape or the reset button returns to full view. **Lyrics:** the spoken sentence is large and kept in the middle, earlier and later ones are dimmed; tap or press Enter on a sentence to play from it; scrolling by hand stops the automatic following and shows *Back to current*. With reduced motion switched on in your device the text jumps instead of gliding. **Minimize** (in the menu) reduces the area to the title and the current line so the dock keeps minimal text; choose it again to expand. Your view and the minimized state are remembered in this browser only. Titles wrap to two lines (with an ellipsis beyond that), picture captions to two lines.
|
||||
**Three lines of text:** driving mode shows the previous sentence (small), the current one (large, the spoken word marked) and the next (small) in a box of fixed height, so nothing moves around it. Tap or press Enter on the previous or next line to jump to that sentence. The text is deliberately not announced by screen readers sentence by sentence; the play/pause button and the position slider stay the way to control it. With reduced motion switched on in your device, lines change without animation. On a phone (375 px wide) the sentences wrap and a very long current sentence scrolls inside its box. Turn the text off with *Show text* in the menu to see only the picture.
|
||||
**Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. The highlight lights up a sentence just before its first word is heard (each sentence starts with a short silence, which is not counted). Older audio, interview-style audio and audio whose text was edited afterwards has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync.
|
||||
**Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. The highlight lights up a sentence just before its first word is heard (each sentence starts with a short silence, which is not counted). Audio from before sentence times were recorded, and audio whose text was edited afterwards, has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync. Interview-voice audio is covered like any other.
|
||||
**Word by word:** the first time an article is played, the app asks the forced-aligner service (the separate `tts-aligner` container, see below) to measure where each word starts and ends and stores that beside the audio. Playback never waits for it: until it is ready (usually well under a minute for a magazine article) the marked word is estimated from word length as before, and it switches to the measured word on its own. The marked word is the last one that has started, so it stays lit through a pause; the *Lyrics* view now marks the spoken word too (as do *Compact* and the playlist text). If the service is not running, nothing changes for you and the app tries again later (at most every 10 minutes per article). Prerequisites: `docker compose up -d tts-aligner` (first start downloads about 1.2 GB of model weights into the `aligner-models` volume; after that it needs no internet). `ALIGNER_URL` empty turns word timing off. Privacy: text and audio go only to your own container on the internal Docker network. Accessibility: the marking is visual only and adds no announcements; with reduced motion nothing animates. API: `POST /api/magazine/{doc}/articles/{id}/align-words` starts it explicitly, `GET` shows progress.
|
||||
**Share a link:** the *Share* button in the toolbar (the whole magazine), the share button in the playlist header and the one at the top left of driving mode (the current article) share a link of the form `<this app's address>/#s-magazine?doc=<magazine>&article=<article>&mode=drive`. Where the browser has a share sheet (phones, and any browser on https) it opens; otherwise the link is copied and a confirmation is announced (on a plain-http address a box with the link is shown to copy by hand). Opening the link goes straight to that article in driving mode with audio, pictures and text; if the browser refuses to start audio without a tap, a large *Tap to play* button appears. A link for the whole magazine opens at its first unfinished article. **Privacy and access:** the link contains only ids, no password or key. It is just an address: whoever opens it needs to reach this app the same way as for any other page (this app has no user accounts; access is controlled by your network or reverse proxy), and an unknown or removed article simply shows a message. Only articles that already have audio can be shared. A link whose parameters are not in the expected form is ignored.
|
||||
|
||||
**Expected result:** the article plays from its headline to its final paragraph.
|
||||
|
||||
@ -77,6 +77,7 @@ which is the writable volume.
|
||||
| `VOICES_DIR` / `OUTPUT_DIR` | voice WAV library |
|
||||
| `TTS_CONTAINER_NAMES` | containers restarted after a voice change |
|
||||
| `DOCKER_HOST` | points at the socket proxy; the raw socket is not mounted |
|
||||
| `ALIGNER_URL` | forced-aligner sidecar for word timing (`http://tts-aligner:8000`; empty = off) |
|
||||
| `llm_url` / `llm_model` / `llm_api_key` | settings.json, OpenAI-compatible LLM |
|
||||
|
||||
## Pending decisions
|
||||
|
||||
3
aligner/.dockerignore
Normal file
3
aligner/.dockerignore
Normal file
@ -0,0 +1,3 @@
|
||||
tests
|
||||
__pycache__
|
||||
*.pyc
|
||||
19
aligner/Dockerfile
Normal file
19
aligner/Dockerfile
Normal file
@ -0,0 +1,19 @@
|
||||
# Forced-aligner sidecar: word-level timing with torchaudio MMS_FA. Not part of the app image.
|
||||
# CPU (default): docker compose up -d --build tts-aligner
|
||||
# GPU (optional): docker compose --profile gpu up -d --build tts-aligner-gpu
|
||||
FROM python:3.12-slim-bookworm
|
||||
ARG TORCH_INDEX=https://download.pytorch.org/whl/cpu
|
||||
ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1 TORCH_HOME=/models/torch HF_HOME=/models/hf
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends libsndfile1 curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
COPY requirements.txt /tmp/requirements.txt
|
||||
RUN pip install --index-url "$TORCH_INDEX" torch==2.10.0 torchaudio==2.10.0 \
|
||||
&& pip install -r /tmp/requirements.txt
|
||||
RUN useradd --uid 10001 --no-create-home --shell /usr/sbin/nologin aligner && mkdir -p /models && chown aligner /models
|
||||
WORKDIR /srv
|
||||
COPY app.py .
|
||||
USER aligner
|
||||
EXPOSE 8000
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8000/health || exit 1
|
||||
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1", "--no-access-log"]
|
||||
230
aligner/app.py
Normal file
230
aligner/app.py
Normal file
@ -0,0 +1,230 @@
|
||||
"""Forced aligner sidecar: where does each word of a known text start and end in its audio?
|
||||
|
||||
POST /align multipart: audio (wav/mp3/flac file), text, language, [start, end] seconds
|
||||
POST /align/batch multipart: audio, spans (JSON [{start,end,text}]), language -- many sentences, one upload
|
||||
GET /health
|
||||
|
||||
Times are seconds in the uploaded audio. Words are the whitespace-separated tokens of the text, in order; a
|
||||
token with no letters (digits, punctuation) cannot be aligned and gets the gap between its neighbours
|
||||
(aligned=false). Model: torchaudio MMS_FA (weights are fetched once into TORCH_HOME, a volume, never baked in).
|
||||
No secrets, no auth: the service is reachable only on the internal Docker network.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
import unicodedata
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
|
||||
|
||||
SAMPLE_RATE = 16000
|
||||
MAX_AUDIO_BYTES = int(os.environ.get("ALIGNER_MAX_AUDIO_BYTES", 40 * 1024 * 1024))
|
||||
MAX_SPAN_SECONDS = float(os.environ.get("ALIGNER_MAX_SPAN_SECONDS", 120))
|
||||
MAX_AUDIO_SECONDS = float(os.environ.get("ALIGNER_MAX_AUDIO_SECONDS", 900))
|
||||
MAX_SPANS = int(os.environ.get("ALIGNER_MAX_SPANS", 400))
|
||||
MAX_TEXT_CHARS = int(os.environ.get("ALIGNER_MAX_TEXT_CHARS", 20000))
|
||||
REQUEST_TIMEOUT = float(os.environ.get("ALIGNER_REQUEST_TIMEOUT", 110))
|
||||
QUEUE_WAIT = float(os.environ.get("ALIGNER_QUEUE_WAIT", 30))
|
||||
DEVICE = os.environ.get("ALIGNER_DEVICE", "cpu")
|
||||
_LANG_RE = re.compile(r"^[A-Za-z]{2,3}(-[A-Za-z0-9]{2,8})?$")
|
||||
|
||||
app = FastAPI(title="aligner", docs_url=None, redoc_url=None, openapi_url=None)
|
||||
_gate = asyncio.Semaphore(1) # one alignment at a time: the model is the scarce resource
|
||||
_state: dict[str, Any] = {"bundle": None, "model": None, "tok": None, "aligner": None, "error": None}
|
||||
_load_lock = threading.Lock()
|
||||
|
||||
|
||||
# ── text ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
def normalize_word(word: str) -> str:
|
||||
"""Romanize one token to the a-z' alphabet of the MMS_FA model (umlauts and accents folded)."""
|
||||
w = unicodedata.normalize("NFKD", word.lower().replace("ß", "ss").replace("æ", "ae").replace("œ", "oe"))
|
||||
w = "".join(c for c in w if not unicodedata.combining(c))
|
||||
return re.sub(r"[^a-z']", "", w)
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
return text.split()
|
||||
|
||||
|
||||
# ── model ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _load() -> None:
|
||||
with _load_lock:
|
||||
if _state["model"] is not None:
|
||||
return
|
||||
import torch # imported lazily: /health works before the model is loaded
|
||||
import torchaudio
|
||||
torch.set_num_threads(int(os.environ.get("ALIGNER_THREADS", "4")))
|
||||
bundle = torchaudio.pipelines.MMS_FA
|
||||
model = bundle.get_model(with_star=False).to(DEVICE).eval()
|
||||
_state.update(bundle=bundle, model=model, tok=bundle.get_tokenizer(), aligner=bundle.get_aligner(), error=None)
|
||||
|
||||
|
||||
def align_samples(samples: np.ndarray, words: list[str]) -> list[tuple[float, float, bool]]:
|
||||
"""Per whitespace token (start, end, aligned) in seconds relative to `samples` (16 kHz mono float32)."""
|
||||
import torch
|
||||
norm = [normalize_word(w) for w in words]
|
||||
keep = [i for i, n in enumerate(norm) if n]
|
||||
out: list[list[Any]] = [[None, None, False] for _ in words]
|
||||
if keep and len(samples) >= 400:
|
||||
wav = torch.from_numpy(samples)[None].to(DEVICE)
|
||||
with torch.inference_mode():
|
||||
emission, _ = _state["model"](wav)
|
||||
spans = _state["aligner"](emission[0], _state["tok"]([norm[i] for i in keep]))
|
||||
frame = len(samples) / emission.shape[1] / SAMPLE_RATE
|
||||
for i, sp in zip(keep, spans):
|
||||
out[i] = [sp[0].start * frame, sp[-1].end * frame, True]
|
||||
# tokens without letters: the gap between their neighbours
|
||||
total = len(samples) / SAMPLE_RATE
|
||||
for i, o in enumerate(out):
|
||||
if o[2]:
|
||||
continue
|
||||
prev_end = next((out[j][1] for j in range(i - 1, -1, -1) if out[j][2]), 0.0)
|
||||
next_start = next((out[j][0] for j in range(i + 1, len(out)) if out[j][2]), total)
|
||||
o[0], o[1] = prev_end, max(prev_end, next_start)
|
||||
return [(float(a), float(b), bool(c)) for a, b, c in out]
|
||||
|
||||
|
||||
# ── audio ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def decode(data: bytes) -> np.ndarray:
|
||||
"""Any libsndfile format (wav, flac, mp3) to 16 kHz mono float32."""
|
||||
import soundfile as sf
|
||||
try:
|
||||
with sf.SoundFile(io.BytesIO(data)) as f:
|
||||
if f.frames / max(1, f.samplerate) > MAX_AUDIO_SECONDS:
|
||||
raise HTTPException(413, f"audio longer than {MAX_AUDIO_SECONDS:.0f} s")
|
||||
x = f.read(dtype="float32", always_2d=True)
|
||||
sr = f.samplerate
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception:
|
||||
raise HTTPException(415, "audio is not a readable wav/mp3/flac file")
|
||||
x = x.mean(axis=1)
|
||||
if sr != SAMPLE_RATE:
|
||||
import torch
|
||||
import torchaudio.functional as F
|
||||
x = F.resample(torch.from_numpy(x), sr, SAMPLE_RATE).numpy()
|
||||
return np.ascontiguousarray(x, dtype=np.float32)
|
||||
|
||||
|
||||
def _check_language(language: str) -> None:
|
||||
if not _LANG_RE.match(language or ""):
|
||||
raise HTTPException(422, "language must be an ISO code such as 'de' or 'en'")
|
||||
|
||||
|
||||
def _check_text(text: str, strict: bool = True) -> list[str]:
|
||||
if not isinstance(text, str) or len(text) > MAX_TEXT_CHARS:
|
||||
raise HTTPException(422, f"text must be a string of at most {MAX_TEXT_CHARS} characters")
|
||||
words = tokenize(text)
|
||||
if not words:
|
||||
raise HTTPException(422, "text is empty")
|
||||
if strict and not any(normalize_word(w) for w in words):
|
||||
raise HTTPException(422, "text has no letters the model can align (latin alphabets only)")
|
||||
return words
|
||||
|
||||
|
||||
def _slice(x: np.ndarray, start: float, end: float) -> tuple[np.ndarray, float]:
|
||||
total = len(x) / SAMPLE_RATE
|
||||
if not (0 <= start < end <= total + 0.05):
|
||||
raise HTTPException(422, f"span {start}-{end} s is outside the audio (0-{total:.2f} s)")
|
||||
if end - start > MAX_SPAN_SECONDS:
|
||||
raise HTTPException(422, f"span longer than {MAX_SPAN_SECONDS:.0f} s")
|
||||
return x[int(start * SAMPLE_RATE):int(min(end, total) * SAMPLE_RATE)], start
|
||||
|
||||
|
||||
def _run(x: np.ndarray, spans: list[tuple[float, float, str]], deadline: float) -> list[list[dict[str, Any]]]:
|
||||
_load()
|
||||
result = []
|
||||
for start, end, text in spans:
|
||||
if time.monotonic() > deadline:
|
||||
raise HTTPException(504, "alignment took longer than the request limit")
|
||||
words = tokenize(text)
|
||||
piece, offset = _slice(x, start, end)
|
||||
times = align_samples(piece, words)
|
||||
result.append([{"word": w, "start": round(offset + a, 3), "end": round(offset + b, 3), "aligned": ok}
|
||||
for w, (a, b, ok) in zip(words, times)])
|
||||
return result
|
||||
|
||||
|
||||
async def _guarded(x: np.ndarray, spans: list[tuple[float, float, str]]) -> list[list[dict[str, Any]]]:
|
||||
try:
|
||||
await asyncio.wait_for(_gate.acquire(), QUEUE_WAIT)
|
||||
except asyncio.TimeoutError:
|
||||
raise HTTPException(503, "aligner busy, retry shortly", headers={"Retry-After": "10"})
|
||||
try:
|
||||
deadline = time.monotonic() + REQUEST_TIMEOUT
|
||||
return await asyncio.wait_for(asyncio.to_thread(_run, x, spans, deadline), REQUEST_TIMEOUT + 5)
|
||||
except asyncio.TimeoutError:
|
||||
raise HTTPException(504, "alignment took longer than the request limit")
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc: # model download/load failure and the like
|
||||
_state["error"] = type(exc).__name__
|
||||
raise HTTPException(503, f"aligner model unavailable ({type(exc).__name__})")
|
||||
finally:
|
||||
_gate.release()
|
||||
|
||||
|
||||
async def _read_audio(audio: UploadFile) -> np.ndarray:
|
||||
data = await audio.read(MAX_AUDIO_BYTES + 1)
|
||||
if len(data) > MAX_AUDIO_BYTES:
|
||||
raise HTTPException(413, "audio file too large")
|
||||
return await asyncio.to_thread(decode, data)
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def _preload() -> None:
|
||||
"""Load (and on first run download) the model in the background so the first request is not slow."""
|
||||
def work() -> None:
|
||||
try:
|
||||
_load()
|
||||
except Exception as exc: # reported by /health, retried on the first request
|
||||
_state["error"] = type(exc).__name__
|
||||
threading.Thread(target=work, name="preload", daemon=True).start()
|
||||
|
||||
|
||||
# ── routes ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok", "model_loaded": _state["model"] is not None, "device": DEVICE,
|
||||
"last_error": _state["error"]}
|
||||
|
||||
|
||||
@app.post("/align")
|
||||
async def align(audio: UploadFile = File(...), text: str = Form(...), language: str = Form("de"),
|
||||
start: float | None = Form(None), end: float | None = Form(None)):
|
||||
_check_language(language)
|
||||
_check_text(text)
|
||||
x = await _read_audio(audio)
|
||||
s = 0.0 if start is None else start
|
||||
e = len(x) / SAMPLE_RATE if end is None else end
|
||||
words = (await _guarded(x, [(s, e, text)]))[0]
|
||||
return {"language": language, "words": words}
|
||||
|
||||
|
||||
@app.post("/align/batch")
|
||||
async def align_batch(audio: UploadFile = File(...), spans: str = Form(...), language: str = Form("de")):
|
||||
_check_language(language)
|
||||
try:
|
||||
raw = json.loads(spans)
|
||||
parsed = [(float(r["start"]), float(r["end"]), str(r["text"])) for r in raw]
|
||||
except (ValueError, KeyError, TypeError):
|
||||
raise HTTPException(422, "spans must be a JSON list of {start, end, text}")
|
||||
if not parsed or len(parsed) > MAX_SPANS:
|
||||
raise HTTPException(422, f"between 1 and {MAX_SPANS} spans are allowed")
|
||||
for _, _, t in parsed:
|
||||
_check_text(t, strict=False) # a sentence of digits only just gets its gap back
|
||||
if sum(len(t) for _, _, t in parsed) > MAX_TEXT_CHARS * 4:
|
||||
raise HTTPException(422, "too much text")
|
||||
x = await _read_audio(audio)
|
||||
return {"language": language, "spans": await _guarded(x, parsed)}
|
||||
6
aligner/requirements.txt
Normal file
6
aligner/requirements.txt
Normal file
@ -0,0 +1,6 @@
|
||||
# torch / torchaudio are installed separately in the Dockerfile (CPU or CUDA wheel index)
|
||||
fastapi==0.115.6
|
||||
uvicorn==0.32.1
|
||||
python-multipart==0.0.20
|
||||
soundfile==0.13.1
|
||||
numpy==2.2.6
|
||||
84
aligner/tests/test_app.py
Normal file
84
aligner/tests/test_app.py
Normal file
@ -0,0 +1,84 @@
|
||||
"""Validation and text-handling tests for the aligner; they never load the model."""
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import unittest
|
||||
import wave
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
import app as aligner # noqa: E402
|
||||
|
||||
|
||||
def wav_bytes(seconds=1.0, sr=16000):
|
||||
buf = io.BytesIO()
|
||||
with wave.open(buf, "wb") as w:
|
||||
w.setnchannels(1); w.setsampwidth(2); w.setframerate(sr)
|
||||
w.writeframes((np.zeros(int(seconds * sr))).astype("<i2").tobytes())
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
class NormalizeTests(unittest.TestCase):
|
||||
def test_romanizes_german(self):
|
||||
self.assertEqual(aligner.normalize_word("Größe,"), "grosse")
|
||||
self.assertEqual(aligner.normalize_word("Über"), "uber")
|
||||
self.assertEqual(aligner.normalize_word("2024."), "")
|
||||
self.assertEqual(aligner.normalize_word("don't"), "don't")
|
||||
|
||||
def test_digits_get_the_gap_between_neighbours(self):
|
||||
fake = aligner.align_samples(np.zeros(16000, dtype=np.float32), ["2024", "3"])
|
||||
self.assertEqual([(a, ok) for a, _, ok in fake], [(0.0, False), (0.0, False)])
|
||||
self.assertAlmostEqual(fake[0][1], 1.0)
|
||||
|
||||
|
||||
class ValidationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.c = TestClient(aligner.app)
|
||||
|
||||
def post(self, url="/align", audio=None, **data):
|
||||
files = {"audio": ("a.wav", audio if audio is not None else wav_bytes(), "audio/wav")}
|
||||
return self.c.post(url, files=files, data=data)
|
||||
|
||||
def test_health_works_without_the_model(self):
|
||||
r = self.c.get("/health")
|
||||
self.assertEqual(r.status_code, 200)
|
||||
self.assertFalse(r.json()["model_loaded"])
|
||||
|
||||
def test_bad_language_text_and_audio_are_rejected(self):
|
||||
self.assertEqual(self.post(text="Hallo Welt", language="../x").status_code, 422)
|
||||
self.assertEqual(self.post(text=" ", language="de").status_code, 422)
|
||||
self.assertEqual(self.post(text="1234 5678", language="de").status_code, 422)
|
||||
self.assertEqual(self.post(text="x" * (aligner.MAX_TEXT_CHARS + 1), language="de").status_code, 422)
|
||||
self.assertEqual(self.post(audio=b"not audio", text="Hallo", language="de").status_code, 415)
|
||||
|
||||
def test_span_must_lie_inside_the_audio(self):
|
||||
self.assertEqual(self.post(text="Hallo", language="de", start="5", end="6").status_code, 422)
|
||||
self.assertEqual(self.post(text="Hallo", language="de", start="0.5", end="0.2").status_code, 422)
|
||||
|
||||
def test_oversized_audio_is_rejected(self):
|
||||
old = aligner.MAX_AUDIO_BYTES
|
||||
aligner.MAX_AUDIO_BYTES = 100
|
||||
try:
|
||||
self.assertEqual(self.post(text="Hallo", language="de").status_code, 413)
|
||||
finally:
|
||||
aligner.MAX_AUDIO_BYTES = old
|
||||
|
||||
def test_batch_validates_spans(self):
|
||||
self.assertEqual(self.post("/align/batch", spans="nonsense", language="de").status_code, 422)
|
||||
self.assertEqual(self.post("/align/batch", spans="[]", language="de").status_code, 422)
|
||||
self.assertEqual(self.post("/align/batch", spans=json.dumps([{"start": 0, "end": 1}]), language="de").status_code, 422)
|
||||
many = json.dumps([{"start": 0, "end": 1, "text": "a"}] * (aligner.MAX_SPANS + 1))
|
||||
self.assertEqual(self.post("/align/batch", spans=many, language="de").status_code, 422)
|
||||
|
||||
def test_letterless_sentence_in_a_batch_needs_no_model(self):
|
||||
# no letters: handled by gap interpolation, but the model still loads first; only check validation passes
|
||||
r = self.post("/align/batch", spans=json.dumps([{"start": 0, "end": 1, "text": "2024"}]), language="de")
|
||||
self.assertNotEqual(r.status_code, 422)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@ -272,7 +272,27 @@ without returning to the contents screen. `magRead(id, {fromStart: true})`
|
||||
is what makes this a fresh start rather than resuming a queued article's own
|
||||
saved progress, which would otherwise interrupt a continuous listen.
|
||||
|
||||
### Word-level highlighting is estimated, not measured
|
||||
### Word-level timing: forced-aligner sidecar
|
||||
|
||||
`aligner/` is a separate image/service (`tts-aligner` in docker-compose.yml): FastAPI + torchaudio `MMS_FA`, CPU wheels
|
||||
by default (`--profile gpu` builds `tts-aligner-gpu` with the cu130 wheel; both answer to the name `tts-aligner`), weights
|
||||
cached in the `aligner-models` volume (downloaded once from dl.fbaipublicfiles.com, never in git), non-root, read-only root,
|
||||
no published port (only `dgx_net`), 3 GB / 4 CPU limits, healthcheck on `GET /health`. `POST /align` (audio, text, language,
|
||||
optional start/end) and `POST /align/batch` (audio + `spans` JSON `[{start,end,text}]`) return `[{word,start,end,aligned}]`
|
||||
per whitespace token, in seconds of the uploaded audio; tokens without letters get the gap between their neighbours
|
||||
(`aligned:false`). Limits: 40 MB upload, 120 s per span, 400 spans, 20 000 chars, 110 s per request, one request at a time
|
||||
(others wait 30 s, then 503), 422/413/415 for invalid input.
|
||||
|
||||
App side (`core/word_alignment.py`, env `ALIGNER_URL`, empty = off): `plan_sentences` gives each sentence its recorded span;
|
||||
`alignment_jobs` makes one job per recorded utterance (exact blocks) or per block (anchored blocks, split afterwards by token
|
||||
count) and drops an interview `Speaker:` label from the text (those tokens get zero-length slots). Jobs are sent in chunks of
|
||||
at most 45 s / 25 spans as 16 kHz mono WAV slices (3 attempts, backoff 1 s/2 s, 3 s connect / 150 s read timeouts). Results are
|
||||
written per record segment as `words: [[start_ms, end_ms], ...]` (record `version` 2, `words_engine` `mms_fa`; segments only when
|
||||
every sentence got words); `read_audio_timings` accepts versions 1 and 2. `GET .../timings` adds `words` (`ready|partial|none|running`),
|
||||
`words_progress` and per-sentence `words` (seconds), and lazily starts the background job (never awaited; cooldown 10 min after
|
||||
any run). The player prefers `wordTimes` (`playlistWordIndex`: last word that has started) over `playlistWordAt`.
|
||||
|
||||
### Word-level highlighting without measured times is estimated
|
||||
|
||||
No forced-alignment or TTS timestamp source exists in this pipeline.
|
||||
`estimateWordTimings` distributes a sentence's *real, measured* audio
|
||||
|
||||
@ -5848,7 +5848,7 @@ def read_audio_timings(mp3_path: Path) -> dict[str, Any] | None:
|
||||
data = json.loads(mp3_path.with_suffix(".timings.json").read_text("utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return data if isinstance(data, dict) and data.get("version") == 1 and isinstance(data.get("segments"), list) else None
|
||||
return data if isinstance(data, dict) and data.get("version") in (1, 2) and isinstance(data.get("segments"), list) else None
|
||||
|
||||
|
||||
def tag_article_mp3(path: Path, article: dict[str, Any], lyrics: str, *,
|
||||
|
||||
233
core/word_alignment.py
Normal file
233
core/word_alignment.py
Normal file
@ -0,0 +1,233 @@
|
||||
"""Word-level timing of a narrated article, measured by the forced-aligner sidecar (aligner/).
|
||||
|
||||
The MP3 is rendered sentence by sentence, so `audio/<id>.timings.json` already says where every sentence sits.
|
||||
This module asks the sidecar where each *word* of a sentence starts and ends inside that span and records the
|
||||
answer as `words: [[start_ms, end_ms], ...]` per segment (the file becomes version 2; version 1 readers simply
|
||||
ignore the key). Never blocks playback: it runs in a background thread, retries with backoff, and when the
|
||||
sidecar is down it gives up quietly -- the player then interpolates words inside each sentence as before.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Sequence
|
||||
|
||||
import requests
|
||||
|
||||
from core import epub_export
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
CHUNK_SECONDS = 45.0 # audio per sidecar request: bounded time per call and real progress
|
||||
CHUNK_MAX_SPANS = 25
|
||||
ATTEMPTS = 3
|
||||
CONNECT_TIMEOUT = 3.0
|
||||
READ_TIMEOUT = 150.0
|
||||
FAILURE_COOLDOWN = 600.0 # after a failed run, do not retry on every timings request
|
||||
WORDS_VERSION = 2
|
||||
|
||||
_jobs: dict[tuple[str, str], dict[str, Any]] = {}
|
||||
_lock = threading.Lock()
|
||||
|
||||
|
||||
def aligner_url() -> str:
|
||||
return os.environ.get("ALIGNER_URL", "").strip().rstrip("/")
|
||||
|
||||
|
||||
# ── planning ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def alignment_jobs(units: Sequence[dict[str, Any]], blocks: Sequence[dict[str, Any]],
|
||||
interview: bool) -> list[dict[str, Any]]:
|
||||
"""Group sentence units into alignment jobs: [{units: [idx...], start, end, text, lead}] (ms).
|
||||
|
||||
A block whose sentences all have their own recorded utterance gets one job per sentence (tightest span).
|
||||
A block whose spans were only anchored (counts disagreed) is aligned as one span and split afterwards.
|
||||
`lead` counts leading tokens that were never spoken (an interview "Speaker:" label) and get no time.
|
||||
"""
|
||||
from core.magazine import _INTERVIEW_LABEL_RE
|
||||
by_block: dict[int, list[int]] = {}
|
||||
for idx, u in enumerate(units):
|
||||
by_block.setdefault(u["block"], []).append(idx)
|
||||
jobs: list[dict[str, Any]] = []
|
||||
for block, idxs in by_block.items():
|
||||
lead = 0
|
||||
if interview:
|
||||
m = _INTERVIEW_LABEL_RE.match(units[idxs[0]]["text"])
|
||||
if m:
|
||||
lead = len(units[idxs[0]]["text"].split()) - len(m.group(2).split())
|
||||
exact = all(units[i].get("nth") is not None for i in idxs)
|
||||
groups = [[i] for i in idxs] if exact else [idxs]
|
||||
for g in groups:
|
||||
text = " ".join(units[i]["text"] for i in g)
|
||||
first_lead = lead if g[0] == idxs[0] else 0
|
||||
spoken = " ".join(text.split()[first_lead:])
|
||||
if not spoken:
|
||||
continue
|
||||
jobs.append({"units": g, "start": units[g[0]]["start"], "end": units[g[-1]]["end"],
|
||||
"text": spoken, "lead": first_lead})
|
||||
return jobs
|
||||
|
||||
|
||||
def _chunks(jobs: Sequence[dict[str, Any]]) -> list[list[dict[str, Any]]]:
|
||||
out: list[list[dict[str, Any]]] = []
|
||||
cur: list[dict[str, Any]] = []
|
||||
for j in jobs:
|
||||
if cur and (len(cur) >= CHUNK_MAX_SPANS or (j["end"] - cur[0]["start"]) / 1000 > CHUNK_SECONDS):
|
||||
out.append(cur)
|
||||
cur = []
|
||||
cur.append(j)
|
||||
if cur:
|
||||
out.append(cur)
|
||||
return out
|
||||
|
||||
|
||||
# ── sidecar client ─────────────────────────────────────────────────────────────
|
||||
|
||||
class AlignerError(RuntimeError):
|
||||
"""The sidecar could not be used (down, busy too long, rejected the request)."""
|
||||
|
||||
|
||||
def _post_chunk(url: str, wav: bytes, spans: list[dict[str, Any]], language: str,
|
||||
sleep: Callable[[float], None] = time.sleep) -> list[list[dict[str, Any]]]:
|
||||
last: Exception | None = None
|
||||
for attempt in range(ATTEMPTS):
|
||||
try:
|
||||
r = requests.post(f"{url}/align/batch", files={"audio": ("chunk.wav", wav, "audio/wav")},
|
||||
data={"spans": json.dumps(spans), "language": language},
|
||||
timeout=(CONNECT_TIMEOUT, READ_TIMEOUT))
|
||||
if r.status_code == 200:
|
||||
return r.json()["spans"]
|
||||
if 400 <= r.status_code < 500 and r.status_code != 429:
|
||||
raise AlignerError(f"aligner rejected the request ({r.status_code})")
|
||||
last = AlignerError(f"aligner answered {r.status_code}")
|
||||
except requests.RequestException as exc:
|
||||
last = AlignerError(f"aligner unreachable ({type(exc).__name__})")
|
||||
except (ValueError, KeyError) as exc:
|
||||
last = AlignerError(f"aligner sent an unreadable answer ({type(exc).__name__})")
|
||||
if attempt < ATTEMPTS - 1:
|
||||
sleep(2 ** attempt)
|
||||
raise last or AlignerError("aligner failed")
|
||||
|
||||
|
||||
def _wav_slice(audio: Any, start_ms: int, end_ms: int) -> bytes:
|
||||
buf = io.BytesIO()
|
||||
audio[start_ms:end_ms].set_frame_rate(16000).set_channels(1).export(buf, format="wav")
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
# ── the job ────────────────────────────────────────────────────────────────────
|
||||
|
||||
def align_words(mp3: Path, units: Sequence[dict[str, Any]], jobs: Sequence[dict[str, Any]], language: str, *,
|
||||
url: str, progress: Callable[[int, int], None] | None = None,
|
||||
post: Callable[..., Any] = _post_chunk) -> dict[int, list[list[int]]]:
|
||||
"""Ask the sidecar for the words of every job; returns {unit index: [[start_ms, end_ms], ...]} for the
|
||||
units it could time (a unit's list has one pair per whitespace token, label tokens first at zero length)."""
|
||||
from pydub import AudioSegment
|
||||
audio = AudioSegment.from_file(mp3)
|
||||
chunks = _chunks(jobs)
|
||||
result: dict[int, list[list[int]]] = {}
|
||||
done = 0
|
||||
for chunk in chunks:
|
||||
lo = max(0, int(chunk[0]["start"]) - 100)
|
||||
hi = min(len(audio), int(chunk[-1]["end"]) + 100)
|
||||
spans = [{"start": round((j["start"] - lo) / 1000, 3), "end": round(min(j["end"], hi) / 1000 - lo / 1000, 3),
|
||||
"text": j["text"]} for j in chunk]
|
||||
answer = post(url, _wav_slice(audio, lo, hi), spans, language)
|
||||
for j, words in zip(chunk, answer):
|
||||
pairs = [[int(round(lo + w["start"] * 1000)), int(round(lo + w["end"] * 1000))] for w in words]
|
||||
pairs = [[j["start"], j["start"]]] * j["lead"] + pairs
|
||||
at = 0
|
||||
for i in j["units"]:
|
||||
n = len(units[i]["text"].split())
|
||||
result[i] = pairs[at:at + n]
|
||||
at += n
|
||||
done += 1
|
||||
if progress:
|
||||
progress(done, len(jobs))
|
||||
return result
|
||||
|
||||
|
||||
def build_segment_words(units: Sequence[dict[str, Any]], per_unit: dict[int, list[list[int]]],
|
||||
n_segments: int) -> dict[int, list[list[int]]]:
|
||||
"""Per record segment, the words of all its sentences in order -- only for segments fully covered."""
|
||||
out: dict[int, list[list[int]]] = {}
|
||||
bad: set[int] = set()
|
||||
for idx, u in enumerate(units):
|
||||
k = u["seg"]
|
||||
words = per_unit.get(idx)
|
||||
if words is None or len(words) != len(u["text"].split()):
|
||||
bad.add(k)
|
||||
continue
|
||||
out.setdefault(k, []).extend(words)
|
||||
return {k: v for k, v in out.items() if k not in bad and 0 <= k < n_segments}
|
||||
|
||||
|
||||
def write_words(mp3: Path, segment_words: dict[int, list[list[int]]]) -> bool:
|
||||
"""Add `words` to the record (atomic replace; everything else in the file is kept as it was)."""
|
||||
path = mp3.with_suffix(".timings.json")
|
||||
try:
|
||||
rec = json.loads(path.read_text("utf-8"))
|
||||
segs = rec["segments"]
|
||||
for k, words in segment_words.items():
|
||||
segs[k]["words"] = words
|
||||
rec["version"] = WORDS_VERSION
|
||||
rec["words_engine"] = "mms_fa"
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(rec), "utf-8")
|
||||
tmp.replace(path)
|
||||
return True
|
||||
except (OSError, ValueError, KeyError, IndexError):
|
||||
log.warning("Could not write word timings for %s", mp3.name)
|
||||
return False
|
||||
|
||||
|
||||
# ── background job registry ────────────────────────────────────────────────────
|
||||
|
||||
def status(doc_id: str, article_id: str) -> dict[str, Any]:
|
||||
with _lock:
|
||||
return dict(_jobs.get((doc_id, article_id)) or {"state": "idle", "done": 0, "total": 0})
|
||||
|
||||
|
||||
def should_autostart(doc_id: str, article_id: str) -> bool:
|
||||
if not aligner_url():
|
||||
return False
|
||||
st = status(doc_id, article_id)
|
||||
if st["state"] == "running":
|
||||
return False
|
||||
# Whatever the last outcome, do not re-run on every request: a run that finished without covering every
|
||||
# sentence (or after the audio was re-rendered) is retried after the cooldown.
|
||||
return time.time() - st.get("finished", 0) >= FAILURE_COOLDOWN
|
||||
|
||||
|
||||
def start(doc_id: str, article_id: str, run: Callable[[Callable[[int, int], None]], dict[str, Any]]) -> bool:
|
||||
"""Run `run(progress)` in a daemon thread unless this article is already being aligned."""
|
||||
key = (doc_id, article_id)
|
||||
with _lock:
|
||||
if (_jobs.get(key) or {}).get("state") == "running":
|
||||
return False
|
||||
_jobs[key] = {"state": "running", "done": 0, "total": 0, "started": time.time()}
|
||||
|
||||
def progress(done: int, total: int) -> None:
|
||||
with _lock:
|
||||
_jobs[key].update(done=done, total=total)
|
||||
|
||||
def target() -> None:
|
||||
try:
|
||||
info = run(progress)
|
||||
end = {"state": "done", **info}
|
||||
except AlignerError as exc:
|
||||
log.warning("Word alignment for %s skipped: %s", article_id, exc)
|
||||
end = {"state": "failed", "error": str(exc)}
|
||||
except Exception as exc: # never let a worker thread die noisily
|
||||
log.exception("Word alignment for %s failed", article_id)
|
||||
end = {"state": "failed", "error": type(exc).__name__}
|
||||
with _lock:
|
||||
_jobs[key].update(end, finished=time.time())
|
||||
|
||||
threading.Thread(target=target, name=f"align-{article_id}", daemon=True).start()
|
||||
return True
|
||||
@ -83,6 +83,8 @@ services:
|
||||
- TTS_CONTAINER_NAMES=tts-faster-qwen3-tts-voiceclone,tts-faster-qwen3-tts-streaming
|
||||
- VOICE_DESIGN_MODEL=Qwen3-TTS-12Hz-1.7B-VoiceDesign
|
||||
- DOCKER_HOST=tcp://docker-socket-proxy:2375
|
||||
# Word-level forced aligner sidecar; empty disables word timing (sentence timing keeps working)
|
||||
- ALIGNER_URL=${ALIGNER_URL-http://tts-aligner:8000}
|
||||
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
@ -90,6 +92,86 @@ services:
|
||||
networks:
|
||||
- dgx_net
|
||||
|
||||
# Word-level forced aligner (torchaudio MMS_FA). Separate image so the app stays torch-free. Reachable only on
|
||||
# the Compose network (no published port); the app calls http://tts-aligner:8000. Weights (~1.2 GB) are
|
||||
# downloaded once into the named volume; after that the service needs no outbound access.
|
||||
tts-aligner:
|
||||
build:
|
||||
context: ./aligner
|
||||
image: tts-aligner:local
|
||||
container_name: tts-aligner
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
cap_drop:
|
||||
- ALL
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
tmpfs:
|
||||
- /tmp:size=256m,mode=1777
|
||||
mem_limit: "${ALIGNER_MEM_LIMIT:-3g}"
|
||||
cpus: "${ALIGNER_CPUS:-4}"
|
||||
environment:
|
||||
- ALIGNER_DEVICE=cpu
|
||||
- ALIGNER_THREADS=4
|
||||
expose:
|
||||
- "8000"
|
||||
volumes:
|
||||
- aligner-models:/models:rw
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
- dgx_net
|
||||
|
||||
# Optional GPU variant: `docker compose --profile gpu up -d --build tts-aligner-gpu` (stop tts-aligner first;
|
||||
# both answer to the name tts-aligner). Off by default because the DGX GPU is often full.
|
||||
tts-aligner-gpu:
|
||||
profiles: ["gpu"]
|
||||
build:
|
||||
context: ./aligner
|
||||
args:
|
||||
TORCH_INDEX: https://download.pytorch.org/whl/cu130
|
||||
image: tts-aligner:gpu
|
||||
container_name: tts-aligner-gpu
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
cap_drop:
|
||||
- ALL
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
tmpfs:
|
||||
- /tmp:size=256m,mode=1777
|
||||
mem_limit: "${ALIGNER_MEM_LIMIT:-3g}"
|
||||
environment:
|
||||
- ALIGNER_DEVICE=cuda
|
||||
expose:
|
||||
- "8000"
|
||||
volumes:
|
||||
- aligner-models:/models:rw
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
dgx_net:
|
||||
aliases:
|
||||
- tts-aligner
|
||||
|
||||
networks:
|
||||
dgx_net:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
aligner-models:
|
||||
|
||||
@ -35,6 +35,7 @@ from fastapi import APIRouter, File, HTTPException, Request, UploadFile
|
||||
from fastapi.responses import FileResponse, Response
|
||||
|
||||
from core import magazine as mag
|
||||
from core import word_alignment
|
||||
from core.config import _load_settings
|
||||
from core.constants import CONFIG_DIR
|
||||
from core.database import (
|
||||
@ -754,11 +755,86 @@ async def magazine_article_timings(doc_id: str, article_id: str):
|
||||
k = 0
|
||||
seen.add(u["block"])
|
||||
flat.append(on[k] if k is not None and k < len(on) else 0)
|
||||
return {"available": True, "quality": quality, "sentences": [
|
||||
sentences = [
|
||||
{"text": u["text"], "block": u["block"], "block_id": blocks[u["block"]].get("id"),
|
||||
"start": round(u["start"] / 1000, 3), "end": round(u["end"] / 1000, 3),
|
||||
"speech": round((u["start"] + min(max(0, o), max(0, u["end"] - u["start"] - 10))) / 1000, 3)}
|
||||
for u, o in zip(units, flat)]}
|
||||
for u, o in zip(units, flat)]
|
||||
words_state = _attach_words(sentences, units, segs)
|
||||
if words_state != "ready" and word_alignment.should_autostart(doc_id, article_id):
|
||||
# Lazily, in the background: playback never waits for it; the next request carries the words.
|
||||
_start_word_alignment(d, doc_id, article_id)
|
||||
words_state = "running"
|
||||
return {"available": True, "quality": quality, "words": words_state,
|
||||
"words_progress": word_alignment.status(doc_id, article_id), "sentences": sentences}
|
||||
|
||||
|
||||
def _attach_words(sentences: list[dict[str, Any]], units: list[dict[str, Any]], segs: list[dict[str, Any]]) -> str:
|
||||
"""Add `words: [[start, end], ...]` (seconds, one pair per whitespace token) to every sentence whose
|
||||
segment has recorded word times that fit its text. Returns ready | partial | none."""
|
||||
have = 0
|
||||
cursor: dict[int, int] = {}
|
||||
for s, u in zip(sentences, units):
|
||||
k = u.get("seg")
|
||||
w = segs[k].get("words") if k is not None and 0 <= k < len(segs) else None
|
||||
n = len(u["text"].split())
|
||||
at = cursor.get(k, 0)
|
||||
cursor[k] = at + n
|
||||
if isinstance(w, list) and len(w) >= at + n and all(isinstance(p, list) and len(p) == 2 for p in w[at:at + n]):
|
||||
s["words"] = [[round(a / 1000, 3), round(b / 1000, 3)] for a, b in w[at:at + n]]
|
||||
have += 1
|
||||
if not have:
|
||||
return "none"
|
||||
return "ready" if have == len(sentences) else "partial"
|
||||
|
||||
|
||||
def _start_word_alignment(d: Path, doc_id: str, article_id: str) -> bool:
|
||||
url = word_alignment.aligner_url()
|
||||
if not url:
|
||||
return False
|
||||
|
||||
def run(progress):
|
||||
from core import epub_export as ee
|
||||
layout = mag.load_layout(d)
|
||||
target = next((a for a in mag.load_articles(d) if a.get("id") == article_id), None)
|
||||
mp3 = d / "audio" / f"{article_id}.mp3"
|
||||
timings = mag.read_audio_timings(mp3) if mp3.is_file() else None
|
||||
if target is None or not timings:
|
||||
return {"words": 0}
|
||||
blocks = mag.article_text(layout, target, fuse=True, glossary=target.get("glossary"))
|
||||
units, quality = ee.plan_sentences(blocks, float(timings.get("duration_ms") or 0), timings,
|
||||
mag.spoken_chars_for(blocks, timings))
|
||||
if quality == "estimated":
|
||||
return {"words": 0}
|
||||
jobs = word_alignment.alignment_jobs(units, blocks, bool(timings.get("interview")))
|
||||
language = mag.detect_article_language(b["text"] for b in blocks)
|
||||
per_unit = word_alignment.align_words(mp3, units, jobs, language, url=url, progress=progress)
|
||||
seg_words = word_alignment.build_segment_words(units, per_unit, len(timings["segments"]))
|
||||
word_alignment.write_words(mp3, seg_words)
|
||||
return {"words": sum(len(v) for v in seg_words.values())}
|
||||
|
||||
return word_alignment.start(doc_id, article_id, run)
|
||||
|
||||
|
||||
@router.get("/api/magazine/{doc_id}/articles/{article_id}/align-words")
|
||||
async def magazine_align_words_status(doc_id: str, article_id: str):
|
||||
"""Progress of the word-timing job: {state: idle|running|done|failed, done, total, error?, enabled}."""
|
||||
_require_magazine(doc_id)
|
||||
article_id = _article_id(article_id)
|
||||
return {**word_alignment.status(doc_id, article_id), "enabled": bool(word_alignment.aligner_url())}
|
||||
|
||||
|
||||
@router.post("/api/magazine/{doc_id}/articles/{article_id}/align-words", status_code=202)
|
||||
async def magazine_align_words(doc_id: str, article_id: str):
|
||||
"""Measure word times for this article's audio with the aligner sidecar (background job, idempotent)."""
|
||||
d, _ = _require_magazine(doc_id)
|
||||
article_id = _article_id(article_id)
|
||||
if not word_alignment.aligner_url():
|
||||
raise HTTPException(503, "Word alignment is not configured (ALIGNER_URL is empty)")
|
||||
if not (d / "audio" / f"{article_id}.timings.json").is_file():
|
||||
raise HTTPException(404, "This article has no sentence timings yet -- render its audio first")
|
||||
started = _start_word_alignment(d, doc_id, article_id)
|
||||
return {**word_alignment.status(doc_id, article_id), "started": started}
|
||||
|
||||
|
||||
@router.post("/api/magazine/{doc_id}/articles/auto-link")
|
||||
|
||||
@ -5141,8 +5141,11 @@ function playlistTimeline(duration) {
|
||||
if (s.type === 'caption') return { start: cursor, end: cursor, speech: cursor, skip: true };
|
||||
const t = exact[n++];
|
||||
cursor = t.end;
|
||||
const speech = Number.isFinite(t.speech) ? Math.min(t.end, Math.max(t.start, t.speech)) : t.start;
|
||||
return { start: t.start, end: t.end, speech, skip: false };
|
||||
let speech = Number.isFinite(t.speech) ? Math.min(t.end, Math.max(t.start, t.speech)) : t.start;
|
||||
// Measured word times (forced aligner) when they fit this sentence's words; else the length-based estimate.
|
||||
const wt = t.words && t.words.length === s.words.length ? t.words : null;
|
||||
if (wt) speech = Math.min(t.end, Math.max(t.start, wt[0][0]));
|
||||
return { start: t.start, end: t.end, speech, skip: false, wordTimes: wt };
|
||||
});
|
||||
} else {
|
||||
const weights = sentences.map((s) => s.type === 'caption' ? 0 : Math.max(6, s.text.replace(/\s+/g, ' ').length));
|
||||
@ -5191,6 +5194,32 @@ function playlistLocal(span, time) {
|
||||
}
|
||||
|
||||
// Which word is being said, by the share of the sentence's characters spoken so far.
|
||||
// Measured word times for one sentence: [[start, end], ...] in seconds, or null when absent or malformed.
|
||||
function magCleanWordTimes(w) {
|
||||
if (!Array.isArray(w) || !w.length) return null;
|
||||
const out = w.map((p) => (Array.isArray(p) && p.length === 2 ? [Number(p[0]), Number(p[1])] : null));
|
||||
return out.every((p) => p && Number.isFinite(p[0]) && Number.isFinite(p[1])) ? out : null;
|
||||
}
|
||||
|
||||
// The word being said at `time`: with measured word times the last word that has started (so a pause keeps
|
||||
// the word just spoken lit, and the lead-in before the first word lights none); otherwise the share of the
|
||||
// sentence's characters spoken so far (playlistWordAt).
|
||||
function playlistWordIndex(index, timeline, time) {
|
||||
const span = timeline[index];
|
||||
const sentence = state.playlistSentences[index];
|
||||
if (span && span.wordTimes) {
|
||||
const wt = span.wordTimes;
|
||||
if (time < wt[0][0] - 0.02) return -1;
|
||||
let lo = 0, hi = wt.length - 1, found = 0;
|
||||
while (lo <= hi) {
|
||||
const mid = (lo + hi) >> 1;
|
||||
if (wt[mid][0] <= time + 0.02) { found = mid; lo = mid + 1; } else hi = mid - 1;
|
||||
}
|
||||
return found;
|
||||
}
|
||||
return playlistWordAt(sentence && sentence.words, playlistLocal(span, time));
|
||||
}
|
||||
|
||||
function playlistWordAt(words, local) {
|
||||
if (!words || !words.length) return -1;
|
||||
const lens = words.map((w) => String(w).length + 1);
|
||||
@ -5253,7 +5282,7 @@ function magPlaylistSync() {
|
||||
const index = playlistIndexAt(timeline, time);
|
||||
const span = timeline[index];
|
||||
const local = playlistLocal(span, time);
|
||||
const wordIdx = playlistWordAt(state.playlistSentences[index]?.words, local);
|
||||
const wordIdx = playlistWordIndex(index, timeline, time);
|
||||
const follow = $('mag-playlist-karaoke')?.checked !== false;
|
||||
document.querySelectorAll('#mag-playlist-transcript [data-playlist-sentence]').forEach((el) => {
|
||||
el.classList.toggle('is-current', follow && Number(el.dataset.playlistSentence) === index);
|
||||
@ -5284,7 +5313,7 @@ function magPlaylistFrame() {
|
||||
const time = audio.currentTime;
|
||||
const timeline = playlistTimeline(playlistDuration(a));
|
||||
const index = playlistIndexAt(timeline, time);
|
||||
const wordIdx = playlistWordAt(state.playlistSentences[index]?.words, playlistLocal(timeline[index], time));
|
||||
const wordIdx = playlistWordIndex(index, timeline, time);
|
||||
if (index !== state.syncIndex || wordIdx !== state.syncWord || Math.abs(time - (state.syncTime || 0)) >= 0.25) magPlaylistSync();
|
||||
}
|
||||
|
||||
@ -5302,6 +5331,26 @@ function magSyncLoopStart() {
|
||||
state.syncRaf = requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
// The server measures word times in the background the first time an article is played; pick them up when
|
||||
// they are ready without interrupting playback (the timeline cache key changes, so it rebuilds on the next frame).
|
||||
async function magPollWordTimes(a, token, attempt) {
|
||||
if (attempt > 40) return;
|
||||
await new Promise((r) => setTimeout(r, attempt < 3 ? 4000 : 15000));
|
||||
if (token !== state.playlistLoadToken || state.playlistCurrent !== a.id) return;
|
||||
const t = await api('/api/magazine/' + state.doc.id + '/articles/' + a.id + '/timings').catch(() => null);
|
||||
if (token !== state.playlistLoadToken || !t || !t.available) return;
|
||||
if (t.words === 'ready' || t.words === 'partial') {
|
||||
const exact = state.playlistExact;
|
||||
if (exact && exact.length === t.sentences.length) {
|
||||
t.sentences.forEach((u, i) => { exact[i].words = magCleanWordTimes(u.words); });
|
||||
state.playlistTimelineKey = null;
|
||||
state.wordTimesLoaded = a.id;
|
||||
}
|
||||
}
|
||||
const st = t.words_progress && t.words_progress.state;
|
||||
if (t.words !== 'ready' && (t.words === 'running' || st === 'running' || t.words === 'partial' && st !== 'done')) magPollWordTimes(a, token, attempt + 1);
|
||||
}
|
||||
|
||||
async function loadPlaylistCompanion(a) {
|
||||
const token = ++state.playlistLoadToken;
|
||||
state.playlistSentences = [];
|
||||
@ -5320,12 +5369,14 @@ async function loadPlaylistCompanion(a) {
|
||||
if (token !== state.playlistLoadToken || state.playlistCurrent !== a.id) return;
|
||||
state.playlistSentences = splitSentences(data.blocks || []);
|
||||
state.playlistExact = timings && timings.available && Array.isArray(timings.sentences)
|
||||
? timings.sentences.map((u) => ({ start: Number(u.start), end: Number(u.end), speech: u.speech == null ? NaN : Number(u.speech) }))
|
||||
? timings.sentences.map((u) => ({ start: Number(u.start), end: Number(u.end), speech: u.speech == null ? NaN : Number(u.speech),
|
||||
words: magCleanWordTimes(u.words) }))
|
||||
.filter((u) => Number.isFinite(u.start) && Number.isFinite(u.end))
|
||||
: null;
|
||||
state.playlistTimelineKey = null;
|
||||
state.playlistImages = articleImages(a);
|
||||
renderPlaylistCompanion();
|
||||
if (timings && timings.available && (timings.words === 'running' || timings.words === 'partial')) magPollWordTimes(a, token, 0);
|
||||
} catch (_) {
|
||||
if (token === state.playlistLoadToken) renderPlaylistCompanion();
|
||||
}
|
||||
@ -5890,6 +5941,7 @@ function magDrivingSync(audio, duration, pct, sentenceIndex, local, timeline) {
|
||||
const time = $('mag-driving-time');
|
||||
if (time) time.textContent = magFmtClock(audio.currentTime) + ' / ' + magFmtClock(duration);
|
||||
const sentences = state.playlistSentences || [];
|
||||
const wi = timeline ? playlistWordIndex(sentenceIndex, timeline, audio.currentTime) : playlistWordAt(sentences[sentenceIndex]?.words, local || 0);
|
||||
const lyrics = $('mag-driving-lyrics');
|
||||
if (lyrics) {
|
||||
lyrics.hidden = !prefs.text || !sentences.length;
|
||||
@ -5932,7 +5984,6 @@ function magDrivingSync(audio, duration, pct, sentenceIndex, local, timeline) {
|
||||
const cur = lyrics.querySelector('.mag-driving-line.is-current');
|
||||
lyrics.querySelectorAll('.mag-dw.is-current').forEach((w) => w.classList.remove('is-current'));
|
||||
const words = cur ? cur.querySelectorAll('.mag-dw') : [];
|
||||
const wi = playlistWordAt(sentences[sentenceIndex]?.words, local || 0);
|
||||
if (words[wi]) words[wi].classList.add('is-current');
|
||||
}
|
||||
}
|
||||
@ -5952,7 +6003,7 @@ function magDrivingSync(audio, duration, pct, sentenceIndex, local, timeline) {
|
||||
const noimg = $('mag-driving-noimg');
|
||||
if (noimg) noimg.hidden = !!show || !prefs.images;
|
||||
}
|
||||
magDrivingSyncFull(sentences, sentenceIndex);
|
||||
magDrivingSyncFull(sentences, sentenceIndex, wi);
|
||||
}
|
||||
|
||||
// ── Views, whole-text "lyrics", picture zoom ──────────────────────────────────────────────────────
|
||||
@ -6012,7 +6063,7 @@ function magDrivingBackToCurrent() {
|
||||
magDrivingScrollToCurrent(true);
|
||||
}
|
||||
|
||||
function magDrivingSyncFull(sentences, index) {
|
||||
function magDrivingSyncFull(sentences, index, wi) {
|
||||
const box = $('mag-driving-full');
|
||||
const prefs = magDrivingPrefs();
|
||||
if (!box || prefs.view !== 'lyrics' || !prefs.text) return;
|
||||
@ -6020,7 +6071,8 @@ function magDrivingSyncFull(sentences, index) {
|
||||
if (_drvFull.key !== key) {
|
||||
_drvFull.key = key; _drvFull.index = -1; _drvFull.follow = true;
|
||||
box.innerHTML = sentences.map((sent, i) =>
|
||||
'<p class="mag-driving-fline" data-i="' + i + '" tabindex="-1" role="button">' + esc(sent.text || '') + '</p>').join('');
|
||||
'<p class="mag-driving-fline" data-i="' + i + '" tabindex="-1" role="button">' +
|
||||
(sent.words || [sent.text || '']).map((w) => '<span class="mag-fw">' + esc(w) + '</span>').join(' ') + '</p>').join('');
|
||||
box.scrollTop = 0;
|
||||
}
|
||||
if (!_drvFull.wired) {
|
||||
@ -6050,10 +6102,21 @@ function magDrivingSyncFull(sentences, index) {
|
||||
const cur = lines[index];
|
||||
if (cur) { cur.classList.add('is-current'); cur.setAttribute('aria-current', 'true'); if (!box.contains(document.activeElement) || document.activeElement === box) cur.tabIndex = 0; }
|
||||
_drvFull.index = index;
|
||||
_drvFull.word = -1;
|
||||
if (_drvFull.follow) { _drvFull.programmatic = Date.now(); magDrivingScrollToCurrent(true); }
|
||||
} else if (_drvFull.follow && !_drvFull.settled) {
|
||||
_drvFull.settled = true; magDrivingScrollToCurrent(false);
|
||||
}
|
||||
// Per-word highlight inside the current line (touches the DOM only when the word changes).
|
||||
if (_drvFull.word !== wi) {
|
||||
const line = box.children[index];
|
||||
if (line) {
|
||||
line.querySelectorAll('.mag-fw.is-current').forEach((w) => w.classList.remove('is-current'));
|
||||
const w = wi >= 0 ? line.querySelectorAll('.mag-fw')[wi] : null;
|
||||
if (w) w.classList.add('is-current');
|
||||
}
|
||||
_drvFull.word = wi;
|
||||
}
|
||||
}
|
||||
|
||||
// Picture zoom: buttons, wheel, keys, double tap/click, pinch and drag-to-pan. State lives in _drvZoom.
|
||||
|
||||
@ -5482,6 +5482,7 @@ input[list="lang-codes"] {
|
||||
}
|
||||
@keyframes mag-line-in { from { opacity: 0; transform: translateY(10px); } to { opacity: 1; transform: none; } }
|
||||
@keyframes mag-line-fade { from { opacity: 0; } to { opacity: .6; } }
|
||||
.mag-fw.is-current { background: var(--accent); color: var(--on-accent, #fff); border-radius: 4px; padding: 0 3px; }
|
||||
.mag-dw.is-current { background: var(--accent); color: var(--on-accent, #fff); border-radius: 4px; padding: 0 3px; }
|
||||
.mag-driving-lyrics { position: relative; }
|
||||
|
||||
|
||||
@ -3154,3 +3154,105 @@ test.describe('Driving mode layout and views', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
// ── Word-level highlight from measured word times (forced aligner), with fallback to interpolation ──────────
|
||||
test.describe('Driving mode: measured word times drive the word highlight', () => {
|
||||
const W_START = [0.7, 1.2, 1.6, 2.3], W_END = [1.1, 1.5, 2.2, 2.9]; // seconds into each 3 s sentence
|
||||
const EFF = [0, 3.55, 6.55, 9.55];
|
||||
const N_WORDS = [4, 4, 3, 4]; // words in 'Alpha one is here.' ... 'Delta four ends it.'
|
||||
const expectedWord = (t) => {
|
||||
const i = EFF.reduce((n, e, k) => (e <= t ? k : n), 0), rel = t - 3 * i;
|
||||
if (rel < W_START[0] - 0.02) return [i, -1];
|
||||
return [i, W_START.slice(0, N_WORDS[i]).reduce((n, s, k) => (s <= rel + 0.02 ? k : n), 0)];
|
||||
};
|
||||
|
||||
async function openWords(page, { withWords = true, speed } = {}) {
|
||||
await mockBackend(page, { articleStatus: 'approved', hasAudio: true });
|
||||
const blocks = [{ id: 'b1', page: 3, bbox: [0, 0, 1, 1], type: 'body',
|
||||
text: 'Alpha one is here. Bravo two is next. Charlie three follows. Delta four ends it.', reading_order: 0 }];
|
||||
await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/text*`, route => route.fulfill({ json: { article: ARTICLE_APPROVED, blocks } }));
|
||||
await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/timings`, route => route.fulfill({ json: { available: true, quality: 'exact',
|
||||
words: withWords ? 'ready' : 'none',
|
||||
sentences: [0, 3, 6, 9].map((s, i) => ({ text: 'S' + i, block: 0, start: s, end: s + 3, speech: s + 0.7,
|
||||
...(withWords ? { words: W_START.slice(0, N_WORDS[i]).map((w, k) => [s + w, s + W_END[k]]) } : {}) })) } }));
|
||||
await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/audio`, route => {
|
||||
const wav = silentWav(12000), range = /bytes=(\d+)-(\d*)/.exec(route.request().headers().range || '');
|
||||
if (!range) return route.fulfill({ body: wav, contentType: 'audio/wav', headers: { 'Accept-Ranges': 'bytes' } });
|
||||
const from = Number(range[1]), to = range[2] ? Number(range[2]) : wav.length - 1;
|
||||
return route.fulfill({ status: 206, body: wav.subarray(from, to + 1), contentType: 'audio/wav',
|
||||
headers: { 'Accept-Ranges': 'bytes', 'Content-Range': `bytes ${from}-${to}/${wav.length}` } });
|
||||
});
|
||||
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
|
||||
await navTo(page, 's-magazine');
|
||||
await page.locator('#mag-library-list .mag-card').first().click();
|
||||
await page.getByRole('button', { name: 'Playlist' }).click();
|
||||
await page.locator('.mag-playlist-track-play').click();
|
||||
await expect(page.locator('#mag-playlist-transcript [data-playlist-sentence]')).toHaveCount(4);
|
||||
await page.evaluate(() => window.magOpenDriving());
|
||||
await expect(page.locator('#mag-driving')).toBeVisible();
|
||||
await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').duration)).toBeGreaterThan(11);
|
||||
if (speed) await page.evaluate((v) => { const a = document.getElementById('mag-playlist-audio'); a.defaultPlaybackRate = v; a.playbackRate = v; }, speed);
|
||||
}
|
||||
const lit = (page) => page.evaluate(() => {
|
||||
const idx = (sel, wsel, attr) => {
|
||||
const line = document.querySelector(sel); if (!line) return null;
|
||||
const ws = [...line.querySelectorAll(wsel)]; return [Number(line.dataset[attr]), ws.findIndex((w) => w.classList.contains('is-current'))];
|
||||
};
|
||||
return { t: document.getElementById('mag-playlist-audio').currentTime,
|
||||
compact: idx('#mag-driving-lyrics .mag-driving-line.is-current', '.mag-dw', 'i'),
|
||||
lyrics: idx('#mag-driving-full .mag-driving-fline.is-current', '.mag-fw', 'i'),
|
||||
list: idx('#mag-playlist-transcript .is-current', '.mag-w', 'playlistSentence') };
|
||||
});
|
||||
const seekPaused = (page, t) => page.evaluate((x) => { const a = document.getElementById('mag-playlist-audio'); a.pause(); a.currentTime = x; }, t);
|
||||
|
||||
test('word follows the measured times at several positions and seeks, in compact, lyrics and playlist views', async ({ page }) => {
|
||||
await openWords(page);
|
||||
await page.evaluate(() => window.magDrivingSetView('compact'));
|
||||
for (const t of [0.3, 0.9, 1.35, 2.5, 3.8, 4.3, 5.0, 3.0, 8.4, 11.2, 0.8, 7.0]) {
|
||||
await seekPaused(page, t);
|
||||
const want = expectedWord(t);
|
||||
await expect.poll(async () => (await lit(page)).compact, { message: 'compact at ' + t }).toEqual(want);
|
||||
await expect.poll(async () => (await lit(page)).list, { message: 'playlist at ' + t }).toEqual(want);
|
||||
}
|
||||
await page.evaluate(() => window.magDrivingSetView('lyrics'));
|
||||
for (const t of [1.0, 2.0, 4.0, 9.0, 6.0]) {
|
||||
await seekPaused(page, t);
|
||||
await expect.poll(async () => (await lit(page)).lyrics, { message: 'lyrics at ' + t }).toEqual(expectedWord(t));
|
||||
}
|
||||
});
|
||||
|
||||
test('measured words differ from the length estimate: the long-pause word stays lit until the next starts', async ({ page }) => {
|
||||
await openWords(page);
|
||||
await seekPaused(page, 1.52 + 0.0); // word 1 ended at 1.5; next starts at 1.6: word 1 stays lit
|
||||
await expect.poll(async () => (await lit(page)).list).toEqual([0, 1]);
|
||||
await seekPaused(page, 1.7);
|
||||
await expect.poll(async () => (await lit(page)).list).toEqual([0, 2]);
|
||||
});
|
||||
|
||||
for (const speed of [1.5, 2]) {
|
||||
test(`playing at ${speed}x the lit word matches the measured table`, async ({ page }) => {
|
||||
await openWords(page, { speed });
|
||||
await page.evaluate(() => { const a = document.getElementById('mag-playlist-audio'); a.currentTime = 0.4; a.play(); });
|
||||
const wall0 = Date.now(); let checked = 0;
|
||||
while (Date.now() - wall0 < 5000) {
|
||||
const s = await lit(page);
|
||||
const near = [...EFF, ...W_START.flatMap((w) => [0, 3, 6, 9].map((b) => b + w))].some((e) => Math.abs(s.t - e) < 0.07 * speed + 0.04);
|
||||
if (!near && s.compact) { expect(s.compact, 'at ' + s.t).toEqual(expectedWord(s.t)); checked++; }
|
||||
if (s.t > 11.5) break;
|
||||
await page.waitForTimeout(50);
|
||||
}
|
||||
expect(checked).toBeGreaterThan(4);
|
||||
});
|
||||
}
|
||||
|
||||
test('without measured words the highlight falls back to the interpolation and still moves', async ({ page }) => {
|
||||
await openWords(page, { withWords: false });
|
||||
await page.evaluate(() => window.magDrivingSetView('compact'));
|
||||
await seekPaused(page, 1.0);
|
||||
await expect.poll(async () => (await lit(page)).compact?.[1]).toBeGreaterThanOrEqual(0);
|
||||
await seekPaused(page, 2.9);
|
||||
await expect.poll(async () => (await lit(page)).compact?.[1]).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
});
|
||||
|
||||
228
tests/test_word_alignment.py
Normal file
228
tests/test_word_alignment.py
Normal file
@ -0,0 +1,228 @@
|
||||
"""Word-timing job: planning, sidecar client, v2 record format and graceful fallback (sidecar mocked)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
_CONFIG = tempfile.mkdtemp(prefix="ttvc-wordalign-test-")
|
||||
os.environ.setdefault("CONFIG_DIR", _CONFIG)
|
||||
|
||||
import requests
|
||||
|
||||
from core import epub_export, word_alignment as wa
|
||||
|
||||
|
||||
def units_for(record, blocks):
|
||||
return epub_export.plan_sentences(blocks, record["duration_ms"], record, None)
|
||||
|
||||
|
||||
BLOCKS = [{"id": "b1", "text": "Erster Satz hier. Zweiter Satz."}, {"id": "b2", "text": "Dritter Satz."}]
|
||||
RECORD = {"version": 1, "interview": False, "duration_ms": 6000, "segments": [
|
||||
{"chars": len(BLOCKS[0]["text"]), "times": [[0, 2000], [2000, 3000]]},
|
||||
{"chars": len(BLOCKS[1]["text"]), "times": [[3000, 6000]]}]}
|
||||
|
||||
|
||||
def fake_post_factory(calls):
|
||||
def post(url, wav, spans, language, **_):
|
||||
calls.append((url, spans, language))
|
||||
# one word per token, evenly spread over the span
|
||||
out = []
|
||||
for sp in spans:
|
||||
toks = sp["text"].split()
|
||||
step = (sp["end"] - sp["start"]) / len(toks)
|
||||
out.append([{"word": t, "start": round(sp["start"] + i * step, 3),
|
||||
"end": round(sp["start"] + (i + 1) * step, 3), "aligned": True} for i, t in enumerate(toks)])
|
||||
return out
|
||||
return post
|
||||
|
||||
|
||||
class PlanningTests(unittest.TestCase):
|
||||
def test_exact_blocks_get_one_job_per_sentence(self):
|
||||
units, q = units_for(RECORD, BLOCKS)
|
||||
self.assertEqual(q, "exact")
|
||||
jobs = wa.alignment_jobs(units, BLOCKS, False)
|
||||
self.assertEqual([(j["start"], j["end"], j["text"]) for j in jobs],
|
||||
[(0, 2000, "Erster Satz hier."), (2000, 3000, "Zweiter Satz."), (3000, 6000, "Dritter Satz.")])
|
||||
|
||||
def test_anchored_block_is_aligned_as_one_span(self):
|
||||
rec = json.loads(json.dumps(RECORD))
|
||||
rec["segments"][0]["times"] = [[0, 3000]] # one utterance for two sentences
|
||||
units, q = units_for(rec, BLOCKS)
|
||||
self.assertEqual(q, "anchored")
|
||||
jobs = wa.alignment_jobs(units, BLOCKS, False)
|
||||
self.assertEqual(jobs[0]["text"], "Erster Satz hier. Zweiter Satz.")
|
||||
self.assertEqual(jobs[0]["units"], [0, 1])
|
||||
|
||||
def test_speaker_label_is_not_sent_and_gets_a_zero_length_slot(self):
|
||||
blocks = [{"id": "b1", "text": "Frage Eins: Wie geht es? Gut."}]
|
||||
rec = {"version": 1, "interview": True, "duration_ms": 4000, "segments": [
|
||||
{"chars": len("Wie geht es? Gut."), "times": [[0, 2000], [2000, 4000]]}]}
|
||||
units, q = epub_export.plan_sentences(blocks, 4000, rec, [len("Wie geht es? Gut.")])
|
||||
self.assertEqual(q, "exact")
|
||||
jobs = wa.alignment_jobs(units, blocks, True)
|
||||
self.assertEqual(jobs[0]["text"], "Wie geht es?")
|
||||
self.assertEqual(jobs[0]["lead"], 2)
|
||||
|
||||
|
||||
class JobTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.d = Path(tempfile.mkdtemp(prefix="ttvc-wa-"))
|
||||
self.addCleanup(shutil.rmtree, self.d, True)
|
||||
from pydub import AudioSegment
|
||||
from pydub.generators import Sine
|
||||
self.mp3 = self.d / "a1.mp3"
|
||||
Sine(300).to_audio_segment(duration=6000).export(self.mp3, format="mp3")
|
||||
self.rec_path = self.d / "a1.timings.json"
|
||||
self.rec_path.write_text(json.dumps(RECORD))
|
||||
self.units, _ = units_for(RECORD, BLOCKS)
|
||||
self.jobs = wa.alignment_jobs(self.units, BLOCKS, False)
|
||||
|
||||
def test_words_are_written_as_version_2_and_old_fields_survive(self):
|
||||
calls = []
|
||||
got = wa.align_words(self.mp3, self.units, self.jobs, "de", url="http://x", post=fake_post_factory(calls))
|
||||
seg_words = wa.build_segment_words(self.units, got, 2)
|
||||
self.assertTrue(wa.write_words(self.mp3, seg_words))
|
||||
rec = json.loads(self.rec_path.read_text())
|
||||
self.assertEqual(rec["version"], 2)
|
||||
self.assertEqual(rec["segments"][0]["times"], RECORD["segments"][0]["times"]) # v1 data untouched
|
||||
self.assertEqual(len(rec["segments"][0]["words"]), 5) # 3 + 2 tokens
|
||||
self.assertEqual(len(rec["segments"][1]["words"]), 2)
|
||||
first, last = rec["segments"][0]["words"][0], rec["segments"][0]["words"][-1]
|
||||
self.assertAlmostEqual(first[0], 0, delta=150)
|
||||
self.assertAlmostEqual(last[1], 3000, delta=150)
|
||||
# the v1 reader still accepts it
|
||||
from core import magazine as mag
|
||||
self.assertIsNotNone(mag.read_audio_timings(self.mp3))
|
||||
self.assertEqual(calls[0][2], "de")
|
||||
|
||||
def test_chunks_are_bounded_and_progress_reported(self):
|
||||
many = [{"units": [0], "start": i * 10000, "end": i * 10000 + 9000, "text": "a", "lead": 0} for i in range(7)]
|
||||
self.assertTrue(all(len(c) <= 4 for c in wa._chunks(many))) # 45 s per request
|
||||
seen = []
|
||||
wa.align_words(self.mp3, self.units, self.jobs, "de", url="http://x",
|
||||
post=fake_post_factory([]), progress=lambda a, b: seen.append((a, b)))
|
||||
self.assertEqual(seen[-1], (3, 3))
|
||||
|
||||
def test_incomplete_segment_gets_no_words(self):
|
||||
got = {0: [[0, 1]] * 3} # sentence 1 missing for segment 0
|
||||
self.assertEqual(wa.build_segment_words(self.units, got, 2), {})
|
||||
|
||||
|
||||
class ClientFallbackTests(unittest.TestCase):
|
||||
def test_retries_with_backoff_then_raises_when_sidecar_is_down(self):
|
||||
sleeps = []
|
||||
with mock.patch.object(wa.requests, "post", side_effect=requests.ConnectionError("down")) as p:
|
||||
with self.assertRaises(wa.AlignerError):
|
||||
wa._post_chunk("http://x", b"w", [{"start": 0, "end": 1, "text": "a"}], "de", sleep=sleeps.append)
|
||||
self.assertEqual(p.call_count, wa.ATTEMPTS)
|
||||
self.assertEqual(sleeps, [1, 2])
|
||||
|
||||
def test_client_error_is_not_retried(self):
|
||||
resp = mock.Mock(status_code=422)
|
||||
with mock.patch.object(wa.requests, "post", return_value=resp) as p:
|
||||
with self.assertRaises(wa.AlignerError):
|
||||
wa._post_chunk("http://x", b"w", [], "de", sleep=lambda s: None)
|
||||
self.assertEqual(p.call_count, 1)
|
||||
|
||||
def test_busy_then_ok(self):
|
||||
ok = mock.Mock(status_code=200); ok.json.return_value = {"spans": [[]]}
|
||||
with mock.patch.object(wa.requests, "post", side_effect=[mock.Mock(status_code=503), ok]):
|
||||
self.assertEqual(wa._post_chunk("http://x", b"w", [], "de", sleep=lambda s: None), [[]])
|
||||
|
||||
def test_failed_job_is_recorded_and_not_restarted_within_cooldown(self):
|
||||
def run(progress):
|
||||
raise wa.AlignerError("aligner unreachable")
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": "http://x"}):
|
||||
self.assertTrue(wa.should_autostart("docF", "a1"))
|
||||
self.assertTrue(wa.start("docF", "a1", run))
|
||||
for _ in range(100):
|
||||
if wa.status("docF", "a1")["state"] != "running":
|
||||
break
|
||||
time.sleep(0.02)
|
||||
st = wa.status("docF", "a1")
|
||||
self.assertEqual(st["state"], "failed")
|
||||
self.assertFalse(wa.should_autostart("docF", "a1"))
|
||||
|
||||
def test_disabled_without_url(self):
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}):
|
||||
self.assertFalse(wa.should_autostart("docX", "a1"))
|
||||
|
||||
|
||||
class RouteTests(unittest.TestCase):
|
||||
"""GET /timings carries words when present and never waits for the job."""
|
||||
|
||||
def setUp(self):
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
import routes.magazine as routes
|
||||
self.routes = routes
|
||||
self.lib = Path(tempfile.mkdtemp(prefix="ttvc-lib-"))
|
||||
self.addCleanup(shutil.rmtree, self.lib, True)
|
||||
p = mock.patch.object(routes, "_LIB", self.lib); p.start(); self.addCleanup(p.stop)
|
||||
app = FastAPI(); app.include_router(routes.router)
|
||||
self.client = TestClient(app)
|
||||
self.doc = self.client.post("/api/magazine/docs", json={"title": "T"}).json()["id"]
|
||||
d = self.d = self.lib / self.doc
|
||||
blocks = [{"id": "b1", "page": 1, "bbox": [0, 0, 10, 10], "type": "body", "text": BLOCKS[0]["text"], "reading_order": 1},
|
||||
{"id": "b2", "page": 1, "bbox": [0, 40, 10, 10], "type": "body", "text": BLOCKS[1]["text"], "reading_order": 2}]
|
||||
(d / "layout.json").write_text(json.dumps({"pages": [{"page": 1, "width": 600, "height": 800}], "blocks": blocks, "images": []}))
|
||||
(d / "articles.json").write_text(json.dumps({"articles": [
|
||||
{"id": "a1", "title": "T", "pages": [1], "blocks": ["b1", "b2"], "images": [], "status": "approved"}]}))
|
||||
(d / "audio").mkdir(exist_ok=True)
|
||||
(d / "audio" / "a1.mp3").write_bytes(b"x")
|
||||
self.rec = d / "audio" / "a1.timings.json"
|
||||
self.rec.write_text(json.dumps(dict(RECORD, interview=True)))
|
||||
self.url = f"/api/magazine/{self.doc}/articles/a1/timings"
|
||||
|
||||
def test_without_sidecar_the_answer_is_unchanged_and_instant(self):
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}):
|
||||
body = self.client.get(self.url).json()
|
||||
self.assertTrue(body["available"])
|
||||
self.assertEqual(body["words"], "none")
|
||||
self.assertNotIn("words", body["sentences"][0])
|
||||
|
||||
def test_words_are_returned_per_sentence(self):
|
||||
rec = json.loads(self.rec.read_text()); rec["version"] = 2
|
||||
rec["segments"][0]["words"] = [[0, 400], [500, 900], [1000, 1900], [2100, 2500], [2600, 2900]]
|
||||
rec["segments"][1]["words"] = [[3100, 3500], [3600, 5900]]
|
||||
self.rec.write_text(json.dumps(rec))
|
||||
body = self.client.get(self.url).json()
|
||||
self.assertEqual(body["words"], "ready")
|
||||
self.assertEqual(body["sentences"][0]["words"], [[0.0, 0.4], [0.5, 0.9], [1.0, 1.9]])
|
||||
self.assertEqual(body["sentences"][1]["words"], [[2.1, 2.5], [2.6, 2.9]])
|
||||
self.assertEqual(body["sentences"][2]["words"][1], [3.6, 5.9])
|
||||
|
||||
def test_mismatching_words_are_ignored(self):
|
||||
rec = json.loads(self.rec.read_text()); rec["segments"][0]["words"] = [[0, 1]]
|
||||
self.rec.write_text(json.dumps(rec))
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}):
|
||||
body = self.client.get(self.url).json()
|
||||
self.assertNotIn("words", body["sentences"][0])
|
||||
|
||||
def test_first_request_starts_the_job_in_background(self):
|
||||
started = []
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": "http://x"}), \
|
||||
mock.patch.object(self.routes, "_start_word_alignment", lambda *a: started.append(a) or True):
|
||||
body = self.client.get(self.url).json()
|
||||
self.assertEqual(body["words"], "running")
|
||||
self.assertEqual(len(started), 1)
|
||||
self.assertTrue(body["available"])
|
||||
|
||||
def test_explicit_endpoint_requires_configuration_and_timings(self):
|
||||
url = f"/api/magazine/{self.doc}/articles/a1/align-words"
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": ""}):
|
||||
self.assertEqual(self.client.post(url).status_code, 503)
|
||||
with mock.patch.dict(os.environ, {"ALIGNER_URL": "http://x"}):
|
||||
self.rec.unlink()
|
||||
self.assertEqual(self.client.post(url).status_code, 404)
|
||||
self.assertIn("state", self.client.get(url).json())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Reference in New Issue
Block a user