fix(magazine): record where speech starts in each sentence so the highlight is not early

The TTS pads every sentence with 0.4-0.9 s of silence; boundaries are exact but the
highlight lit up that long before the words. Record per-sentence speech onsets at render
time, measure them from the decoded MP3 for older audio, and expose them as 'speech' in
/timings.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -11,6 +11,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
 - **Filter the page grid by status.** Each legend entry is now a toggle button with the number of pages in that status; choose one or several to show only pages that have any of them. It combines with *Show only pages that need me*, the summary says how many pages are shown, *Show all pages* clears everything, a status no page has is greyed out, and the buttons announce their on/off state and work from the keyboard.
 
 ### Fixed
+- **Audio and text now line up when the voice actually starts.** Measured on real rendered articles (an 8-sentence and a 71-sentence MP3): the recorded sentence boundaries are exact (no drift over 10 minutes, no encoder-delay offset: the MP3 carries a LAME/Info header, so browsers trim the encoder delay), but the voice engine pads every sentence with 0.4 to 0.9 s of silence before the first word, so the highlight lit up about 0.7 s early. New renders now record where speech begins in each sentence (`onsets` in `<id>.timings.json`), older audio is measured from the decoded MP3 on request (in memory, nothing is written), and `/timings` returns it as `speech` per sentence.
 - **The item pictures on the character concept sheet show only objects, never hands, feet or people.** The lower-right tiles are now asked for as "a high-quality concept art prop sheet featuring isolated character accessories" on a plain, neutral studio background, still life, strictly inanimate objects, with no people, hands, faces or body parts; the item name (German names included) and the book's world and style stay in the prompt. A body-part list was also added to the item tile's negative prompt, which like the rest only acts if the cfg is raised above 1.0. Existing sheets need to be regenerated.
 - **Driving mode and the playlist highlight the sentence that is actually being spoken.** The player guessed where each sentence sits in the MP3 from its text length, so the highlighted line and the picture drifted from the voice (worst after seeking). The renderer already records the exact time of every sentence (`audio/<id>.timings.json`); a new endpoint `GET /api/magazine/{doc}/articles/{id}/timings` returns them (`{available, sentences: [{text, block, block_id, start, end}]}`, seconds, from the same sentence planner the EPUB uses) and the player uses them, falling back to the old estimate when an MP3 has no usable record (older audio, interview audio, text edited since). The transcript is still fetched with captions, but the MP3 is built without them, so a caption now gets no time slot (the estimate no longer spends audio time on captions either). The highlight is derived only from the audio's current time against the timeline, so the slider, skip buttons, tapping a sentence, lock-screen seek and resume all stay in sync. Covered by a backend test (`ArticleTimingsRouteTests`) and a Playwright test with seeking.
 - **Magazine player buttons stay round on a phone.** In the full-screen player at 375 px wide (iPhone SE) the five transport buttons were 52+76+110+76+52 px plus gaps, wider than the screen, so they shrank unevenly: the big play button became a tall oval and the 15-second skip buttons were squeezed. The buttons now have a fixed 1:1 size that scales down on narrow screens (play 64-84 px, others 44-56 px) and never shrink or stretch, and the player no longer clips its top or bottom when the content is taller than the screen.

Manual.md

@@ -289,7 +289,7 @@ turned into a podcast discussion between two or three voices you pick.
    large touch target and keyboard reachable; the speed also applies to the
    playlist player.
    **Hands-free:** while a track plays, the lock screen, notification shade, Bluetooth headset and car display show the article title, magazine and first-page artwork and offer play/pause, previous/next article and ±15 s (and seeking, where the device offers it). In driving mode the screen is kept awake while audio plays and allowed to sleep when it is paused or you leave. Both are optional browser features; where one is missing or refused, playback itself is unaffected. Opening driving mode resumes the first unfinished article at the position you left it.
-   **Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. Older audio, interview-style audio and audio whose text was edited afterwards has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync.
+   **Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. The highlight lights up a sentence just before its first word is heard (each sentence starts with a short silence, which is not counted). Older audio, interview-style audio and audio whose text was edited afterwards has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync.
    **Share a link:** the *Share* button in the toolbar (the whole magazine), the share button in the playlist header and the one at the top left of driving mode (the current article) share a link of the form `<this app's address>/#s-magazine?doc=<magazine>&article=<article>&mode=drive`. Where the browser has a share sheet (phones, and any browser on https) it opens; otherwise the link is copied and a confirmation is announced (on a plain-http address a box with the link is shown to copy by hand). Opening the link goes straight to that article in driving mode with audio, pictures and text; if the browser refuses to start audio without a tap, a large *Tap to play* button appears. A link for the whole magazine opens at its first unfinished article. **Privacy and access:** the link contains only ids, no password or key. It is just an address: whoever opens it needs to reach this app the same way as for any other page (this app has no user accounts; access is controlled by your network or reverse proxy), and an unknown or removed article simply shows a message. Only articles that already have audio can be shared. A link whose parameters are not in the expected form is ignored.
 
 **Expected result:** the article plays from its headline to its final paragraph.

core/magazine.py

@@ -5719,11 +5719,13 @@ def synthesize_article_audio(doc_dir: Path, article: dict[str, Any], layout: dic
 
     combined = AudioSegment.empty()
     spans: list[tuple[int, int]] = []        # [start_ms, end_ms) of every utterance in the final MP3
+    onsets: list[int] = []                   # ms from each utterance's start to its first audible sound
     for i, (sentence, sentence_voice) in enumerate(utterances):
         audio_bytes, media_type = _tts_request_audio(sentence, sentence_voice, settings)
         fmt = "wav" if audio_bytes[:4] == b"RIFF" or "wav" in media_type.lower() else None
         piece = AudioSegment.from_file(_io.BytesIO(audio_bytes), format=fmt)
         spans.append((len(combined), len(combined) + len(piece)))
+        onsets.append(_speech_onset_ms(piece))
         combined += piece
         if progress_cb:
             progress_cb(i + 1, total)
@@ -5739,7 +5741,7 @@ def synthesize_article_audio(doc_dir: Path, article: dict[str, Any], layout: dic
         # Tags (title, artist, cover, lyrics) are a convenience; losing them must not throw away minutes of
         # synthesis. The audio and its timings are complete without them.
         logging.getLogger(__name__).warning("mutagen is not installed: %s was saved without ID3 tags", out_path.name)
-    write_audio_timings(out_path, segments, utterance_segment, spans, interview=bool(interview_voice))
+    write_audio_timings(out_path, segments, utterance_segment, spans, interview=bool(interview_voice), onsets=onsets)
     return out_path
 
 
@@ -5761,8 +5763,49 @@ def mp3_duration_seconds(path: Path) -> float | None:
         return None
 
 
+def _speech_onset_ms(piece: Any) -> int:
+    """Milliseconds of leading silence in one rendered sentence (TTS pads each clip with 0.4-0.9 s).
+
+    The sentence's span starts where its clip starts, but the words only begin after this lead-in; the
+    player uses it to light a sentence when it is heard rather than when the previous one ends.
+    Quiet means 32 dB below the clip's own peak, so it adapts to the voice's loudness.
+    """
+    try:
+        from pydub.silence import detect_leading_silence
+        if len(piece) < 50 or piece.max_dBFS == float("-inf"):
+            return 0
+        lead = detect_leading_silence(piece, silence_threshold=max(-70.0, piece.max_dBFS - 32.0), chunk_size=10)
+        return int(min(max(0, lead), max(0, len(piece) - 10)))
+    except Exception:
+        return 0
+
+
+_ONSET_CACHE: dict[tuple[str, int, int], list[int]] = {}
+
+
+def measure_speech_onsets(mp3_path: Path, timings: dict[str, Any]) -> list[int] | None:
+    """Per-sentence speech onsets (ms, relative to the sentence start) for an MP3 rendered before they were
+    recorded, measured on the decoded file itself. Cached in memory by file identity; never writes anything."""
+    try:
+        st = mp3_path.stat()
+        key = (str(mp3_path), st.st_mtime_ns, st.st_size)
+        if key in _ONSET_CACHE:
+            return _ONSET_CACHE[key]
+        from pydub import AudioSegment
+        audio = AudioSegment.from_file(mp3_path)
+        times = [t for seg in timings.get("segments", []) for t in seg.get("times", [])]
+        out = [_speech_onset_ms(audio[int(a):int(b)]) for a, b in times]
+        if len(_ONSET_CACHE) > 64:
+            _ONSET_CACHE.clear()
+        _ONSET_CACHE[key] = out
+        return out
+    except Exception:
+        return None
+
+
 def write_audio_timings(mp3_path: Path, segments: Sequence[dict[str, Any]], utterance_segment: Sequence[int],
-                        spans: Sequence[tuple[int, int]], *, interview: bool) -> None:
+                        spans: Sequence[tuple[int, int]], *, interview: bool,
+                        onsets: Sequence[int] | None = None) -> None:
     """Record where every spoken sentence sits in the MP3 (`<id>.timings.json` beside it).
 
     Measured, not estimated: the sentences were rendered one by one and joined, so each one's length is
@@ -5771,12 +5814,15 @@ def write_audio_timings(mp3_path: Path, segments: Sequence[dict[str, Any]], utte
     write only costs the EPUB its exact timings.
     """
     per_segment: list[list[list[int]]] = [[] for _ in segments]
-    for n, (start, end) in zip(utterance_segment, spans):
+    per_onset: list[list[int]] = [[] for _ in segments]
+    for k, (n, (start, end)) in enumerate(zip(utterance_segment, spans)):
         per_segment[n].append([int(start), int(end)])
+        per_onset[n].append(int(onsets[k]) if onsets and k < len(onsets) else 0)
     record = {
         "version": 1, "interview": bool(interview), "duration_ms": int(spans[-1][1]) if spans else 0,
-        "segments": [{"chars": len(str(seg.get("text") or "")), "times": times}
-                     for seg, times in zip(segments, per_segment)],
+        "segments": [{"chars": len(str(seg.get("text") or "")), "times": times,
+                      **({"onsets": on} if onsets else {})}
+                     for seg, times, on in zip(segments, per_segment, per_onset)],
     }
     try:
         tmp = mp3_path.with_suffix(".timings.tmp")

routes/magazine.py

@@ -730,9 +730,18 @@ async def magazine_article_timings(doc_id: str, article_id: str):
     units, quality = epub_export.plan_sentences(blocks, float(timings["duration_ms"]), timings)
     if quality == "estimated":
         return none
+    # Speech onset: the TTS pads every sentence with a lead-in of silence, so the words start later than the
+    # span does. Recorded at render time; measured from the decoded MP3 for audio rendered before that.
+    flat = [o for seg in timings["segments"] for o in (seg.get("onsets") or [])]
+    if len(flat) != len(units):
+        flat = (await asyncio.to_thread(mag.measure_speech_onsets, mp3, timings)) if quality == "exact" else None
+    if not flat or len(flat) != len(units):
+        flat = [0] * len(units)
     return {"available": True, "quality": quality, "sentences": [
         {"text": u["text"], "block": u["block"], "block_id": blocks[u["block"]].get("id"),
-         "start": round(u["start"] / 1000, 3), "end": round(u["end"] / 1000, 3)} for u in units]}
+         "start": round(u["start"] / 1000, 3), "end": round(u["end"] / 1000, 3),
+         "speech": round((u["start"] + min(max(0, o), max(0, u["end"] - u["start"] - 10))) / 1000, 3)}
+        for u, o in zip(units, flat)]}
 
 
 @router.post("/api/magazine/{doc_id}/articles/auto-link")

tests/test_audio_onsets.py

@@ -0,0 +1,42 @@
+"""Speech onsets: the lead-in of silence TTS puts before each sentence is measured and recorded."""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from pydub import AudioSegment
+from pydub.generators import Sine
+
+from core import magazine as mag
+
+
+def _clip(lead_ms: int, tone_ms: int = 400) -> AudioSegment:
+    return AudioSegment.silent(lead_ms, frame_rate=24000) + Sine(300, sample_rate=24000).to_audio_segment(tone_ms).apply_gain(-6)
+
+
+def test_onset_is_the_leading_silence():
+    assert abs(mag._speech_onset_ms(_clip(700)) - 700) <= 20
+    assert mag._speech_onset_ms(_clip(0)) <= 10
+
+
+def test_silent_or_tiny_clip_has_no_onset():
+    assert mag._speech_onset_ms(AudioSegment.silent(500, frame_rate=24000)) == 0
+    assert mag._speech_onset_ms(AudioSegment.silent(20, frame_rate=24000)) == 0
+
+
+def test_onsets_are_written_per_segment(tmp_path):
+    mp3 = tmp_path / "a.mp3"
+    segs = [{"text": "One. Two."}, {"text": "Three."}]
+    mag.write_audio_timings(mp3, segs, [0, 0, 1], [(0, 1000), (1000, 2000), (2000, 3000)], interview=False, onsets=[700, 650, 500])
+    rec = mag.read_audio_timings(mp3)
+    assert [s["onsets"] for s in rec["segments"]] == [[700, 650], [500]]
+    assert [s["times"] for s in rec["segments"]] == [[[0, 1000], [1000, 2000]], [[2000, 3000]]]
+
+
+def test_old_audio_onsets_are_measured_from_the_file(tmp_path):
+    mp3 = tmp_path / "old.mp3"
+    (_clip(600) + _clip(800)).export(mp3, format="mp3")
+    total = int(AudioSegment.from_file(mp3).duration_seconds * 1000)
+    timings = {"version": 1, "segments": [{"chars": 1, "times": [[0, 1000], [1000, total]]}]}
+    got = mag.measure_speech_onsets(mp3, timings)
+    assert got is not None and abs(got[0] - 600) <= 60 and abs(got[1] - 800) <= 60