diff --git a/CHANGELOG.md b/CHANGELOG.md index b887102..65c024a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi - **Filter the page grid by status.** Each legend entry is now a toggle button with the number of pages in that status; choose one or several to show only pages that have any of them. It combines with *Show only pages that need me*, the summary says how many pages are shown, *Show all pages* clears everything, a status no page has is greyed out, and the buttons announce their on/off state and work from the keyboard. ### Fixed +- **Driving mode and the playlist highlight the sentence that is actually being spoken.** The player guessed where each sentence sits in the MP3 from its text length, so the highlighted line and the picture drifted from the voice (worst after seeking). The renderer already records the exact time of every sentence (`audio/.timings.json`); a new endpoint `GET /api/magazine/{doc}/articles/{id}/timings` returns them (`{available, sentences: [{text, block, block_id, start, end}]}`, seconds, from the same sentence planner the EPUB uses) and the player uses them, falling back to the old estimate when an MP3 has no usable record (older audio, interview audio, text edited since). The transcript is still fetched with captions, but the MP3 is built without them, so a caption now gets no time slot (the estimate no longer spends audio time on captions either). The highlight is derived only from the audio's current time against the timeline, so the slider, skip buttons, tapping a sentence, lock-screen seek and resume all stay in sync. Covered by a backend test (`ArticleTimingsRouteTests`) and a Playwright test with seeking. - **Magazine player buttons stay round on a phone.** In the full-screen player at 375 px wide (iPhone SE) the five transport buttons were 52+76+110+76+52 px plus gaps, wider than the screen, so they shrank unevenly: the big play button became a tall oval and the 15-second skip buttons were squeezed. The buttons now have a fixed 1:1 size that scales down on narrow screens (play 64-84 px, others 44-56 px) and never shrink or stretch, and the player no longer clips its top or bottom when the content is taller than the screen. - **No gaps on the character concept sheet.** After the divider lines were removed, light cream gutters remained between the three poses and above the lower pictures, and the left pose's cape and the right pose's sword were clipped at them. The composer cut the generated triptych into thirds and pasted each into an inset panel (8 px per side), and the lower pictures had 12-18 px padding. The poses are now pasted as one whole image with no cut and no inset, and the lower pictures sit flush under it. The generation prompt never asked for gaps, so it is unchanged. Sheets already stored have the gaps baked in and need to be regenerated. - **No divider lines on the character concept sheet.** The Konzeptbild composer drew light lines between the three poses and between the top and bottom rows, which cut through capes and swords. They are gone. Sheets already stored have the lines baked into the image file and need to be regenerated to lose them. diff --git a/Manual.md b/Manual.md index 95dc717..5ab10ae 100644 --- a/Manual.md +++ b/Manual.md @@ -289,6 +289,7 @@ turned into a podcast discussion between two or three voices you pick. large touch target and keyboard reachable; the speed also applies to the playlist player. **Hands-free:** while a track plays, the lock screen, notification shade, Bluetooth headset and car display show the article title, magazine and first-page artwork and offer play/pause, previous/next article and ±15 s (and seeking, where the device offers it). In driving mode the screen is kept awake while audio plays and allowed to sleep when it is paused or you leave. Both are optional browser features; where one is missing or refused, playback itself is unaffected. Opening driving mode resumes the first unfinished article at the position you left it. + **Text and picture follow the voice:** for audio generated since sentence times were recorded, the highlighted sentence (playlist and driving mode) and the picture shown come from the exact time each sentence was spoken, so they stay right after the slider, the 15-second buttons, a tap on a sentence, the lock screen or a resume. Captions are shown but are not in the MP3, so they are never highlighted. Older audio, interview-style audio and audio whose text was edited afterwards has no usable record and is followed by an estimate from sentence length (less exact); regenerate the audio to get exact sync. **Expected result:** the article plays from its headline to its final paragraph. Page furniture (running heads, page numbers), adverts, sidebars and pull quotes diff --git a/routes/magazine.py b/routes/magazine.py index 8f9ea72..cf86645 100644 --- a/routes/magazine.py +++ b/routes/magazine.py @@ -707,6 +707,34 @@ async def magazine_article_text(doc_id: str, article_id: str, return {"article": target, "blocks": blocks, "asides": mag.article_asides(layout, target, articles)} +@router.get("/api/magazine/{doc_id}/articles/{article_id}/timings") +async def magazine_article_timings(doc_id: str, article_id: str): + """Exact per-sentence times of the article's MP3, for the player's text and picture sync. + + {available, sentences: [{text, block, block_id, start, end}]} with start/end in seconds, over the narrated + blocks only (captions are not in the MP3, so they get no slot). `available` is false when the MP3 has no + usable timing record (older audio, an interview render, or text edited since): the player then estimates. + """ + from core import epub_export + d, _ = _require_magazine(doc_id) + article_id = _article_id(article_id) + target = next((a for a in mag.load_articles(d) if a.get("id") == article_id), None) + if target is None: + raise HTTPException(404, "Article not found") + none = {"available": False, "quality": "estimated", "sentences": []} + mp3 = d / "audio" / f"{article_id}.mp3" + timings = mag.read_audio_timings(mp3) if mp3.is_file() else None + if not timings or not timings.get("duration_ms"): + return none + blocks = mag.article_text(mag.load_layout(d), target, fuse=True, glossary=target.get("glossary")) + units, quality = epub_export.plan_sentences(blocks, float(timings["duration_ms"]), timings) + if quality == "estimated": + return none + return {"available": True, "quality": quality, "sentences": [ + {"text": u["text"], "block": u["block"], "block_id": blocks[u["block"]].get("id"), + "start": round(u["start"] / 1000, 3), "end": round(u["end"] / 1000, 3)} for u in units]} + + @router.post("/api/magazine/{doc_id}/articles/auto-link") async def magazine_auto_link(doc_id: str): """Link continuation pages into the articles they belong to (see mag.auto_link_pages). Undoable.""" diff --git a/static/js/magazine.js b/static/js/magazine.js index 7b7888e..ca88168 100644 --- a/static/js/magazine.js +++ b/static/js/magazine.js @@ -29,6 +29,7 @@ const state = { pagesCols: 8, // "All pages" grid column count, user-adjustable 1-12 playlistCurrent: null, // article id of the MP3 playlist's current track playlistSentences: [], // transcript aligned approximately to the exported MP3 + playlistExact: null, // exact [{start,end}] per narrated sentence from /timings, or null (estimate) playlistImages: [], // figures/tables/charts shown alongside that transcript playlistLoadToken: 0, // prevents a slow text fetch replacing a newer track playlistLastSaved: 0, @@ -5117,15 +5118,49 @@ function savePlaylistProgress(completed, immediate) { renderPlaylistTracks(); } +// One {start, end, skip} per displayed sentence, in seconds on the MP3's own clock. +// The text is fetched with captions (shown), but the MP3 is built without them, so a caption sentence has no +// slot (skip, zero width, sitting where the narration reaches it). Narrated sentences use the exact times the +// renderer recorded (state.playlistExact, one entry per narrated sentence, from /timings) when there are any, +// else a length-based estimate over `duration`. The highlight only ever reads audio.currentTime against this, +// so seeking by slider, skip buttons, click, lock screen or resume cannot desync it. function playlistTimeline(duration) { - const weights = state.playlistSentences.map((s) => Math.max(6, s.text.replace(/\s+/g, ' ').length)); - const total = weights.reduce((sum, n) => sum + n, 0) || 1; - let cursor = 0; - return weights.map((weight) => { - const start = cursor / total * duration; - cursor += weight; - return { start, end: cursor / total * duration }; - }); + const sentences = state.playlistSentences; + const exact = state.playlistExact; + const key = state.playlistCurrent + '|' + sentences.length + '|' + (exact ? exact.length : 0) + '|' + Math.round(duration * 100); + if (state.playlistTimelineKey === key && state.playlistTimelineCache) return state.playlistTimelineCache; + const narrated = sentences.filter((s) => s.type !== 'caption'); + let result; + if (exact && exact.length === narrated.length && exact.length) { + let n = 0, cursor = 0; + result = sentences.map((s) => { + if (s.type === 'caption') return { start: cursor, end: cursor, skip: true }; + const t = exact[n++]; + cursor = t.end; + return { start: t.start, end: t.end, skip: false }; + }); + } else { + const weights = sentences.map((s) => s.type === 'caption' ? 0 : Math.max(6, s.text.replace(/\s+/g, ' ').length)); + const total = weights.reduce((sum, n) => sum + n, 0) || 1; + let cursor = 0; + result = weights.map((weight) => { + const start = cursor / total * duration; + cursor += weight; + return { start, end: cursor / total * duration, skip: weight === 0 }; + }); + } + state.playlistTimelineKey = key; + state.playlistTimelineCache = result; + return result; +} + +// The sentence the audio is in: the first narrated slot that has not ended yet; past the end, the last one. +function playlistIndexAt(timeline, time) { + let index = timeline.findIndex((t) => !t.skip && time < t.end); + if (index < 0) { + for (let i = timeline.length - 1; i >= 0; i--) if (!timeline[i].skip) { index = i; break; } + } + return Math.max(0, index); } function magPlaylistSeekSentence(index) { @@ -5175,8 +5210,7 @@ function magPlaylistSync() { if (!audio || !a) return; const duration = playlistDuration(a); const timeline = playlistTimeline(duration); - let index = timeline.findIndex((t) => audio.currentTime < t.end); - if (index < 0) index = Math.max(0, timeline.length - 1); + const index = playlistIndexAt(timeline, audio.currentTime); const follow = $('mag-playlist-karaoke')?.checked !== false; document.querySelectorAll('#mag-playlist-transcript [data-playlist-sentence]').forEach((el) => { el.classList.toggle('is-current', follow && Number(el.dataset.playlistSentence) === index); @@ -5206,12 +5240,22 @@ async function loadPlaylistCompanion(a) { const token = ++state.playlistLoadToken; state.playlistSentences = []; state.playlistImages = []; + state.playlistExact = null; renderPlaylistCompanion(); try { await ensureLayout(); - const data = await api('/api/magazine/' + state.doc.id + '/articles/' + a.id + '/text?captions=1&descriptions=0'); + const base = '/api/magazine/' + state.doc.id + '/articles/' + a.id; + // Timings are optional: older audio has none and the player then estimates. + const [data, timings] = await Promise.all([ + api(base + '/text?captions=1&descriptions=0'), + api(base + '/timings').catch(() => null), + ]); if (token !== state.playlistLoadToken || state.playlistCurrent !== a.id) return; state.playlistSentences = splitSentences(data.blocks || []); + state.playlistExact = timings && timings.available && Array.isArray(timings.sentences) + ? timings.sentences.map((u) => ({ start: Number(u.start), end: Number(u.end) })) + .filter((u) => Number.isFinite(u.start) && Number.isFinite(u.end)) + : null; state.playlistImages = articleImages(a); renderPlaylistCompanion(); } catch (_) { diff --git a/tests/magazine.spec.js b/tests/magazine.spec.js index 81a3c1c..35cc76c 100644 --- a/tests/magazine.spec.js +++ b/tests/magazine.spec.js @@ -96,6 +96,9 @@ async function mockBackend(page, { articleStatus = 'detected', hasAudio = false, await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/text*`, route => route.fulfill({ json: { article, blocks: LAYOUT.blocks }, })); + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/timings`, route => route.fulfill({ + json: { available: false, quality: 'estimated', sentences: [] }, + })); await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/audio`, route => route.fulfill({ body: silentWav(2000), contentType: 'audio/wav', })); @@ -1666,6 +1669,54 @@ test.describe('Hands-free listening', () => { await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').currentTime)).toBeGreaterThan(1); }); + test('driving-mode text follows the recorded sentence times and gives captions no slot', async ({ page }) => { + await mockBackend(page, { articleStatus: 'approved', hasAudio: true }); + const blocks = [ + { id: 'b1', page: 3, bbox: [0, 0, 1, 1], type: 'title', text: 'Heading Here', reading_order: 0 }, + { id: 'b3', page: 3, bbox: [0, 0, 1, 1], type: 'body', text: 'First one. Second one.', reading_order: 1 }, + { id: 'c1', page: 3, bbox: [0, 0, 1, 1], type: 'caption', text: 'A caption here.', reading_order: 2 }, + { id: 'b4', page: 3, bbox: [0, 0, 1, 1], type: 'body', text: 'Last one.', reading_order: 3 }, + ]; + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/text*`, route => route.fulfill({ json: { article: ARTICLE_APPROVED, blocks } })); + // Recorded times are far from what text length would suggest (the estimate puts 0.25 s in the heading). + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/timings`, route => route.fulfill({ json: { available: true, quality: 'exact', sentences: [ + { text: 'Heading Here', block: 0, start: 0, end: 0.2 }, { text: 'First one.', block: 1, start: 0.2, end: 0.3 }, + { text: 'Second one.', block: 1, start: 0.3, end: 1.9 }, { text: 'Last one.', block: 2, start: 1.9, end: 2.0 }] } })); + // Seeking needs a server that answers Range requests, like the real audio route does. + await page.route(`**/api/magazine/${DOC_ID}/articles/article-001/audio`, route => { + const wav = silentWav(2000), range = /bytes=(\d+)-(\d*)/.exec(route.request().headers().range || ''); + if (!range) return route.fulfill({ body: wav, contentType: 'audio/wav', headers: { 'Accept-Ranges': 'bytes' } }); + const from = Number(range[1]), to = range[2] ? Number(range[2]) : wav.length - 1; + return route.fulfill({ status: 206, body: wav.subarray(from, to + 1), contentType: 'audio/wav', + headers: { 'Accept-Ranges': 'bytes', 'Content-Range': `bytes ${from}-${to}/${wav.length}` } }); + }); + await page.goto('/', { waitUntil: 'domcontentloaded' }); + await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 }); + await navTo(page, 's-magazine'); + await page.locator('#mag-library-list .mag-card').first().click(); + await page.getByRole('button', { name: 'Playlist' }).click(); + await page.locator('.mag-playlist-track-play').click(); + await expect(page.locator('#mag-playlist-transcript [data-playlist-sentence]')).toHaveCount(5); + await page.evaluate(() => window.magOpenDriving()); + await expect(page.locator('#mag-driving')).toBeVisible(); + await expect(page.locator('#mag-driving-lyrics .mag-driving-line')).toHaveCount(5); + await page.evaluate(() => document.getElementById('mag-playlist-audio').pause()); + const at = async (t) => { + await page.evaluate((x) => { document.getElementById('mag-playlist-audio').currentTime = x; }, t); + await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').currentTime)).toBeCloseTo(t, 1); + await page.evaluate(() => window.magPlaylistSync()); + return page.evaluate(() => ({ + driving: document.querySelector('#mag-driving-lyrics .mag-driving-line.is-current')?.dataset.i, + list: document.querySelector('#mag-playlist-transcript .is-current')?.dataset.playlistSentence, + })); + }; + // sentence index 3 is the caption: it never gets the highlight, and "Last one." is index 4 + expect(await at(0.25)).toEqual({ driving: '1', list: '1' }); + expect(await at(1.0)).toEqual({ driving: '2', list: '2' }); + expect(await at(1.95)).toEqual({ driving: '4', list: '4' }); + expect(await at(0.05)).toEqual({ driving: '0', list: '0' }); // seeking back works the same way + }); + test('the screen is kept awake only while driving mode is open and playing', async ({ page }) => { await startPlayback(page); expect(await page.evaluate(() => window.__wl.requested)).toBe(0); // reading view: no lock diff --git a/tests/test_magazine_routes.py b/tests/test_magazine_routes.py index f99e5b8..4cb1966 100644 --- a/tests/test_magazine_routes.py +++ b/tests/test_magazine_routes.py @@ -2330,3 +2330,67 @@ class DeletePictureRouteTests(unittest.TestCase): self.assertIn("fig", {b["id"] for b in layout["blocks"]}) self.assertIn("p1-region-fig", {i["id"] for i in layout["images"]}) self.assertEqual(arts["a1"]["blocks"], ["fig", "txt"]) + + +class ArticleTimingsRouteTests(unittest.TestCase): + """GET .../articles//timings: exact sentence times for the player, none for captions.""" + + def setUp(self): + from fastapi import FastAPI + from fastapi.testclient import TestClient + import routes.magazine as routes + from unittest import mock + self.lib = Path(tempfile.mkdtemp(prefix="ttvc-lib-")) + self.addCleanup(shutil.rmtree, self.lib, True) + p = mock.patch.object(routes, "_LIB", self.lib) + p.start(); self.addCleanup(p.stop) + app = FastAPI(); app.include_router(routes.router) + self.client = TestClient(app) + self.doc = self.client.post("/api/magazine/docs", json={"title": "T"}).json()["id"] + self.d = self.lib / self.doc + self.t1, self.t2 = "Erster Satz. Zweiter Satz.", "Dritter Satz." + blocks = [ + {"id": "b1", "page": 1, "bbox": [0, 0, 10, 10], "type": "body", "text": self.t1, "reading_order": 1}, + {"id": "c1", "page": 1, "bbox": [0, 20, 10, 10], "type": "caption", "text": "Bildunterschrift hier.", "reading_order": 2}, + {"id": "b2", "page": 1, "bbox": [0, 40, 10, 10], "type": "body", "text": self.t2, "reading_order": 3}, + ] + (self.d / "layout.json").write_text(json.dumps({"pages": [{"page": 1, "width": 600, "height": 800}], "blocks": blocks, "images": []})) + (self.d / "articles.json").write_text(json.dumps({"articles": [ + {"id": "a1", "title": "T", "pages": [1], "blocks": ["b1", "c1", "b2"], "images": [], "status": "approved"}]})) + (self.d / "audio").mkdir(exist_ok=True) + + def url(self, art="a1"): + return f"/api/magazine/{self.doc}/articles/{art}/timings" + + def write(self, record): + (self.d / "audio" / "a1.mp3").write_bytes(b"x") + (self.d / "audio" / "a1.timings.json").write_text(json.dumps(record)) + + def test_exact_timings_cover_narrated_sentences_only_and_skip_captions(self): + self.write({"version": 1, "interview": False, "duration_ms": 6000, "segments": [ + {"chars": len(self.t1), "times": [[0, 1500], [1500, 3000]]}, + {"chars": len(self.t2), "times": [[3000, 6000]]}]}) + r = self.client.get(self.url()) + self.assertEqual(r.status_code, 200, r.text) + body = r.json() + self.assertTrue(body["available"]) + self.assertEqual(body["quality"], "exact") + self.assertEqual([(s["text"], s["start"], s["end"]) for s in body["sentences"]], + [("Erster Satz.", 0.0, 1.5), ("Zweiter Satz.", 1.5, 3.0), ("Dritter Satz.", 3.0, 6.0)]) + self.assertNotIn("c1", [s["block_id"] for s in body["sentences"]]) # no slot for the caption + + def test_no_timing_record_means_not_available(self): + self.assertEqual(self.client.get(self.url()).json()["available"], False) + (self.d / "audio" / "a1.mp3").write_bytes(b"x") + self.assertEqual(self.client.get(self.url()).json()["sentences"], []) + + def test_stale_or_interview_records_are_not_used(self): + self.write({"version": 1, "interview": False, "duration_ms": 6000, "segments": [ + {"chars": 3, "times": [[0, 1]]}, {"chars": len(self.t2), "times": [[1, 2]]}]}) + self.assertFalse(self.client.get(self.url()).json()["available"]) # text changed since the render + self.write({"version": 1, "interview": True, "duration_ms": 6000, "segments": []}) + self.assertFalse(self.client.get(self.url()).json()["available"]) + + def test_unknown_article_is_404_and_bad_id_is_400(self): + self.assertEqual(self.client.get(self.url("nope")).status_code, 404) + self.assertEqual(self.client.get(self.url("a b")).status_code, 400)