diff --git a/CHANGELOG.md b/CHANGELOG.md index d1ba52c..7fd2d23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ## [Unreleased] ### Added +- **Hands-free listening.** The article player now registers with the browser's Media Session: title, magazine and first-page artwork on the lock screen / headset / car display, with play, pause, previous/next article, ±15 s and seek, plus a position bar. While driving mode is open and audio is playing the screen is kept awake (Screen Wake Lock), and released on pause or exit; a refused lock never affects playback. - **EPUB 3 with Media Overlays.** *EPUB with audio* in the contents toolbar (and `GET /api/magazine/{id}/epub`) packages the approved articles that have audio into a book whose text highlights as the audio plays. MP3 export now records each sentence's real start and end (`audio/.timings.json`, no text), so new audio highlights exactly; older audio is placed by estimate and the book's description says so. Validated with W3C EPUBCheck 5.2.1 (0 errors, 0 warnings) on a real 12-minute two-article issue. - **Magazine reading view shows sidebars.** A sidebar was kept out of narration and therefore never shown at all. The article text endpoint now also returns `asides` and the reflowed view displays each one where it sits, labelled "Sidebar · not read aloud · p. N", as plain text that is neither a sentence nor highlighted. Only for pages the article alone occupies (on a shared page there is no telling which article a sidebar belongs to); not for advertisement pages; a sidebar you ticked into the narration stays narrated. - **Magazine review: extraction diff.** A region whose text you edited by hand is marked *Edited*, with *Show changes* (word-level comparison against the text as first recognised; removals struck through and marked "−", additions underlined and marked "+") and *Restore recognised text*. The first reading is kept in a new `ocr_text` field across repeated edits; re-reading the region or typing the original back clears it. @@ -18,6 +19,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi - **"Order by clicking" now keeps its controls on screen.** *Save order*, *Undo last* and *Cancel* lived in the toolbar row and could scroll out of reach, leaving the task unfinishable. While ordering they are a floating panel pinned to the bottom-right of the window (full width on a phone): a progress meter, a large *Save order* button that turns green when every box is numbered, *Undo last*, and *Cancel* (also **Esc**, except while typing in a field). It sits above the Back/Approve bar and clear of the page image. Tests scroll the real scroll container, and were verified to fail against the old layout. ### Fixed +- **"Listen" on an article that already has audio did nothing until the native play button was pressed.** The player element is `preload="none"`, and starting a track waited for `loadedmetadata`, which such an element never fires. Choosing a track now sets `preload` so it loads and plays (and resumes at the saved position). - **MP3 export no longer throws away a finished synthesis when ID3 tagging is unavailable, and article durations no longer read 0 without mutagen.** The deployed image lacked `mutagen` although requirements.txt lists it, so tagging raised after every sentence had been rendered and the contents showed no MP3 lengths. Tagging is now best-effort (a warning), and durations fall back to ffprobe. Rebuild the image to get the tags back. - Docker: added an allow-list `.dockerignore`, so a build no longer uploads `config/` (user data, settings, keys), logs, tests or the git history into the build context; only the files the Dockerfile copies are sent. New image `tts-voice-creator-clone-and-design-2:latest` (also `:1.55.0`) includes `mutagen`; the previous image is kept as `:rollback-20260930`. diff --git a/Manual.md b/Manual.md index f9b5f7d..601102f 100644 --- a/Manual.md +++ b/Manual.md @@ -254,6 +254,7 @@ turned into a podcast discussion between two or three voices you pick. **Text** / **Images** toggles that show or hide the article text as karaoke (the whole article scrolls, the sentence being read is marked and the current word is highlighted; word timing is estimated from the MP3 position) and the article's current picture. Everything is a large touch target and keyboard reachable; the speed also applies to the playlist player. + **Hands-free:** while a track plays, the lock screen, notification shade, Bluetooth headset and car display show the article title, magazine and first-page artwork and offer play/pause, previous/next article and ±15 s (and seeking, where the device offers it). In driving mode the screen is kept awake while audio plays and allowed to sleep when it is paused or you leave. Both are optional browser features; where one is missing or refused, playback itself is unaffected. Opening driving mode resumes the first unfinished article at the position you left it. **Expected result:** the article plays from its headline to its final paragraph. Page furniture (running heads, page numbers), adverts, sidebars and pull quotes diff --git a/TODO.md b/TODO.md index e034f2d..74da19f 100644 --- a/TODO.md +++ b/TODO.md @@ -720,8 +720,8 @@ casting. The original PDF remains the visual source of truth while approved arti *(EPUB 3 Media Overlay export done 2026-09-30, EPUBCheck-clean, with exact sentence timings for newly rendered audio; word-level highlighting is still a character-length estimate within each sentence, not forced alignment; the in-app MP3 player still aligns its transcript by estimate and could use the new timing records)* -- [~] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume. - *(driving mode done: two oversized controls, reuses whatever is already playing/queued; not mobile-specific)* +- [x] Add mobile/driving mode with large controls, automatic article advance, and hands-free resume. + *(driving mode with large controls, queue auto-advance and resume at the saved position; 2026-09-30: Media Session for lock screen / headset / car controls and artwork, screen wake lock while playing)* - [~] Add OCR-engine benchmark comparisons and per-page fallback selection. *(per-page fallback done, now a real two-engine chain: vision OCR escalates when a page needs OCR or scores poorly, KDL Frontier Parser nano is the vision fallback, tesseract is the last resort; no formal benchmark)* diff --git a/static/js/magazine.js b/static/js/magazine.js index 769ce6b..73d8cd0 100644 --- a/static/js/magazine.js +++ b/static/js/magazine.js @@ -4820,6 +4820,9 @@ function magPlaylistPlay(articleId, options) { if (audio) { state.playlistSwitching = true; audio.dataset.article = articleId; + // The element ships with preload="none" (nothing is fetched until a track is chosen). Choosing one must + // load it: with "none" the browser never fires loadedmetadata, and the code below waits for exactly that. + audio.preload = 'auto'; audio.src = '/api/magazine/' + state.doc.id + '/articles/' + articleId + '/audio'; queueMicrotask(() => { state.playlistSwitching = false; }); const saved = (state.doc.progress || {})[articleId] || {}; @@ -5093,6 +5096,7 @@ function magToggleDriving() { const overlay = $('mag-driving'); if (overlay) overlay.hidden = !state.driving; if (state.driving) renderDriving(); + magDrivingWake(); } function magOpenDriving() { @@ -5290,12 +5294,85 @@ function magDrivingPrevious() { } } +// ── Lock screen, headset and car controls; screen stays awake while driving ─── +// Hands-free means the phone in a pocket or a dashboard mount must still be controllable: the Media Session +// API puts title, artwork, position and play/pause/next/previous/±15 s on the lock screen, in a notification, +// on a Bluetooth headset or a car's display. Everything is optional: a browser without it simply plays. +const MAG_SEEK_STEP = 15; + +function magMediaSessionSync() { + if (!('mediaSession' in navigator)) return; + const audio = $('mag-playlist-audio'); + const a = state.articles.find((x) => x.id === state.playlistCurrent); + if (!audio || !a) return; + try { + const first = (a.pages || [])[0]; + navigator.mediaSession.metadata = new MediaMetadata({ + title: a.title || 'Untitled', + artist: a.author || state.doc?.title || '', + album: [state.doc?.title, state.doc?.issue].filter(Boolean).join(' — '), + artwork: first ? [{ src: '/api/magazine/' + state.doc.id + '/pages/' + first, type: 'image/png' }] : [], + }); + navigator.mediaSession.playbackState = audio.paused ? 'paused' : 'playing'; + } catch (_) { /* metadata is presentation only */ } +} + +function magMediaSessionPosition() { + if (!('mediaSession' in navigator) || !navigator.mediaSession.setPositionState) return; + const audio = $('mag-playlist-audio'); + const now = Date.now(); + if (!audio || now - (state._msPosAt || 0) < 1000) return; // once a second is plenty for a lock screen + state._msPosAt = now; + if (!Number.isFinite(audio.duration) || audio.duration <= 0) return; + try { + navigator.mediaSession.setPositionState({ + duration: audio.duration, playbackRate: audio.playbackRate || 1, position: Math.min(audio.currentTime, audio.duration), + }); + } catch (_) { /* inconsistent state during a track change: skip this tick */ } +} + +function magWireMediaSession() { + if (!('mediaSession' in navigator) || state._msWired) return; + state._msWired = true; + const audio = () => $('mag-playlist-audio'); + const handlers = { + play: () => audio()?.play().catch(() => {}), + pause: () => audio()?.pause(), + stop: () => audio()?.pause(), + previoustrack: () => magDrivingPrevious(), + nexttrack: () => magDrivingNext(), + seekbackward: (d) => magDrivingSkip(-(d?.seekOffset || MAG_SEEK_STEP)), + seekforward: (d) => magDrivingSkip(d?.seekOffset || MAG_SEEK_STEP), + seekto: (d) => { const el = audio(); if (el && Number.isFinite(d?.seekTime)) el.currentTime = d.seekTime; }, + }; + Object.entries(handlers).forEach(([action, fn]) => { + try { navigator.mediaSession.setActionHandler(action, fn); } catch (_) { /* action not supported here */ } + }); +} + +// Keep the screen on while driving mode is open AND audio is playing; let it sleep otherwise (battery). +async function magDrivingWake() { + if (!('wakeLock' in navigator)) return; + const audio = $('mag-playlist-audio'); + const want = !!(state.driving && audio && !audio.paused); + try { + if (want && !state.wakeLock) { + state.wakeLock = await navigator.wakeLock.request('screen'); + state.wakeLock.addEventListener?.('release', () => { state.wakeLock = null; }); + } else if (!want && state.wakeLock) { + const lock = state.wakeLock; + state.wakeLock = null; + await lock.release(); + } + } catch (_) { state.wakeLock = null; /* refused (battery saver, hidden tab): playback is unaffected */ } +} + function wirePlaylistAudio() { const audio = $('mag-playlist-audio'); if (!audio || audio.dataset.wired) return; audio.dataset.wired = '1'; audio.addEventListener('ended', _magPlaylistNext); - audio.addEventListener('timeupdate', () => { magPlaylistSync(); savePlaylistProgress(false, false); }); + audio.addEventListener('timeupdate', () => { magPlaylistSync(); savePlaylistProgress(false, false); magMediaSessionPosition(); }); audio.addEventListener('loadedmetadata', () => { const a = state.articles.find((x) => x.id === state.playlistCurrent); if (a && Number.isFinite(audio.duration)) a.audio_seconds = audio.duration; @@ -5303,11 +5380,16 @@ function wirePlaylistAudio() { magPlaylistSync(); }); audio.addEventListener('loadedmetadata', magApplyPlaybackSpeed); - audio.addEventListener('play', syncDrivingPlayButton); + audio.addEventListener('play', () => { syncDrivingPlayButton(); magWireMediaSession(); magMediaSessionSync(); magDrivingWake(); }); audio.addEventListener('pause', () => { if (!state.playlistSwitching) savePlaylistProgress(false, true); syncDrivingPlayButton(); + magMediaSessionSync(); + magDrivingWake(); }); + audio.addEventListener('loadedmetadata', magMediaSessionSync); + // The browser drops a screen wake lock when the tab is hidden; take it again on return. + document.addEventListener('visibilitychange', () => { if (document.visibilityState === 'visible') magDrivingWake(); }); } // ── Boot ───────────────────────────────────────────────────────────────────── diff --git a/tests/magazine.spec.js b/tests/magazine.spec.js index eb46744..1d01e9e 100644 --- a/tests/magazine.spec.js +++ b/tests/magazine.spec.js @@ -1617,3 +1617,81 @@ test('no EPUB link for an approved article without audio', async ({ page }) => { await page.waitForSelector('.mag-toc-row'); await expect(page.locator('#mag-epub-link')).toBeHidden(); }); + +test.describe('Hands-free listening', () => { + async function startPlayback(page) { + await page.addInitScript(() => { + window.__ms = { actions: {}, positions: 0 }; + const ms = navigator.mediaSession; + const set = ms.setActionHandler.bind(ms); + ms.setActionHandler = (name, fn) => { window.__ms.actions[name] = fn; return set(name, fn); }; + const pos = ms.setPositionState.bind(ms); + ms.setPositionState = (s) => { window.__ms.positions++; return pos(s); }; + window.__wl = { requested: 0, released: 0 }; + Object.defineProperty(navigator, 'wakeLock', { configurable: true, value: { + request: async () => { window.__wl.requested++; return { addEventListener() {}, release: async () => { window.__wl.released++; } }; } } }); + }); + await mockBackend(page, { articleStatus: 'approved', hasAudio: true }); + await page.goto('/', { waitUntil: 'domcontentloaded' }); + await page.waitForSelector('#app-skeleton', { state: 'detached' }); + await navTo(page, 's-magazine'); + await page.locator('#mag-library-list .mag-card').first().click(); + await page.locator('.mag-toc-row.is-approved').getByRole('button', { name: /Listen|Continue/ }).click(); + await expect.poll(() => page.evaluate(() => { const a = document.getElementById('mag-playlist-audio'); return !!a && !a.paused; })).toBe(true); + } + + test('lock screen shows the article and offers play, pause, next, previous and seek', async ({ page }) => { + await startPlayback(page); + const meta = await page.evaluate(() => ({ title: navigator.mediaSession.metadata?.title, album: navigator.mediaSession.metadata?.album, + state: navigator.mediaSession.playbackState, actions: Object.keys(window.__ms.actions).sort() })); + expect(meta.title).toBe('The Future of Batteries'); + expect(meta.album).toContain('Scientific American'); + expect(meta.state).toBe('playing'); + expect(meta.actions).toEqual(['nexttrack', 'pause', 'play', 'previoustrack', 'seekbackward', 'seekforward', 'seekto', 'stop']); + await expect.poll(() => page.evaluate(() => window.__ms.positions)).toBeGreaterThan(0); + }); + + test('pause and play from the lock screen control the real player', async ({ page }) => { + await startPlayback(page); + await page.evaluate(() => window.__ms.actions.pause()); + await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').paused)).toBe(true); + await expect.poll(() => page.evaluate(() => navigator.mediaSession.playbackState)).toBe('paused'); + await page.evaluate(() => window.__ms.actions.play()); + await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').paused)).toBe(false); + }); + + test('seekto and seekforward move the playhead', async ({ page }) => { + await startPlayback(page); + await page.evaluate(() => window.__ms.actions.seekto({ seekTime: 1.2 })); + await expect.poll(() => page.evaluate(() => document.getElementById('mag-playlist-audio').currentTime)).toBeGreaterThan(1); + }); + + test('the screen is kept awake only while driving mode is open and playing', async ({ page }) => { + await startPlayback(page); + expect(await page.evaluate(() => window.__wl.requested)).toBe(0); // reading view: no lock + await page.evaluate(() => window.magOpenDriving()); + await expect.poll(() => page.evaluate(() => window.__wl.requested)).toBe(1); + await page.evaluate(() => window.__ms.actions.pause()); // paused: let the screen sleep + await expect.poll(() => page.evaluate(() => window.__wl.released)).toBe(1); + await page.evaluate(() => window.__ms.actions.play()); + await expect.poll(() => page.evaluate(() => window.__wl.requested)).toBe(2); + await page.evaluate(() => window.magToggleDriving()); // leaving driving mode releases it + await expect.poll(() => page.evaluate(() => window.__wl.released)).toBe(2); + }); + + test('a refused wake lock never stops playback', async ({ page }) => { + await page.addInitScript(() => { + Object.defineProperty(navigator, 'wakeLock', { configurable: true, value: { request: async () => { throw new Error('denied'); } } }); + }); + await mockBackend(page, { articleStatus: 'approved', hasAudio: true }); + await page.goto('/', { waitUntil: 'domcontentloaded' }); + await page.waitForSelector('#app-skeleton', { state: 'detached' }); + await navTo(page, 's-magazine'); + await page.locator('#mag-library-list .mag-card').first().click(); + await page.locator('.mag-toc-row.is-approved').getByRole('button', { name: /Listen|Continue/ }).click(); + await expect.poll(() => page.evaluate(() => !document.getElementById('mag-playlist-audio').paused)).toBe(true); + await page.evaluate(() => window.magOpenDriving()); + await expect(page.locator('#mag-driving')).toBeVisible(); + await expect.poll(() => page.evaluate(() => !document.getElementById('mag-playlist-audio').paused)).toBe(true); + }); +});