fix(voices): robust LibriVox search and expose export_provenance_default
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
7a86e51faa
commit
c0a5d54763
@ -6,6 +6,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
||||
---
|
||||
|
||||
## [Unreleased]
|
||||
### Fixed
|
||||
- **LibriVox search returned 502 "unavailable" after 20-30 s.** Cause: the LibriVox API is erratic (10-40 s for any request, measured from host and container; egress/DNS/TLS fine) and the old call asked for `extended=1` (all chapters of 15 books) with a 30 s timeout, mapping every failure to one generic message. Now the list is light (no `extended`), the first chapter is fetched per book on demand (`GET /api/voice-sources/librivox/book/{id}`, UI "Load preview" / on import), timeout 25 s with one retry and backoff, failures cached 30 s, distinct messages (timeout / unreachable / HTTP status / unreadable) and a privacy-safe log line (exception class and query length only). Host/redirect rules unchanged.
|
||||
- **`export_provenance_default` missing from settings defaults.** It is now in the defaults (`true`), so `GET /api/settings` exposes it.
|
||||
### Added
|
||||
- **Audiobook emotion tests.** Playwright `tests/audiobook-emotion.spec.js` (mocked routes) covers span-cast emotion -> `instruct` in `/api/tts-preview` (dialogue only, English tag, narration empty) and the line cache key; `tests/test_tts_instruct_payload.py` covers the instruct passthrough.
|
||||
- **LibriVox and Wikimedia Commons voice sources.** New tab in Get Voices Online searches the official LibriVox feed and the Commons MediaWiki API through `GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` (fixed hosts, query cap 80, 10 results, 30 s timeout, 10 min cache, 1 s throttle, descriptive User-Agent, result URLs host-checked). Results show licence and a preview; "Import clip" trims (start/length) and stores provenance (LibriVox: public domain + check-the-reader note; Wikimedia: licence + attribution). `archive.org` and `upload.wikimedia.org` added to the audio proxy allowlist. Tests: `tests/test_voice_sources_new.py`, `tests/voice-sources-new.spec.js`. See `architecture/voice-sources.md`.
|
||||
|
||||
@ -4,7 +4,7 @@ Status: implemented. Code: `core/open_voice_sources.py`, routes in `routes/sourc
|
||||
|
||||
## Verified API facts (checked 2026-10-04, docs and live calls)
|
||||
LibriVox
|
||||
- `GET https://librivox.org/api/feed/audiobooks?title=^<q>&format=json&extended=1&limit=N`. A bare `title=` is an EXACT match (404 `{"error":...}` otherwise); the `^` prefix gives a prefix match. No language parameter: the language is filtered client-side on `language` ("English"). `extended=1` adds `sections[]` with `listen_url` (archive.org MP3, 64 kbps), `playtime` (s) and `readers[]`.
|
||||
- `GET https://librivox.org/api/feed/audiobooks?title=^<q>&format=json&limit=15` (light list, no `extended`; the API takes 10-40 s even so). Audio is resolved per book on demand: `?id=<n>&format=json&extended=1`, first section whose `listen_url` is on an allowed host, exposed as `GET /api/voice-sources/librivox/book/{id}` (`{audio_url, section, reader, duration}`). Upstream calls: timeout 25 s, one retry after 1.5 s, failures cached 30 s, 502 messages distinguish timeout / unreachable / HTTP status / unreadable; logs carry exception class and query length only. A bare `title=` is an EXACT match (404 `{"error":...}` otherwise); the `^` prefix gives a prefix match. No language parameter: the language is filtered client-side on `language` ("English"). `extended=1` adds `sections[]` with `listen_url` (archive.org MP3, 64 kbps), `playtime` (s) and `readers[]`.
|
||||
- The docs state no rate limit and no terms. The extended feed is slow (about 14 s for 5-25 books), so timeout is 30 s, limit 15, results cached 10 min, one upstream call per second.
|
||||
- Licence (librivox.org/pages/public-domain): recordings are donated to the public domain, use without permission or attribution allowed (credit appreciated). Caveats: guaranteed public domain only under US law; source texts are pre-1923 works; translations are separate works; readers are volunteers and the page of a project may show a reader-specific note, so the UI/provenance note tells the user to check the reader's declared licence. A reader's voice is still a person's voice: public domain is not consent for voice cloning of a recognisable person; the provenance basis is a record, not a legal clearance.
|
||||
- Audio is hosted on archive.org (`www.archive.org/download/...` redirects to `iaNNN.us.archive.org`).
|
||||
@ -22,7 +22,7 @@ Wikimedia Commons
|
||||
- Licence/attribution values sent by the browser at import are only length-capped, not re-fetched (single-user LAN tool).
|
||||
|
||||
## Endpoints
|
||||
`GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` : `q` required, max 80 chars, control chars stripped; `lang` 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: `{source, title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url, attribution}`. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server).
|
||||
`GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` : `q` required, max 80 chars, control chars stripped; `lang` 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: `{source, book_id (librivox), title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url (librivox: empty until resolved), attribution}`. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server).
|
||||
|
||||
`POST /api/quick-import-voice` accepts `source_kind` (`librivox`|`wikimedia`), `licence`, `attribution`, `page_url`, `start`, `length` (clip, 3-120 s, default 0/30). Provenance: LibriVox -> `public_domain`, source = book page, note says to check the reader's licence. Wikimedia -> `licensed` (or `public_domain` when licence says public domain/CC0), licence, note `Attribution: ...`; `meta.attribution` also stored.
|
||||
|
||||
|
||||
@ -348,6 +348,7 @@ def _load_settings() -> dict:
|
||||
"vision_ocr_model": _VISION_OCR_MODEL_DEFAULT,
|
||||
"vision_ocr_enabled": True,
|
||||
"inline_tags_enabled": False,
|
||||
"export_provenance_default": True,
|
||||
"vision_ocr_fallback_url": _VISION_OCR_FALLBACK_DEFAULT,
|
||||
"vision_ocr_fallback_model": _VISION_OCR_FALLBACK_MODEL_DEFAULT,
|
||||
"engine_local_urls": {},
|
||||
|
||||
@ -2,6 +2,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
@ -17,7 +19,10 @@ USER_AGENT = "TTSVoiceCreator/1.0 (self-hosted voice library; official-API clien
|
||||
MAX_Q = 80
|
||||
MAX_RESULTS = 10
|
||||
MAX_BYTES = 8 * 1024 * 1024
|
||||
TIMEOUT = 30 # the LibriVox extended feed takes ~14 s
|
||||
TIMEOUT = 25 # LibriVox is erratic: 10-40 s even for small feeds (measured 2026-10); one retry follows
|
||||
RETRIES = 1
|
||||
BACKOFF = 1.5
|
||||
NEG_TTL = 30 # seconds an upstream failure is remembered (stops hammering a slow API)
|
||||
CACHE_TTL = 600
|
||||
CACHE_MAX = 200
|
||||
MIN_INTERVAL = 1.0 # seconds between upstream calls per source (MediaWiki asks for serial requests)
|
||||
@ -37,6 +42,8 @@ _cache: dict[tuple, tuple[float, list]] = {}
|
||||
_last_call: dict[str, float] = {}
|
||||
_locks = {s: threading.Lock() for s in SOURCES}
|
||||
_cache_lock = threading.Lock()
|
||||
_neg: dict[tuple, tuple[float, str]] = {}
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SourceError(Exception):
|
||||
@ -76,35 +83,54 @@ def _text(value, limit=200) -> str:
|
||||
return _CTRL.sub(" ", html.unescape(_TAGS.sub("", str(value or "")))).strip()[:limit]
|
||||
|
||||
|
||||
def _fetch(url: str, params: dict, src: str) -> dict:
|
||||
r = requests.get(url, params=params, headers={"User-Agent": USER_AGENT, "Accept": "application/json"},
|
||||
timeout=(5, TIMEOUT), allow_redirects=False, stream=True)
|
||||
if r.status_code == 404 and src == "librivox":
|
||||
return {}
|
||||
r.raise_for_status()
|
||||
body = b""
|
||||
for chunk in r.iter_content(65536):
|
||||
body += chunk
|
||||
if len(body) > MAX_BYTES:
|
||||
raise SourceError("Upstream response too large", 502)
|
||||
return json.loads(body.decode("utf-8", "replace"))
|
||||
|
||||
|
||||
def _get_json(url: str, params: dict) -> dict:
|
||||
wait = 0.0
|
||||
src = "librivox" if url == LIBRIVOX_API else "wikimedia"
|
||||
with _locks[src]:
|
||||
wait = MIN_INTERVAL - (time.monotonic() - _last_call.get(src, 0.0))
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
_last_call[src] = time.monotonic()
|
||||
try:
|
||||
r = requests.get(url, params=params, headers={"User-Agent": USER_AGENT, "Accept": "application/json"},
|
||||
timeout=TIMEOUT, allow_redirects=False, stream=True)
|
||||
if r.status_code == 404 and src == "librivox":
|
||||
return {}
|
||||
r.raise_for_status()
|
||||
body = b""
|
||||
for chunk in r.iter_content(65536):
|
||||
body += chunk
|
||||
if len(body) > MAX_BYTES:
|
||||
raise SourceError("Upstream response too large", 502)
|
||||
import json
|
||||
return json.loads(body.decode("utf-8", "replace"))
|
||||
except SourceError:
|
||||
raise
|
||||
except Exception:
|
||||
raise SourceError(f"{src} is unavailable right now", 502)
|
||||
for attempt in range(RETRIES + 1):
|
||||
wait = MIN_INTERVAL - (time.monotonic() - _last_call.get(src, 0.0))
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
_last_call[src] = time.monotonic()
|
||||
try:
|
||||
return _fetch(url, params, src)
|
||||
except SourceError:
|
||||
raise
|
||||
except Exception as e:
|
||||
# privacy-safe: exception class and query length only, never the query text
|
||||
log.warning("%s fetch attempt %d failed: %s (query length %d)", src, attempt + 1, type(e).__name__,
|
||||
len(str(params.get("title") or params.get("gsrsearch") or "")))
|
||||
if attempt < RETRIES:
|
||||
time.sleep(BACKOFF * (2 ** attempt))
|
||||
continue
|
||||
if isinstance(e, requests.Timeout):
|
||||
msg = f"{src} did not answer in time (its API is slow); try again in a minute"
|
||||
elif isinstance(e, requests.HTTPError):
|
||||
code = getattr(e.response, "status_code", "?")
|
||||
msg = f"{src} refused the request (HTTP {code})"
|
||||
elif isinstance(e, requests.ConnectionError):
|
||||
msg = f"{src} is unreachable from this server"
|
||||
else:
|
||||
msg = f"{src} returned an unreadable answer"
|
||||
raise SourceError(msg, 502)
|
||||
|
||||
|
||||
def _librivox_items(q: str, lang: str) -> list[dict]:
|
||||
data = _get_json(LIBRIVOX_API, {"title": "^" + q, "format": "json", "extended": 1, "limit": 15}) # "^" = prefix match; bare title is exact
|
||||
# Light list (no extended=1: sections made the feed several times bigger and slower); audio is resolved per book.
|
||||
data = _get_json(LIBRIVOX_API, {"title": "^" + q, "format": "json", "limit": 15}) # "^" = prefix match; bare title is exact
|
||||
want = LANG_NAMES.get(lang, "").lower()
|
||||
out = []
|
||||
for b in (data.get("books") or []) if isinstance(data, dict) else []:
|
||||
@ -112,28 +138,49 @@ def _librivox_items(q: str, lang: str) -> list[dict]:
|
||||
continue
|
||||
if want and str(b.get("language") or "").lower() != want:
|
||||
continue
|
||||
secs = [s for s in (b.get("sections") or []) if isinstance(s, dict) and host_ok(s.get("listen_url") or "", AUDIO_HOSTS)]
|
||||
src_url = b.get("url_librivox") or ""
|
||||
if not secs or not host_ok(src_url, PAGE_HOSTS):
|
||||
src_url, bid = b.get("url_librivox") or "", str(b.get("id") or "")
|
||||
if not bid.isdigit() or len(bid) > 9 or not host_ok(src_url, PAGE_HOSTS):
|
||||
continue
|
||||
s = secs[0]
|
||||
readers = ", ".join(_text(r.get("display_name"), 60) for r in (s.get("readers") or []) if isinstance(r, dict))
|
||||
authors = ", ".join(_text(f"{a.get('first_name', '')} {a.get('last_name', '')}", 60) for a in (b.get("authors") or []) if isinstance(a, dict))
|
||||
try:
|
||||
dur = int(s.get("playtime") or 0)
|
||||
dur = int(b.get("totaltimesecs") or 0)
|
||||
except (TypeError, ValueError):
|
||||
dur = 0
|
||||
out.append({
|
||||
"source": "librivox", "title": _text(f"{b.get('title')} - {s.get('title')}"), "reader": readers, "author": authors,
|
||||
"source": "librivox", "book_id": bid, "title": _text(b.get("title")), "reader": "", "author": authors,
|
||||
"language": _text(b.get("language"), 40), "duration": dur,
|
||||
"licence": "Public domain (LibriVox, USA)", "licence_url": "https://librivox.org/pages/public-domain/",
|
||||
"source_url": src_url, "audio_url": s["listen_url"], "attribution": "",
|
||||
"source_url": src_url, "audio_url": "", "attribution": "",
|
||||
})
|
||||
if len(out) >= MAX_RESULTS:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def librivox_first_section(book_id: str) -> dict:
|
||||
"""First playable section of one book (title, reader, duration, host-checked audio_url)."""
|
||||
book_id = str(book_id or "")
|
||||
if not (book_id.isdigit() and len(book_id) <= 9):
|
||||
raise SourceError("invalid book id")
|
||||
key = ("librivox-book", book_id)
|
||||
cached = _cache_get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
data = _cached_call(key, lambda: _get_json(LIBRIVOX_API, {"id": book_id, "format": "json", "extended": 1}))
|
||||
for b in (data.get("books") or []) if isinstance(data, dict) else []:
|
||||
for s in (b.get("sections") or []) if isinstance(b, dict) else []:
|
||||
if isinstance(s, dict) and host_ok(s.get("listen_url") or "", AUDIO_HOSTS):
|
||||
readers = ", ".join(_text(r.get("display_name"), 60) for r in (s.get("readers") or []) if isinstance(r, dict))
|
||||
try:
|
||||
dur = int(s.get("playtime") or 0)
|
||||
except (TypeError, ValueError):
|
||||
dur = 0
|
||||
res = {"audio_url": s["listen_url"], "section": _text(s.get("title")), "reader": readers, "duration": dur}
|
||||
_cache_put(key, res)
|
||||
return res
|
||||
raise SourceError("no playable audio found for this book", 404)
|
||||
|
||||
|
||||
def _wikimedia_items(q: str) -> list[dict]:
|
||||
data = _get_json(COMMONS_API, {
|
||||
"action": "query", "format": "json", "generator": "search", "gsrnamespace": 6, "gsrlimit": 20,
|
||||
@ -163,21 +210,46 @@ def _wikimedia_items(q: str) -> list[dict]:
|
||||
return out
|
||||
|
||||
|
||||
def _cache_get(key):
|
||||
with _cache_lock:
|
||||
hit = _cache.get(key)
|
||||
if hit and time.monotonic() - hit[0] < CACHE_TTL:
|
||||
return hit[1]
|
||||
return None
|
||||
|
||||
|
||||
def _cache_put(key, value):
|
||||
with _cache_lock:
|
||||
if len(_cache) >= CACHE_MAX:
|
||||
_cache.pop(min(_cache, key=lambda k: _cache[k][0]))
|
||||
_cache[key] = (time.monotonic(), value)
|
||||
|
||||
|
||||
def _cached_call(key, fn):
|
||||
"""Run fn; a SourceError is remembered for NEG_TTL seconds and re-raised instantly meanwhile."""
|
||||
with _cache_lock:
|
||||
bad = _neg.get(key)
|
||||
if bad and time.monotonic() - bad[0] < NEG_TTL:
|
||||
raise SourceError(bad[1], 502)
|
||||
try:
|
||||
return fn()
|
||||
except SourceError as e:
|
||||
if e.status >= 500:
|
||||
with _cache_lock:
|
||||
_neg[key] = (time.monotonic(), str(e))
|
||||
raise
|
||||
|
||||
|
||||
def search(source: str, q: str, lang: str = "") -> list[dict]:
|
||||
if source not in SOURCES:
|
||||
raise SourceError("unknown source")
|
||||
q, lang = clean_query(q), clean_lang(lang)
|
||||
key = (source, q.lower(), lang if source == "librivox" else "")
|
||||
now = time.monotonic()
|
||||
with _cache_lock:
|
||||
hit = _cache.get(key)
|
||||
if hit and now - hit[0] < CACHE_TTL:
|
||||
return hit[1]
|
||||
items = _librivox_items(q, lang) if source == "librivox" else _wikimedia_items(q)
|
||||
with _cache_lock:
|
||||
if len(_cache) >= CACHE_MAX:
|
||||
_cache.pop(min(_cache, key=lambda k: _cache[k][0]))
|
||||
_cache[key] = (now, items)
|
||||
hit = _cache_get(key)
|
||||
if hit is not None:
|
||||
return hit
|
||||
items = _cached_call(key, lambda: _librivox_items(q, lang) if source == "librivox" else _wikimedia_items(q))
|
||||
_cache_put(key, items)
|
||||
return items
|
||||
|
||||
|
||||
|
||||
@ -535,6 +535,15 @@ async def voice_sources_custom(request: Request):
|
||||
return _voice_sources_payload(data.get("urls") if isinstance(data, dict) else None)
|
||||
|
||||
|
||||
@router.get("/api/voice-sources/librivox/book/{book_id}")
|
||||
async def librivox_book_audio(book_id: str):
|
||||
from core import open_voice_sources as ovs
|
||||
try:
|
||||
return await asyncio.to_thread(ovs.librivox_first_section, book_id)
|
||||
except ovs.SourceError as e:
|
||||
raise HTTPException(e.status, str(e))
|
||||
|
||||
|
||||
@router.get("/api/voice-sources/{source}")
|
||||
async def open_voice_source_search(source: str, q: str = "", lang: str = ""):
|
||||
from core import open_voice_sources as ovs
|
||||
|
||||
4
static/dist/main.min.js
vendored
4
static/dist/main.min.js
vendored
File diff suppressed because one or more lines are too long
@ -199,7 +199,9 @@ function renderOpenSources() {
|
||||
$('os-list').innerHTML = _osItems.map((it, i) => `<div class="voice-source-card">
|
||||
<div class="voice-source-title"><strong>${escHtml(it.title)}</strong><span>${escHtml([it.reader, it.language, osDuration(it.duration)].filter(Boolean).join(' · '))}</span></div>
|
||||
<div class="voice-source-desc">Licence: ${it.licence_url ? `<a href="${escHtml(it.licence_url)}" target="_blank" rel="noopener">${escHtml(it.licence)}</a>` : escHtml(it.licence)}</div>
|
||||
<audio controls preload="none" aria-label="Preview ${escHtml(it.title)}" src="/api/proxy-audio?url=${encodeURIComponent(it.audio_url)}"></audio>
|
||||
${it.audio_url
|
||||
? `<audio controls preload="none" aria-label="Preview ${escHtml(it.title)}" src="/api/proxy-audio?url=${encodeURIComponent(it.audio_url)}"></audio>`
|
||||
: `<button class="btn-secondary os-load" type="button" data-i="${i}">Load preview</button>`}
|
||||
<div class="voice-source-actions">
|
||||
<a class="btn-secondary" href="${escHtml(it.source_url)}" target="_blank" rel="noopener">Source page</a>
|
||||
<button class="btn-primary os-import" type="button" data-i="${i}">Import clip</button>
|
||||
@ -223,8 +225,19 @@ async function searchOpenSources(ev) {
|
||||
_osItems = []; renderOpenSources(); st.textContent = 'Search failed: ' + e.message;
|
||||
} finally { btn.disabled = false; }
|
||||
}
|
||||
// LibriVox list items carry only book_id; the first chapter's audio is fetched on demand.
|
||||
async function resolveOpenSource(it) {
|
||||
if (it.audio_url || !it.book_id) return;
|
||||
const r = await fetch(`/api/voice-sources/librivox/book/${encodeURIComponent(it.book_id)}`);
|
||||
const d = await r.json().catch(() => ({}));
|
||||
if (!r.ok) throw new Error(d.detail || r.statusText);
|
||||
Object.assign(it, {audio_url: d.audio_url, reader: d.reader || it.reader, duration: d.duration || it.duration,
|
||||
title: d.section ? `${it.title} - ${d.section}` : it.title});
|
||||
}
|
||||
async function importOpenSource(it) {
|
||||
const st = $('os-status');
|
||||
st.textContent = 'Looking up audio...';
|
||||
try { await resolveOpenSource(it); } catch (e) { st.textContent = 'Lookup failed: ' + e.message; return; }
|
||||
const lang = sourceLanguageCode(it.language || $('os-lang').value || 'English');
|
||||
const body = {
|
||||
audio_url: it.audio_url, voice_id: suggestedVoiceIdFromSourceName(it.title, lang, 'N'), transcript: '', lang,
|
||||
@ -243,7 +256,14 @@ async function importOpenSource(it) {
|
||||
} catch (e) { st.textContent = 'Import failed: ' + e.message; toast('Import failed: ' + e.message, 'error'); }
|
||||
}
|
||||
$('os-form')?.addEventListener('submit', searchOpenSources);
|
||||
$('os-list')?.addEventListener('click', e => {
|
||||
$('os-list')?.addEventListener('click', async e => {
|
||||
const l = e.target.closest('.os-load');
|
||||
if (l && _osItems[+l.dataset.i]) {
|
||||
l.disabled = true; $('os-status').textContent = 'Loading preview...';
|
||||
try { await resolveOpenSource(_osItems[+l.dataset.i]); renderOpenSources(); $('os-status').textContent = 'Preview ready.'; }
|
||||
catch (err) { l.disabled = false; $('os-status').textContent = 'Preview failed: ' + err.message; }
|
||||
return;
|
||||
}
|
||||
const b = e.target.closest('.os-import');
|
||||
if (b && _osItems[+b.dataset.i]) importOpenSource(_osItems[+b.dataset.i]);
|
||||
});
|
||||
|
||||
@ -13,10 +13,12 @@ from core import open_voice_sources as ovs
|
||||
import routes.sources as src
|
||||
|
||||
|
||||
def book(i=1, lang="English", url="https://www.archive.org/download/x/a.mp3"):
|
||||
return {"title": f"Book {i}", "language": lang, "url_librivox": f"https://librivox.org/b{i}/",
|
||||
"authors": [{"first_name": "Jane", "last_name": "Austen"}],
|
||||
"sections": [{"title": "Ch 1", "listen_url": url, "playtime": "600", "readers": [{"display_name": "Ann"}]}]}
|
||||
def book(i=1, lang="English", url="https://www.archive.org/download/x/a.mp3", sections=False):
|
||||
b = {"id": str(i), "title": f"Book {i}", "language": lang, "url_librivox": f"https://librivox.org/b{i}/",
|
||||
"totaltimesecs": 600, "authors": [{"first_name": "Jane", "last_name": "Austen"}]}
|
||||
if sections:
|
||||
b["sections"] = [{"title": "Ch 1", "listen_url": url, "playtime": "600", "readers": [{"display_name": "Ann"}]}]
|
||||
return b
|
||||
|
||||
|
||||
def commons(**over):
|
||||
@ -56,16 +58,31 @@ class ValidationTests(Base):
|
||||
|
||||
|
||||
class MappingTests(Base):
|
||||
def test_librivox_mapping_filter_cap_and_bad_hosts(self):
|
||||
books = [book(i) for i in range(15)] + [book(99, url="https://evil.example/a.mp3"), book(98, lang="German")]
|
||||
def test_librivox_list_is_light_filtered_and_capped(self):
|
||||
books = [book(i) for i in range(15)] + [book(98, lang="German"), dict(book(97), url_librivox="https://evil.example/x/"), dict(book(96), id="x;1")]
|
||||
with mock.patch.object(ovs, "_get_json", return_value={"books": books}) as g:
|
||||
items = ovs.search("librivox", "pride", "en")
|
||||
self.assertEqual(len(items), ovs.MAX_RESULTS)
|
||||
self.assertEqual(g.call_args[0][0], ovs.LIBRIVOX_API)
|
||||
self.assertNotIn("extended", g.call_args[0][1])
|
||||
it = items[0]
|
||||
self.assertEqual((it["reader"], it["author"], it["language"], it["duration"]), ("Ann", "Jane Austen", "English", 600))
|
||||
self.assertEqual((it["author"], it["language"], it["duration"], it["audio_url"], it["book_id"]), ("Jane Austen", "English", 600, "", "0"))
|
||||
self.assertTrue(it["licence_url"].startswith("https://librivox.org/"))
|
||||
self.assertTrue(all("evil" not in i["audio_url"] and i["language"] == "English" for i in items))
|
||||
self.assertTrue(all(i["language"] == "English" and "evil" not in i["source_url"] for i in items))
|
||||
|
||||
def test_librivox_first_section(self):
|
||||
bad = book(2, url="https://evil.example/a.mp3", sections=True)
|
||||
bad["sections"].append({"title": "Ch 2", "listen_url": "https://ia1.us.archive.org/b.mp3", "playtime": "5", "readers": []})
|
||||
with mock.patch.object(ovs, "_get_json", return_value={"books": [bad]}) as g:
|
||||
r = ovs.librivox_first_section("2")
|
||||
self.assertEqual(g.call_args[0][1]["extended"], 1)
|
||||
self.assertEqual((r["audio_url"], r["section"], r["duration"]), ("https://ia1.us.archive.org/b.mp3", "Ch 2", 5))
|
||||
for badid in ("", "abc", "1;2", "1" * 10):
|
||||
with self.assertRaises(ovs.SourceError):
|
||||
ovs.librivox_first_section(badid)
|
||||
with mock.patch.object(ovs, "_get_json", return_value={"books": [book(3, url="https://evil.example/a")]}):
|
||||
with self.assertRaises(ovs.SourceError) as cm:
|
||||
ovs.librivox_first_section("3")
|
||||
self.assertEqual(cm.exception.status, 404)
|
||||
|
||||
def test_wikimedia_mapping(self):
|
||||
with mock.patch.object(ovs, "_get_json", return_value=commons()):
|
||||
@ -117,6 +134,41 @@ class CacheThrottleTests(Base):
|
||||
self.assertNotIn("secret", str(cm.exception))
|
||||
|
||||
|
||||
class RetryErrorTests(Base):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
ovs._neg.clear()
|
||||
|
||||
def test_timeout_retried_once_then_clear_message_and_negative_cache(self):
|
||||
with mock.patch.object(ovs.requests, "get", side_effect=ovs.requests.Timeout("secret")) as g:
|
||||
with self.assertRaises(ovs.SourceError) as cm:
|
||||
ovs.search("librivox", "Faust")
|
||||
self.assertEqual(g.call_count, 2)
|
||||
self.assertIn("in time", str(cm.exception))
|
||||
self.assertEqual(cm.exception.status, 502)
|
||||
with self.assertRaises(ovs.SourceError): # remembered: no further upstream call
|
||||
ovs.search("librivox", "Faust")
|
||||
self.assertEqual(g.call_count, 2)
|
||||
|
||||
def test_retry_recovers_and_messages_differ(self):
|
||||
resp = mock.Mock(status_code=200)
|
||||
resp.raise_for_status.return_value = None
|
||||
resp.iter_content.return_value = [b'{"books": []}']
|
||||
with mock.patch.object(ovs.requests, "get", side_effect=[ovs.requests.ConnectionError("x"), resp]):
|
||||
self.assertEqual(ovs._get_json(ovs.LIBRIVOX_API, {}), {"books": []})
|
||||
for exc, text in ((ovs.requests.ConnectionError("x"), "unreachable"), (ValueError("x"), "unreadable")):
|
||||
with mock.patch.object(ovs.requests, "get", side_effect=exc):
|
||||
with self.assertRaises(ovs.SourceError) as cm:
|
||||
ovs._get_json(ovs.LIBRIVOX_API, {})
|
||||
self.assertIn(text, str(cm.exception))
|
||||
|
||||
|
||||
class SettingsTests(unittest.TestCase):
|
||||
def test_export_provenance_default_exposed(self):
|
||||
from core import config
|
||||
self.assertIs(config._load_settings().get("export_provenance_default"), True)
|
||||
|
||||
|
||||
class RouteTests(Base):
|
||||
def test_search_route_errors_and_allowlist(self):
|
||||
with self.assertRaises(src.HTTPException) as cm:
|
||||
|
||||
@ -41,3 +41,26 @@ test('search error is shown in the status region', async ({ page }) => {
|
||||
await page.evaluate(() => document.getElementById('os-form').requestSubmit());
|
||||
await expect(page.locator('#os-status')).toContainText('unavailable');
|
||||
});
|
||||
|
||||
test('librivox list without audio: preview and import resolve the chapter on demand', async ({ page }) => {
|
||||
let imported = null, resolved = 0;
|
||||
const light = [{ ...ITEMS[0], book_id: '855', audio_url: '', reader: '' }];
|
||||
await page.route('**/api/voice-sources/librivox/book/855', r => { resolved++; return r.fulfill({ json: { audio_url: 'https://www.archive.org/download/x/a.mp3', section: 'Ch 1', reader: 'Ann', duration: 600 } }); });
|
||||
await page.route('**/api/voice-sources/librivox?*', r => r.fulfill({ json: { source: 'librivox', items: light } }));
|
||||
await page.route('**/api/proxy-audio*', r => r.fulfill({ body: '', contentType: 'audio/mpeg' }));
|
||||
await page.route('**/api/quick-import-voice', r => { imported = r.request().postDataJSON(); return r.fulfill({ json: { voice_id: 'EN_N_Pride' } }); });
|
||||
await page.route('**/api/voices*', r => r.fulfill({ json: [] }));
|
||||
await page.route('**/api/settings*', r => r.fulfill({ json: {} }));
|
||||
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForFunction(() => typeof window.navTo === 'function' && !!document.getElementById('os-form'), null, { timeout: 15_000 });
|
||||
await page.evaluate(() => window.navTo('s-studio'));
|
||||
await page.locator('.gvo-tab[data-src="open"]').click();
|
||||
await page.fill('#os-q', 'pride');
|
||||
await page.press('#os-q', 'Enter');
|
||||
await page.locator('.os-load').click();
|
||||
await expect(page.locator('#os-list audio')).toHaveCount(1);
|
||||
await page.locator('.os-import').click();
|
||||
await expect(page.locator('#os-status')).toContainText('Saved as EN_N_Pride');
|
||||
expect(resolved).toBe(1);
|
||||
expect(imported.audio_url).toContain('archive.org');
|
||||
});
|
||||
|
||||
Loading…
Reference in New Issue
Block a user