diff --git a/CHANGELOG.md b/CHANGELOG.md index fb49c41..51cfa54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi --- ## [Unreleased] +### Fixed +- **LibriVox search returned 502 "unavailable" after 20-30 s.** Cause: the LibriVox API is erratic (10-40 s for any request, measured from host and container; egress/DNS/TLS fine) and the old call asked for `extended=1` (all chapters of 15 books) with a 30 s timeout, mapping every failure to one generic message. Now the list is light (no `extended`), the first chapter is fetched per book on demand (`GET /api/voice-sources/librivox/book/{id}`, UI "Load preview" / on import), timeout 25 s with one retry and backoff, failures cached 30 s, distinct messages (timeout / unreachable / HTTP status / unreadable) and a privacy-safe log line (exception class and query length only). Host/redirect rules unchanged. +- **`export_provenance_default` missing from settings defaults.** It is now in the defaults (`true`), so `GET /api/settings` exposes it. ### Added - **Audiobook emotion tests.** Playwright `tests/audiobook-emotion.spec.js` (mocked routes) covers span-cast emotion -> `instruct` in `/api/tts-preview` (dialogue only, English tag, narration empty) and the line cache key; `tests/test_tts_instruct_payload.py` covers the instruct passthrough. - **LibriVox and Wikimedia Commons voice sources.** New tab in Get Voices Online searches the official LibriVox feed and the Commons MediaWiki API through `GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` (fixed hosts, query cap 80, 10 results, 30 s timeout, 10 min cache, 1 s throttle, descriptive User-Agent, result URLs host-checked). Results show licence and a preview; "Import clip" trims (start/length) and stores provenance (LibriVox: public domain + check-the-reader note; Wikimedia: licence + attribution). `archive.org` and `upload.wikimedia.org` added to the audio proxy allowlist. Tests: `tests/test_voice_sources_new.py`, `tests/voice-sources-new.spec.js`. See `architecture/voice-sources.md`. diff --git a/architecture/voice-sources.md b/architecture/voice-sources.md index aca203d..739ee65 100644 --- a/architecture/voice-sources.md +++ b/architecture/voice-sources.md @@ -4,7 +4,7 @@ Status: implemented. Code: `core/open_voice_sources.py`, routes in `routes/sourc ## Verified API facts (checked 2026-10-04, docs and live calls) LibriVox -- `GET https://librivox.org/api/feed/audiobooks?title=^&format=json&extended=1&limit=N`. A bare `title=` is an EXACT match (404 `{"error":...}` otherwise); the `^` prefix gives a prefix match. No language parameter: the language is filtered client-side on `language` ("English"). `extended=1` adds `sections[]` with `listen_url` (archive.org MP3, 64 kbps), `playtime` (s) and `readers[]`. +- `GET https://librivox.org/api/feed/audiobooks?title=^&format=json&limit=15` (light list, no `extended`; the API takes 10-40 s even so). Audio is resolved per book on demand: `?id=&format=json&extended=1`, first section whose `listen_url` is on an allowed host, exposed as `GET /api/voice-sources/librivox/book/{id}` (`{audio_url, section, reader, duration}`). Upstream calls: timeout 25 s, one retry after 1.5 s, failures cached 30 s, 502 messages distinguish timeout / unreachable / HTTP status / unreadable; logs carry exception class and query length only. A bare `title=` is an EXACT match (404 `{"error":...}` otherwise); the `^` prefix gives a prefix match. No language parameter: the language is filtered client-side on `language` ("English"). `extended=1` adds `sections[]` with `listen_url` (archive.org MP3, 64 kbps), `playtime` (s) and `readers[]`. - The docs state no rate limit and no terms. The extended feed is slow (about 14 s for 5-25 books), so timeout is 30 s, limit 15, results cached 10 min, one upstream call per second. - Licence (librivox.org/pages/public-domain): recordings are donated to the public domain, use without permission or attribution allowed (credit appreciated). Caveats: guaranteed public domain only under US law; source texts are pre-1923 works; translations are separate works; readers are volunteers and the page of a project may show a reader-specific note, so the UI/provenance note tells the user to check the reader's declared licence. A reader's voice is still a person's voice: public domain is not consent for voice cloning of a recognisable person; the provenance basis is a record, not a legal clearance. - Audio is hosted on archive.org (`www.archive.org/download/...` redirects to `iaNNN.us.archive.org`). @@ -22,7 +22,7 @@ Wikimedia Commons - Licence/attribution values sent by the browser at import are only length-capped, not re-fetched (single-user LAN tool). ## Endpoints -`GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` : `q` required, max 80 chars, control chars stripped; `lang` 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: `{source, title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url, attribution}`. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server). +`GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` : `q` required, max 80 chars, control chars stripped; `lang` 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: `{source, book_id (librivox), title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url (librivox: empty until resolved), attribution}`. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server). `POST /api/quick-import-voice` accepts `source_kind` (`librivox`|`wikimedia`), `licence`, `attribution`, `page_url`, `start`, `length` (clip, 3-120 s, default 0/30). Provenance: LibriVox -> `public_domain`, source = book page, note says to check the reader's licence. Wikimedia -> `licensed` (or `public_domain` when licence says public domain/CC0), licence, note `Attribution: ...`; `meta.attribution` also stored. diff --git a/core/config.py b/core/config.py index 327627a..6489ab7 100644 --- a/core/config.py +++ b/core/config.py @@ -348,6 +348,7 @@ def _load_settings() -> dict: "vision_ocr_model": _VISION_OCR_MODEL_DEFAULT, "vision_ocr_enabled": True, "inline_tags_enabled": False, + "export_provenance_default": True, "vision_ocr_fallback_url": _VISION_OCR_FALLBACK_DEFAULT, "vision_ocr_fallback_model": _VISION_OCR_FALLBACK_MODEL_DEFAULT, "engine_local_urls": {}, diff --git a/core/open_voice_sources.py b/core/open_voice_sources.py index fe4ae83..b21ba62 100644 --- a/core/open_voice_sources.py +++ b/core/open_voice_sources.py @@ -2,6 +2,8 @@ from __future__ import annotations import html +import json +import logging import re import threading import time @@ -17,7 +19,10 @@ USER_AGENT = "TTSVoiceCreator/1.0 (self-hosted voice library; official-API clien MAX_Q = 80 MAX_RESULTS = 10 MAX_BYTES = 8 * 1024 * 1024 -TIMEOUT = 30 # the LibriVox extended feed takes ~14 s +TIMEOUT = 25 # LibriVox is erratic: 10-40 s even for small feeds (measured 2026-10); one retry follows +RETRIES = 1 +BACKOFF = 1.5 +NEG_TTL = 30 # seconds an upstream failure is remembered (stops hammering a slow API) CACHE_TTL = 600 CACHE_MAX = 200 MIN_INTERVAL = 1.0 # seconds between upstream calls per source (MediaWiki asks for serial requests) @@ -37,6 +42,8 @@ _cache: dict[tuple, tuple[float, list]] = {} _last_call: dict[str, float] = {} _locks = {s: threading.Lock() for s in SOURCES} _cache_lock = threading.Lock() +_neg: dict[tuple, tuple[float, str]] = {} +log = logging.getLogger(__name__) class SourceError(Exception): @@ -76,35 +83,54 @@ def _text(value, limit=200) -> str: return _CTRL.sub(" ", html.unescape(_TAGS.sub("", str(value or "")))).strip()[:limit] +def _fetch(url: str, params: dict, src: str) -> dict: + r = requests.get(url, params=params, headers={"User-Agent": USER_AGENT, "Accept": "application/json"}, + timeout=(5, TIMEOUT), allow_redirects=False, stream=True) + if r.status_code == 404 and src == "librivox": + return {} + r.raise_for_status() + body = b"" + for chunk in r.iter_content(65536): + body += chunk + if len(body) > MAX_BYTES: + raise SourceError("Upstream response too large", 502) + return json.loads(body.decode("utf-8", "replace")) + + def _get_json(url: str, params: dict) -> dict: - wait = 0.0 src = "librivox" if url == LIBRIVOX_API else "wikimedia" with _locks[src]: - wait = MIN_INTERVAL - (time.monotonic() - _last_call.get(src, 0.0)) - if wait > 0: - time.sleep(wait) - _last_call[src] = time.monotonic() - try: - r = requests.get(url, params=params, headers={"User-Agent": USER_AGENT, "Accept": "application/json"}, - timeout=TIMEOUT, allow_redirects=False, stream=True) - if r.status_code == 404 and src == "librivox": - return {} - r.raise_for_status() - body = b"" - for chunk in r.iter_content(65536): - body += chunk - if len(body) > MAX_BYTES: - raise SourceError("Upstream response too large", 502) - import json - return json.loads(body.decode("utf-8", "replace")) - except SourceError: - raise - except Exception: - raise SourceError(f"{src} is unavailable right now", 502) + for attempt in range(RETRIES + 1): + wait = MIN_INTERVAL - (time.monotonic() - _last_call.get(src, 0.0)) + if wait > 0: + time.sleep(wait) + _last_call[src] = time.monotonic() + try: + return _fetch(url, params, src) + except SourceError: + raise + except Exception as e: + # privacy-safe: exception class and query length only, never the query text + log.warning("%s fetch attempt %d failed: %s (query length %d)", src, attempt + 1, type(e).__name__, + len(str(params.get("title") or params.get("gsrsearch") or ""))) + if attempt < RETRIES: + time.sleep(BACKOFF * (2 ** attempt)) + continue + if isinstance(e, requests.Timeout): + msg = f"{src} did not answer in time (its API is slow); try again in a minute" + elif isinstance(e, requests.HTTPError): + code = getattr(e.response, "status_code", "?") + msg = f"{src} refused the request (HTTP {code})" + elif isinstance(e, requests.ConnectionError): + msg = f"{src} is unreachable from this server" + else: + msg = f"{src} returned an unreadable answer" + raise SourceError(msg, 502) def _librivox_items(q: str, lang: str) -> list[dict]: - data = _get_json(LIBRIVOX_API, {"title": "^" + q, "format": "json", "extended": 1, "limit": 15}) # "^" = prefix match; bare title is exact + # Light list (no extended=1: sections made the feed several times bigger and slower); audio is resolved per book. + data = _get_json(LIBRIVOX_API, {"title": "^" + q, "format": "json", "limit": 15}) # "^" = prefix match; bare title is exact want = LANG_NAMES.get(lang, "").lower() out = [] for b in (data.get("books") or []) if isinstance(data, dict) else []: @@ -112,28 +138,49 @@ def _librivox_items(q: str, lang: str) -> list[dict]: continue if want and str(b.get("language") or "").lower() != want: continue - secs = [s for s in (b.get("sections") or []) if isinstance(s, dict) and host_ok(s.get("listen_url") or "", AUDIO_HOSTS)] - src_url = b.get("url_librivox") or "" - if not secs or not host_ok(src_url, PAGE_HOSTS): + src_url, bid = b.get("url_librivox") or "", str(b.get("id") or "") + if not bid.isdigit() or len(bid) > 9 or not host_ok(src_url, PAGE_HOSTS): continue - s = secs[0] - readers = ", ".join(_text(r.get("display_name"), 60) for r in (s.get("readers") or []) if isinstance(r, dict)) authors = ", ".join(_text(f"{a.get('first_name', '')} {a.get('last_name', '')}", 60) for a in (b.get("authors") or []) if isinstance(a, dict)) try: - dur = int(s.get("playtime") or 0) + dur = int(b.get("totaltimesecs") or 0) except (TypeError, ValueError): dur = 0 out.append({ - "source": "librivox", "title": _text(f"{b.get('title')} - {s.get('title')}"), "reader": readers, "author": authors, + "source": "librivox", "book_id": bid, "title": _text(b.get("title")), "reader": "", "author": authors, "language": _text(b.get("language"), 40), "duration": dur, "licence": "Public domain (LibriVox, USA)", "licence_url": "https://librivox.org/pages/public-domain/", - "source_url": src_url, "audio_url": s["listen_url"], "attribution": "", + "source_url": src_url, "audio_url": "", "attribution": "", }) if len(out) >= MAX_RESULTS: break return out +def librivox_first_section(book_id: str) -> dict: + """First playable section of one book (title, reader, duration, host-checked audio_url).""" + book_id = str(book_id or "") + if not (book_id.isdigit() and len(book_id) <= 9): + raise SourceError("invalid book id") + key = ("librivox-book", book_id) + cached = _cache_get(key) + if cached is not None: + return cached + data = _cached_call(key, lambda: _get_json(LIBRIVOX_API, {"id": book_id, "format": "json", "extended": 1})) + for b in (data.get("books") or []) if isinstance(data, dict) else []: + for s in (b.get("sections") or []) if isinstance(b, dict) else []: + if isinstance(s, dict) and host_ok(s.get("listen_url") or "", AUDIO_HOSTS): + readers = ", ".join(_text(r.get("display_name"), 60) for r in (s.get("readers") or []) if isinstance(r, dict)) + try: + dur = int(s.get("playtime") or 0) + except (TypeError, ValueError): + dur = 0 + res = {"audio_url": s["listen_url"], "section": _text(s.get("title")), "reader": readers, "duration": dur} + _cache_put(key, res) + return res + raise SourceError("no playable audio found for this book", 404) + + def _wikimedia_items(q: str) -> list[dict]: data = _get_json(COMMONS_API, { "action": "query", "format": "json", "generator": "search", "gsrnamespace": 6, "gsrlimit": 20, @@ -163,21 +210,46 @@ def _wikimedia_items(q: str) -> list[dict]: return out +def _cache_get(key): + with _cache_lock: + hit = _cache.get(key) + if hit and time.monotonic() - hit[0] < CACHE_TTL: + return hit[1] + return None + + +def _cache_put(key, value): + with _cache_lock: + if len(_cache) >= CACHE_MAX: + _cache.pop(min(_cache, key=lambda k: _cache[k][0])) + _cache[key] = (time.monotonic(), value) + + +def _cached_call(key, fn): + """Run fn; a SourceError is remembered for NEG_TTL seconds and re-raised instantly meanwhile.""" + with _cache_lock: + bad = _neg.get(key) + if bad and time.monotonic() - bad[0] < NEG_TTL: + raise SourceError(bad[1], 502) + try: + return fn() + except SourceError as e: + if e.status >= 500: + with _cache_lock: + _neg[key] = (time.monotonic(), str(e)) + raise + + def search(source: str, q: str, lang: str = "") -> list[dict]: if source not in SOURCES: raise SourceError("unknown source") q, lang = clean_query(q), clean_lang(lang) key = (source, q.lower(), lang if source == "librivox" else "") - now = time.monotonic() - with _cache_lock: - hit = _cache.get(key) - if hit and now - hit[0] < CACHE_TTL: - return hit[1] - items = _librivox_items(q, lang) if source == "librivox" else _wikimedia_items(q) - with _cache_lock: - if len(_cache) >= CACHE_MAX: - _cache.pop(min(_cache, key=lambda k: _cache[k][0])) - _cache[key] = (now, items) + hit = _cache_get(key) + if hit is not None: + return hit + items = _cached_call(key, lambda: _librivox_items(q, lang) if source == "librivox" else _wikimedia_items(q)) + _cache_put(key, items) return items diff --git a/routes/sources.py b/routes/sources.py index 4ba6ba0..282f185 100644 --- a/routes/sources.py +++ b/routes/sources.py @@ -535,6 +535,15 @@ async def voice_sources_custom(request: Request): return _voice_sources_payload(data.get("urls") if isinstance(data, dict) else None) +@router.get("/api/voice-sources/librivox/book/{book_id}") +async def librivox_book_audio(book_id: str): + from core import open_voice_sources as ovs + try: + return await asyncio.to_thread(ovs.librivox_first_section, book_id) + except ovs.SourceError as e: + raise HTTPException(e.status, str(e)) + + @router.get("/api/voice-sources/{source}") async def open_voice_source_search(source: str, q: str = "", lang: str = ""): from core import open_voice_sources as ovs diff --git a/static/dist/main.min.js b/static/dist/main.min.js index 9c599cc..bec0fe1 100644 --- a/static/dist/main.min.js +++ b/static/dist/main.min.js @@ -205,11 +205,11 @@ `}).join("")+(more>0?`

${more} more matches. Narrow the search or filters to see them.

`:"")}async function loadGetVoices(force=!1){if(initGetVoiceSourcesEditor(),_voiceSourcePayload&&!force){renderGetVoices();return}const urls=getEditableVoiceSourceUrls(),list=$("getvoices-list");if(!urls.length){list&&(list.innerHTML='

Add at least one source URL, then scrape again.

'),$("getvoices-status").textContent="No sources.";return}list&&(list.innerHTML=loadingMarkup("Scraping voice sources",`Fetching ${urls.length} source${urls.length===1?"":"s"} from the editable list.`,6)),$("getvoices-status").textContent="Scraping...",$("getvoices-refresh-btn").disabled=!0;try{const r=await fetch("/api/voice-sources",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({urls})});if(!r.ok){const text=await r.text().catch(()=>"");let message=r.statusText||`HTTP ${r.status}`;try{message=JSON.parse(text).detail||message}catch{text&&(message=text.slice(0,160))}throw new Error(message)}_voiceSourcePayload=await r.json(),renderGetVoices();const errors=(_voiceSourcePayload.errors||[]).length;$("getvoices-status").textContent=`${_voiceSourcePayload.total||0} found${errors?`, ${errors} source errors`:""}`}catch(e){list&&(list.innerHTML=`

Scrape failed: ${escHtml(e.message)}

Check that the TTS Voice Creator backend is restarted and that at least one source URL is reachable.

`),$("getvoices-status").textContent="Scrape failed",toast("Voice source scrape failed: "+e.message,"error")}finally{$("getvoices-refresh-btn").disabled=!1}}["getvoices-search","getvoices-source-filter","getvoices-language-filter","getvoices-gender-filter","getvoices-filetype-filter","getvoices-direct-only"].forEach(id=>{const el=$(id);el&&el.addEventListener(id==="getvoices-search"?"input":"change",renderGetVoices)}),(_a=$("getvoices-refresh-btn"))==null||_a.addEventListener("click",()=>loadGetVoices(!0)),(_b=$("getvoices-reset-sources-btn"))==null||_b.addEventListener("click",()=>{const box=$("getvoices-sources");box&&(box.value=sourceTextareaValueFromDefaults(),localStorage.setItem(VOICE_SOURCE_STORAGE_KEY,box.value),_voiceSourcePayload=null,renderGetVoices(),$("getvoices-status").textContent="Source list reset.")});let _osItems=[];function osDuration(sec){return sec=Math.round(sec||0),sec?`${Math.floor(sec/60)}:${String(sec%60).padStart(2,"0")}`:""}function renderOpenSources(){$("os-list").innerHTML=_osItems.map((it,i)=>`
${escHtml(it.title)}${escHtml([it.reader,it.language,osDuration(it.duration)].filter(Boolean).join(" \xB7 "))}
Licence: ${it.licence_url?`${escHtml(it.licence)}`:escHtml(it.licence)}
- + ${it.audio_url?``:``}
Source page -
`).join("")}async function searchOpenSources(ev){ev==null||ev.preventDefault();const st=$("os-status"),btn=$("os-search-btn"),q=($("os-q").value||"").trim();if(!q){st.textContent="Enter a search term.";return}btn.disabled=!0,st.textContent="Searching...";try{const p=new URLSearchParams({q,lang:$("os-lang").value}),r=await fetch(`/api/voice-sources/${encodeURIComponent($("os-source").value)}?${p}`),d=await r.json().catch(()=>({}));if(!r.ok)throw new Error(d.detail||r.statusText);_osItems=d.items||[],renderOpenSources(),st.textContent=`${_osItems.length} result${_osItems.length===1?"":"s"}.`}catch(e){_osItems=[],renderOpenSources(),st.textContent="Search failed: "+e.message}finally{btn.disabled=!1}}async function importOpenSource(it){const st=$("os-status"),lang=sourceLanguageCode(it.language||$("os-lang").value||"English"),body={audio_url:it.audio_url,voice_id:suggestedVoiceIdFromSourceName(it.title,lang,"N"),transcript:"",lang,source_kind:it.source,page_url:it.source_url,licence:it.licence,attribution:it.attribution||"",name:it.title,group:it.source,origin:"cloned",start:+$("os-start").value||0,length:+$("os-len").value||30,note:`${it.reader||""} ${it.licence}`.trim()};st.textContent="Importing...";try{const r=await fetch("/api/quick-import-voice",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify(body)}),d=await r.json().catch(()=>({}));if(!r.ok)throw new Error(d.detail||r.statusText);st.textContent=`Saved as ${d.voice_id} (source and licence stored).`,toast(`Saved as ${d.voice_id}`,"success"),_libraryLoadPromise=null}catch(e){st.textContent="Import failed: "+e.message,toast("Import failed: "+e.message,"error")}}(_c=$("os-form"))==null||_c.addEventListener("submit",searchOpenSources),(_d=$("os-list"))==null||_d.addEventListener("click",e=>{const b=e.target.closest(".os-import");b&&_osItems[+b.dataset.i]&&importOpenSource(_osItems[+b.dataset.i])});const SOURCE_LANGUAGE_CODES={english:"EN",german:"DE",deutsch:"DE",french:"FR",spanish:"ES",japanese:"JA",korean:"KO",italian:"IT",portuguese:"PT",russian:"RU",arabic:"AR",polish:"PL",dutch:"NL",swedish:"SV",turkish:"TR",hindi:"HI",chinese:"ZH"};function sourceLanguageCode(language){const raw=String(language||"").trim();return/^[A-Za-z]{2}$/.test(raw)?raw.toUpperCase():SOURCE_LANGUAGE_CODES[raw.toLowerCase()]||"EN"}function sourceGenderCode(gender,name=""){const text=`${gender||""} ${name||""}`.toLowerCase();return/female|woman|girl|\bf\b/.test(text)?"F":/male|man|boy|\bm\b/.test(text)?"M":"N"}function suggestedVoiceIdFromSourceName(name,language="EN",gender="N"){const base=String(name||"SourceVoice").normalize("NFKD").replace(/[\u0300-\u036f]/g,"").replace(/[^A-Za-z0-9]+/g,"_").replace(/^_+|_+$/g,"").slice(0,48)||"SourceVoice";return`${language||"EN"}_${gender||"N"}_${base}`}function setLibAddSourcePreview2(meta={}){const sourceBox=document.querySelector(".lib-add-source-box");if(!sourceBox)return;let preview=$("lib-add-source-preview");if(preview||(preview=document.createElement("div"),preview.id="lib-add-source-preview",preview.className="lib-add-source-preview",sourceBox.appendChild(preview)),!meta.name&&!meta.imageUrl){preview.classList.remove("open"),preview.innerHTML="";return}const image=meta.imageUrl?``:'
';preview.innerHTML=`${image}
${escHtml(meta.name||"Source voice")}${escHtml([meta.language,meta.gender,meta.kind].filter(Boolean).join(" \xB7 ")||"Source metadata will be saved with the voice")}
`,preview.classList.add("open")}async function getSourceVoiceInLibrary(meta){if(!meta.url){toast("This source has no direct audio URL","error");return}const lang=sourceLanguageCode(meta.language),gender=sourceGenderCode(meta.gender,meta.name),voiceId=suggestedVoiceIdFromSourceName(meta.name,lang,gender);toast(`Importing ${meta.name||"voice"}\u2026`,"info");try{const r=await fetch("/api/quick-import-voice",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({audio_url:meta.url,voice_id:voiceId,transcript:""})});if(!r.ok){const e=await r.json().catch(()=>({}));throw new Error(e.detail||r.statusText)}const d=await r.json();let srcTag="online";try{meta.pageUrl&&(srcTag=new URL(meta.pageUrl).hostname.replace(/^www\./,"").split(".")[0]||"online")}catch{}typeof saveMeta=="function"&&await saveMeta(d.voice_id,{name:meta.name||d.voice_id,tag:srcTag,group:srcTag,origin:"cloned",gender:(gender||"").toUpperCase(),note:(meta.description||"").slice(0,180)}).catch(()=>{}),meta.imageUrl&&await fetch("/api/voice/picture-url",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({voice_id:d.voice_id,image_url:meta.imageUrl})}).catch(()=>{}),_pendingSelectId=d.voice_id,navTo("s-voices"),_libraryLoadPromise=null,await loadVoiceLibrary(),toast(`Saved as ${d.voice_id}`,"success")}catch(e){toast("Import failed: "+e.message,"error")}}(_e=$("getvoices-list"))==null||_e.addEventListener("click",async e=>{const getBtn=e.target.closest(".get-source-voice");if(getBtn){await getSourceVoiceInLibrary({url:getBtn.dataset.url||"",name:getBtn.dataset.name||"",imageUrl:getBtn.dataset.image||"",language:getBtn.dataset.language||"",gender:getBtn.dataset.gender||"",kind:getBtn.dataset.kind||"",description:getBtn.dataset.description||"",pageUrl:getBtn.dataset.page||""});return}const btn=e.target.closest(".copy-source-url");btn&&(await copyText(btn.dataset.url||""),toast("Source URL copied","success"))}),Object.assign(root,{DEFAULT_VOICE_SOURCE_URLS,VOICE_SOURCE_STORAGE_KEY,_voiceSourcePayload,_voiceSourceItems,sourceTextareaValueFromDefaults,initGetVoiceSourcesEditor,getEditableVoiceSourceUrls,getVoiceSourceItems,voiceSourceSearchText,setOptions,renderGetVoices,loadGetVoices,SOURCE_LANGUAGE_CODES,sourceLanguageCode,sourceGenderCode,suggestedVoiceIdFromSourceName,setLibAddSourcePreview:setLibAddSourcePreview2,getSourceVoiceInLibrary,searchOpenSources,importOpenSource}),root.TTVC=root.TTVC||{},root.TTVC.voiceSources=root.TTVC.voiceSources||{},Object.assign(root.TTVC.voiceSources,{list:getVoiceSourceItems,load:loadGetVoices,import:getSourceVoiceInLibrary,sourceLanguageCode,sourceGenderCode})}(window),function(){var _a,_b,_c,_d,_e,_f,_g,_h,_i,_j,_k,_l;const $2=id=>document.getElementById(id),grid=$2("fa-grid");if(!grid)return;let _page=1,_audio=null,_playingCard=null;const esc=s=>String(s==null?"":s).replace(/&/g,"&").replace(//g,">").replace(/"/g,""");function fmtCount(n){return n=+n||0,n>=1e6?(n/1e6).toFixed(1).replace(/\.0$/,"")+"M":n>=1e3?(n/1e3).toFixed(1).replace(/\.0$/,"")+"K":String(n)}const FLAG={EN:"\u{1F1EC}\u{1F1E7}",DE:"\u{1F1E9}\u{1F1EA}",FR:"\u{1F1EB}\u{1F1F7}",ES:"\u{1F1EA}\u{1F1F8}",IT:"\u{1F1EE}\u{1F1F9}",PT:"\u{1F1F5}\u{1F1F9}",NL:"\u{1F1F3}\u{1F1F1}",JA:"\u{1F1EF}\u{1F1F5}",ZH:"\u{1F1E8}\u{1F1F3}",KO:"\u{1F1F0}\u{1F1F7}",RU:"\u{1F1F7}\u{1F1FA}",AR:"\u{1F1F8}\u{1F1E6}",PL:"\u{1F1F5}\u{1F1F1}"};function status2(msg,show=!0){const bar=$2("fa-status-bar"),txt=$2("fa-status-text");bar&&(bar.hidden=!show),txt&&(txt.textContent=msg||"")}function activeVals(group){return[...document.querySelectorAll(`#fa-filter-pop .fa-pchips[data-group="${group}"] .fa-pchip.active`)].map(c=>c.dataset.val).filter(Boolean)}function filterCount(){var _a2;return activeVals("gender").length+activeVals("age").length+activeVals("tag").length+((_a2=$2("fa-tag"))!=null&&_a2.value.trim()?1:0)}function updateFilterBadge(){const n=filterCount(),b=$2("fa-filter-badge"),btn=$2("fa-filter-btn");b&&(b.textContent=String(n),b.hidden=n===0),btn==null||btn.classList.toggle("active",n>0)}function params(){var _a2,_b2,_c2,_d2;const p=new URLSearchParams({page:String(_page),page_size:"24"}),s=(_a2=$2("fa-search"))==null?void 0:_a2.value.trim();s&&p.set("search",s);const l=(_b2=$2("fa-lang"))==null?void 0:_b2.value;l&&p.set("language",l);const sort=(_c2=$2("fa-sort"))==null?void 0:_c2.value;sort&&p.set("sort_by",sort);const g=activeVals("gender")[0];g&&p.set("gender",g);const a=activeVals("age")[0];a&&p.set("age",a),activeVals("tag").forEach(t2=>p.append("tag",t2));const t=(_d2=$2("fa-tag"))==null?void 0:_d2.value.trim();return t&&p.append("tag",t),p}function card(v){const can=!!v.sample_audio,meta=[v.language?`${FLAG[v.language]||"\u{1F310}"} ${esc(v.language)}`:"",v.gender?`${esc(v.gender)}`:"",v.age?`${esc(v.age)}`:""].join(""),avatar=v.image?``:"";return`
+
`).join("")}async function searchOpenSources(ev){ev==null||ev.preventDefault();const st=$("os-status"),btn=$("os-search-btn"),q=($("os-q").value||"").trim();if(!q){st.textContent="Enter a search term.";return}btn.disabled=!0,st.textContent="Searching...";try{const p=new URLSearchParams({q,lang:$("os-lang").value}),r=await fetch(`/api/voice-sources/${encodeURIComponent($("os-source").value)}?${p}`),d=await r.json().catch(()=>({}));if(!r.ok)throw new Error(d.detail||r.statusText);_osItems=d.items||[],renderOpenSources(),st.textContent=`${_osItems.length} result${_osItems.length===1?"":"s"}.`}catch(e){_osItems=[],renderOpenSources(),st.textContent="Search failed: "+e.message}finally{btn.disabled=!1}}async function resolveOpenSource(it){if(it.audio_url||!it.book_id)return;const r=await fetch(`/api/voice-sources/librivox/book/${encodeURIComponent(it.book_id)}`),d=await r.json().catch(()=>({}));if(!r.ok)throw new Error(d.detail||r.statusText);Object.assign(it,{audio_url:d.audio_url,reader:d.reader||it.reader,duration:d.duration||it.duration,title:d.section?`${it.title} - ${d.section}`:it.title})}async function importOpenSource(it){const st=$("os-status");st.textContent="Looking up audio...";try{await resolveOpenSource(it)}catch(e){st.textContent="Lookup failed: "+e.message;return}const lang=sourceLanguageCode(it.language||$("os-lang").value||"English"),body={audio_url:it.audio_url,voice_id:suggestedVoiceIdFromSourceName(it.title,lang,"N"),transcript:"",lang,source_kind:it.source,page_url:it.source_url,licence:it.licence,attribution:it.attribution||"",name:it.title,group:it.source,origin:"cloned",start:+$("os-start").value||0,length:+$("os-len").value||30,note:`${it.reader||""} ${it.licence}`.trim()};st.textContent="Importing...";try{const r=await fetch("/api/quick-import-voice",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify(body)}),d=await r.json().catch(()=>({}));if(!r.ok)throw new Error(d.detail||r.statusText);st.textContent=`Saved as ${d.voice_id} (source and licence stored).`,toast(`Saved as ${d.voice_id}`,"success"),_libraryLoadPromise=null}catch(e){st.textContent="Import failed: "+e.message,toast("Import failed: "+e.message,"error")}}(_c=$("os-form"))==null||_c.addEventListener("submit",searchOpenSources),(_d=$("os-list"))==null||_d.addEventListener("click",async e=>{const l=e.target.closest(".os-load");if(l&&_osItems[+l.dataset.i]){l.disabled=!0,$("os-status").textContent="Loading preview...";try{await resolveOpenSource(_osItems[+l.dataset.i]),renderOpenSources(),$("os-status").textContent="Preview ready."}catch(err){l.disabled=!1,$("os-status").textContent="Preview failed: "+err.message}return}const b=e.target.closest(".os-import");b&&_osItems[+b.dataset.i]&&importOpenSource(_osItems[+b.dataset.i])});const SOURCE_LANGUAGE_CODES={english:"EN",german:"DE",deutsch:"DE",french:"FR",spanish:"ES",japanese:"JA",korean:"KO",italian:"IT",portuguese:"PT",russian:"RU",arabic:"AR",polish:"PL",dutch:"NL",swedish:"SV",turkish:"TR",hindi:"HI",chinese:"ZH"};function sourceLanguageCode(language){const raw=String(language||"").trim();return/^[A-Za-z]{2}$/.test(raw)?raw.toUpperCase():SOURCE_LANGUAGE_CODES[raw.toLowerCase()]||"EN"}function sourceGenderCode(gender,name=""){const text=`${gender||""} ${name||""}`.toLowerCase();return/female|woman|girl|\bf\b/.test(text)?"F":/male|man|boy|\bm\b/.test(text)?"M":"N"}function suggestedVoiceIdFromSourceName(name,language="EN",gender="N"){const base=String(name||"SourceVoice").normalize("NFKD").replace(/[\u0300-\u036f]/g,"").replace(/[^A-Za-z0-9]+/g,"_").replace(/^_+|_+$/g,"").slice(0,48)||"SourceVoice";return`${language||"EN"}_${gender||"N"}_${base}`}function setLibAddSourcePreview2(meta={}){const sourceBox=document.querySelector(".lib-add-source-box");if(!sourceBox)return;let preview=$("lib-add-source-preview");if(preview||(preview=document.createElement("div"),preview.id="lib-add-source-preview",preview.className="lib-add-source-preview",sourceBox.appendChild(preview)),!meta.name&&!meta.imageUrl){preview.classList.remove("open"),preview.innerHTML="";return}const image=meta.imageUrl?``:'
';preview.innerHTML=`${image}
${escHtml(meta.name||"Source voice")}${escHtml([meta.language,meta.gender,meta.kind].filter(Boolean).join(" \xB7 ")||"Source metadata will be saved with the voice")}
`,preview.classList.add("open")}async function getSourceVoiceInLibrary(meta){if(!meta.url){toast("This source has no direct audio URL","error");return}const lang=sourceLanguageCode(meta.language),gender=sourceGenderCode(meta.gender,meta.name),voiceId=suggestedVoiceIdFromSourceName(meta.name,lang,gender);toast(`Importing ${meta.name||"voice"}\u2026`,"info");try{const r=await fetch("/api/quick-import-voice",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({audio_url:meta.url,voice_id:voiceId,transcript:""})});if(!r.ok){const e=await r.json().catch(()=>({}));throw new Error(e.detail||r.statusText)}const d=await r.json();let srcTag="online";try{meta.pageUrl&&(srcTag=new URL(meta.pageUrl).hostname.replace(/^www\./,"").split(".")[0]||"online")}catch{}typeof saveMeta=="function"&&await saveMeta(d.voice_id,{name:meta.name||d.voice_id,tag:srcTag,group:srcTag,origin:"cloned",gender:(gender||"").toUpperCase(),note:(meta.description||"").slice(0,180)}).catch(()=>{}),meta.imageUrl&&await fetch("/api/voice/picture-url",{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify({voice_id:d.voice_id,image_url:meta.imageUrl})}).catch(()=>{}),_pendingSelectId=d.voice_id,navTo("s-voices"),_libraryLoadPromise=null,await loadVoiceLibrary(),toast(`Saved as ${d.voice_id}`,"success")}catch(e){toast("Import failed: "+e.message,"error")}}(_e=$("getvoices-list"))==null||_e.addEventListener("click",async e=>{const getBtn=e.target.closest(".get-source-voice");if(getBtn){await getSourceVoiceInLibrary({url:getBtn.dataset.url||"",name:getBtn.dataset.name||"",imageUrl:getBtn.dataset.image||"",language:getBtn.dataset.language||"",gender:getBtn.dataset.gender||"",kind:getBtn.dataset.kind||"",description:getBtn.dataset.description||"",pageUrl:getBtn.dataset.page||""});return}const btn=e.target.closest(".copy-source-url");btn&&(await copyText(btn.dataset.url||""),toast("Source URL copied","success"))}),Object.assign(root,{DEFAULT_VOICE_SOURCE_URLS,VOICE_SOURCE_STORAGE_KEY,_voiceSourcePayload,_voiceSourceItems,sourceTextareaValueFromDefaults,initGetVoiceSourcesEditor,getEditableVoiceSourceUrls,getVoiceSourceItems,voiceSourceSearchText,setOptions,renderGetVoices,loadGetVoices,SOURCE_LANGUAGE_CODES,sourceLanguageCode,sourceGenderCode,suggestedVoiceIdFromSourceName,setLibAddSourcePreview:setLibAddSourcePreview2,getSourceVoiceInLibrary,searchOpenSources,importOpenSource}),root.TTVC=root.TTVC||{},root.TTVC.voiceSources=root.TTVC.voiceSources||{},Object.assign(root.TTVC.voiceSources,{list:getVoiceSourceItems,load:loadGetVoices,import:getSourceVoiceInLibrary,sourceLanguageCode,sourceGenderCode})}(window),function(){var _a,_b,_c,_d,_e,_f,_g,_h,_i,_j,_k,_l;const $2=id=>document.getElementById(id),grid=$2("fa-grid");if(!grid)return;let _page=1,_audio=null,_playingCard=null;const esc=s=>String(s==null?"":s).replace(/&/g,"&").replace(//g,">").replace(/"/g,""");function fmtCount(n){return n=+n||0,n>=1e6?(n/1e6).toFixed(1).replace(/\.0$/,"")+"M":n>=1e3?(n/1e3).toFixed(1).replace(/\.0$/,"")+"K":String(n)}const FLAG={EN:"\u{1F1EC}\u{1F1E7}",DE:"\u{1F1E9}\u{1F1EA}",FR:"\u{1F1EB}\u{1F1F7}",ES:"\u{1F1EA}\u{1F1F8}",IT:"\u{1F1EE}\u{1F1F9}",PT:"\u{1F1F5}\u{1F1F9}",NL:"\u{1F1F3}\u{1F1F1}",JA:"\u{1F1EF}\u{1F1F5}",ZH:"\u{1F1E8}\u{1F1F3}",KO:"\u{1F1F0}\u{1F1F7}",RU:"\u{1F1F7}\u{1F1FA}",AR:"\u{1F1F8}\u{1F1E6}",PL:"\u{1F1F5}\u{1F1F1}"};function status2(msg,show=!0){const bar=$2("fa-status-bar"),txt=$2("fa-status-text");bar&&(bar.hidden=!show),txt&&(txt.textContent=msg||"")}function activeVals(group){return[...document.querySelectorAll(`#fa-filter-pop .fa-pchips[data-group="${group}"] .fa-pchip.active`)].map(c=>c.dataset.val).filter(Boolean)}function filterCount(){var _a2;return activeVals("gender").length+activeVals("age").length+activeVals("tag").length+((_a2=$2("fa-tag"))!=null&&_a2.value.trim()?1:0)}function updateFilterBadge(){const n=filterCount(),b=$2("fa-filter-badge"),btn=$2("fa-filter-btn");b&&(b.textContent=String(n),b.hidden=n===0),btn==null||btn.classList.toggle("active",n>0)}function params(){var _a2,_b2,_c2,_d2;const p=new URLSearchParams({page:String(_page),page_size:"24"}),s=(_a2=$2("fa-search"))==null?void 0:_a2.value.trim();s&&p.set("search",s);const l=(_b2=$2("fa-lang"))==null?void 0:_b2.value;l&&p.set("language",l);const sort=(_c2=$2("fa-sort"))==null?void 0:_c2.value;sort&&p.set("sort_by",sort);const g=activeVals("gender")[0];g&&p.set("gender",g);const a=activeVals("age")[0];a&&p.set("age",a),activeVals("tag").forEach(t2=>p.append("tag",t2));const t=(_d2=$2("fa-tag"))==null?void 0:_d2.value.trim();return t&&p.append("tag",t),p}function card(v){const can=!!v.sample_audio,meta=[v.language?`${FLAG[v.language]||"\u{1F310}"} ${esc(v.language)}`:"",v.gender?`${esc(v.gender)}`:"",v.age?`${esc(v.age)}`:""].join(""),avatar=v.image?``:"";return`
${avatar} ${can?'':""} diff --git a/static/js/voice-sources.js b/static/js/voice-sources.js index 180c1fa..05c95ed 100644 --- a/static/js/voice-sources.js +++ b/static/js/voice-sources.js @@ -199,7 +199,9 @@ function renderOpenSources() { $('os-list').innerHTML = _osItems.map((it, i) => `
${escHtml(it.title)}${escHtml([it.reader, it.language, osDuration(it.duration)].filter(Boolean).join(' · '))}
Licence: ${it.licence_url ? `${escHtml(it.licence)}` : escHtml(it.licence)}
- + ${it.audio_url + ? `` + : ``}
Source page @@ -223,8 +225,19 @@ async function searchOpenSources(ev) { _osItems = []; renderOpenSources(); st.textContent = 'Search failed: ' + e.message; } finally { btn.disabled = false; } } +// LibriVox list items carry only book_id; the first chapter's audio is fetched on demand. +async function resolveOpenSource(it) { + if (it.audio_url || !it.book_id) return; + const r = await fetch(`/api/voice-sources/librivox/book/${encodeURIComponent(it.book_id)}`); + const d = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(d.detail || r.statusText); + Object.assign(it, {audio_url: d.audio_url, reader: d.reader || it.reader, duration: d.duration || it.duration, + title: d.section ? `${it.title} - ${d.section}` : it.title}); +} async function importOpenSource(it) { const st = $('os-status'); + st.textContent = 'Looking up audio...'; + try { await resolveOpenSource(it); } catch (e) { st.textContent = 'Lookup failed: ' + e.message; return; } const lang = sourceLanguageCode(it.language || $('os-lang').value || 'English'); const body = { audio_url: it.audio_url, voice_id: suggestedVoiceIdFromSourceName(it.title, lang, 'N'), transcript: '', lang, @@ -243,7 +256,14 @@ async function importOpenSource(it) { } catch (e) { st.textContent = 'Import failed: ' + e.message; toast('Import failed: ' + e.message, 'error'); } } $('os-form')?.addEventListener('submit', searchOpenSources); -$('os-list')?.addEventListener('click', e => { +$('os-list')?.addEventListener('click', async e => { + const l = e.target.closest('.os-load'); + if (l && _osItems[+l.dataset.i]) { + l.disabled = true; $('os-status').textContent = 'Loading preview...'; + try { await resolveOpenSource(_osItems[+l.dataset.i]); renderOpenSources(); $('os-status').textContent = 'Preview ready.'; } + catch (err) { l.disabled = false; $('os-status').textContent = 'Preview failed: ' + err.message; } + return; + } const b = e.target.closest('.os-import'); if (b && _osItems[+b.dataset.i]) importOpenSource(_osItems[+b.dataset.i]); }); diff --git a/tests/test_voice_sources_new.py b/tests/test_voice_sources_new.py index b3c514f..d59abf7 100644 --- a/tests/test_voice_sources_new.py +++ b/tests/test_voice_sources_new.py @@ -13,10 +13,12 @@ from core import open_voice_sources as ovs import routes.sources as src -def book(i=1, lang="English", url="https://www.archive.org/download/x/a.mp3"): - return {"title": f"Book {i}", "language": lang, "url_librivox": f"https://librivox.org/b{i}/", - "authors": [{"first_name": "Jane", "last_name": "Austen"}], - "sections": [{"title": "Ch 1", "listen_url": url, "playtime": "600", "readers": [{"display_name": "Ann"}]}]} +def book(i=1, lang="English", url="https://www.archive.org/download/x/a.mp3", sections=False): + b = {"id": str(i), "title": f"Book {i}", "language": lang, "url_librivox": f"https://librivox.org/b{i}/", + "totaltimesecs": 600, "authors": [{"first_name": "Jane", "last_name": "Austen"}]} + if sections: + b["sections"] = [{"title": "Ch 1", "listen_url": url, "playtime": "600", "readers": [{"display_name": "Ann"}]}] + return b def commons(**over): @@ -56,16 +58,31 @@ class ValidationTests(Base): class MappingTests(Base): - def test_librivox_mapping_filter_cap_and_bad_hosts(self): - books = [book(i) for i in range(15)] + [book(99, url="https://evil.example/a.mp3"), book(98, lang="German")] + def test_librivox_list_is_light_filtered_and_capped(self): + books = [book(i) for i in range(15)] + [book(98, lang="German"), dict(book(97), url_librivox="https://evil.example/x/"), dict(book(96), id="x;1")] with mock.patch.object(ovs, "_get_json", return_value={"books": books}) as g: items = ovs.search("librivox", "pride", "en") self.assertEqual(len(items), ovs.MAX_RESULTS) - self.assertEqual(g.call_args[0][0], ovs.LIBRIVOX_API) + self.assertNotIn("extended", g.call_args[0][1]) it = items[0] - self.assertEqual((it["reader"], it["author"], it["language"], it["duration"]), ("Ann", "Jane Austen", "English", 600)) + self.assertEqual((it["author"], it["language"], it["duration"], it["audio_url"], it["book_id"]), ("Jane Austen", "English", 600, "", "0")) self.assertTrue(it["licence_url"].startswith("https://librivox.org/")) - self.assertTrue(all("evil" not in i["audio_url"] and i["language"] == "English" for i in items)) + self.assertTrue(all(i["language"] == "English" and "evil" not in i["source_url"] for i in items)) + + def test_librivox_first_section(self): + bad = book(2, url="https://evil.example/a.mp3", sections=True) + bad["sections"].append({"title": "Ch 2", "listen_url": "https://ia1.us.archive.org/b.mp3", "playtime": "5", "readers": []}) + with mock.patch.object(ovs, "_get_json", return_value={"books": [bad]}) as g: + r = ovs.librivox_first_section("2") + self.assertEqual(g.call_args[0][1]["extended"], 1) + self.assertEqual((r["audio_url"], r["section"], r["duration"]), ("https://ia1.us.archive.org/b.mp3", "Ch 2", 5)) + for badid in ("", "abc", "1;2", "1" * 10): + with self.assertRaises(ovs.SourceError): + ovs.librivox_first_section(badid) + with mock.patch.object(ovs, "_get_json", return_value={"books": [book(3, url="https://evil.example/a")]}): + with self.assertRaises(ovs.SourceError) as cm: + ovs.librivox_first_section("3") + self.assertEqual(cm.exception.status, 404) def test_wikimedia_mapping(self): with mock.patch.object(ovs, "_get_json", return_value=commons()): @@ -117,6 +134,41 @@ class CacheThrottleTests(Base): self.assertNotIn("secret", str(cm.exception)) +class RetryErrorTests(Base): + def setUp(self): + super().setUp() + ovs._neg.clear() + + def test_timeout_retried_once_then_clear_message_and_negative_cache(self): + with mock.patch.object(ovs.requests, "get", side_effect=ovs.requests.Timeout("secret")) as g: + with self.assertRaises(ovs.SourceError) as cm: + ovs.search("librivox", "Faust") + self.assertEqual(g.call_count, 2) + self.assertIn("in time", str(cm.exception)) + self.assertEqual(cm.exception.status, 502) + with self.assertRaises(ovs.SourceError): # remembered: no further upstream call + ovs.search("librivox", "Faust") + self.assertEqual(g.call_count, 2) + + def test_retry_recovers_and_messages_differ(self): + resp = mock.Mock(status_code=200) + resp.raise_for_status.return_value = None + resp.iter_content.return_value = [b'{"books": []}'] + with mock.patch.object(ovs.requests, "get", side_effect=[ovs.requests.ConnectionError("x"), resp]): + self.assertEqual(ovs._get_json(ovs.LIBRIVOX_API, {}), {"books": []}) + for exc, text in ((ovs.requests.ConnectionError("x"), "unreachable"), (ValueError("x"), "unreadable")): + with mock.patch.object(ovs.requests, "get", side_effect=exc): + with self.assertRaises(ovs.SourceError) as cm: + ovs._get_json(ovs.LIBRIVOX_API, {}) + self.assertIn(text, str(cm.exception)) + + +class SettingsTests(unittest.TestCase): + def test_export_provenance_default_exposed(self): + from core import config + self.assertIs(config._load_settings().get("export_provenance_default"), True) + + class RouteTests(Base): def test_search_route_errors_and_allowlist(self): with self.assertRaises(src.HTTPException) as cm: diff --git a/tests/voice-sources-new.spec.js b/tests/voice-sources-new.spec.js index ff54adc..f58b70b 100644 --- a/tests/voice-sources-new.spec.js +++ b/tests/voice-sources-new.spec.js @@ -41,3 +41,26 @@ test('search error is shown in the status region', async ({ page }) => { await page.evaluate(() => document.getElementById('os-form').requestSubmit()); await expect(page.locator('#os-status')).toContainText('unavailable'); }); + +test('librivox list without audio: preview and import resolve the chapter on demand', async ({ page }) => { + let imported = null, resolved = 0; + const light = [{ ...ITEMS[0], book_id: '855', audio_url: '', reader: '' }]; + await page.route('**/api/voice-sources/librivox/book/855', r => { resolved++; return r.fulfill({ json: { audio_url: 'https://www.archive.org/download/x/a.mp3', section: 'Ch 1', reader: 'Ann', duration: 600 } }); }); + await page.route('**/api/voice-sources/librivox?*', r => r.fulfill({ json: { source: 'librivox', items: light } })); + await page.route('**/api/proxy-audio*', r => r.fulfill({ body: '', contentType: 'audio/mpeg' })); + await page.route('**/api/quick-import-voice', r => { imported = r.request().postDataJSON(); return r.fulfill({ json: { voice_id: 'EN_N_Pride' } }); }); + await page.route('**/api/voices*', r => r.fulfill({ json: [] })); + await page.route('**/api/settings*', r => r.fulfill({ json: {} })); + await page.goto('/', { waitUntil: 'domcontentloaded' }); + await page.waitForFunction(() => typeof window.navTo === 'function' && !!document.getElementById('os-form'), null, { timeout: 15_000 }); + await page.evaluate(() => window.navTo('s-studio')); + await page.locator('.gvo-tab[data-src="open"]').click(); + await page.fill('#os-q', 'pride'); + await page.press('#os-q', 'Enter'); + await page.locator('.os-load').click(); + await expect(page.locator('#os-list audio')).toHaveCount(1); + await page.locator('.os-import').click(); + await expect(page.locator('#os-status')).toContainText('Saved as EN_N_Pride'); + expect(resolved).toBe(1); + expect(imported.audio_url).toContain('archive.org'); +});