feat(voices): LibriVox and Wikimedia Commons voice sources

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -7,6 +7,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
 
 ## [Unreleased]
 ### Added
+- **LibriVox and Wikimedia Commons voice sources.** New tab in Get Voices Online searches the official LibriVox feed and the Commons MediaWiki API through `GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` (fixed hosts, query cap 80, 10 results, 30 s timeout, 10 min cache, 1 s throttle, descriptive User-Agent, result URLs host-checked). Results show licence and a preview; "Import clip" trims (start/length) and stores provenance (LibriVox: public domain + check-the-reader note; Wikimedia: licence + attribution). `archive.org` and `upload.wikimedia.org` added to the audio proxy allowlist. Tests: `tests/test_voice_sources_new.py`, `tests/voice-sources-new.spec.js`. See `architecture/voice-sources.md`.
 - **Provenance tags on exported audio.** Magazine MP3 downloads, `audio.zip` and the MP3s inside the EPUB get ID3 tags (`synthetic_speech=true`, `generator`, `generator_version`, `generated_date`, comment; voice name only on opt-in) via one ffmpeg stream copy of a temp copy; stored audio is untouched. Per-export checkboxes on the magazine export row ("Mark as synthetic speech", "include voice name"), default from new setting `export_provenance_default` (on). `POST /api/export/provenance` and `python -m core.export_provenance file.mp3` read the tags back. Inaudible AudioSeal watermarking is researched (MIT code and weights, torch needed, weak message recovery after MP3/AAC) but only a stub `watermark(path)` that reports unavailable. Spoken disclosure is not implemented yet. Tests: `tests/test_watermark_tags.py`, `tests/magazine.spec.js`. See `architecture/watermarking.md`.
 - **Inline text tags for TTS (setting `inline_tags_enabled`, default OFF).** `[pause 500ms]`/`[pause 1.2s]` (max 10 s each, 60 s per text), `[spell]USA[/spell]` (German letter names for German text), `[slow]..[/slow]` / `[fast]..[/fast]` (0.85 / 1.15 via ffmpeg atempo, innermost wins, unclosed ends at paragraph). Known control tags (e.g. `[laugh]`) are stripped, never spoken; `[1]` and `[sic]` stay. Rendered in `POST /api/tts-preview` (also used by audiobook line previews), `/v1/audio/speech` (non-streaming), `/speak` and the MCP speak tool through one shared `ssml_lite.render_with_tags`; stripped only (never spoken) in streaming, spoken chat replies and magazine MP3 (timings stay exact). Settings -> General switch (accessible, DE/EN) and a syntax hint under the Try It Out and Read Aloud text boxes. Tests: `tests/test_ssml_lite.py`, `tests/test_inline_tags_paths.py`, `tests/inline-tags.spec.js`.
 - **Optional vocal isolation for URL/YouTube imports (default OFF).** `POST /api/isolate-vocals {id}` runs the Demucs CLI (`htdemucs`, two stems) on a registered temp audio and registers the vocals as a new id; the original stays valid. Needs `VOCAL_ISOLATION=1`, a `demucs` binary (or `VOCAL_ISOLATION_CMD`) and the weights already on disk; the app never downloads anything and nothing is added to the main image. Limits: 15 min input, 15 min runtime (`VOCAL_ISOLATION_MAX_SECONDS`/`_TIMEOUT`), one job at a time, cancelled on client disconnect, temp files always removed. `GET /api/version` reports `features.vocal_isolation`; the "Remove background music" button (clone page URL card and voice library add-URL) appears only then, with an aria-live status and a play original / play without music toggle. Tests: `tests/test_vocal_isolation.py`, `tests/vocal-isolation.spec.js`. See `architecture/vocal-isolation.md`.

Manual.md

@@ -45,6 +45,17 @@ Errors: damaged, tampered, oversized or unexpected files are rejected with a mes
 Mobile / accessibility: both buttons are at least 44 px high and keyboard operable; the file chooser is the browser's own.
 Offline: works fully offline. Privacy: a bundle contains the voice, its source/consent notes and any consent recording; share it only with people you trust.
 
+## LibriVox and Wikimedia Commons voices
+
+Purpose: find short speech samples with an open licence and import a clip as a voice.
+Prerequisites: Voice library; internet access from the server.
+Steps: Get Voices Online -> tab "LibriVox & Wikimedia"; choose source, type a term (LibriVox matches the start of a title; the language filter works for LibriVox only), press Search (can take about 15 s). Play the preview, set clip start and length (seconds), press "Import clip".
+Result: the clip is saved as a voice; its "Source & consent" shows source page, licence and, for Wikimedia, the attribution text. LibriVox is recorded as public domain with a reminder to check the reader's licence; Wikimedia as licensed (or public domain).
+Errors: shown in the status line under the search box (bad input, source unavailable, trim outside the audio).
+Licences: CC BY / CC BY-SA need attribution when you publish results; read `architecture/voice-sources.md` before publishing anything cloned from these voices.
+Mobile/accessibility: 44 px controls, labelled fields, status is announced, previews are native audio players.
+Offline: needs internet. Privacy: your search term goes to librivox.org / archive.org or commons.wikimedia.org / upload.wikimedia.org; nothing else.
+
 ## Voice source & consent
 
 Purpose: record where a voice comes from and why you may use it (own voice, speaker consented, licensed, public domain, unknown).

architecture/voice-sources.md

@@ -0,0 +1,30 @@
+# Voice sources: LibriVox and Wikimedia Commons
+
+Status: implemented. Code: `core/open_voice_sources.py`, routes in `routes/sources.py`, UI in `static/js/voice-sources.js` (tab "LibriVox & Wikimedia" in Get Voices Online).
+
+## Verified API facts (checked 2026-10-04, docs and live calls)
+LibriVox
+- `GET https://librivox.org/api/feed/audiobooks?title=^<q>&format=json&extended=1&limit=N`. A bare `title=` is an EXACT match (404 `{"error":...}` otherwise); the `^` prefix gives a prefix match. No language parameter: the language is filtered client-side on `language` ("English"). `extended=1` adds `sections[]` with `listen_url` (archive.org MP3, 64 kbps), `playtime` (s) and `readers[]`.
+- The docs state no rate limit and no terms. The extended feed is slow (about 14 s for 5-25 books), so timeout is 30 s, limit 15, results cached 10 min, one upstream call per second.
+- Licence (librivox.org/pages/public-domain): recordings are donated to the public domain, use without permission or attribution allowed (credit appreciated). Caveats: guaranteed public domain only under US law; source texts are pre-1923 works; translations are separate works; readers are volunteers and the page of a project may show a reader-specific note, so the UI/provenance note tells the user to check the reader's declared licence. A reader's voice is still a person's voice: public domain is not consent for voice cloning of a recognisable person; the provenance basis is a record, not a legal clearance.
+- Audio is hosted on archive.org (`www.archive.org/download/...` redirects to `iaNNN.us.archive.org`).
+
+Wikimedia Commons
+- `GET https://commons.wikimedia.org/w/api.php?action=query&generator=search&gsrnamespace=6&gsrsearch=filetype:audio <q>&prop=imageinfo&iiprop=url|extmetadata|mediatype&maxlag=5`. Per-file `extmetadata`: `LicenseShortName`, `LicenseUrl`, `Artist` (HTML, stripped), `AttributionRequired`. `inlanguage:` does not filter audio files, so `lang` is ignored for Wikimedia.
+- Etiquette (mediawiki.org API:Etiquette): descriptive User-Agent with contact info is mandatory, requests serial, use `maxlag`, back off on errors; clients can be blocked without notice. Our UA is `TTSVoiceCreator/1.0 (self-hosted voice library; official-API client)` (no personal data; Wikimedia prefers a contact URL/email, set one via a future env var if blocked or high volume).
+- Licences are per file. Attribution duties: CC BY and CC BY-SA require author, title, licence (with link) and a note of changes (trimming/cloning counts); CC BY-SA also requires derivatives under the same licence (a cloned voice model is arguably a derivative: treat as unresolved, prefer CC0/public domain/CC BY for anything published). Public domain/CC0: none. Some files carry non-copyright restrictions (personality rights).
+
+## NOT safe / not done
+- No HTML scraping of librivox.org, archive.org or commons pages; JSON APIs only.
+- No user-supplied URLs: API hosts are constants; every URL from a response must be https on an allowlisted host (`archive.org`, `upload.wikimedia.org`, page hosts `librivox.org`, `commons.wikimedia.org`) or the item is dropped. `quick-import-voice` with `source_kind` re-checks the audio host.
+- Redirects are not followed for API calls; generic `/api/proxy-audio` (existing) follows redirects and now also allows the two audio hosts.
+- Files without a licence in `extmetadata` are dropped. The licence string is trusted from Commons metadata and can be wrong; the user must still check.
+- Licence/attribution values sent by the browser at import are only length-capped, not re-fetched (single-user LAN tool).
+
+## Endpoints
+`GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` : `q` required, max 80 chars, control chars stripped; `lang` 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: `{source, title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url, attribution}`. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server).
+
+`POST /api/quick-import-voice` accepts `source_kind` (`librivox`|`wikimedia`), `licence`, `attribution`, `page_url`, `start`, `length` (clip, 3-120 s, default 0/30). Provenance: LibriVox -> `public_domain`, source = book page, note says to check the reader's licence. Wikimedia -> `licensed` (or `public_domain` when licence says public domain/CC0), licence, note `Attribution: ...`; `meta.attribution` also stored.
+
+## Cache and throttle
+In-process dict, TTL 600 s, max 200 entries (evict oldest); per-source lock with at least 1 s between upstream calls. Lost on restart; fine for one container.

core/open_voice_sources.py

@@ -0,0 +1,196 @@
+"""LibriVox and Wikimedia Commons voice search (official APIs only, fixed hosts). See architecture/voice-sources.md."""
+from __future__ import annotations
+
+import html
+import re
+import threading
+import time
+from urllib.parse import urlsplit
+
+import requests
+
+from core.provenance import validate_provenance
+
+LIBRIVOX_API = "https://librivox.org/api/feed/audiobooks"
+COMMONS_API = "https://commons.wikimedia.org/w/api.php"
+USER_AGENT = "TTSVoiceCreator/1.0 (self-hosted voice library; official-API client)"  # no personal data
+MAX_Q = 80
+MAX_RESULTS = 10
+MAX_BYTES = 8 * 1024 * 1024
+TIMEOUT = 30  # the LibriVox extended feed takes ~14 s
+CACHE_TTL = 600
+CACHE_MAX = 200
+MIN_INTERVAL = 1.0  # seconds between upstream calls per source (MediaWiki asks for serial requests)
+
+# host suffixes allowed in result URLs (checked on every URL taken from an API response)
+AUDIO_HOSTS = ("archive.org", "upload.wikimedia.org")
+PAGE_HOSTS = ("librivox.org", "archive.org", "commons.wikimedia.org")
+SOURCES = ("librivox", "wikimedia")
+LANG_NAMES = {
+    "en": "English", "de": "German", "fr": "French", "es": "Spanish", "it": "Italian", "pt": "Portuguese",
+    "nl": "Dutch", "pl": "Polish", "ru": "Russian", "ja": "Japanese", "zh": "Chinese", "sv": "Swedish",
+}
+_CTRL = re.compile(r"[\x00-\x1f\x7f]")
+_TAGS = re.compile(r"<[^>]*>")
+
+_cache: dict[tuple, tuple[float, list]] = {}
+_last_call: dict[str, float] = {}
+_locks = {s: threading.Lock() for s in SOURCES}
+_cache_lock = threading.Lock()
+
+
+class SourceError(Exception):
+    """Bad input (400) or upstream failure (502); message is safe to show."""
+    def __init__(self, msg: str, status: int = 400):
+        super().__init__(msg)
+        self.status = status
+
+
+def host_ok(url: str, suffixes: tuple[str, ...]) -> bool:
+    try:
+        p = urlsplit(str(url))
+        h = (p.hostname or "").lower()
+    except ValueError:
+        return False
+    return p.scheme == "https" and not p.username and not p.password and any(h == s or h.endswith("." + s) for s in suffixes)
+
+
+def clean_query(q: str) -> str:
+    q = _CTRL.sub(" ", str(q or "")).strip()
+    q = re.sub(r"\s+", " ", q)
+    if not q:
+        raise SourceError("q is required")
+    if len(q) > MAX_Q:
+        raise SourceError(f"q is too long (max {MAX_Q})")
+    return q
+
+
+def clean_lang(lang: str) -> str:
+    lang = str(lang or "").strip().lower()
+    if lang and not re.fullmatch(r"[a-z]{2,3}", lang):
+        raise SourceError("lang must be a 2-3 letter code")
+    return lang
+
+
+def _text(value, limit=200) -> str:
+    return _CTRL.sub(" ", html.unescape(_TAGS.sub("", str(value or "")))).strip()[:limit]
+
+
+def _get_json(url: str, params: dict) -> dict:
+    wait = 0.0
+    src = "librivox" if url == LIBRIVOX_API else "wikimedia"
+    with _locks[src]:
+        wait = MIN_INTERVAL - (time.monotonic() - _last_call.get(src, 0.0))
+        if wait > 0:
+            time.sleep(wait)
+        _last_call[src] = time.monotonic()
+        try:
+            r = requests.get(url, params=params, headers={"User-Agent": USER_AGENT, "Accept": "application/json"},
+                             timeout=TIMEOUT, allow_redirects=False, stream=True)
+            if r.status_code == 404 and src == "librivox":
+                return {}
+            r.raise_for_status()
+            body = b""
+            for chunk in r.iter_content(65536):
+                body += chunk
+                if len(body) > MAX_BYTES:
+                    raise SourceError("Upstream response too large", 502)
+            import json
+            return json.loads(body.decode("utf-8", "replace"))
+        except SourceError:
+            raise
+        except Exception:
+            raise SourceError(f"{src} is unavailable right now", 502)
+
+
+def _librivox_items(q: str, lang: str) -> list[dict]:
+    data = _get_json(LIBRIVOX_API, {"title": "^" + q, "format": "json", "extended": 1, "limit": 15})  # "^" = prefix match; bare title is exact
+    want = LANG_NAMES.get(lang, "").lower()
+    out = []
+    for b in (data.get("books") or []) if isinstance(data, dict) else []:
+        if not isinstance(b, dict):
+            continue
+        if want and str(b.get("language") or "").lower() != want:
+            continue
+        secs = [s for s in (b.get("sections") or []) if isinstance(s, dict) and host_ok(s.get("listen_url") or "", AUDIO_HOSTS)]
+        src_url = b.get("url_librivox") or ""
+        if not secs or not host_ok(src_url, PAGE_HOSTS):
+            continue
+        s = secs[0]
+        readers = ", ".join(_text(r.get("display_name"), 60) for r in (s.get("readers") or []) if isinstance(r, dict))
+        authors = ", ".join(_text(f"{a.get('first_name', '')} {a.get('last_name', '')}", 60) for a in (b.get("authors") or []) if isinstance(a, dict))
+        try:
+            dur = int(s.get("playtime") or 0)
+        except (TypeError, ValueError):
+            dur = 0
+        out.append({
+            "source": "librivox", "title": _text(f"{b.get('title')} - {s.get('title')}"), "reader": readers, "author": authors,
+            "language": _text(b.get("language"), 40), "duration": dur,
+            "licence": "Public domain (LibriVox, USA)", "licence_url": "https://librivox.org/pages/public-domain/",
+            "source_url": src_url, "audio_url": s["listen_url"], "attribution": "",
+        })
+        if len(out) >= MAX_RESULTS:
+            break
+    return out
+
+
+def _wikimedia_items(q: str) -> list[dict]:
+    data = _get_json(COMMONS_API, {
+        "action": "query", "format": "json", "generator": "search", "gsrnamespace": 6, "gsrlimit": 20,
+        "gsrsearch": f"filetype:audio {q}", "prop": "imageinfo", "iiprop": "url|extmetadata|mediatype", "maxlag": 5,
+    })
+    pages = ((data.get("query") or {}).get("pages") or {}) if isinstance(data, dict) else {}
+    out = []
+    for p in sorted((x for x in pages.values() if isinstance(x, dict)), key=lambda x: x.get("index", 0)):
+        info = (p.get("imageinfo") or [{}])[0]
+        meta = info.get("extmetadata") or {}
+        mv = lambda k: (meta.get(k) or {}).get("value", "")
+        lic = _text(mv("LicenseShortName"), 80)
+        audio, page = info.get("url") or "", info.get("descriptionurl") or ""
+        if info.get("mediatype") != "AUDIO" or not lic or not host_ok(audio, AUDIO_HOSTS) or not host_ok(page, PAGE_HOSTS):
+            continue
+        licurl = mv("LicenseUrl") if host_ok(mv("LicenseUrl"), ("creativecommons.org", "wikimedia.org")) else ""
+        title = _text(re.sub(r"^File:|\.[A-Za-z0-9]+$", "", str(p.get("title") or "")))
+        artist = _text(mv("Artist"), 120)
+        out.append({
+            "source": "wikimedia", "title": title, "reader": artist, "author": artist, "language": "",
+            "duration": round(float(info.get("duration") or 0)), "licence": lic, "licence_url": licurl,
+            "source_url": page, "audio_url": audio,
+            "attribution": _text(f"{artist + ' / ' if artist else ''}{title}, {lic}, via Wikimedia Commons", 300),
+        })
+        if len(out) >= MAX_RESULTS:
+            break
+    return out
+
+
+def search(source: str, q: str, lang: str = "") -> list[dict]:
+    if source not in SOURCES:
+        raise SourceError("unknown source")
+    q, lang = clean_query(q), clean_lang(lang)
+    key = (source, q.lower(), lang if source == "librivox" else "")
+    now = time.monotonic()
+    with _cache_lock:
+        hit = _cache.get(key)
+        if hit and now - hit[0] < CACHE_TTL:
+            return hit[1]
+    items = _librivox_items(q, lang) if source == "librivox" else _wikimedia_items(q)
+    with _cache_lock:
+        if len(_cache) >= CACHE_MAX:
+            _cache.pop(min(_cache, key=lambda k: _cache[k][0]))
+        _cache[key] = (now, items)
+    return items
+
+
+def import_provenance_for(kind: str, source_url: str, licence: str, attribution: str) -> dict:
+    """Provenance prefill for an import from one of these sources (raises ValueError on bad input)."""
+    if kind == "librivox":
+        return validate_provenance({
+            "basis": "public_domain", "source": source_url, "licence": "Public domain (LibriVox)",
+            "note": "LibriVox recordings are donated to the public domain (guaranteed in the USA only). "
+                    "Check the reader's declared licence on the LibriVox page before use."})
+    if kind == "wikimedia":
+        pd = re.search(r"public domain|\bcc0\b|\bpd\b", licence, re.I)
+        return validate_provenance({
+            "basis": "public_domain" if pd else "licensed", "source": source_url, "licence": licence,
+            "note": ("Attribution: " + attribution)[:1000]})
+    raise ValueError("unknown source_kind")

routes/sources.py

@@ -153,6 +153,8 @@ _PROXY_AUDIO_DOMAINS: set[str] = {
     "sample-files.com",
     "platform.r2.fish.audio",
     "fish.audio",
+    "archive.org",
+    "upload.wikimedia.org",
 }
 _GDRIVE_MIME_EXT = {
     "audio/mpeg": "mp3", "audio/mp3": "mp3",
@@ -533,6 +535,15 @@ async def voice_sources_custom(request: Request):
     return _voice_sources_payload(data.get("urls") if isinstance(data, dict) else None)
 
 
+@router.get("/api/voice-sources/{source}")
+async def open_voice_source_search(source: str, q: str = "", lang: str = ""):
+    from core import open_voice_sources as ovs
+    try:
+        return {"source": source, "items": await asyncio.to_thread(ovs.search, source, q, lang)}
+    except ovs.SourceError as e:
+        raise HTTPException(e.status, str(e))
+
+
 @router.post("/api/import-source-audio")
 async def import_source_audio(request: Request):
     data = await request.json()
@@ -594,6 +605,16 @@ async def proxy_audio(url: str):
 async def quick_import_voice(request: Request):
     data = await request.json()
     audio_url = _source_import_url(str(data.get("audio_url") or ""))
+    source_kind = str(data.get("source_kind") or "")
+    if source_kind:
+        from core import open_voice_sources as ovs
+        if not ovs.host_ok(audio_url, ovs.AUDIO_HOSTS):
+            raise HTTPException(400, "audio_url host is not allowed for this source")
+        try:
+            kind_prov = ovs.import_provenance_for(
+                source_kind, str(data.get("page_url") or ""), str(data.get("licence") or ""), str(data.get("attribution") or ""))
+        except ValueError as e:
+            raise HTTPException(400, str(e))
     voice_id = re.sub(r"[^A-Za-z0-9_\-\.]", "_", str(data.get("voice_id") or "").strip())[:80]
     if not voice_id:
         raise HTTPException(400, "voice_id is required")
@@ -619,6 +640,17 @@ async def quick_import_voice(request: Request):
                         raise HTTPException(413, "Downloaded audio is too large")
                     f.write(chunk)
         wav = _to_wav_24k(dest)
+        if source_kind:  # sample clip: trim to [start, start+length] (length 3..120 s)
+            from pydub import AudioSegment
+            try:
+                t0 = max(0.0, float(data.get("start") or 0))
+                ln = min(120.0, max(3.0, float(data.get("length") or 30)))
+            except (TypeError, ValueError):
+                raise HTTPException(400, "start/length must be numbers")
+            seg = AudioSegment.from_wav(str(wav))[int(t0 * 1000):int((t0 + ln) * 1000)]
+            if len(seg) < 1000:
+                raise HTTPException(400, "Trim range is outside the audio")
+            seg.export(str(wav), format="wav")
     except HTTPException:
         raise
     except Exception as e:
@@ -653,6 +685,10 @@ async def quick_import_voice(request: Request):
         if data.get(field) is not None:
             meta[field] = str(data.get(field) or "").strip()[:500]
     from core.provenance import import_provenance
+    if source_kind:
+        meta["provenance"] = kind_prov
+        meta["attribution"] = str(data.get("attribution") or "")[:500]
+    else:
         meta["provenance"] = import_provenance(str(data.get("page_url") or audio_url))  # basis stays "unknown"
     if auto_prepare and not transcript:
         meta["reference_transcript_status"] = "pending"

static/js/fishaudio-browser.js

@@ -245,7 +245,7 @@
 (function () {
   const tabs = document.getElementById('gvo-tabs');
   if (!tabs) return;
-  const map = { direct: 'tab-getvoices', fish: 'fa-browser-card', eleven: 'el-browser-card' };
+  const map = { direct: 'tab-getvoices', fish: 'fa-browser-card', open: 'open-sources-card', eleven: 'el-browser-card' };
   function show(src) {
     if (!map[src]) src = 'direct';
     Object.entries(map).forEach(([k, id]) => {

static/js/voice-sources.js

@@ -192,6 +192,62 @@ $('getvoices-reset-sources-btn')?.addEventListener('click', () => {
   renderGetVoices();
   $('getvoices-status').textContent = 'Source list reset.';
 });
+// -- LibriVox / Wikimedia Commons (server-side official-API search) ----------
+let _osItems = [];
+function osDuration(sec) { sec = Math.round(sec || 0); return sec ? `${Math.floor(sec / 60)}:${String(sec % 60).padStart(2, '0')}` : ''; }
+function renderOpenSources() {
+  $('os-list').innerHTML = _osItems.map((it, i) => `<div class="voice-source-card">
+    <div class="voice-source-title"><strong>${escHtml(it.title)}</strong><span>${escHtml([it.reader, it.language, osDuration(it.duration)].filter(Boolean).join(' · '))}</span></div>
+    <div class="voice-source-desc">Licence: ${it.licence_url ? `<a href="${escHtml(it.licence_url)}" target="_blank" rel="noopener">${escHtml(it.licence)}</a>` : escHtml(it.licence)}</div>
+    <audio controls preload="none" aria-label="Preview ${escHtml(it.title)}" src="/api/proxy-audio?url=${encodeURIComponent(it.audio_url)}"></audio>
+    <div class="voice-source-actions">
+      <a class="btn-secondary" href="${escHtml(it.source_url)}" target="_blank" rel="noopener">Source page</a>
+      <button class="btn-primary os-import" type="button" data-i="${i}">Import clip</button>
+    </div></div>`).join('');
+}
+async function searchOpenSources(ev) {
+  ev?.preventDefault();
+  const st = $('os-status'), btn = $('os-search-btn');
+  const q = ($('os-q').value || '').trim();
+  if (!q) { st.textContent = 'Enter a search term.'; return; }
+  btn.disabled = true; st.textContent = 'Searching...';
+  try {
+    const p = new URLSearchParams({q, lang: $('os-lang').value});
+    const r = await fetch(`/api/voice-sources/${encodeURIComponent($('os-source').value)}?${p}`);
+    const d = await r.json().catch(() => ({}));
+    if (!r.ok) throw new Error(d.detail || r.statusText);
+    _osItems = d.items || [];
+    renderOpenSources();
+    st.textContent = `${_osItems.length} result${_osItems.length === 1 ? '' : 's'}.`;
+  } catch (e) {
+    _osItems = []; renderOpenSources(); st.textContent = 'Search failed: ' + e.message;
+  } finally { btn.disabled = false; }
+}
+async function importOpenSource(it) {
+  const st = $('os-status');
+  const lang = sourceLanguageCode(it.language || $('os-lang').value || 'English');
+  const body = {
+    audio_url: it.audio_url, voice_id: suggestedVoiceIdFromSourceName(it.title, lang, 'N'), transcript: '', lang,
+    source_kind: it.source, page_url: it.source_url, licence: it.licence, attribution: it.attribution || '',
+    name: it.title, group: it.source, origin: 'cloned', start: +$('os-start').value || 0, length: +$('os-len').value || 30,
+    note: `${it.reader || ''} ${it.licence}`.trim(),
+  };
+  st.textContent = 'Importing...';
+  try {
+    const r = await fetch('/api/quick-import-voice', {method: 'POST', headers: {'Content-Type': 'application/json'}, body: JSON.stringify(body)});
+    const d = await r.json().catch(() => ({}));
+    if (!r.ok) throw new Error(d.detail || r.statusText);
+    st.textContent = `Saved as ${d.voice_id} (source and licence stored).`;
+    toast(`Saved as ${d.voice_id}`, 'success');
+    _libraryLoadPromise = null;
+  } catch (e) { st.textContent = 'Import failed: ' + e.message; toast('Import failed: ' + e.message, 'error'); }
+}
+$('os-form')?.addEventListener('submit', searchOpenSources);
+$('os-list')?.addEventListener('click', e => {
+  const b = e.target.closest('.os-import');
+  if (b && _osItems[+b.dataset.i]) importOpenSource(_osItems[+b.dataset.i]);
+});
+
 const SOURCE_LANGUAGE_CODES = {
   english:'EN', german:'DE', deutsch:'DE', french:'FR', spanish:'ES', japanese:'JA', korean:'KO',
   italian:'IT', portuguese:'PT', russian:'RU', arabic:'AR', polish:'PL', dutch:'NL', swedish:'SV',
@@ -304,7 +360,7 @@ Object.assign(root, {
   sourceTextareaValueFromDefaults, initGetVoiceSourcesEditor, getEditableVoiceSourceUrls,
   getVoiceSourceItems, voiceSourceSearchText, setOptions, renderGetVoices, loadGetVoices,
   SOURCE_LANGUAGE_CODES, sourceLanguageCode, sourceGenderCode, suggestedVoiceIdFromSourceName,
-  setLibAddSourcePreview, getSourceVoiceInLibrary,
+  setLibAddSourcePreview, getSourceVoiceInLibrary, searchOpenSources, importOpenSource,
 });
 root.TTVC = root.TTVC || {};
 root.TTVC.voiceSources = root.TTVC.voiceSources || {};

static/sections/s-studio.html

@@ -10,6 +10,7 @@
       <div class="gvo-tabs" id="gvo-tabs" role="tablist">
         <button class="gvo-tab active" data-src="direct" role="tab"><span class="mdi mdi-link-variant"></span> Direct sources <span class="gvo-tab-sub">scrape URLs</span></button>
         <button class="gvo-tab" data-src="fish" role="tab"><span class="mdi mdi-fishbowl-outline"></span> Fish.audio <span class="gvo-tab-sub">2M+ clonable</span></button>
+        <button class="gvo-tab" data-src="open" role="tab"><span class="mdi mdi-book-open-variant"></span> LibriVox &amp; Wikimedia <span class="gvo-tab-sub">open licences</span></button>
         <button class="gvo-tab" data-src="eleven" role="tab"><span class="gvo-eleven-logo">11</span> ElevenLabs <span class="gvo-tab-sub">12k shared</span></button>
       </div>
 
@@ -43,6 +44,21 @@
         </div>
       </div><!-- /tab-getvoices -->
 
+      <div class="card" id="open-sources-card" style="display:none">
+        <h2>LibriVox &amp; Wikimedia Commons</h2>
+        <p class="note">Official APIs only. Searching sends your query to librivox.org and archive.org (audio) or commons.wikimedia.org and upload.wikimedia.org; nothing else leaves this server. LibriVox recordings are public domain (USA); check the reader's licence on the LibriVox page. Wikimedia files carry their own licence; CC BY / CC BY-SA need attribution (saved with the voice). LibriVox searches match the start of a title; the language filter applies to LibriVox only. A search can take about 15 seconds.</p>
+        <form class="getvoices-toolbar" id="os-form">
+          <select id="os-source" aria-label="Source"><option value="librivox">LibriVox</option><option value="wikimedia">Wikimedia Commons</option></select>
+          <input type="search" id="os-q" maxlength="80" aria-label="Search term" placeholder="Title or search term" autocomplete="off" required>
+          <select id="os-lang" aria-label="Language (LibriVox)"><option value="">Language: any</option><option value="en">English</option><option value="de">German</option><option value="fr">French</option><option value="es">Spanish</option><option value="it">Italian</option><option value="nl">Dutch</option></select>
+          <label>Clip start (s) <input type="number" id="os-start" min="0" value="0" style="width:5em"></label>
+          <label>Length (s) <input type="number" id="os-len" min="3" max="120" value="30" style="width:5em"></label>
+          <button class="btn-primary" type="submit" id="os-search-btn">Search</button>
+        </form>
+        <p class="note" id="os-status" role="status" aria-live="polite"></p>
+        <div class="getvoices-grid" id="os-list"></div>
+      </div>
+
       <!-- ElevenLabs Voice Library Browser -->
       <div class="card el-browser" id="el-browser-card">
 

static/style.css

@@ -6026,3 +6026,8 @@ html[data-mag-night] #toast.success { color: var(--green); } html[data-mag-night
 .insp-prov-input:focus-visible, .insp-prov-panel button:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
 .insp-prov-panel button { min-height: 44px; min-width: 44px; }
 .insp-prov-rec { display: flex; flex-wrap: wrap; align-items: center; gap: 10px; }
+
+/* LibriVox / Wikimedia Commons search card: 44px touch targets */
+#open-sources-card .getvoices-toolbar { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; }
+#open-sources-card button, #open-sources-card input, #open-sources-card select { min-height: 44px; }
+#open-sources-card .voice-source-actions a { display: inline-flex; align-items: center; min-height: 44px; }

tests/test_voice_sources_new.py

@@ -0,0 +1,139 @@
+"""LibriVox / Wikimedia Commons voice sources: validation, caps, host checks, mapping, provenance, cache/throttle (HTTP mocked)."""
+import asyncio
+import os
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+os.environ.setdefault("CONFIG_DIR", tempfile.mkdtemp(prefix="ttvc-vs-test-config-"))
+from core import open_voice_sources as ovs
+import routes.sources as src
+
+
+def book(i=1, lang="English", url="https://www.archive.org/download/x/a.mp3"):
+    return {"title": f"Book {i}", "language": lang, "url_librivox": f"https://librivox.org/b{i}/",
+            "authors": [{"first_name": "Jane", "last_name": "Austen"}],
+            "sections": [{"title": "Ch 1", "listen_url": url, "playtime": "600", "readers": [{"display_name": "Ann"}]}]}
+
+
+def commons(**over):
+    info = {"url": "https://upload.wikimedia.org/a.opus", "descriptionurl": "https://commons.wikimedia.org/wiki/File:A.opus",
+            "mediatype": "AUDIO", "duration": 12.4,
+            "extmetadata": {"LicenseShortName": {"value": "CC BY-SA 4.0"}, "Artist": {"value": "<a href='x'>Bob &amp; Co</a>"},
+                            "LicenseUrl": {"value": "https://creativecommons.org/licenses/by-sa/4.0"}}}
+    info.update(over)
+    return {"query": {"pages": {"1": {"title": "File:A.opus", "index": 1, "imageinfo": [info]}}}}
+
+
+class Base(unittest.TestCase):
+    def setUp(self):
+        ovs._cache.clear()
+        ovs._last_call.clear()
+        p = mock.patch.object(ovs.time, "sleep")
+        p.start()
+        self.addCleanup(p.stop)
+
+
+class ValidationTests(Base):
+    def test_query_and_lang(self):
+        for bad in ("", "   ", "x" * 81):
+            with self.assertRaises(ovs.SourceError):
+                ovs.clean_query(bad)
+        self.assertEqual(ovs.clean_query(" a\x00 \n b "), "a b")
+        with self.assertRaises(ovs.SourceError):
+            ovs.clean_lang("en;rm")
+        with self.assertRaises(ovs.SourceError):
+            ovs.search("evil", "x")
+
+    def test_host_check(self):
+        self.assertTrue(ovs.host_ok("https://ia80.us.archive.org/x.mp3", ovs.AUDIO_HOSTS))
+        for bad in ("http://archive.org/x", "https://evilarchive.org/x", "https://archive.org.evil.com/x",
+                    "https://u:p@archive.org/x", "https://127.0.0.1/x", "file:///etc/passwd"):
+            self.assertFalse(ovs.host_ok(bad, ovs.AUDIO_HOSTS), bad)
+
+
+class MappingTests(Base):
+    def test_librivox_mapping_filter_cap_and_bad_hosts(self):
+        books = [book(i) for i in range(15)] + [book(99, url="https://evil.example/a.mp3"), book(98, lang="German")]
+        with mock.patch.object(ovs, "_get_json", return_value={"books": books}) as g:
+            items = ovs.search("librivox", "pride", "en")
+        self.assertEqual(len(items), ovs.MAX_RESULTS)
+        self.assertEqual(g.call_args[0][0], ovs.LIBRIVOX_API)
+        it = items[0]
+        self.assertEqual((it["reader"], it["author"], it["language"], it["duration"]), ("Ann", "Jane Austen", "English", 600))
+        self.assertTrue(it["licence_url"].startswith("https://librivox.org/"))
+        self.assertTrue(all("evil" not in i["audio_url"] and i["language"] == "English" for i in items))
+
+    def test_wikimedia_mapping(self):
+        with mock.patch.object(ovs, "_get_json", return_value=commons()):
+            it = ovs.search("wikimedia", "speech")[0]
+        self.assertEqual((it["title"], it["reader"], it["licence"], it["duration"]), ("A", "Bob & Co", "CC BY-SA 4.0", 12))
+        self.assertIn("Bob & Co / A, CC BY-SA 4.0", it["attribution"])
+
+    def test_wikimedia_drops_unsafe_or_unlicensed(self):
+        for over in ({"url": "https://evil.example/a.opus"}, {"mediatype": "VIDEO"}, {"extmetadata": {}}):
+            ovs._cache.clear()
+            with mock.patch.object(ovs, "_get_json", return_value=commons(**over)):
+                self.assertEqual(ovs.search("wikimedia", "speech"), [])
+
+
+class ProvenanceTests(Base):
+    def test_librivox(self):
+        p = ovs.import_provenance_for("librivox", "https://librivox.org/b/", "", "")
+        self.assertEqual(p["basis"], "public_domain")
+        self.assertIn("reader's declared licence", p["note"])
+        self.assertEqual(p["source"], "https://librivox.org/b/")
+
+    def test_wikimedia(self):
+        p = ovs.import_provenance_for("wikimedia", "https://commons.wikimedia.org/wiki/File:A.opus", "CC BY 4.0", "Bob / A")
+        self.assertEqual((p["basis"], p["licence"]), ("licensed", "CC BY 4.0"))
+        self.assertIn("Bob / A", p["note"])
+        self.assertEqual(ovs.import_provenance_for("wikimedia", "https://c.org/", "Public domain", "")["basis"], "public_domain")
+        with self.assertRaises(ValueError):
+            ovs.import_provenance_for("other", "", "", "")
+
+
+class CacheThrottleTests(Base):
+    def test_cache_hit_skips_upstream(self):
+        with mock.patch.object(ovs, "_get_json", return_value={"books": [book()]}) as g:
+            ovs.search("librivox", "Pride"); ovs.search("librivox", "pride")
+        self.assertEqual(g.call_count, 1)
+
+    def test_throttle_sleeps_and_errors_are_generic(self):
+        resp = mock.Mock(status_code=200)
+        resp.raise_for_status.return_value = None
+        resp.iter_content.return_value = [b'{"books": []}']
+        with mock.patch.object(ovs.requests, "get", return_value=resp) as g:
+            ovs._get_json(ovs.LIBRIVOX_API, {}); ovs._get_json(ovs.LIBRIVOX_API, {})
+        self.assertTrue(ovs.time.sleep.called)
+        self.assertFalse(g.call_args.kwargs["allow_redirects"])
+        self.assertIn("User-Agent", g.call_args.kwargs["headers"])
+        with mock.patch.object(ovs.requests, "get", side_effect=OSError("secret host")):
+            with self.assertRaises(ovs.SourceError) as cm:
+                ovs._get_json(ovs.COMMONS_API, {})
+        self.assertNotIn("secret", str(cm.exception))
+
+
+class RouteTests(Base):
+    def test_search_route_errors_and_allowlist(self):
+        with self.assertRaises(src.HTTPException) as cm:
+            asyncio.run(src.open_voice_source_search("librivox", ""))
+        self.assertEqual(cm.exception.status_code, 400)
+        self.assertIn("archive.org", src._PROXY_AUDIO_DOMAINS)
+        self.assertIn("upload.wikimedia.org", src._PROXY_AUDIO_DOMAINS)
+
+    def test_quick_import_rejects_foreign_host(self):
+        req = mock.Mock()
+        async def js(): return {"audio_url": "https://example.com/a.mp3", "voice_id": "x", "source_kind": "librivox"}
+        req.json = js
+        with mock.patch.object(src, "_source_import_url", side_effect=lambda u: u):
+            with self.assertRaises(src.HTTPException) as cm:
+                asyncio.run(src.quick_import_voice(req))
+        self.assertEqual(cm.exception.status_code, 400)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/voice-sources-new.spec.js

@@ -0,0 +1,43 @@
+// @ts-check
+// LibriVox / Wikimedia search + import in "Get Voices Online" (mocked routes only).
+const { test, expect } = require('@playwright/test');
+
+const ITEMS = [{ source: 'librivox', title: 'Pride - Ch 1', reader: 'Ann', author: 'Jane Austen', language: 'English', duration: 600,
+  licence: 'Public domain (LibriVox, USA)', licence_url: 'https://librivox.org/pages/public-domain/', source_url: 'https://librivox.org/b/',
+  audio_url: 'https://www.archive.org/download/x/a.mp3', attribution: '' }];
+
+test('search shows licence, import sends provenance fields', async ({ page }) => {
+  let imported = null, searched = '';
+  await page.route('**/api/voice-sources/librivox*', r => { searched = r.request().url(); return r.fulfill({ json: { source: 'librivox', items: ITEMS } }); });
+  await page.route('**/api/proxy-audio*', r => r.fulfill({ body: '', contentType: 'audio/mpeg' }));
+  await page.route('**/api/quick-import-voice', r => { imported = r.request().postDataJSON(); return r.fulfill({ json: { voice_id: 'EN_N_Pride' } }); });
+  await page.route('**/api/voices*', r => r.fulfill({ json: [] }));
+  await page.route('**/api/settings*', r => r.fulfill({ json: {} }));
+  await page.goto('/', { waitUntil: 'domcontentloaded' });
+  await page.waitForFunction(() => typeof window.navTo === 'function' && !!document.getElementById('os-form'), null, { timeout: 15_000 });
+  await page.evaluate(() => window.navTo('s-studio'));
+  await page.locator('.gvo-tab[data-src="open"]').click();
+  await page.fill('#os-q', 'pride');
+  await page.press('#os-q', 'Enter');
+  await expect(page.locator('#os-list')).toContainText('Public domain (LibriVox, USA)');
+  expect(searched).toContain('q=pride');
+  const b = page.locator('.os-import');
+  const h = await b.evaluate(el => el.getBoundingClientRect().height);
+  expect(h).toBeGreaterThanOrEqual(44);
+  await b.click();
+  await expect(page.locator('#os-status')).toContainText('Saved as EN_N_Pride');
+  expect(imported).toMatchObject({ source_kind: 'librivox', page_url: 'https://librivox.org/b/', start: 0, length: 30 });
+});
+
+test('search error is shown in the status region', async ({ page }) => {
+  await page.route('**/api/voice-sources/wikimedia*', r => r.fulfill({ status: 502, json: { detail: 'wikimedia is unavailable right now' } }));
+  await page.route('**/api/voices*', r => r.fulfill({ json: [] }));
+  await page.goto('/', { waitUntil: 'domcontentloaded' });
+  await page.waitForFunction(() => typeof window.navTo === 'function' && !!document.getElementById('os-form'), null, { timeout: 15_000 });
+  await page.evaluate(() => window.navTo('s-studio'));
+  await page.locator('.gvo-tab[data-src="open"]').click();
+  await page.selectOption('#os-source', 'wikimedia');
+  await page.fill('#os-q', 'speech');
+  await page.evaluate(() => document.getElementById('os-form').requestSubmit());
+  await expect(page.locator('#os-status')).toContainText('unavailable');
+});