tts-voice-creator-clone-and-design-2/architecture/voice-sources.md

Voice sources: LibriVox and Wikimedia Commons

Status: implemented. Code: core/open_voice_sources.py, routes in routes/sources.py, UI in static/js/voice-sources.js (tab "LibriVox & Wikimedia" in Get Voices Online).

Verified API facts (checked 2026-10-04, docs and live calls)

LibriVox

• GET https://librivox.org/api/feed/audiobooks?title=^<q>&format=json&limit=15 (light list, no extended; the API takes 10-40 s even so). Audio is resolved per book on demand: ?id=<n>&format=json&extended=1, first section whose listen_url is on an allowed host, exposed as GET /api/voice-sources/librivox/book/{id} ({audio_url, section, reader, duration}). Upstream calls: timeout 25 s, one retry after 1.5 s, failures cached 30 s, 502 messages distinguish timeout / unreachable / HTTP status / unreadable; logs carry exception class and query length only. A bare title= is an EXACT match (404 {"error":...} otherwise); the ^ prefix gives a prefix match. No language parameter: the language is filtered client-side on language ("English"). extended=1 adds sections[] with listen_url (archive.org MP3, 64 kbps), playtime (s) and readers[].
• The docs state no rate limit and no terms. The extended feed is slow (about 14 s for 5-25 books), so timeout is 30 s, limit 15, results cached 10 min, one upstream call per second.
• Licence (librivox.org/pages/public-domain): recordings are donated to the public domain, use without permission or attribution allowed (credit appreciated). Caveats: guaranteed public domain only under US law; source texts are pre-1923 works; translations are separate works; readers are volunteers and the page of a project may show a reader-specific note, so the UI/provenance note tells the user to check the reader's declared licence. A reader's voice is still a person's voice: public domain is not consent for voice cloning of a recognisable person; the provenance basis is a record, not a legal clearance.
• Audio is hosted on archive.org (www.archive.org/download/... redirects to iaNNN.us.archive.org).

Wikimedia Commons

• GET https://commons.wikimedia.org/w/api.php?action=query&generator=search&gsrnamespace=6&gsrsearch=filetype:audio <q>&prop=imageinfo&iiprop=url|extmetadata|mediatype&maxlag=5. Per-file extmetadata: LicenseShortName, LicenseUrl, Artist (HTML, stripped), AttributionRequired. inlanguage: does not filter audio files, so lang is ignored for Wikimedia.
• Etiquette (mediawiki.org API:Etiquette): descriptive User-Agent with contact info is mandatory, requests serial, use maxlag, back off on errors; clients can be blocked without notice. Our UA is TTSVoiceCreator/1.0 (self-hosted voice library; official-API client) (no personal data; Wikimedia prefers a contact URL/email, set one via a future env var if blocked or high volume).
• Licences are per file. Attribution duties: CC BY and CC BY-SA require author, title, licence (with link) and a note of changes (trimming/cloning counts); CC BY-SA also requires derivatives under the same licence (a cloned voice model is arguably a derivative: treat as unresolved, prefer CC0/public domain/CC BY for anything published). Public domain/CC0: none. Some files carry non-copyright restrictions (personality rights).

NOT safe / not done

• No HTML scraping of librivox.org, archive.org or commons pages; JSON APIs only.
• No user-supplied URLs: API hosts are constants; every URL from a response must be https on an allowlisted host (archive.org, upload.wikimedia.org, page hosts librivox.org, commons.wikimedia.org) or the item is dropped. quick-import-voice with source_kind re-checks the audio host.
• Redirects are not followed for API calls; generic /api/proxy-audio (existing) follows redirects and now also allows the two audio hosts.
• Files without a licence in extmetadata are dropped. The licence string is trusted from Commons metadata and can be wrong; the user must still check.
• Licence/attribution values sent by the browser at import are only length-capped, not re-fetched (single-user LAN tool).

Endpoints

GET /api/voice-sources/{librivox|wikimedia}?q=&lang= : q required, max 80 chars, control chars stripped; lang 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: {source, book_id (librivox), title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url (librivox: empty until resolved), attribution}. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server).

POST /api/quick-import-voice accepts source_kind (librivox|wikimedia), licence, attribution, page_url, start, length (clip, 3-120 s, default 0/30). Provenance: LibriVox -> public_domain, source = book page, note says to check the reader's licence. Wikimedia -> licensed (or public_domain when licence says public domain/CC0), licence, note Attribution: ...; meta.attribution also stored.

Cache and throttle

In-process dict, TTL 600 s, max 200 entries (evict oldest); per-source lock with at least 1 s between upstream calls. Lost on restart; fine for one container.