31 lines
5.1 KiB
Markdown
31 lines
5.1 KiB
Markdown
# Voice sources: LibriVox and Wikimedia Commons
|
|
|
|
Status: implemented. Code: `core/open_voice_sources.py`, routes in `routes/sources.py`, UI in `static/js/voice-sources.js` (tab "LibriVox & Wikimedia" in Get Voices Online).
|
|
|
|
## Verified API facts (checked 2026-10-04, docs and live calls)
|
|
LibriVox
|
|
- `GET https://librivox.org/api/feed/audiobooks?title=^<q>&format=json&limit=15` (light list, no `extended`; the API takes 10-40 s even so). Audio is resolved per book on demand: `?id=<n>&format=json&extended=1`, first section whose `listen_url` is on an allowed host, exposed as `GET /api/voice-sources/librivox/book/{id}` (`{audio_url, section, reader, duration}`). Upstream calls: timeout 25 s, one retry after 1.5 s, failures cached 30 s, 502 messages distinguish timeout / unreachable / HTTP status / unreadable; logs carry exception class and query length only. A bare `title=` is an EXACT match (404 `{"error":...}` otherwise); the `^` prefix gives a prefix match. No language parameter: the language is filtered client-side on `language` ("English"). `extended=1` adds `sections[]` with `listen_url` (archive.org MP3, 64 kbps), `playtime` (s) and `readers[]`.
|
|
- The docs state no rate limit and no terms. The extended feed is slow (about 14 s for 5-25 books), so timeout is 30 s, limit 15, results cached 10 min, one upstream call per second.
|
|
- Licence (librivox.org/pages/public-domain): recordings are donated to the public domain, use without permission or attribution allowed (credit appreciated). Caveats: guaranteed public domain only under US law; source texts are pre-1923 works; translations are separate works; readers are volunteers and the page of a project may show a reader-specific note, so the UI/provenance note tells the user to check the reader's declared licence. A reader's voice is still a person's voice: public domain is not consent for voice cloning of a recognisable person; the provenance basis is a record, not a legal clearance.
|
|
- Audio is hosted on archive.org (`www.archive.org/download/...` redirects to `iaNNN.us.archive.org`).
|
|
|
|
Wikimedia Commons
|
|
- `GET https://commons.wikimedia.org/w/api.php?action=query&generator=search&gsrnamespace=6&gsrsearch=filetype:audio <q>&prop=imageinfo&iiprop=url|extmetadata|mediatype&maxlag=5`. Per-file `extmetadata`: `LicenseShortName`, `LicenseUrl`, `Artist` (HTML, stripped), `AttributionRequired`. `inlanguage:` does not filter audio files, so `lang` is ignored for Wikimedia.
|
|
- Etiquette (mediawiki.org API:Etiquette): descriptive User-Agent with contact info is mandatory, requests serial, use `maxlag`, back off on errors; clients can be blocked without notice. Our UA is `TTSVoiceCreator/1.0 (self-hosted voice library; official-API client)` (no personal data; Wikimedia prefers a contact URL/email, set one via a future env var if blocked or high volume).
|
|
- Licences are per file. Attribution duties: CC BY and CC BY-SA require author, title, licence (with link) and a note of changes (trimming/cloning counts); CC BY-SA also requires derivatives under the same licence (a cloned voice model is arguably a derivative: treat as unresolved, prefer CC0/public domain/CC BY for anything published). Public domain/CC0: none. Some files carry non-copyright restrictions (personality rights).
|
|
|
|
## NOT safe / not done
|
|
- No HTML scraping of librivox.org, archive.org or commons pages; JSON APIs only.
|
|
- No user-supplied URLs: API hosts are constants; every URL from a response must be https on an allowlisted host (`archive.org`, `upload.wikimedia.org`, page hosts `librivox.org`, `commons.wikimedia.org`) or the item is dropped. `quick-import-voice` with `source_kind` re-checks the audio host.
|
|
- Redirects are not followed for API calls; generic `/api/proxy-audio` (existing) follows redirects and now also allows the two audio hosts.
|
|
- Files without a licence in `extmetadata` are dropped. The licence string is trusted from Commons metadata and can be wrong; the user must still check.
|
|
- Licence/attribution values sent by the browser at import are only length-capped, not re-fetched (single-user LAN tool).
|
|
|
|
## Endpoints
|
|
`GET /api/voice-sources/{librivox|wikimedia}?q=&lang=` : `q` required, max 80 chars, control chars stripped; `lang` 2-3 letters; max 10 items; 400 on bad input, 502 with a generic message on upstream failure. Item: `{source, book_id (librivox), title, reader, author, language, duration (s), licence, licence_url, source_url, audio_url (librivox: empty until resolved), attribution}`. Privacy: the query goes to librivox.org / commons.wikimedia.org; previews and import go to archive.org / upload.wikimedia.org (via this server).
|
|
|
|
`POST /api/quick-import-voice` accepts `source_kind` (`librivox`|`wikimedia`), `licence`, `attribution`, `page_url`, `start`, `length` (clip, 3-120 s, default 0/30). Provenance: LibriVox -> `public_domain`, source = book page, note says to check the reader's licence. Wikimedia -> `licensed` (or `public_domain` when licence says public domain/CC0), licence, note `Attribution: ...`; `meta.attribution` also stored.
|
|
|
|
## Cache and throttle
|
|
In-process dict, TTL 600 s, max 200 entries (evict oldest); per-source lock with at least 1 s between upstream calls. Lost on restart; fine for one container.
|