3.8 KiB
Magazine sources (find and import an issue)
Status: implemented (core/magazine_sources.py, routes/magazine_sources.py, magazine.js "Find an issue online").
Goal
Search legitimate catalogues from inside the app and import an issue as a normal magazine project. Ingestion stays a separate, existing step, so nothing is narrated before the review.
Providers (one interface: search, resolve_pdf, allowed_host)
| Provider | Kind | Notes |
|---|---|---|
| Internet Archive | fixed | documented advanced-search + metadata API; loan-only items excluded (-access-restricted-item); prefers the original PDF over derived text PDFs |
| OPDS catalogue | user-configured (magazine_opds_catalogs setting) |
standard Atom feed; search via the catalogue's own OpenSearch template; only PDF acquisition links |
Deliberately unsupported: sites that redistribute commercial titles without the publisher's licence (see TODO.md). Candidates later: publisher free issues / feeds, DOAJ/arXiv/PMC. Each would be another provider.
API
GET /api/magazine-sources->{sources:[{key,label,kind}]}GET /api/magazine-sources/{key}/search?q=&page=->{items:[{id,title,author,date,summary,license,page_url,thumb_url}]}; 404 unknown source, 502 with a safe message when the source failsPOST /api/magazine-sources/import {source,item_id,title}->{id,job_id}; the download runs as a background job (magazine-source-import, poll/api/jobs/{id}); 409 while another import runs
Security boundaries
- SSRF: a download is fetched only from the provider's own host (archive.org, or the catalogue's host). Item ids are validated (IA identifier regex; OPDS id must be same-host). Redirects are followed manually and re-checked.
- XML: feeds with a DOCTYPE or entity declaration are refused (billion-laughs / XXE class); feed size capped at 4 MB.
- Limits: 400 MB per PDF, connect/read timeouts, one import at a time,
%PDFmagic checked on the first chunk, temp file removed on failure. - No credentials are stored or sent; catalogue URLs with embedded credentials are rejected.
- Remote text is rendered as text in the UI (escaped), links open with
noopener noreferrer. - Rate limits/terms: the Internet Archive asks for a descriptive User-Agent and modest request rates; searches are user-initiated, one page of 20 per request.
Error behaviour
Every provider failure is a SourceError with a user-safe message (unreachable, HTTP status, rate-limited, not a PDF, too large, different host).
Offline / sync
Needs the network only for search and download. No synchronisation state.
Catalogues, paging and login (2026-09-30)
GET/PUT /api/magazine-sources/cataloguesmanage the OPDS list, validated at the boundary (<= 10;http(s)with a host; no credentials in the address; a login needs both a username andpassword_env, a capitalised variable name; duplicates and wrong shapes are refused and change nothing). The GET tells whether the named variable is set, never its value.- Login is HTTP Basic: username from the settings, password read from the environment variable named by
password_env. It is sent only to the catalogue's own host (every hop, including redirects, must match it; a cross-host redirect ornextlink is refused), used for both search and download, and appears in no response, log line or error message. - A search walks the catalogue's
rel="next"links (at most 5 result pages, loop-protected, same host only) until the requested page of 20 can be filled, since entries without a PDF are dropped.
Known limitations
No thumbnails shown yet (returned in the API). Basic auth only (no OAuth/cookies). Each search page re-reads the catalogue from its first page (bounded at 5), there is no result cache. Anyone who can edit the catalogue list can name any environment variable as the password source, so the list is an administrator setting.