51 lines
3.8 KiB
Markdown
51 lines
3.8 KiB
Markdown
# Magazine sources (find and import an issue)
|
|
|
|
Status: implemented (core/magazine_sources.py, routes/magazine_sources.py, magazine.js "Find an issue online").
|
|
|
|
## Goal
|
|
Search legitimate catalogues from inside the app and import an issue as a normal magazine project. Ingestion
|
|
stays a separate, existing step, so nothing is narrated before the review.
|
|
|
|
## Providers (one interface: `search`, `resolve_pdf`, `allowed_host`)
|
|
| Provider | Kind | Notes |
|
|
|---|---|---|
|
|
| Internet Archive | fixed | documented advanced-search + metadata API; loan-only items excluded (`-access-restricted-item`); prefers the original PDF over derived text PDFs |
|
|
| OPDS catalogue | user-configured (`magazine_opds_catalogs` setting) | standard Atom feed; search via the catalogue's own OpenSearch template; only PDF acquisition links |
|
|
|
|
Deliberately unsupported: sites that redistribute commercial titles without the publisher's licence (see TODO.md).
|
|
Candidates later: publisher free issues / feeds, DOAJ/arXiv/PMC. Each would be another provider.
|
|
|
|
## API
|
|
- `GET /api/magazine-sources` -> `{sources:[{key,label,kind}]}`
|
|
- `GET /api/magazine-sources/{key}/search?q=&page=` -> `{items:[{id,title,author,date,summary,license,page_url,thumb_url}]}`; 404 unknown source, 502 with a safe message when the source fails
|
|
- `POST /api/magazine-sources/import {source,item_id,title}` -> `{id,job_id}`; the download runs as a background job (`magazine-source-import`, poll `/api/jobs/{id}`); 409 while another import runs
|
|
|
|
## Security boundaries
|
|
- SSRF: a download is fetched only from the provider's own host (archive.org, or the catalogue's host). Item ids are validated (IA identifier regex; OPDS id must be same-host). Redirects are followed manually and re-checked.
|
|
- XML: feeds with a DOCTYPE or entity declaration are refused (billion-laughs / XXE class); feed size capped at 4 MB.
|
|
- Limits: 400 MB per PDF, connect/read timeouts, one import at a time, `%PDF` magic checked on the first chunk, temp file removed on failure.
|
|
- No credentials are stored or sent; catalogue URLs with embedded credentials are rejected.
|
|
- Remote text is rendered as text in the UI (escaped), links open with `noopener noreferrer`.
|
|
- Rate limits/terms: the Internet Archive asks for a descriptive User-Agent and modest request rates; searches are user-initiated, one page of 20 per request.
|
|
|
|
## Error behaviour
|
|
Every provider failure is a `SourceError` with a user-safe message (unreachable, HTTP status, rate-limited, not a PDF, too large, different host).
|
|
|
|
## Offline / sync
|
|
Needs the network only for search and download. No synchronisation state.
|
|
|
|
## Catalogues, paging and login (2026-09-30)
|
|
- `GET/PUT /api/magazine-sources/catalogues` manage the OPDS list, validated at the boundary (<= 10; `http(s)` with a host; no credentials in the
|
|
address; a login needs both a username and `password_env`, a capitalised variable name; duplicates and wrong shapes are refused and
|
|
change nothing). The GET tells whether the named variable is set, never its value.
|
|
- Login is HTTP Basic: username from the settings, password read from the environment variable named by `password_env`. It is sent only
|
|
to the catalogue's own host (every hop, including redirects, must match it; a cross-host redirect or `next` link is refused), used for
|
|
both search and download, and appears in no response, log line or error message.
|
|
- A search walks the catalogue's `rel="next"` links (at most 5 result pages, loop-protected, same host only) until the requested page of 20
|
|
can be filled, since entries without a PDF are dropped.
|
|
|
|
## Known limitations
|
|
No thumbnails shown yet (returned in the API). Basic auth only (no OAuth/cookies). Each search page re-reads the catalogue from its first page
|
|
(bounded at 5), there is no result cache. Anyone who can edit the catalogue list can name any environment variable as the password source,
|
|
so the list is an administrator setting.
|