mARTin-B78
c3c5c55f17
Also reuses the existing magFmtClock for the contents list (the duplicate tripped the frontend collision gate). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
39 lines
2.7 KiB
Markdown
39 lines
2.7 KiB
Markdown
# Magazine sources (find and import an issue)
|
|
|
|
Status: implemented (core/magazine_sources.py, routes/magazine_sources.py, magazine.js "Find an issue online").
|
|
|
|
## Goal
|
|
Search legitimate catalogues from inside the app and import an issue as a normal magazine project. Ingestion
|
|
stays a separate, existing step, so nothing is narrated before the review.
|
|
|
|
## Providers (one interface: `search`, `resolve_pdf`, `allowed_host`)
|
|
| Provider | Kind | Notes |
|
|
|---|---|---|
|
|
| Internet Archive | fixed | documented advanced-search + metadata API; loan-only items excluded (`-access-restricted-item`); prefers the original PDF over derived text PDFs |
|
|
| OPDS catalogue | user-configured (`magazine_opds_catalogs` setting) | standard Atom feed; search via the catalogue's own OpenSearch template; only PDF acquisition links |
|
|
|
|
Deliberately unsupported: sites that redistribute commercial titles without the publisher's licence (see TODO.md).
|
|
Candidates later: publisher free issues / feeds, DOAJ/arXiv/PMC. Each would be another provider.
|
|
|
|
## API
|
|
- `GET /api/magazine-sources` -> `{sources:[{key,label,kind}]}`
|
|
- `GET /api/magazine-sources/{key}/search?q=&page=` -> `{items:[{id,title,author,date,summary,license,page_url,thumb_url}]}`; 404 unknown source, 502 with a safe message when the source fails
|
|
- `POST /api/magazine-sources/import {source,item_id,title}` -> `{id,job_id}`; the download runs as a background job (`magazine-source-import`, poll `/api/jobs/{id}`); 409 while another import runs
|
|
|
|
## Security boundaries
|
|
- SSRF: a download is fetched only from the provider's own host (archive.org, or the catalogue's host). Item ids are validated (IA identifier regex; OPDS id must be same-host). Redirects are followed manually and re-checked.
|
|
- XML: feeds with a DOCTYPE or entity declaration are refused (billion-laughs / XXE class); feed size capped at 4 MB.
|
|
- Limits: 400 MB per PDF, connect/read timeouts, one import at a time, `%PDF` magic checked on the first chunk, temp file removed on failure.
|
|
- No credentials are stored or sent; catalogue URLs with embedded credentials are rejected.
|
|
- Remote text is rendered as text in the UI (escaped), links open with `noopener noreferrer`.
|
|
- Rate limits/terms: the Internet Archive asks for a descriptive User-Agent and modest request rates; searches are user-initiated, one page of 20 per request.
|
|
|
|
## Error behaviour
|
|
Every provider failure is a `SourceError` with a user-safe message (unreachable, HTTP status, rate-limited, not a PDF, too large, different host).
|
|
|
|
## Offline / sync
|
|
Needs the network only for search and download. No synchronisation state.
|
|
|
|
## Known limitations
|
|
OPDS catalogues requiring login; OPDS pagination beyond the first feed page; no thumbnails shown yet (returned in the API).
|