mARTin-B78/tts-voice-creator-clone-and-design-2

mARTin-B78 c3c5c55f17 feat(magazine): find and import issues from the Internet Archive and OPDS catalogues

Also reuses the existing magFmtClock for the contents list (the duplicate
tripped the frontend collision gate).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

2026-09-29 17:35:08 +02:00

2.7 KiB

Raw Blame History

Magazine sources (find and import an issue)

Status: implemented (core/magazine_sources.py, routes/magazine_sources.py, magazine.js "Find an issue online").

Goal

Search legitimate catalogues from inside the app and import an issue as a normal magazine project. Ingestion stays a separate, existing step, so nothing is narrated before the review.

Providers (one interface: `search`, `resolve_pdf`, `allowed_host`)

Provider	Kind	Notes
Internet Archive	fixed	documented advanced-search + metadata API; loan-only items excluded (`-access-restricted-item`); prefers the original PDF over derived text PDFs
OPDS catalogue	user-configured (`magazine_opds_catalogs` setting)	standard Atom feed; search via the catalogue's own OpenSearch template; only PDF acquisition links

Deliberately unsupported: sites that redistribute commercial titles without the publisher's licence (see TODO.md). Candidates later: publisher free issues / feeds, DOAJ/arXiv/PMC. Each would be another provider.

API

GET /api/magazine-sources -> {sources:[{key,label,kind}]}
GET /api/magazine-sources/{key}/search?q=&page= -> {items:[{id,title,author,date,summary,license,page_url,thumb_url}]}; 404 unknown source, 502 with a safe message when the source fails
POST /api/magazine-sources/import {source,item_id,title} -> {id,job_id}; the download runs as a background job (magazine-source-import, poll /api/jobs/{id}); 409 while another import runs

Security boundaries

SSRF: a download is fetched only from the provider's own host (archive.org, or the catalogue's host). Item ids are validated (IA identifier regex; OPDS id must be same-host). Redirects are followed manually and re-checked.
XML: feeds with a DOCTYPE or entity declaration are refused (billion-laughs / XXE class); feed size capped at 4 MB.
Limits: 400 MB per PDF, connect/read timeouts, one import at a time, %PDF magic checked on the first chunk, temp file removed on failure.
No credentials are stored or sent; catalogue URLs with embedded credentials are rejected.
Remote text is rendered as text in the UI (escaped), links open with noopener noreferrer.
Rate limits/terms: the Internet Archive asks for a descriptive User-Agent and modest request rates; searches are user-initiated, one page of 20 per request.

Error behaviour

Every provider failure is a SourceError with a user-safe message (unreachable, HTTP status, rate-limited, not a PDF, too large, different host).

Offline / sync

Needs the network only for search and download. No synchronisation state.

Known limitations

OPDS catalogues requiring login; OPDS pagination beyond the first feed page; no thumbnails shown yet (returned in the API).

2.7 KiB Raw Blame History