diff --git a/CHANGELOG.md b/CHANGELOG.md index b93877d..1eca82c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ### Fixed - **Voice benchmark failures now say why.** Designed voices are benchmarked through the Voice Design engine; when that service is down the row showed a bare "-" and the raw "Max retries exceeded" text. The error now names the service and the container to start, the Factor and WPM columns show `ERR` beside the time column, and a batch stops after three identical "unreachable" failures instead of retrying every remaining voice. ### Added +- **Find an issue online (magazines).** A search panel in the magazine library finds issues in the Internet Archive and in any OPDS catalogue you configure (Calibre-Web, Kavita, Komga), shows the source's licence, and imports the PDF straight into a new magazine project, then starts ingestion. SSRF-safe (downloads only from the provider's own host), size/time bounded, feeds with DTDs refused. See `architecture/magazine-sources.md`. - **Magazine contents list shows synthesis state.** Each article row says "Synthesised · m:ss" (the MP3's exact length) or "Not synthesised" (approved, no MP3 yet), and has an inline play/pause button that streams the MP3 without opening the article. Icon plus words, screen-reader labelled, works at phone width. - `scripts/compare_magazines.py`: read-only comparison of two magazine folders (words, boxes, reading order, articles, quality numbers) for re-import checks. diff --git a/Manual.md b/Manual.md index 9f9432e..bc7b3fe 100644 --- a/Manual.md +++ b/Manual.md @@ -338,6 +338,35 @@ Open an article and expand **Pronunciation & spelling** under the player. **Mobile / accessibility:** the lists are ordinary buttons with labels ("Replace X with Y"); marks are underlined, not colour-only. **Privacy:** the agent sends only the flagged word and its surrounding sentence to the LLM you configured in Settings; without an LLM nothing leaves the machine. +### Find an issue online + +**Purpose.** Pull a magazine issue into the app without leaving for a browser and uploading by hand. + +**Prerequisites.** Internet access for the Internet Archive; for your own library, an OPDS catalogue +(Calibre-Web, Kavita or Komga all serve one) added under `magazine_opds_catalogs` in the app settings as +`[{"name": "NAS", "url": "http://nas.local:8083/opds/"}]`. Catalogues that need a login are not supported yet. + +**Steps.** +1. Open Magazines. In *Find an issue online* pick a source, type a title or subject, press Search. +2. Read the licence line on each result. The source states it, the app does not judge it. +3. Press *Import* on a result. The PDF downloads (progress is announced under the search box), the magazine + opens and ingestion starts, exactly as after an upload. + +**Expected result.** A new magazine project with the PDF stored locally, then the normal review-before-narration flow. + +**Errors.** A source that is down, rate-limiting or returns something other than a PDF gives a plain message +under the search box and nothing is created that you must clean up beyond the empty project shown in your library. +Lending-only Internet Archive items are never listed. Only one download runs at a time; a PDF larger than +400 MB is refused. + +**Mobile / accessibility.** The form stacks on a phone; results are a list with labelled Import buttons; status +text is a polite live region, so a screen reader hears "N results" and download progress. + +**Offline.** Search and import need the network; everything after import works offline. + +**Privacy.** Your search words go to the chosen source (archive.org, or your own catalogue). Nothing else is +sent. Websites that redistribute commercial magazines without a licence are deliberately not supported. + ### Quality check / run (whole magazine) **Purpose:** find every page where text is not inside a box, boxes overlap or cut off letters, or OCR still looks wrong. diff --git a/TODO.md b/TODO.md index 86f609a..48b2719 100644 --- a/TODO.md +++ b/TODO.md @@ -745,7 +745,7 @@ casting. The original PDF remains the visual source of truth while approved arti ### P2 — Finding magazines to read -- [ ] **Browse and search online PDF magazine sources from inside the app**, so an issue can be found and pulled in +- [~] **Browse and search online PDF magazine sources from inside the app** *(done 2026-09-29: Internet Archive + OPDS catalogues, see architecture/magazine-sources.md; still open: publisher free issues/feeds, open-access journals, OPDS login and pagination)*, so an issue can be found and pulled in without leaving for a browser and uploading by hand. Wanted: a catalogue of sources, a search across them, and a one-click "ingest this issue". **Blocked on a source decision, and that decision is the whole task.** The two sites named as examples diff --git a/architecture/magazine-sources.md b/architecture/magazine-sources.md new file mode 100644 index 0000000..3206751 --- /dev/null +++ b/architecture/magazine-sources.md @@ -0,0 +1,38 @@ +# Magazine sources (find and import an issue) + +Status: implemented (core/magazine_sources.py, routes/magazine_sources.py, magazine.js "Find an issue online"). + +## Goal +Search legitimate catalogues from inside the app and import an issue as a normal magazine project. Ingestion +stays a separate, existing step, so nothing is narrated before the review. + +## Providers (one interface: `search`, `resolve_pdf`, `allowed_host`) +| Provider | Kind | Notes | +|---|---|---| +| Internet Archive | fixed | documented advanced-search + metadata API; loan-only items excluded (`-access-restricted-item`); prefers the original PDF over derived text PDFs | +| OPDS catalogue | user-configured (`magazine_opds_catalogs` setting) | standard Atom feed; search via the catalogue's own OpenSearch template; only PDF acquisition links | + +Deliberately unsupported: sites that redistribute commercial titles without the publisher's licence (see TODO.md). +Candidates later: publisher free issues / feeds, DOAJ/arXiv/PMC. Each would be another provider. + +## API +- `GET /api/magazine-sources` -> `{sources:[{key,label,kind}]}` +- `GET /api/magazine-sources/{key}/search?q=&page=` -> `{items:[{id,title,author,date,summary,license,page_url,thumb_url}]}`; 404 unknown source, 502 with a safe message when the source fails +- `POST /api/magazine-sources/import {source,item_id,title}` -> `{id,job_id}`; the download runs as a background job (`magazine-source-import`, poll `/api/jobs/{id}`); 409 while another import runs + +## Security boundaries +- SSRF: a download is fetched only from the provider's own host (archive.org, or the catalogue's host). Item ids are validated (IA identifier regex; OPDS id must be same-host). Redirects are followed manually and re-checked. +- XML: feeds with a DOCTYPE or entity declaration are refused (billion-laughs / XXE class); feed size capped at 4 MB. +- Limits: 400 MB per PDF, connect/read timeouts, one import at a time, `%PDF` magic checked on the first chunk, temp file removed on failure. +- No credentials are stored or sent; catalogue URLs with embedded credentials are rejected. +- Remote text is rendered as text in the UI (escaped), links open with `noopener noreferrer`. +- Rate limits/terms: the Internet Archive asks for a descriptive User-Agent and modest request rates; searches are user-initiated, one page of 20 per request. + +## Error behaviour +Every provider failure is a `SourceError` with a user-safe message (unreachable, HTTP status, rate-limited, not a PDF, too large, different host). + +## Offline / sync +Needs the network only for search and download. No synchronisation state. + +## Known limitations +OPDS catalogues requiring login; OPDS pagination beyond the first feed page; no thumbnails shown yet (returned in the API). diff --git a/core/config.py b/core/config.py index e4c43a1..c60b471 100644 --- a/core/config.py +++ b/core/config.py @@ -46,6 +46,8 @@ _SETTINGS_KEYS = { "magazine_layout_url", "magazine_layout_model", "vision_ocr_url", "vision_ocr_model", "vision_ocr_enabled", "vision_ocr_fallback_url", "vision_ocr_fallback_model", + # OPDS catalogues to search for magazine issues: [{"name": ..., "url": ...}], no credentials. + "magazine_opds_catalogs", "engine_local_urls", "engine_container_names", "engine_api_keys", "custom_engine_cards", # Character portrait generation (cloud APIs + local ComfyUI) "image_gen_provider", "image_gen_model", @@ -276,6 +278,8 @@ def _normalize_settings(s: dict) -> dict: s["engine_api_keys"] = {} if not isinstance(s.get("custom_engine_cards"), list): s["custom_engine_cards"] = [] + if not isinstance(s.get("magazine_opds_catalogs"), list): + s["magazine_opds_catalogs"] = [] return s @@ -348,6 +352,7 @@ def _load_settings() -> dict: "engine_container_names": {}, "engine_api_keys": {}, "custom_engine_cards": [], + "magazine_opds_catalogs": [], "image_gen_provider": "", "image_gen_model": "", "comfyui_url": "http://host.docker.internal:8188", diff --git a/core/magazine_sources.py b/core/magazine_sources.py new file mode 100644 index 0000000..984d019 --- /dev/null +++ b/core/magazine_sources.py @@ -0,0 +1,338 @@ +"""Legitimate places to find a magazine issue to import: OPDS catalogues and the Internet Archive. + +Each source is a provider behind the same small interface (search + resolve a PDF), so one going away +or changing its API does not break the others. Nothing here scrapes a website: OPDS is the open +catalogue standard (Calibre-Web, Kavita and Komga serve it) and the Internet Archive has a documented +search and metadata API. See TODO.md ("Finding magazines to read") for why commercial redistribution +sites are deliberately not supported. + +Security boundaries: + * a download is only ever fetched from the provider's own host (the OPDS catalogue's host, or + archive.org), never from a URL an arbitrary feed entry names -- a feed cannot aim the server at + another machine (SSRF); + * redirects are followed by hand and re-checked against the same host rule; + * size and time are bounded, and the result must really be a PDF. +""" +from __future__ import annotations + +import re +from dataclasses import dataclass, asdict +from typing import Iterable, Protocol +from urllib.parse import urljoin, urlsplit, quote + +import requests +import xml.etree.ElementTree as ET + +USER_AGENT = "tts-voice-creator-magazine-import/1.0 (self-hosted; contact via repository)" +SEARCH_TIMEOUT = (5, 20) # connect, read (seconds) +DOWNLOAD_TIMEOUT = (5, 60) +MAX_DOWNLOAD_BYTES = 400 * 1024 * 1024 +MAX_FEED_BYTES = 4 * 1024 * 1024 +MAX_REDIRECTS = 4 +PAGE_SIZE = 20 + +_ATOM = "{http://www.w3.org/2005/Atom}" +_OPDS_ACQ = "http://opds-spec.org/acquisition" +_OPDS_IMAGE = ("http://opds-spec.org/image", "http://opds-spec.org/image/thumbnail") +_IA_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}") + + +class SourceError(Exception): + """A source could not be searched or downloaded; the message is safe to show the user.""" + + +@dataclass +class SourceItem: + id: str # provider-specific handle, e.g. the Internet Archive identifier or an OPDS entry URL + title: str + summary: str = "" + author: str = "" + date: str = "" + license: str = "" # a licence URL or name when the source states one, else empty (unknown) + page_url: str = "" # human-readable page for the item + thumb_url: str = "" + downloadable: bool = True + + def as_dict(self) -> dict: + return asdict(self) + + +class Provider(Protocol): + key: str + label: str + + def search(self, query: str, page: int = 1) -> list[SourceItem]: ... + def resolve_pdf(self, item_id: str) -> str: ... + def allowed_host(self, url: str) -> bool: ... + + +# ── helpers ────────────────────────────────────────────────────────────────── + +def _clip(value, limit: int) -> str: + return re.sub(r"\s+", " ", str(value or "")).strip()[:limit] + + +def _get(session: requests.Session, url: str, *, timeout, stream: bool = False, host_ok=None, + accept: str = "*/*") -> requests.Response: + """GET with manual, host-checked redirects.""" + for _ in range(MAX_REDIRECTS + 1): + if host_ok is not None and not host_ok(url): + raise SourceError("The source pointed to a different host; refusing to follow it") + try: + resp = session.get(url, timeout=timeout, stream=stream, allow_redirects=False, + headers={"User-Agent": USER_AGENT, "Accept": accept}) + except requests.RequestException as e: + raise SourceError(f"Could not reach the source ({type(e).__name__})") from e + if resp.is_redirect or resp.status_code in (301, 302, 303, 307, 308): + target = resp.headers.get("location") or "" + resp.close() + if not target: + raise SourceError("The source sent an empty redirect") + url = urljoin(url, target) + continue + if resp.status_code == 429: + resp.close() + raise SourceError("The source is rate-limiting requests; try again in a minute") + if resp.status_code >= 400: + resp.close() + raise SourceError(f"The source answered HTTP {resp.status_code}") + return resp + raise SourceError("Too many redirects") + + +def _read_limited(resp: requests.Response, limit: int) -> bytes: + out = bytearray() + for chunk in resp.iter_content(64 * 1024): + out.extend(chunk) + if len(out) > limit: + resp.close() + raise SourceError("The response is larger than the allowed size") + return bytes(out) + + +def parse_xml_safely(payload: bytes) -> ET.Element: + """Parse a feed while refusing DTDs and entity declarations (billion-laughs / XXE class).""" + head = payload[:4096].lower() + if b" int: + """Download the item's PDF to `dest` (a pathlib.Path); returns the byte count. + + Written to a temporary sibling first and moved into place only once it is verified to be a PDF. + """ + from pathlib import Path + dest = Path(dest) + session = session or requests.Session() + url = provider.resolve_pdf(item_id) + resp = _get(session, url, timeout=DOWNLOAD_TIMEOUT, stream=True, host_ok=provider.allowed_host, + accept="application/pdf,*/*") + tmp = dest.with_suffix(".part") + total = 0 + try: + declared = int(resp.headers.get("content-length") or 0) + if declared > MAX_DOWNLOAD_BYTES: + raise SourceError("This PDF is larger than the allowed size") + with open(tmp, "wb") as fh: + first = True + for chunk in resp.iter_content(256 * 1024): + if not chunk: + continue + if first: + if not chunk.lstrip()[:5].startswith(b"%PDF"): + raise SourceError("The source did not return a PDF") + first = False + total += len(chunk) + if total > MAX_DOWNLOAD_BYTES: + raise SourceError("This PDF is larger than the allowed size") + fh.write(chunk) + if progress: + progress(total, declared) + if total == 0: + raise SourceError("The source returned an empty file") + tmp.replace(dest) + return total + except BaseException: + tmp.unlink(missing_ok=True) + raise + finally: + resp.close() + + +# ── Internet Archive ───────────────────────────────────────────────────────── + +class InternetArchiveProvider: + """archive.org: documented advanced-search and metadata APIs; only freely downloadable items are listed.""" + key = "internet-archive" + label = "Internet Archive" + _HOST = "archive.org" + + def __init__(self, session: requests.Session | None = None): + self._s = session or requests.Session() + + def allowed_host(self, url: str) -> bool: + parts = urlsplit(url) + host = (parts.hostname or "").lower() + return parts.scheme == "https" and (host == self._HOST or host.endswith("." + self._HOST)) + + def search(self, query: str, page: int = 1) -> list[SourceItem]: + q = _clip(query, 200) + if not q: + raise SourceError("Enter something to search for") + # Lucene-escape quotes only; the rest is passed as the user's free text. + text = q.replace('"', " ") + # Loan-only scans are not downloadable, and a PDF is what this app ingests. + lucene = f'({text}) AND mediatype:texts AND format:pdf AND -access-restricted-item:true AND (subject:magazine OR collection:magazine_rack OR collection:periodicals)' + params = { + "q": lucene, "fl[]": ["identifier", "title", "description", "creator", "date", "licenseurl"], + "rows": PAGE_SIZE, "page": max(1, int(page)), "output": "json", + } + try: + resp = self._s.get(f"https://{self._HOST}/advancedsearch.php", params=params, timeout=SEARCH_TIMEOUT, + headers={"User-Agent": USER_AGENT}) + except requests.RequestException as e: + raise SourceError(f"Could not reach the Internet Archive ({type(e).__name__})") from e + if resp.status_code == 429: + raise SourceError("The Internet Archive is rate-limiting requests; try again in a minute") + if resp.status_code != 200: + raise SourceError(f"The Internet Archive answered HTTP {resp.status_code}") + try: + docs = resp.json()["response"]["docs"] + except (ValueError, KeyError, TypeError) as e: + raise SourceError("The Internet Archive returned an unexpected answer") from e + return list(self._items(docs)) + + @staticmethod + def _first(v) -> str: + return _clip(v[0] if isinstance(v, list) and v else v, 500) + + def _items(self, docs: Iterable[dict]) -> Iterable[SourceItem]: + for d in docs or []: + ident = self._first(d.get("identifier")) + if not _IA_ID.fullmatch(ident): + continue + yield SourceItem( + id=ident, title=self._first(d.get("title")) or ident, + summary=_clip(self._first(d.get("description")), 400), author=self._first(d.get("creator")), + date=self._first(d.get("date"))[:10], license=self._first(d.get("licenseurl")), + page_url=f"https://{self._HOST}/details/{quote(ident)}", + thumb_url=f"https://{self._HOST}/services/img/{quote(ident)}", + ) + + def resolve_pdf(self, item_id: str) -> str: + if not _IA_ID.fullmatch(item_id or ""): + raise SourceError("Not a valid Internet Archive identifier") + resp = _get(self._s, f"https://{self._HOST}/metadata/{quote(item_id)}", timeout=SEARCH_TIMEOUT, + host_ok=self.allowed_host, accept="application/json") + try: + meta = resp.json() + except ValueError as e: + raise SourceError("The Internet Archive returned an unexpected answer") from e + finally: + resp.close() + if (meta.get("metadata") or {}).get("access-restricted-item") in ("true", True): + raise SourceError("This item is lending-only and cannot be downloaded") + files = [f for f in meta.get("files") or [] if str(f.get("name", "")).lower().endswith(".pdf")] + # The original upload beats a derived "Text PDF"; among equals the largest is the full issue. + def rank(f): + return (0 if f.get("source") == "original" else 1, -int(f.get("size") or 0)) + files.sort(key=rank) + if not files: + raise SourceError("This item has no PDF file") + return f"https://{self._HOST}/download/{quote(item_id)}/{quote(str(files[0]['name']))}" + + +# ── OPDS ───────────────────────────────────────────────────────────────────── + +class OpdsProvider: + """A user-configured OPDS catalogue (their own Calibre-Web/Kavita/Komga, or any public one).""" + + def __init__(self, key: str, label: str, root_url: str, session: requests.Session | None = None): + self.key = key + self.label = label + self._root = root_url + self._host = (urlsplit(root_url).hostname or "").lower() + self._s = session or requests.Session() + + def allowed_host(self, url: str) -> bool: + parts = urlsplit(url) + return parts.scheme in ("http", "https") and (parts.hostname or "").lower() == self._host + + def _feed(self, url: str) -> ET.Element: + resp = _get(self._s, url, timeout=SEARCH_TIMEOUT, stream=True, host_ok=self.allowed_host, + accept="application/atom+xml,application/xml,text/xml,*/*") + try: + return parse_xml_safely(_read_limited(resp, MAX_FEED_BYTES)) + finally: + resp.close() + + def _search_url(self) -> str: + """The catalogue's own OpenSearch template if it advertises one, else the conventional path.""" + root = self._feed(self._root) + for link in root.findall(f"{_ATOM}link"): + if link.get("rel") == "search" and link.get("type", "").startswith("application/atom+xml"): + href = urljoin(self._root, link.get("href", "")) + if "{searchTerms}" in href: + return href + return urljoin(self._root, "search?query={searchTerms}") + + def search(self, query: str, page: int = 1) -> list[SourceItem]: + q = _clip(query, 200) + if not q: + raise SourceError("Enter something to search for") + url = self._search_url().replace("{searchTerms}", quote(q)) + items = list(self._entries(self._feed(url), url)) + start = (max(1, int(page)) - 1) * PAGE_SIZE + return items[start:start + PAGE_SIZE] + + def _entries(self, root: ET.Element, base: str) -> Iterable[SourceItem]: + for e in root.findall(f"{_ATOM}entry"): + pdf = "" + thumb = "" + page_url = "" + for link in e.findall(f"{_ATOM}link"): + href = urljoin(base, link.get("href", "")) + rel = link.get("rel", "") + if rel.startswith(_OPDS_ACQ) and "pdf" in (link.get("type") or "").lower() and not pdf: + pdf = href + elif rel in _OPDS_IMAGE and not thumb: + thumb = href + elif rel == "alternate" and not page_url: + page_url = href + # An entry with no PDF is a navigation entry or an EPUB-only book; not importable here. + if not pdf or not self.allowed_host(pdf): + continue + author = e.find(f"{_ATOM}author/{_ATOM}name") + yield SourceItem( + id=pdf, title=_clip(e.findtext(f"{_ATOM}title"), 300) or "Untitled", + summary=_clip(e.findtext(f"{_ATOM}summary") or e.findtext(f"{_ATOM}content"), 400), + author=_clip(author.text if author is not None else "", 200), + date=_clip(e.findtext(f"{_ATOM}updated"), 10), + page_url=page_url if self.allowed_host(page_url or "") else "", + thumb_url=thumb if self.allowed_host(thumb or "") else "", + ) + + def resolve_pdf(self, item_id: str) -> str: + # The id *is* the acquisition URL the search returned; accept only this catalogue's host. + if not self.allowed_host(item_id or ""): + raise SourceError("That item does not belong to this catalogue") + return item_id + + +def build_providers(catalogs: list[dict] | None, session: requests.Session | None = None) -> dict[str, Provider]: + """The Internet Archive plus every configured OPDS catalogue, keyed for the API.""" + providers: dict[str, Provider] = {InternetArchiveProvider.key: InternetArchiveProvider(session)} + for n, cat in enumerate(catalogs or [], 1): + url = str((cat or {}).get("url") or "").strip() + parts = urlsplit(url) + if parts.scheme not in ("http", "https") or not parts.hostname or parts.username or parts.password: + continue + key = f"opds-{n}" + providers[key] = OpdsProvider(key, _clip((cat or {}).get("name"), 80) or parts.hostname, url, session) + return providers diff --git a/routes/magazine_sources.py b/routes/magazine_sources.py new file mode 100644 index 0000000..75fce13 --- /dev/null +++ b/routes/magazine_sources.py @@ -0,0 +1,110 @@ +"""Find a magazine issue in a legitimate source and import it as a new magazine project. + +Search and download go through core.magazine_sources (Internet Archive + configured OPDS catalogues). +An import only creates the project and stores source.pdf; ingestion stays the existing, separate step so +the user still reviews before anything is narrated. +""" +from __future__ import annotations + +import asyncio +import logging +import re +import time +import uuid + +from fastapi import APIRouter, HTTPException, Request + +from core import magazine_sources as ms +from core.config import _load_settings +from core.constants import CONFIG_DIR +from core.database import background_job_create, background_job_update +from routes.magazine import _write_meta + +router = APIRouter() + +_LIB = CONFIG_DIR / "reader_library" +_import_lock = asyncio.Lock() # one download at a time: polite to the source and to the disk +_KEY = re.compile(r"[a-z0-9-]{1,40}") + + +def _providers() -> dict: + return ms.build_providers(_load_settings().get("magazine_opds_catalogs") or []) + + +def _provider(key: str): + provider = _providers().get(key if _KEY.fullmatch(key or "") else "") + if provider is None: + raise HTTPException(404, "Unknown source") + return provider + + +@router.get("/api/magazine-sources") +async def magazine_sources_list(): + return {"sources": [{"key": p.key, "label": p.label, "kind": "opds" if p.key.startswith("opds-") else "archive"} + for p in _providers().values()]} + + +@router.get("/api/magazine-sources/{key}/search") +async def magazine_sources_search(key: str, q: str = "", page: int = 1): + provider = _provider(key) + try: + items = await asyncio.to_thread(provider.search, q, min(max(page, 1), 50)) + except ms.SourceError as e: + raise HTTPException(502, str(e)) + return {"source": key, "items": [i.as_dict() for i in items]} + + +def _download_sync(doc_id: str, job_id: str, key: str, item_id: str) -> None: + dest = _LIB / doc_id / "source.pdf" + try: + provider = _provider(key) + + def progress(done: int, total: int) -> None: + mb = done / 1e6 + detail = f"Downloaded {mb:.0f} of {total / 1e6:.0f} MB" if total else f"Downloaded {mb:.0f} MB" + background_job_update(job_id, stage="download", detail=detail) + + background_job_update(job_id, stage="download", detail="Contacting the source") + size = ms.download_pdf(provider, item_id, dest, progress=progress) + background_job_update(job_id, stage="done", detail="Downloaded", done=True, + result={"doc_id": doc_id, "bytes": size}) + except ms.SourceError as e: + background_job_update(job_id, stage="error", detail="Download failed", done=True, error=str(e)[:300]) + except HTTPException as e: + background_job_update(job_id, stage="error", detail="Download failed", done=True, error=str(e.detail)[:300]) + except Exception: + logging.exception("Magazine source download failed") + background_job_update(job_id, stage="error", detail="Download failed", done=True, + error="The download failed unexpectedly") + + +@router.post("/api/magazine-sources/import") +async def magazine_sources_import(request: Request): + body = await request.json() + key = str(body.get("source") or "") + item_id = str(body.get("item_id") or "") + provider = _provider(key) + if not item_id or len(item_id) > 2000: + raise HTTPException(400, "item_id is required") + if _import_lock.locked(): + raise HTTPException(409, "Another import is still downloading") + title = re.sub(r"\s+", " ", str(body.get("title") or "")).strip()[:300] or "Untitled magazine" + doc_id = uuid.uuid4().hex + d = _LIB / doc_id + d.mkdir(parents=True, exist_ok=True) + now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + _write_meta(d, { + "kind": "magazine", "title": title, "issue": str(body.get("issue") or "").strip()[:120], + "publisher": "", "created": now, "updated": now, "ingest": {"status": "empty"}, + "default_voice": "", "podcast": {"hosts": []}, "progress": {}, + "source": {"provider": provider.label, "item": item_id[:500]}, + }) + job_id = uuid.uuid4().hex + background_job_create(job_id, "magazine-source-import", fingerprint=doc_id) + + async def run(): + async with _import_lock: + await asyncio.to_thread(_download_sync, doc_id, job_id, key, item_id) + + asyncio.create_task(run()) + return {"id": doc_id, "job_id": job_id} diff --git a/server.py b/server.py index 347d049..170e3e3 100644 --- a/server.py +++ b/server.py @@ -19,7 +19,7 @@ from starlette.datastructures import MutableHeaders from core.constants import STATIC_DIR, _BufferHandler, _MAX_JSON_BYTES, _MAX_UPLOAD_BYTES, _MAX_RAW_REQUEST_BYTES, _REQUIRE_EXTERNAL_API_KEY from core.config import _load_settings, _ensure_external_api_key from core.voice_index import refresh_voice_index_background -from routes import admin, settings, library, stt, sources, docker, tts, conversation, reader, characters, rehearsals_db, recovery, history, jobs, magazine +from routes import admin, settings, library, stt, sources, docker, tts, conversation, reader, characters, rehearsals_db, recovery, history, jobs, magazine, magazine_sources logger = logging.getLogger("uvicorn.error") @@ -347,6 +347,7 @@ app.include_router(recovery.router) app.include_router(history.router) app.include_router(jobs.router) app.include_router(magazine.router) +app.include_router(magazine_sources.router) # ── Static files ────────────────────────────────────────────────────────────── diff --git a/static/js/magazine.js b/static/js/magazine.js index bd91469..a3800d0 100644 --- a/static/js/magazine.js +++ b/static/js/magazine.js @@ -88,15 +88,6 @@ function fmtDuration(seconds) { return m < 60 ? m + ' min' : Math.floor(m / 60) + ' h ' + (m % 60) + ' min'; } -// Exact clock time (m:ss, h:mm:ss) -- the estimate above is deliberately coarse, this one names a real MP3. -function fmtClock(seconds) { - const s = Math.max(0, Math.round(seconds || 0)); - const ss = String(s % 60).padStart(2, '0'); - return s >= 3600 - ? Math.floor(s / 3600) + ':' + String(Math.floor(s / 60) % 60).padStart(2, '0') + ':' + ss - : Math.floor(s / 60) + ':' + ss; -} - // The contents list doubles as the playlist: one shared