Compare commits
2 Commits
9942f920bc
...
5adba6b4c0
| Author | SHA1 | Date | |
|---|---|---|---|
| 5adba6b4c0 | |||
| e81fa64448 |
27
CHANGELOG.md
27
CHANGELOG.md
@ -7,7 +7,34 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- **Split-Screen Article Review, "OCR Again", Reading-Order Fixes.** Review shows the rendered page on the left with a
|
||||||
|
numbered box per detected region and the extracted text on the right with matching numbers — clicking either side
|
||||||
|
highlights the other. Building this immediately surfaced a real bug on live production data (see Fixed below).
|
||||||
|
A per-page **"OCR again"** button forces that one page through the configured vision engine again without touching
|
||||||
|
article grouping or approvals (`reocr_page`); a magazine-level **"OCR again (whole magazine)"** re-reads every page
|
||||||
|
and re-runs detection, with a confirmation when articles were already reviewed since that resets their status.
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
- **Two Real Magazine Columns Interleaved Into One.** Found live using the new split-screen review's numbered boxes on
|
||||||
|
a real scanned c't magazine page: a photo caption at 49.9% of the page width sat in the gutter between two genuinely
|
||||||
|
separate columns and merged them into a single detected column, so every block on the page came back `column=0` and
|
||||||
|
reading order fell back to pure top-to-bottom — splicing the two columns' sentences together (a box the review
|
||||||
|
numbered 6, physically above the box numbered 1, was read after it). Column boundary detection now excludes blocks
|
||||||
|
much wider than the page's median block width; separately, the real gutter measured on that page was itself only
|
||||||
|
~2.1% of the page width, narrower than `detect_columns`' 3.5% threshold — lowered to 1.5%.
|
||||||
|
- **Magazine "Forgets" the Open Document.** A full page reload or the browser's own Back button tears down all
|
||||||
|
in-memory state; the magazine section always landed back on the empty library with no way back to what was open.
|
||||||
|
Now persists the last-opened magazine id and reopens it automatically. The actual root cause of a reported duplicate:
|
||||||
|
Studio/Reader's book library and Library → Books both listed every document with no `kind` filter, so the magazine
|
||||||
|
was already showing up there disguised as a plain book — it had not vanished, it was misfiled, which is what led to
|
||||||
|
a re-upload and a genuine duplicate under a different kind. Both listings now exclude `kind: "magazine"`.
|
||||||
|
- **Confidence Badge Layout (Studio Characters Review).** The badge rendered on its own line far below the speaker
|
||||||
|
name because both the name and the article text were `display: block`, forcing a line break regardless of markup
|
||||||
|
order. Wrapped the name, badge and check button in a shared flex row so the badge now sits directly beside the name.
|
||||||
|
- **docker-socket-proxy Crash Loop.** `read_only: true` left the container's own entrypoint nowhere to write its
|
||||||
|
`haproxy.cfg`, crash-looping it and leaving the app without Docker access. `cap_drop: ALL`, `no-new-privileges` and
|
||||||
|
the `:ro` socket mount already provide the real hardening, so `read_only` was removed for this one service.
|
||||||
- **Magazine Dropzone Opened the Dropped File in the Browser Instead of Accepting It.** `dragover`'s `preventDefault()`
|
- **Magazine Dropzone Opened the Dropped File in the Browser Instead of Accepting It.** `dragover`'s `preventDefault()`
|
||||||
was gated on `dataTransfer.types` including `'Files'`, which Chromium does not reliably populate synchronously on
|
was gated on `dataTransfer.types` including `'Files'`, which Chromium does not reliably populate synchronously on
|
||||||
every `dragover` event during a real OS file drag — the condition failed often enough that the element never
|
every `dragover` event during a real OS file drag — the condition failed often enough that the element never
|
||||||
|
|||||||
@ -78,6 +78,24 @@ magazines change column count per page — a fixed grid is wrong. Blocks spannin
|
|||||||
>60% of the page width sort ahead of both columns, so a full-width headline
|
>60% of the page width sort ahead of both columns, so a full-width headline
|
||||||
precedes the body it introduces.
|
precedes the body it introduces.
|
||||||
|
|
||||||
|
**Column boundary detection excludes unusually wide blocks, and the gutter
|
||||||
|
threshold is narrower than it looks.** Confirmed live on a real scanned c't
|
||||||
|
page, using the split-screen review's numbered boxes (built specifically to
|
||||||
|
make exactly this kind of error visible): a photo caption at 49.9% of the
|
||||||
|
page width — under the 60% "spans the page" threshold above, so not given
|
||||||
|
special handling — sat in the whitespace gap between two genuinely separate
|
||||||
|
columns and merged them into one detected column. Every block on the page
|
||||||
|
came back `column=0`, and reading order fell back to pure top-to-bottom,
|
||||||
|
splicing the two real columns' sentences together — the box numbered 6 in
|
||||||
|
review (a caption physically above the box numbered 1) was read *after* it.
|
||||||
|
Column boundaries are now detected from blocks close to the page's median
|
||||||
|
width only, and the real gutter measured on that page was itself only ~2.1%
|
||||||
|
of the page width, narrower than the 3.5% `detect_columns` started with — now
|
||||||
|
1.5%. A single block much narrower than the surrounding column (a stray
|
||||||
|
caption fragment) can still form a spurious extra column of its own; the
|
||||||
|
main failure (two real columns interleaved) is fixed, that residual
|
||||||
|
narrower one is not.
|
||||||
|
|
||||||
**Furniture is identified by repetition, not position.** A running head and a
|
**Furniture is identified by repetition, not position.** A running head and a
|
||||||
headline both sit at the top of a page. What separates them is that the running
|
headline both sit at the top of a page. What separates them is that the running
|
||||||
head recurs on many pages at the same edge; `find_repeated_lines` needs the
|
head recurs on many pages at the same edge; `find_repeated_lines` needs the
|
||||||
|
|||||||
@ -141,13 +141,20 @@ def bbox_overlaps(a: Sequence[float], b: Sequence[float], tol: float = 0.0) -> b
|
|||||||
|
|
||||||
|
|
||||||
def detect_columns(blocks: Sequence[Block], page_width: float,
|
def detect_columns(blocks: Sequence[Block], page_width: float,
|
||||||
min_gap_ratio: float = 0.035) -> list[tuple[float, float]]:
|
min_gap_ratio: float = 0.015) -> list[tuple[float, float]]:
|
||||||
"""Infer the column grid from where blocks are *absent*.
|
"""Infer the column grid from where blocks are *absent*.
|
||||||
|
|
||||||
Magazines vary the column count per page, so a fixed grid is wrong. The
|
Magazines vary the column count per page, so a fixed grid is wrong. The
|
||||||
reliable signal is vertical whitespace running the height of the page: we
|
reliable signal is vertical whitespace running the height of the page: we
|
||||||
project every block onto the x axis, then split wherever the uncovered gap
|
project every block onto the x axis, then split wherever the uncovered gap
|
||||||
is wider than a few percent of the page. Returns (left, right) spans.
|
is wider than a few percent of the page. Returns (left, right) spans.
|
||||||
|
|
||||||
|
1.5%, not the rounder 3.5% this started at: measured on a real scanned
|
||||||
|
c't magazine page, the actual gutter between two genuine columns was
|
||||||
|
only ~2.1% of the page width -- narrower than the old threshold, which
|
||||||
|
made detect_columns see one merged column and, combined with a wide
|
||||||
|
caption bridging the same gap (see assign_reading_order), produced
|
||||||
|
every block on the page landing in column=0.
|
||||||
"""
|
"""
|
||||||
spans = sorted(
|
spans = sorted(
|
||||||
((b.bbox[0], b.bbox[0] + b.bbox[2]) for b in blocks if b.bbox[2] > 1),
|
((b.bbox[0], b.bbox[0] + b.bbox[2]) for b in blocks if b.bbox[2] > 1),
|
||||||
@ -171,8 +178,23 @@ def assign_reading_order(blocks: list[Block], page_width: float) -> list[Block]:
|
|||||||
|
|
||||||
Sorting purely by y interleaves parallel columns, which is precisely the
|
Sorting purely by y interleaves parallel columns, which is precisely the
|
||||||
failure that splices one article's sentence into another's.
|
failure that splices one article's sentence into another's.
|
||||||
|
|
||||||
|
Column BOUNDARIES are detected from blocks close to the page's typical
|
||||||
|
width only. Confirmed live on a real scanned magazine: a caption running
|
||||||
|
under a photo, at 49.9% of the page width -- under the "spans the whole
|
||||||
|
page" threshold below, so not given special handling -- sat squarely in
|
||||||
|
the whitespace gap between two real text columns and bridged them into
|
||||||
|
one merged span, which meant EVERY block on the page got column=0 and
|
||||||
|
reading order fell back to pure top-to-bottom, splicing the two
|
||||||
|
columns' sentences together (the exact failure this function exists to
|
||||||
|
prevent). Every block, wide ones included, is still assigned to
|
||||||
|
whichever column its centre or overlap lands in once boundaries are
|
||||||
|
known -- only the boundary DETECTION excludes them.
|
||||||
"""
|
"""
|
||||||
columns = detect_columns(blocks, page_width)
|
widths = sorted(b.bbox[2] for b in blocks if b.bbox[2] > 1)
|
||||||
|
median_w = widths[len(widths) // 2] if widths else 0
|
||||||
|
narrow = [b for b in blocks if median_w <= 0 or b.bbox[2] <= median_w * 1.4]
|
||||||
|
columns = detect_columns(narrow if len(narrow) >= 2 else blocks, page_width)
|
||||||
|
|
||||||
def column_of(b: Block) -> int:
|
def column_of(b: Block) -> int:
|
||||||
centre = b.bbox[0] + b.bbox[2] / 2
|
centre = b.bbox[0] + b.bbox[2] / 2
|
||||||
@ -1003,6 +1025,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
|
|||||||
vision_ocr_fallback_url: str | None = None,
|
vision_ocr_fallback_url: str | None = None,
|
||||||
vision_ocr_fallback_model: str | None = None,
|
vision_ocr_fallback_model: str | None = None,
|
||||||
vision_ocr_timeout: float = 150.0,
|
vision_ocr_timeout: float = 150.0,
|
||||||
|
force_vision_ocr: bool = False,
|
||||||
progress: Any = None) -> dict[str, Any]:
|
progress: Any = None) -> dict[str, Any]:
|
||||||
"""Run the full extraction for one PDF, writing page renders and images.
|
"""Run the full extraction for one PDF, writing page renders and images.
|
||||||
|
|
||||||
@ -1111,7 +1134,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
|
|||||||
ocr_quality_score(" ".join(b.text for b in blocks))
|
ocr_quality_score(" ".join(b.text for b in blocks))
|
||||||
if not needs_ocr else 0.0
|
if not needs_ocr else 0.0
|
||||||
)
|
)
|
||||||
needs_escalation = needs_ocr or native_quality < _VISION_QUALITY_THRESHOLD
|
needs_escalation = force_vision_ocr or needs_ocr or native_quality < _VISION_QUALITY_THRESHOLD
|
||||||
|
|
||||||
vision_blocks: list[Block] | None = None
|
vision_blocks: list[Block] | None = None
|
||||||
if vision_enabled and needs_escalation:
|
if vision_enabled and needs_escalation:
|
||||||
@ -1326,6 +1349,69 @@ def build_articles(layout: dict[str, Any]) -> list[dict[str, Any]]:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def reocr_page(doc_dir: Path, layout: dict[str, Any], page_no: int,
|
||||||
|
vision_ocr_url: str, vision_ocr_model: str,
|
||||||
|
vision_ocr_fallback_url: str | None = None,
|
||||||
|
vision_ocr_fallback_model: str | None = None,
|
||||||
|
timeout: float = 150.0) -> dict[str, Any] | None:
|
||||||
|
"""Force one page through the vision OCR chain again with the engine the
|
||||||
|
user picked, replacing only that page's blocks in an already-ingested
|
||||||
|
layout.
|
||||||
|
|
||||||
|
Unlike extract_pages()'s automatic escalation, this always calls the
|
||||||
|
engine regardless of the page's existing text quality -- a user pressing
|
||||||
|
"OCR again" has already judged the result unusable and is not asking for
|
||||||
|
another quality gate. Deliberately does NOT call build_articles(): an
|
||||||
|
article the user already reviewed and approved must not be silently
|
||||||
|
re-grouped and lose that approval just because one of its pages was
|
||||||
|
re-read. The caller decides whether to re-run detection separately.
|
||||||
|
|
||||||
|
Returns the updated layout, or None if the page has no rendered PNG or
|
||||||
|
the engine returned nothing usable (network failure, empty response) --
|
||||||
|
the caller keeps the existing blocks in that case, never discards good
|
||||||
|
data for a failed retry.
|
||||||
|
"""
|
||||||
|
fitz = _require_fitz()
|
||||||
|
png = doc_dir / "pages" / f"{page_no:04d}.png"
|
||||||
|
if not png.is_file():
|
||||||
|
return None
|
||||||
|
|
||||||
|
page_meta = next((p for p in layout.get("pages", []) if p["page"] == page_no), None)
|
||||||
|
if not page_meta:
|
||||||
|
return None
|
||||||
|
|
||||||
|
engines = [(vision_ocr_url, vision_ocr_model),
|
||||||
|
(vision_ocr_fallback_url, vision_ocr_fallback_model)]
|
||||||
|
regions = vision_ocr_page_chain(png, engines, timeout=timeout)
|
||||||
|
if not regions:
|
||||||
|
return None
|
||||||
|
|
||||||
|
scale_x = page_meta["width"] / max(1.0, _png_width(png, fitz))
|
||||||
|
new_blocks = build_vision_blocks(regions, page_no, scale_x, id_prefix="r")
|
||||||
|
|
||||||
|
all_blocks = [Block(**b) for b in layout.get("blocks", [])]
|
||||||
|
kept = [b for b in all_blocks if b.page != page_no]
|
||||||
|
|
||||||
|
# The baseline must come from the rest of the document, not this page
|
||||||
|
# alone -- see classify_blocks' docstring on why a single page (which may
|
||||||
|
# have no body copy of its own) cannot be its own reference.
|
||||||
|
doc_base = body_font_size(kept) or body_font_size(all_blocks)
|
||||||
|
classify_blocks(new_blocks, page_meta["height"], page_meta["width"], base_size=doc_base)
|
||||||
|
mark_advert_pages(new_blocks)
|
||||||
|
mark_pullquotes(new_blocks)
|
||||||
|
assign_reading_order(new_blocks, page_meta["width"])
|
||||||
|
mark_sidebars(new_blocks, page_meta["width"])
|
||||||
|
|
||||||
|
merged = kept + new_blocks
|
||||||
|
page_heights = {p["page"]: p["height"] for p in layout.get("pages", [])}
|
||||||
|
cluster_running_furniture(merged, page_heights)
|
||||||
|
|
||||||
|
layout = dict(layout)
|
||||||
|
layout["blocks"] = [b.to_json() for b in merged]
|
||||||
|
layout["vision_ocr_pages"] = int(layout.get("vision_ocr_pages") or 0) + 1
|
||||||
|
return layout
|
||||||
|
|
||||||
|
|
||||||
def save_layout(doc_dir: Path, layout: dict[str, Any]) -> None:
|
def save_layout(doc_dir: Path, layout: dict[str, Any]) -> None:
|
||||||
(doc_dir / "layout.json").write_text(
|
(doc_dir / "layout.json").write_text(
|
||||||
json.dumps(layout, ensure_ascii=False), "utf-8")
|
json.dumps(layout, ensure_ascii=False), "utf-8")
|
||||||
|
|||||||
@ -5,7 +5,13 @@ services:
|
|||||||
image: tecnativa/docker-socket-proxy:0.3.0
|
image: tecnativa/docker-socket-proxy:0.3.0
|
||||||
container_name: tts-voice-creator-docker-proxy
|
container_name: tts-voice-creator-docker-proxy
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
read_only: true
|
# NOT read_only: the entrypoint sed's its own haproxy.cfg.template into
|
||||||
|
# haproxy.cfg at startup, in the same directory the template ships in --
|
||||||
|
# read_only crash-looped it ("Read-only file system"), and a tmpfs mount
|
||||||
|
# over that directory only trades one failure for another by hiding the
|
||||||
|
# template it needs to read (confirmed live: "No such file or
|
||||||
|
# directory"). The real hardening here is cap_drop: ALL,
|
||||||
|
# no-new-privileges, and the Docker socket itself being mounted :ro.
|
||||||
cap_drop:
|
cap_drop:
|
||||||
- ALL
|
- ALL
|
||||||
security_opt:
|
security_opt:
|
||||||
|
|||||||
@ -111,7 +111,8 @@ async def magazine_create(request: Request):
|
|||||||
def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
|
def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
|
||||||
vision_ocr_url: str | None, vision_ocr_model: str | None,
|
vision_ocr_url: str | None, vision_ocr_model: str | None,
|
||||||
vision_ocr_fallback_url: str | None = None,
|
vision_ocr_fallback_url: str | None = None,
|
||||||
vision_ocr_fallback_model: str | None = None) -> None:
|
vision_ocr_fallback_model: str | None = None,
|
||||||
|
force_vision_ocr: bool = False) -> None:
|
||||||
"""Run the extraction pipeline; recorded as a restart-safe job."""
|
"""Run the extraction pipeline; recorded as a restart-safe job."""
|
||||||
d = _LIB / doc_id
|
d = _LIB / doc_id
|
||||||
pdf = d / "source.pdf"
|
pdf = d / "source.pdf"
|
||||||
@ -131,6 +132,7 @@ def _ingest_sync(doc_id: str, job_id: str, ocr_lang: str, render_dpi: int,
|
|||||||
vision_ocr_model=vision_ocr_model,
|
vision_ocr_model=vision_ocr_model,
|
||||||
vision_ocr_fallback_url=vision_ocr_fallback_url,
|
vision_ocr_fallback_url=vision_ocr_fallback_url,
|
||||||
vision_ocr_fallback_model=vision_ocr_fallback_model,
|
vision_ocr_fallback_model=vision_ocr_fallback_model,
|
||||||
|
force_vision_ocr=force_vision_ocr,
|
||||||
progress=progress)
|
progress=progress)
|
||||||
mag.save_layout(d, layout)
|
mag.save_layout(d, layout)
|
||||||
|
|
||||||
@ -203,6 +205,13 @@ async def magazine_ingest(doc_id: str, request: Request):
|
|||||||
if not (vision_ocr_fallback_url and vision_ocr_fallback_model):
|
if not (vision_ocr_fallback_url and vision_ocr_fallback_model):
|
||||||
vision_ocr_fallback_url = vision_ocr_fallback_model = None
|
vision_ocr_fallback_url = vision_ocr_fallback_model = None
|
||||||
|
|
||||||
|
# "OCR again" for the whole magazine: skip the quality gate and send
|
||||||
|
# every page through the vision engine regardless of how good its
|
||||||
|
# existing text already looks. Only meaningful with vision OCR actually
|
||||||
|
# enabled -- silently ignored otherwise, since there is no engine to
|
||||||
|
# force a page through.
|
||||||
|
force_vision_ocr = bool(body.get("force_vision_ocr")) and vision_ocr_url is not None
|
||||||
|
|
||||||
job_id = uuid.uuid4().hex
|
job_id = uuid.uuid4().hex
|
||||||
background_job_create(job_id, "magazine-ingest", fingerprint=doc_id)
|
background_job_create(job_id, "magazine-ingest", fingerprint=doc_id)
|
||||||
meta["ingest"] = {"status": "running", "job_id": job_id}
|
meta["ingest"] = {"status": "running", "job_id": job_id}
|
||||||
@ -211,7 +220,8 @@ async def magazine_ingest(doc_id: str, request: Request):
|
|||||||
asyncio.create_task(
|
asyncio.create_task(
|
||||||
asyncio.to_thread(_ingest_sync, doc_id, job_id, ocr_lang, render_dpi,
|
asyncio.to_thread(_ingest_sync, doc_id, job_id, ocr_lang, render_dpi,
|
||||||
vision_ocr_url, vision_ocr_model,
|
vision_ocr_url, vision_ocr_model,
|
||||||
vision_ocr_fallback_url, vision_ocr_fallback_model)
|
vision_ocr_fallback_url, vision_ocr_fallback_model,
|
||||||
|
force_vision_ocr)
|
||||||
)
|
)
|
||||||
return {"job_id": job_id}
|
return {"job_id": job_id}
|
||||||
|
|
||||||
@ -356,6 +366,70 @@ async def magazine_page_image(doc_id: str, page: int):
|
|||||||
return FileResponse(png, media_type="image/png")
|
return FileResponse(png, media_type="image/png")
|
||||||
|
|
||||||
|
|
||||||
|
def _reocr_page_sync(doc_id: str, job_id: str, page: int,
|
||||||
|
vision_ocr_url: str, vision_ocr_model: str,
|
||||||
|
vision_ocr_fallback_url: str | None,
|
||||||
|
vision_ocr_fallback_model: str | None) -> None:
|
||||||
|
""""OCR again" on one page with the engine the user picked. Restart-safe
|
||||||
|
job for the same reason ingestion is: a vision OCR call can take up to
|
||||||
|
~40s (confirmed live), too long for a plain request/response cycle.
|
||||||
|
"""
|
||||||
|
d = _LIB / doc_id
|
||||||
|
try:
|
||||||
|
layout = mag.load_layout(d)
|
||||||
|
if not layout:
|
||||||
|
raise FileNotFoundError("No layout yet -- run ingestion first")
|
||||||
|
background_job_update(job_id, stage="reocr", detail=f"Re-reading page {page}")
|
||||||
|
updated = mag.reocr_page(d, layout, page, vision_ocr_url, vision_ocr_model,
|
||||||
|
vision_ocr_fallback_url, vision_ocr_fallback_model)
|
||||||
|
if updated is None:
|
||||||
|
raise RuntimeError("The OCR engine returned nothing usable for this page")
|
||||||
|
mag.save_layout(d, updated)
|
||||||
|
background_job_update(job_id, stage="done", done=True,
|
||||||
|
detail=f"Page {page} re-read", result={"page": page})
|
||||||
|
except Exception as exc: # noqa: BLE001 - surfaced to UI
|
||||||
|
background_job_update(job_id, stage="failed", done=True, error=str(exc)[:1000])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/api/magazine/{doc_id}/pages/{page}/reocr")
|
||||||
|
async def magazine_reocr_page(doc_id: str, page: int, request: Request):
|
||||||
|
"""Force one page through the configured vision OCR engine again,
|
||||||
|
regardless of its current text quality -- for when the user has already
|
||||||
|
judged the result unusable. Only replaces that page's blocks; article
|
||||||
|
boundaries and approvals are left untouched (call the ingest endpoint's
|
||||||
|
detection separately if the article needs re-grouping too).
|
||||||
|
"""
|
||||||
|
d, _ = _require_magazine(doc_id)
|
||||||
|
if page < 1 or page > 10000:
|
||||||
|
raise HTTPException(400, "Bad page")
|
||||||
|
if not (d / "pages" / f"{page:04d}.png").is_file():
|
||||||
|
raise HTTPException(404, "Page not rendered -- run ingestion first")
|
||||||
|
|
||||||
|
body = {}
|
||||||
|
try:
|
||||||
|
body = await request.json()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
settings = _load_settings()
|
||||||
|
vision_ocr_url = str(body.get("vision_ocr_url") or settings.get("vision_ocr_url") or "").strip()
|
||||||
|
vision_ocr_model = str(body.get("vision_ocr_model") or settings.get("vision_ocr_model") or "").strip()
|
||||||
|
if not (vision_ocr_url and vision_ocr_model):
|
||||||
|
raise HTTPException(400, "No vision OCR engine configured -- set one in Settings")
|
||||||
|
vision_ocr_fallback_url = str(
|
||||||
|
body.get("vision_ocr_fallback_url") or settings.get("vision_ocr_fallback_url") or "").strip() or None
|
||||||
|
vision_ocr_fallback_model = str(
|
||||||
|
body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip() or None
|
||||||
|
|
||||||
|
job_id = uuid.uuid4().hex
|
||||||
|
background_job_create(job_id, "magazine-reocr-page", fingerprint=f"{doc_id}:{page}")
|
||||||
|
asyncio.create_task(
|
||||||
|
asyncio.to_thread(_reocr_page_sync, doc_id, job_id, page,
|
||||||
|
vision_ocr_url, vision_ocr_model,
|
||||||
|
vision_ocr_fallback_url, vision_ocr_fallback_model)
|
||||||
|
)
|
||||||
|
return {"job_id": job_id}
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/magazine/{doc_id}/images/{name}")
|
@router.get("/api/magazine/{doc_id}/images/{name}")
|
||||||
async def magazine_image(doc_id: str, name: str):
|
async def magazine_image(doc_id: str, name: str):
|
||||||
d, _ = _require_magazine(doc_id)
|
d, _ = _require_magazine(doc_id)
|
||||||
|
|||||||
@ -653,15 +653,28 @@
|
|||||||
background: rgba(37,99,235,.06); border: 1px solid var(--accent); border-radius: 4px;
|
background: rgba(37,99,235,.06); border: 1px solid var(--accent); border-radius: 4px;
|
||||||
padding: 4px 6px; outline: none;
|
padding: 4px 6px; outline: none;
|
||||||
}
|
}
|
||||||
|
/* .ab-cv-spk-row wraps the speaker name together with its confidence badge
|
||||||
|
and check button (see _abRowFromSegment) so the badge sits directly beside
|
||||||
|
the name instead of dropping to its own line below it — before this, both
|
||||||
|
.ab-cv-spk and .ab-cv-txt were `display: block`, which forced a line break
|
||||||
|
between the name and the badge that followed it in the DOM regardless of
|
||||||
|
how close together the markup already put them. */
|
||||||
|
.ab-cv-spk-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: baseline;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 8px;
|
||||||
|
margin-bottom: 2px;
|
||||||
|
}
|
||||||
.ab-cv-spk {
|
.ab-cv-spk {
|
||||||
display: block;
|
|
||||||
font-weight: 700;
|
font-weight: 700;
|
||||||
letter-spacing: 0.08em;
|
letter-spacing: 0.08em;
|
||||||
text-transform: uppercase;
|
text-transform: uppercase;
|
||||||
margin-bottom: 2px;
|
}
|
||||||
|
.ab-cv-row:not(.is-narr):not(.is-processing) .ab-cv-spk-row {
|
||||||
|
padding-left: 120px;
|
||||||
}
|
}
|
||||||
.ab-cv-row:not(.is-narr):not(.is-processing) .ab-cv-spk {
|
.ab-cv-row:not(.is-narr):not(.is-processing) .ab-cv-spk {
|
||||||
padding-left: 120px;
|
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
border-radius: 4px;
|
border-radius: 4px;
|
||||||
transition: background 0.15s;
|
transition: background 0.15s;
|
||||||
@ -685,9 +698,7 @@
|
|||||||
/*max-width: 500px;*/
|
/*max-width: 500px;*/
|
||||||
}
|
}
|
||||||
.ab-cv-row.is-narr .ab-cv-spk {
|
.ab-cv-row.is-narr .ab-cv-spk {
|
||||||
display: block;
|
|
||||||
font-weight: 600;
|
font-weight: 600;
|
||||||
padding-left: 0;
|
|
||||||
font-size: 11px;
|
font-size: 11px;
|
||||||
}
|
}
|
||||||
.ab-cv-row.is-processing {
|
.ab-cv-row.is-processing {
|
||||||
|
|||||||
@ -1867,8 +1867,7 @@ STRIKTE FORMAT- UND TEXTREGELN:
|
|||||||
<button class="ab-cv-row-tool ab-cv-merge-prev" type="button" title="Merge with previous segment"><span class="mdi mdi-arrow-collapse-up"></span></button>
|
<button class="ab-cv-row-tool ab-cv-merge-prev" type="button" title="Merge with previous segment"><span class="mdi mdi-arrow-collapse-up"></span></button>
|
||||||
<button class="ab-cv-row-tool ab-cv-merge-next" type="button" title="Merge with next segment"><span class="mdi mdi-arrow-collapse-down"></span></button>
|
<button class="ab-cv-row-tool ab-cv-merge-next" type="button" title="Merge with next segment"><span class="mdi mdi-arrow-collapse-down"></span></button>
|
||||||
</span>
|
</span>
|
||||||
<span class="ab-cv-spk" style="color:${c}" title="Click to assign character">${escHtml(speakerName)}${s.emotion ? ' <span style="text-transform:lowercase; font-weight:normal; opacity:0.8">(' + escHtml(s.emotion) + ')</span>' : ''}</span>
|
<span class="ab-cv-spk-row"><span class="ab-cv-spk" style="color:${c}" title="Click to assign character">${escHtml(speakerName)}${s.emotion ? ' <span style="text-transform:lowercase; font-weight:normal; opacity:0.8">(' + escHtml(s.emotion) + ')</span>' : ''}</span>${_abConfBadgeHtml(s)}${_abCheckBtnHtml(s)}</span>
|
||||||
${_abConfBadgeHtml(s)}${_abCheckBtnHtml(s)}
|
|
||||||
|
|
||||||
<span class="ab-cv-txt">${_abMarkQuotes(highlightText(_abDisplayText(s)))}</span>`;
|
<span class="ab-cv-txt">${_abMarkQuotes(highlightText(_abDisplayText(s)))}</span>`;
|
||||||
if (typeof s.conf === 'number') row.dataset.conf = String(s.conf);
|
if (typeof s.conf === 'number') row.dataset.conf = String(s.conf);
|
||||||
|
|||||||
@ -66,6 +66,9 @@ async function libraryRenderBooks() {
|
|||||||
list.innerHTML = _libSkeleton(4);
|
list.innerHTML = _libSkeleton(4);
|
||||||
let all = [];
|
let all = [];
|
||||||
try { const r = await fetch(LIB_READER_API); if (r.ok) all = (await r.json()).docs || []; } catch (_) { all = []; }
|
try { const r = await fetch(LIB_READER_API); if (r.ok) all = (await r.json()).docs || []; } catch (_) { all = []; }
|
||||||
|
// Magazines share this store (kind: "magazine") but live in their own
|
||||||
|
// library at Studio -> Magazines -- see the matching fix in reader.js.
|
||||||
|
all = all.filter(d => d.kind !== 'magazine');
|
||||||
if (!all.length) {
|
if (!all.length) {
|
||||||
list.innerHTML = '<div class="lib-empty"><span class="mdi mdi-book-open-outline"></span><p>No books yet.</p>'
|
list.innerHTML = '<div class="lib-empty"><span class="mdi mdi-book-open-outline"></span><p>No books yet.</p>'
|
||||||
+ '<p class="lib-empty-hint">Open <b>Read Aloud</b>, import a PDF or text, and save it to your library.</p></div>';
|
+ '<p class="lib-empty-hint">Open <b>Read Aloud</b>, import a PDF or text, and save it to your library.</p></div>';
|
||||||
|
|||||||
@ -193,18 +193,32 @@ async function magCreate() {
|
|||||||
|
|
||||||
// ── Ingestion ────────────────────────────────────────────────────────────────
|
// ── Ingestion ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// Remembers which magazine was open so a full page reload or the browser's
|
||||||
|
// own Back button (which, unlike in-app navigation, tears down all JS state)
|
||||||
|
// lands back on it instead of the empty library — confirmed live as the
|
||||||
|
// cause of a user re-uploading the same PDF, creating a duplicate under a
|
||||||
|
// completely different kind ("book", via Studio/Reader's own import) because
|
||||||
|
// the magazine they had been working on appeared to have vanished.
|
||||||
|
const MAG_LAST_DOC_KEY = 'ttsvc_mag_last_doc';
|
||||||
|
|
||||||
async function magOpen(docId) {
|
async function magOpen(docId) {
|
||||||
try {
|
try {
|
||||||
state.doc = await api('/api/reader/docs/' + docId);
|
state.doc = await api('/api/reader/docs/' + docId);
|
||||||
state.doc.id = docId;
|
state.doc.id = docId;
|
||||||
await magRefreshArticles();
|
await magRefreshArticles();
|
||||||
magShow('contents');
|
magShow('contents');
|
||||||
|
try { localStorage.setItem(MAG_LAST_DOC_KEY, docId); } catch (_) {}
|
||||||
if ((state.doc.ingest || {}).status === 'running') pollIngest();
|
if ((state.doc.ingest || {}).status === 'running') pollIngest();
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
say('Could not open magazine: ' + e.message, 'err');
|
say('Could not open magazine: ' + e.message, 'err');
|
||||||
|
try { localStorage.removeItem(MAG_LAST_DOC_KEY); } catch (_) {}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function magForgetLastDoc() {
|
||||||
|
try { localStorage.removeItem(MAG_LAST_DOC_KEY); } catch (_) {}
|
||||||
|
}
|
||||||
|
|
||||||
async function magIngest() {
|
async function magIngest() {
|
||||||
if (!state.doc) return;
|
if (!state.doc) return;
|
||||||
try {
|
try {
|
||||||
@ -219,9 +233,42 @@ async function magIngest() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// "OCR again" for the whole issue: forces every page through the vision
|
||||||
|
// engine regardless of its current quality, then re-detects articles --
|
||||||
|
// which is why this asks for confirmation. A single page's "OCR again" in
|
||||||
|
// review does not touch article grouping; re-reading the whole document
|
||||||
|
// does, since detection runs again over the freshly re-read text.
|
||||||
|
async function magReocrMagazine() {
|
||||||
|
if (!state.doc) return;
|
||||||
|
const hasReviewed = state.articles.some((a) => a.status !== 'detected');
|
||||||
|
if (hasReviewed && !confirm(
|
||||||
|
'Re-reading the whole magazine runs article detection again. ' +
|
||||||
|
'Any articles you already approved or rejected will need reviewing again. Continue?'
|
||||||
|
)) return;
|
||||||
|
try {
|
||||||
|
await api('/api/magazine/' + state.doc.id + '/ingest', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ force_vision_ocr: true }),
|
||||||
|
});
|
||||||
|
pollIngest();
|
||||||
|
} catch (e) {
|
||||||
|
say('Could not start OCR: ' + e.message, 'err');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// "Analysing page 28 of 204" -> a real percentage, not just spinner text.
|
||||||
|
function magParsePageProgress(detail) {
|
||||||
|
const m = /page (\d+) of (\d+)/i.exec(detail || '');
|
||||||
|
if (!m) return null;
|
||||||
|
const page = parseInt(m[1], 10), total = parseInt(m[2], 10);
|
||||||
|
if (!total) return null;
|
||||||
|
return { page, total, pct: Math.min(100, Math.round((page / total) * 100)) };
|
||||||
|
}
|
||||||
|
|
||||||
async function pollIngest() {
|
async function pollIngest() {
|
||||||
const bar = $('mag-ingest-status');
|
const bar = $('mag-ingest-status');
|
||||||
if (bar) bar.hidden = false;
|
if (bar) { bar.hidden = false; bar.className = 'mag-status is-running'; }
|
||||||
for (;;) {
|
for (;;) {
|
||||||
let data;
|
let data;
|
||||||
try {
|
try {
|
||||||
@ -230,11 +277,27 @@ async function pollIngest() {
|
|||||||
const ing = data.ingest || {};
|
const ing = data.ingest || {};
|
||||||
const detail = (data.job || {}).detail || '';
|
const detail = (data.job || {}).detail || '';
|
||||||
if (bar) {
|
if (bar) {
|
||||||
bar.innerHTML = ing.status === 'running'
|
if (ing.status === 'running') {
|
||||||
? '<span class="mdi mdi-progress-clock"></span> ' + esc(detail || 'Analysing…')
|
bar.className = 'mag-status is-running';
|
||||||
: ing.status === 'failed'
|
const p = magParsePageProgress(detail);
|
||||||
? '<span class="mdi mdi-alert"></span> Ingestion failed: ' + esc(ing.error || '')
|
bar.innerHTML =
|
||||||
: '<span class="mdi mdi-check"></span> ' + (ing.articles || 0) + ' articles detected';
|
'<div class="mag-ingest-row">' +
|
||||||
|
'<span class="mdi mdi-progress-clock mag-ingest-icon"></span>' +
|
||||||
|
'<div class="mag-ingest-text">' +
|
||||||
|
'<strong>' + esc(detail || 'Analysing…') + '</strong>' +
|
||||||
|
(p ? '<div class="mag-ingest-track"><div class="mag-ingest-fill" style="width:' + p.pct + '%"></div></div>' : '') +
|
||||||
|
'</div>' +
|
||||||
|
(p ? '<span class="mag-ingest-pct">' + p.pct + '%</span>' : '') +
|
||||||
|
'</div>';
|
||||||
|
} else if (ing.status === 'failed') {
|
||||||
|
bar.className = 'mag-status is-failed';
|
||||||
|
bar.innerHTML = '<div class="mag-ingest-row"><span class="mdi mdi-alert mag-ingest-icon"></span>' +
|
||||||
|
'<span>Ingestion failed: ' + esc(ing.error || '') + '</span></div>';
|
||||||
|
} else {
|
||||||
|
bar.className = 'mag-status is-done';
|
||||||
|
bar.innerHTML = '<div class="mag-ingest-row"><span class="mdi mdi-check-circle mag-ingest-icon"></span>' +
|
||||||
|
'<span>' + (ing.articles || 0) + ' article' + (ing.articles === 1 ? '' : 's') + ' detected</span></div>';
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if (ing.status !== 'running') {
|
if (ing.status !== 'running') {
|
||||||
if (ing.status === 'done') await magRefreshArticles();
|
if (ing.status === 'done') await magRefreshArticles();
|
||||||
@ -325,6 +388,11 @@ async function magReview(articleId) {
|
|||||||
if (!article) return;
|
if (!article) return;
|
||||||
state.article = article;
|
state.article = article;
|
||||||
await ensureLayout();
|
await ensureLayout();
|
||||||
|
// The whole review is one page at a time -- showing every page of a
|
||||||
|
// 10-page feature stacked at once was unreadable and made the boxes tiny.
|
||||||
|
state.reviewSpan = magReviewSpan(article);
|
||||||
|
state.reviewPage = state.reviewSpan[0] || null;
|
||||||
|
_magReviewChecked = null;
|
||||||
magShow('review');
|
magShow('review');
|
||||||
renderReview();
|
renderReview();
|
||||||
}
|
}
|
||||||
@ -346,18 +414,35 @@ function blocksForPages(pages) {
|
|||||||
.sort((a, b) => a.page - b.page || a.reading_order - b.reading_order);
|
.sort((a, b) => a.page - b.page || a.reading_order - b.reading_order);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Offer neighbouring pages too, so a boundary that started a page late (or a
|
||||||
|
// jump tail that should have been included) can be corrected without
|
||||||
|
// leaving the screen.
|
||||||
|
function magReviewSpan(article) {
|
||||||
|
if (!article.pages.length) return [];
|
||||||
|
return Array.from(new Set(article.pages.concat(
|
||||||
|
[article.pages[0] - 1, article.pages[article.pages.length - 1] + 1])))
|
||||||
|
.filter((p) => p >= 1 && p <= (state.layout.page_count || 0)).sort((x, y) => x - y);
|
||||||
|
}
|
||||||
|
|
||||||
|
function magReviewGoPage(page) {
|
||||||
|
if (!state.reviewSpan.includes(page)) return;
|
||||||
|
state.reviewPage = page;
|
||||||
|
renderReview();
|
||||||
|
}
|
||||||
|
|
||||||
function renderReview() {
|
function renderReview() {
|
||||||
const host = $('mag-review');
|
const host = $('mag-review');
|
||||||
const a = state.article;
|
const a = state.article;
|
||||||
if (!host || !a) return;
|
if (!host || !a) return;
|
||||||
const included = new Set(a.blocks || []);
|
const included = new Set(a.blocks || []);
|
||||||
// Offer neighbouring pages too, so a boundary that started a page late can be
|
|
||||||
// corrected without leaving the screen.
|
|
||||||
const span = a.pages.length
|
|
||||||
? Array.from(new Set(a.pages.concat([a.pages[0] - 1, a.pages[a.pages.length - 1] + 1])))
|
|
||||||
.filter((p) => p >= 1 && p <= (state.layout.page_count || 0)).sort((x, y) => x - y)
|
|
||||||
: [];
|
|
||||||
const conf = confidenceLabel(a.confidence);
|
const conf = confidenceLabel(a.confidence);
|
||||||
|
const page = state.reviewPage;
|
||||||
|
const pageMeta = (state.layout.pages || []).find((p) => p.page === page);
|
||||||
|
const pageBlocks = page == null ? [] : blocksForPages([page]);
|
||||||
|
|
||||||
|
const spanIdx = state.reviewSpan.indexOf(page);
|
||||||
|
const prevPage = spanIdx > 0 ? state.reviewSpan[spanIdx - 1] : null;
|
||||||
|
const nextPage = spanIdx >= 0 && spanIdx < state.reviewSpan.length - 1 ? state.reviewSpan[spanIdx + 1] : null;
|
||||||
|
|
||||||
let html =
|
let html =
|
||||||
'<div class="mag-review-head">' +
|
'<div class="mag-review-head">' +
|
||||||
@ -366,27 +451,65 @@ function renderReview() {
|
|||||||
'<div class="mag-pill ' + conf.cls + '">' + esc(conf.text) + '</div>' +
|
'<div class="mag-pill ' + conf.cls + '">' + esc(conf.text) + '</div>' +
|
||||||
'</div>' +
|
'</div>' +
|
||||||
'<p class="muted">Untick anything that is not part of this article — a sidebar, an advert, ' +
|
'<p class="muted">Untick anything that is not part of this article — a sidebar, an advert, ' +
|
||||||
'or a neighbouring piece. Only ticked text is read aloud.</p>';
|
'or a neighbouring piece. Only ticked text is read aloud. Click a numbered region on the page, ' +
|
||||||
|
'or its matching entry on the right, to line the two up.</p>';
|
||||||
|
|
||||||
let currentPage = null;
|
html += '<div class="mag-review-pagenav">' +
|
||||||
html += '<div class="mag-blocks">';
|
'<button type="button" class="btn-sm" ' + (prevPage ? '' : 'disabled') +
|
||||||
blocksForPages(span).forEach((b) => {
|
' onclick="magReviewGoPage(' + prevPage + ')"><span class="mdi mdi-chevron-left"></span></button>' +
|
||||||
if (b.page !== currentPage) {
|
'<span>Page ' + (page ?? '—') + '</span>' +
|
||||||
currentPage = b.page;
|
'<button type="button" class="btn-sm" ' + (nextPage ? '' : 'disabled') +
|
||||||
html += '<h5 class="mag-page-head">Page ' + b.page +
|
' onclick="magReviewGoPage(' + nextPage + ')"><span class="mdi mdi-chevron-right"></span></button>' +
|
||||||
' <button type="button" class="btn-sm" onclick="magPeekPage(' + b.page + ')">' +
|
'<span class="mag-review-pagenav-spacer"></span>' +
|
||||||
'Show original</button></h5>';
|
'<button type="button" class="btn-sm" id="mag-reocr-btn" onclick="magReocrPage()">' +
|
||||||
|
'<span class="mdi mdi-text-recognition"></span> OCR again (this page)</button>' +
|
||||||
|
'<span id="mag-reocr-status" class="muted"></span>' +
|
||||||
|
'</div>';
|
||||||
|
|
||||||
|
html += '<div class="mag-review-split">';
|
||||||
|
|
||||||
|
// Left: the rendered page with a numbered box per block.
|
||||||
|
html += '<div class="mag-review-page">';
|
||||||
|
if (page != null) {
|
||||||
|
html += '<div class="mag-review-page-wrap">' +
|
||||||
|
'<img src="/api/magazine/' + state.doc.id + '/pages/' + page + '" alt="Page ' + page + '">';
|
||||||
|
pageBlocks.forEach((b, i) => {
|
||||||
|
if (!pageMeta) return;
|
||||||
|
const on = included.has(b.id);
|
||||||
|
const left = (b.bbox[0] / pageMeta.width * 100).toFixed(2);
|
||||||
|
const top = (b.bbox[1] / pageMeta.height * 100).toFixed(2);
|
||||||
|
const w = (b.bbox[2] / pageMeta.width * 100).toFixed(2);
|
||||||
|
const h = (b.bbox[3] / pageMeta.height * 100).toFixed(2);
|
||||||
|
html += '<div class="mag-review-box' + (on ? '' : ' is-excluded') + '" data-block="' + esc(b.id) +
|
||||||
|
'" style="left:' + left + '%;top:' + top + '%;width:' + w + '%;height:' + h + '%" ' +
|
||||||
|
'onclick="magReviewFocusBlock(\'' + esc(b.id) + '\')" title="' + esc(b.type) + '">' +
|
||||||
|
'<span class="mag-review-box-num">' + (i + 1) + '</span></div>';
|
||||||
|
});
|
||||||
|
html += '</div>';
|
||||||
|
} else {
|
||||||
|
html += '<p class="muted">No page to show.</p>';
|
||||||
}
|
}
|
||||||
|
html += '</div>';
|
||||||
|
|
||||||
|
// Right: the same blocks as text, numbered to match.
|
||||||
|
html += '<div class="mag-blocks">';
|
||||||
|
pageBlocks.forEach((b, i) => {
|
||||||
const narratable = ['title', 'deck', 'byline', 'body'].includes(b.type);
|
const narratable = ['title', 'deck', 'byline', 'body'].includes(b.type);
|
||||||
const on = included.has(b.id);
|
const on = included.has(b.id);
|
||||||
html += '<label class="mag-block' + (narratable ? '' : ' is-aux') + '">' +
|
html += '<label class="mag-block' + (narratable ? '' : ' is-aux') + '" data-block-row="' + esc(b.id) + '" ' +
|
||||||
'<input type="checkbox" ' + (on ? 'checked' : '') + ' data-block="' + esc(b.id) + '">' +
|
'onmouseenter="magReviewHighlightBox(\'' + esc(b.id) + '\')" onmouseleave="magReviewHighlightBox(null)">' +
|
||||||
|
'<span class="mag-block-num">' + (i + 1) + '</span>' +
|
||||||
|
'<input type="checkbox" ' + (on ? 'checked' : '') + ' data-block="' + esc(b.id) + '" onchange="magReviewToggleBox(this)">' +
|
||||||
'<span class="mag-block-type">' + esc(b.type) + '</span>' +
|
'<span class="mag-block-type">' + esc(b.type) + '</span>' +
|
||||||
'<span class="mag-block-text">' + esc(b.text.slice(0, 220)) +
|
'<span class="mag-block-text">' + esc(b.text.slice(0, 220)) +
|
||||||
(b.text.length > 220 ? '…' : '') + '</span></label>';
|
(b.text.length > 220 ? '…' : '') + '</span></label>';
|
||||||
});
|
});
|
||||||
html += '</div>' +
|
if (!pageBlocks.length) html += '<p class="muted">Nothing detected on this page.</p>';
|
||||||
'<div class="mag-review-foot">' +
|
html += '</div>';
|
||||||
|
|
||||||
|
html += '</div>'; // /mag-review-split
|
||||||
|
|
||||||
|
html += '<div class="mag-review-foot">' +
|
||||||
'<button type="button" class="btn" onclick="magShow(\'contents\')">Back to contents</button>' +
|
'<button type="button" class="btn" onclick="magShow(\'contents\')">Back to contents</button>' +
|
||||||
'<button type="button" class="btn" onclick="magSetStatus(\'rejected\')">Not an article</button>' +
|
'<button type="button" class="btn" onclick="magSetStatus(\'rejected\')">Not an article</button>' +
|
||||||
'<button type="button" class="btn primary" onclick="magApprove()">Approve</button>' +
|
'<button type="button" class="btn primary" onclick="magApprove()">Approve</button>' +
|
||||||
@ -394,11 +517,74 @@ function renderReview() {
|
|||||||
host.innerHTML = html;
|
host.innerHTML = html;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Tracks every ticked block across ALL reviewed pages of this article, not
|
||||||
|
// just the one currently on screen -- switching pages must not forget a
|
||||||
|
// change made on a previous one.
|
||||||
|
let _magReviewChecked = null;
|
||||||
|
|
||||||
|
function magReviewToggleBox(input) {
|
||||||
|
if (!_magReviewChecked) _magReviewChecked = new Set(state.article.blocks || []);
|
||||||
|
if (input.checked) _magReviewChecked.add(input.dataset.block);
|
||||||
|
else _magReviewChecked.delete(input.dataset.block);
|
||||||
|
const box = document.querySelector('.mag-review-box[data-block="' + CSS.escape(input.dataset.block) + '"]');
|
||||||
|
if (box) box.classList.toggle('is-excluded', !input.checked);
|
||||||
|
}
|
||||||
|
|
||||||
|
function magReviewFocusBlock(blockId) {
|
||||||
|
const cb = document.querySelector('#mag-review input[data-block="' + CSS.escape(blockId) + '"]');
|
||||||
|
if (cb) { cb.checked = !cb.checked; magReviewToggleBox(cb); }
|
||||||
|
const row = document.querySelector('#mag-review [data-block-row="' + CSS.escape(blockId) + '"]');
|
||||||
|
if (row) row.scrollIntoView({ block: 'nearest', behavior: prefersReducedMotion() ? 'auto' : 'smooth' });
|
||||||
|
}
|
||||||
|
|
||||||
|
function magReviewHighlightBox(blockId) {
|
||||||
|
document.querySelectorAll('.mag-review-box.is-linked').forEach((el) => el.classList.remove('is-linked'));
|
||||||
|
if (!blockId) return;
|
||||||
|
const box = document.querySelector('.mag-review-box[data-block="' + CSS.escape(blockId) + '"]');
|
||||||
|
if (box) box.classList.add('is-linked');
|
||||||
|
}
|
||||||
|
|
||||||
function selectedBlockIds() {
|
function selectedBlockIds() {
|
||||||
|
if (_magReviewChecked) return Array.from(_magReviewChecked);
|
||||||
return Array.from(document.querySelectorAll('#mag-review input[data-block]:checked'))
|
return Array.from(document.querySelectorAll('#mag-review input[data-block]:checked'))
|
||||||
.map((el) => el.dataset.block);
|
.map((el) => el.dataset.block);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function magReocrPage() {
|
||||||
|
const page = state.reviewPage;
|
||||||
|
if (page == null) return;
|
||||||
|
const btn = $('mag-reocr-btn');
|
||||||
|
const statusEl = $('mag-reocr-status');
|
||||||
|
if (btn) btn.disabled = true;
|
||||||
|
if (statusEl) statusEl.textContent = 'Reading page ' + page + '…';
|
||||||
|
let jobId;
|
||||||
|
try {
|
||||||
|
const r = await api('/api/magazine/' + state.doc.id + '/pages/' + page + '/reocr', {
|
||||||
|
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}',
|
||||||
|
});
|
||||||
|
jobId = r.job_id;
|
||||||
|
} catch (e) {
|
||||||
|
if (statusEl) statusEl.textContent = '';
|
||||||
|
if (btn) btn.disabled = false;
|
||||||
|
return say('Could not start OCR: ' + e.message, 'err');
|
||||||
|
}
|
||||||
|
for (;;) {
|
||||||
|
let job;
|
||||||
|
try { job = await api('/api/jobs/' + jobId); } catch (_) { break; }
|
||||||
|
if (job.done) {
|
||||||
|
if (job.error) say('OCR failed: ' + job.error, 'err');
|
||||||
|
else say('Page ' + page + ' re-read', 'ok');
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
await new Promise((r) => setTimeout(r, 2000));
|
||||||
|
}
|
||||||
|
if (btn) btn.disabled = false;
|
||||||
|
if (statusEl) statusEl.textContent = '';
|
||||||
|
state.layout = null;
|
||||||
|
await ensureLayout();
|
||||||
|
renderReview();
|
||||||
|
}
|
||||||
|
|
||||||
async function magApprove() {
|
async function magApprove() {
|
||||||
const a = state.article;
|
const a = state.article;
|
||||||
if (!a) return;
|
if (!a) return;
|
||||||
@ -969,11 +1155,27 @@ function syncDrivingPlayButton() {
|
|||||||
|
|
||||||
// ── Boot ─────────────────────────────────────────────────────────────────────
|
// ── Boot ─────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
function magInit() {
|
async function magInit() {
|
||||||
if (!$('mag-library-list')) return; // section not mounted
|
if (!$('mag-library-list')) return; // section not mounted
|
||||||
magLoadLibrary();
|
magLoadLibrary();
|
||||||
magLoadVoices();
|
magLoadVoices();
|
||||||
setupMagDropzone();
|
setupMagDropzone();
|
||||||
|
let lastId = null;
|
||||||
|
try { lastId = localStorage.getItem(MAG_LAST_DOC_KEY); } catch (_) {}
|
||||||
|
if (lastId) {
|
||||||
|
// Confirmed live: without this, a full reload or the browser's own Back
|
||||||
|
// button (as opposed to in-app navigation) always dropped back to the
|
||||||
|
// empty library, making an in-progress magazine look like it had been
|
||||||
|
// lost. If it genuinely no longer exists (deleted elsewhere), fall back
|
||||||
|
// to the library rather than getting stuck on a permanent error toast.
|
||||||
|
try {
|
||||||
|
await api('/api/reader/docs/' + lastId);
|
||||||
|
await magOpen(lastId);
|
||||||
|
return;
|
||||||
|
} catch (_) {
|
||||||
|
magForgetLastDoc();
|
||||||
|
}
|
||||||
|
}
|
||||||
magShow('library');
|
magShow('library');
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -981,6 +1183,12 @@ root.magOpen = magOpen;
|
|||||||
root.magCreate = magCreate;
|
root.magCreate = magCreate;
|
||||||
root.magIngest = magIngest;
|
root.magIngest = magIngest;
|
||||||
root.magReview = magReview;
|
root.magReview = magReview;
|
||||||
|
root.magReviewGoPage = magReviewGoPage;
|
||||||
|
root.magReviewFocusBlock = magReviewFocusBlock;
|
||||||
|
root.magReviewHighlightBox = magReviewHighlightBox;
|
||||||
|
root.magReviewToggleBox = magReviewToggleBox;
|
||||||
|
root.magReocrPage = magReocrPage;
|
||||||
|
root.magReocrMagazine = magReocrMagazine;
|
||||||
root.magApprove = magApprove;
|
root.magApprove = magApprove;
|
||||||
root.magSetStatus = magSetStatus;
|
root.magSetStatus = magSetStatus;
|
||||||
root.magRead = magRead;
|
root.magRead = magRead;
|
||||||
|
|||||||
@ -1717,6 +1717,12 @@ async function readerRenderLibrary() {
|
|||||||
if (!card || !list) return;
|
if (!card || !list) return;
|
||||||
let all = [];
|
let all = [];
|
||||||
try { const r = await fetch(READER_API); if (r.ok) all = (await r.json()).docs || []; } catch (_) { all = []; }
|
try { const r = await fetch(READER_API); if (r.ok) all = (await r.json()).docs || []; } catch (_) { all = []; }
|
||||||
|
// Magazines share this store (kind: "magazine") but are a different
|
||||||
|
// feature with their own library at Studio -> Magazines -- listing them
|
||||||
|
// here too let one masquerade as a plain book, which is how a user's
|
||||||
|
// in-progress magazine ended up re-uploaded as a duplicate (confirmed
|
||||||
|
// live: it never actually vanished, it was just misfiled here).
|
||||||
|
all = all.filter(d => d.kind !== 'magazine');
|
||||||
if (!all.length) { list.innerHTML = '<div style="padding: 20px; text-align: center; color: var(--subtext);">No saved books yet.</div>'; return; }
|
if (!all.length) { list.innerHTML = '<div style="padding: 20px; text-align: center; color: var(--subtext);">No saved books yet.</div>'; return; }
|
||||||
list.innerHTML = all.map(rec => {
|
list.innerHTML = all.map(rec => {
|
||||||
const total = rec.sentenceCount || 0;
|
const total = rec.sentenceCount || 0;
|
||||||
|
|||||||
@ -41,7 +41,7 @@
|
|||||||
<!-- ── Contents ─────────────────────────────────────────────────────── -->
|
<!-- ── Contents ─────────────────────────────────────────────────────── -->
|
||||||
<div id="mag-view-contents" hidden>
|
<div id="mag-view-contents" hidden>
|
||||||
<div class="mag-bar">
|
<div class="mag-bar">
|
||||||
<button type="button" class="btn-sm" onclick="magShow('library')">
|
<button type="button" class="btn-sm" onclick="magForgetLastDoc();magShow('library')">
|
||||||
<span class="mdi mdi-arrow-left"></span> All magazines
|
<span class="mdi mdi-arrow-left"></span> All magazines
|
||||||
</button>
|
</button>
|
||||||
<h3 id="mag-doc-title">Magazine</h3>
|
<h3 id="mag-doc-title">Magazine</h3>
|
||||||
@ -52,6 +52,9 @@
|
|||||||
<button type="button" class="btn-sm" onclick="magIngest()">
|
<button type="button" class="btn-sm" onclick="magIngest()">
|
||||||
<span class="mdi mdi-refresh"></span> Re-run detection
|
<span class="mdi mdi-refresh"></span> Re-run detection
|
||||||
</button>
|
</button>
|
||||||
|
<button type="button" class="btn-sm" onclick="magReocrMagazine()" title="Send every page through the configured OCR engine again, then re-detect articles">
|
||||||
|
<span class="mdi mdi-text-recognition"></span> OCR again (whole magazine)
|
||||||
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div id="mag-ingest-status" class="mag-status" hidden></div>
|
<div id="mag-ingest-status" class="mag-status" hidden></div>
|
||||||
|
|||||||
@ -4765,8 +4765,29 @@ input[list="lang-codes"] {
|
|||||||
.mag-bar { display: flex; align-items: center; gap: 12px; margin-bottom: 14px; flex-wrap: wrap; }
|
.mag-bar { display: flex; align-items: center; gap: 12px; margin-bottom: 14px; flex-wrap: wrap; }
|
||||||
.mag-bar h3 { margin: 0; }
|
.mag-bar h3 { margin: 0; }
|
||||||
.mag-bar-actions { margin-left: auto; }
|
.mag-bar-actions { margin-left: auto; }
|
||||||
.mag-status { padding: 8px 12px; border-radius: var(--radius); background: var(--panel);
|
.mag-status { padding: 12px 14px; border-radius: var(--radius); background: var(--panel);
|
||||||
color: var(--subtext); margin-bottom: 12px; }
|
color: var(--subtext); margin-bottom: 12px; border-left: 3px solid var(--border); }
|
||||||
|
.mag-status.is-running { border-left-color: var(--accent); }
|
||||||
|
.mag-status.is-done { border-left-color: var(--green); }
|
||||||
|
.mag-status.is-failed { border-left-color: var(--red); }
|
||||||
|
.mag-ingest-row { display: flex; align-items: center; gap: 10px; }
|
||||||
|
.mag-ingest-icon { font-size: 20px; flex-shrink: 0; }
|
||||||
|
.mag-status.is-running .mag-ingest-icon { color: var(--accent); animation: mag-spin 1.6s linear infinite; }
|
||||||
|
.mag-status.is-done .mag-ingest-icon { color: var(--green); }
|
||||||
|
.mag-status.is-failed .mag-ingest-icon { color: var(--red); }
|
||||||
|
.mag-ingest-text { flex: 1; min-width: 0; display: flex; flex-direction: column; gap: 6px; }
|
||||||
|
.mag-ingest-text strong { font-weight: 500; color: var(--text); font-size: 13px;
|
||||||
|
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||||
|
.mag-ingest-track { height: 6px; border-radius: 3px; background: var(--border); overflow: hidden; }
|
||||||
|
.mag-ingest-fill { height: 100%; background: var(--accent); border-radius: 3px;
|
||||||
|
transition: width .4s ease; }
|
||||||
|
.mag-ingest-pct { font-size: 13px; font-weight: 600; color: var(--accent); flex-shrink: 0;
|
||||||
|
min-width: 3.2em; text-align: right; }
|
||||||
|
@keyframes mag-spin { from { transform: rotate(0deg); } to { transform: rotate(360deg); } }
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
.mag-status.is-running .mag-ingest-icon { animation: none; }
|
||||||
|
.mag-ingest-fill { transition: none; }
|
||||||
|
}
|
||||||
|
|
||||||
.mag-pill { display: inline-block; padding: 2px 8px; border-radius: 999px; font-size: 12px;
|
.mag-pill { display: inline-block; padding: 2px 8px; border-radius: 999px; font-size: 12px;
|
||||||
background: var(--panel); color: var(--subtext); border: 1px solid var(--border); }
|
background: var(--panel); color: var(--subtext); border: 1px solid var(--border); }
|
||||||
@ -4791,15 +4812,43 @@ input[list="lang-codes"] {
|
|||||||
.mag-review-head input { padding: 8px 10px; border: 1px solid var(--border); border-radius: var(--radius);
|
.mag-review-head input { padding: 8px 10px; border: 1px solid var(--border); border-radius: var(--radius);
|
||||||
background: var(--surface); color: var(--text); }
|
background: var(--surface); color: var(--text); }
|
||||||
.mag-page-head { margin: 16px 0 6px; font-size: 13px; color: var(--subtext); }
|
.mag-page-head { margin: 16px 0 6px; font-size: 13px; color: var(--subtext); }
|
||||||
.mag-block { display: grid; grid-template-columns: auto 90px 1fr; gap: 10px; align-items: start;
|
|
||||||
|
/* ── Review page navigator + "OCR again" ─────────────────────────────────── */
|
||||||
|
.mag-review-pagenav { display: flex; align-items: center; gap: 10px; margin-bottom: 12px; }
|
||||||
|
.mag-review-pagenav span { font-size: 13px; color: var(--subtext); min-width: 60px; text-align: center; }
|
||||||
|
.mag-review-pagenav-spacer { flex: 1; min-width: 0; }
|
||||||
|
|
||||||
|
/* ── Split screen: rendered page (left) + extracted blocks (right) ───────── */
|
||||||
|
.mag-review-split { display: grid; grid-template-columns: 1.1fr 1fr; gap: 18px; align-items: start; }
|
||||||
|
.mag-review-page { position: sticky; top: 12px; }
|
||||||
|
.mag-review-page-wrap { position: relative; border: 1px solid var(--border); border-radius: var(--radius);
|
||||||
|
overflow: hidden; background: var(--panel); }
|
||||||
|
.mag-review-page-wrap img { width: 100%; height: auto; display: block; }
|
||||||
|
.mag-review-box { position: absolute; border: 2px solid var(--accent); background: rgba(37,99,235,.10);
|
||||||
|
cursor: pointer; transition: background .12s, border-color .12s; }
|
||||||
|
.mag-review-box:hover { background: rgba(37,99,235,.22); }
|
||||||
|
.mag-review-box.is-excluded { border-color: var(--subtext); background: rgba(120,120,120,.10); }
|
||||||
|
.mag-review-box.is-excluded:hover { background: rgba(120,120,120,.22); }
|
||||||
|
.mag-review-box.is-linked { border-color: var(--yellow); background: rgba(217,119,6,.22); z-index: 2; }
|
||||||
|
.mag-review-box-num { position: absolute; top: -1px; left: -1px; min-width: 16px; padding: 0 3px;
|
||||||
|
background: var(--accent); color: var(--on-accent); font-size: 10px; font-weight: 700;
|
||||||
|
line-height: 16px; text-align: center; border-radius: 0 0 4px 0; }
|
||||||
|
.mag-review-box.is-excluded .mag-review-box-num { background: var(--subtext); }
|
||||||
|
|
||||||
|
.mag-block { display: grid; grid-template-columns: auto auto 90px 1fr; gap: 10px; align-items: start;
|
||||||
padding: 8px 10px; border: 1px solid var(--border); border-radius: var(--radius);
|
padding: 8px 10px; border: 1px solid var(--border); border-radius: var(--radius);
|
||||||
background: var(--surface); margin-bottom: 4px; cursor: pointer; }
|
background: var(--surface); margin-bottom: 4px; cursor: pointer; }
|
||||||
.mag-block.is-aux { opacity: .68; }
|
.mag-block.is-aux { opacity: .68; }
|
||||||
.mag-block:focus-within { border-color: var(--accent); box-shadow: 0 0 0 3px rgba(37,99,235,.14); }
|
.mag-block:focus-within { border-color: var(--accent); box-shadow: 0 0 0 3px rgba(37,99,235,.14); }
|
||||||
|
.mag-block-num { font-size: 11px; font-weight: 700; color: var(--accent); min-width: 1.4em; }
|
||||||
.mag-block-type { font-size: 11px; text-transform: uppercase; letter-spacing: .05em; color: var(--subtext); }
|
.mag-block-type { font-size: 11px; text-transform: uppercase; letter-spacing: .05em; color: var(--subtext); }
|
||||||
.mag-block-text { font-size: 14px; line-height: 1.45; }
|
.mag-block-text { font-size: 14px; line-height: 1.45; }
|
||||||
.mag-review-foot { display: flex; gap: 8px; justify-content: flex-end; margin-top: 16px;
|
.mag-review-foot { display: flex; gap: 8px; justify-content: flex-end; margin-top: 16px;
|
||||||
position: sticky; bottom: 0; padding: 12px 0; background: var(--bg); }
|
position: sticky; bottom: 0; padding: 12px 0; background: var(--bg); }
|
||||||
|
@media (max-width: 980px) {
|
||||||
|
.mag-review-split { grid-template-columns: 1fr; }
|
||||||
|
.mag-review-page { position: static; }
|
||||||
|
}
|
||||||
|
|
||||||
.mag-controls { display: flex; gap: 12px; align-items: center; flex-wrap: wrap; margin-bottom: 14px;
|
.mag-controls { display: flex; gap: 12px; align-items: center; flex-wrap: wrap; margin-bottom: 14px;
|
||||||
padding: 10px 12px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--surface); }
|
padding: 10px 12px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--surface); }
|
||||||
@ -4874,7 +4923,7 @@ input[list="lang-codes"] {
|
|||||||
.mag-new-grid, .mag-review-head { grid-template-columns: 1fr; }
|
.mag-new-grid, .mag-review-head { grid-template-columns: 1fr; }
|
||||||
.mag-toc-row { grid-template-columns: 1fr; }
|
.mag-toc-row { grid-template-columns: 1fr; }
|
||||||
.mag-toc-actions { justify-content: flex-start; }
|
.mag-toc-actions { justify-content: flex-start; }
|
||||||
.mag-block { grid-template-columns: auto 1fr; }
|
.mag-block { grid-template-columns: auto auto 1fr; }
|
||||||
.mag-block-type { display: none; }
|
.mag-block-type { display: none; }
|
||||||
.mag-article { font-size: 18px; }
|
.mag-article { font-size: 18px; }
|
||||||
.mag-driving-controls { gap: 20px; }
|
.mag-driving-controls { gap: 20px; }
|
||||||
|
|||||||
@ -9,6 +9,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import os
|
import os
|
||||||
import tempfile
|
import tempfile
|
||||||
|
import shutil
|
||||||
import pathlib
|
import pathlib
|
||||||
import unittest
|
import unittest
|
||||||
from unittest import mock
|
from unittest import mock
|
||||||
@ -37,6 +38,7 @@ from core.magazine import (
|
|||||||
build_articles,
|
build_articles,
|
||||||
build_vision_blocks,
|
build_vision_blocks,
|
||||||
classify_blocks,
|
classify_blocks,
|
||||||
|
reocr_page,
|
||||||
detect_columns,
|
detect_columns,
|
||||||
estimate_font_size_from_area,
|
estimate_font_size_from_area,
|
||||||
estimate_listen_seconds,
|
estimate_listen_seconds,
|
||||||
@ -103,6 +105,36 @@ class ColumnGeometryTests(unittest.TestCase):
|
|||||||
order = [b.id for b in assign_reading_order(blocks, PAGE_W)]
|
order = [b.id for b in assign_reading_order(blocks, PAGE_W)]
|
||||||
self.assertEqual(order[0], "head")
|
self.assertEqual(order[0], "head")
|
||||||
|
|
||||||
|
def test_a_wide_caption_does_not_bridge_two_real_columns(self):
|
||||||
|
"""Grounded in real data from a live scanned c't magazine page:
|
||||||
|
every block came back column=0, and reading order fell back to pure
|
||||||
|
top-to-bottom, splicing the two columns' sentences together. Cause:
|
||||||
|
a caption running under a photo at 49.9% of the page width -- under
|
||||||
|
the "spans the whole page" threshold, so not given special handling
|
||||||
|
-- sat in the whitespace gap between two real columns and merged
|
||||||
|
them into one. Page width and block widths below are the real
|
||||||
|
measured values (in points), simplified to two blocks per column."""
|
||||||
|
page_w = 583.2
|
||||||
|
blocks = [
|
||||||
|
block("caption", 1, 31.3, 268.0, 291.3, 23.4, "Caption under a photo, spanning most of the page."),
|
||||||
|
block("l1", 1, 138.4, 345.4, 201.7, 55.8, "Left column body copy."),
|
||||||
|
block("l2", 1, 138.9, 401.8, 198.9, 21.4, "Left column continues."),
|
||||||
|
block("r1", 1, 352.4, 58.0, 199.2, 10.6, "Right column body copy."),
|
||||||
|
block("r2", 1, 352.7, 79.8, 201.0, 44.0, "Right column continues."),
|
||||||
|
]
|
||||||
|
ordered = assign_reading_order(blocks, page_w)
|
||||||
|
cols = {b.id: b.column for b in ordered}
|
||||||
|
self.assertNotEqual(cols["l1"], cols["r1"],
|
||||||
|
"left and right column blocks landed in the same detected column")
|
||||||
|
# The left column (l1, l2) must not be interleaved with the right
|
||||||
|
# column (r1, r2) by y-position -- confirmed live as the actual
|
||||||
|
# symptom: an earlier left-column block sorted after a later
|
||||||
|
# right-column one purely because both were column=0.
|
||||||
|
order = [b.id for b in ordered]
|
||||||
|
self.assertLess(order.index("l1"), order.index("r2"))
|
||||||
|
self.assertEqual(cols["l1"], cols["l2"])
|
||||||
|
self.assertEqual(cols["r1"], cols["r2"])
|
||||||
|
|
||||||
|
|
||||||
class ClassificationTests(unittest.TestCase):
|
class ClassificationTests(unittest.TestCase):
|
||||||
def test_body_size_is_not_hijacked_by_display_type(self):
|
def test_body_size_is_not_hijacked_by_display_type(self):
|
||||||
@ -1019,5 +1051,108 @@ class VisionEngineChainTests(unittest.TestCase):
|
|||||||
self.assertIsNone(vision_ocr_page_chain(pathlib.Path("/fake.png"), []))
|
self.assertIsNone(vision_ocr_page_chain(pathlib.Path("/fake.png"), []))
|
||||||
|
|
||||||
|
|
||||||
|
class ReocrPageTests(unittest.TestCase):
|
||||||
|
"""User-triggered "OCR again" on one page -- must replace only that
|
||||||
|
page's blocks, never touch build_articles() (an already-approved
|
||||||
|
article's boundaries must not silently change), and never discard good
|
||||||
|
existing blocks on a failed retry."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.mkdtemp(prefix="ttvc-reocr-test-")
|
||||||
|
self.doc_dir = pathlib.Path(self.tmp)
|
||||||
|
(self.doc_dir / "pages").mkdir()
|
||||||
|
(self.doc_dir / "pages" / "0005.png").write_bytes(b"fake-png")
|
||||||
|
self.layout = {
|
||||||
|
"page_count": 10,
|
||||||
|
"pages": [{"page": 5, "width": 595.0, "height": 842.0}],
|
||||||
|
"blocks": [
|
||||||
|
{"id": "b0005-000", "page": 5, "bbox": [50, 60, 470, 50],
|
||||||
|
"type": BLOCK_TITLE, "text": "Old garbled title", "reading_order": 0,
|
||||||
|
"confidence": 0.5, "font_size": 26.0, "font_name": "", "bold": True,
|
||||||
|
"column": 0, "source": "ocr"},
|
||||||
|
{"id": "b0009-000", "page": 9, "bbox": [50, 60, 300, 400],
|
||||||
|
"type": BLOCK_BODY, "text": "Untouched other page copy. " * 40,
|
||||||
|
"reading_order": 0, "confidence": 0.85, "font_size": 10.0,
|
||||||
|
"font_name": "", "bold": False, "column": 0, "source": "native"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
self._fitz_patch = mock.patch("core.magazine._require_fitz", return_value=mock.Mock())
|
||||||
|
self._pngw_patch = mock.patch("core.magazine._png_width", return_value=595.0)
|
||||||
|
self._fitz_patch.start()
|
||||||
|
self._pngw_patch.start()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self._fitz_patch.stop()
|
||||||
|
self._pngw_patch.stop()
|
||||||
|
shutil.rmtree(self.tmp, ignore_errors=True)
|
||||||
|
|
||||||
|
def _mock_chain(self, regions):
|
||||||
|
return mock.patch("core.magazine.vision_ocr_page_chain", return_value=regions)
|
||||||
|
|
||||||
|
def test_missing_png_returns_none(self):
|
||||||
|
(self.doc_dir / "pages" / "0005.png").unlink()
|
||||||
|
out = reocr_page(self.doc_dir, self.layout, 5, "http://e", "m")
|
||||||
|
self.assertIsNone(out)
|
||||||
|
|
||||||
|
def test_page_not_in_layout_returns_none(self):
|
||||||
|
out = reocr_page(self.doc_dir, self.layout, 99, "http://e", "m")
|
||||||
|
self.assertIsNone(out)
|
||||||
|
|
||||||
|
def test_engine_returning_nothing_returns_none_and_keeps_old_blocks(self):
|
||||||
|
with self._mock_chain(None):
|
||||||
|
out = reocr_page(self.doc_dir, self.layout, 5, "http://e", "m")
|
||||||
|
self.assertIsNone(out)
|
||||||
|
# The original (unsuccessful) caller-visible layout is untouched --
|
||||||
|
# reocr_page never mutates its input on failure.
|
||||||
|
self.assertEqual(len(self.layout["blocks"]), 2)
|
||||||
|
|
||||||
|
def test_successful_reocr_replaces_only_the_target_pages_blocks(self):
|
||||||
|
regions = [{"label": "title", "bbox": [605, 294, 914, 336],
|
||||||
|
"text": "1. Irrtum: KI ist ein allgemeiner Jobkiller"}]
|
||||||
|
with self._mock_chain(regions):
|
||||||
|
out = reocr_page(self.doc_dir, self.layout, 5, "http://e", "m")
|
||||||
|
self.assertIsNotNone(out)
|
||||||
|
pages = {b["page"] for b in out["blocks"]}
|
||||||
|
self.assertEqual(pages, {5, 9})
|
||||||
|
page5 = [b for b in out["blocks"] if b["page"] == 5]
|
||||||
|
self.assertEqual(len(page5), 1)
|
||||||
|
self.assertEqual(page5[0]["text"], "1. Irrtum: KI ist ein allgemeiner Jobkiller")
|
||||||
|
self.assertEqual(page5[0]["type"], BLOCK_TITLE)
|
||||||
|
# Page 9's block is byte-for-byte the original -- never regrouped or
|
||||||
|
# reclassified just because a different page was re-OCR'd.
|
||||||
|
page9 = [b for b in out["blocks"] if b["page"] == 9]
|
||||||
|
self.assertEqual(page9[0]["text"], self.layout["blocks"][1]["text"])
|
||||||
|
|
||||||
|
def test_vision_ocr_pages_counter_increments(self):
|
||||||
|
regions = [{"label": "text", "bbox": [50, 60, 500, 400], "text": "Some real text. " * 10}]
|
||||||
|
with self._mock_chain(regions):
|
||||||
|
out = reocr_page(self.doc_dir, self.layout, 5, "http://e", "m")
|
||||||
|
self.assertEqual(out["vision_ocr_pages"], 1)
|
||||||
|
|
||||||
|
def test_baseline_uses_the_rest_of_the_document_not_the_new_page_alone(self):
|
||||||
|
"""A page reduced to one block would otherwise become its own
|
||||||
|
font-size reference (see classify_blocks' docstring): with only this
|
||||||
|
page's own text to measure against, its ratio is always exactly 1.0,
|
||||||
|
which would make it read as body regardless of what it actually is.
|
||||||
|
Judged against page 9's real ~10pt body baseline instead, a
|
||||||
|
similarly-sized estimate for ordinary paragraph text should also
|
||||||
|
land as body -- not because the ratio is trivially 1, but because
|
||||||
|
the two independently-measured sizes genuinely agree."""
|
||||||
|
text = "Body copy text repeated many times to fill the box. " * 4
|
||||||
|
regions = [{"label": "text", "bbox": [50, 60, 250, 60], "text": text}]
|
||||||
|
with self._mock_chain(regions):
|
||||||
|
out = reocr_page(self.doc_dir, self.layout, 5, "http://e", "m")
|
||||||
|
page5 = [b for b in out["blocks"] if b["page"] == 5][0]
|
||||||
|
self.assertEqual(page5["type"], BLOCK_BODY)
|
||||||
|
|
||||||
|
def test_fallback_engine_is_passed_through_the_chain(self):
|
||||||
|
with mock.patch("core.magazine.vision_ocr_page_chain") as mocked:
|
||||||
|
mocked.return_value = [{"label": "text", "bbox": [0, 0, 10, 10], "text": "x y z real words"}]
|
||||||
|
reocr_page(self.doc_dir, self.layout, 5, "http://primary", "m1",
|
||||||
|
vision_ocr_fallback_url="http://fallback", vision_ocr_fallback_model="m2")
|
||||||
|
engines_arg = mocked.call_args[0][1]
|
||||||
|
self.assertEqual(engines_arg, [("http://primary", "m1"), ("http://fallback", "m2")])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user