Compare commits
3 Commits
81e2c6b4ef
...
916d19d22f
| Author | SHA1 | Date | |
|---|---|---|---|
| 916d19d22f | |||
| 884bbf33aa | |||
| 28d6439604 |
11
CHANGELOG.md
11
CHANGELOG.md
@ -7,6 +7,14 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
---
|
||||
|
||||
## [1.56.2] — 2026-09-30
|
||||
### Fixed
|
||||
- **A turned page could keep showing its old picture, which made turning look broken (and invited pressing the buttons again).** Page pictures had no cache header until recently, and a browser may treat an unchanged URL as fresh for days, so after a successful turn the review could keep showing the old image. The layout now stamps every page with its picture's version (the file's modification time) and every page picture URL (review, page grid, reading pane, cover, lock-screen artwork) carries it, so a changed picture is a new URL in any browser. Turns also add up (right + half turn = three quarters), so the toolbar now says how far the page is turned from the scan ("Turned 270° clockwise from the scan") with an **Undo turns** button back to the PDF's own orientation, and a second turn or re-read of a page that is still being processed is refused (HTTP 409) instead of stacking. `extract_pages` records a stored turn in the page's layout entry, so it survives a re-import visibly. Verified end to end with the real OCR engines on a scratch copy of c't p. 156: turn 90 gives an upright 886x1039 page read at word-F1 0.878, Undo turns restores the 1039x886 original and clears the stored turn.
|
||||
- **A page's picture could vanish from the review (seen on c't p. 176).** One region held a 3,408-character token with no spaces (`Giro + 2.1.1.1.1…`, the OCR engine stuck in a loop). Its width was taken as the minimum width of the list column, which widened to 20,305 px and squeezed the page-picture column to 2 px. The review grid's tracks can now shrink (`minmax(0, …)`), and text in the review list, reading view, figure diffs and page cards wraps anywhere, so no content can collapse the layout again; a long URL or a loop now simply wraps. The cause is also handled: *Clean up boxes* (and every import and re-read, which run it) re-type a region that is mostly one endless repeated letter/digit token as OCR noise (kept, visible, reversible in the type picker, not read aloud). Leader dots, rules, section numbers, phone numbers, prices and URLs are not affected; run over all three of your magazines (about 6,000 regions) the rule flags exactly that one region on page 176 in each and nothing else. To clear it on an existing magazine press *Clean up boxes* or *Fix this page automatically*.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## [1.56.1] — 2026-09-30
|
||||
@ -4616,7 +4624,8 @@ Initial feature-complete release.
|
||||
|
||||
---
|
||||
|
||||
[Unreleased]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.1...HEAD
|
||||
[Unreleased]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.2...HEAD
|
||||
[1.56.2]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.1...v1.56.2
|
||||
[1.56.1]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.0...v1.56.1
|
||||
[1.56.0]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.55.0...v1.56.0
|
||||
[1.55.0]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.54.0...v1.55.0
|
||||
|
||||
@ -407,6 +407,13 @@ of the two pages the buttons act on, as with the other page tools.
|
||||
are re-measured. On a test page, reading it sideways scored word-F1 0.02 and reading it after turning 0.88 (an ordinary page scores
|
||||
about 0.88). The turn is remembered: re-running detection or re-importing does not flip the page back.
|
||||
|
||||
**Keeping track of turns.** Turns add up: a right turn plus a half turn is three quarters. The toolbar says so in words (for example
|
||||
"Turned 270° clockwise from the scan") and offers **Undo turns**, which returns the page to the orientation the PDF itself gives it. Only one
|
||||
turn or re-read of a page can run at a time: pressing a button again while one is still working is refused with a message, instead of stacking
|
||||
a second turn. The turned picture appears as soon as the page is re-read; page pictures are requested under a version number, so a browser
|
||||
that remembered the old picture fetches the new one. Note what a PDF's own orientation can be: some pages are stored sideways on purpose
|
||||
(c't page 156 is a landscape page that reads sideways), and one **Turn right** is then the whole fix.
|
||||
|
||||
**What is replaced.** The page's boxes and text are replaced by the new reading, so corrections you made to that page by hand are lost, and
|
||||
articles that used the page go back to "not approved" where the new boxes do not line up with the old ones. **Undo does not cover a turn**
|
||||
(it changes the page image); if you turned the wrong way, press the opposite button.
|
||||
|
||||
@ -477,6 +477,16 @@ Verified end-to-end against a synthetic 8-page issue (contents page, two-column
|
||||
feature with a byline and a page-8 jump, full-page advert, second feature,
|
||||
reviews) and against an image-only rendering of the same file to exercise OCR.
|
||||
|
||||
## Endless-token regions and unbreakable text
|
||||
|
||||
A recogniser stuck in a loop can emit one token with no spaces at all (c't p. 176: `Giro + 2.1.1.1.1…`, 3,408 characters). `_looks_like_repetition_loop`
|
||||
ignores a repeated unit without a real word on purpose (leader dots, rules), so it did not see it. `glued_repetition_share(text)` counts the share of a text
|
||||
covered by glued runs of a short unit (<= 6 characters) containing a letter or digit, repeated 30+ times; `clean_up_regions` re-types a region of at least 60
|
||||
characters where that share is >= 0.5 as `ocr_noise` (furniture, reversible). Measured on the three c't magazines (~6,000 regions): exactly one region each, none
|
||||
other. Independently of the data, the review must survive such text: a grid track is min-content unless told otherwise, so one unbreakable word widened the
|
||||
list column to 20,305 px and left the page picture 2 px. `.mag-review-split` uses `minmax(0, ...)` tracks, the list containers have `min-width: 0` and the text
|
||||
classes `overflow-wrap: anywhere`.
|
||||
|
||||
## Turning a page scanned sideways
|
||||
|
||||
`POST /api/magazine/{id}/pages/{page}/rotate {degrees: 90|180|270|-90}` starts the same background job as "OCR again"
|
||||
|
||||
@ -1251,6 +1251,24 @@ def _looks_like_repetition_loop(text: str) -> bool:
|
||||
for m in _REPETITION_RUN.finditer(text or ""))
|
||||
|
||||
|
||||
# A recogniser stuck in a loop can also emit one endless token with no spaces at all ("2.1.1.1.1.1 ..." thousands of
|
||||
# characters long, seen on a real shopping-advert page). _looks_like_repetition_loop deliberately ignores a repeated unit
|
||||
# without a real word, so leader dots and rules survive -- but a short unit that contains a letter or digit, repeated
|
||||
# this many times back to back, is never real text: a phone number, a price, a section number or a URL do not come
|
||||
# close to 30 identical repeats.
|
||||
_GLUED_RUN = re.compile(r"(\S{1,6}?)\1{29,}")
|
||||
_GLUED_UNIT_CONTENT = re.compile(r"[A-Za-z0-9À-ÖØ-öø-ÿ]")
|
||||
|
||||
|
||||
def glued_repetition_share(text: str) -> float:
|
||||
"""The share of `text` taken up by glued runs of a short letter/digit unit repeated 30+ times (0.0 - 1.0)."""
|
||||
text = text or ""
|
||||
if not text:
|
||||
return 0.0
|
||||
covered = sum(m.end() - m.start() for m in _GLUED_RUN.finditer(text) if _GLUED_UNIT_CONTENT.search(m.group(1)))
|
||||
return covered / len(text)
|
||||
|
||||
|
||||
def vision_ocr_page(png_path: Path, base_url: str, model: str,
|
||||
timeout: float = 150.0) -> list[dict[str, Any]] | None:
|
||||
"""OCR one rendered page with a connected vision-capable OCR engine.
|
||||
@ -2652,6 +2670,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
|
||||
"page": page_no,
|
||||
"width": rect.width,
|
||||
"height": rect.height,
|
||||
"rotation": extra_turn,
|
||||
"blocks": blocks,
|
||||
"image_boxes": image_boxes,
|
||||
"images": page_images,
|
||||
@ -2716,7 +2735,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
|
||||
|
||||
return {
|
||||
"page_count": doc.page_count,
|
||||
"pages": [{"page": p["page"], "width": p["width"], "height": p["height"]}
|
||||
"pages": [{"page": p["page"], "width": p["width"], "height": p["height"],
|
||||
**({"rotation": p["rotation"]} if p.get("rotation") else {})}
|
||||
for p in raw_pages],
|
||||
"blocks": [b.to_json() for b in all_blocks],
|
||||
"images": all_images,
|
||||
@ -4755,8 +4775,11 @@ def clean_up_regions(layout: dict[str, Any], pages: Iterable[int] | None = None)
|
||||
if text.startswith("(no text recognised"):
|
||||
continue
|
||||
if (len(re.sub(r"[\W_]", "", text)) <= 2 or text.lower().startswith("[non-text")
|
||||
or b["bbox"][3] < _MIN_TEXT_BOX_HEIGHT):
|
||||
mark(b, "ocr_noise") # includes the engine's own "[Non-Text]" and hairline boxes
|
||||
or b["bbox"][3] < _MIN_TEXT_BOX_HEIGHT
|
||||
or (len(text) >= 60 and glued_repetition_share(text) >= 0.5)):
|
||||
# includes the engine's own "[Non-Text]", hairline boxes, and a region that is mostly one endless
|
||||
# repeated token (a recogniser stuck in a loop)
|
||||
mark(b, "ocr_noise")
|
||||
elif len(text) <= 80 and any(_area_inside(b["bbox"], p) >= 0.9 for p in pictures.get(page_no, [])):
|
||||
mark(b, "in_picture")
|
||||
for small in text_boxes:
|
||||
|
||||
@ -40,6 +40,7 @@ from core.constants import CONFIG_DIR
|
||||
from core.database import (
|
||||
background_job_create,
|
||||
background_job_get,
|
||||
background_job_list,
|
||||
background_job_update,
|
||||
)
|
||||
from core import spelling
|
||||
@ -551,9 +552,20 @@ async def magazine_layout(doc_id: str, page: int | None = None):
|
||||
layout = dict(layout)
|
||||
layout["blocks"] = [b for b in layout.get("blocks", []) if b.get("page") == page]
|
||||
layout["pages"] = [p for p in layout.get("pages", []) if p.get("page") == page]
|
||||
# The page picture's URL carries this number, so a turned or re-rendered page is fetched again by every browser
|
||||
# whatever it cached earlier (a browser may treat an unchanged URL as fresh for days).
|
||||
layout = dict(layout)
|
||||
layout["pages"] = [{**p, "image_v": _page_image_version(d, p.get("page"))} for p in layout.get("pages", [])]
|
||||
return layout
|
||||
|
||||
|
||||
def _page_image_version(d: Path, page: Any) -> int:
|
||||
try:
|
||||
return int((d / "pages" / f"{int(page):04d}.png").stat().st_mtime)
|
||||
except (OSError, TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _next_article_id(articles: list[dict]) -> str:
|
||||
nums = [int(m.group(1)) for a in articles
|
||||
if (m := re.fullmatch(r"article-(\d+)", a.get("id") or ""))]
|
||||
@ -1054,6 +1066,22 @@ def _reocr_page_sync(doc_id: str, job_id: str, page: int,
|
||||
background_job_update(job_id, stage="failed", done=True, error=str(exc)[:1000])
|
||||
|
||||
|
||||
_PAGE_JOB_KINDS = ("magazine-rotate-page", "magazine-reocr-page")
|
||||
_PAGE_JOB_STALE_SECONDS = 15 * 60 # a job that has not reported for this long died with the server
|
||||
|
||||
|
||||
def _page_job_running(doc_id: str, page: int) -> bool:
|
||||
"""True while a turn or re-read of this page is in flight. Two at once would read and rewrite the same image and
|
||||
layout; pressing the button again because the first one seemed to do nothing must not stack a second turn."""
|
||||
now = time.time()
|
||||
for kind in _PAGE_JOB_KINDS:
|
||||
for job in background_job_list(kind, 50):
|
||||
if (job and not job.get("done") and job.get("fingerprint") == f"{doc_id}:{page}"
|
||||
and now - float(job.get("updated") or 0) < _PAGE_JOB_STALE_SECONDS):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
@router.post("/api/magazine/{doc_id}/pages/{page}/reocr")
|
||||
async def magazine_reocr_page(doc_id: str, page: int, request: Request):
|
||||
"""Force one page through the configured vision OCR engine again,
|
||||
@ -1083,6 +1111,8 @@ async def magazine_reocr_page(doc_id: str, page: int, request: Request):
|
||||
vision_ocr_fallback_model = str(
|
||||
body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip() or None
|
||||
|
||||
if _page_job_running(doc_id, page):
|
||||
raise HTTPException(409, f"Page {page} is still being read; wait for it to finish")
|
||||
job_id = uuid.uuid4().hex
|
||||
background_job_create(job_id, "magazine-reocr-page", fingerprint=f"{doc_id}:{page}")
|
||||
asyncio.create_task(
|
||||
@ -1119,6 +1149,8 @@ async def magazine_rotate_page(doc_id: str, page: int, request: Request):
|
||||
url, model, fb_url, fb_model = _vision_args(settings)
|
||||
if not (url and model):
|
||||
raise HTTPException(400, "No vision OCR engine configured -- set one in Settings (turning re-reads the page)")
|
||||
if _page_job_running(doc_id, page):
|
||||
raise HTTPException(409, f"Page {page} is still being turned or read; wait for it to finish before turning it again")
|
||||
job_id = uuid.uuid4().hex
|
||||
background_job_create(job_id, "magazine-rotate-page", fingerprint=f"{doc_id}:{page}")
|
||||
asyncio.create_task(asyncio.to_thread(_reocr_page_sync, doc_id, job_id, page, url, model, fb_url, fb_model, degrees))
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
<meta name="format-detection" content="telephone=no">
|
||||
<meta name="color-scheme" content="light dark">
|
||||
<meta name="theme-color" content="#2563EB">
|
||||
<meta name="app-version" content="1.56.1">
|
||||
<meta name="app-version" content="1.56.2">
|
||||
<link rel="manifest" href="/manifest.webmanifest">
|
||||
<link rel="icon" href="/static/icon.svg" type="image/svg+xml">
|
||||
<link rel="apple-touch-icon" href="/static/icon.svg">
|
||||
@ -27,12 +27,12 @@
|
||||
|
||||
<!-- ── Core styles (local — no CDN dependency for first paint) ────────── -->
|
||||
<link rel="stylesheet" href="/static/vendor/mdi/materialdesignicons.min.css">
|
||||
<link rel="stylesheet" href="/static/style.css?v=1.56.1">
|
||||
<link rel="stylesheet" href="/static/css/resource-diagnostics.css?v=1.56.1">
|
||||
<link rel="stylesheet" href="/static/css/background-jobs.css?v=1.56.1">
|
||||
<link rel="stylesheet" href="/static/css/voice-design.css?v=1.56.1">
|
||||
<link rel="stylesheet" href="/static/css/rehearser.css?v=1.56.1">
|
||||
<link rel="stylesheet" href="/static/css/audiobook.css?v=1.56.1">
|
||||
<link rel="stylesheet" href="/static/style.css?v=1.56.2">
|
||||
<link rel="stylesheet" href="/static/css/resource-diagnostics.css?v=1.56.2">
|
||||
<link rel="stylesheet" href="/static/css/background-jobs.css?v=1.56.2">
|
||||
<link rel="stylesheet" href="/static/css/voice-design.css?v=1.56.2">
|
||||
<link rel="stylesheet" href="/static/css/rehearser.css?v=1.56.2">
|
||||
<link rel="stylesheet" href="/static/css/audiobook.css?v=1.56.2">
|
||||
|
||||
|
||||
<!-- ── Flag icons — non-blocking (loaded async, icons appear after JS) ── -->
|
||||
@ -403,7 +403,7 @@ window.toggleNavTree = function(treeId, chevronId) {
|
||||
</script>
|
||||
|
||||
<!-- loader.js: fetches sections → loads JS modules → removes skeleton -->
|
||||
<script src="/static/loader.js?v=1.56.1"></script>
|
||||
<script src="/static/loader.js?v=1.56.2"></script>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@ -175,6 +175,15 @@ function setupMagIngestPreviewResize() {
|
||||
});
|
||||
}
|
||||
|
||||
// The URL of a page's picture. It carries the picture's version (its modification time, stamped on each page by the
|
||||
// layout endpoint), so when a page is turned or re-rendered the URL changes and every browser fetches it again instead
|
||||
// of trusting a copy it cached under the old one.
|
||||
function magPageSrc(page, docId) {
|
||||
const id = docId || state.doc?.id;
|
||||
const meta = (state.layout?.pages || []).find((p) => p.page === Number(page));
|
||||
return '/api/magazine/' + id + '/pages/' + page + (meta && meta.image_v ? '?v=' + meta.image_v : '');
|
||||
}
|
||||
|
||||
// ── Library ──────────────────────────────────────────────────────────────────
|
||||
|
||||
async function magLoadLibrary() {
|
||||
@ -773,7 +782,7 @@ async function pollIngest() {
|
||||
if (data.preview.page !== lastPreviewPage) {
|
||||
lastPreviewPage = data.preview.page;
|
||||
const img = $('mag-ingest-preview-img');
|
||||
if (img) img.src = '/api/magazine/' + state.doc.id + '/pages/' + data.preview.page;
|
||||
if (img) img.src = '/api/magazine/' + state.doc.id + '/pages/' + data.preview.page + '?t=' + Date.now();
|
||||
}
|
||||
magRenderPreviewBoxes(data.preview.page, data.preview.boxes || [], data.preview.stage);
|
||||
const textEl = $('mag-ingest-preview-text');
|
||||
@ -1242,7 +1251,7 @@ function renderPagesView(host) {
|
||||
};
|
||||
html += '<div class="mag-card-wrap" data-page="' + p + '"><div class="mag-card is-page-' + st + (guide.needs ? ' needs-you' : '') + '">' +
|
||||
'<button type="button" class="mag-card-thumb-btn" title="Review page ' + p + '" aria-label="Review page ' + p + '" onclick="' + reviewJs + '">' +
|
||||
'<img class="mag-card-thumb" loading="lazy" alt="Page ' + p + '" src="/api/magazine/' + esc(state.doc.id) + '/pages/' + p + '"></button>' +
|
||||
'<img class="mag-card-thumb" loading="lazy" alt="Page ' + p + '" src="' + esc(magPageSrc(p)) + '"></button>' +
|
||||
'<span class="mag-card-title">Page ' + p + '</span>' +
|
||||
'<span class="mag-card-count">' + (arts.length ? arts.length + (arts.length === 1 ? ' article' : ' articles') : 'no article') + '</span>' +
|
||||
magQualityBadge(quality.get(p), p) +
|
||||
@ -1793,7 +1802,7 @@ function magReviewPageColumn(page, wrapId, interactive) {
|
||||
let pageHtml = '';
|
||||
if (page != null) {
|
||||
pageHtml = '<div class="mag-review-page-wrap' + (interactive ? '' : ' is-context-page') + '" id="' + wrapId + '">' +
|
||||
'<img src="/api/magazine/' + state.doc.id + '/pages/' + page + '" alt="Page ' + page + '">' +
|
||||
'<img src="' + esc(magPageSrc(page)) + '" alt="Page ' + page + '">' +
|
||||
(interactive ? '<div id="mag-draw-rect" class="mag-review-box mag-draw-rect" hidden></div>' : '');
|
||||
regions.forEach((r, i) => {
|
||||
if (!pageMeta) return;
|
||||
@ -2003,6 +2012,7 @@ function renderReview() {
|
||||
'<button type="button" class="btn-sm" id="mag-turn-flip" onclick="magRotatePage(180)" ' +
|
||||
'title="The page is upside down: turn it 180° and read it again">' +
|
||||
'<span class="mdi mdi-flip-vertical" aria-hidden="true"></span> Turn 180°</button>' +
|
||||
magTurnStateHtml() +
|
||||
'</span>' +
|
||||
'<button type="button" class="btn-sm" id="mag-reocr-btn" onclick="magReocrPage()">' +
|
||||
'<span class="mdi mdi-text-recognition"></span> OCR again (this page)</button>' +
|
||||
@ -3618,6 +3628,22 @@ async function magReocrPage() {
|
||||
renderReview();
|
||||
}
|
||||
|
||||
// How far this page has been turned from the PDF's own orientation, in words, with a way back. Turns add up (a right
|
||||
// turn and a half turn is three quarters), so pressing the buttons more than once is easy to lose track of.
|
||||
function magTurnNow(page) {
|
||||
const meta = (state.layout?.pages || []).find((p) => p.page === Number(page));
|
||||
return ((Number(meta?.rotation) || 0) % 360 + 360) % 360;
|
||||
}
|
||||
|
||||
function magTurnStateHtml() {
|
||||
const now = magTurnNow(state.reviewPage);
|
||||
if (!now) return '';
|
||||
return '<span class="mag-turn-state" id="mag-turn-state" role="status">Turned ' + now + '° clockwise from the scan</span>' +
|
||||
'<button type="button" class="btn-sm" id="mag-turn-undo" onclick="magRotatePage(' + (360 - now) + ')" ' +
|
||||
'title="Turn the page back to the orientation the PDF gives it, and read it again">' +
|
||||
'<span class="mdi mdi-backup-restore" aria-hidden="true"></span> Undo turns</button>';
|
||||
}
|
||||
|
||||
// A page scanned sideways or upside down: turn the page itself, then read it again. The boxes and text
|
||||
// produced from the wrong orientation are replaced, so this asks first; approvals carry over where the new
|
||||
// regions line up with the old ones. Turning is not covered by Undo (it rewrites the page image), so the
|
||||
@ -3634,7 +3660,7 @@ async function magRotatePage(degrees) {
|
||||
{ title: what + '?', okLabel: 'Turn and read again' });
|
||||
if (!ok) return;
|
||||
magOfferUndo(null); // an older Undo would put old-orientation boxes on the turned page
|
||||
const buttons = ['mag-turn-left', 'mag-turn-right', 'mag-turn-flip', 'mag-reocr-btn'].map($).filter(Boolean);
|
||||
const buttons = ['mag-turn-left', 'mag-turn-right', 'mag-turn-flip', 'mag-turn-undo', 'mag-reocr-btn'].map($).filter(Boolean);
|
||||
const statusEl = $('mag-reocr-status');
|
||||
const busy = (on, text) => { buttons.forEach((b) => { b.disabled = on; }); if (statusEl) statusEl.textContent = text || ''; };
|
||||
busy(true, 'Turning page ' + page + '…');
|
||||
@ -4732,7 +4758,7 @@ function showPage(page, bbox) {
|
||||
const img = $('mag-page-img');
|
||||
const box = $('mag-page-box');
|
||||
if (!img) return;
|
||||
const src = '/api/magazine/' + state.doc.id + '/pages/' + page;
|
||||
const src = magPageSrc(page);
|
||||
if (img.getAttribute('src') !== src) img.setAttribute('src', src);
|
||||
img.setAttribute('alt', 'Page ' + page + ' of ' + (state.doc.title || 'the magazine'));
|
||||
const label = $('mag-page-label');
|
||||
@ -5165,7 +5191,7 @@ function magPlaylistPlay(articleId, options) {
|
||||
const cover = $('mag-playlist-cover');
|
||||
if (cover) {
|
||||
const firstPage = (a.pages || [])[0];
|
||||
if (firstPage) { cover.src = '/api/magazine/' + state.doc.id + '/pages/' + firstPage; cover.hidden = false; }
|
||||
if (firstPage) { cover.src = magPageSrc(firstPage); cover.hidden = false; }
|
||||
else cover.hidden = true;
|
||||
}
|
||||
renderPlaylistTracks();
|
||||
@ -5636,7 +5662,7 @@ function magMediaSessionSync() {
|
||||
title: a.title || 'Untitled',
|
||||
artist: a.author || state.doc?.title || '',
|
||||
album: [state.doc?.title, state.doc?.issue].filter(Boolean).join(' — '),
|
||||
artwork: first ? [{ src: '/api/magazine/' + state.doc.id + '/pages/' + first, type: 'image/png' }] : [],
|
||||
artwork: first ? [{ src: magPageSrc(first), type: 'image/png' }] : [],
|
||||
});
|
||||
navigator.mediaSession.playbackState = audio.paused ? 'paused' : 'playing';
|
||||
} catch (_) { /* metadata is presentation only */ }
|
||||
|
||||
@ -5083,7 +5083,7 @@ input[list="lang-codes"] {
|
||||
.mag-review-pagenav-spacer { flex: 1; min-width: 0; }
|
||||
|
||||
/* ── Split screen: rendered page (left) + extracted blocks (right) ───────── */
|
||||
.mag-review-split { display: grid; grid-template-columns: 1.1fr 1fr; gap: 18px; align-items: start; }
|
||||
.mag-review-split { display: grid; grid-template-columns: minmax(0, 1.1fr) minmax(0, 1fr); gap: 18px; align-items: start; } /* minmax(0,..): unbreakable content must not widen a column */
|
||||
.mag-review-page { position: sticky; top: 12px; }
|
||||
.mag-review-page-wrap { position: relative; border: 1px solid var(--border); border-radius: var(--radius);
|
||||
overflow: hidden; background: var(--panel); }
|
||||
@ -5649,3 +5649,9 @@ details.mag-report-group summary { cursor: pointer; margin-bottom: 6px; }
|
||||
.mag-turn-group { display: inline-flex; gap: 4px; padding: 2px; border: 1px dashed var(--border); border-radius: var(--radius); }
|
||||
.mag-type-picker-list.opens-up { top: auto; bottom: 100%; margin: 0 0 2px; }
|
||||
.mag-type-picker-list { overflow-y: auto; }
|
||||
|
||||
/* Content that cannot break (an OCR loop, a long URL) must never widen the review or push the page picture out:
|
||||
grid tracks are min-content by default, so every track that holds text is allowed to shrink and text wraps anywhere. */
|
||||
.mag-review-split > *, .mag-blocks, .mag-block { min-width: 0; }
|
||||
.mag-block-text, .mag-diff, .mag-article, .mag-driving-lyrics, .mag-page-article-title, .mag-toc-title { overflow-wrap: anywhere; }
|
||||
.mag-turn-state { align-self: center; padding: 0 6px; font-size: 12px; font-weight: 600; color: var(--subtext); }
|
||||
|
||||
@ -2189,3 +2189,123 @@ test.describe('Draw region panel layering', () => {
|
||||
expect(box.x + box.width).toBeLessThanOrEqual(vp.width);
|
||||
});
|
||||
});
|
||||
|
||||
test.describe('Unbreakable text cannot collapse the review', () => {
|
||||
const HUGE = 'Giro + 2.' + '1.'.repeat(2500);
|
||||
async function openWith(page, width = 1600) {
|
||||
await page.setViewportSize({ width, height: 900 });
|
||||
await mockBackend(page);
|
||||
const layout = { ...LAYOUT, blocks: [...LAYOUT.blocks,
|
||||
{ id: 'b9', page: 3, bbox: [50, 500, 200, 40], type: 'body', text: HUGE, reading_order: 3, confidence: 0.5, font_size: 10, font_name: '', bold: false, column: 0, source: 'vision_ocr' }] };
|
||||
await page.route(`**/api/magazine/${DOC_ID}/layout*`, route => route.fulfill({ json: layout }));
|
||||
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
|
||||
await navTo(page, 's-magazine');
|
||||
await page.locator('#mag-library-list .mag-card').first().click();
|
||||
await page.locator('#mag-contents-view-toggle').click();
|
||||
await page.locator('.mag-card-wrap').filter({ hasText: 'Page 3' }).getByRole('button', { name: /^Review/ }).click();
|
||||
await page.waitForSelector('#mag-undo-btn');
|
||||
await expect(page.locator('[data-block-row="b9"]')).toBeVisible();
|
||||
}
|
||||
|
||||
test('the page picture keeps its column and the list stays inside the window', async ({ page }) => {
|
||||
await openWith(page);
|
||||
const m = await page.evaluate(() => {
|
||||
const wrap = document.getElementById('mag-review-page-wrap').getBoundingClientRect();
|
||||
const split = document.querySelector('.mag-review-split');
|
||||
const cols = getComputedStyle(split).gridTemplateColumns.split(' ').map(parseFloat);
|
||||
const list = document.querySelector('.mag-blocks').getBoundingClientRect();
|
||||
return { wrapW: wrap.width, cols, listRight: list.right, vw: window.innerWidth, docW: document.documentElement.scrollWidth };
|
||||
});
|
||||
expect(m.wrapW).toBeGreaterThan(250); // the page picture is not squeezed to nothing
|
||||
expect(Math.min(...m.cols)).toBeGreaterThan(250); // both columns have real width
|
||||
expect(m.listRight).toBeLessThanOrEqual(m.vw + 1); // the list does not run off the window
|
||||
expect(m.docW).toBeLessThanOrEqual(m.vw + 1); // and the page gets no sideways scroll
|
||||
});
|
||||
|
||||
test('the same holds on a phone, where the columns stack', async ({ page }) => {
|
||||
await openWith(page, 390);
|
||||
const m = await page.evaluate(() => ({
|
||||
wrapW: document.getElementById('mag-review-page-wrap').getBoundingClientRect().width,
|
||||
docW: document.documentElement.scrollWidth, vw: window.innerWidth }));
|
||||
expect(m.wrapW).toBeGreaterThan(200);
|
||||
expect(m.docW).toBeLessThanOrEqual(m.vw + 1);
|
||||
});
|
||||
|
||||
test('the huge token is still shown, wrapped, so it can be fixed or deleted', async ({ page }) => {
|
||||
await openWith(page);
|
||||
const row = page.locator('[data-block-row="b9"] .mag-block-text');
|
||||
await expect(row).toContainText('Giro + 2.1.1.1');
|
||||
const box = await row.boundingBox();
|
||||
expect(box.width).toBeLessThan(1600);
|
||||
expect(box.height).toBeGreaterThan(100); // wrapped onto many lines instead of one 20,000 px line
|
||||
await expect(page.locator('[data-block-row="b9"]').getByRole('button', { name: /Delete region/ })).toBeVisible();
|
||||
});
|
||||
});
|
||||
|
||||
test.describe('A turned page shows at once, and its net turn is visible', () => {
|
||||
async function open(page, { rotation = 0 } = {}) {
|
||||
await mockBackend(page);
|
||||
const state = { v: 100, rotation, rotates: [] };
|
||||
await page.route(`**/api/magazine/${DOC_ID}/layout*`, route => route.fulfill({
|
||||
json: { ...LAYOUT, pages: LAYOUT.pages.map(p => ({ ...p, image_v: state.v, ...(state.rotation ? { rotation: state.rotation } : {}) })) } }));
|
||||
await page.route(`**/api/magazine/${DOC_ID}/pages/3/rotate`, route => {
|
||||
state.rotates.push(route.request().postDataJSON());
|
||||
state.v += 50; state.rotation = (state.rotation + ((state.rotates.at(-1).degrees % 360) + 360)) % 360; // what the server would now report
|
||||
return route.fulfill({ json: { job_id: 'rot1', degrees: state.rotates.at(-1).degrees } });
|
||||
});
|
||||
await page.route('**/api/jobs/rot1', route => route.fulfill({ json: { done: true, error: null, detail: 'done' } }));
|
||||
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
|
||||
await navTo(page, 's-magazine');
|
||||
await page.locator('#mag-library-list .mag-card').first().click();
|
||||
await page.locator('#mag-contents-view-toggle').click();
|
||||
await page.locator('.mag-card-wrap').filter({ hasText: 'Page 3' }).getByRole('button', { name: /^Review/ }).click();
|
||||
await page.waitForSelector('#mag-undo-btn');
|
||||
return state;
|
||||
}
|
||||
|
||||
test('every picture of a page carries its version, so a changed picture is a new URL', async ({ page }) => {
|
||||
await open(page);
|
||||
await expect(page.locator('#mag-review-page-wrap img')).toHaveAttribute('src', new RegExp(`/pages/3\\?v=100$`));
|
||||
});
|
||||
|
||||
test('after a turn the page picture is requested again under its new version', async ({ page }) => {
|
||||
const state = await open(page);
|
||||
await page.getByRole('button', { name: /Turn right/ }).click();
|
||||
await page.getByRole('alertdialog').getByRole('button', { name: 'Turn and read again' }).click();
|
||||
await expect.poll(() => state.rotates.length).toBe(1);
|
||||
await expect(page.locator('#mag-review-page-wrap img')).toHaveAttribute('src', /\/pages\/3\?v=150$/);
|
||||
});
|
||||
|
||||
test('the page grid thumbnails use the version too', async ({ page }) => {
|
||||
await open(page);
|
||||
await page.getByRole('button', { name: /^Contents$/ }).click();
|
||||
await expect(page.locator('.mag-card-wrap[data-page="3"] img.mag-card-thumb')).toHaveAttribute('src', /\/pages\/3\?v=100$/);
|
||||
});
|
||||
|
||||
test('a page that has not been turned shows no turn state', async ({ page }) => {
|
||||
await open(page);
|
||||
await expect(page.locator('#mag-turn-state')).toHaveCount(0);
|
||||
await expect(page.getByRole('button', { name: /Undo turns/ })).toHaveCount(0);
|
||||
});
|
||||
|
||||
test('a turned page says by how much, and Undo turns sends what returns it to the PDF orientation', async ({ page }) => {
|
||||
const state = await open(page, { rotation: 270 });
|
||||
await expect(page.locator('#mag-turn-state')).toHaveText('Turned 270° clockwise from the scan');
|
||||
await page.getByRole('button', { name: /Undo turns/ }).click();
|
||||
await page.getByRole('alertdialog').getByRole('button', { name: 'Turn and read again' }).click();
|
||||
await expect.poll(() => state.rotates).toEqual([{ degrees: 90 }]); // 270 + 90 = a full circle
|
||||
await expect(page.locator('#mag-turn-state')).toHaveCount(0); // and the state line is gone once it is back to 0
|
||||
});
|
||||
|
||||
test('the server refusing a second turn while one runs is explained and the buttons come back', async ({ page }) => {
|
||||
await open(page);
|
||||
await page.route(`**/api/magazine/${DOC_ID}/pages/3/rotate`, route => route.fulfill({
|
||||
status: 409, json: { detail: 'Page 3 is still being turned or read; wait for it to finish before turning it again' } }));
|
||||
await page.getByRole('button', { name: /Turn right/ }).click();
|
||||
await page.getByRole('alertdialog').getByRole('button', { name: 'Turn and read again' }).click();
|
||||
await expect(page.locator('.toast, #toast, [role="status"]').filter({ hasText: 'still being turned or read' }).first()).toBeVisible();
|
||||
await expect(page.getByRole('button', { name: /Turn right/ })).toBeEnabled();
|
||||
});
|
||||
});
|
||||
|
||||
@ -298,6 +298,8 @@ class RenderTests(unittest.TestCase):
|
||||
self.assertTrue(any(b["text"].startswith("A sentence") for b in plain["blocks"]))
|
||||
turned = mag.extract_pages(d / "source.pdf", d / "turned", render_dpi=50, page_rotations={1: 90})
|
||||
self.assertEqual((turned["pages"][0]["width"], turned["pages"][0]["height"]), (200.0, 300.0))
|
||||
self.assertEqual(turned["pages"][0].get("rotation"), 90) # the stored turn is visible in the layout again
|
||||
self.assertNotIn("rotation", plain["pages"][0])
|
||||
with Image.open(d / "turned" / "pages" / "0001.png") as im, Image.open(d / "plain" / "pages" / "0001.png") as ref:
|
||||
self.assertEqual(im.size, (ref.size[1], ref.size[0]))
|
||||
# The PDF's own text boxes belong to the unturned page; on a turned page only OCR of the render counts.
|
||||
@ -311,6 +313,75 @@ class RenderTests(unittest.TestCase):
|
||||
self.assertEqual([x["text"] for x in a["blocks"]], [x["text"] for x in b["blocks"]])
|
||||
|
||||
|
||||
class ImageVersionAndJobGuardTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
import routes.magazine as routes
|
||||
self.routes = routes
|
||||
self.lib = Path(tempfile.mkdtemp(prefix="ttvc-lib-"))
|
||||
p = mock.patch.object(routes, "_LIB", self.lib)
|
||||
p.start(); self.addCleanup(p.stop)
|
||||
app = FastAPI(); app.include_router(routes.router)
|
||||
self.client = TestClient(app)
|
||||
self.doc = self.client.post("/api/magazine/docs", json={"title": "T"}).json()["id"]
|
||||
self.d = self.lib / self.doc
|
||||
(self.d / "pages").mkdir()
|
||||
self.png = self.d / "pages" / "0002.png"
|
||||
Image.new("RGB", (50, 80), "white").save(self.png)
|
||||
(self.d / "layout.json").write_text(json.dumps({"pages": [{"page": 2, "width": 50.0, "height": 80.0}, {"page": 3, "width": 50.0, "height": 80.0}],
|
||||
"blocks": [], "images": []}))
|
||||
|
||||
def test_each_page_is_stamped_with_its_picture_version_and_it_changes_with_the_picture(self):
|
||||
first = self.client.get(f"/api/magazine/{self.doc}/layout").json()["pages"]
|
||||
self.assertEqual(first[0]["image_v"], int(self.png.stat().st_mtime))
|
||||
self.assertEqual(first[1]["image_v"], 0) # page 3 has no picture: 0, not an error
|
||||
os.utime(self.png, (self.png.stat().st_atime, self.png.stat().st_mtime + 100))
|
||||
again = self.client.get(f"/api/magazine/{self.doc}/layout").json()["pages"]
|
||||
self.assertEqual(again[0]["image_v"], first[0]["image_v"] + 100)
|
||||
|
||||
def test_the_stamp_is_also_there_when_one_page_is_asked_for(self):
|
||||
pages = self.client.get(f"/api/magazine/{self.doc}/layout?page=2").json()["pages"]
|
||||
self.assertEqual([p["page"] for p in pages], [2])
|
||||
self.assertGreater(pages[0]["image_v"], 0)
|
||||
|
||||
def test_the_stamp_is_not_written_into_the_saved_layout(self):
|
||||
self.client.get(f"/api/magazine/{self.doc}/layout")
|
||||
self.assertNotIn("image_v", json.loads((self.d / "layout.json").read_text())["pages"][0])
|
||||
|
||||
def job(self, **kw):
|
||||
base = {"done": False, "fingerprint": f"{self.doc}:2", "updated": __import__("time").time()}
|
||||
base.update(kw)
|
||||
return base
|
||||
|
||||
def running(self, jobs, page=2):
|
||||
with mock.patch.object(self.routes, "background_job_list", side_effect=lambda kind, limit=50: jobs if kind == "magazine-rotate-page" else []):
|
||||
return self.routes._page_job_running(self.doc, page)
|
||||
|
||||
def test_a_job_in_flight_for_the_same_page_counts(self):
|
||||
self.assertTrue(self.running([self.job()]))
|
||||
|
||||
def test_finished_stale_and_other_page_jobs_do_not(self):
|
||||
self.assertFalse(self.running([self.job(done=True)]))
|
||||
self.assertFalse(self.running([self.job(updated=__import__("time").time() - 3600)])) # died with the server
|
||||
self.assertFalse(self.running([self.job(fingerprint=f"{self.doc}:3")]))
|
||||
self.assertFalse(self.running([self.job(fingerprint="other:2")]))
|
||||
self.assertFalse(self.running([]))
|
||||
|
||||
def test_a_second_turn_or_re_read_while_one_runs_is_refused_and_changes_nothing(self):
|
||||
before = self.png.read_bytes()
|
||||
settings = {"vision_ocr_url": "http://ocr", "vision_ocr_model": "m"}
|
||||
with mock.patch.object(self.routes, "_load_settings", return_value=settings), \
|
||||
mock.patch.object(self.routes, "_page_job_running", return_value=True), \
|
||||
mock.patch.object(self.routes.asyncio, "create_task") as task:
|
||||
turn = self.client.post(f"/api/magazine/{self.doc}/pages/2/rotate", json={"degrees": 90})
|
||||
reread = self.client.post(f"/api/magazine/{self.doc}/pages/2/reocr", json={})
|
||||
self.assertEqual((turn.status_code, reread.status_code), (409, 409))
|
||||
self.assertIn("wait for it to finish", turn.json()["detail"])
|
||||
task.assert_not_called()
|
||||
self.assertEqual(self.png.read_bytes(), before)
|
||||
|
||||
|
||||
class EndpointTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
from fastapi import FastAPI
|
||||
|
||||
@ -946,6 +946,43 @@ class CleanUpBoxesTests(unittest.TestCase):
|
||||
self.assertEqual(types["long"][0], "body") # real text caught in a loose frame stays
|
||||
self.assertEqual(types["outside"][0], "body")
|
||||
|
||||
def test_a_region_that_is_one_endless_repeated_token_is_noise_but_real_text_with_repeats_is_not(self):
|
||||
"""Seen on a real advert page: 'Giro + 2.1.1.1.1 ...' thousands of characters long, no spaces."""
|
||||
from core.magazine import clean_up_regions
|
||||
junk = "Giro + " + "2." + "1." * 1700
|
||||
layout = self.base([
|
||||
self.blk("loop", 10, 10, 300, 40, junk),
|
||||
self.blk("digits", 10, 60, 300, 40, "0" * 120),
|
||||
self.blk("dots", 10, 110, 300, 12, "Kapitel 1 " + "." * 200 + " 17", "caption"),
|
||||
self.blk("toc", 10, 150, 300, 12, "2.1.1 Einleitung 2.1.2 Methode 2.1.3 Ergebnisse 2.1.4 Diskussion 2.2 Fazit 3.1.1.1 Anhang"),
|
||||
self.blk("url", 10, 170, 300, 12, "https://example.org/artikel/2026/09/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0?ref=newsletter&utm_source=heise"),
|
||||
self.blk("prose", 10, 190, 300, 40, "Ein ganz normaler Absatz mit genug Woertern, der nichts Verdaechtiges enthaelt. " * 3),
|
||||
self.blk("short", 10, 240, 300, 12, "1." * 10),
|
||||
])
|
||||
types = self.types(clean_up_regions(layout)[0])
|
||||
self.assertEqual(types["loop"], ("furniture", "ocr_noise"))
|
||||
self.assertEqual(types["digits"], ("furniture", "ocr_noise"))
|
||||
for kept in ("dots", "toc", "url", "prose", "short"):
|
||||
self.assertNotEqual(types[kept][1], "ocr_noise", kept)
|
||||
|
||||
def test_the_repetition_share_counts_only_letter_or_digit_units_repeated_thirty_times(self):
|
||||
from core.magazine import glued_repetition_share
|
||||
self.assertGreater(glued_repetition_share("1." * 100), 0.9)
|
||||
self.assertGreater(glued_repetition_share("ab" * 40), 0.9)
|
||||
self.assertEqual(glued_repetition_share("." * 500), 0.0) # leader dots and rules are not text, so not counted
|
||||
self.assertEqual(glued_repetition_share("- " * 100), 0.0)
|
||||
self.assertEqual(glued_repetition_share("1." * 29), 0.0) # one short of the threshold
|
||||
self.assertEqual(glued_repetition_share(""), 0.0)
|
||||
self.assertAlmostEqual(glued_repetition_share("Hallo " + "xy" * 50 + " Welt"), 100 / 111, places=2)
|
||||
|
||||
def test_clean_up_is_idempotent_for_the_new_rule(self):
|
||||
from core.magazine import clean_up_regions
|
||||
layout = self.base([self.blk("loop", 10, 10, 300, 40, "2." + "1." * 500)])
|
||||
once = clean_up_regions(layout)[0]
|
||||
twice, report = clean_up_regions(once)
|
||||
self.assertEqual(self.types(twice)["loop"], ("furniture", "ocr_noise"))
|
||||
self.assertEqual(report["ocr_noise"], []) # nothing left to re-type the second time
|
||||
|
||||
def test_one_or_two_characters_are_noise_but_a_phone_number_is_not(self):
|
||||
from core.magazine import clean_up_regions
|
||||
layout = self.base([
|
||||
|
||||
Loading…
Reference in New Issue
Block a user