Compare commits

...

3 Commits

Author SHA1 Message Date
916d19d22f Release v1.56.2
Some checks are pending
Test and SonarQube / quality (push) Waiting to run
2026-09-30 15:01:34 +02:00
884bbf33aa fix(magazine): a turned page shows at once (versioned picture URLs), turns are visible and undoable, one turn at a time
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 13:35:25 +02:00
28d6439604 fix(magazine): unbreakable text can no longer collapse the review; endless repeated tokens are OCR noise
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 13:13:40 +02:00
12 changed files with 362 additions and 21 deletions

View File

@ -7,6 +7,14 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
## [Unreleased]
---
## [1.56.2] — 2026-09-30
### Fixed
- **A turned page could keep showing its old picture, which made turning look broken (and invited pressing the buttons again).** Page pictures had no cache header until recently, and a browser may treat an unchanged URL as fresh for days, so after a successful turn the review could keep showing the old image. The layout now stamps every page with its picture's version (the file's modification time) and every page picture URL (review, page grid, reading pane, cover, lock-screen artwork) carries it, so a changed picture is a new URL in any browser. Turns also add up (right + half turn = three quarters), so the toolbar now says how far the page is turned from the scan ("Turned 270° clockwise from the scan") with an **Undo turns** button back to the PDF's own orientation, and a second turn or re-read of a page that is still being processed is refused (HTTP 409) instead of stacking. `extract_pages` records a stored turn in the page's layout entry, so it survives a re-import visibly. Verified end to end with the real OCR engines on a scratch copy of c't p. 156: turn 90 gives an upright 886x1039 page read at word-F1 0.878, Undo turns restores the 1039x886 original and clears the stored turn.
- **A page's picture could vanish from the review (seen on c't p. 176).** One region held a 3,408-character token with no spaces (`Giro + 2.1.1.1.1…`, the OCR engine stuck in a loop). Its width was taken as the minimum width of the list column, which widened to 20,305 px and squeezed the page-picture column to 2 px. The review grid's tracks can now shrink (`minmax(0, …)`), and text in the review list, reading view, figure diffs and page cards wraps anywhere, so no content can collapse the layout again; a long URL or a loop now simply wraps. The cause is also handled: *Clean up boxes* (and every import and re-read, which run it) re-type a region that is mostly one endless repeated letter/digit token as OCR noise (kept, visible, reversible in the type picker, not read aloud). Leader dots, rules, section numbers, phone numbers, prices and URLs are not affected; run over all three of your magazines (about 6,000 regions) the rule flags exactly that one region on page 176 in each and nothing else. To clear it on an existing magazine press *Clean up boxes* or *Fix this page automatically*.
---
## [1.56.1] — 2026-09-30
@ -4616,7 +4624,8 @@ Initial feature-complete release.
---
[Unreleased]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.1...HEAD
[Unreleased]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.2...HEAD
[1.56.2]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.1...v1.56.2
[1.56.1]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.56.0...v1.56.1
[1.56.0]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.55.0...v1.56.0
[1.55.0]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.54.0...v1.55.0

View File

@ -407,6 +407,13 @@ of the two pages the buttons act on, as with the other page tools.
are re-measured. On a test page, reading it sideways scored word-F1 0.02 and reading it after turning 0.88 (an ordinary page scores
about 0.88). The turn is remembered: re-running detection or re-importing does not flip the page back.
**Keeping track of turns.** Turns add up: a right turn plus a half turn is three quarters. The toolbar says so in words (for example
"Turned 270° clockwise from the scan") and offers **Undo turns**, which returns the page to the orientation the PDF itself gives it. Only one
turn or re-read of a page can run at a time: pressing a button again while one is still working is refused with a message, instead of stacking
a second turn. The turned picture appears as soon as the page is re-read; page pictures are requested under a version number, so a browser
that remembered the old picture fetches the new one. Note what a PDF's own orientation can be: some pages are stored sideways on purpose
(c't page 156 is a landscape page that reads sideways), and one **Turn right** is then the whole fix.
**What is replaced.** The page's boxes and text are replaced by the new reading, so corrections you made to that page by hand are lost, and
articles that used the page go back to "not approved" where the new boxes do not line up with the old ones. **Undo does not cover a turn**
(it changes the page image); if you turned the wrong way, press the opposite button.

View File

@ -1 +1 @@
1.56.1
1.56.2

View File

@ -477,6 +477,16 @@ Verified end-to-end against a synthetic 8-page issue (contents page, two-column
feature with a byline and a page-8 jump, full-page advert, second feature,
reviews) and against an image-only rendering of the same file to exercise OCR.
## Endless-token regions and unbreakable text
A recogniser stuck in a loop can emit one token with no spaces at all (c't p. 176: `Giro + 2.1.1.1.1…`, 3,408 characters). `_looks_like_repetition_loop`
ignores a repeated unit without a real word on purpose (leader dots, rules), so it did not see it. `glued_repetition_share(text)` counts the share of a text
covered by glued runs of a short unit (<= 6 characters) containing a letter or digit, repeated 30+ times; `clean_up_regions` re-types a region of at least 60
characters where that share is >= 0.5 as `ocr_noise` (furniture, reversible). Measured on the three c't magazines (~6,000 regions): exactly one region each, none
other. Independently of the data, the review must survive such text: a grid track is min-content unless told otherwise, so one unbreakable word widened the
list column to 20,305 px and left the page picture 2 px. `.mag-review-split` uses `minmax(0, ...)` tracks, the list containers have `min-width: 0` and the text
classes `overflow-wrap: anywhere`.
## Turning a page scanned sideways
`POST /api/magazine/{id}/pages/{page}/rotate {degrees: 90|180|270|-90}` starts the same background job as "OCR again"

View File

@ -1251,6 +1251,24 @@ def _looks_like_repetition_loop(text: str) -> bool:
for m in _REPETITION_RUN.finditer(text or ""))
# A recogniser stuck in a loop can also emit one endless token with no spaces at all ("2.1.1.1.1.1 ..." thousands of
# characters long, seen on a real shopping-advert page). _looks_like_repetition_loop deliberately ignores a repeated unit
# without a real word, so leader dots and rules survive -- but a short unit that contains a letter or digit, repeated
# this many times back to back, is never real text: a phone number, a price, a section number or a URL do not come
# close to 30 identical repeats.
_GLUED_RUN = re.compile(r"(\S{1,6}?)\1{29,}")
_GLUED_UNIT_CONTENT = re.compile(r"[A-Za-z0-9À-ÖØ-öø-ÿ]")
def glued_repetition_share(text: str) -> float:
"""The share of `text` taken up by glued runs of a short letter/digit unit repeated 30+ times (0.0 - 1.0)."""
text = text or ""
if not text:
return 0.0
covered = sum(m.end() - m.start() for m in _GLUED_RUN.finditer(text) if _GLUED_UNIT_CONTENT.search(m.group(1)))
return covered / len(text)
def vision_ocr_page(png_path: Path, base_url: str, model: str,
timeout: float = 150.0) -> list[dict[str, Any]] | None:
"""OCR one rendered page with a connected vision-capable OCR engine.
@ -2652,6 +2670,7 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
"page": page_no,
"width": rect.width,
"height": rect.height,
"rotation": extra_turn,
"blocks": blocks,
"image_boxes": image_boxes,
"images": page_images,
@ -2716,7 +2735,8 @@ def extract_pages(pdf_path: Path, out_dir: Path, render_dpi: int = 110,
return {
"page_count": doc.page_count,
"pages": [{"page": p["page"], "width": p["width"], "height": p["height"]}
"pages": [{"page": p["page"], "width": p["width"], "height": p["height"],
**({"rotation": p["rotation"]} if p.get("rotation") else {})}
for p in raw_pages],
"blocks": [b.to_json() for b in all_blocks],
"images": all_images,
@ -4755,8 +4775,11 @@ def clean_up_regions(layout: dict[str, Any], pages: Iterable[int] | None = None)
if text.startswith("(no text recognised"):
continue
if (len(re.sub(r"[\W_]", "", text)) <= 2 or text.lower().startswith("[non-text")
or b["bbox"][3] < _MIN_TEXT_BOX_HEIGHT):
mark(b, "ocr_noise") # includes the engine's own "[Non-Text]" and hairline boxes
or b["bbox"][3] < _MIN_TEXT_BOX_HEIGHT
or (len(text) >= 60 and glued_repetition_share(text) >= 0.5)):
# includes the engine's own "[Non-Text]", hairline boxes, and a region that is mostly one endless
# repeated token (a recogniser stuck in a loop)
mark(b, "ocr_noise")
elif len(text) <= 80 and any(_area_inside(b["bbox"], p) >= 0.9 for p in pictures.get(page_no, [])):
mark(b, "in_picture")
for small in text_boxes:

View File

@ -40,6 +40,7 @@ from core.constants import CONFIG_DIR
from core.database import (
background_job_create,
background_job_get,
background_job_list,
background_job_update,
)
from core import spelling
@ -551,9 +552,20 @@ async def magazine_layout(doc_id: str, page: int | None = None):
layout = dict(layout)
layout["blocks"] = [b for b in layout.get("blocks", []) if b.get("page") == page]
layout["pages"] = [p for p in layout.get("pages", []) if p.get("page") == page]
# The page picture's URL carries this number, so a turned or re-rendered page is fetched again by every browser
# whatever it cached earlier (a browser may treat an unchanged URL as fresh for days).
layout = dict(layout)
layout["pages"] = [{**p, "image_v": _page_image_version(d, p.get("page"))} for p in layout.get("pages", [])]
return layout
def _page_image_version(d: Path, page: Any) -> int:
try:
return int((d / "pages" / f"{int(page):04d}.png").stat().st_mtime)
except (OSError, TypeError, ValueError):
return 0
def _next_article_id(articles: list[dict]) -> str:
nums = [int(m.group(1)) for a in articles
if (m := re.fullmatch(r"article-(\d+)", a.get("id") or ""))]
@ -1054,6 +1066,22 @@ def _reocr_page_sync(doc_id: str, job_id: str, page: int,
background_job_update(job_id, stage="failed", done=True, error=str(exc)[:1000])
_PAGE_JOB_KINDS = ("magazine-rotate-page", "magazine-reocr-page")
_PAGE_JOB_STALE_SECONDS = 15 * 60 # a job that has not reported for this long died with the server
def _page_job_running(doc_id: str, page: int) -> bool:
"""True while a turn or re-read of this page is in flight. Two at once would read and rewrite the same image and
layout; pressing the button again because the first one seemed to do nothing must not stack a second turn."""
now = time.time()
for kind in _PAGE_JOB_KINDS:
for job in background_job_list(kind, 50):
if (job and not job.get("done") and job.get("fingerprint") == f"{doc_id}:{page}"
and now - float(job.get("updated") or 0) < _PAGE_JOB_STALE_SECONDS):
return True
return False
@router.post("/api/magazine/{doc_id}/pages/{page}/reocr")
async def magazine_reocr_page(doc_id: str, page: int, request: Request):
"""Force one page through the configured vision OCR engine again,
@ -1083,6 +1111,8 @@ async def magazine_reocr_page(doc_id: str, page: int, request: Request):
vision_ocr_fallback_model = str(
body.get("vision_ocr_fallback_model") or settings.get("vision_ocr_fallback_model") or "").strip() or None
if _page_job_running(doc_id, page):
raise HTTPException(409, f"Page {page} is still being read; wait for it to finish")
job_id = uuid.uuid4().hex
background_job_create(job_id, "magazine-reocr-page", fingerprint=f"{doc_id}:{page}")
asyncio.create_task(
@ -1119,6 +1149,8 @@ async def magazine_rotate_page(doc_id: str, page: int, request: Request):
url, model, fb_url, fb_model = _vision_args(settings)
if not (url and model):
raise HTTPException(400, "No vision OCR engine configured -- set one in Settings (turning re-reads the page)")
if _page_job_running(doc_id, page):
raise HTTPException(409, f"Page {page} is still being turned or read; wait for it to finish before turning it again")
job_id = uuid.uuid4().hex
background_job_create(job_id, "magazine-rotate-page", fingerprint=f"{doc_id}:{page}")
asyncio.create_task(asyncio.to_thread(_reocr_page_sync, doc_id, job_id, page, url, model, fb_url, fb_model, degrees))

View File

@ -10,7 +10,7 @@
<meta name="format-detection" content="telephone=no">
<meta name="color-scheme" content="light dark">
<meta name="theme-color" content="#2563EB">
<meta name="app-version" content="1.56.1">
<meta name="app-version" content="1.56.2">
<link rel="manifest" href="/manifest.webmanifest">
<link rel="icon" href="/static/icon.svg" type="image/svg+xml">
<link rel="apple-touch-icon" href="/static/icon.svg">
@ -27,12 +27,12 @@
<!-- ── Core styles (local — no CDN dependency for first paint) ────────── -->
<link rel="stylesheet" href="/static/vendor/mdi/materialdesignicons.min.css">
<link rel="stylesheet" href="/static/style.css?v=1.56.1">
<link rel="stylesheet" href="/static/css/resource-diagnostics.css?v=1.56.1">
<link rel="stylesheet" href="/static/css/background-jobs.css?v=1.56.1">
<link rel="stylesheet" href="/static/css/voice-design.css?v=1.56.1">
<link rel="stylesheet" href="/static/css/rehearser.css?v=1.56.1">
<link rel="stylesheet" href="/static/css/audiobook.css?v=1.56.1">
<link rel="stylesheet" href="/static/style.css?v=1.56.2">
<link rel="stylesheet" href="/static/css/resource-diagnostics.css?v=1.56.2">
<link rel="stylesheet" href="/static/css/background-jobs.css?v=1.56.2">
<link rel="stylesheet" href="/static/css/voice-design.css?v=1.56.2">
<link rel="stylesheet" href="/static/css/rehearser.css?v=1.56.2">
<link rel="stylesheet" href="/static/css/audiobook.css?v=1.56.2">
<!-- ── Flag icons — non-blocking (loaded async, icons appear after JS) ── -->
@ -403,7 +403,7 @@ window.toggleNavTree = function(treeId, chevronId) {
</script>
<!-- loader.js: fetches sections → loads JS modules → removes skeleton -->
<script src="/static/loader.js?v=1.56.1"></script>
<script src="/static/loader.js?v=1.56.2"></script>
</body>
</html>

View File

@ -175,6 +175,15 @@ function setupMagIngestPreviewResize() {
});
}
// The URL of a page's picture. It carries the picture's version (its modification time, stamped on each page by the
// layout endpoint), so when a page is turned or re-rendered the URL changes and every browser fetches it again instead
// of trusting a copy it cached under the old one.
function magPageSrc(page, docId) {
const id = docId || state.doc?.id;
const meta = (state.layout?.pages || []).find((p) => p.page === Number(page));
return '/api/magazine/' + id + '/pages/' + page + (meta && meta.image_v ? '?v=' + meta.image_v : '');
}
// ── Library ──────────────────────────────────────────────────────────────────
async function magLoadLibrary() {
@ -773,7 +782,7 @@ async function pollIngest() {
if (data.preview.page !== lastPreviewPage) {
lastPreviewPage = data.preview.page;
const img = $('mag-ingest-preview-img');
if (img) img.src = '/api/magazine/' + state.doc.id + '/pages/' + data.preview.page;
if (img) img.src = '/api/magazine/' + state.doc.id + '/pages/' + data.preview.page + '?t=' + Date.now();
}
magRenderPreviewBoxes(data.preview.page, data.preview.boxes || [], data.preview.stage);
const textEl = $('mag-ingest-preview-text');
@ -1242,7 +1251,7 @@ function renderPagesView(host) {
};
html += '<div class="mag-card-wrap" data-page="' + p + '"><div class="mag-card is-page-' + st + (guide.needs ? ' needs-you' : '') + '">' +
'<button type="button" class="mag-card-thumb-btn" title="Review page ' + p + '" aria-label="Review page ' + p + '" onclick="' + reviewJs + '">' +
'<img class="mag-card-thumb" loading="lazy" alt="Page ' + p + '" src="/api/magazine/' + esc(state.doc.id) + '/pages/' + p + '"></button>' +
'<img class="mag-card-thumb" loading="lazy" alt="Page ' + p + '" src="' + esc(magPageSrc(p)) + '"></button>' +
'<span class="mag-card-title">Page ' + p + '</span>' +
'<span class="mag-card-count">' + (arts.length ? arts.length + (arts.length === 1 ? ' article' : ' articles') : 'no article') + '</span>' +
magQualityBadge(quality.get(p), p) +
@ -1793,7 +1802,7 @@ function magReviewPageColumn(page, wrapId, interactive) {
let pageHtml = '';
if (page != null) {
pageHtml = '<div class="mag-review-page-wrap' + (interactive ? '' : ' is-context-page') + '" id="' + wrapId + '">' +
'<img src="/api/magazine/' + state.doc.id + '/pages/' + page + '" alt="Page ' + page + '">' +
'<img src="' + esc(magPageSrc(page)) + '" alt="Page ' + page + '">' +
(interactive ? '<div id="mag-draw-rect" class="mag-review-box mag-draw-rect" hidden></div>' : '');
regions.forEach((r, i) => {
if (!pageMeta) return;
@ -2003,6 +2012,7 @@ function renderReview() {
'<button type="button" class="btn-sm" id="mag-turn-flip" onclick="magRotatePage(180)" ' +
'title="The page is upside down: turn it 180° and read it again">' +
'<span class="mdi mdi-flip-vertical" aria-hidden="true"></span> Turn 180°</button>' +
magTurnStateHtml() +
'</span>' +
'<button type="button" class="btn-sm" id="mag-reocr-btn" onclick="magReocrPage()">' +
'<span class="mdi mdi-text-recognition"></span> OCR again (this page)</button>' +
@ -3618,6 +3628,22 @@ async function magReocrPage() {
renderReview();
}
// How far this page has been turned from the PDF's own orientation, in words, with a way back. Turns add up (a right
// turn and a half turn is three quarters), so pressing the buttons more than once is easy to lose track of.
function magTurnNow(page) {
const meta = (state.layout?.pages || []).find((p) => p.page === Number(page));
return ((Number(meta?.rotation) || 0) % 360 + 360) % 360;
}
function magTurnStateHtml() {
const now = magTurnNow(state.reviewPage);
if (!now) return '';
return '<span class="mag-turn-state" id="mag-turn-state" role="status">Turned ' + now + '° clockwise from the scan</span>' +
'<button type="button" class="btn-sm" id="mag-turn-undo" onclick="magRotatePage(' + (360 - now) + ')" ' +
'title="Turn the page back to the orientation the PDF gives it, and read it again">' +
'<span class="mdi mdi-backup-restore" aria-hidden="true"></span> Undo turns</button>';
}
// A page scanned sideways or upside down: turn the page itself, then read it again. The boxes and text
// produced from the wrong orientation are replaced, so this asks first; approvals carry over where the new
// regions line up with the old ones. Turning is not covered by Undo (it rewrites the page image), so the
@ -3634,7 +3660,7 @@ async function magRotatePage(degrees) {
{ title: what + '?', okLabel: 'Turn and read again' });
if (!ok) return;
magOfferUndo(null); // an older Undo would put old-orientation boxes on the turned page
const buttons = ['mag-turn-left', 'mag-turn-right', 'mag-turn-flip', 'mag-reocr-btn'].map($).filter(Boolean);
const buttons = ['mag-turn-left', 'mag-turn-right', 'mag-turn-flip', 'mag-turn-undo', 'mag-reocr-btn'].map($).filter(Boolean);
const statusEl = $('mag-reocr-status');
const busy = (on, text) => { buttons.forEach((b) => { b.disabled = on; }); if (statusEl) statusEl.textContent = text || ''; };
busy(true, 'Turning page ' + page + '…');
@ -4732,7 +4758,7 @@ function showPage(page, bbox) {
const img = $('mag-page-img');
const box = $('mag-page-box');
if (!img) return;
const src = '/api/magazine/' + state.doc.id + '/pages/' + page;
const src = magPageSrc(page);
if (img.getAttribute('src') !== src) img.setAttribute('src', src);
img.setAttribute('alt', 'Page ' + page + ' of ' + (state.doc.title || 'the magazine'));
const label = $('mag-page-label');
@ -5165,7 +5191,7 @@ function magPlaylistPlay(articleId, options) {
const cover = $('mag-playlist-cover');
if (cover) {
const firstPage = (a.pages || [])[0];
if (firstPage) { cover.src = '/api/magazine/' + state.doc.id + '/pages/' + firstPage; cover.hidden = false; }
if (firstPage) { cover.src = magPageSrc(firstPage); cover.hidden = false; }
else cover.hidden = true;
}
renderPlaylistTracks();
@ -5636,7 +5662,7 @@ function magMediaSessionSync() {
title: a.title || 'Untitled',
artist: a.author || state.doc?.title || '',
album: [state.doc?.title, state.doc?.issue].filter(Boolean).join(' — '),
artwork: first ? [{ src: '/api/magazine/' + state.doc.id + '/pages/' + first, type: 'image/png' }] : [],
artwork: first ? [{ src: magPageSrc(first), type: 'image/png' }] : [],
});
navigator.mediaSession.playbackState = audio.paused ? 'paused' : 'playing';
} catch (_) { /* metadata is presentation only */ }

View File

@ -5083,7 +5083,7 @@ input[list="lang-codes"] {
.mag-review-pagenav-spacer { flex: 1; min-width: 0; }
/* ── Split screen: rendered page (left) + extracted blocks (right) ───────── */
.mag-review-split { display: grid; grid-template-columns: 1.1fr 1fr; gap: 18px; align-items: start; }
.mag-review-split { display: grid; grid-template-columns: minmax(0, 1.1fr) minmax(0, 1fr); gap: 18px; align-items: start; } /* minmax(0,..): unbreakable content must not widen a column */
.mag-review-page { position: sticky; top: 12px; }
.mag-review-page-wrap { position: relative; border: 1px solid var(--border); border-radius: var(--radius);
overflow: hidden; background: var(--panel); }
@ -5649,3 +5649,9 @@ details.mag-report-group summary { cursor: pointer; margin-bottom: 6px; }
.mag-turn-group { display: inline-flex; gap: 4px; padding: 2px; border: 1px dashed var(--border); border-radius: var(--radius); }
.mag-type-picker-list.opens-up { top: auto; bottom: 100%; margin: 0 0 2px; }
.mag-type-picker-list { overflow-y: auto; }
/* Content that cannot break (an OCR loop, a long URL) must never widen the review or push the page picture out:
grid tracks are min-content by default, so every track that holds text is allowed to shrink and text wraps anywhere. */
.mag-review-split > *, .mag-blocks, .mag-block { min-width: 0; }
.mag-block-text, .mag-diff, .mag-article, .mag-driving-lyrics, .mag-page-article-title, .mag-toc-title { overflow-wrap: anywhere; }
.mag-turn-state { align-self: center; padding: 0 6px; font-size: 12px; font-weight: 600; color: var(--subtext); }

View File

@ -2189,3 +2189,123 @@ test.describe('Draw region panel layering', () => {
expect(box.x + box.width).toBeLessThanOrEqual(vp.width);
});
});
test.describe('Unbreakable text cannot collapse the review', () => {
const HUGE = 'Giro + 2.' + '1.'.repeat(2500);
async function openWith(page, width = 1600) {
await page.setViewportSize({ width, height: 900 });
await mockBackend(page);
const layout = { ...LAYOUT, blocks: [...LAYOUT.blocks,
{ id: 'b9', page: 3, bbox: [50, 500, 200, 40], type: 'body', text: HUGE, reading_order: 3, confidence: 0.5, font_size: 10, font_name: '', bold: false, column: 0, source: 'vision_ocr' }] };
await page.route(`**/api/magazine/${DOC_ID}/layout*`, route => route.fulfill({ json: layout }));
await page.goto('/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
await navTo(page, 's-magazine');
await page.locator('#mag-library-list .mag-card').first().click();
await page.locator('#mag-contents-view-toggle').click();
await page.locator('.mag-card-wrap').filter({ hasText: 'Page 3' }).getByRole('button', { name: /^Review/ }).click();
await page.waitForSelector('#mag-undo-btn');
await expect(page.locator('[data-block-row="b9"]')).toBeVisible();
}
test('the page picture keeps its column and the list stays inside the window', async ({ page }) => {
await openWith(page);
const m = await page.evaluate(() => {
const wrap = document.getElementById('mag-review-page-wrap').getBoundingClientRect();
const split = document.querySelector('.mag-review-split');
const cols = getComputedStyle(split).gridTemplateColumns.split(' ').map(parseFloat);
const list = document.querySelector('.mag-blocks').getBoundingClientRect();
return { wrapW: wrap.width, cols, listRight: list.right, vw: window.innerWidth, docW: document.documentElement.scrollWidth };
});
expect(m.wrapW).toBeGreaterThan(250); // the page picture is not squeezed to nothing
expect(Math.min(...m.cols)).toBeGreaterThan(250); // both columns have real width
expect(m.listRight).toBeLessThanOrEqual(m.vw + 1); // the list does not run off the window
expect(m.docW).toBeLessThanOrEqual(m.vw + 1); // and the page gets no sideways scroll
});
test('the same holds on a phone, where the columns stack', async ({ page }) => {
await openWith(page, 390);
const m = await page.evaluate(() => ({
wrapW: document.getElementById('mag-review-page-wrap').getBoundingClientRect().width,
docW: document.documentElement.scrollWidth, vw: window.innerWidth }));
expect(m.wrapW).toBeGreaterThan(200);
expect(m.docW).toBeLessThanOrEqual(m.vw + 1);
});
test('the huge token is still shown, wrapped, so it can be fixed or deleted', async ({ page }) => {
await openWith(page);
const row = page.locator('[data-block-row="b9"] .mag-block-text');
await expect(row).toContainText('Giro + 2.1.1.1');
const box = await row.boundingBox();
expect(box.width).toBeLessThan(1600);
expect(box.height).toBeGreaterThan(100); // wrapped onto many lines instead of one 20,000 px line
await expect(page.locator('[data-block-row="b9"]').getByRole('button', { name: /Delete region/ })).toBeVisible();
});
});
test.describe('A turned page shows at once, and its net turn is visible', () => {
async function open(page, { rotation = 0 } = {}) {
await mockBackend(page);
const state = { v: 100, rotation, rotates: [] };
await page.route(`**/api/magazine/${DOC_ID}/layout*`, route => route.fulfill({
json: { ...LAYOUT, pages: LAYOUT.pages.map(p => ({ ...p, image_v: state.v, ...(state.rotation ? { rotation: state.rotation } : {}) })) } }));
await page.route(`**/api/magazine/${DOC_ID}/pages/3/rotate`, route => {
state.rotates.push(route.request().postDataJSON());
state.v += 50; state.rotation = (state.rotation + ((state.rotates.at(-1).degrees % 360) + 360)) % 360; // what the server would now report
return route.fulfill({ json: { job_id: 'rot1', degrees: state.rotates.at(-1).degrees } });
});
await page.route('**/api/jobs/rot1', route => route.fulfill({ json: { done: true, error: null, detail: 'done' } }));
await page.goto('/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
await navTo(page, 's-magazine');
await page.locator('#mag-library-list .mag-card').first().click();
await page.locator('#mag-contents-view-toggle').click();
await page.locator('.mag-card-wrap').filter({ hasText: 'Page 3' }).getByRole('button', { name: /^Review/ }).click();
await page.waitForSelector('#mag-undo-btn');
return state;
}
test('every picture of a page carries its version, so a changed picture is a new URL', async ({ page }) => {
await open(page);
await expect(page.locator('#mag-review-page-wrap img')).toHaveAttribute('src', new RegExp(`/pages/3\\?v=100$`));
});
test('after a turn the page picture is requested again under its new version', async ({ page }) => {
const state = await open(page);
await page.getByRole('button', { name: /Turn right/ }).click();
await page.getByRole('alertdialog').getByRole('button', { name: 'Turn and read again' }).click();
await expect.poll(() => state.rotates.length).toBe(1);
await expect(page.locator('#mag-review-page-wrap img')).toHaveAttribute('src', /\/pages\/3\?v=150$/);
});
test('the page grid thumbnails use the version too', async ({ page }) => {
await open(page);
await page.getByRole('button', { name: /^Contents$/ }).click();
await expect(page.locator('.mag-card-wrap[data-page="3"] img.mag-card-thumb')).toHaveAttribute('src', /\/pages\/3\?v=100$/);
});
test('a page that has not been turned shows no turn state', async ({ page }) => {
await open(page);
await expect(page.locator('#mag-turn-state')).toHaveCount(0);
await expect(page.getByRole('button', { name: /Undo turns/ })).toHaveCount(0);
});
test('a turned page says by how much, and Undo turns sends what returns it to the PDF orientation', async ({ page }) => {
const state = await open(page, { rotation: 270 });
await expect(page.locator('#mag-turn-state')).toHaveText('Turned 270° clockwise from the scan');
await page.getByRole('button', { name: /Undo turns/ }).click();
await page.getByRole('alertdialog').getByRole('button', { name: 'Turn and read again' }).click();
await expect.poll(() => state.rotates).toEqual([{ degrees: 90 }]); // 270 + 90 = a full circle
await expect(page.locator('#mag-turn-state')).toHaveCount(0); // and the state line is gone once it is back to 0
});
test('the server refusing a second turn while one runs is explained and the buttons come back', async ({ page }) => {
await open(page);
await page.route(`**/api/magazine/${DOC_ID}/pages/3/rotate`, route => route.fulfill({
status: 409, json: { detail: 'Page 3 is still being turned or read; wait for it to finish before turning it again' } }));
await page.getByRole('button', { name: /Turn right/ }).click();
await page.getByRole('alertdialog').getByRole('button', { name: 'Turn and read again' }).click();
await expect(page.locator('.toast, #toast, [role="status"]').filter({ hasText: 'still being turned or read' }).first()).toBeVisible();
await expect(page.getByRole('button', { name: /Turn right/ })).toBeEnabled();
});
});

View File

@ -298,6 +298,8 @@ class RenderTests(unittest.TestCase):
self.assertTrue(any(b["text"].startswith("A sentence") for b in plain["blocks"]))
turned = mag.extract_pages(d / "source.pdf", d / "turned", render_dpi=50, page_rotations={1: 90})
self.assertEqual((turned["pages"][0]["width"], turned["pages"][0]["height"]), (200.0, 300.0))
self.assertEqual(turned["pages"][0].get("rotation"), 90) # the stored turn is visible in the layout again
self.assertNotIn("rotation", plain["pages"][0])
with Image.open(d / "turned" / "pages" / "0001.png") as im, Image.open(d / "plain" / "pages" / "0001.png") as ref:
self.assertEqual(im.size, (ref.size[1], ref.size[0]))
# The PDF's own text boxes belong to the unturned page; on a turned page only OCR of the render counts.
@ -311,6 +313,75 @@ class RenderTests(unittest.TestCase):
self.assertEqual([x["text"] for x in a["blocks"]], [x["text"] for x in b["blocks"]])
class ImageVersionAndJobGuardTests(unittest.TestCase):
def setUp(self):
from fastapi import FastAPI
from fastapi.testclient import TestClient
import routes.magazine as routes
self.routes = routes
self.lib = Path(tempfile.mkdtemp(prefix="ttvc-lib-"))
p = mock.patch.object(routes, "_LIB", self.lib)
p.start(); self.addCleanup(p.stop)
app = FastAPI(); app.include_router(routes.router)
self.client = TestClient(app)
self.doc = self.client.post("/api/magazine/docs", json={"title": "T"}).json()["id"]
self.d = self.lib / self.doc
(self.d / "pages").mkdir()
self.png = self.d / "pages" / "0002.png"
Image.new("RGB", (50, 80), "white").save(self.png)
(self.d / "layout.json").write_text(json.dumps({"pages": [{"page": 2, "width": 50.0, "height": 80.0}, {"page": 3, "width": 50.0, "height": 80.0}],
"blocks": [], "images": []}))
def test_each_page_is_stamped_with_its_picture_version_and_it_changes_with_the_picture(self):
first = self.client.get(f"/api/magazine/{self.doc}/layout").json()["pages"]
self.assertEqual(first[0]["image_v"], int(self.png.stat().st_mtime))
self.assertEqual(first[1]["image_v"], 0) # page 3 has no picture: 0, not an error
os.utime(self.png, (self.png.stat().st_atime, self.png.stat().st_mtime + 100))
again = self.client.get(f"/api/magazine/{self.doc}/layout").json()["pages"]
self.assertEqual(again[0]["image_v"], first[0]["image_v"] + 100)
def test_the_stamp_is_also_there_when_one_page_is_asked_for(self):
pages = self.client.get(f"/api/magazine/{self.doc}/layout?page=2").json()["pages"]
self.assertEqual([p["page"] for p in pages], [2])
self.assertGreater(pages[0]["image_v"], 0)
def test_the_stamp_is_not_written_into_the_saved_layout(self):
self.client.get(f"/api/magazine/{self.doc}/layout")
self.assertNotIn("image_v", json.loads((self.d / "layout.json").read_text())["pages"][0])
def job(self, **kw):
base = {"done": False, "fingerprint": f"{self.doc}:2", "updated": __import__("time").time()}
base.update(kw)
return base
def running(self, jobs, page=2):
with mock.patch.object(self.routes, "background_job_list", side_effect=lambda kind, limit=50: jobs if kind == "magazine-rotate-page" else []):
return self.routes._page_job_running(self.doc, page)
def test_a_job_in_flight_for_the_same_page_counts(self):
self.assertTrue(self.running([self.job()]))
def test_finished_stale_and_other_page_jobs_do_not(self):
self.assertFalse(self.running([self.job(done=True)]))
self.assertFalse(self.running([self.job(updated=__import__("time").time() - 3600)])) # died with the server
self.assertFalse(self.running([self.job(fingerprint=f"{self.doc}:3")]))
self.assertFalse(self.running([self.job(fingerprint="other:2")]))
self.assertFalse(self.running([]))
def test_a_second_turn_or_re_read_while_one_runs_is_refused_and_changes_nothing(self):
before = self.png.read_bytes()
settings = {"vision_ocr_url": "http://ocr", "vision_ocr_model": "m"}
with mock.patch.object(self.routes, "_load_settings", return_value=settings), \
mock.patch.object(self.routes, "_page_job_running", return_value=True), \
mock.patch.object(self.routes.asyncio, "create_task") as task:
turn = self.client.post(f"/api/magazine/{self.doc}/pages/2/rotate", json={"degrees": 90})
reread = self.client.post(f"/api/magazine/{self.doc}/pages/2/reocr", json={})
self.assertEqual((turn.status_code, reread.status_code), (409, 409))
self.assertIn("wait for it to finish", turn.json()["detail"])
task.assert_not_called()
self.assertEqual(self.png.read_bytes(), before)
class EndpointTests(unittest.TestCase):
def setUp(self):
from fastapi import FastAPI

View File

@ -946,6 +946,43 @@ class CleanUpBoxesTests(unittest.TestCase):
self.assertEqual(types["long"][0], "body") # real text caught in a loose frame stays
self.assertEqual(types["outside"][0], "body")
def test_a_region_that_is_one_endless_repeated_token_is_noise_but_real_text_with_repeats_is_not(self):
"""Seen on a real advert page: 'Giro + 2.1.1.1.1 ...' thousands of characters long, no spaces."""
from core.magazine import clean_up_regions
junk = "Giro + " + "2." + "1." * 1700
layout = self.base([
self.blk("loop", 10, 10, 300, 40, junk),
self.blk("digits", 10, 60, 300, 40, "0" * 120),
self.blk("dots", 10, 110, 300, 12, "Kapitel 1 " + "." * 200 + " 17", "caption"),
self.blk("toc", 10, 150, 300, 12, "2.1.1 Einleitung 2.1.2 Methode 2.1.3 Ergebnisse 2.1.4 Diskussion 2.2 Fazit 3.1.1.1 Anhang"),
self.blk("url", 10, 170, 300, 12, "https://example.org/artikel/2026/09/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0?ref=newsletter&utm_source=heise"),
self.blk("prose", 10, 190, 300, 40, "Ein ganz normaler Absatz mit genug Woertern, der nichts Verdaechtiges enthaelt. " * 3),
self.blk("short", 10, 240, 300, 12, "1." * 10),
])
types = self.types(clean_up_regions(layout)[0])
self.assertEqual(types["loop"], ("furniture", "ocr_noise"))
self.assertEqual(types["digits"], ("furniture", "ocr_noise"))
for kept in ("dots", "toc", "url", "prose", "short"):
self.assertNotEqual(types[kept][1], "ocr_noise", kept)
def test_the_repetition_share_counts_only_letter_or_digit_units_repeated_thirty_times(self):
from core.magazine import glued_repetition_share
self.assertGreater(glued_repetition_share("1." * 100), 0.9)
self.assertGreater(glued_repetition_share("ab" * 40), 0.9)
self.assertEqual(glued_repetition_share("." * 500), 0.0) # leader dots and rules are not text, so not counted
self.assertEqual(glued_repetition_share("- " * 100), 0.0)
self.assertEqual(glued_repetition_share("1." * 29), 0.0) # one short of the threshold
self.assertEqual(glued_repetition_share(""), 0.0)
self.assertAlmostEqual(glued_repetition_share("Hallo " + "xy" * 50 + " Welt"), 100 / 111, places=2)
def test_clean_up_is_idempotent_for_the_new_rule(self):
from core.magazine import clean_up_regions
layout = self.base([self.blk("loop", 10, 10, 300, 40, "2." + "1." * 500)])
once = clean_up_regions(layout)[0]
twice, report = clean_up_regions(once)
self.assertEqual(self.types(twice)["loop"], ("furniture", "ocr_noise"))
self.assertEqual(report["ocr_noise"], []) # nothing left to re-type the second time
def test_one_or_two_characters_are_noise_but_a_phone_number_is_not(self):
from core.magazine import clean_up_regions
layout = self.base([