diff --git a/CHANGELOG.md b/CHANGELOG.md index 2328503..670f3b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,12 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi - `vision_ocr_page` routes its request through the cache layer; the default `live` mode behaves exactly as before. ### Fixed - **A boy rendered as an adult:** `_character_age_category` read `age`, but sheets store `age_estimate`. +- **Characters' data no longer bleeds into each other through shared labels or names.** A real-library audit (270 records) traced Arthag's (book 02) copy of Kolon's species and relationships and 16 whole-sheet copies (Zwerg/Kolon = Arthag, Mann = Fremder, Alchimist = Promos ...) to identity matching on any shared token ("Zwerg", "der Zwerg", "Mann") with "longer text wins" merging, and to `csRosterCanonicalName` renaming a sheet to the first cast name found among its aliases instead of its own name. Now only a real name links two profiles (`static/js/character-identity.js`, `core/character_records.py`), the own name always wins (`csFindMergeKey`, `csRosterCanonicalName`, `clUpsert` writes to the exact-name record before any alias scan, no seed rename onto a name that has its own record). Tests: `tests/character-identity.test.mjs`. +- **Other cast members' names are no longer stored as a character's alias or real name** (Kolon carried "Sharraz Garthai"/"Sharraz"): `_charsheets_parse` drops them in cast-target mode, keeping relations such as "Tochter des Promos". +- **Model "thinking" no longer lands in profile fields** ("Warte: ...", "Nein, Kolon ist ...", "Der Text sagt ..."): every text field is cut before such markers on extraction. +- **The story-moment picture no longer shows another character's scene:** only a source quote that names the character is used, otherwise its own mannerisms (16 of 270 lower panels named only another cast member). +- **Age is read from German and English wording:** "Junge", "Mädchen", "boy", "girl", number words and "Sommer/Winter" ages, the estimate's own "im Text als Junge beschrieben" clause, and age-tagged quotes; Marrad ("fast acht Sommer") is now a child. "15 Jahre älter" is no longer read as a 15-year-old. +- **New read-only audit and dry-run cleanup for saved characters:** `scripts/audit_character_records.py`, `scripts/clean_character_records.py` (dry run by default; `--apply` writes a backup first). See architecture/character-extraction.md. Tests: `tests/test_character_record_hygiene.py`. - **German "hat" (has) became a hood:** "hat dünne Arme" turned into "soft wool hood" in the prompt; `_period_wording` now only maps English "hat" and leaves German text alone. - **Dwarf body shape and beard no longer leak onto human characters' images.** The dwarf body lock ("one adult dwarf", short legs, thick full beard) was switched on by the word "Zwerg" anywhere in the species field, including extraction notes about other characters. In the real library this hit Marrad (a thin human boy, "Mensch (oder Zwerg, je nach Kontext …)") and Kolon and Arthag in book 02 ("Mensch (… 'Der Zwerg' wird separat erwähnt … Warte: 'Der Zwerg lächelte grimmig' …)"), whose prompt Subject also carried that sentence about another character. Species now comes only from the field's own leading assertion: long or quoting bracket notes are dropped, alternatives ("Mensch oder Zwerg") never count as a dwarf, and a *visual species* override is honoured for the body lock too. Seven of 270 records change; tests in `tests/test_character_species_isolation.py`. diff --git a/Manual.md b/Manual.md index b70170b..77eef04 100644 --- a/Manual.md +++ b/Manual.md @@ -97,6 +97,18 @@ brackets such as "Mensch (oder Zwerg, je nach Kontext …)" or a long remark abo characters is ignored, so such a character is drawn as a human. If a character really is a dwarf but the field says "Mensch (…)", change the field to "Zwerg" and regenerate. +**Story moment:** the lower picture uses a quote from the character's *Sources* that +names this character (by name, real name or a real alias such as "Murlok von +Mengbilla"; "der Zwerg" alone does not count). A quote that only names someone else is +never used; without a fitting quote the character's own mannerisms or abilities are +shown instead. To change that fallback, edit the character's *Mannerisms*. + +**Age:** a child or teenager is drawn as one. The app reads the age estimate ("12 +Jahre", "zwölfjährig", "fast acht Sommer", "15 years old"), age quotes in *Sources*, +and words like "Kind", "Junge", "Mädchen", "boy", "girl". "Junger Mann" / "Junge Frau" +stay adults. If a hedged estimate ("Junges Mädchen / Junger Mann") is drawn wrongly, +write the age as a number in *Estimated age* and regenerate. + **Privacy and offline:** only the prompts and the profile picture go to your own ComfyUI; nothing leaves your network. Without a reachable ComfyUI nothing is generated. @@ -127,6 +139,31 @@ continues. Set `PROMPT_TRANSLATION=off` to disable translation entirely. **Privacy:** character text goes only to the LLM you configured; cached translations stay in the local database. +### Keeping characters apart (profiles and the character library) + +**Purpose:** each character's profile holds only that character's facts. + +**What the app does:** two profiles are treated as the same person only when they share +a real name. Labels many people share ("Zwerg", "der Zwerg", "Mann", "Fremder", +"Soldat", "Ork-General") never join profiles. A profile is saved to the record with +exactly its name. When the cast is known, another cast member's name is never kept as +an alias or first/last name (relations such as "Tochter des Promos" stay), and notes +in which the model talks to itself ("Warte: ...", "Der Text sagt ...") are removed. + +**Checking an existing library (administrators):** on the server, run +`python3 scripts/audit_character_records.py` for a read-only report of copied profiles, +names of other characters in aliases, leaked notes and foreign quotes. To see what a +cleanup would change, run `python3 scripts/clean_character_records.py --out `: +it writes `report.md` and changes nothing. Only with `--apply` (after you have read the +report) does it save a backup to the folder and change the listed records; copied +profiles are only reported, you decide which one to keep (delete or recast the other). + +**Errors:** a profile that was copied or mixed up before this fix keeps its wrong text +until you edit it, delete the copy, or recast that character. + +**Mobile, accessibility, offline:** no interface change. **Privacy:** the scripts run +locally against your own database; nothing is sent anywhere. + --- ## Magazines diff --git a/architecture/character-extraction.md b/architecture/character-extraction.md new file mode 100644 index 0000000..6e69487 --- /dev/null +++ b/architecture/character-extraction.md @@ -0,0 +1,105 @@ +# Character extraction, identity merging and record hygiene + +Status: 2026-10-03. Companion to `character-visual-identity.md` (how a saved sheet +becomes image prompts). This document covers how sheets are extracted from the book, +how they are merged into library records, and the rules that keep one character's +data out of another's. + +## Pipeline + +1. **Cast roster** (Identify Characters / audiobook casting) names the speakers. +2. **Extraction** (`static/js/character-sheets.js` `csGenerate`): the book is cut into + ~4000-character passages; each passage goes to `POST /api/character-sheets` + (`routes/conversation.py` `_charsheets_prepare` / `_charsheets_parse`) together with + the roster and a short summary of the sheets so far. One LLM call returns sheets for + **every** character the passage reveals (batch per passage, not per character). + - *Cast-target mode* (roster known, `csForReader`): only sheets whose name exactly + equals a roster name are kept and merged by exact name (`strictRoster`). + - *Discovery mode* (no roster, `csForRehearser`): sheets are merged by identity + (`csFindMergeKey`), then `csConsolidateAliasDuplicates` folds alias-linked entries. + - Field merge is **"longer text wins"** per field; sources (max 12) and inventory + are unioned. +3. **Canonical names** (`csCanonicalizeRosterNames` / `csRosterCanonicalName`): the + roster name is the display name; other names found by the LLM become aliases. +4. **Save** (`static/js/characters-library.js` `clUpsert`): a roster-authoritative sheet + is written by its book+name key; other sheets look for an existing record first. +5. **Image prompts** read the saved record (`character-visual-identity.md`); the + story-moment panel uses one quote from `sources` (`_character_scene_evidence`). + +## Root causes found on the real library (270 records, 4 books) + +Evidence from a read-only snapshot, the `config/regression_snapshots/*` exports of +2026-08-31 and the per-update backups in `config/backups/characters/`. + +| Symptom | Cause (proven) | +|---|---| +| Arthag (book 02) has Kolon's species, relationships and motivation word for word; both records date from the 2026-07-17 batch and were already identical on 2026-08-31 | The extraction code of that time filtered target-mode sheets by **substring** roster match and merged by **any shared identity token** (title still counted). Merging is "longer text wins", so Kolon's long fields overwrote Arthag's. Today's target mode uses exact names, but discovery mode (`csFindMergeKey`, `csConsolidateAliasDuplicates`) and the library upsert (`clSameIdentity`) still linked records over a single shared token such as "Zwerg". | +| 16 records in book 02/03 (Zwerg, Kolon, Inquisitor, Soldat, Alter Soldat, Alte, Ritter, Navigator, Fremder, Alchimist, Kriegerin, Mädchen, ...) are complete copies, sources included, of another record (Arthag, Marcian, Veteran, Alrik, Mann, Promos, Nyrilla, Marina, ...) | Born in two bulk saves on 2026-08-16 (06:41 and 07:13, ~20 ms apart, two books), the day after `csRosterCanonicalName` shipped. The alias lists of each pair are the same list minus each record's own name, i.e. **one sheet was written under two names**. `csRosterCanonicalName` returned the *first roster entry* found among all of a sheet's tokens, not the sheet's own name, so Arthag (aliases "der Zwerg, Zwerg, ..., Kolon") became "Zwerg"/"Kolon" and the save created those records from Arthag's sheet. (The exact caller of that batch is not recoverable; the rename path is the only code that turns one sheet into another name.) | +| Kolon (b02) aliases "Sharraz Garthai, ..., Sharraz, Zwerg, Kriegshäuptling", first/last name Sharraz Garthai | The LLM, given a passage about both, wrote the other person's name into Kolon's sheet; its own relationships field says "dient ... Sharraz Garthai". Aliases were accepted verbatim; only first/last name had a roster check (client side). 87 records carry another record's name as alias or real name. | +| Mann/Fremder (b03), Fremde/Zerwas (KONTROLLE) share species text | Same as row 2 (copied sheet), see `duplicated_sheets` in the audit. | +| Story-moment quote about another cast member | `_character_scene_evidence` chose by line_hint and verb score only. Sources hold whatever the passage quoted, including other characters' lines (also inherited through the merges above). 16 picked quotes named only another cast member; 85 did not name the character at all. | +| "Warte: ...", "Nein, Kolon ist ...", "Der Text sagt ..." in species/backstory/notes (12 records) | Reasoning-model deliberation written into field values; nothing validated them. | +| Marrad (a boy, "fast acht Sommer") rendered as an adult; Fremder b01 ("mindestens 15 Jahre älter") as a teen | `_character_age_category` knew no boy/girl nouns ("Junge", "Mädchen", "boy", "girl"), no "Sommer/Winter" ages, and read "15 Jahre älter" as an age. | + +What could **not** be proven without live LLM runs: how often today's prompt still +produces cross-character aliases or merged facts for a multi-character passage. To +test: run *Recast selected* for Kolon and Arthag on book 02 with the local model and +check that Kolon's aliases contain no roster name and Arthag's species is his own. + +## Rules (implemented) + +**Identity evidence** (`core/character_records.py`, mirrored in +`static/js/character-identity.js`, word lists kept equal by a unit test): a shared +token proves "same person" only if it is a real name. Generic = only common nouns, +roles, ranks, kinship, species, age/look adjectives or name particles ("Zwerg", "Mann", +"Fremder", "Alter Soldat", "Ork-General", "Bürgerwehroffizier", "von"), or starting with +an article ("der Zwerg", "der Tunneltreiber"). Used by `csFindMergeKey`, +`csConsolidateAliasDuplicates`, `csRosterCanonicalName` and `clSameIdentity`. + +**Own name first:** `csFindMergeKey` targets an entry with exactly the sheet's name; +`csRosterCanonicalName` keeps the sheet's own name when it is a cast name; `clUpsert` +writes to the record with exactly this book+name before any alias scan; `csForReader` +never renames a saved sheet onto a cast name that has its own saved record. + +**Per-character isolation at parse time** (`_charsheets_parse(raw, roster)`, roster +only in cast-target mode): aliases and first/last/full names that are another cast +member's name, or a name-like alias containing one ("Sharraz Garthai", "Schlange +Gamba"), are dropped; descriptions naming a relative ("Tochter des Promos") stay. +Every text field is cut before reasoning markers (`strip_thinking_leakage`). +Source quotes are kept: some name another character but are about this one. + +**Story moment:** only a source quote that names the character (name, real name, +genitive, non-generic alias) is eligible; otherwise the character's own mannerisms / +capabilities text is used, never somebody else's quote. + +**Age:** explicit age (digits or number words with Jahre/jährig/Sommer/Winter/years/ +summers/winters; not "15 Jahre älter") from the age estimate, age-tagged source +quotes, labels, physical; then the estimate's evidence clause ("im Text als Junge +beschrieben" / "described in the text as a boy"); then child/teen words and boy/girl +nouns in label fields. A hedged "Junges Mädchen / Junger Mann" with a minor signal is a +minor. German "Junge Frau"/"Junger Mann" are adults. Works on German and translated +English text. + +## Audit and cleanup (SOP) + +- `python3 scripts/audit_character_records.py [DB] [--book X] [--details|--json]`: + read-only (`mode=ro`, `query_only`). Reports duplicated sheets, alias collisions, + generic-only alias links, thinking leakage, foreign source quotes and scene quotes + about another cast member. Run on a copy when the app is busy. +- `python3 scripts/clean_character_records.py [DB] --out DIR`: **dry run** (default), + writes `report.json` / `report.md`, database untouched. Plans: cut reasoning leakage; + remove an alias / real name of another cast member only when the record's own + relationships name that person as someone else; everything else is *review only*. + `--drop-foreign-sources` additionally removes source quotes naming only another cast + member (lossy, off by default). Duplicated sheets are reported, never changed. +- `--apply --out DIR` (owner approval only) writes `backup.json` with every affected row + first, then updates those rows in one transaction. +- **Rollback:** for each entry in `backup.json`, write its `sheet` and `updated` back: + `UPDATE characters SET sheet=?, updated=? WHERE id=?` (or restore the DB backup taken + before applying). Stop the container or make sure no recast runs meanwhile. + +Snapshot of 2026-10-03 (before): 33 duplicated-sheet pairs, 161 alias collisions in 87 +records, 61 generic-only alias links, 12 records with leakage, 139 foreign source quotes +in 89 records, 16 scene quotes about another cast member (0 with the fixed picker). +Dry run: 12 records would change (12 leakage cuts, 6 evidenced identity removals), 70 +identity collisions listed for review. diff --git a/architecture/character-visual-identity.md b/architecture/character-visual-identity.md index e315c6c..bd075b9 100644 --- a/architecture/character-visual-identity.md +++ b/architecture/character-visual-identity.md @@ -57,6 +57,9 @@ active changes only that form. ## Image prompt hygiene and the turnaround (1.34.x) +How sheets are extracted and merged, which identity tokens count, the story-moment quote rule and the +age rules: see `character-extraction.md`. + * **Setting.** Prompts describe the books as *The Dark Eye* high-fantasy medieval dress (about 900-1300 AD: wool and linen tunics, belted robes, leather, mail where the sheet names it). The earlier "early Renaissance 1420-1500" wording pulled uniforms, coats, hats and boots that read as musketeer/cowboy/wartime. * **Wording.** Extracted costume text is rewritten at prompt time (`_period_wording`: Uniform, Weste, Hut, Sporen, Jacke, Schal … become surcoat, jerkin, hood, boots, tunic …). Stored sheets are not changed. * **Injuries and dirt.** `_strip_injury_clauses` removes clauses about blood, wounds, bandages, burns, limping, torn clothes, soot and dirt from appearance, costume and the story moment; the rest of the sentence stays. The portrait baseline asks for clean skin and garments. diff --git a/core/character_records.py b/core/character_records.py new file mode 100644 index 0000000..f5ca14b --- /dev/null +++ b/core/character_records.py @@ -0,0 +1,339 @@ +"""Identity and hygiene rules for extracted character records. + +Pure functions (no I/O) shared by the character-sheet extraction parser +(routes/conversation.py), the scene-quote picker for image prompts, and the +read-only audit / dry-run cleanup scripts in scripts/. + +Why these exist (all found on the real library, 2026-10-03, see +architecture/character-extraction.md): + +* Two records were merged, or one record's sheet was copied into another, + because they shared ONE identity token such as "Zwerg", "der Zwerg", "Mann" + or "Fremder". Those are common nouns, roles and titles, not names: the book + calls many people "der Zwerg". Only a non-generic token is evidence that two + sheets describe the same person. +* The LLM put other cast members' names into a character's aliases (Kolon in + book 02 carried "Sharraz Garthai", "Sharraz" and "Kriegshäuptling", all + separate cast records). +* Reasoning text leaked into descriptive fields ("Warte: ...", "Nein, Kolon + ist ...", "Der Text sagt ..."). +* The story-moment quote was picked by line_hint/verb score only, so a quote + about another cast member could become this character's lower panel. + +The generic-word list is mirrored in static/js/character-identity.js; a unit +test keeps the two in sync. +""" +from __future__ import annotations + +import re +from typing import Iterable + +# Articles / determiners that make an alias a description rather than a name +# ("der Zwerg", "der Tunneltreiber", "the Executioner"). +ARTICLES = frozenset(( + "der die das den dem des ein eine einer eines einem einen " + "the a an" +).split()) + +# Common nouns, roles, ranks, kinship words, species and age/look adjectives. +# A token made only of these words (after articles) is generic: it can label +# many different people and is never evidence of shared identity. +GENERIC_IDENTITY_WORDS = frozenset(( + # people, age, kinship (de) + "mann frau junge mädchen kind knabe bub greis greisin alte alter altes alten " + "jung junger junges jungen menge männer frauen leute person gestalt wesen gast " + "vater mutter sohn tochter bruder schwester onkel tante witwe witwer " + # species / kinds (de) + "mensch menschen zwerg zwergin zwerge elf elfe elfen auelfe ork orks orkin " + "vampir vampirin dämon monster kreatur " + # roles, ranks, professions (de) + "krieger kriegerin wache wachen wächter reiter schreiber adept adepten " + "zauberer zauberin magier magierin hexe inquisitor schmied oberst obrist " + "general offizier offizierin hauptmann kommandant kommandantin kapitän " + "admiral großadmiral marschall fremde fremder fremden stimme stimmen sklave " + "sklavin sklaven nachbar nachbarin bote botin soldat soldatin soldaten " + "ritter ritterin prinz prinzessin könig königin kaiser kaiserin graf gräfin " + "baron baronin herzog herzogin junker herr herrin dame diener dienerin magd " + "knecht händler kaufmann wirt wirtin bauer bäuerin fischer jäger priester " + "priesterin geweihter mönch nonne schamane druide amazone henker veteran " + "navigator begleiter begleiterin gefährte gefährtin anführer anführerin " + "häuptling kriegshäuptling führer militärführer meister lehrer schüler " + "bürger bürgerin bürgersohn weber kutscher bäckerstochter erzähler " + "torwächter feldherr heerführer befehlshaber hauptfrau " + # descriptive adjectives that often head a role label (de) + "erster erste zweiter zweite dritter dritte von van zu vom zum zur de del " + "vermummt vermummte vermummter vermummten groß große großer kleine kleiner " + "dunkle dunkler blonde blonder rote roter tote toter schwarzhaarige " + "unbekannt unbekannte unbekannter irre unsicher " + # English + "man woman boy girl child kid stranger old young person figure creature " + "father mother son daughter brother sister " + "human dwarf elf orc vampire demon monster " + "guard soldier warrior knight officer general colonel captain commander " + "admiral marshal prince princess king queen lord lady sir master servant " + "merchant smith blacksmith priest mage wizard witch shaman druid " + "executioner veteran navigator companion leader chief voice crowd " + "messenger scribe inquisitor narrator unknown" +).split()) + +# A compound ending in one of these (length >= 5) is still a role label: +# "Bürgerwehroffizier", "Kriegshäuptling", "Ork-General" (split on '-'). +_GENERIC_SUFFIXES = tuple(sorted((w for w in GENERIC_IDENTITY_WORDS if len(w) >= 5), key=len, reverse=True)) + +_ALIAS_SPLIT_RE = re.compile( + r"[,;/|]|\baka\b|\baka\.\b|\balias(?:es)?\b|\bgenannt\b|\bnamens\b|\bcalled\b|\bknown as\b", + re.IGNORECASE, +) +_WORD_RE = re.compile(r"[\w'’]+", re.UNICODE) + + +def normalize_identity_token(token: str) -> str: + """Lower-case, drop parentheticals and surrounding punctuation.""" + text = re.sub(r"\([^)]*\)", " ", str(token or "")) + text = re.sub(r"\s+", " ", text).strip(" \t\n.,;:!?\"'»«„“”‚‘’").lower() + return text + + +def split_identity_tokens(value) -> list[str]: + """Split an aliases/name value into individual name strings.""" + if value is None: + return [] + if isinstance(value, (list, tuple)): + value = ", ".join(str(v) for v in value if v) + parts = [p.strip() for p in _ALIAS_SPLIT_RE.split(str(value))] + return [p for p in parts if p and len(p) <= 80 and not re.match(r"^(needs?:|complete$)", p, re.I)] + + +def is_generic_identity_token(token: str) -> bool: + """True when ``token`` is a common noun / role / title, not a name. + + "Zwerg", "der Zwerg", "Alter Soldat", "Ork-General", "Bürgerwehroffizier", + "der Tunneltreiber" (article + description) are generic; "Arthag", + "Sharraz Garthai", "Admiral Sanin" (contains a name) are not. + """ + norm = normalize_identity_token(token) + if not norm: + return True + words = [w for w in re.split(r"[\s\-]+", norm) if w] + if not words: + return True + if words[0] in ARTICLES: + # "der Henker", "der Mann mit dem roten Umhang": a description. + return True + for word in words: + bare = word.strip("'’.") + if len(bare) <= 1 or bare in ARTICLES or bare in GENERIC_IDENTITY_WORDS: + continue + if any(bare.endswith(suffix) and len(bare) > len(suffix) for suffix in _GENERIC_SUFFIXES): + continue + return False + return True + + +def identity_tokens(name: str, sheet: dict | None, include_generic: bool = False) -> set[str]: + """Normalized identity tokens of one character: name, real names, aliases.""" + sheet = sheet or {} + out: set[str] = set() + values = [name, sheet.get("first_name"), sheet.get("last_name"), sheet.get("full_name")] + first, last = str(sheet.get("first_name") or "").strip(), str(sheet.get("last_name") or "").strip() + if first and last: + values.append(f"{first} {last}") + for value in values: + for part in split_identity_tokens(value): + norm = normalize_identity_token(part) + if norm and (include_generic or not is_generic_identity_token(part)): + out.add(norm) + for part in split_identity_tokens(sheet.get("aliases")): + norm = normalize_identity_token(part) + if norm and (include_generic or not is_generic_identity_token(part)): + out.add(norm) + own = normalize_identity_token(name) + if own: + out.add(own) # the record's own name always identifies it, even "Fremder" + return out + + +def shared_identity_evidence(a_tokens: Iterable[str], b_tokens: Iterable[str]) -> set[str]: + """Shared tokens that actually prove identity (generic ones never do).""" + a = {normalize_identity_token(t) for t in a_tokens} + b = {normalize_identity_token(t) for t in b_tokens} + return {t for t in a & b if t and not is_generic_identity_token(t)} + + +def text_mentions_token(text: str, token: str) -> bool: + """Whole-word, case-insensitive mention; a German genitive -s counts.""" + token = normalize_identity_token(token) + if len(token) < 2: + return False + pattern = r"(? bool: + """True when the quote names this character by name or a non-generic alias.""" + return any(text_mentions_token(quote, token) for token in identity_tokens(name, sheet)) + + +def quote_mentions_other_cast(quote: str, own_name: str, own_sheet: dict | None, + cast: Iterable[tuple[str, dict]]) -> list[str]: + """Names of OTHER cast members (non-generic names only) the quote mentions.""" + own = identity_tokens(own_name, own_sheet) + hits = [] + for other_name, _other_sheet in cast: + if normalize_identity_token(other_name) in own or is_generic_identity_token(other_name): + continue + if text_mentions_token(quote, other_name): + hits.append(other_name) + return hits + + +# ── Reasoning ("thinking") leakage ───────────────────────────────────────── + +# Phrases that only appear when the model narrates its own deliberation inside +# a field value. Calibrated on the real library: every hit there was leakage, +# and none of these occur in legitimate descriptive text of the 270 records. +_THINKING_MARKER_RE = re.compile( + r"|" + r"|\b(?:aber\s+)?(?:warte|wait)\b\s*[:,.!]" + r"|\bhmm+\b" + r"|\bmoment mal\b" + r"|\blass mich\b|\blet me\b" + r"|\bes ist mehrdeutig\b|\bit(?:'s| is) ambiguous\b" + r"|\b(?:der\s+)?text sagt\b|\bthe text says\b" + r"|\bnein,\s+[A-ZÄÖÜ][\w-]*\s+ist\b|\bno,\s+[A-Z][\w-]*\s+is\b" + r"|\bich werde (?:mich|fokussieren|annehmen)\b|\bfokussiere ich\b" + r"|\b(?:korrektur|correction)\s*:", + re.IGNORECASE, +) +# Field values the model writes as prose about the character. Identity/name +# fields are included: leakage there is just as wrong. +TEXT_FIELDS = ( + "aliases", "first_name", "last_name", "full_name", "title", "profession", + "age_estimate", "race_species", "languages", "nationality_background", + "social_class", "archetype", "physical", "clothing", "alignment", "arc_note", + "attribute_high", "attribute_low", "skills", "capabilities", "backstory", + "relationships", "motivation", "fears", "mannerisms", "communication_style", + "voice_pattern", "secret", "conflict_style", "win_condition", "reputation", + "religious_beliefs", "notes", +) + + +def has_thinking_leakage(value) -> bool: + return isinstance(value, str) and bool(_THINKING_MARKER_RE.search(value)) + + +def strip_thinking_leakage(value: str) -> str: + """Cut a field value before the first deliberation marker. + + The cut backs up to the start of the clause that holds the marker (the + last sentence end, ';' or unclosed '(' before it), so "Mensch (oder + Ork-Verbündeter, ... Text sagt ...)" keeps only "Mensch". Text without a + marker is returned unchanged. + """ + if not isinstance(value, str): + return value + text = re.sub(r".*?", "", value, flags=re.DOTALL | re.IGNORECASE) + match = _THINKING_MARKER_RE.search(text) + if not match: + return text.strip() if text != value else value + head = text[:match.start()] + # Back up to the start of the clause holding the marker. + depth, cut = 0, 0 + for i, ch in enumerate(head): + if ch == "(": + depth += 1 + if depth == 1: + open_at = i + elif ch == ")": + depth = max(0, depth - 1) + if depth > 0: + cut = open_at # the marker sits inside an unclosed parenthetical + head = head[:cut] + else: + ends = [m.end() for m in re.finditer(r"[.;!?](?:\s|$)", head)] + head = head[:ends[-1]] if ends else "" + return head.strip().rstrip(",;:-–— ").strip() + + +def sanitize_sheet_text_fields(sheet: dict) -> dict[str, tuple[str, str]]: + """Strip reasoning leakage from ``sheet`` in place; return {field: (old, new)}.""" + changes: dict[str, tuple[str, str]] = {} + for field in TEXT_FIELDS: + value = sheet.get(field) + if not has_thinking_leakage(value): + continue + clean = strip_thinking_leakage(value) + if clean != value: + sheet[field] = clean + changes[field] = (value, clean) + return changes + + +def drop_foreign_identity(name: str, sheet: dict, cast_names: Iterable[str]) -> list[str]: + """Remove aliases / real-name fields that are ANOTHER cast member's name. + + An alias that equals a different, separately cast character's name is the + model mixing two people up in one passage (Kolon -> "Sharraz Garthai"), + never a proven alternate name of this one. Mutates ``sheet``; returns the + removed values (as "field=value"). + """ + own = normalize_identity_token(name) + # Generic cast labels ("Zwerg", "Mann", "Fremder") are no identity claim: + # Kolon may well be "der Zwerg". Only real names of others are removed. + others = {normalize_identity_token(n) for n in cast_names if n and not is_generic_identity_token(n)} + others.discard(own) + others.discard("") + if not others: + return [] + # Real names of other cast members embedded in a longer alias + # ("Sharraz Garthai" when "Sharraz" is cast, "Schlange Gamba"). Exact word + # match only: "Darrags Sohn" (a kinship description) stays. + named_others = [o for o in others if o not in own.split()] + + def foreign(value: str) -> bool: + norm = normalize_identity_token(value) + if not norm or norm == own: + return False + if norm in others: + return True + for other in named_others: + match = re.search(r"(? list[dict]: + """Source entries whose quote names this character (name or real alias).""" + sources = sheet.get("sources") if isinstance(sheet.get("sources"), list) else [] + return [s for s in sources if isinstance(s, dict) + and quote_mentions_identity(str(s.get("quote") or ""), name, sheet)] diff --git a/routes/conversation.py b/routes/conversation.py index 93571bb..27d938e 100644 --- a/routes/conversation.py +++ b/routes/conversation.py @@ -37,6 +37,7 @@ from core.voice_index import rebuild_voice_index from core.database import background_job_create, background_job_get, background_job_store_artifact, background_job_update from core.retry import request_with_retries from core import prompt_translation as _pt +from core import character_records as _cr from core.character_image_check import checker_unavailable, failure_count, summarise from core.attribution_spans import ( SpanRequestError, build_messages as _span_build_messages, max_tokens_for as _span_max_tokens, @@ -529,7 +530,7 @@ def _english_prompt_inputs(name: str, sheet: dict, book_profile: dict | None = N if isinstance(value, str) or (isinstance(value, list) and value and all(isinstance(v, str) for v in value)): fields[f"sheet.{key}"] = value if not _visual_prompt_value(sheet.get("visual_scene")).strip(): - evidence = _character_scene_evidence(sheet) + evidence = _character_scene_evidence(sheet, name) if evidence and not evidence.startswith(_DEFAULT_SCENE_EVIDENCE_PREFIX): fields["scene_evidence"] = evidence for key in _PROMPT_TRANSLATE_BOOK_FIELDS: @@ -1262,11 +1263,22 @@ def _charsheets_prepare(data: dict) -> dict: "payload": payload, "llm_url": llm_url, "model": model, "api_key": _settings.get("llm_api_key") or "sk-dummy-key", "timeout_seconds": timeout_seconds, + # In cast-target mode the roster is authoritative: another roster + # name can never be an alias of this character (see _charsheets_parse). + "roster": [str(n) for n in known] if target_mode else [], } -def _charsheets_parse(raw: str) -> dict: - """Turn the LLM's raw answer into normalised {sheets, characters}.""" +def _charsheets_parse(raw: str, roster: list | None = None) -> dict: + """Turn the LLM's raw answer into normalised {sheets, characters}. + + Every descriptive field is cleaned of reasoning leakage ("Warte: ...", + "Der Text sagt ...") and, when ``roster`` (the authoritative cast list of a + cast-target pass) is given, aliases / real-name fields that are ANOTHER + cast member's name are dropped: in book 02 Kolon's sheet carried + "Sharraz", "Sharraz Garthai" and "Kriegshäuptling", all separate cast + records, because one passage mentioned them together. + """ content = re.sub(r".*?", "", raw, flags=re.DOTALL).strip() or raw sheets = [] for cand in (content, _extract_json_block(content)): @@ -1287,6 +1299,13 @@ def _charsheets_parse(raw: str) -> dict: name = str(s.get("name") or "").strip() if not name: continue + leaked = _cr.sanitize_sheet_text_fields(s) + if leaked: + print(f"[character-sheets] stripped reasoning leakage from {name!r}: {sorted(leaked)}") + if roster: + dropped = _cr.drop_foreign_identity(name, s, roster) + if dropped: + print(f"[character-sheets] dropped other cast members' names from {name!r}: {dropped}") alias_raw = str(s.get("aliases") or "").strip() alias_parts = [p.strip() for p in alias_splitter.split(alias_raw) if p.strip()] alias_parts = [p for p in alias_parts if len(p) <= 80 and not re.match(r"^(needs?:|complete$)", p, re.I)] @@ -1399,7 +1418,7 @@ async def character_sheets(request: Request): raise except Exception as e: raise HTTPException(502, f"LLM character-sheet generation failed: {e}") - return _charsheets_parse(raw) + return _charsheets_parse(raw, prep["roster"]) @router.post("/api/book-profile/generate") @@ -1557,7 +1576,7 @@ async def character_sheets_stream(request: Request): raw_parts.append(delta["content"]) if t: yield f'data: {json.dumps({"t": t})}\n\n' - result = _charsheets_parse("".join(raw_parts)) + result = _charsheets_parse("".join(raw_parts), prep["roster"]) yield f'data: {json.dumps({"done": True, "result": result})}\n\n' except GeneratorExit: raise @@ -2162,8 +2181,22 @@ def _character_ranged_weapon_lock(sheet: dict) -> str: return f"Any ranged weapon shown is {shape}, drawn and aimed by hand exactly like that object. " +_AGE_NUMBER_WORDS = { + "ein": 1, "eins": 1, "zwei": 2, "drei": 3, "vier": 4, "fünf": 5, "fuenf": 5, "sechs": 6, + "sieben": 7, "acht": 8, "neun": 9, "zehn": 10, "elf": 11, "zwölf": 12, "zwoelf": 12, + "dreizehn": 13, "vierzehn": 14, "fünfzehn": 15, "sechzehn": 16, "siebzehn": 17, + "achtzehn": 18, "neunzehn": 19, "zwanzig": 20, + "one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, + "nine": 9, "ten": 10, "eleven": 11, "twelve": 12, "thirteen": 13, "fourteen": 14, + "fifteen": 15, "sixteen": 16, "seventeen": 17, "eighteen": 18, "nineteen": 19, "twenty": 20, +} +# "12 Jahre", "zwölfjährig", "fast acht Sommer", "15 years old", "eight winters". +# Not "15 Jahre älter" / "vor 20 Jahren": a difference or a time span is no age. _EXPLICIT_AGE_RE = re.compile( - r"\b(\d{1,2})[\s-]*(?:jahre|jährig|j[äa]hrige[rns]?|years?[\s-]*old|y\.?o\.?)\b", re.IGNORECASE, + r"\b(\d{1,2}|" + "|".join(sorted(_AGE_NUMBER_WORDS, key=len, reverse=True)) + r")[\s-]*" + r"(?:jahre|jährig|j[äa]hrige[rns]?|sommer|winter|years?(?:[\s-]*old)?|summers|winters|y\.?o\.?)\b" + r"(?!\s*(?:älter|jünger|aelter|juenger|older|younger|her\b|ago|zuvor|später|lang|long|later))", + re.IGNORECASE, ) _CHILD_KEYWORD_RE = re.compile( r"\b(kind|kleinkind|kindlich|s[äa]ugling|kleinkindalter|" @@ -2173,6 +2206,44 @@ _TEEN_KEYWORD_RE = re.compile( r"\b(jugendlich(?:e|er|es)?|teenager|halbw[üu]chsig(?:e|er|es)?|heranwachsend(?:e|er|es)?|" r"teen(?:ager)?|adolescent)\b", re.IGNORECASE, ) +# A boy/girl noun says "minor" without saying how old. German "Junge(n)" is the +# noun only when capitalised and not followed by another capitalised word +# ("Junge Frau" = young woman, "Junger Mann" = young man: adults). +_MINOR_NOUN_RE = re.compile( + r"\b(?:mädchen|maedchen|knabe|knaben|bub|buben|boy|boys|girl|girls|lad|lass)\b", re.IGNORECASE, +) +_GERMAN_BOY_NOUN_RE = re.compile(r"\bJungen?\b(?!\s+[A-ZÄÖÜ])") +# "im Text als Junge beschrieben", "described in the text as a boy": the +# extractor's own evidence clause decides a hedged "A / B" estimate. +_AGE_EVIDENCE_CLAUSE_RE = re.compile( + r"(?:im\s+text|laut\s+text|described(?:\s+in\s+the\s+text)?|according\s+to\s+the\s+text)" + r"\s+(?:als\s+|as\s+)?(?:an?\s+|ein(?:e|en)?\s+)?([^(),;/]{2,40})", + re.IGNORECASE, +) + + +def _age_category_of_text(text: str, label: bool) -> str: + """'child' / 'teen' / 'adult' from one piece of text, '' when it says nothing. + + ``label`` marks short label fields (age estimate, archetype, aliases ...), + where a bare boy/girl noun describes the character itself; in prose + ("physical") such a noun may be somebody else, so only explicit child / + teen words count there. + """ + if not text: + return "" + match = _EXPLICIT_AGE_RE.search(text) + if match: + raw = match.group(1).lower() + age = int(raw) if raw.isdigit() else _AGE_NUMBER_WORDS.get(raw, 99) + return "child" if age < 13 else "teen" if age < 18 else "adult" + if _CHILD_KEYWORD_RE.search(text): + return "child" + if _TEEN_KEYWORD_RE.search(text): + return "teen" + if label and (_MINOR_NOUN_RE.search(text) or _GERMAN_BOY_NOUN_RE.search(text)): + return "teen" + return "" def _character_age_category(sheet: dict) -> str: @@ -2182,32 +2253,49 @@ def _character_age_category(sheet: dict) -> str: Every render template previously hard-coded "EXACTLY ONE ADULT PERSON" for every character with zero exceptions, so a genuinely child or teenage cast member was always rendered as a full adult — confirmed as a - real gap, not just an edge case, once asked about directly. An explicit - numeric age wins when present; otherwise this falls back to age-bearing - words anywhere in the established profile text. + real gap, not just an edge case, once asked about directly. + + Order: an explicit age (digits or number words, "Jahre"/"Sommer"/"years"/ + "winters") wins, also from source quotes the extractor tagged as age + evidence ("Fast acht Sommer, Kommandant." for Marrad); then the age + estimate's own evidence clause ("im Text als Junge beschrieben"); then + child / teen words, and boy/girl nouns in label fields. Works on the + German record and on its English translation alike. A hedged + "Junges Mädchen / Junger Mann" with any minor signal is a minor: rendering + a boy as a grown man was the reported failure, the opposite has not been + seen. """ - text = " ".join(filter(None, ( + sources = sheet.get("sources") if isinstance(sheet.get("sources"), list) else [] + age_quotes = " ".join( + _visual_prompt_value(s.get("quote")) for s in sources + if isinstance(s, dict) and re.search(r"(? str: return f"{value} — one complete portable object matching this exact noun, silhouette, and function, resting on the sweep as a tabletop still-life object" -def _character_scene_evidence(sheet: dict) -> str: - """Choose one representative, source-grounded action for the lower panel.""" +def _character_scene_evidence(sheet: dict, name: str = "") -> str: + """Choose one representative, source-grounded action for the lower panel. + + Only a quote that names THIS character (its name, real name or a + non-generic alias) is eligible. Extraction reads passages with several + characters, so a sheet's sources can hold another cast member's line; on + the real library 16 of 270 lower panels showed a quote naming only another + cast member (e.g. Arthag's "... Nyrilla drehte sich zum Zwerg um ..."). + Without an eligible quote the character's own mannerisms/capabilities + text is used instead of somebody else's moment. + """ override = _visual_prompt_value(sheet.get("visual_scene")).strip() if override: return override[:700] + own_name = str(name or sheet.get("name") or "").strip() sources = sheet.get("sources") if isinstance(sheet.get("sources"), list) else [] + if own_name: + sources = _cr.scene_sources_about_character(own_name, sheet) priorities = { "action": 12, "mannerisms": 10, "capabilities": 9, "conflict_style": 8, "communication_style": 7, @@ -2691,7 +2791,7 @@ def _build_compact_character_render_prompts(name: str, book: str, sheet: dict, # structured fallback already uses _primary_visual_role when no edited # Subject is available. props = _character_prop_item_list(sheet) - scene_evidence = _positive_conditioning_text(_period_wording(_strip_injury_clauses(_character_scene_evidence(sheet)))) + scene_evidence = _positive_conditioning_text(_period_wording(_strip_injury_clauses(_character_scene_evidence(sheet, name)))) has_headwear = _character_has_established_headwear(sheet) # Saved prompts were written before the wording/injury clean-up existed. portrait_subject = _positive_conditioning_text(_period_wording(_strip_injury_clauses(portrait_subject))) diff --git a/scripts/audit_character_records.py b/scripts/audit_character_records.py new file mode 100644 index 0000000..e315e99 --- /dev/null +++ b/scripts/audit_character_records.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Read-only audit of saved character records for extraction/merge damage. + +Usage: + python3 scripts/audit_character_records.py [DB_PATH] [--book SUBSTRING] [--json] [--details] + +DB_PATH defaults to config/tts_creator.db. The database is opened with +SQLite's read-only URI mode (``mode=ro``) and ``PRAGMA query_only``; nothing is +written. Run it against a copy when the app is busy. + +Reports, per book: + duplicated_sheets two records with the same long field text / the same + sources list (one sheet copied into two records) + alias_collisions an alias or first/last name that is another cast + member's record name in the same book + generic_alias_links two records linked only by a generic alias such as + "Zwerg" / "der Zwerg" / "Mann" (never identity proof) + thinking_leakage reasoning text in a field ("Warte:", "Text sagt", ...) + foreign_sources source quotes that name another cast member but not + this character + scene_quote_other the story-moment quote the image prompt would pick + names another cast member and not this character +See architecture/character-extraction.md. +""" +from __future__ import annotations + +import argparse +import collections +import json +import os +import sqlite3 +import sys +import tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT)) + +from core import character_records as cr # noqa: E402 + +DUP_FIELDS = ("race_species", "relationships", "physical", "backstory", "motivation", "clothing", + "profession", "age_estimate", "skills", "capabilities", "fears", "mannerisms", "secret") +DUP_MIN_CHARS = 25 + + +def load_records(db_path: str, book_filter: str = "") -> list[dict]: + uri = Path(db_path).resolve().as_uri() + "?mode=ro" + con = sqlite3.connect(uri, uri=True) + try: + con.execute("PRAGMA query_only = ON") + rows = con.execute("SELECT id, book, name, sheet, created FROM characters").fetchall() + finally: + con.close() + out = [] + for rid, book, name, sheet, created in rows: + if book_filter and book_filter.lower() not in str(book).lower(): + continue + try: + data = json.loads(sheet or "{}") + except json.JSONDecodeError: + data = {} + out.append({"id": rid, "book": book or "", "name": name or "", "sheet": data if isinstance(data, dict) else {}, + "created": created or ""}) + return out + + +def _picker(): + """The story-moment picker the image prompts use (current code).""" + os.environ.setdefault("CONFIG_DIR", tempfile.mkdtemp(prefix="ttvc-audit-config-")) + try: + import routes.conversation as conv # noqa: WPS433 + except Exception as exc: # pragma: no cover - environment without app deps + print(f"[audit] scene-quote check skipped: {exc}", file=sys.stderr) + return None + + def pick(name: str, sheet: dict) -> str: + sheet = dict(sheet) + sheet.setdefault("name", name) + sheet.pop("visual_scene", None) # audit the extracted evidence, not a manual override + try: + return conv._character_scene_evidence(sheet, name) + except TypeError: # older signature without the name argument + return conv._character_scene_evidence(sheet) + return pick + + +def audit(records: list[dict], with_scene: bool = True) -> dict: + by_book: dict[str, list[dict]] = collections.defaultdict(list) + for rec in records: + by_book[rec["book"]].append(rec) + issues: dict[str, list] = {k: [] for k in ( + "duplicated_sheets", "alias_collisions", "generic_alias_links", + "thinking_leakage", "foreign_sources", "scene_quote_other")} + pick = _picker() if with_scene else None + for book, recs in by_book.items(): + cast = [(r["name"], r["sheet"]) for r in recs] + names_lower = {cr.normalize_identity_token(r["name"]): r["name"] for r in recs} + for i, a in enumerate(recs): + for b in recs[i + 1:]: + same = [f for f in DUP_FIELDS + if len(str(a["sheet"].get(f) or "")) >= DUP_MIN_CHARS and a["sheet"].get(f) == b["sheet"].get(f)] + same_sources = bool(a["sheet"].get("sources")) and a["sheet"].get("sources") == b["sheet"].get("sources") + if len(same) >= 2 or same_sources or "race_species" in same: + issues["duplicated_sheets"].append({ + "book": book, "a": a["name"], "b": b["name"], "fields": same, "same_sources": same_sources, + "a_created": a["created"], "b_created": b["created"]}) + ta = cr.identity_tokens(a["name"], a["sheet"], include_generic=True) + tb = cr.identity_tokens(b["name"], b["sheet"], include_generic=True) + shared = ta & tb + if shared and not cr.shared_identity_evidence(ta, tb): + issues["generic_alias_links"].append({"book": book, "a": a["name"], "b": b["name"], + "shared": sorted(shared)}) + for rec in recs: + name, sheet = rec["name"], rec["sheet"] + own = cr.normalize_identity_token(name) + for alias in cr.split_identity_tokens(sheet.get("aliases")): + other = names_lower.get(cr.normalize_identity_token(alias)) + if other and cr.normalize_identity_token(other) != own: + issues["alias_collisions"].append({"book": book, "name": name, "field": "aliases", "value": alias}) + for field in ("first_name", "last_name", "full_name"): + other = names_lower.get(cr.normalize_identity_token(sheet.get(field) or "")) + if other and cr.normalize_identity_token(other) != own: + issues["alias_collisions"].append({"book": book, "name": name, "field": field, + "value": sheet.get(field)}) + leaked = [f for f in cr.TEXT_FIELDS if cr.has_thinking_leakage(sheet.get(f))] + if leaked: + issues["thinking_leakage"].append({"book": book, "name": name, "fields": leaked}) + for src in sheet.get("sources") or []: + quote = str((src or {}).get("quote") or "") if isinstance(src, dict) else "" + if not quote or cr.quote_mentions_identity(quote, name, sheet): + continue + others = cr.quote_mentions_other_cast(quote, name, sheet, cast) + if others: + issues["foreign_sources"].append({"book": book, "name": name, "others": others, + "quote": quote[:160]}) + if pick is not None: + quote = pick(name, sheet) + source_quotes = {str((x or {}).get("quote") or "").strip()[:700] + for x in sheet.get("sources") or [] if isinstance(x, dict)} + # Only a picked SOURCE quote counts; the fallback is the + # character's own mannerisms/capabilities text. + if quote and quote.strip() in source_quotes and not cr.quote_mentions_identity(quote, name, sheet): + others = cr.quote_mentions_other_cast(quote, name, sheet, cast) + if others: + issues["scene_quote_other"].append({"book": book, "name": name, "others": others, + "quote": quote[:160]}) + summary = { + "records": len(records), + "books": len(by_book), + "duplicated_sheet_pairs": len(issues["duplicated_sheets"]), + "alias_collisions": len(issues["alias_collisions"]), + "records_with_alias_collisions": len({(x["book"], x["name"]) for x in issues["alias_collisions"]}), + "generic_alias_links": len(issues["generic_alias_links"]), + "records_with_thinking_leakage": len(issues["thinking_leakage"]), + "foreign_source_quotes": len(issues["foreign_sources"]), + "records_with_foreign_sources": len({(x["book"], x["name"]) for x in issues["foreign_sources"]}), + "scene_quote_about_other": len(issues["scene_quote_other"]), + } + return {"summary": summary, "issues": issues} + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("db", nargs="?", default=str(ROOT / "config" / "tts_creator.db")) + parser.add_argument("--book", default="", help="only books whose title contains this text") + parser.add_argument("--json", action="store_true", help="print the full result as JSON") + parser.add_argument("--details", action="store_true", help="list every finding") + parser.add_argument("--no-scene", action="store_true", help="skip the image-prompt scene-quote check") + args = parser.parse_args(argv) + if not Path(args.db).is_file(): + print(f"database not found: {args.db}", file=sys.stderr) + return 2 + result = audit(load_records(args.db, args.book), with_scene=not args.no_scene) + if args.json: + print(json.dumps(result, ensure_ascii=False, indent=2)) + return 0 + for key, value in result["summary"].items(): + print(f"{key:32} {value}") + if args.details: + for kind, items in result["issues"].items(): + print(f"\n## {kind} ({len(items)})") + for item in items: + print(" " + json.dumps(item, ensure_ascii=False)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/clean_character_records.py b/scripts/clean_character_records.py new file mode 100644 index 0000000..e7c52b2 --- /dev/null +++ b/scripts/clean_character_records.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +"""Dry-run-first cleanup of extraction damage in saved character records. + +Usage: + python3 scripts/clean_character_records.py [DB_PATH] [--book SUBSTRING] [--out DIR] + python3 scripts/clean_character_records.py [DB_PATH] --apply --out DIR # owner approval only + +Without ``--apply`` nothing is written to the database: the database is opened +read-only and the planned changes are written as a report (report.json + +report.md) into ``--out`` (default: a new temp directory). + +With ``--apply`` the script first writes ``backup.json`` (every affected row, +unchanged) and the report into ``--out``, then updates only those rows in one +transaction. Restore = write the backed-up ``sheet`` values back (see +architecture/character-extraction.md, "Cleanup and rollback"). + +Deterministic, evidence-based fixes only: + 1. reasoning leakage ("Warte: ...", "Der Text sagt ...") cut from text fields + 2. aliases / first / last / full names that are ANOTHER cast member's record + name in the same book removed + 3. only with --drop-foreign-sources: source quotes that name another cast + member but not this character removed (lossy, off by default) +Duplicated sheets (one sheet copied into two records) are only REPORTED: which +record is genuine needs the owner's judgement. +""" +from __future__ import annotations + +import argparse +import copy +import datetime as _dt +import json +import sqlite3 +import sys +import tempfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT)) + +from core import character_records as cr # noqa: E402 +from scripts.audit_character_records import audit, load_records # noqa: E402 + + +def plan_changes(records: list[dict], drop_foreign_sources: bool = False) -> list[dict]: + """Return one entry per record that would change: {id, book, name, changes, before, after}.""" + by_book: dict[str, list[dict]] = {} + for rec in records: + by_book.setdefault(rec["book"], []).append(rec) + # Records that are copies of one another (same sources / long fields) are + # probably one person cast twice: their names in each other's aliases are + # not "foreign". That duplication is reported for the owner instead. + copies: dict[tuple[str, str], set[str]] = {} + for dup in audit(records, with_scene=False)["issues"]["duplicated_sheets"]: + copies.setdefault((dup["book"], dup["a"]), set()).add(dup["b"]) + copies.setdefault((dup["book"], dup["b"]), set()).add(dup["a"]) + plans = [] + for book, recs in by_book.items(): + cast = [(r["name"], r["sheet"]) for r in recs] + for rec in recs: + twins = copies.get((book, rec["name"]), set()) + cast_names = [r["name"] for r in recs if r["name"] not in twins] + before = rec["sheet"] + after = copy.deepcopy(before) + changes: list[dict] = [] + for field, (old, new) in cr.sanitize_sheet_text_fields(after).items(): + changes.append({"kind": "thinking_leakage", "field": field, "old": old, "new": new}) + # Only where the record contradicts itself: its own relationships + # name that person as somebody else (Kolon "dient ... Sharraz + # Garthai" yet alias "Sharraz"). Other collisions may be one person + # cast twice (Promos / Alchimist): listed for review, never applied. + probe = copy.deepcopy(after) + relationships = str(after.get("relationships") or "") + real_name = f"{after.get('first_name') or ''} {after.get('last_name') or ''}".strip() + for removed in cr.drop_foreign_identity(rec["name"], probe, cast_names): + field, _, value = removed.partition("=") + probe_text = real_name if field in ("first_name", "last_name") else value + related = [n for n in cast_names if cr.normalize_identity_token(n) != cr.normalize_identity_token(rec["name"]) + and not cr.is_generic_identity_token(n) and cr.text_mentions_token(probe_text, n) + and cr.text_mentions_token(relationships, n)] + kind = "foreign_identity" if related else "review_identity" + changes.append({"kind": kind, "field": field, "removed": value, + **({"evidence": f"relationships names {', '.join(related)}"} if related else {})}) + contradicted = {c["removed"] for c in changes if c["kind"] == "foreign_identity" and c["field"] == "aliases"} + if contradicted: + after["aliases"] = ", ".join(a for a in cr.split_identity_tokens(after.get("aliases")) if a not in contradicted) + for c in changes: + if c["kind"] == "foreign_identity" and c["field"] in ("first_name", "last_name", "full_name"): + after[c["field"]] = "" + sources = after.get("sources") if isinstance(after.get("sources"), list) else [] + kept = [] + for src in sources: + quote = str(src.get("quote") or "") if isinstance(src, dict) else "" + if drop_foreign_sources and quote and not cr.quote_mentions_identity(quote, rec["name"], after): + others = cr.quote_mentions_other_cast(quote, rec["name"], after, cast) + if others: + changes.append({"kind": "foreign_source", "field": "sources", "removed": quote[:240], + "names": others}) + continue + kept.append(src) + if len(kept) != len(sources): + after["sources"] = kept + if changes: + plans.append({"id": rec["id"], "book": book, "name": rec["name"], "changes": changes, + "before": before, "after": after}) + return plans + + +def write_report(out: Path, plans: list[dict], audit_result: dict, applied: bool) -> None: + out.mkdir(parents=True, exist_ok=True) + slim = [{k: v for k, v in p.items() if k not in ("before", "after")} for p in plans] + (out / "report.json").write_text(json.dumps({ + "applied": applied, "records_changed": len(plans), "plans": slim, + "duplicated_sheets_report_only": audit_result["issues"]["duplicated_sheets"], + "audit_summary": audit_result["summary"], + }, ensure_ascii=False, indent=2), encoding="utf-8") + counts: dict[str, int] = {} + for p in plans: + for c in p["changes"]: + counts[c["kind"]] = counts.get(c["kind"], 0) + 1 + lines = [f"# Character record cleanup ({'APPLIED' if applied else 'dry run, nothing written'})", "", + f"Records that would change: {sum(1 for p in plans if p['after'] != p['before'])}", ""] + lines += [f"- {kind}: {n}" for kind, n in sorted(counts.items())] + lines += ["", f"Duplicated sheets (report only, owner decides): {len(audit_result['issues']['duplicated_sheets'])}", ""] + for p in plans: + lines.append(f"## {p['book']} | {p['name']}") + for c in p["changes"]: + if c["kind"] == "thinking_leakage": + lines.append(f"- {c['field']}: cut to {c['new']!r} (was {len(c['old'])} chars)") + elif c["kind"] == "foreign_identity": + lines.append(f"- {c['field']}: remove {c['removed']!r} ({c['evidence']})") + elif c["kind"] == "review_identity": + lines.append(f"- REVIEW ONLY (not changed): {c['field']} {c['removed']!r} is another record's name") + else: + lines.append(f"- source quote naming {', '.join(c['names'])}: remove {c['removed'][:100]!r}") + lines.append("") + (out / "report.md").write_text("\n".join(lines), encoding="utf-8") + + +def apply_plans(db_path: str, plans: list[dict], out: Path) -> None: + out.mkdir(parents=True, exist_ok=True) + plans = [p for p in plans if p["after"] != p["before"]] # review-only entries are never written + con = sqlite3.connect(db_path) + try: + ids = [p["id"] for p in plans] + rows = [] + for rid in ids: + row = con.execute("SELECT id, book, name, sheet, updated FROM characters WHERE id = ?", (rid,)).fetchone() + if row: + rows.append(dict(zip(("id", "book", "name", "sheet", "updated"), row))) + backup = out / "backup.json" + backup.write_text(json.dumps(rows, ensure_ascii=False, indent=1), encoding="utf-8") + if len(rows) != len(ids): + raise SystemExit("records changed since planning; nothing written") + now = _dt.datetime.now(_dt.timezone.utc).isoformat().replace("+00:00", "Z") + with con: + for p in plans: + con.execute("UPDATE characters SET sheet = ?, updated = ? WHERE id = ?", + (json.dumps(p["after"], ensure_ascii=False), now, p["id"])) + finally: + con.close() + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("db", nargs="?", default=str(ROOT / "config" / "tts_creator.db")) + parser.add_argument("--book", default="") + parser.add_argument("--out", default="", help="report/backup directory (default: new temp dir)") + parser.add_argument("--drop-foreign-sources", action="store_true", + help="also remove source quotes that name another cast member but not this one " + "(lossy: some are about this character by pronoun/role; image prompts already skip them)") + parser.add_argument("--apply", action="store_true", + help="write the changes (backup first). Only with the owner's approval.") + args = parser.parse_args(argv) + if not Path(args.db).is_file(): + print(f"database not found: {args.db}", file=sys.stderr) + return 2 + out = Path(args.out) if args.out else Path(tempfile.mkdtemp(prefix="ttvc-character-cleanup-")) + records = load_records(args.db, args.book) # read-only connection + plans = plan_changes(records, drop_foreign_sources=args.drop_foreign_sources) + audit_result = audit(records, with_scene=False) + if args.apply: + if not args.out: + print("--apply needs an explicit --out directory for the backup", file=sys.stderr) + return 2 + apply_plans(args.db, plans, out) + write_report(out, plans, audit_result, applied=args.apply) + kinds: dict[str, int] = {} + for p in plans: + for c in p["changes"]: + kinds[c["kind"]] = kinds.get(c["kind"], 0) + 1 + changed = sum(1 for p in plans if p["after"] != p["before"]) + print(f"{'applied' if args.apply else 'dry run'}: {changed} records would change {kinds}; report: {out}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/static/js/character-identity.js b/static/js/character-identity.js new file mode 100644 index 0000000..a6a5c72 --- /dev/null +++ b/static/js/character-identity.js @@ -0,0 +1,81 @@ +// ── Character identity rules ────────────────────────────────────────────────── +// Which alias/name tokens are evidence that two character sheets describe the +// SAME person. A token made only of common nouns, roles, ranks, species or an +// article-led description ("Zwerg", "der Zwerg", "Mann", "Fremder", +// "Alter Soldat", "Ork-General", "der Tunneltreiber") labels many people and +// is never such evidence. On the real library a shared "Zwerg"/"Mann" token +// merged or copied unrelated sheets (Arthag/Kolon, Mann/Fremder in book 03). +// +// Pure functions, no DOM: loaded in the browser before characters-library.js +// and character-sheets.js, and importable from Node for unit tests. The word +// list mirrors core/character_records.py (a Python test keeps them in sync). +(function (root) { +'use strict'; + +const ARTICLES = new Set(('a an das dem den der des die ein eine einem einen einer eines the').split(' ')); +const GENERIC_WORDS = new Set(( + 'adept adepten admiral alte alten alter altes amazone anführer anführerin auelfe baron ' + + 'baronin bauer befehlshaber begleiter begleiterin blacksmith blonde blonder bote botin boy brother ' + + 'bruder bub bäckerstochter bäuerin bürger bürgerin bürgersohn captain chief child colonel commander ' + + 'companion creature crowd dame daughter de del demon diener dienerin dritte dritter ' + + 'druid druide dunkle dunkler dwarf dämon elf elfe elfen erste erster erzähler ' + + 'executioner father feldherr figure fischer frau frauen fremde fremden fremder führer gast ' + + 'gefährte gefährtin general gestalt geweihter girl graf greis greisin groß großadmiral große ' + + 'großer gräfin guard hauptfrau hauptmann heerführer henker herr herrin herzog herzogin hexe ' + + 'human händler häuptling inquisitor irre jung junge jungen junger junges junker jäger ' + + 'kaiser kaiserin kapitän kaufmann kid kind king kleine kleiner knabe knecht knight ' + + 'kommandant kommandantin kreatur krieger kriegerin kriegshäuptling kutscher könig königin lady leader lehrer ' + + 'leute lord magd mage magier magierin man mann marschall marshal master meister ' + + 'menge mensch menschen merchant messenger militärführer monster mother mutter mädchen männer mönch ' + + 'nachbar nachbarin narrator navigator nonne oberst obrist officer offizier offizierin old onkel ' + + 'orc ork orkin orks person priest priester priesterin prince princess prinz prinzessin ' + + 'queen reiter ritter ritterin rote roter schamane schmied schreiber schwarzhaarige schwester schüler ' + + 'scribe servant shaman sir sister sklave sklaven sklavin smith sohn soldat soldaten ' + + 'soldatin soldier son stimme stimmen stranger tante tochter torwächter tote toter unbekannt ' + + 'unbekannte unbekannter unknown unsicher vampir vampire vampirin van vater vermummt vermummte vermummten ' + + 'vermummter veteran voice vom von wache wachen warrior weber wesen wirt wirtin ' + + 'witch witwe witwer wizard woman wächter young zauberer zauberin zu zum zur ' + + 'zweite zweiter zwerg zwerge zwergin' +).split(' ')); +const GENERIC_SUFFIXES = [...GENERIC_WORDS].filter(w => w.length >= 5).sort((a, b) => b.length - a.length); + +function normalizeToken(token) { + return String(token == null ? '' : token) + .replace(/\([^)]*\)/g, ' ') + .replace(/\s+/g, ' ') + .trim() + .replace(/^[\s.,;:!?"'»«„“”‚‘’]+|[\s.,;:!?"'»«„“”‚‘’]+$/g, '') + .toLowerCase(); +} + +function isGenericIdentityToken(token) { + const norm = normalizeToken(token); + if (!norm) return true; + const words = norm.split(/[\s-]+/).filter(Boolean); + if (!words.length) return true; + if (ARTICLES.has(words[0])) return true; // "der Henker": a description + return words.every(word => { + const bare = word.replace(/^['’.]+|['’.]+$/g, ''); + if (bare.length <= 1 || ARTICLES.has(bare) || GENERIC_WORDS.has(bare)) return true; + return GENERIC_SUFFIXES.some(sfx => bare.length > sfx.length && bare.endsWith(sfx)); + }); +} + +// Shared tokens of two lower-cased token collections that prove identity. +function sharedIdentityEvidence(a, b) { + const left = new Set([...(a || [])].map(normalizeToken)); + const out = []; + for (const t of (b || [])) { + const n = normalizeToken(t); + if (n && left.has(n) && !isGenericIdentityToken(n) && !out.includes(n)) out.push(n); + } + return out; +} + +const characterIdentityApi = { isGenericIdentityToken, sharedIdentityEvidence, normalizeToken, GENERIC_WORDS, ARTICLES }; +root.TTVC = root.TTVC || {}; +root.TTVC.characterIdentity = characterIdentityApi; +root.ttvcIsGenericIdentityToken = isGenericIdentityToken; +root.ttvcSharedIdentityEvidence = sharedIdentityEvidence; +if (typeof module !== 'undefined' && module.exports) module.exports = characterIdentityApi; +})(typeof window !== 'undefined' ? window : globalThis); diff --git a/static/js/character-sheets.js b/static/js/character-sheets.js index eaea5fe..5819074 100644 --- a/static/js/character-sheets.js +++ b/static/js/character-sheets.js @@ -353,11 +353,26 @@ function csSheetMatchesRoster(sheet, roster = []) { // display name. Character-sheet extraction may discover a real name, role, or // epithet and put it in `name`, but that must never replace the name the user // sees in Characters found / the cast. Keep those discoveries in aliases. +// Identity evidence: a shared token proves "same person" only when it is a +// real name, never a common noun / role / title such as "Zwerg", "der Zwerg", +// "Mann" or "Fremder" (static/js/character-identity.js). +function csIsGenericIdentityToken(token) { + const ci = (typeof window !== 'undefined' && window.TTVC && window.TTVC.characterIdentity) || null; + return ci ? ci.isGenericIdentityToken(token) : false; +} + function csRosterCanonicalName(sheet, roster = []) { const names = csCleanRoster(roster); if (!names.length) return ''; + // The sheet's own name wins whenever it is itself a cast name. Picking the + // FIRST roster entry found among all its tokens renamed the book-03 Arthag + // sheet (aliases "der Zwerg, Zwerg, ..., Kolon") to "Zwerg"/"Kolon", and the + // save then created those records as full copies of Arthag's sheet. + const own = String(sheet?.name || '').trim().toLowerCase(); + const ownMatch = own ? names.find(n => n.toLowerCase() === own) : ''; + if (ownMatch) return ownMatch; const tokens = csNameTokens(sheet); - const exact = names.find(n => tokens.has(n.toLowerCase())); + const exact = names.find(n => tokens.has(n.toLowerCase()) && !csIsGenericIdentityToken(n)); return exact || ''; } @@ -471,13 +486,19 @@ function csEvidenceWindowText(text, needles, before = 2, after = 2) { } function csFindMergeKey(map, sheet) { + const ownKey = (sheet.name || '').trim().toLowerCase(); + // An entry with exactly this name is always the target. + if (ownKey && map.has(ownKey)) return ownKey; const incoming = csNameTokens(sheet); - if (!incoming.size) return (sheet.name || '').trim().toLowerCase(); + if (!incoming.size) return ownKey; for (const [key, existing] of map.entries()) { const existingNames = csNameTokens(existing); - for (const n of incoming) if (existingNames.has(n)) return key; + // Only a shared real name links two sheets; "Zwerg", "der Zwerg", + // "Mann", "Fremder" are labels many different people share, and merging + // on them let the longer field text of one character overwrite another's. + for (const n of incoming) if (existingNames.has(n) && !csIsGenericIdentityToken(n)) return key; } - return (sheet.name || '').trim().toLowerCase(); + return ownKey; } function csMergeAliases(existing, incoming) { @@ -580,7 +601,9 @@ function csConsolidateAliasDuplicates(map, establishedKeys, establishedCreated, changed = false; for (const [key, s] of map) { const aliasKeys = csSplitIdentityTokens(s.aliases, { aliases: true }).map(a => a.toLowerCase()); - const dupKey = aliasKeys.find(a => a !== key && map.has(a)); + // A generic alias ("Mann", "Fremder", "Zwerg") naming another entry is + // no proof that the two are one person. + const dupKey = aliasKeys.find(a => a !== key && map.has(a) && !csIsGenericIdentityToken(a)); if (!dupKey) continue; const other = map.get(dupKey); // Established (pre-existing library record) beats a same-run stub. If @@ -2503,10 +2526,15 @@ async function csForReader(opts = {}) { last_name: rec?.sheet?.last_name || '', full_name: rec?.sheet?.full_name || '', }, knownRoster); + const savedNames = new Set((existing || []).map(rec => String(rec?.name || '').trim().toLowerCase()).filter(Boolean)); seedSheets = (existing || []).filter(stillCast).map(rec => { const sheet = csSeedSheet(rec); const rosterName = csRosterCanonicalName(sheet, knownRoster); - if (rosterName && sheet.name.toLowerCase() !== rosterName.toLowerCase()) { + // Never rename a saved sheet onto a cast name that has its own saved + // record: both seeds would claim one key, and the renamed sheet was + // saved as a copy under the other name (book 03: Zwerg/Kolon = Arthag). + if (rosterName && sheet.name.toLowerCase() !== rosterName.toLowerCase() + && !savedNames.has(rosterName.toLowerCase())) { sheet.aliases = csMergeAliases( { name: rosterName, aliases: sheet.aliases }, { name: sheet.name, aliases: '' }, @@ -2714,5 +2742,7 @@ Object.assign(root.TTVC.characterSheets, { show: csShow, forReader: csForReader, forRehearser: csForRehearser, + // Pure identity/merge helpers, exported for tests/character-identity.test.mjs. + _identity: { csFindMergeKey, csRosterCanonicalName, csMerge, csConsolidateAliasDuplicates, csCanonicalizeRosterNames }, }); })(window); diff --git a/static/js/characters-library.js b/static/js/characters-library.js index 6af6a30..9a5cbf2 100644 --- a/static/js/characters-library.js +++ b/static/js/characters-library.js @@ -136,7 +136,11 @@ function clSameIdentity(rec, book, sheet) { if (String(rec?.book || '').trim().toLowerCase() !== String(book || '').trim().toLowerCase()) return false; const a = clIdentityNames(rec); const b = clIdentityNames(sheet); - for (const n of b) if (a.has(n)) return true; + // Only a shared real name is evidence. "Zwerg", "der Zwerg", "Mann", + // "Fremder" (static/js/character-identity.js) label many people; matching + // on them redirected one character's sheet into another's record. + const ci = (typeof window !== 'undefined' && window.TTVC && window.TTVC.characterIdentity) || null; + for (const n of b) if (a.has(n) && !(ci && ci.isGenericIdentityToken(n))) return true; return false; } @@ -219,7 +223,11 @@ async function clUpsert(book, sheet, knownId) { ? await clGet(authoritativeId).catch(() => null) : knownId ? await clGet(knownId).catch(() => null) - : (await clGetAll().catch(() => [])).find(r => clSameIdentity(r, bk, sheet)); + // A record that already has exactly this name is the target; the alias + // scan is only for a sheet that has no record of its own yet. Scanning + // first let "Kolon" (aliases incl. "Sharraz") land in Sharraz's record. + : (await clGet(clKey(bk, name)).catch(() => null)) + || (await clGetAll().catch(() => [])).find(r => clSameIdentity(r, bk, sheet)); const id = authoritativeId || aliasPrev?.id || knownId || clKey(bk, name); const now = new Date(); const prev = aliasPrev || await clGet(id).catch(() => null); diff --git a/static/module-manifest.json b/static/module-manifest.json index 0c56ba3..ac0d85c 100644 --- a/static/module-manifest.json +++ b/static/module-manifest.json @@ -47,6 +47,7 @@ "rehearser-library", "rehearser-stage", "rehearser-voice-picker", + "character-identity", "characters-library", "reader", "audiobook/network", @@ -73,6 +74,7 @@ "rehearser-library", "rehearser-stage", "rehearser-voice-picker", + "character-identity", "characters-library", "reader", "audiobook/network", @@ -103,6 +105,7 @@ "rehearser-library", "rehearser-stage", "rehearser-voice-picker", + "character-identity", "characters-library", "reader", "audiobook/network", diff --git a/tests/character-identity.test.mjs b/tests/character-identity.test.mjs new file mode 100644 index 0000000..c14dee7 --- /dev/null +++ b/tests/character-identity.test.mjs @@ -0,0 +1,123 @@ +// Character identity / merge rules. Run: npm run test:unit +// Fixtures are shaped like the real book-02/03 records found broken on 2026-10-03 +// (see architecture/character-extraction.md). +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { createRequire } from 'node:module'; +import { readFileSync } from 'node:fs'; +import vm from 'node:vm'; + +const ci = createRequire(import.meta.url)('../static/js/character-identity.js'); +const BOOK = 'Das Jahr des Greifen 03'; + +function loadBrowserFiles(files, extra = {}) { + const el = new Proxy(function () {}, { get: (t, k) => (k === Symbol.toPrimitive ? () => '' : el), apply: () => el }); + const sandbox = { + console, setTimeout, clearTimeout, URL, Promise, Map, Set, JSON, + document: { querySelectorAll: () => [], querySelector: () => null, getElementById: () => null, + addEventListener() {}, createElement: () => el, body: el }, + $: () => null, toast() {}, indexedDB: { open: () => ({}) }, + ...extra, + }; + sandbox.window = sandbox; + sandbox.globalThis = sandbox; + vm.createContext(sandbox); + for (const f of files) vm.runInContext(readFileSync(new URL(`../static/js/${f}`, import.meta.url), 'utf8'), sandbox, { filename: f }); + return sandbox; +} + +test('generic labels are not names', () => { + for (const t of ['Zwerg', 'der Zwerg', 'Mann', 'Fremder', 'Fremde', 'Alter Soldat', 'Ork-General', 'Bürgerwehroffizier', + 'der Tunneltreiber', 'Kriegshäuptling', 'Erster Offizier', 'junges Mädchen', 'the Executioner']) { + assert.equal(ci.isGenericIdentityToken(t), true, t); + } + for (const t of ['Arthag', 'Kolon', 'Sharraz Garthai', 'Admiral Sanin', 'Schlange Gamba', 'Gutfolg', 'Marcus the Executioner']) { + assert.equal(ci.isGenericIdentityToken(t), false, t); + } + assert.deepEqual(ci.sharedIdentityEvidence(['arthag', 'der zwerg', 'zwerg'], ['kolon', 'zwerg', 'der zwerg']), []); + assert.deepEqual(ci.sharedIdentityEvidence(['marcian', 'inquisitor'], ['inquisitor', 'marcian']), ['marcian']); +}); + +// Book 03: Arthag's saved aliases contained "der Zwerg, Zwerg, ..., Kolon" and the +// cast roster listed Zwerg and Kolon before Arthag. The roster canonicaliser +// picked the first roster name among ALL tokens, renaming Arthag's sheet, and +// the save created Zwerg/Kolon as full copies of Arthag. +test('a sheet keeps its own cast name instead of a roster name found in its aliases', () => { + const sb = loadBrowserFiles(['character-identity.js', 'character-sheets.js']); + const { csRosterCanonicalName } = sb.TTVC.characterSheets._identity; + const arthag = { name: 'Arthag', aliases: 'der Zwerg, Zwerg, Schreiber, Begleiter, Kolon, der Tunneltreiber' }; + assert.equal(csRosterCanonicalName(arthag, ['Zwerg', 'Kolon', 'Arthag']), 'Arthag'); + // A legacy record whose name is not cast maps only through a real name, never "Zwerg". + const legacy = { name: 'Amboßzwerg', aliases: 'Zwerg, der Zwerg' }; + assert.equal(csRosterCanonicalName(legacy, ['Zwerg', 'Kolon']), ''); + assert.equal(csRosterCanonicalName({ name: 'Murlok', aliases: 'Zerwas, Vampir' }, ['Vampir', 'Zerwas']), 'Zerwas'); +}); + +test('discovery merge does not fold two characters together over a generic alias', () => { + const sb = loadBrowserFiles(['character-identity.js', 'character-sheets.js']); + const { csMerge } = sb.TTVC.characterSheets._identity; + const map = new Map(); + csMerge(map, [{ name: 'Arthag', aliases: 'der Zwerg, Zwerg', race_species: 'Zwerg', relationships: 'Nyrilla (Gefährtin)', sources: [] }]); + csMerge(map, [{ name: 'Kolon', aliases: 'Zwerg, Ork-General', race_species: 'Mensch (oder Ork-Verbündeter, Kontext unklar ...)', + relationships: 'Führt die Orks; dient Sharraz Garthai; arbeitet mit Drasdech', sources: [] }]); + assert.deepEqual([...map.keys()].sort(), ['arthag', 'kolon']); + assert.equal(map.get('arthag').race_species, 'Zwerg'); + assert.equal(map.get('arthag').relationships, 'Nyrilla (Gefährtin)'); + // A real shared name still merges (the alias proves the identity). + csMerge(map, [{ name: 'Murlok von Mengbilla', aliases: 'Arthag', notes: 'x', sources: [] }]); + assert.deepEqual([...map.keys()].sort(), ['arthag', 'kolon']); +}); + +test('consolidation ignores a generic alias that happens to be another entry name', () => { + const sb = loadBrowserFiles(['character-identity.js', 'character-sheets.js']); + const { csConsolidateAliasDuplicates } = sb.TTVC.characterSheets._identity; + const map = new Map([ + ['mann', { name: 'Mann', aliases: 'Fremde, Offizier, Fremder, Gutfolg', physical: 'rothaarig', sources: [], inventory: [] }], + ['fremder', { name: 'Fremder', aliases: '', physical: 'Kapuze', sources: [], inventory: [] }], + ]); + csConsolidateAliasDuplicates(map, new Set(), new Map(), []); + assert.deepEqual([...map.keys()].sort(), ['fremder', 'mann']); + assert.equal(map.get('fremder').physical, 'Kapuze'); +}); + +function libraryWithStore(records) { + const store = new Map(records.map(r => [r.id, structuredClone(r)])); + const writes = []; + const fetch = async (url, opts = {}) => { + const u = String(url); + const json = (body, status = 200) => ({ ok: status < 400, status, json: async () => body }); + if (u === '/api/characters' || u.startsWith('/api/characters?')) return json({ characters: [...store.values()] }); + const id = decodeURIComponent(u.replace('/api/characters/', '')); + if (!opts.method || opts.method === 'GET') return store.has(id) ? json(store.get(id)) : json({}, 404); + if (opts.method === 'PUT') { const rec = JSON.parse(opts.body); store.set(id, rec); writes.push(id); return json(rec); } + return json({ ok: true }); + }; + const sb = loadBrowserFiles(['character-identity.js', 'characters-library.js'], { fetch }); + return { sb, store, writes }; +} + +// Book 02: Kolon's extracted aliases named "Sharraz" (a separate cast record). +// The upsert scanned all records for ANY shared token before looking at the +// record that already carries the name, so Kolon's sheet could land in Sharraz. +test('library upsert writes to the record with the exact name, not an alias-linked one', async () => { + const { sb, store, writes } = libraryWithStore([ + { id: `${BOOK}::sharraz`.toLowerCase(), book: BOOK, name: 'Sharraz', sheet: { name: 'Sharraz', race_species: 'Ork' } }, + { id: `${BOOK}::kolon`.toLowerCase(), book: BOOK, name: 'Kolon', sheet: { name: 'Kolon', race_species: 'Zwerg' } }, + ]); + await sb.clUpsert(BOOK, { name: 'Kolon', aliases: 'Sharraz, Ork-General', physical: 'weißer Bart' }); + assert.deepEqual(writes, [`${BOOK}::kolon`.toLowerCase()]); + assert.equal(store.get(`${BOOK}::sharraz`.toLowerCase()).sheet.physical, undefined); +}); + +test('library upsert does not redirect a new character into a record sharing only "Zwerg"', async () => { + const { sb, store, writes } = libraryWithStore([ + { id: `${BOOK}::arthag`.toLowerCase(), book: BOOK, name: 'Arthag', + sheet: { name: 'Arthag', aliases: 'der Zwerg, Zwerg', race_species: 'Zwerg' } }, + ]); + await sb.clUpsert(BOOK, { name: 'Kolon', aliases: 'Zwerg, der Zwerg', race_species: 'Zwerg', relationships: 'Sharraz Garthai' }); + assert.deepEqual(writes, [`${BOOK}::kolon`.toLowerCase()]); + assert.equal(store.get(`${BOOK}::arthag`.toLowerCase()).sheet.relationships, undefined); + // A real shared name still finds the existing record (dedupe of a renamed sheet). + await sb.clUpsert(BOOK, { name: 'Amboßzwerg Arthag', aliases: 'Arthag', notes: 'Schreiber' }); + assert.equal(writes.at(-1), `${BOOK}::arthag`.toLowerCase()); +}); diff --git a/tests/test_character_record_hygiene.py b/tests/test_character_record_hygiene.py new file mode 100644 index 0000000..e5cdaa4 --- /dev/null +++ b/tests/test_character_record_hygiene.py @@ -0,0 +1,268 @@ +"""Character extraction / merge hygiene (real-library audit, 2026-10-03). + +Fixtures are shaped like the broken real records (see +architecture/character-extraction.md): +* Kolon (book 02): race_species with the model's reasoning ("Warte: ...", + "Nein, Kolon ist ...") and aliases/first/last naming Sharraz (a separate + cast record). +* Arthag (book 03): a lower-panel quote about Nyrilla picked over Arthag's own. +* Mann / Fremder (book 03): one sheet stored in two records. +* Marrad (book 02): a boy whose hedged age estimate rendered as an adult. +""" +from __future__ import annotations + +import hashlib +import json +import os +import re +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +os.environ.setdefault("CONFIG_DIR", tempfile.mkdtemp(prefix="ttvc-hygiene-test-config-")) + +import routes.conversation as conv # noqa: E402 +from core import character_records as cr # noqa: E402 +from scripts import audit_character_records as audit_script # noqa: E402 +from scripts import clean_character_records as clean_script # noqa: E402 + +ROOT = Path(__file__).resolve().parent.parent + +KOLON_SPECIES = ( + "Mensch (oder Ork-Verbündeter, Kontext unklar, aber 'Der Zwerg' wird separat erwähnt, Kolon scheint " + "der strategische Kopf zu sein - Text sagt 'Der Zwerg lächelte grimmig' im selben Absatz wie Kolon. " + "Warte: 'Der Zwerg lächelte grimmig ... Es wäre allein sein Verdienst ...'. Es ist mehrdeutig. " + "Nein, Kolon ist in der Liste der bekannten Charaktere.)" +) +KOLON_RELATIONSHIPS = "Führt die Orks; dient oder steht im Dienst von Sharraz Garthai; arbeitet mit Drasdech." + + +class GenericTokenTests(unittest.TestCase): + def test_labels_are_generic_names_are_not(self): + for token in ("Zwerg", "der Zwerg", "Mann", "Fremder", "Fremde", "Alter Soldat", "Ork-General", + "Bürgerwehroffizier", "der Tunneltreiber", "Kriegshäuptling", "Erster Offizier", "Von"): + self.assertTrue(cr.is_generic_identity_token(token), token) + for token in ("Arthag", "Kolon", "Sharraz Garthai", "Admiral Sanin", "Schlange Gamba", "Gutfolg"): + self.assertFalse(cr.is_generic_identity_token(token), token) + + def test_shared_generic_alias_is_no_identity_evidence(self): + arthag = cr.identity_tokens("Arthag", {"aliases": "der Zwerg, Zwerg, Schreiber"}, include_generic=True) + kolon = cr.identity_tokens("Kolon", {"aliases": "Zwerg, der Zwerg"}, include_generic=True) + self.assertTrue(arthag & kolon) + self.assertEqual(cr.shared_identity_evidence(arthag, kolon), set()) + + def test_js_word_list_matches_python(self): + js = (ROOT / "static" / "js" / "character-identity.js").read_text(encoding="utf-8") + block = re.search(r"const GENERIC_WORDS = new Set\(\((.*?)\)\.split", js, re.S).group(1) + words = set(" ".join(re.findall(r"'([^']*)'", block)).split()) + self.assertEqual(words, set(cr.GENERIC_IDENTITY_WORDS)) + arts = re.search(r"const ARTICLES = new Set\(\('([^']*)'\)", js).group(1).split() + self.assertEqual(set(arts), set(cr.ARTICLES)) + + +class ThinkingLeakageTests(unittest.TestCase): + def test_real_leak_shapes_are_cut(self): + self.assertEqual(cr.strip_thinking_leakage(KOLON_SPECIES), "Mensch") + self.assertEqual(cr.strip_thinking_leakage( + "Prinz des Reiches, begleitet von Alrik (oder ist Alrik selbst der Prinz? Der Text sagt 'Alrik " + "hatte ihr bis zuletzt nicht glauben wollen'. Es gibt Verwechslungsgefahr.)"), + "Prinz des Reiches, begleitet von Alrik") + self.assertEqual(cr.strip_thinking_leakage( + "Schwarzhaarig. Groß. Wait, 'Die Frau bückte die Bannerträgerin' is likely a typo."), + "Schwarzhaarig. Groß.") + self.assertEqual(cr.strip_thinking_leakage("Kolon or Arthag?Zwerg"), "Zwerg") + + def test_clean_text_is_untouched(self): + clean = ("Sie hat ihren Auftraggeber (Inquisition) verraten oder ist sich ihrer eigenen Gefühle " + "für Zerwas bewusst.") + self.assertFalse(cr.has_thinking_leakage(clean)) + self.assertEqual(cr.strip_thinking_leakage(clean), clean) + self.assertEqual(cr.strip_thinking_leakage(None), None) + + +class ParseIsolationTests(unittest.TestCase): + RAW = json.dumps({"sheets": [{ + "name": "Kolon", "aliases": "Sharraz Garthai, General, Ork-General, Sharraz, Zwerg, Kriegshäuptling", + "first_name": "Sharraz", "last_name": "Garthai", "race_species": KOLON_SPECIES, + "relationships": KOLON_RELATIONSHIPS, "physical": "Kleiner Mann mit einem weißen Bart.", + "sources": [{"page": 142, "quote": "Kolon kratzte sich mit dem Griffel am Hinterkopf.", "line_hint": "mannerisms"}], + }, { + "name": "Marina", "aliases": "Tochter des Promos, das Mädchen", "race_species": "Mensch", + }]}) + ROSTER = ["Kolon", "Arthag", "Sharraz", "Zwerg", "Kriegshäuptling", "Marina", "Promos"] + + def test_roster_names_never_become_aliases_and_leakage_is_stripped(self): + out = conv._charsheets_parse(self.RAW, self.ROSTER) + kolon = next(s for s in out["sheets"] if s["name"] == "Kolon") + aliases = [a.strip() for a in kolon["aliases"].split(",")] + self.assertNotIn("Sharraz", aliases) + self.assertNotIn("Sharraz Garthai", aliases) + self.assertIn("Ork-General", aliases) # a generic title is data, not a claim + self.assertEqual((kolon["first_name"], kolon["last_name"]), ("", "")) + self.assertEqual(kolon["race_species"], "Mensch") + self.assertEqual(kolon["relationships"], KOLON_RELATIONSHIPS) + marina = next(s for s in out["sheets"] if s["name"] == "Marina") + self.assertIn("Tochter des Promos", marina["aliases"]) # a relation, not a name + + def test_without_roster_aliases_are_kept_but_leakage_still_stripped(self): + out = conv._charsheets_parse(self.RAW) + kolon = out["sheets"][0] + self.assertIn("Sharraz", kolon["aliases"]) + self.assertEqual(kolon["race_species"], "Mensch") + + def test_roster_only_passed_in_cast_target_mode(self): + base = {"text": "Kolon sprach.", "known_characters": ["Kolon", "Sharraz"], "llm_url": "http://x/v1"} + self.assertEqual(conv._charsheets_prepare(dict(base, target_mode=True))["roster"], ["Kolon", "Sharraz"]) + self.assertEqual(conv._charsheets_prepare(dict(base, target_mode=False))["roster"], []) + + +class SceneQuoteTests(unittest.TestCase): + ARTHAG = { + "name": "Arthag", "aliases": "der Zwerg, Zwerg, Schreiber", + "mannerisms": "Hockt in Ecken und rechnet auf Wachstafeln.", + "sources": [ + {"quote": "»Was soll die Frage?« Nyrilla drehte sich zum Zwerg um und rief laut.", "line_hint": "action"}, + {"quote": "Arthag wog prüfend seine Streitaxt.", "line_hint": "capabilities"}, + ], + } + + def test_quote_naming_the_character_beats_a_higher_scored_foreign_one(self): + self.assertEqual(conv._character_scene_evidence(dict(self.ARTHAG)), "Arthag wog prüfend seine Streitaxt.") + + def test_only_foreign_quotes_fall_back_to_own_text(self): + sheet = dict(self.ARTHAG, sources=[self.ARTHAG["sources"][0], + {"quote": "Marcian: »Und wer meldet sich?«", "line_hint": "action"}]) + self.assertEqual(conv._character_scene_evidence(sheet), sheet["mannerisms"]) + + def test_real_alias_and_genitive_count(self): + sheet = {"name": "Zerwas", "aliases": "Murlok von Mengbilla, der Vampir", + "sources": [{"quote": "Murlok von Mengbilla lächelte und hob den Becher.", "line_hint": "action"}]} + self.assertIn("Murlok", conv._character_scene_evidence(sheet)) + self.assertTrue(cr.quote_mentions_identity("Marrads Augen funkelten.", "Marrad", {})) + self.assertFalse(cr.quote_mentions_identity("Der Vampir lächelte.", "Zerwas", sheet)) + + def test_name_argument_used_when_sheet_has_no_name(self): + sheet = {k: v for k, v in self.ARTHAG.items() if k != "name"} + self.assertEqual(conv._character_scene_evidence(sheet, "Arthag"), "Arthag wog prüfend seine Streitaxt.") + + +class AgeCategoryTests(unittest.TestCase): + MARRAD = "Junges Mädchen / Junger Mann (im Text als Junge beschrieben)" + + def test_marrad_german_english_and_quote(self): + self.assertEqual(conv._character_age_category({"age_estimate": self.MARRAD}), "teen") + self.assertEqual(conv._character_age_category( + {"age_estimate": "Young girl / young man (described in the text as a boy)"}), "teen") + self.assertEqual(conv._character_age_category({ + "age_estimate": self.MARRAD, + "sources": [{"quote": "Fast acht Sommer, Kommandant.", "line_hint": "age_estimate"}]}), "child") + self.assertEqual(conv._age_body_lock({"age_estimate": self.MARRAD})[0], "EXACTLY ONE TEENAGE PERSON.") + + def test_adults_stay_adult(self): + for value in ("Junger Mann", "young man", "Junge Frau", "Erwachsen", "jung", + "Mittleres Alter (mindestens 15 Jahre älter als in seiner Akademiezeit)", + "jung bis mittelalterlich (ca. 25-40 Jahre)"): + self.assertEqual(conv._character_age_category({"age_estimate": value}), "adult", value) + # A boy noun in prose may be someone else; only label fields count. + self.assertEqual(conv._character_age_category({"physical": "trägt einen Jungen auf den Schultern"}), "adult") + # A non-age source hint never feeds the age. + self.assertEqual(conv._character_age_category({ + "age_estimate": "Erwachsen", "sources": [{"quote": "Acht Sommer diente er.", "line_hint": "image"}]}), + "adult") + + def test_number_words_and_seasons(self): + self.assertEqual(conv._character_age_category({"age_estimate": "zwölfjährig"}), "child") + self.assertEqual(conv._character_age_category({"age_estimate": "etwa fünfzehn Winter"}), "teen") + self.assertEqual(conv._character_age_category({"age_estimate": "about eight summers"}), "child") + self.assertEqual(conv._character_age_category({"age_estimate": "Mädchen"}), "teen") + self.assertEqual(conv._character_age_category({"age_estimate": "girl"}), "teen") + + +def _make_db(path: Path) -> None: + con = sqlite3.connect(path) + con.execute("CREATE TABLE characters (id TEXT PRIMARY KEY, book TEXT, name TEXT, tags TEXT, voice TEXT, " + "image TEXT, sheet TEXT, analysis TEXT, created TEXT, updated TEXT)") + book = "Buch 02" + copied = {"race_species": "Ork (vermutlich, da im Lager und als Druide)", + "relationships": "Uigar Kai (dient ihm, hält den Steigbügel)", "physical": "rothaarig, erfrorenes Gesicht", + "sources": [{"page": 188, "quote": "Langsam richtete sich der Fremde aus dem Wald auf.", "line_hint": "physical"}]} + rows = [ + ("Kolon", {"aliases": "Sharraz Garthai, Sharraz, Zwerg", "first_name": "Sharraz", "last_name": "Garthai", + "race_species": KOLON_SPECIES, "relationships": KOLON_RELATIONSHIPS, + "sources": [{"quote": "Kolon brummte wütend vor sich hin.", "line_hint": "mannerisms"}]}), + ("Arthag", {"aliases": "der Zwerg, Zwerg", "race_species": KOLON_SPECIES, "relationships": KOLON_RELATIONSHIPS, + "mannerisms": "Spuckt angewidert aus.", + "sources": [{"quote": "Arthag stand ratlos vor der Versammlung und rief laut.", "line_hint": "action"}]}), + ("Marrad", {"sources": [{"quote": "Arthag stand ratlos vor der Versammlung und rief laut.", "line_hint": "action"}, + {"quote": "Statt einer Antwort salutierte er.", "line_hint": "mannerisms"}]}), + ("Sharraz", {"race_species": "Ork"}), + ("Mann", dict(copied, aliases="Fremde, Fremder, Gutfolg")), + ("Fremder", dict(copied, aliases="Mann, Fremde, Gutfolg")), + ("Promos", {"aliases": "Alchimist", "physical": "alter Mann mit zitternden Händen und Brille"}), + ("Alchimist", {"aliases": "Promos", "physical": "kleiner Alter im fleckigen Kittel"}), + ] + for name, sheet in rows: + con.execute("INSERT INTO characters (id, book, name, sheet, created, updated) VALUES (?,?,?,?,?,?)", + (f"{book}::{name}".lower(), book, name, json.dumps(dict(sheet, name=name)), "2026-07-17", "2026-07-17")) + con.commit() + con.close() + + +def _sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +class AuditAndCleanupScriptTests(unittest.TestCase): + def setUp(self): + self.tmp = Path(tempfile.mkdtemp(prefix="ttvc-hygiene-db-")) + self.db = self.tmp / "chars.db" + _make_db(self.db) + + def test_audit_reports_and_never_writes(self): + before = _sha(self.db) + result = audit_script.audit(audit_script.load_records(str(self.db))) + self.assertEqual(_sha(self.db), before) + summary, issues = result["summary"], result["issues"] + pairs = {frozenset((d["a"], d["b"])) for d in issues["duplicated_sheets"]} + self.assertIn(frozenset(("Kolon", "Arthag")), pairs) + self.assertIn(frozenset(("Mann", "Fremder")), pairs) + self.assertIn(("Kolon", "Sharraz"), {(c["name"], c["value"]) for c in issues["alias_collisions"]}) + self.assertEqual({x["name"] for x in issues["thinking_leakage"]}, {"Kolon", "Arthag"}) + self.assertIn("Marrad", {x["name"] for x in issues["foreign_sources"]}) + self.assertEqual(summary["scene_quote_about_other"], 0) # the fixed picker skips them + self.assertGreaterEqual(summary["generic_alias_links"], 1) + + def test_cleanup_dry_run_writes_report_only(self): + before = _sha(self.db) + out = self.tmp / "dry" + self.assertEqual(clean_script.main([str(self.db), "--out", str(out)]), 0) + self.assertEqual(_sha(self.db), before) + report = json.loads((out / "report.json").read_text(encoding="utf-8")) + self.assertFalse(report["applied"]) + self.assertFalse((out / "backup.json").exists()) + kolon = next(p for p in report["plans"] if p["name"] == "Kolon") + kinds = {(c["kind"], c["field"]) for c in kolon["changes"]} + self.assertIn(("thinking_leakage", "race_species"), kinds) + self.assertIn(("foreign_identity", "aliases"), kinds) + promos = [p for p in report["plans"] if p["name"] == "Promos"] + self.assertTrue(all(c["kind"] == "review_identity" for p in promos for c in p["changes"])) + + def test_apply_needs_out_and_writes_backup_first(self): + self.assertEqual(clean_script.main([str(self.db), "--apply"]), 2) + out = self.tmp / "applied" + self.assertEqual(clean_script.main([str(self.db), "--apply", "--out", str(out)]), 0) + backup = json.loads((out / "backup.json").read_text(encoding="utf-8")) + self.assertIn("Kolon", {r["name"] for r in backup}) + con = sqlite3.connect(self.db) + sheets = {n: json.loads(s) for n, s in con.execute("SELECT name, sheet FROM characters")} + con.close() + self.assertEqual(sheets["Kolon"]["race_species"], "Mensch") + self.assertNotIn("Sharraz", sheets["Kolon"]["aliases"]) + self.assertEqual(sheets["Promos"]["aliases"], "Alchimist") # review-only, untouched + self.assertEqual(sheets["Mann"], json.loads(json.dumps(sheets["Mann"]))) + self.assertEqual(len(sheets["Marrad"]["sources"]), 2) # sources only with the opt-in flag + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_prompt_translation.py b/tests/test_prompt_translation.py index d52bdf7..4366885 100644 --- a/tests/test_prompt_translation.py +++ b/tests/test_prompt_translation.py @@ -218,13 +218,15 @@ class ConversationWiringTests(unittest.TestCase): self.assertEqual(sheet_en["physical"], BOY["physical"]) def test_scene_evidence_and_book_profile_translated(self): - table = {"Der Zwerg hob die Axt und rief laut.": "The dwarf raised the axe and shouted loudly.", + # The quote must name the character: a quote that does not is never + # picked (see tests/test_character_record_hygiene.py). + table = {"Kolon hob die Axt und rief laut.": "Kolon raised the axe and shouted loudly.", "Ein dunkler Wald im Norden": "A dark forest in the north"} - sheet = {"sources": [{"quote": "Der Zwerg hob die Axt und rief laut.", "line_hint": "action"}]} + sheet = {"sources": [{"quote": "Kolon hob die Axt und rief laut.", "line_hint": "action"}]} sheet_en, book_en, _, _ = conv._english_prompt_inputs( "Kolon", sheet, {"setting": "Ein dunkler Wald im Norden"}, None, {"llm_url": ""}, chat=FakeChat(table=table), cache=pt.MemoryCache()) - self.assertEqual(sheet_en["visual_scene"], "The dwarf raised the axe and shouted loudly.") + self.assertEqual(sheet_en["visual_scene"], "Kolon raised the axe and shouted loudly.") self.assertEqual(book_en["setting"], "A dark forest in the north")