tts-voice-creator-clone-and-design-2/architecture/character-extraction.md

Character extraction, identity merging and record hygiene

Status: 2026-10-03. Companion to character-visual-identity.md (how a saved sheet becomes image prompts). This document covers how sheets are extracted from the book, how they are merged into library records, and the rules that keep one character's data out of another's.

Pipeline

1. Cast roster (Identify Characters / audiobook casting) names the speakers.
2. Extraction (static/js/character-sheets.js csGenerate): the book is cut into ~4000-character passages; each passage goes to POST /api/character-sheets (routes/conversation.py _charsheets_prepare / _charsheets_parse) together with the roster and a short summary of the sheets so far. One LLM call returns sheets for every character the passage reveals (batch per passage, not per character).
   • Cast-target mode (roster known, csForReader): only sheets whose name exactly equals a roster name are kept and merged by exact name (strictRoster).
   • Discovery mode (no roster, csForRehearser): sheets are merged by identity (csFindMergeKey), then csConsolidateAliasDuplicates folds alias-linked entries.
   • Field merge is "longer text wins" per field; sources (max 12) and inventory are unioned.
3. Canonical names (csCanonicalizeRosterNames / csRosterCanonicalName): the roster name is the display name; other names found by the LLM become aliases.
4. Save (static/js/characters-library.js clUpsert): a roster-authoritative sheet is written by its book+name key; other sheets look for an existing record first.
5. Image prompts read the saved record (character-visual-identity.md); the story-moment panel uses one quote from sources (_character_scene_evidence).

Root causes found on the real library (270 records, 4 books)

Evidence from a read-only snapshot, the config/regression_snapshots/* exports of 2026-08-31 and the per-update backups in config/backups/characters/.

Symptom	Cause (proven)
Arthag (book 02) has Kolon's species, relationships and motivation word for word; both records date from the 2026-07-17 batch and were already identical on 2026-08-31	The extraction code of that time filtered target-mode sheets by substring roster match and merged by any shared identity token (title still counted). Merging is "longer text wins", so Kolon's long fields overwrote Arthag's. Today's target mode uses exact names, but discovery mode (csFindMergeKey, csConsolidateAliasDuplicates) and the library upsert (clSameIdentity) still linked records over a single shared token such as "Zwerg".
16 records in book 02/03 (Zwerg, Kolon, Inquisitor, Soldat, Alter Soldat, Alte, Ritter, Navigator, Fremder, Alchimist, Kriegerin, Mädchen, ...) are complete copies, sources included, of another record (Arthag, Marcian, Veteran, Alrik, Mann, Promos, Nyrilla, Marina, ...)	Born in two bulk saves on 2026-08-16 (06:41 and 07:13, ~20 ms apart, two books), the day after csRosterCanonicalName shipped. The alias lists of each pair are the same list minus each record's own name, i.e. one sheet was written under two names. csRosterCanonicalName returned the first roster entry found among all of a sheet's tokens, not the sheet's own name, so Arthag (aliases "der Zwerg, Zwerg, ..., Kolon") became "Zwerg"/"Kolon" and the save created those records from Arthag's sheet. (The exact caller of that batch is not recoverable; the rename path is the only code that turns one sheet into another name.)
Kolon (b02) aliases "Sharraz Garthai, ..., Sharraz, Zwerg, Kriegshäuptling", first/last name Sharraz Garthai	The LLM, given a passage about both, wrote the other person's name into Kolon's sheet; its own relationships field says "dient ... Sharraz Garthai". Aliases were accepted verbatim; only first/last name had a roster check (client side). 87 records carry another record's name as alias or real name.
Mann/Fremder (b03), Fremde/Zerwas (KONTROLLE) share species text	Same as row 2 (copied sheet), see duplicated_sheets in the audit.
Story-moment quote about another cast member	_character_scene_evidence chose by line_hint and verb score only. Sources hold whatever the passage quoted, including other characters' lines (also inherited through the merges above). 16 picked quotes named only another cast member; 85 did not name the character at all.
"Warte: ...", "Nein, Kolon ist ...", "Der Text sagt ..." in species/backstory/notes (12 records)	Reasoning-model deliberation written into field values; nothing validated them.
Marrad (a boy, "fast acht Sommer") rendered as an adult; Fremder b01 ("mindestens 15 Jahre älter") as a teen	_character_age_category knew no boy/girl nouns ("Junge", "Mädchen", "boy", "girl"), no "Sommer/Winter" ages, and read "15 Jahre älter" as an age.

What could not be proven without live LLM runs: how often today's prompt still produces cross-character aliases or merged facts for a multi-character passage. To test: run Recast selected for Kolon and Arthag on book 02 with the local model and check that Kolon's aliases contain no roster name and Arthag's species is his own.

Rules (implemented)

Identity evidence (core/character_records.py, mirrored in static/js/character-identity.js, word lists kept equal by a unit test): a shared token proves "same person" only if it is a real name. Generic = only common nouns, roles, ranks, kinship, species, age/look adjectives or name particles ("Zwerg", "Mann", "Fremder", "Alter Soldat", "Ork-General", "Bürgerwehroffizier", "von"), or starting with an article ("der Zwerg", "der Tunneltreiber"). Used by csFindMergeKey, csConsolidateAliasDuplicates, csRosterCanonicalName and clSameIdentity.

Own name first: csFindMergeKey targets an entry with exactly the sheet's name; csRosterCanonicalName keeps the sheet's own name when it is a cast name; clUpsert writes to the record with exactly this book+name before any alias scan; csForReader never renames a saved sheet onto a cast name that has its own saved record.

Per-character isolation at parse time (_charsheets_parse(raw, roster), roster only in cast-target mode): aliases and first/last/full names that are another cast member's name, or a name-like alias containing one ("Sharraz Garthai", "Schlange Gamba"), are dropped; descriptions naming a relative ("Tochter des Promos") stay. Every text field is cut before reasoning markers (strip_thinking_leakage). Source quotes are kept: some name another character but are about this one.

Story moment: only a source quote that names the character (name, real name, genitive, non-generic alias) is eligible; otherwise the character's own mannerisms / capabilities text is used, never somebody else's quote.

Age: explicit age (digits or number words with Jahre/jährig/Sommer/Winter/years/ summers/winters; not "15 Jahre älter") from the age estimate, age-tagged source quotes, labels, physical; then the estimate's evidence clause ("im Text als Junge beschrieben" / "described in the text as a boy"); then child/teen words and boy/girl nouns in label fields. A hedged "Junges Mädchen / Junger Mann" with a minor signal is a minor. German "Junge Frau"/"Junger Mann" are adults. Works on German and translated English text.

Audit and cleanup (SOP)

• python3 scripts/audit_character_records.py [DB] [--book X] [--details|--json]: read-only (mode=ro, query_only). Reports duplicated sheets, alias collisions, generic-only alias links, thinking leakage, foreign source quotes and scene quotes about another cast member. Run on a copy when the app is busy.
• python3 scripts/clean_character_records.py [DB] --out DIR: dry run (default), writes report.json / report.md, database untouched. Plans: cut reasoning leakage; remove an alias / real name of another cast member only when the record's own relationships name that person as someone else; everything else is review only. --drop-foreign-sources additionally removes source quotes naming only another cast member (lossy, off by default). Duplicated sheets are reported, never changed.
• --apply --out DIR (owner approval only) writes backup.json with every affected row first, then updates those rows in one transaction.
• Rollback: for each entry in backup.json, write its sheet and updated back: UPDATE characters SET sheet=?, updated=? WHERE id=? (or restore the DB backup taken before applying). Stop the container or make sure no recast runs meanwhile.

Snapshot of 2026-10-03 (before): 33 duplicated-sheet pairs, 161 alias collisions in 87 records, 61 generic-only alias links, 12 records with leakage, 139 foreign source quotes in 89 records, 16 scene quotes about another cast member (0 with the fixed picker). Dry run: 12 records would change (12 leakage cuts, 6 evidenced identity removals), 70 identity collisions listed for review.