tts-voice-creator-clone-and-design-2/architecture/character-extraction.md

# Character extraction, identity merging and record hygiene

Status: 2026-10-03. Companion to `character-visual-identity.md` (how a saved sheet
becomes image prompts). This document covers how sheets are extracted from the book,
how they are merged into library records, and the rules that keep one character's
data out of another's.

## Pipeline

1. **Cast roster** (Identify Characters / audiobook casting) names the speakers.
2. **Extraction** (`static/js/character-sheets.js` `csGenerate`): the book is cut into
   ~4000-character passages; each passage goes to `POST /api/character-sheets`
   (`routes/conversation.py` `_charsheets_prepare` / `_charsheets_parse`) together with
   the roster and a short summary of the sheets so far. One LLM call returns sheets for
   **every** character the passage reveals (batch per passage, not per character).
   - *Cast-target mode* (roster known, `csForReader`): only sheets whose name exactly
     equals a roster name are kept and merged by exact name (`strictRoster`).
   - *Discovery mode* (no roster, `csForRehearser`): sheets are merged by identity
     (`csFindMergeKey`), then `csConsolidateAliasDuplicates` folds alias-linked entries.
   - Field merge is **"longer text wins"** per field; sources (max 12) and inventory
     are unioned.
3. **Canonical names** (`csCanonicalizeRosterNames` / `csRosterCanonicalName`): the
   roster name is the display name; other names found by the LLM become aliases.
4. **Save** (`static/js/characters-library.js` `clUpsert`): a roster-authoritative sheet
   is written by its book+name key; other sheets look for an existing record first.
5. **Image prompts** read the saved record (`character-visual-identity.md`); the
   story-moment panel uses one quote from `sources` (`_character_scene_evidence`).

## Root causes found on the real library (270 records, 4 books)

Evidence from a read-only snapshot, the `config/regression_snapshots/*` exports of
2026-08-31 and the per-update backups in `config/backups/characters/`.

| Symptom | Cause (proven) |
|---|---|
| Arthag (book 02) has Kolon's species, relationships and motivation word for word; both records date from the 2026-07-17 batch and were already identical on 2026-08-31 | The extraction code of that time filtered target-mode sheets by **substring** roster match and merged by **any shared identity token** (title still counted). Merging is "longer text wins", so Kolon's long fields overwrote Arthag's. Today's target mode uses exact names, but discovery mode (`csFindMergeKey`, `csConsolidateAliasDuplicates`) and the library upsert (`clSameIdentity`) still linked records over a single shared token such as "Zwerg". |
| 16 records in book 02/03 (Zwerg, Kolon, Inquisitor, Soldat, Alter Soldat, Alte, Ritter, Navigator, Fremder, Alchimist, Kriegerin, Mädchen, ...) are complete copies, sources included, of another record (Arthag, Marcian, Veteran, Alrik, Mann, Promos, Nyrilla, Marina, ...) | Born in two bulk saves on 2026-08-16 (06:41 and 07:13, ~20 ms apart, two books), the day after `csRosterCanonicalName` shipped. The alias lists of each pair are the same list minus each record's own name, i.e. **one sheet was written under two names**. `csRosterCanonicalName` returned the *first roster entry* found among all of a sheet's tokens, not the sheet's own name, so Arthag (aliases "der Zwerg, Zwerg, ..., Kolon") became "Zwerg"/"Kolon" and the save created those records from Arthag's sheet. (The exact caller of that batch is not recoverable; the rename path is the only code that turns one sheet into another name.) |
| Kolon (b02) aliases "Sharraz Garthai, ..., Sharraz, Zwerg, Kriegshäuptling", first/last name Sharraz Garthai | The LLM, given a passage about both, wrote the other person's name into Kolon's sheet; its own relationships field says "dient ... Sharraz Garthai". Aliases were accepted verbatim; only first/last name had a roster check (client side). 87 records carry another record's name as alias or real name. |
| Mann/Fremder (b03), Fremde/Zerwas (KONTROLLE) share species text | Same as row 2 (copied sheet), see `duplicated_sheets` in the audit. |
| Story-moment quote about another cast member | `_character_scene_evidence` chose by line_hint and verb score only. Sources hold whatever the passage quoted, including other characters' lines (also inherited through the merges above). 16 picked quotes named only another cast member; 85 did not name the character at all. |
| "Warte: ...", "Nein, Kolon ist ...", "Der Text sagt ..." in species/backstory/notes (12 records) | Reasoning-model deliberation written into field values; nothing validated them. |
| Marrad (a boy, "fast acht Sommer") rendered as an adult; Fremder b01 ("mindestens 15 Jahre älter") as a teen | `_character_age_category` knew no boy/girl nouns ("Junge", "Mädchen", "boy", "girl"), no "Sommer/Winter" ages, and read "15 Jahre älter" as an age. |

What could **not** be proven without live LLM runs: how often today's prompt still
produces cross-character aliases or merged facts for a multi-character passage. To
test: run *Recast selected* for Kolon and Arthag on book 02 with the local model and
check that Kolon's aliases contain no roster name and Arthag's species is his own.

## Rules (implemented)

**Identity evidence** (`core/character_records.py`, mirrored in
`static/js/character-identity.js`, word lists kept equal by a unit test): a shared
token proves "same person" only if it is a real name. Generic = only common nouns,
roles, ranks, kinship, species, age/look adjectives or name particles ("Zwerg", "Mann",
"Fremder", "Alter Soldat", "Ork-General", "Bürgerwehroffizier", "von"), or starting with
an article ("der Zwerg", "der Tunneltreiber"). Used by `csFindMergeKey`,
`csConsolidateAliasDuplicates`, `csRosterCanonicalName` and `clSameIdentity`.

**Own name first:** `csFindMergeKey` targets an entry with exactly the sheet's name;
`csRosterCanonicalName` keeps the sheet's own name when it is a cast name; `clUpsert`
writes to the record with exactly this book+name before any alias scan; `csForReader`
never renames a saved sheet onto a cast name that has its own saved record.

**Per-character isolation at parse time** (`_charsheets_parse(raw, roster)`, roster
only in cast-target mode): aliases and first/last/full names that are another cast
member's name, or a name-like alias containing one ("Sharraz Garthai", "Schlange
Gamba"), are dropped; descriptions naming a relative ("Tochter des Promos") stay.
Every text field is cut before reasoning markers (`strip_thinking_leakage`).
Source quotes are kept: some name another character but are about this one.

**Story moment:** only a source quote that names the character (name, real name,
genitive, non-generic alias) is eligible; otherwise the character's own mannerisms /
capabilities text is used, never somebody else's quote.

**Age:** explicit age (digits or number words with Jahre/jährig/Sommer/Winter/years/
summers/winters; not "15 Jahre älter") from the age estimate, age-tagged source
quotes, labels, physical; then the estimate's evidence clause ("im Text als Junge
beschrieben" / "described in the text as a boy"); then child/teen words and boy/girl
nouns in label fields. A hedged "Junges Mädchen / Junger Mann" with a minor signal is a
minor. German "Junge Frau"/"Junger Mann" are adults. Works on German and translated
English text.

## Audit and cleanup (SOP)

- `python3 scripts/audit_character_records.py [DB] [--book X] [--details|--json]`:
  read-only (`mode=ro`, `query_only`). Reports duplicated sheets, alias collisions,
  generic-only alias links, thinking leakage, foreign source quotes and scene quotes
  about another cast member. Run on a copy when the app is busy.
- `python3 scripts/clean_character_records.py [DB] --out DIR`: **dry run** (default),
  writes `report.json` / `report.md`, database untouched. Plans: cut reasoning leakage;
  remove an alias / real name of another cast member only when the record's own
  relationships name that person as someone else; everything else is *review only*.
  `--drop-foreign-sources` additionally removes source quotes naming only another cast
  member (lossy, off by default). Duplicated sheets are reported, never changed.
- `--apply --out DIR` (owner approval only) writes `backup.json` with every affected row
  first, then updates those rows in one transaction.
- **Rollback:** for each entry in `backup.json`, write its `sheet` and `updated` back:
  `UPDATE characters SET sheet=?, updated=? WHERE id=?` (or restore the DB backup taken
  before applying). Stop the container or make sure no recast runs meanwhile.

Snapshot of 2026-10-03 (before): 33 duplicated-sheet pairs, 161 alias collisions in 87
records, 61 generic-only alias links, 12 records with leakage, 139 foreign source quotes
in 89 records, 16 scene quotes about another cast member (0 with the fixed picker).
Dry run: 12 records would change (12 leakage cuts, 6 evidenced identity removals), 70
identity collisions listed for review.