mARTin-B78/tts-voice-creator-clone-and-design-2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(characters): automatic vision check for generated character images

Browse Source 
The model only reports observations; expectations come from the sheet and the
comparison is in code. Fail-safe: an unreachable checker or unreadable answer
is a failure. Calibrated 14/14 on panels with known defects using the already
loaded LLM; Ling 3.0 Flash VL (free) scored 8/14 with 6 unreadable answers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
mARTin-B78 2026-09-21 10:23:42 +02:00
parent 16df95ec97
commit ac8a6c7a24
5 changed files with 452 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -8,6 +8,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
## [Unreleased]
### Added
- **Automatic check for generated character images** (`POST /api/character-image-check`). A vision model reports what it actually sees in a portrait, turnaround or scene — figure count, which way each figure faces, whether heads and feet are in frame, the apparent gender, pointed or round ears, wounds, plain or scenery background, and whether two images show the same person — and the app compares that with the character sheet. An unavailable checker or an unreadable answer counts as a failure, never a pass. It uses the LLM that is already loaded (no model swapping); calibrated at 14/14 on panels with known defects.
- **LiteLLM model-group aliases in the model list.** The Active Language Model / OCR model dropdown now also lists the gateway's router aliases (ocr, default, cheap, vision, local-stt …), which LiteLLM's `/models` omits; they are read from `/router/settings` and shown first. Gateways without that endpoint are unaffected.
### Changed

31
architecture/character-visual-identity.md
View File

@ -64,3 +64,34 @@ active changes only that form.
* **The turnaround stays one T2I triptych.** Drawing front/side/rear one by one with the edit model was tried (2026-09-20) and reverted the next day: it returned three near-identical front views, mixed rendering styles between panels, cropped bodies, and copied the wireframe pose guide into the costume — exactly the failure the triptych was introduced to fix. The portrait/sheet face mismatch is still open and needs a different approach.
* **Props** get an object-only period sentence, so a tile no longer turns into a portrait of the character.
* Older saved prompts are cleaned the same way at render time. Existing image pairs are unchanged until regenerated.
## Automatic image check (`POST /api/character-image-check`)
A vision model reports **only what it sees** — figure count, the view of each figure left to right,
whether every head and foot is in frame, whether the figures are the same person, apparent gender,
ear shape (pointed/round/hidden), wounds, and background (plain/scenery). The expected values come from
the character sheet and the comparison happens in code (`core/character_image_check.py`). Asking a model
"is this a woman, as required?" invites agreement; asking "what gender is this?" does not, and the
expected gender is never revealed in the question.
* Kinds: `portrait`, `turnaround`, `scene`, `same_person` (profile vs. a sheet figure).
* **Fail-safe:** an unreachable checker, an unreadable answer, or a missing observation is a failure,
never a pass. An unreadable answer is returned verbatim (truncated) so it can be diagnosed.
* Thinking is switched off: a reasoning model spent the whole budget deliberating over a confusing image
and returned nothing.
* **Model:** the active LLM by default (`llm_model`), overridable with the `image_check_model` setting.
The model already loaded reads images, so a check never makes llama-swap unload it for a separate
vision model on a GPU with no room for both.
**Calibration (2026-09-21)** on seven panels whose defects were known (three front views, elf ears on a
human, a wireframe rendered as a figure, cut-off feet, a man where the sheet says woman, a forest behind a
studio portrait, and one correct same-person pair), run twice:
| model | correct | wrong | unreadable | per check |
|---|---|---|---|---|
| local Qwen3.8-27B (`default`) | 14/14 | 0 | 0 | ~4 s |
| Ling 3.0 Flash VL (`vision`, free, OpenRouter) | 8/14 | 0 | 6 | ~17 s |
Ling never judged wrongly but on every turnaround ignored the JSON-only instruction and answered in prose.
It also has free-tier rate limits and sends the images to a third party.

178
core/character_image_check.py Normal file
View File

@ -0,0 +1,178 @@
"""Automatic quality check for generated character images.
A vision model is asked only to *report what it sees* (how many figures, which way each one
faces, whether feet are in frame, the apparent gender, ...). The expected values come from the
character sheet and the comparison happens here, in code. Asking a model "is this a woman, as
required?" invites agreement; asking "what gender is this person?" does not.
Pure functions only: building the request, parsing the answer, and judging it. The route owns
the network call, so every piece here is testable without a model.
"""
from __future__ import annotations
import json
import re
PANEL_KINDS = ("portrait", "turnaround", "scene", "same_person")
# Observation fields the model must fill in, per panel kind. Wording is deliberately neutral.
_FIELDS = {
"portrait": {
"person_count": "integer: how many people are visible",
"apparent_gender": '"female", "male" or "unclear"',
"ears": '"pointed", "round" or "hidden" (hidden = covered by hair or headwear)',
"wounds_or_blood": "true if any fresh wound, cut, blood, bruise or bandage is visible",
"background": '"plain" (empty studio backdrop) or "scenery" (landscape, forest, room, sky)',
},
"turnaround": {
"figure_count": "integer: how many full or partial human figures are in the image",
"views_left_to_right": 'list, one entry per figure from left to right, each "front", "side", "back" or "three-quarter"',
"all_feet_visible": "true only if the feet of every figure are completely inside the image",
"all_heads_visible": "true only if the whole head of every figure is inside the image",
"same_person_in_all": "true if every figure is clearly the same person (face, hair, clothing)",
"apparent_gender": '"female", "male" or "unclear" (of the person shown)',
"ears": '"pointed", "round" or "hidden"',
"wounds_or_blood": "true if any fresh wound, cut, blood, bruise or bandage is visible",
"background": '"plain" (empty studio backdrop) or "scenery"',
},
"scene": {
"main_person_gender": '"female", "male" or "unclear": the most prominent person in the foreground',
"ears": '"pointed", "round" or "hidden" (of that person)',
"wounds_or_blood": "true if any fresh wound, cut, blood, bruise or bandage is visible",
},
"same_person": {
"same_person": "true if the two images clearly show the same individual (face shape, features, hair colour and style)",
},
}
_INSTRUCTIONS = (
"You inspect generated images for a production pipeline. Report only what is actually "
"visible; do not guess what was intended. Answer with ONE JSON object and nothing else, "
"with exactly these keys:\n"
)
def expected_from_sheet(sheet: dict, *, is_elf: bool, age_category: str) -> dict:
"""What a correct image must show, derived from the character sheet."""
gender = str((sheet or {}).get("gender") or "").strip().lower()
gender = {"weiblich": "female", "männlich": "male", "maennlich": "male"}.get(gender, gender)
return {
"gender": gender if gender in ("female", "male") else "",
"ears": "pointed" if is_elf else "round",
"age_category": age_category,
}
def build_messages(kind: str, image_uris: list[str]) -> list[dict]:
if kind not in _FIELDS:
raise ValueError(f"unknown panel kind: {kind}")
wanted = 2 if kind == "same_person" else 1
if len(image_uris) != wanted:
raise ValueError(f"{kind} needs {wanted} image(s), got {len(image_uris)}")
keys = "\n".join(f'- "{key}": {desc}' for key, desc in _FIELDS[kind].items())
lead = "Compare the two images." if kind == "same_person" else "Inspect the image."
content: list[dict] = [{"type": "text", "text": f"{lead}\n{_INSTRUCTIONS}{keys}"}]
content += [{"type": "image_url", "image_url": {"url": uri}} for uri in image_uris]
return [{"role": "user", "content": content}]
def parse_observation(raw: str, kind: str) -> dict:
"""Pull the JSON object out of the answer and keep only the expected keys."""
text = re.sub(r"<think>.*?</think>", "", raw or "", flags=re.DOTALL).strip()
fence = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
candidate = fence.group(1) if fence else text[text.find("{"): text.rfind("}") + 1]
try:
data = json.loads(candidate)
except (ValueError, TypeError):
return {}
if not isinstance(data, dict):
return {}
return {key: data[key] for key in _FIELDS.get(kind, {}) if key in data}
def _truthy(value) -> bool | None:
if isinstance(value, bool):
return value
if isinstance(value, str) and value.strip().lower() in ("true", "yes", "ja"):
return True
if isinstance(value, str) and value.strip().lower() in ("false", "no", "nein"):
return False
return None
def judge(kind: str, observed: dict, expected: dict) -> dict:
"""Compare observations with expectations. Returns {passed, failures:[...], observed}.
A missing or unreadable observation counts as a failure: an image we could not verify is
not an image we may call good.
"""
failures: list[str] = []
def need(key: str) -> object:
if key not in observed:
failures.append(f"{key}: not reported")
return None
return observed[key]
def check_gender(key: str) -> None:
value = need(key)
want = expected.get("gender")
if value is None or not want:
return
if str(value).strip().lower() != want:
failures.append(f"gender: shows {value}, sheet says {want}")
def check_ears() -> None:
value = str(need("ears") or "").strip().lower()
want = expected.get("ears")
if value == "pointed" and want == "round":
failures.append("ears: pointed, but the character is not an elf")
elif value == "round" and want == "pointed":
failures.append("ears: round, but the character is an elf")
def check_false(key: str, message: str) -> None:
value = _truthy(need(key))
if value is True:
failures.append(message)
def check_true(key: str, message: str) -> None:
value = _truthy(need(key))
if value is not True and key in observed:
failures.append(message)
if kind == "portrait":
count = need("person_count")
if count is not None and str(count).strip() != "1":
failures.append(f"person_count: {count}, expected exactly one person")
check_gender("apparent_gender")
check_ears()
check_false("wounds_or_blood", "wounds or blood visible")
if str(need("background") or "").strip().lower() == "scenery":
failures.append("background: scenery instead of a plain studio backdrop")
elif kind == "turnaround":
count = need("figure_count")
if count is not None and str(count).strip() != "3":
failures.append(f"figure_count: {count}, expected three")
views = need("views_left_to_right")
if views is not None:
normalised = [str(v).strip().lower() for v in views] if isinstance(views, list) else []
if normalised != ["front", "side", "back"]:
failures.append(f"views: {normalised or views}, expected front, side, back")
check_true("all_feet_visible", "feet cut off")
check_true("all_heads_visible", "head cut off")
check_true("same_person_in_all", "the figures are not the same person")
check_gender("apparent_gender")
check_ears()
check_false("wounds_or_blood", "wounds or blood visible")
if str(need("background") or "").strip().lower() == "scenery":
failures.append("background: scenery instead of a plain studio backdrop")
elif kind == "scene":
check_gender("main_person_gender")
check_ears()
check_false("wounds_or_blood", "wounds or blood visible")
elif kind == "same_person":
check_true("same_person", "the sheet shows a different person than the profile picture")
else:
failures.append(f"unknown panel kind: {kind}")
return {"passed": not failures, "failures": failures, "observed": observed}

81
routes/conversation.py
View File

@ -2641,6 +2641,87 @@ def _build_compact_character_render_prompts(name: str, book: str, sheet: dict,
return portrait, front_view, side_view, back_view, scene_prompt, prop_prompts
# ── Automatic image check ───────────────────────────────────────────────────
_IMAGE_CHECK_MAX_URI = 12_000_000 # a data URI; a 1536x1536 PNG is ~3-5 MB
def _image_check_expected(sheet: dict) -> dict:
from core.character_image_check import expected_from_sheet
sheet = _character_form_sheet(sheet or {})
return expected_from_sheet(
sheet,
is_elf=_character_ear_shape(sheet).startswith("pointed"),
age_category=_character_age_category(sheet),
)
def _image_check_run(kind: str, image_uris: list[str], sheet: dict, settings: dict,
model: str = "", timeout: float = 180) -> dict:
"""Ask the vision model what it sees, then judge that against the sheet. Never raises:
an unreachable or unreadable checker is reported as a failed check, not a pass."""
from core.character_image_check import build_messages, judge, parse_observation
llm_url = str(settings.get("llm_url") or "").rstrip("/")
# The active LLM by default: the one already loaded reads images, so a check never forces
# llama-swap to unload it for a separate vision model on a GPU that has no room for both.
model = model or str(settings.get("image_check_model") or settings.get("llm_model") or "")
expected = _image_check_expected(sheet)
# Thinking off: a reasoning model spent the whole budget deliberating over a confusing image
# (a wireframe in a costume) and returned no answer. Observation needs no deliberation.
payload = {
"model": model,
"messages": build_messages(kind, image_uris),
"temperature": 0,
"max_tokens": 800,
"chat_template_kwargs": {"enable_thinking": False},
}
headers = {"Authorization": f"Bearer {settings.get('llm_api_key') or 'sk-dummy-key'}"}
try:
resp = _post_llm_chat_completion(llm_url, payload, headers, timeout)
if resp.status_code in (400, 422): # a backend that rejects the (Qwen-style) thinking switch
payload = {k: v for k, v in payload.items() if k != "chat_template_kwargs"}
resp = _post_llm_chat_completion(llm_url, payload, headers, timeout)
if resp.status_code >= 400:
raise RuntimeError(f"HTTP {resp.status_code}: {_response_error_text(resp)[:160]}")
msg = resp.json()["choices"][0]["message"]
raw = (msg.get("content") or _reasoning_text(msg) or "").strip()
except Exception as exc:
return {"passed": False, "failures": [f"checker unavailable: {exc}"], "observed": {},
"expected": expected, "model": model}
observed = parse_observation(raw, kind)
verdict = judge(kind, observed, expected)
if not observed:
# Say what came back, so an unreadable answer is diagnosable rather than just "not reported".
verdict["failures"] = [f"checker gave an unreadable answer: {raw[:300]!r}"] + verdict["failures"]
return dict(verdict, expected=expected, model=model)
@router.post("/api/character-image-check")
async def character_image_check(request: Request):
"""Check one generated panel against the character sheet.
Body: {kind: "portrait"|"turnaround"|"scene"|"same_person", images: [data URI, ...],
sheet: {...}, model?: str}
Returns: {passed, failures:[str], observed:{...}, expected:{...}, model}
"""
from core.character_image_check import PANEL_KINDS
data = await request.json()
if not isinstance(data, dict):
raise HTTPException(400, "Body must be an object")
kind = str(data.get("kind") or "")
if kind not in PANEL_KINDS:
raise HTTPException(400, f"kind must be one of {', '.join(PANEL_KINDS)}")
images = data.get("images")
if not isinstance(images, list) or not images or not all(
isinstance(i, str) and i.startswith("data:image/") and len(i) <= _IMAGE_CHECK_MAX_URI for i in images
):
raise HTTPException(400, "images must be a list of image data URIs")
if len(images) != (2 if kind == "same_person" else 1):
raise HTTPException(400, f"{kind} takes {2 if kind == 'same_person' else 1} image(s)")
sheet = data.get("sheet") if isinstance(data.get("sheet"), dict) else {}
model = str(data.get("model") or "").strip()[:200]
return await asyncio.to_thread(_image_check_run, kind, images, sheet, _load_settings(), model)
@router.post("/api/character-generate-prompts")
async def character_generate_prompts(request: Request):
"""Turn an already-extracted character sheet into four ready-to-use external

161
tests/test_character_image_check.py Normal file
View File

@ -0,0 +1,161 @@
"""Automatic character image check: request shape, parsing, judging, and the route."""
from __future__ import annotations
import os
import tempfile
import unittest
from unittest import mock
os.environ.setdefault("CONFIG_DIR", tempfile.mkdtemp(prefix="ttvc-imgcheck-test-config-"))
from fastapi import FastAPI
from fastapi.testclient import TestClient
from core.character_image_check import build_messages, expected_from_sheet, judge, parse_observation
import routes.conversation as conversation
PNG = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
WOMAN = expected_from_sheet({"gender": "female"}, is_elf=False, age_category="adult")
GOOD_TURNAROUND = {
"figure_count": 3, "views_left_to_right": ["front", "side", "back"],
"all_feet_visible": True, "all_heads_visible": True, "same_person_in_all": True,
"apparent_gender": "female", "ears": "round", "wounds_or_blood": False, "background": "plain",
}
class RequestTests(unittest.TestCase):
def test_the_question_asks_for_observations_not_for_agreement(self):
text = build_messages("turnaround", [PNG])[0]["content"][0]["text"]
self.assertIn("views_left_to_right", text)
self.assertIn("Report only what is actually visible", text)
self.assertNotIn("female", text.split("keys:")[0]) # the expected gender is never revealed
def test_same_person_sends_both_images(self):
parts = build_messages("same_person", [PNG, PNG])[0]["content"]
self.assertEqual(sum(1 for p in parts if p["type"] == "image_url"), 2)
def test_wrong_image_counts_are_rejected(self):
with self.assertRaises(ValueError):
build_messages("same_person", [PNG])
with self.assertRaises(ValueError):
build_messages("portrait", [PNG, PNG])
class ParseTests(unittest.TestCase):
def test_reads_fenced_json_and_drops_unknown_keys(self):
raw = '<think>hmm</think>```json\n{"apparent_gender":"male","extra":1}\n```'
self.assertEqual(parse_observation(raw, "portrait"), {"apparent_gender": "male"})
def test_garbage_gives_an_empty_observation(self):
self.assertEqual(parse_observation("I cannot tell.", "portrait"), {})
class JudgeTests(unittest.TestCase):
def test_a_correct_turnaround_passes(self):
self.assertTrue(judge("turnaround", GOOD_TURNAROUND, WOMAN)["passed"])
def test_three_front_views_fail(self):
out = judge("turnaround", dict(GOOD_TURNAROUND, views_left_to_right=["front", "front", "front"]), WOMAN)
self.assertFalse(out["passed"])
self.assertTrue(any("views" in f for f in out["failures"]))
def test_cut_off_feet_fail(self):
out = judge("turnaround", dict(GOOD_TURNAROUND, all_feet_visible=False), WOMAN)
self.assertIn("feet cut off", out["failures"])
def test_a_man_where_the_sheet_says_woman_fails(self):
out = judge("scene", {"main_person_gender": "male", "ears": "round", "wounds_or_blood": False}, WOMAN)
self.assertEqual(out["failures"], ["gender: shows male, sheet says female"])
def test_elf_ears_on_a_human_fail_but_an_elf_needs_them(self):
self.assertFalse(judge("portrait", {"person_count": 1, "apparent_gender": "female", "ears": "pointed",
"wounds_or_blood": False, "background": "plain"}, WOMAN)["passed"])
elf = expected_from_sheet({"gender": "female"}, is_elf=True, age_category="adult")
self.assertTrue(judge("portrait", {"person_count": 1, "apparent_gender": "female", "ears": "pointed",
"wounds_or_blood": False, "background": "plain"}, elf)["passed"])
def test_hidden_ears_are_not_held_against_the_image(self):
self.assertTrue(judge("portrait", {"person_count": 1, "apparent_gender": "female", "ears": "hidden",
"wounds_or_blood": False, "background": "plain"}, WOMAN)["passed"])
def test_scenery_behind_a_studio_portrait_fails(self):
out = judge("portrait", {"person_count": 1, "apparent_gender": "female", "ears": "round",
"wounds_or_blood": False, "background": "scenery"}, WOMAN)
self.assertFalse(out["passed"])
def test_wounds_fail(self):
out = judge("turnaround", dict(GOOD_TURNAROUND, wounds_or_blood=True), WOMAN)
self.assertIn("wounds or blood visible", out["failures"])
def test_an_observation_the_model_did_not_give_is_a_failure(self):
self.assertFalse(judge("turnaround", {}, WOMAN)["passed"])
def test_no_gender_on_the_sheet_means_gender_is_not_judged(self):
unknown = expected_from_sheet({}, is_elf=False, age_category="adult")
out = judge("scene", {"main_person_gender": "male", "ears": "round", "wounds_or_blood": False}, unknown)
self.assertTrue(out["passed"])
def test_a_different_person_from_the_profile_fails(self):
self.assertFalse(judge("same_person", {"same_person": False}, WOMAN)["passed"])
class RouteTests(unittest.TestCase):
def setUp(self):
app = FastAPI()
app.include_router(conversation.router)
self.client = TestClient(app)
def answer(self, content, status=200):
r = mock.Mock(); r.status_code = status
r.json.return_value = {"choices": [{"message": {"content": content}}]}
r.text = content
return r
def test_judges_the_models_observation_against_the_sheet(self):
with mock.patch.object(conversation, "_post_llm_chat_completion",
return_value=self.answer('{"main_person_gender":"male","ears":"round","wounds_or_blood":false}')):
out = self.client.post("/api/character-image-check",
json={"kind": "scene", "images": [PNG], "sheet": {"gender": "female"}}).json()
self.assertFalse(out["passed"])
self.assertEqual(out["failures"], ["gender: shows male, sheet says female"])
self.assertEqual(out["expected"]["gender"], "female")
def test_a_hedged_species_is_judged_as_human(self):
sheet = {"gender": "female", "race_species": "Mensch (oder Halbelfe, je nach Kontext, hier nicht spezifiziert)"}
with mock.patch.object(conversation, "_post_llm_chat_completion",
return_value=self.answer('{"main_person_gender":"female","ears":"pointed","wounds_or_blood":false}')):
out = self.client.post("/api/character-image-check",
json={"kind": "scene", "images": [PNG], "sheet": sheet}).json()
self.assertEqual(out["expected"]["ears"], "round")
self.assertIn("ears: pointed, but the character is not an elf", out["failures"])
def test_thinking_is_switched_off_and_dropped_if_the_backend_refuses_it(self):
good = self.answer('{"main_person_gender":"female","ears":"round","wounds_or_blood":false}')
with mock.patch.object(conversation, "_post_llm_chat_completion",
side_effect=[self.answer("unsupported", status=400), good]) as llm:
out = self.client.post("/api/character-image-check",
json={"kind": "scene", "images": [PNG], "sheet": {"gender": "female"}}).json()
self.assertTrue(out["passed"])
self.assertEqual(llm.call_args_list[0].args[1]["chat_template_kwargs"], {"enable_thinking": False})
self.assertNotIn("chat_template_kwargs", llm.call_args_list[1].args[1])
def test_an_unreachable_checker_is_a_failure_not_a_pass(self):
with mock.patch.object(conversation, "_post_llm_chat_completion", side_effect=RuntimeError("down")):
out = self.client.post("/api/character-image-check",
json={"kind": "portrait", "images": [PNG], "sheet": {}}).json()
self.assertFalse(out["passed"])
self.assertTrue(out["failures"][0].startswith("checker unavailable"))
def test_bad_requests_are_rejected_before_any_model_call(self):
with mock.patch.object(conversation, "_post_llm_chat_completion") as llm:
for body in ({"kind": "nope", "images": [PNG]},
{"kind": "portrait", "images": ["http://evil/x.png"]},
{"kind": "portrait", "images": []},
{"kind": "same_person", "images": [PNG]}):
with self.subTest(body=str(body)[:50]):
self.assertEqual(self.client.post("/api/character-image-check", json=body).status_code, 400)
llm.assert_not_called()
if __name__ == "__main__":
unittest.main()