diff --git a/CHANGELOG.md b/CHANGELOG.md index b565670..e25dea1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi --- ## [Unreleased] +### Changed +- Conversation Playground: the metrics (Token Usage and STT / LLM first token / TTS first audio TTFB charts) now sit in the right-hand Latency column below Turn history instead of a second "Metrics" tab; the Conversation | Metrics tablist is removed. Charts gain axis titles ("Time (ms)", "Time"), a legend and three y-ticks. + ### Added - Conversation Playground: Metrics view (Conversation | Metrics tabs) with Pipecat-style TTFB line charts for STT, LLM first token and TTS, plus session Token Usage (prompt/completion/total, "n/a" when the backend reports none). The stream now requests `stream_options.include_usage` (dropped and retried on 400/422) and forwards `usage` in the final stats event; the TTS value is now the time to the first audio chunk instead of including the remaining LLM generation time. - Conversation Playground: a Thinking on/off switch next to the Language Model (default off, setting `conversation_thinking`). Off sends `chat_template_kwargs.enable_thinking=false` (plus `think:false` for Ollama) with each reply request and never touches the system prompt; the empty-reply error now says to turn Thinking off. diff --git a/Manual.md b/Manual.md index a72d673..1df28f3 100644 --- a/Manual.md +++ b/Manual.md @@ -878,19 +878,19 @@ offline cache yet. **Privacy:** the setting is stored in the local app settings only; nothing extra is sent anywhere. -## Conversation Playground: Metrics view +## Conversation Playground: Metrics (Latency column) **Purpose:** see how fast each stage responds over a session, like the Pipecat Playground "Metrics" tab: Token Usage (prompt / completion / total) and a TTFB line chart each for STT, LLM (first token) and TTS (time to the first audio chunk, not the end of playback). **Prerequisites:** at least one completed turn in Conversation. Token counts need a backend that reports `usage` in streaming replies (Ollama, llama.cpp, vLLM and llama-swap do). -**Usage:** switch the Conversation | Metrics tabs at the top (click, tap, or Left/Right/Home/End arrows). Each completed turn adds one point (X = time of day HH:MM:SS, Y = ms). Use "Show as table" under a chart for the numbers. The Clear button resets charts and token totals. +**Usage:** scroll the right-hand Latency column below "Turn history" to Token Usage and the three chart boxes (STT, LLM first token, TTS first audio). Each completed turn adds one point (X = time of day HH:MM:SS, Y = "Time (ms)", legend "TTFB (STT|LLM|TTS)"). Use "Show as table" under a chart for the numbers. The Clear button resets charts and token totals. -**Expected result:** three charts with one point per turn (last 50 kept) and summed token counts. Voice turns include STT; typed turns have no STT point. The Latency panel in the Conversation tab is unchanged. +**Expected result:** three charts with one point per turn (last 50 kept) and summed token counts. Voice turns include STT; typed turns have no STT point. The Latency panel and Turn history above are unchanged. **Errors:** a backend that sends no usage shows "n/a" (never 0). Failed turns add no point. With no turns yet, "No turns yet." is shown. -**Mobile / accessibility:** charts stack on phones; 44 px tabs (tablist, aria-selected); every chart has a text summary (e.g. "LLM first token, last 142 ms, 5 points") and a table alternative; values are drawn as line plus markers, not colour only; no animation. +**Mobile / accessibility:** on phones the column stacks below the chat and the chart boxes stay full width; every chart has a text summary (e.g. "LLM first token, last 142 ms, 5 points") and a table alternative; values are drawn as line plus markers with a distinct colour per chart, never colour only; no animation. **Offline:** works with a local LLM; no network beyond your configured endpoint. diff --git a/PROJECT_MAP.md b/PROJECT_MAP.md index 089f74f..777c819 100644 --- a/PROJECT_MAP.md +++ b/PROJECT_MAP.md @@ -58,7 +58,7 @@ Two self-hosted Docker instances, both port 7890: | Voice library / clone / design | `routes/library.py`, `core/voice.py` | `library.js` | mature | | Voice archetype gallery (40 own German-first design voices, cached previews) | `core/voice_archetypes.py`, `routes/archetypes.py` | `voice-archetypes.js`, `s-design.html` | **new** — see [architecture/voice-archetypes.md](architecture/voice-archetypes.md) | | Read Aloud (books, PDFs) | `routes/reader.py` | `reader.js` | mature | -| Conversation Metrics view (TTFB charts STT/LLM/TTS, token usage) | `routes/conversation.py` (`_usage_from_chunk`, `_post_llm_stream`) | `conversation.js`, `s-conversation.html` | **new 2026-10-06** — in-memory, last 50 turns | +| Conversation metrics in the Latency column (TTFB charts STT/LLM/TTS, token usage) | `routes/conversation.py` (`_usage_from_chunk`, `_post_llm_stream`) | `conversation.js`, `s-conversation.html` | **new 2026-10-06** — in-memory, last 50 turns | | Audiobook casting (span mode: deterministic quote spans, LLM names speakers only; verify/recast) | `routes/conversation.py`, `core/attribution_spans.py` | `audiobook*.js`, `audiobook-spans.js`, `audiobook-cast-spans.js` | **new 2026-09-19** — see [architecture/audiobook-attribution.md](architecture/audiobook-attribution.md) | | Characters (avatar/concept identity, appearance forms) | `routes/characters.py`, `routes/conversation.py` | `library-characters.js`, `character-sheets.js` | **reworked 2026-09-19** — see [architecture/character-visual-identity.md](architecture/character-visual-identity.md) | | Rehearser | `routes/rehearsals_db.py` | `rehearser-*.js` | mature | diff --git a/static/js/conversation.js b/static/js/conversation.js index 471021c..5de0483 100644 --- a/static/js/conversation.js +++ b/static/js/conversation.js @@ -573,7 +573,7 @@ $('s-import-voices-file')?.addEventListener('change', async function () { } - // ── Metrics view (TTFB charts + token usage; in-memory, last 50 turns) ──── + // ── Metrics (latency column; TTFB charts + token usage; in-memory, last 50 turns) ──── const METRIC_MAX = 50; let metricTurns = []; // {ts, stt, llm, tts} (ms, null when not measured) let metricUsage = null; // {p, c, t} summed over turns whose server reported usage @@ -595,22 +595,25 @@ $('s-import-voices-file')?.addEventListener('change', async function () { renderMetrics(); } - function chartSvg(pts, label) { - const W = 300, H = 140, L = 40, R = 10, T = 8, B = 22; + function chartSvg(pts, label, k) { + const W = 300, H = 160, L = 46, R = 10, T = 8, B = 36; const max = Math.max(...pts.map(p => p.v), 1); const t0 = pts[0].ts, span = pts[pts.length - 1].ts - t0; const x = ts => L + (span ? (ts - t0) / span : 0.5) * (W - L - R); const y = v => T + (1 - v / max) * (H - T - B); const line = pts.map(p => `${x(p.ts).toFixed(1)},${y(p.v).toFixed(1)}`).join(' '); const dots = pts.map(p => ``).join(''); + const yt = [0, max / 2, max].map(v => `${Math.round(v)}`).join(''); return ` - ${max} - 0 - ${hms(t0)} - ${span ? `${hms(pts[pts.length - 1].ts)}` : ''} - ${dots}`; + ${yt} + ${hms(t0)} + ${span ? `${hms(pts[pts.length - 1].ts)}` : ''} + ${tr('Time')} + ${tr('Time (ms)')} + ● TTFB (${k.toUpperCase()}) + ${dots}`; } function renderMetrics() { @@ -624,29 +627,12 @@ $('s-import-voices-file')?.addEventListener('change', async function () { const summary = pts.length ? `${tr(what)}, ${tr('last')} ${pts[pts.length - 1].v} ms, ${pts.length} ${tr('points')}` : `${tr(what)}: ${tr('no data')}`; const rows = pts.map(p => `${hms(p.ts)}${p.v} ms`).join(''); return `

${title}

- ${pts.length ? chartSvg(pts, summary) : ''}

${summary}

+ ${pts.length ? chartSvg(pts, summary, k) : ''}

${summary}

${tr('Show as table')}${rows}
${summary}
${tr('Time')}TTFB
`; }).join(''); } - (function initMetricTabs() { - const tabs = [$('conv-tab-chat'), $('conv-tab-metrics')].filter(Boolean); - const panels = [$('conv-main-view'), $('conv-metrics-view')]; - if (tabs.length < 2) return; - function select(i, focus) { - tabs.forEach((tb, j) => { tb.setAttribute('aria-selected', String(i === j)); tb.tabIndex = i === j ? 0 : -1; if (panels[j]) panels[j].hidden = i !== j; }); - if (focus) tabs[i].focus(); - } - tabs.forEach((tb, i) => { - tb.addEventListener('click', () => select(i)); - tb.addEventListener('keydown', e => { - const k = { ArrowRight: (i + 1) % 2, ArrowLeft: (i + 1) % 2, Home: 0, End: 1 }[e.key]; - if (k != null) { e.preventDefault(); select(k, true); } - }); - }); - })(); - // ── Recording ──────────────────────────────────────────────────────────── function startRecTimer() { recStart = Date.now(); diff --git a/static/js/i18n.js b/static/js/i18n.js index db42306..854bcfe 100644 --- a/static/js/i18n.js +++ b/static/js/i18n.js @@ -95,9 +95,8 @@ const I18N_DICT = { 'on': 'an', 'off': 'aus', 'Off = faster answers. On = better reasoning but slower, and some models may return an empty reply.': 'Aus = schnellere Antworten. An = besseres Schlussfolgern, aber langsamer; manche Modelle liefern eine leere Antwort.', // Conversation metrics view - 'Metrics': 'Metriken', 'Conversation views': 'Unterhaltungsansichten', - 'Token Usage': 'Token-Verbrauch', 'TTFB Metrics': 'TTFB-Metriken', - 'Prompt tokens': 'Prompt-Token', 'Completion tokens': 'Completion-Token', 'Total tokens': 'Token gesamt', + 'Token Usage': 'Token-Verbrauch', 'Time (ms)': 'Zeit (ms)', + 'Prompt': 'Prompt', 'Completion': 'Completion', 'n/a': 'k. A.', 'LLM first token': 'LLM erstes Token', 'TTS first audio': 'TTS erstes Audio', 'last': 'zuletzt', 'points': 'Punkte', 'no data': 'keine Daten', 'Show as table': 'Als Tabelle anzeigen', 'Time': 'Zeit', // Inline text tags diff --git a/static/sections/s-conversation.html b/static/sections/s-conversation.html index b3f5bee..8396b01 100644 --- a/static/sections/s-conversation.html +++ b/static/sections/s-conversation.html @@ -6,11 +6,6 @@ -
- - -
-

Engines & prompt

@@ -58,7 +53,7 @@
-
+
@@ -147,22 +142,20 @@
No turns yet.
+ + +
Token Usage
+
+
Promptn/a
+
Completionn/a
+
Totaln/a
+
+

No turns yet.

+
- - - diff --git a/static/style.css b/static/style.css index 4c15937..89469b3 100644 --- a/static/style.css +++ b/static/style.css @@ -6041,27 +6041,22 @@ html[data-mag-night] #toast.success { color: var(--green); } html[data-mag-night .conv-llm-hint { flex-basis:100%; margin:0; font-size:12px; } @media (max-width: 640px) { #conv-llm-endpoint, #conv-llm-url { flex-basis:100%; width:100% !important; } } -/* Conversation: Conversation | Metrics tabs and the metrics view */ -.conv-tabs { display:flex; gap:4px; margin:0 0 12px; border-bottom:1px solid var(--border); } -.conv-tab { min-height:44px; min-width:44px; padding:0 16px; background:none; border:0; border-bottom:3px solid transparent; color:var(--subtext); font:inherit; font-weight:600; cursor:pointer; } -.conv-tab[aria-selected="true"] { color:var(--text); border-bottom-color:var(--accent); } -.conv-tab:focus-visible { outline:2px solid var(--accent); outline-offset:-2px; } -.conv-main[hidden], .conv-metrics[hidden] { display:none; } -.conv-metrics { margin-top:14px; } -.conv-metrics-h { margin:14px 0 8px; font-size:14px; } -.conv-tokens { display:grid; grid-template-columns:repeat(3, 1fr); gap:10px; } -.conv-token { background:var(--surface); border:1px solid var(--border); border-radius:var(--radius); padding:12px; display:flex; flex-direction:column; gap:4px; } -.conv-token-l { font-size:12px; color:var(--subtext); } -.conv-token strong { font-size:22px; font-variant-numeric:tabular-nums; } -.conv-charts { display:grid; grid-template-columns:repeat(3, 1fr); gap:10px; } -.conv-chart { background:var(--surface); border:1px solid var(--border); border-radius:var(--radius); padding:12px; min-width:0; } -.conv-chart h4 { margin:0 0 6px; font-size:13px; } +/* Conversation: metrics (token usage + TTFB charts) in the latency column */ +.conv-tokens { display:grid; grid-template-columns:repeat(3, 1fr); gap:6px; } +.conv-token { background:var(--bg); border:1px solid var(--border); border-radius:var(--radius); padding:6px 8px; display:flex; flex-direction:column; gap:2px; min-width:0; } +.conv-token-l { font-size:11px; color:var(--subtext); } +.conv-token strong { font-size:16px; font-variant-numeric:tabular-nums; } +.conv-charts { display:flex; flex-direction:column; gap:10px; } +.conv-chart { background:var(--bg); border:1px solid var(--border); border-radius:var(--radius); padding:8px; min-width:0; --cm-c:var(--accent); } +.conv-chart[data-metric="llm"] { --cm-c:#d97706; } +.conv-chart[data-metric="tts"] { --cm-c:#16a34a; } +.conv-chart h4 { margin:0 0 4px; font-size:13px; } .conv-chart svg { width:100%; height:auto; display:block; } .conv-chart .cm-axis { stroke:var(--border); } -.conv-chart .cm-line { stroke:var(--accent); fill:none; stroke-width:2; } -.conv-chart .cm-dot { fill:var(--surface); stroke:var(--accent); stroke-width:2; } +.conv-chart .cm-line { stroke:var(--cm-c); fill:none; stroke-width:2; stroke-linejoin:round; } +.conv-chart .cm-dot { fill:var(--surface); stroke:var(--cm-c); stroke-width:2; } .conv-chart text { fill:var(--subtext); font-size:10px; } +.conv-chart text.cm-legend { fill:var(--cm-c); } .conv-chart details summary { cursor:pointer; min-height:44px; display:flex; align-items:center; font-size:12px; color:var(--subtext); } .conv-chart table { width:100%; font-size:12px; border-collapse:collapse; font-variant-numeric:tabular-nums; } .conv-chart th, .conv-chart td { text-align:left; padding:2px 4px; border-bottom:1px solid var(--border); } -@media (max-width: 767px) { .conv-tokens, .conv-charts { grid-template-columns:1fr; } } diff --git a/tests/conversation-metrics.spec.js b/tests/conversation-metrics.spec.js index 18e3a61..eaf6c59 100644 --- a/tests/conversation-metrics.spec.js +++ b/tests/conversation-metrics.spec.js @@ -37,16 +37,19 @@ test('3 turns -> 3 charts with 3 points, token sums, latency panel intact', asyn for (const id of ['cpv-stt', 'cpv-ttft', 'cpv-llm', 'cpv-tts', 'cpv-total']) await expect(page.locator('#' + id)).not.toHaveText('—'); await expect(page.locator('#conv-stats-panel .conv-stats-head', { hasText: 'Turn history' })).toBeVisible(); - await page.getByRole('tab', { name: 'Metrics' }).click(); - await expect(page.locator('#conv-metrics-view')).toBeVisible(); - await expect(page.locator('#conv-main-view')).toBeHidden(); - await expect(page.locator('.conv-chart')).toHaveCount(3); + await expect(page.getByRole('tab')).toHaveCount(0); + await expect(page.locator('#conv-stats-panel .conv-chart')).toHaveCount(3); for (const k of ['stt', 'llm', 'tts']) { await expect(page.locator(`.conv-chart[data-metric="${k}"] circle`)).toHaveCount(3); await expect(page.locator(`.conv-chart[data-metric="${k}"] tbody tr`)).toHaveCount(3); } await expect(page.locator('.conv-chart[data-metric="llm"] svg')).toHaveAttribute('aria-label', 'LLM first token, last 230 ms, 3 points'); - await expect(page.locator('.conv-chart[data-metric="llm"] svg text').first()).toBeAttached(); + await expect(page.locator('.conv-chart[data-metric="llm"] svg')).toContainText('Time (ms)'); + await expect(page.locator('.conv-chart[data-metric="llm"] svg')).toContainText('TTFB (LLM)'); + const d = page.locator('.conv-chart[data-metric="stt"] details'); + await expect(d).not.toHaveAttribute('open', ''); + await d.locator('summary').click(); + await expect(d.locator('table')).toBeVisible(); await expect(page.locator('#cm-tok-prompt')).toHaveText('30'); await expect(page.locator('#cm-tok-completion')).toHaveText('15'); await expect(page.locator('#cm-tok-total')).toHaveText('45'); @@ -62,23 +65,13 @@ test('3 turns -> 3 charts with 3 points, token sums, latency panel intact', asyn test('no usage from the server shows n/a, never 0', async ({ page }) => { await boot(page, { usage: false }); await turn(page, 1); - await page.getByRole('tab', { name: 'Metrics' }).click(); for (const id of ['prompt', 'completion', 'total']) await expect(page.locator('#cm-tok-' + id)).toHaveText('n/a'); await expect(page.locator('.conv-chart[data-metric="llm"] circle')).toHaveCount(1); }); -test('empty state, keyboard toggle, 44px targets, German', async ({ page }) => { +test('empty state and German labels', async ({ page }) => { await boot(page, { lang: 'de' }); - const chat = page.locator('#conv-tab-chat'), met = page.locator('#conv-tab-metrics'); - await expect(chat).toHaveText('Unterhaltung'); - expect((await met.boundingBox())?.height).toBeGreaterThanOrEqual(44); - await chat.focus(); - await page.keyboard.press('ArrowRight'); - await expect(met).toHaveAttribute('aria-selected', 'true'); - await expect(met).toBeFocused(); await expect(page.locator('#cm-empty')).toBeVisible(); await expect(page.locator('#cm-tok-total')).toHaveText('k. A.'); - await page.keyboard.press('ArrowLeft'); - await expect(chat).toHaveAttribute('aria-selected', 'true'); - await expect(page.locator('#conv-main-view')).toBeVisible(); + await expect(page.locator('#conv-stats-panel .conv-stats-head', { hasText: 'Token-Verbrauch' })).toBeVisible(); });