From 9b169037a4960e84272bde7290d80b6b4306ffc1 Mon Sep 17 00:00:00 2001 From: mARTin-B78 Date: Sun, 4 Oct 2026 17:35:18 +0200 Subject: [PATCH] feat(tts): inline tags in generate, audiobook and magazine paths Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 2 +- Manual.md | 4 +- architecture/inline-tags.md | 10 ++- core/magazine.py | 5 ++ core/ssml_lite.py | 9 +++ routes/conversation.py | 11 ++- routes/tts.py | 31 +++++--- static/js/i18n.js | 4 + static/js/settings.js | 6 ++ static/sections/s-reader.html | 1 + static/sections/s-settings.html | 8 ++ static/sections/s-tryout.html | 1 + tests/inline-tags.spec.js | 44 ++++++++++ tests/test_inline_tags_paths.py | 137 ++++++++++++++++++++++++++++++++ 14 files changed, 256 insertions(+), 17 deletions(-) create mode 100644 tests/inline-tags.spec.js create mode 100644 tests/test_inline_tags_paths.py diff --git a/CHANGELOG.md b/CHANGELOG.md index c0a2892..f90fdcc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ## [Unreleased] ### Added - **Provenance tags on exported audio.** Magazine MP3 downloads, `audio.zip` and the MP3s inside the EPUB get ID3 tags (`synthetic_speech=true`, `generator`, `generator_version`, `generated_date`, comment; voice name only on opt-in) via one ffmpeg stream copy of a temp copy; stored audio is untouched. Per-export checkboxes on the magazine export row ("Mark as synthetic speech", "include voice name"), default from new setting `export_provenance_default` (on). `POST /api/export/provenance` and `python -m core.export_provenance file.mp3` read the tags back. Inaudible AudioSeal watermarking is researched (MIT code and weights, torch needed, weak message recovery after MP3/AAC) but only a stub `watermark(path)` that reports unavailable. Spoken disclosure is not implemented yet. Tests: `tests/test_watermark_tags.py`, `tests/magazine.spec.js`. See `architecture/watermarking.md`. -- **Inline text tags for TTS (setting `inline_tags_enabled`, default OFF).** `[pause 500ms]`/`[pause 1.2s]` (max 10 s each, 60 s per text), `[spell]USA[/spell]` (German letter names for German text), `[slow]..[/slow]` / `[fast]..[/fast]` (0.85 / 1.15 via ffmpeg atempo, innermost wins, unclosed ends at paragraph). Known control tags (e.g. `[laugh]`) are stripped, never spoken; `[1]` and `[sic]` stay. Only `POST /api/tts-preview` renders them; other paths are listed in `architecture/inline-tags.md`. Tests: `tests/test_ssml_lite.py`. +- **Inline text tags for TTS (setting `inline_tags_enabled`, default OFF).** `[pause 500ms]`/`[pause 1.2s]` (max 10 s each, 60 s per text), `[spell]USA[/spell]` (German letter names for German text), `[slow]..[/slow]` / `[fast]..[/fast]` (0.85 / 1.15 via ffmpeg atempo, innermost wins, unclosed ends at paragraph). Known control tags (e.g. `[laugh]`) are stripped, never spoken; `[1]` and `[sic]` stay. Rendered in `POST /api/tts-preview` (also used by audiobook line previews), `/v1/audio/speech` (non-streaming), `/speak` and the MCP speak tool through one shared `ssml_lite.render_with_tags`; stripped only (never spoken) in streaming, spoken chat replies and magazine MP3 (timings stay exact). Settings -> General switch (accessible, DE/EN) and a syntax hint under the Try It Out and Read Aloud text boxes. Tests: `tests/test_ssml_lite.py`, `tests/test_inline_tags_paths.py`, `tests/inline-tags.spec.js`. - **Optional vocal isolation for URL/YouTube imports (default OFF).** `POST /api/isolate-vocals {id}` runs the Demucs CLI (`htdemucs`, two stems) on a registered temp audio and registers the vocals as a new id; the original stays valid. Needs `VOCAL_ISOLATION=1`, a `demucs` binary (or `VOCAL_ISOLATION_CMD`) and the weights already on disk; the app never downloads anything and nothing is added to the main image. Limits: 15 min input, 15 min runtime (`VOCAL_ISOLATION_MAX_SECONDS`/`_TIMEOUT`), one job at a time, cancelled on client disconnect, temp files always removed. `GET /api/version` reports `features.vocal_isolation`; the "Remove background music" button (clone page URL card and voice library add-URL) appears only then, with an aria-live status and a play original / play without music toggle. Tests: `tests/test_vocal_isolation.py`, `tests/vocal-isolation.spec.js`. See `architecture/vocal-isolation.md`. - **Voice export/import bundles.** `GET /api/voices/{id}/export` streams a `.voice.zip` (audio, `reference.txt`, `meta.json` with seed, design prompt, rating/tags, `provenance`, `audio_problems`; optional picture and consent recording; `manifest.json` with format version, app version and sha256 per file; no temp files). `POST /api/voices/import` validates at the boundary: 150 MB upload cap, fixed flat member names only (zip-slip/path traversal rejected), per-file, total and compression-ratio caps, hashes verified, audio decoded, meta and provenance re-validated; a taken name gets a `_2` suffix, an identical voice is a no-op, nothing is overwritten. Voice library toolbar: "Export selected" and "Import bundle" (44px). Tests: `tests/test_voice_bundle.py`, `tests/voice-bundle.spec.js`. See `architecture/voice-bundle.md`. - **Consent/provenance field per voice.** Each voice gets `provenance` in its `.meta.json` (`basis`: own_voice, consented, licensed, public_domain or unknown; `source`, `licence`, `note`, optional `recorded_consent`, `set_at`). Existing voices count as `unknown`; `quick-import-voice` prefills the source URL (basis stays `unknown`). New `PUT /api/voices/{id}/provenance` (enum, length caps, control characters stripped, http(s) URLs only, no credentials) and `POST`/`GET /api/voices/{id}/consent-recording` (audio only, 10 MB cap, stored as `.consent`, moves/deletes with the voice). The voice inspector has a "Source & consent" section (typed note or MediaRecorder recording) and voices without a recorded basis show a "Source unknown" badge. Generation is not blocked. Data stays local. Tests: `tests/test_voice_provenance.py`, `tests/voice-provenance.spec.js`. diff --git a/Manual.md b/Manual.md index 2697c77..d8ef41b 100644 --- a/Manual.md +++ b/Manual.md @@ -9,7 +9,7 @@ Living manual. Every feature updates this file in the same commit as the code. --- ## Inline tags in the text (optional) -Purpose: pauses, spelled-out acronyms and slower/faster passages inside one text. Prerequisite: set `inline_tags_enabled` to true in the settings (default off; when off the tags are spoken literally). +Purpose: pauses, spelled-out acronyms and slower/faster passages inside one text. Prerequisite: switch on **Settings -> General -> Inline text tags** (default off; when off the tags are spoken literally). A syntax hint sits under the text boxes in Try It Out and Read Aloud. | Tag | Result | |---|---| @@ -17,7 +17,7 @@ Purpose: pauses, spelled-out acronyms and slower/faster passages inside one text | `[spell]USA[/spell]` | letters one by one (German letter names for German text) | | `[slow]..[/slow]` / `[fast]..[/fast]` | 0.85x / 1.15x speed, same pitch; innermost wins; unclosed ends at the paragraph | -Known control tags such as `[laugh]` are removed, never read; `[1]` and `[sic]` stay. Works with every engine in the Voice Generator preview/generate path (speed via ffmpeg, so no engine support needed). Not yet in the OpenAI-compatible proxy, streaming, chat or audiobook/magazine audio. Errors: a failing engine reports the usual TTS error. Mobile/accessibility/privacy: no UI change, text stays local. +Known control tags such as `[laugh]` are removed, never read; `[1]` and `[sic]` stay. Rendered (pauses, spell, speed) in: Try It Out/generate, Read Aloud and audiobook line previews (both call `/api/tts-preview`), the OpenAI-compatible `/v1/audio/speech` (non-streaming), and `/speak` and the MCP speak tool. Tags are only removed, never spoken, in: streaming playback and streaming `/v1/audio/speech`, spoken chat replies, and magazine MP3 (so recorded sentence timings stay exact). Speed uses ffmpeg, so no engine support is needed. Errors: a failing engine reports the usual TTS error. Mobile/accessibility: the switch is a labelled checkbox with `role=switch` (keyboard Space, 44 px row); German and English labels. Privacy: text stays local. ## Design a Voice: attribute chips diff --git a/architecture/inline-tags.md b/architecture/inline-tags.md index e665936..37f2ff0 100644 --- a/architecture/inline-tags.md +++ b/architecture/inline-tags.md @@ -18,6 +18,10 @@ Only these names are removed and never spoken (opening or closing, optional shor `parse(text, language)` -> segments `{text, speed, pause_ms}` (tags never reach a segment, so any sentence chunker run on a segment cannot cut a tag) -> `render()` synthesises per segment, stretches, joins with silence -> one WAV. Duration therefore already includes pauses and stretching. ## Coverage -Covered: `POST /api/tts-preview` (main generate path; seed-finder calls only get tags stripped via `plain_text`). -Not covered: `/v1/audio/speech` proxy, streaming playback, conversation/chat, audiobook casting and magazine MP3/read-aloud (timing records `write_audio_timings` assume untagged text; tags there would be spoken literally). Engine speed params are not used: `atempo` works the same for every engine. -Rollback: set `inline_tags_enabled` to false (default). +`ssml_lite.render_with_tags(text, synth, language, enabled)` is the one shared entry (plain `synth(text)` unless enabled and tagged); async routes bridge to it through `routes.tts._generate_with_tags`. +- Rendered: `POST /api/tts-preview` (normal generate; audiobook line previews and Read Aloud call it from the browser, there is no server-side audiobook synthesis), `POST /v1/audio/speech` non-streaming branches (virtual, voice_design, magpie, nvidia, fish, default), `/speak` and the MCP speak tool. Route sounds and `_prepare_proxy_audio` run after rendering, so durations include pauses. +- Stripped only (`plain_text`, never spoken): streaming `/v1/audio/speech`, sentence-streamed chat replies (`_make_tts_task`; a tag cut by the sentence splitter before it reaches the task can leak), seed-finder calls. +- Magazine MP3 (`synthesize_article_audio`): tags are stripped from each prepared segment before sentence splitting, never rendered. Reason: `write_audio_timings` records per-sentence spans and character counts; inserted silence or stretched clips would need extra span types. Spans are measured on the audio actually joined, so they stay exact. +- Off (default): text is passed through unchanged, tags are spoken literally. +Engine speed params are not used: `atempo` works the same for every engine. +Rollback: set `inline_tags_enabled` to false (Settings -> General switch). diff --git a/core/magazine.py b/core/magazine.py index 9bc4b80..53ed738 100644 --- a/core/magazine.py +++ b/core/magazine.py @@ -5753,6 +5753,11 @@ def synthesize_article_audio(doc_dir: Path, article: dict[str, Any], layout: dic utterance_segment: list[int] = [] # which segment each utterance came from, for the timing record for n, segment in enumerate(segments): spoken = segment.get("spoken") or prepare_text_for_tts(str(segment["text"]), language, glossary) + if settings.get("inline_tags_enabled"): + # Tags are stripped here, never rendered: pauses/stretch would shift the recorded sentence timings. + from core.ssml_lite import has_tags, plain_text + if has_tags(spoken): + spoken = plain_text(spoken, language) segment_voice = interview_voice if segment.get("slot") == 1 else voice segment["voice"] = segment_voice for sentence in split_sentences_for_tts(spoken): diff --git a/core/ssml_lite.py b/core/ssml_lite.py index 04ec9e0..58f79df 100644 --- a/core/ssml_lite.py +++ b/core/ssml_lite.py @@ -165,3 +165,12 @@ def render(text: str, synth: Callable[[str], tuple[bytes, str]], tempo: Callable out = io.BytesIO() combined.export(out, format="wav") return out.getvalue(), "audio/wav" + + +def render_with_tags(text: str, synth: Callable[[str], tuple[bytes, str]], language: str = "auto", + enabled: bool = False) -> tuple[bytes, str]: + """Shared entry for every path that can render tags: plain `synth(text)` unless enabled and tagged.""" + if not (enabled and has_tags(text)): + return synth(text) + from core.audio import _change_tempo + return render(text, synth, _change_tempo, language) diff --git a/routes/conversation.py b/routes/conversation.py index faf86ed..b37778a 100644 --- a/routes/conversation.py +++ b/routes/conversation.py @@ -45,6 +45,7 @@ from core.attribution_spans import ( parse_response as _span_parse_response, validate_spans as _span_validate, validate_only_ids as _span_validate_only_ids, ) +from core import ssml_lite from core.tts_helpers import _preview_request_audio from routes.stt import _transcribe_audio, _clean_stt_backend @@ -5670,7 +5671,9 @@ async def speak(request: Request): instruct = str(data.get("instruct") or "") try: audio, media_type = await asyncio.to_thread( - _preview_request_audio, text, voice, settings, instruct, backend + ssml_lite.render_with_tags, text, + lambda t: _preview_request_audio(t, voice, settings, instruct, backend), + "auto", bool(settings.get("inline_tags_enabled")), ) except Exception as e: raise HTTPException(502, f"TTS error: {e}") @@ -5813,7 +5816,9 @@ async def _mcp_tool_speak(args: dict, client_id: str) -> dict: llm_model = settings.get("llm_model") or "" text = await asyncio.to_thread(_rewrite_with_persona_sync, text, persona, llm_url, llm_model) audio, _media_type = await asyncio.to_thread( - _preview_request_audio, text, voice, settings, "", "voice_clone" + ssml_lite.render_with_tags, text, + lambda t: _preview_request_audio(t, voice, settings, "", "voice_clone"), + "auto", bool(settings.get("inline_tags_enabled")), ) audio_b64 = base64.b64encode(audio).decode() return { @@ -6103,6 +6108,8 @@ def _make_tts_task( sem: "asyncio.Semaphore | None", emotion: str = "", ) -> "asyncio.Task": + if settings.get("inline_tags_enabled"): + text = ssml_lite.plain_text(text) or text # sentence-streamed chat reply: tags stripped, never spoken tts_text, instruct = _conv_tts_text_and_instruct(text, emotion, backend) if sem is None: return asyncio.create_task( diff --git a/routes/tts.py b/routes/tts.py index d6dd6f7..d368203 100644 --- a/routes/tts.py +++ b/routes/tts.py @@ -87,6 +87,17 @@ async def _bounded_tts_call(fn, *args): _TTS_GENERATION_SLOTS.release() +async def _generate_with_tags(text: str, language: str, settings: dict, gen_one): + """Run `gen_one(text)` (async, one synthesis) through inline-tag rendering when the flag is on.""" + if not (settings.get("inline_tags_enabled") and ssml_lite.has_tags(text)): + return await gen_one(text) + loop = asyncio.get_running_loop() + return await asyncio.to_thread( + ssml_lite.render_with_tags, text, + lambda t: asyncio.run_coroutine_threadsafe(gen_one(t), loop).result(), language, True, + ) + + def _voice_design_job_set(job_id: str, **fields) -> None: job = _VOICE_DESIGN_JOBS.get(job_id) if job is not None: @@ -834,15 +845,10 @@ async def tts_preview(request: Request): audio, media_type = await _bounded_tts_call( _seed_finder_request_audio, text, voice, settings, int(_overrides["seed"]) ) - elif settings.get("inline_tags_enabled") and ssml_lite.has_tags(text): - from core.audio import _change_tempo - audio, media_type = await _bounded_tts_call( - ssml_lite.render, text, - lambda t: _preview_request_audio(t, voice, settings, instruct, backend), - _change_tempo, str(data.get("language") or "auto"), - ) else: - audio, media_type = await _bounded_tts_call(_preview_request_audio, text, voice, settings, instruct, backend) + async def _one(t): + return await _bounded_tts_call(_preview_request_audio, t, voice, settings, instruct, backend) + audio, media_type = await _generate_with_tags(text, str(data.get("language") or "auto"), settings, _one) except HTTPException: raise except Exception as e: @@ -1145,6 +1151,8 @@ async def openai_speech_proxy(request: Request): route_has_speed = abs(_route_speed(route) - 1.0) > 0.01 if backend == "streaming" and not virtual and response_format == "wav" and not route_has_sounds and not route_has_speed: + if settings.get("inline_tags_enabled"): + text = ssml_lite.plain_text(text, explicit_lang or "auto") or text # streaming: tags stripped, never spoken try: resp = await _bounded_tts_call(_open_tts_stream_response, text, voice, settings, style_instruction) except HTTPException: @@ -1179,7 +1187,7 @@ async def openai_speech_proxy(request: Request): }, ) - try: + async def _gen_one(text): if virtual: _name, preset = virtual instruct = str(preset.get("description", "")).strip() @@ -1222,6 +1230,11 @@ async def openai_speech_proxy(request: Request): audio, media_type = await _bounded_tts_call( _tts_request_audio, text, voice, settings, style_instruction, ) + return audio, media_type + + try: + audio, media_type = await _generate_with_tags( + text, str(data.get("language") or data.get("lang") or "auto"), settings, _gen_one) except HTTPException: raise except Exception as e: diff --git a/static/js/i18n.js b/static/js/i18n.js index 436f27d..276ab22 100644 --- a/static/js/i18n.js +++ b/static/js/i18n.js @@ -37,6 +37,10 @@ const I18N_DICT = { 'No candidate could be generated.': 'Es konnte kein Kandidat erzeugt werden.', 'Speech-to-text unavailable: scored by level and audio problems only.': 'Sprache-zu-Text nicht verfügbar: Wertung nur nach Pegel und Audioproblemen.', 'Auto-pick failed': 'Automatische Wahl fehlgeschlagen', + // Inline text tags + 'Inline text tags': 'Inline-Texttags', + 'Off by default. When on, tags shape the speech in Try It Out, Read Aloud, the speech API and audiobooks; streaming and magazine audio only remove them.': 'Standardmäßig aus. Wenn aktiv, steuern Tags die Sprache in „Ausprobieren“, „Vorlesen“, der Sprach-API und Hörbüchern; Streaming und Magazin-Audio entfernen sie nur.', + 'Inline tags (enable in Settings → General):': 'Inline-Tags (aktivieren unter Einstellungen → Allgemein):', // Brand / sidebar groups 'Voice Creator': 'Voice Creator', 'Clone · Design · Deploy': 'Klonen · Entwerfen · Bereitstellen', diff --git a/static/js/settings.js b/static/js/settings.js index 54af47b..61d8116 100644 --- a/static/js/settings.js +++ b/static/js/settings.js @@ -579,6 +579,12 @@ async function loadSettings() { restoreEngineApiKeyInputs(); const extKeyInp = $('s-external-api-key'); if (extKeyInp) { extKeyInp.value = ''; extKeyInp.placeholder = s.external_api_key === '__configured__' ? 'Configured — regenerate to replace' : ''; } + const inlineTags = $('s-inline-tags-enabled'); + if (inlineTags) inlineTags.checked = !!s.inline_tags_enabled; + if (inlineTags && !inlineTags.dataset.wired) { + inlineTags.dataset.wired = '1'; + inlineTags.addEventListener('change', () => _patchSettings({ inline_tags_enabled: inlineTags.checked })); + } const extKeyReq = $('s-external-api-key-required'); if (extKeyReq) extKeyReq.checked = !!s.external_api_key_required; if (extKeyReq && !extKeyReq.dataset.wired) { diff --git a/static/sections/s-reader.html b/static/sections/s-reader.html index e53d5a6..56ecf2f 100644 --- a/static/sections/s-reader.html +++ b/static/sections/s-reader.html @@ -79,6 +79,7 @@
Paste textPlain text, Markdown, chapters, notes, or draft prose.
+ Inline tags (enable in Settings → General): [pause 500ms] [spell]USA[/spell] [slow]…[/slow] [fast]…[/fast]
diff --git a/static/sections/s-settings.html b/static/sections/s-settings.html index 50e9101..d461275 100644 --- a/static/sections/s-settings.html +++ b/static/sections/s-settings.html @@ -34,6 +34,14 @@ Custom test sentence used when opening the Seed Finder. +
diff --git a/static/sections/s-tryout.html b/static/sections/s-tryout.html index f49aab0..5619e0a 100644 --- a/static/sections/s-tryout.html +++ b/static/sections/s-tryout.html @@ -61,6 +61,7 @@

Text to synthesize

+ Inline tags (enable in Settings → General): [pause 500ms] [spell]USA[/spell] [slow]…[/slow] [fast]…[/fast]