diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e033b9..0902350 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi ## [Unreleased] ### Added +- **Inline text tags for TTS (setting `inline_tags_enabled`, default OFF).** `[pause 500ms]`/`[pause 1.2s]` (max 10 s each, 60 s per text), `[spell]USA[/spell]` (German letter names for German text), `[slow]..[/slow]` / `[fast]..[/fast]` (0.85 / 1.15 via ffmpeg atempo, innermost wins, unclosed ends at paragraph). Known control tags (e.g. `[laugh]`) are stripped, never spoken; `[1]` and `[sic]` stay. Only `POST /api/tts-preview` renders them; other paths are listed in `architecture/inline-tags.md`. Tests: `tests/test_ssml_lite.py`. - **Optional vocal isolation for URL/YouTube imports (default OFF).** `POST /api/isolate-vocals {id}` runs the Demucs CLI (`htdemucs`, two stems) on a registered temp audio and registers the vocals as a new id; the original stays valid. Needs `VOCAL_ISOLATION=1`, a `demucs` binary (or `VOCAL_ISOLATION_CMD`) and the weights already on disk; the app never downloads anything and nothing is added to the main image. Limits: 15 min input, 15 min runtime (`VOCAL_ISOLATION_MAX_SECONDS`/`_TIMEOUT`), one job at a time, cancelled on client disconnect, temp files always removed. `GET /api/version` reports `features.vocal_isolation`; the "Remove background music" button (clone page URL card and voice library add-URL) appears only then, with an aria-live status and a play original / play without music toggle. Tests: `tests/test_vocal_isolation.py`, `tests/vocal-isolation.spec.js`. See `architecture/vocal-isolation.md`. - **Voice export/import bundles.** `GET /api/voices/{id}/export` streams a `.voice.zip` (audio, `reference.txt`, `meta.json` with seed, design prompt, rating/tags, `provenance`, `audio_problems`; optional picture and consent recording; `manifest.json` with format version, app version and sha256 per file; no temp files). `POST /api/voices/import` validates at the boundary: 150 MB upload cap, fixed flat member names only (zip-slip/path traversal rejected), per-file, total and compression-ratio caps, hashes verified, audio decoded, meta and provenance re-validated; a taken name gets a `_2` suffix, an identical voice is a no-op, nothing is overwritten. Voice library toolbar: "Export selected" and "Import bundle" (44px). Tests: `tests/test_voice_bundle.py`, `tests/voice-bundle.spec.js`. See `architecture/voice-bundle.md`. - **Consent/provenance field per voice.** Each voice gets `provenance` in its `.meta.json` (`basis`: own_voice, consented, licensed, public_domain or unknown; `source`, `licence`, `note`, optional `recorded_consent`, `set_at`). Existing voices count as `unknown`; `quick-import-voice` prefills the source URL (basis stays `unknown`). New `PUT /api/voices/{id}/provenance` (enum, length caps, control characters stripped, http(s) URLs only, no credentials) and `POST`/`GET /api/voices/{id}/consent-recording` (audio only, 10 MB cap, stored as `.consent`, moves/deletes with the voice). The voice inspector has a "Source & consent" section (typed note or MediaRecorder recording) and voices without a recorded basis show a "Source unknown" badge. Generation is not blocked. Data stays local. Tests: `tests/test_voice_provenance.py`, `tests/voice-provenance.spec.js`. diff --git a/Manual.md b/Manual.md index 66c1e4e..5d3eaae 100644 --- a/Manual.md +++ b/Manual.md @@ -8,6 +8,17 @@ Living manual. Every feature updates this file in the same commit as the code. --- +## Inline tags in the text (optional) +Purpose: pauses, spelled-out acronyms and slower/faster passages inside one text. Prerequisite: set `inline_tags_enabled` to true in the settings (default off; when off the tags are spoken literally). + +| Tag | Result | +|---|---| +| `[pause 500ms]` / `[pause 1.2s]` | silence, max 10 s each and 60 s per text | +| `[spell]USA[/spell]` | letters one by one (German letter names for German text) | +| `[slow]..[/slow]` / `[fast]..[/fast]` | 0.85x / 1.15x speed, same pitch; innermost wins; unclosed ends at the paragraph | + +Known control tags such as `[laugh]` are removed, never read; `[1]` and `[sic]` stay. Works with every engine in the Voice Generator preview/generate path (speed via ffmpeg, so no engine support needed). Not yet in the OpenAI-compatible proxy, streaming, chat or audiobook/magazine audio. Errors: a failing engine reports the usual TTS error. Mobile/accessibility/privacy: no UI change, text stays local. + ## Design a Voice: attribute chips **Purpose:** build a voice description quickly. **Use:** open Design a Voice, press chips (gender, age, pitch, pace, accent/region, tone/timbre, emotion). Each chip adds its phrase to the description; pressing it again removes only that phrase. Gender, age, pitch and pace allow one choice each. **Free text:** type anything in the box; chips never overwrite your text and are not a limit on what you can describe. **Keyboard/touch:** chips are buttons (Tab, Space/Enter, `aria-pressed`, 44 px targets). **Languages:** chip labels follow the UI language (German/English); the inserted phrase is English. **Offline/privacy:** purely local, nothing is sent until you generate. diff --git a/architecture/inline-tags.md b/architecture/inline-tags.md new file mode 100644 index 0000000..e665936 --- /dev/null +++ b/architecture/inline-tags.md @@ -0,0 +1,23 @@ +# Inline text tags (feature flag `inline_tags_enabled`, default off) + +Engine-independent preprocessing in `core/ssml_lite.py` (pure parser plus `render`). Our engines have no native tags, so the app splits text, synthesises each piece and joins the audio itself. Idea only is borrowed from other tools; no third-party code. + +## Syntax +| Tag | Effect | +|---|---| +| `[pause 500ms]`, `[pause 1.2s]` | generated silence; max 10 s each, 60 s per text; invalid argument is dropped | +| `[spell]USA[/spell]` | letters spoken one by one (`U S A`; German letter names `Uh Ess Ah` when language is German) | +| `[slow]..[/slow]`, `[fast]..[/fast]` | speed factor 0.85 / 1.15 via ffmpeg `atempo` (`core.audio._change_tempo`) on the rendered segment; innermost wins | + +Unclosed `[slow]`/`[fast]`/`[spell]` end at the paragraph break (blank line). + +## Stripping rule +Only these names are removed and never spoken (opening or closing, optional short argument): the four tags above plus `laugh(s/ing) chuckle sigh gasp cough breath breathe inhale exhale whisper shout music sfx noise beat silence emphasis excited sad angry happy calm sniff gulp clears_throat`. Everything else in brackets (`[1]`, `[sic]`, `[Anm.]`) is ordinary prose and untouched. + +## Flow +`parse(text, language)` -> segments `{text, speed, pause_ms}` (tags never reach a segment, so any sentence chunker run on a segment cannot cut a tag) -> `render()` synthesises per segment, stretches, joins with silence -> one WAV. Duration therefore already includes pauses and stretching. + +## Coverage +Covered: `POST /api/tts-preview` (main generate path; seed-finder calls only get tags stripped via `plain_text`). +Not covered: `/v1/audio/speech` proxy, streaming playback, conversation/chat, audiobook casting and magazine MP3/read-aloud (timing records `write_audio_timings` assume untagged text; tags there would be spoken literally). Engine speed params are not used: `atempo` works the same for every engine. +Rollback: set `inline_tags_enabled` to false (default). diff --git a/core/config.py b/core/config.py index c60b471..84d4154 100644 --- a/core/config.py +++ b/core/config.py @@ -45,6 +45,7 @@ _SETTINGS_KEYS = { # LLM so page images never accidentally go to a text-only model. "magazine_layout_url", "magazine_layout_model", "vision_ocr_url", "vision_ocr_model", "vision_ocr_enabled", + "inline_tags_enabled", "vision_ocr_fallback_url", "vision_ocr_fallback_model", # OPDS catalogues to search for magazine issues: [{"name": ..., "url": ...}], no credentials. "magazine_opds_catalogs", @@ -346,6 +347,7 @@ def _load_settings() -> dict: "vision_ocr_url": _VISION_OCR_DEFAULT, "vision_ocr_model": _VISION_OCR_MODEL_DEFAULT, "vision_ocr_enabled": True, + "inline_tags_enabled": False, "vision_ocr_fallback_url": _VISION_OCR_FALLBACK_DEFAULT, "vision_ocr_fallback_model": _VISION_OCR_FALLBACK_MODEL_DEFAULT, "engine_local_urls": {}, diff --git a/core/ssml_lite.py b/core/ssml_lite.py new file mode 100644 index 0000000..04ec9e0 --- /dev/null +++ b/core/ssml_lite.py @@ -0,0 +1,167 @@ +"""Inline text tags for TTS, engine independent (idea only; no third-party code). + +Tags: [pause 500ms] [pause 1.2s], [spell]USA[/spell], [slow]..[/slow], [fast]..[/fast]. +parse() turns text into segments {text, speed, pause_ms}; render() synthesises and joins them. +Only a fixed set of tag names is ever stripped, so "[1]" or "[sic]" stay untouched. +""" +from __future__ import annotations + +import re +from typing import Callable + +MAX_PAUSE_MS = 10_000 +MAX_TOTAL_PAUSE_MS = 60_000 +SPEEDS = {"slow": 0.85, "fast": 1.15} + +# Control-style tags from other tools that must never be spoken. Allowlist, not "any [word]". +STRIP_NAMES = frozenset({ + "laugh", "laughs", "laughing", "chuckle", "sigh", "gasp", "cough", "breath", "breathe", + "inhale", "exhale", "whisper", "shout", "music", "sfx", "noise", "beat", "silence", + "emphasis", "excited", "sad", "angry", "happy", "calm", "sniff", "gulp", "clears_throat", +}) + +_TAG_RE = re.compile(r"\[(/?)([A-Za-z_]+)(?:[ \t]+([^\[\]\n]{0,24}))?\]") +_PAUSE_ARG_RE = re.compile(r"^(\d+(?:\.\d+)?)\s*(ms|s)$", re.I) +_PARA_RE = re.compile(r"(\n\s*\n)") + +_DE_LETTERS = { + "a": "Ah", "b": "Be", "c": "Ze", "d": "De", "e": "Eh", "f": "Eff", "g": "Ge", "h": "Ha", + "i": "Ih", "j": "Jott", "k": "Ka", "l": "Ell", "m": "Emm", "n": "Enn", "o": "Oh", "p": "Pe", + "q": "Ku", "r": "Err", "s": "Ess", "t": "Te", "u": "Uh", "v": "Fau", "w": "We", "x": "Iks", + "y": "Üpsilon", "z": "Zett", "ä": "Ä", "ö": "Ö", "ü": "Ü", "ß": "Ess-Zett", +} + + +def _spell(content: str, language: str) -> str: + chars = [c for c in content if not c.isspace()] + if language == "de": + return " ".join(_DE_LETTERS.get(c.lower(), c) for c in chars) + from core.pronunciations import spell_letters + return spell_letters("".join(chars)) + + +def _lang(text: str, language: str) -> str: + low = (language or "auto").strip().lower() + if low in {"de", "german", "deutsch"} or low.startswith("de-"): + return "de" + if low in {"", "auto"}: + from core.pronunciations import detect_text_language + return detect_text_language(text) + return "en" + + +def _pause_ms(arg: str | None) -> int | None: + m = _PAUSE_ARG_RE.match((arg or "").strip()) + if not m: + return None + ms = float(m.group(1)) * (1000 if m.group(2).lower() == "s" else 1) + return min(int(ms), MAX_PAUSE_MS) + + +def has_tags(text: str) -> bool: + return any(m.group(2).lower() in {"pause", "spell", "slow", "fast"} | STRIP_NAMES + for m in _TAG_RE.finditer(text or "")) + + +def _parse_paragraph(text: str, lang: str) -> list[dict]: + out: list[dict] = [] + speeds: list[float] = [] + spell_from: int | None = None # index in `out` where an open [spell] started + pos = 0 + + def add(chunk: str) -> None: + if chunk: + out.append({"text": chunk, "speed": speeds[-1] if speeds else 1.0, "pause_ms": 0}) + + def close_spell() -> None: + nonlocal spell_from + if spell_from is not None: + joined = "".join(s["text"] for s in out[spell_from:]) + del out[spell_from:] + add(_spell(joined, lang)) + spell_from = None + + for m in _TAG_RE.finditer(text): + name, closing = m.group(2).lower(), m.group(1) == "/" + known = name in {"pause", "spell", "slow", "fast"} or name in STRIP_NAMES + if not known: + continue # ordinary prose such as "[1]" or "[sic]": handled below + add(text[pos:m.start()]) + pos = m.end() + if name == "pause" and not closing: + ms = _pause_ms(m.group(3)) + if ms: + out.append({"text": "", "speed": 1.0, "pause_ms": ms}) + elif name == "spell": + if closing: + close_spell() + elif spell_from is None: + spell_from = len(out) + elif name in SPEEDS: + if closing: + if speeds: + speeds.pop() + else: + speeds.append(SPEEDS[name]) + # anything else in STRIP_NAMES (and a stray [/pause]) is dropped + add(text[pos:]) + close_spell() # unclosed [spell] runs to the end of the paragraph + return out + + +def parse(text: str, language: str = "auto") -> list[dict]: + """Split text into segments; unclosed tags end at the paragraph break; pauses are capped.""" + lang = _lang(text, language) + segments: list[dict] = [] + for part in _PARA_RE.split(text or ""): + if _PARA_RE.fullmatch(part): + if segments and not segments[-1]["pause_ms"]: + segments[-1]["text"] += part + continue + segments.extend(_parse_paragraph(part, lang)) + budget = MAX_TOTAL_PAUSE_MS + merged: list[dict] = [] + for seg in segments: + if seg["pause_ms"]: + seg["pause_ms"] = min(seg["pause_ms"], budget) + budget -= seg["pause_ms"] + if not seg["pause_ms"]: + continue + elif merged and not merged[-1]["pause_ms"] and merged[-1]["speed"] == seg["speed"]: + merged[-1]["text"] += seg["text"] + continue + merged.append(seg) + return [s for s in merged if s["pause_ms"] or s["text"].strip()] + + +def plain_text(text: str, language: str = "auto") -> str: + """Text with all tags resolved and no timing (for paths that cannot render pauses/speed).""" + return re.sub(r"[ \t]+", " ", " ".join(s["text"].strip() for s in parse(text, language) if s["text"].strip())) + + +def render(text: str, synth: Callable[[str], tuple[bytes, str]], tempo: Callable[[bytes, float], bytes], + language: str = "auto") -> tuple[bytes, str]: + """Synthesise every segment, stretch slow/fast ones, join with generated silence -> WAV bytes. + + The result's duration already includes pauses and stretching, so timings measured on it stay right. + """ + import io + from pydub import AudioSegment + + combined = AudioSegment.empty() + for seg in parse(text, language): + if seg["pause_ms"]: + rate = combined.frame_rate if len(combined) else 24000 + combined += AudioSegment.silent(duration=seg["pause_ms"], frame_rate=rate) + continue + audio, media_type = synth(seg["text"].strip()) + fmt = "wav" if audio[:4] == b"RIFF" or "wav" in media_type.lower() else None + piece = AudioSegment.from_file(io.BytesIO(audio), format=fmt) + if abs(seg["speed"] - 1.0) > 0.01: + buf = io.BytesIO() + piece.export(buf, format="wav") + piece = AudioSegment.from_file(io.BytesIO(tempo(buf.getvalue(), seg["speed"])), format="wav") + combined += piece + out = io.BytesIO() + combined.export(out, format="wav") + return out.getvalue(), "audio/wav" diff --git a/routes/tts.py b/routes/tts.py index 341f10a..d6dd6f7 100644 --- a/routes/tts.py +++ b/routes/tts.py @@ -39,6 +39,7 @@ from core.voice import ( ) from core.voice_index import indexed_voices, refresh_voice_index_background from core.validation import _validate_http_url +from core import ssml_lite from core.provider_health import probe_provider from core.registry import _registry_put, TEMP_DIR from core.tts_helpers import ( @@ -828,9 +829,18 @@ async def tts_preview(request: Request): try: if backend == "voice_clone" and data.get("seed_finder") and "seed" in _overrides: + if settings.get("inline_tags_enabled"): + text = ssml_lite.plain_text(text, str(data.get("language") or "auto")) audio, media_type = await _bounded_tts_call( _seed_finder_request_audio, text, voice, settings, int(_overrides["seed"]) ) + elif settings.get("inline_tags_enabled") and ssml_lite.has_tags(text): + from core.audio import _change_tempo + audio, media_type = await _bounded_tts_call( + ssml_lite.render, text, + lambda t: _preview_request_audio(t, voice, settings, instruct, backend), + _change_tempo, str(data.get("language") or "auto"), + ) else: audio, media_type = await _bounded_tts_call(_preview_request_audio, text, voice, settings, instruct, backend) except HTTPException: diff --git a/tests/test_ssml_lite.py b/tests/test_ssml_lite.py new file mode 100644 index 0000000..e2c8dd8 --- /dev/null +++ b/tests/test_ssml_lite.py @@ -0,0 +1,84 @@ +import io +import unittest + +from pydub import AudioSegment +from pydub.generators import Sine + +from core import ssml_lite as S +from core.magazine import split_sentences_for_tts + + +def texts(segs): + return [(s["text"], s["speed"], s["pause_ms"]) for s in segs] + + +class ParseTests(unittest.TestCase): + def test_plain_text_is_one_segment(self): + self.assertEqual(texts(S.parse("Hello there.")), [("Hello there.", 1.0, 0)]) + + def test_pause_units_and_split(self): + self.assertEqual(texts(S.parse("A [pause 500ms] B [pause 1.2s] C")), + [("A ", 1.0, 0), ("", 1.0, 500), (" B ", 1.0, 0), ("", 1.0, 1200), (" C", 1.0, 0)]) + + def test_pause_caps(self): + self.assertEqual(S.parse("A[pause 99s]B")[1]["pause_ms"], 10_000) + many = "A" + "[pause 10s]B" * 10 + self.assertEqual(sum(s["pause_ms"] for s in S.parse(many)), 60_000) + + def test_bad_pause_is_stripped_not_spoken(self): + self.assertEqual(S.plain_text("A [pause soon] B"), "A B") + + def test_speed_nesting_innermost_wins(self): + segs = S.parse("a [slow]b [fast]c[/fast] d[/slow] e") + self.assertEqual([(s["text"].strip(), s["speed"]) for s in segs], + [("a", 1.0), ("b", 0.85), ("c", 1.15), ("d", 0.85), ("e", 1.0)]) + + def test_unclosed_runs_to_paragraph_end(self): + segs = S.parse("[slow]one\n\ntwo") + self.assertEqual([(s["text"].strip(), s["speed"]) for s in segs], [("one", 0.85), ("two", 1.0)]) + + def test_unknown_control_tags_stripped_citations_kept(self): + self.assertEqual(S.plain_text("Hi [laugh] there [/emphasis] [sic] cited [1]."), "Hi there [sic] cited [1].") + self.assertNotIn("[", S.plain_text("[whisper]psst[/whisper]")) + + def test_spell_english_and_german(self): + self.assertEqual(S.plain_text("[spell]USA[/spell]", "en"), "U S A") + self.assertEqual(S.plain_text("[spell]USA[/spell]", "de"), "Uh Ess Ah") + self.assertEqual(S.plain_text("[spell]ZDF[/spell] Nachrichten über Straße"), "Zett De Eff Nachrichten über Straße") + self.assertEqual(S.plain_text("[spell]ab", "en"), "a b") + + def test_segments_never_contain_tags_and_chunker_keeps_them_whole(self): + text = "[slow]First. Second sentence here.[/slow] Third. [pause 1s] Fourth." + for seg in S.parse(text): + self.assertNotIn("[", seg["text"]) + slow = [s for s in S.parse(text) if s["speed"] != 1.0][0] + self.assertEqual(slow["speed"], 0.85) + self.assertIn("First.", slow["text"]) + self.assertIn("Second", slow["text"]) + self.assertGreaterEqual(len(split_sentences_for_tts(slow["text"])), 1) + + def test_has_tags(self): + self.assertTrue(S.has_tags("a [pause 1s]")) + self.assertFalse(S.has_tags("see [1] and [sic]")) + + +class RenderTests(unittest.TestCase): + def test_render_joins_with_silence_and_stretch(self): + tone = Sine(440).to_audio_segment(duration=1000).set_frame_rate(24000) + buf = io.BytesIO() + tone.export(buf, format="wav") + wav = buf.getvalue() + + def tempo(b, speed): # fake: just trims, proves it was called with the factor + tempo.calls.append(speed) + return b + tempo.calls = [] + out, media = S.render("A [pause 500ms] [slow]B[/slow]", lambda t: (wav, "audio/wav"), tempo, "en") + dur = len(AudioSegment.from_file(io.BytesIO(out), format="wav")) + self.assertEqual(media, "audio/wav") + self.assertEqual(tempo.calls, [0.85]) + self.assertAlmostEqual(dur, 2500, delta=30) + + +if __name__ == "__main__": + unittest.main()