tts-voice-creator-clone-and-design-2/architecture/watermarking.md

Provenance tags and watermarking for exported audio

Status: tier 1 implemented (core/export_provenance.py); tier 2 (AudioSeal) stubbed, not installed.

Research (checked 2026-10-04)

AudioSeal (facebookresearch/audioseal)

• Licence: README states code AND model weights are MIT ("full MIT license, including the license for the model weights", since April 2024). Commercial use allowed. Verdict: licence acceptable.
• Requirements: Python >= 3.8, PyTorch >= 1.13, OmegaConf, NumPy (einops for streaming). Weights are fetched from the Hugging Face hub on first use (network download, so must be baked in or mounted for an offline box).
• ARM64/CPU: the host has torch 2.10 (aarch64, cu130), but the app image has no torch (nothing in Dockerfile/requirements). The generator is a small conv/LSTM net at 16 kHz and runs on CPU; no CPU benchmark was found in the docs, so cost per hour of audio is unverified (measure before promising). Docs flag torchaudio backend quirks (needs soundfile or torchaudio 2.1).
• Robustness (paper "Proactive Detection of Voice Cloning with Localized Watermarking", arXiv 2401.17264, and third-party comparisons): detection (is-watermarked) stays high under MP3/AAC (mean AUC about 0.97; 128 kbps AAC detection 1.0), but exact recovery of the 16-bit message under AAC 128k falls to about 0 and newer comparisons report < 60% message accuracy under MP3/AAC. So: only "watermarked yes/no" is dependable after re-encode, not a voice/version id.
• What is detectable: per-sample (1/16000 s) localized probability that audio was watermarked, plus optional 16-bit message. Needs the AudioSeal detector (python+torch); no public third party checks it. Trained at 16 kHz, resampling needed for 22-44 kHz audio. Adversarial removal is possible.

Tier 1 alternatives (no dependencies)

• Provenance tags in the container (ID3v2.3 TXXX/COMM via ffmpeg -metadata): synthetic_speech=true, generator, generator_version, generated_date, optional voice (name only), comment. Honest, trivially strippable, survives only while the container is kept.
• Spoken disclosure ("Dieses Hörbuch wurde synthetisch erzeugt"): not strippable by metadata removal, but needs a TTS call plus a re-encoding concat; NOT implemented yet (see Open decisions).

Design

1. Tier 1 (implemented): on export, one ffmpeg stream copy (-c copy -map_metadata 0 -id3v2_version 3) of the MP3 into a temp copy with the tags. The stored article audio is never changed. Applies to: single article download, audio.zip, and the MP3s inside the EPUB. Query ?provenance=1|0 and &voice=1 per export; absent -> setting export_provenance_default (default true). No user names, paths or secrets are written; voice only when opted in, basename only; ffmpeg errors are not forwarded (they can contain paths).
2. Verify: POST /api/export/provenance (raw audio body, 100 MB cap) returns the tags and synthetic; CLI python -m core.export_provenance file.mp3 (exit 0 if tagged).
3. Tier 2: watermark(path) raises NotImplementedError; watermark_available() is False. Enabling it later means: optional extra image/venv with torch + audioseal and pre-mounted weights, a feature flag, lazy import inside watermark(), and a decision on re-encode (watermark must be applied to PCM before the MP3 encode, i.e. at synthesis time, not on the finished MP3 copy).

Open decisions

• Spoken disclosure: render once per language via the normal TTS path and concat at export (re-encode) - needs owner go-ahead.
• Tier 2: needs torch in a sidecar container (not the main image); recommended only if an inaudible mark is really required, given weak message recovery after AAC/MP3.