feat(voices): optional vocal isolation for URL imports
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
e0daafd4e1
commit
91843e95e6
@ -7,6 +7,7 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
### Added
|
### Added
|
||||||
|
- **Optional vocal isolation for URL/YouTube imports (default OFF).** `POST /api/isolate-vocals {id}` runs the Demucs CLI (`htdemucs`, two stems) on a registered temp audio and registers the vocals as a new id; the original stays valid. Needs `VOCAL_ISOLATION=1`, a `demucs` binary (or `VOCAL_ISOLATION_CMD`) and the weights already on disk; the app never downloads anything and nothing is added to the main image. Limits: 15 min input, 15 min runtime (`VOCAL_ISOLATION_MAX_SECONDS`/`_TIMEOUT`), one job at a time, cancelled on client disconnect, temp files always removed. `GET /api/version` reports `features.vocal_isolation`; the "Remove background music" button (clone page URL card and voice library add-URL) appears only then, with an aria-live status and a play original / play without music toggle. Tests: `tests/test_vocal_isolation.py`, `tests/vocal-isolation.spec.js`. See `architecture/vocal-isolation.md`.
|
||||||
- **Voice export/import bundles.** `GET /api/voices/{id}/export` streams a `<id>.voice.zip` (audio, `reference.txt`, `meta.json` with seed, design prompt, rating/tags, `provenance`, `audio_problems`; optional picture and consent recording; `manifest.json` with format version, app version and sha256 per file; no temp files). `POST /api/voices/import` validates at the boundary: 150 MB upload cap, fixed flat member names only (zip-slip/path traversal rejected), per-file, total and compression-ratio caps, hashes verified, audio decoded, meta and provenance re-validated; a taken name gets a `_2` suffix, an identical voice is a no-op, nothing is overwritten. Voice library toolbar: "Export selected" and "Import bundle" (44px). Tests: `tests/test_voice_bundle.py`, `tests/voice-bundle.spec.js`. See `architecture/voice-bundle.md`.
|
- **Voice export/import bundles.** `GET /api/voices/{id}/export` streams a `<id>.voice.zip` (audio, `reference.txt`, `meta.json` with seed, design prompt, rating/tags, `provenance`, `audio_problems`; optional picture and consent recording; `manifest.json` with format version, app version and sha256 per file; no temp files). `POST /api/voices/import` validates at the boundary: 150 MB upload cap, fixed flat member names only (zip-slip/path traversal rejected), per-file, total and compression-ratio caps, hashes verified, audio decoded, meta and provenance re-validated; a taken name gets a `_2` suffix, an identical voice is a no-op, nothing is overwritten. Voice library toolbar: "Export selected" and "Import bundle" (44px). Tests: `tests/test_voice_bundle.py`, `tests/voice-bundle.spec.js`. See `architecture/voice-bundle.md`.
|
||||||
- **Consent/provenance field per voice.** Each voice gets `provenance` in its `.meta.json` (`basis`: own_voice, consented, licensed, public_domain or unknown; `source`, `licence`, `note`, optional `recorded_consent`, `set_at`). Existing voices count as `unknown`; `quick-import-voice` prefills the source URL (basis stays `unknown`). New `PUT /api/voices/{id}/provenance` (enum, length caps, control characters stripped, http(s) URLs only, no credentials) and `POST`/`GET /api/voices/{id}/consent-recording` (audio only, 10 MB cap, stored as `<voice>.consent`, moves/deletes with the voice). The voice inspector has a "Source & consent" section (typed note or MediaRecorder recording) and voices without a recorded basis show a "Source unknown" badge. Generation is not blocked. Data stays local. Tests: `tests/test_voice_provenance.py`, `tests/voice-provenance.spec.js`.
|
- **Consent/provenance field per voice.** Each voice gets `provenance` in its `.meta.json` (`basis`: own_voice, consented, licensed, public_domain or unknown; `source`, `licence`, `note`, optional `recorded_consent`, `set_at`). Existing voices count as `unknown`; `quick-import-voice` prefills the source URL (basis stays `unknown`). New `PUT /api/voices/{id}/provenance` (enum, length caps, control characters stripped, http(s) URLs only, no credentials) and `POST`/`GET /api/voices/{id}/consent-recording` (audio only, 10 MB cap, stored as `<voice>.consent`, moves/deletes with the voice). The voice inspector has a "Source & consent" section (typed note or MediaRecorder recording) and voices without a recorded basis show a "Source unknown" badge. Generation is not blocked. Data stays local. Tests: `tests/test_voice_provenance.py`, `tests/voice-provenance.spec.js`.
|
||||||
- **Seed Finder: Auto-pick best.** `POST /api/seed-autopick {voice, text, seeds|count}` (default 8, max 16) generates candidates sequentially through the existing temporary-voice path (voices.json is always restored), scores each 0-100 (ASR word accuracy, loudness/clipping, `audio_problems`) and returns all sub-scores plus the winner. Degrades to loudness/problems when STT is unavailable. The UI highlights the winner; saving stays a click on "Use seed". Tests: `tests/test_seed_autopick.py`, `tests/seed-autopick.spec.js`.
|
- **Seed Finder: Auto-pick best.** `POST /api/seed-autopick {voice, text, seeds|count}` (default 8, max 16) generates candidates sequentially through the existing temporary-voice path (voices.json is always restored), scores each 0-100 (ASR word accuracy, loudness/clipping, `audio_problems`) and returns all sub-scores plus the winner. Degrades to loudness/problems when STT is unavailable. The UI highlights the winner; saving stays a click on "Use seed". Tests: `tests/test_seed_autopick.py`, `tests/seed-autopick.spec.js`.
|
||||||
|
|||||||
10
Manual.md
10
Manual.md
@ -14,6 +14,16 @@ Living manual. Every feature updates this file in the same commit as the code.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Remove background music from URL imports (optional)
|
||||||
|
|
||||||
|
- **Purpose:** strip music and background sound from a downloaded clip before cloning, so the clone does not learn the music.
|
||||||
|
- **Prerequisites:** off by default. The server owner sets `VOCAL_ISOLATION=1`, installs Demucs (`pip install demucs` in a venv or sidecar, ARM64 supported; set `VOCAL_ISOLATION_CMD` if it is not on `PATH`) and downloads the `htdemucs` weights once (`955717e8-8726e21a.th`, about 80 MB, into `$TORCH_HOME/hub/checkpoints`). The app never downloads weights itself. Without all of this the button does not appear.
|
||||||
|
- **Steps:** download a YouTube/URL clip, press **Remove background music**, wait (minutes on CPU). The cleaned audio loads in the player or waveform; press **Play original** / **Play without music** to compare. Then trim and save as usual.
|
||||||
|
- **Result:** a new temporary audio; the original is kept until it expires.
|
||||||
|
- **Errors:** clips over 15 minutes are refused (trim first), only one job runs at a time, a timeout or failure shows a message under the button.
|
||||||
|
- **Mobile/accessibility:** buttons are normal buttons; progress and errors are announced through a polite live region.
|
||||||
|
- **Offline/privacy:** runs fully locally; no audio leaves the machine.
|
||||||
|
|
||||||
## Voice export and import bundles
|
## Voice export and import bundles
|
||||||
|
|
||||||
Purpose: back up a voice or move it between machines (DGX and NUC) as one `.voice.zip` file containing the reference audio, transcript, settings (seed, design prompt, rating, tags, source & consent, audio problems), and optionally the picture and the consent recording.
|
Purpose: back up a voice or move it between machines (DGX and NUC) as one `.voice.zip` file containing the reference audio, transcript, settings (seed, design prompt, rating, tags, source & consent, audio problems), and optionally the picture and the consent recording.
|
||||||
|
|||||||
42
architecture/vocal-isolation.md
Normal file
42
architecture/vocal-isolation.md
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
# Vocal isolation for URL/YouTube imports
|
||||||
|
|
||||||
|
Status: implemented behind a flag, default OFF. Goal: remove music/background from a downloaded
|
||||||
|
clip before trimming and cloning (`/api/download-yt` -> `/api/isolate-vocals` -> trim -> `quick-import-voice`).
|
||||||
|
|
||||||
|
## Findings (read-only inspection, 2026-10-04)
|
||||||
|
- App image: `python:3.11-slim-bookworm`, `requirements.txt` has no torch/onnx/demucs. The live app
|
||||||
|
container has no `torch` (`import torch` fails). It is `read_only: true`, `cap_drop: ALL`, 8 GB mem
|
||||||
|
limit, no GPU reservation. Source dirs are bind-mounted read-only.
|
||||||
|
- GPU/torch live in other service containers (tts-faster-qwen3-*, aligner, ...), not reusable: different
|
||||||
|
images, no shared API for source separation.
|
||||||
|
- There is no VAD, denoise or separation anywhere in the app today.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
| Option | Footprint | Verdict |
|
||||||
|
|---|---|---|
|
||||||
|
| Demucs `htdemucs` (MIT code; weights MIT, ~80 MB, from dl.fbaipublicfiles.com via torch hub) in the main image | torch + torchaudio on ARM64 ~ 2-3 GB image growth | rejected: heavy, unconditional |
|
||||||
|
| Demucs in a separate sidecar container | no growth of main image, GPU optional | best long-term, but needs a new image, compose service and network contract |
|
||||||
|
| **Optional external Demucs CLI**, called as a subprocess, only when the flag is on and the binary + weights exist | zero deps in the main image; the owner picks where Demucs lives (venv on a bind mount, or a derived image) | **chosen** |
|
||||||
|
| Lighter: RNNoise/DeepFilterNet/spectral gating | small, but removes noise only, not music/singing | does not meet the goal |
|
||||||
|
|
||||||
|
Why the CLI: `demucs --two-stems=vocals -n htdemucs` is the stable public interface; no import-time cost,
|
||||||
|
no behaviour change when absent. ARM64: `torch`/`torchaudio` publish aarch64 wheels (CUDA wheels exist for
|
||||||
|
DGX Spark / GB10; CPU wheels work everywhere), `demucs` itself is pure Python (`pip install demucs`).
|
||||||
|
A sidecar can later expose the same CLI via `VOCAL_ISOLATION_CMD` (e.g. `docker exec ...` wrapper) without code changes.
|
||||||
|
|
||||||
|
## Safety rules
|
||||||
|
- `VOCAL_ISOLATION=1` required; otherwise the feature reports unavailable and the API answers 503.
|
||||||
|
- Available only if the command resolves AND the htdemucs weights file (`955717e8-*.th`) is already in
|
||||||
|
`$TORCH_HOME/hub/checkpoints` (default `~/.cache/torch`). The app never downloads weights: the owner
|
||||||
|
installs them once (documented in Manual.md). This prevents Demucs' silent first-run download.
|
||||||
|
- Input bounded to `VOCAL_ISOLATION_MAX_SECONDS` (default 900), runtime to `VOCAL_ISOLATION_TIMEOUT`
|
||||||
|
(default 900 s), one job at a time (409 when busy), client disconnect cancels and kills the process.
|
||||||
|
- Work happens in a private temp dir that is always removed; the result is converted to 24 kHz mono WAV
|
||||||
|
and registered as a new temp id. The original id is untouched (compare / undo in the UI).
|
||||||
|
- Subprocess started with an argument list (no shell); the path comes from the registry, never from the client.
|
||||||
|
|
||||||
|
## Contract
|
||||||
|
- `GET /api/version` gains `features.vocal_isolation: bool`.
|
||||||
|
- `POST /api/isolate-vocals {id}` -> `{id, duration}`. Errors: 400 bad id, 404 unknown id, 413 too long,
|
||||||
|
409 busy, 503 disabled/unavailable, 504 timeout, 500 failed (message without internals).
|
||||||
|
- Offline behaviour: fully local; no network access.
|
||||||
82
core/vocal_isolation.py
Normal file
82
core/vocal_isolation.py
Normal file
@ -0,0 +1,82 @@
|
|||||||
|
"""Optional vocal isolation (Demucs CLI). Off unless VOCAL_ISOLATION=1; never downloads weights."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from core.audio import _duration, _to_wav_24k
|
||||||
|
from core.registry import TEMP_DIR
|
||||||
|
|
||||||
|
_WEIGHTS_GLOB = "955717e8*.th" # htdemucs
|
||||||
|
_busy = threading.Lock() # ponytail: one job at a time; a queue only if several users isolate concurrently
|
||||||
|
|
||||||
|
|
||||||
|
class VocalIsolationError(Exception):
|
||||||
|
def __init__(self, message: str, status: int = 500):
|
||||||
|
super().__init__(message)
|
||||||
|
self.status = status
|
||||||
|
|
||||||
|
|
||||||
|
def _cmd() -> str | None:
|
||||||
|
return os.environ.get("VOCAL_ISOLATION_CMD") or shutil.which("demucs")
|
||||||
|
|
||||||
|
|
||||||
|
def _weights_present() -> bool:
|
||||||
|
home = Path(os.environ.get("TORCH_HOME") or Path.home() / ".cache" / "torch")
|
||||||
|
return any((home / "hub" / "checkpoints").glob(_WEIGHTS_GLOB))
|
||||||
|
|
||||||
|
|
||||||
|
def enabled() -> bool:
|
||||||
|
return os.environ.get("VOCAL_ISOLATION", "").strip().lower() in {"1", "true", "yes", "on"}
|
||||||
|
|
||||||
|
|
||||||
|
def available() -> bool:
|
||||||
|
return enabled() and _cmd() is not None and _weights_present()
|
||||||
|
|
||||||
|
|
||||||
|
def isolate_vocals(path: Path, cancel: threading.Event | None = None) -> Path:
|
||||||
|
"""Return a new 24 kHz mono WAV with vocals only; `path` is left untouched."""
|
||||||
|
if not enabled():
|
||||||
|
raise VocalIsolationError("Vocal isolation is turned off (set VOCAL_ISOLATION=1).", 503)
|
||||||
|
cmd = _cmd()
|
||||||
|
if cmd is None or not _weights_present():
|
||||||
|
raise VocalIsolationError("Vocal isolation is not installed (Demucs or its weights are missing).", 503)
|
||||||
|
max_s = float(os.environ.get("VOCAL_ISOLATION_MAX_SECONDS", "900"))
|
||||||
|
timeout = float(os.environ.get("VOCAL_ISOLATION_TIMEOUT", "900"))
|
||||||
|
if _duration(path) > max_s:
|
||||||
|
raise VocalIsolationError(f"Audio is longer than {int(max_s // 60)} minutes; trim it first.", 413)
|
||||||
|
if not _busy.acquire(blocking=False):
|
||||||
|
raise VocalIsolationError("Another vocal isolation is running; try again shortly.", 409)
|
||||||
|
work = Path(tempfile.mkdtemp(prefix="vocal_", dir=TEMP_DIR))
|
||||||
|
try:
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[cmd, "--two-stems=vocals", "-n", "htdemucs", "-o", str(work), str(path)],
|
||||||
|
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||||
|
)
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
while proc.poll() is None:
|
||||||
|
if cancel is not None and cancel.is_set():
|
||||||
|
proc.kill(); proc.wait()
|
||||||
|
raise VocalIsolationError("Vocal isolation was cancelled.", 499)
|
||||||
|
if time.monotonic() > deadline:
|
||||||
|
proc.kill(); proc.wait()
|
||||||
|
raise VocalIsolationError("Vocal isolation timed out.", 504)
|
||||||
|
time.sleep(0.2)
|
||||||
|
found = next(work.glob("*/*/vocals.*"), None)
|
||||||
|
if proc.returncode != 0 or found is None:
|
||||||
|
raise VocalIsolationError("Vocal isolation failed.", 500)
|
||||||
|
staged = TEMP_DIR / f"{uuid.uuid4().hex}_vocals{found.suffix}"
|
||||||
|
shutil.move(str(found), staged)
|
||||||
|
try:
|
||||||
|
return _to_wav_24k(staged)
|
||||||
|
finally:
|
||||||
|
staged.unlink(missing_ok=True)
|
||||||
|
finally:
|
||||||
|
_busy.release()
|
||||||
|
shutil.rmtree(work, ignore_errors=True)
|
||||||
@ -199,7 +199,8 @@ async def browse_dirs_mkdir(payload: dict = Body(...)):
|
|||||||
|
|
||||||
@router.get("/api/version")
|
@router.get("/api/version")
|
||||||
async def get_version():
|
async def get_version():
|
||||||
return {"version": __version__}
|
from core.vocal_isolation import available as _vocal_isolation
|
||||||
|
return {"version": __version__, "features": {"vocal_isolation": _vocal_isolation()}}
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/changelog", response_class=PlainTextResponse)
|
@router.get("/api/changelog", response_class=PlainTextResponse)
|
||||||
|
|||||||
@ -6,6 +6,7 @@ import glob
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
import threading
|
||||||
import uuid
|
import uuid
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from html import unescape
|
from html import unescape
|
||||||
@ -18,7 +19,7 @@ from fastapi.responses import StreamingResponse
|
|||||||
|
|
||||||
from core.config import _load_settings
|
from core.config import _load_settings
|
||||||
from core.constants import _VOICES_DIR_DEFAULT, _MAX_UPLOAD_BYTES, _ALLOW_PRIVATE_DOWNLOADS, CONFIG_DIR, _BENCHMARK_SENTENCES
|
from core.constants import _VOICES_DIR_DEFAULT, _MAX_UPLOAD_BYTES, _ALLOW_PRIVATE_DOWNLOADS, CONFIG_DIR, _BENCHMARK_SENTENCES
|
||||||
from core.registry import _registry_put, TEMP_DIR
|
from core.registry import _registry_put, _registry_get, TEMP_DIR
|
||||||
from core.validation import _validate_http_url
|
from core.validation import _validate_http_url
|
||||||
from core.audio import _to_wav_24k, _duration
|
from core.audio import _to_wav_24k, _duration
|
||||||
from core.voice import (
|
from core.voice import (
|
||||||
@ -722,6 +723,34 @@ async def download_yt(url: str):
|
|||||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})
|
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/api/isolate-vocals")
|
||||||
|
async def isolate_vocals_route(request: Request):
|
||||||
|
"""Remove background music from a registered temp audio id; the original id stays valid."""
|
||||||
|
from core import vocal_isolation as vi
|
||||||
|
data = await request.json()
|
||||||
|
fid = data.get("id") if isinstance(data, dict) else None
|
||||||
|
if not isinstance(fid, str) or not fid:
|
||||||
|
raise HTTPException(400, "id required")
|
||||||
|
src = _registry_get(fid)
|
||||||
|
if src is None or not src.exists():
|
||||||
|
raise HTTPException(404, "Source file not found")
|
||||||
|
cancel = threading.Event()
|
||||||
|
task = asyncio.create_task(asyncio.to_thread(vi.isolate_vocals, src, cancel))
|
||||||
|
try:
|
||||||
|
while not task.done():
|
||||||
|
await asyncio.wait({task}, timeout=1.0)
|
||||||
|
if not task.done() and await request.is_disconnected():
|
||||||
|
cancel.set()
|
||||||
|
wav = task.result()
|
||||||
|
except vi.VocalIsolationError as e:
|
||||||
|
raise HTTPException(e.status, str(e))
|
||||||
|
except Exception:
|
||||||
|
raise HTTPException(500, "Vocal isolation failed")
|
||||||
|
nid = uuid.uuid4().hex
|
||||||
|
_registry_put(nid, wav)
|
||||||
|
return {"id": nid, "duration": _duration(wav)}
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/elevenlabs/voices")
|
@router.get("/api/elevenlabs/voices")
|
||||||
async def elevenlabs_shared_voices(request: Request):
|
async def elevenlabs_shared_voices(request: Request):
|
||||||
settings = _load_settings()
|
settings = _load_settings()
|
||||||
|
|||||||
6
static/dist/main.min.js
vendored
6
static/dist/main.min.js
vendored
File diff suppressed because one or more lines are too long
34
static/js/vocal-isolation.js
Normal file
34
static/js/vocal-isolation.js
Normal file
@ -0,0 +1,34 @@
|
|||||||
|
// Optional "Remove background music" for URL/YouTube imports. Button appears only when the
|
||||||
|
// server reports features.vocal_isolation. Original stays loadable via the toggle (before/after).
|
||||||
|
function setupVocalIsolation({ btn, toggle, msg, getId, getDur, apply }) {
|
||||||
|
const b = $(btn), t = $(toggle), m = $(msg);
|
||||||
|
let pair = null, showing = 'clean';
|
||||||
|
const say = text => { m.textContent = text; status(text); };
|
||||||
|
fetch('/api/version').then(r => r.json()).then(v => { if (v?.features?.vocal_isolation) b.hidden = false; }).catch(() => {});
|
||||||
|
const label = () => { t.textContent = showing === 'clean' ? 'Play original' : 'Play without music'; };
|
||||||
|
b.addEventListener('click', async () => {
|
||||||
|
const id = getId();
|
||||||
|
if (!id) { say('Load audio first, then remove the background music.'); return; }
|
||||||
|
b.disabled = true; t.hidden = true;
|
||||||
|
say('Removing background music. This can take a few minutes.');
|
||||||
|
try {
|
||||||
|
const r = await fetch('/api/isolate-vocals', { method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({ id }) });
|
||||||
|
const d = await r.json().catch(() => ({}));
|
||||||
|
if (!r.ok) throw new Error(d.detail || r.statusText);
|
||||||
|
pair = { orig:id, origDur:getDur(), clean:d.id, cleanDur:d.duration };
|
||||||
|
showing = 'clean'; apply(d.id, d.duration);
|
||||||
|
t.hidden = false; label();
|
||||||
|
say('Background music removed. Use the button next to it to compare with the original.');
|
||||||
|
} catch (e) {
|
||||||
|
say('Could not remove background music: ' + e.message);
|
||||||
|
toast('Vocal isolation failed: ' + e.message, 'error');
|
||||||
|
} finally { b.disabled = false; }
|
||||||
|
});
|
||||||
|
t.addEventListener('click', () => {
|
||||||
|
if (!pair) return;
|
||||||
|
showing = showing === 'clean' ? 'orig' : 'clean';
|
||||||
|
if (showing === 'clean') apply(pair.clean, pair.cleanDur); else apply(pair.orig, pair.origDur);
|
||||||
|
label();
|
||||||
|
say(showing === 'clean' ? 'Playing the version without background music.' : 'Playing the original audio.');
|
||||||
|
});
|
||||||
|
}
|
||||||
@ -1133,3 +1133,6 @@ Object.assign(root.TTVC.voiceClone, {
|
|||||||
design: { run: runVoiceDesign, buildPayload: voiceDesignPayload },
|
design: { run: runVoiceDesign, buildPayload: voiceDesignPayload },
|
||||||
});
|
});
|
||||||
})(window);
|
})(window);
|
||||||
|
|
||||||
|
setupVocalIsolation({ btn:'yt-iso-btn', toggle:'yt-iso-toggle', msg:'yt-iso-msg',
|
||||||
|
getId:() => currentFileId, getDur:() => parseFloat($('trim-end').value) || 0, apply:(id, dur) => loadAudioId(id, dur) });
|
||||||
|
|||||||
@ -1646,6 +1646,9 @@ $('lib-add-url-btn').addEventListener('click', () => {
|
|||||||
};
|
};
|
||||||
});
|
});
|
||||||
|
|
||||||
|
setupVocalIsolation({ btn:'lib-iso-btn', toggle:'lib-iso-toggle', msg:'lib-iso-msg',
|
||||||
|
getId:() => libAddState.id, getDur:() => libAddState.duration, apply:(id, dur) => loadLibAddAudio(id, dur, 'Audio') });
|
||||||
|
|
||||||
$('lib-add-rec-start').addEventListener('click', async () => {
|
$('lib-add-rec-start').addEventListener('click', async () => {
|
||||||
try {
|
try {
|
||||||
await ensureLibAddMicMonitor();
|
await ensureLibAddMicMonitor();
|
||||||
|
|||||||
@ -11,6 +11,7 @@
|
|||||||
"image-cache",
|
"image-cache",
|
||||||
"integrations",
|
"integrations",
|
||||||
"routing",
|
"routing",
|
||||||
|
"vocal-isolation",
|
||||||
"voice-clone",
|
"voice-clone",
|
||||||
"voice-library",
|
"voice-library",
|
||||||
"tts-preview",
|
"tts-preview",
|
||||||
|
|||||||
@ -83,6 +83,11 @@
|
|||||||
<button class="btn-primary" id="yt-btn">Download</button>
|
<button class="btn-primary" id="yt-btn">Download</button>
|
||||||
</div>
|
</div>
|
||||||
<div id="yt-progress"></div>
|
<div id="yt-progress"></div>
|
||||||
|
<div class="btn-row">
|
||||||
|
<button class="btn-secondary" id="yt-iso-btn" hidden>Remove background music</button>
|
||||||
|
<button class="btn-secondary" id="yt-iso-toggle" hidden></button>
|
||||||
|
</div>
|
||||||
|
<div class="note" id="yt-iso-msg" role="status" aria-live="polite"></div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
</div><!-- /clone-srcpanel -->
|
</div><!-- /clone-srcpanel -->
|
||||||
|
|||||||
@ -232,6 +232,11 @@
|
|||||||
<input type="text" id="lib-add-url" placeholder="Paste a direct MP3/WAV, YouTube, or Aiartes URL">
|
<input type="text" id="lib-add-url" placeholder="Paste a direct MP3/WAV, YouTube, or Aiartes URL">
|
||||||
<button class="btn-primary" id="lib-add-url-btn">Download URL</button>
|
<button class="btn-primary" id="lib-add-url-btn">Download URL</button>
|
||||||
</div>
|
</div>
|
||||||
|
<div class="btn-row">
|
||||||
|
<button class="btn-secondary" id="lib-iso-btn" hidden>Remove background music</button>
|
||||||
|
<button class="btn-secondary" id="lib-iso-toggle" hidden></button>
|
||||||
|
</div>
|
||||||
|
<div class="note" id="lib-iso-msg" role="status" aria-live="polite"></div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="lib-add-source-box lib-add-record-box">
|
<div class="lib-add-source-box lib-add-record-box">
|
||||||
|
|||||||
119
tests/test_vocal_isolation.py
Normal file
119
tests/test_vocal_isolation.py
Normal file
@ -0,0 +1,119 @@
|
|||||||
|
"""vocal isolation: flag off, unavailable, success, timeout, cleanup (Demucs mocked)."""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||||
|
|
||||||
|
from pydub import AudioSegment
|
||||||
|
from pydub.generators import Sine
|
||||||
|
|
||||||
|
from core import vocal_isolation as vi
|
||||||
|
from core.registry import TEMP_DIR
|
||||||
|
|
||||||
|
|
||||||
|
def wav(path, ms=1000):
|
||||||
|
Sine(300).to_audio_segment(ms).export(str(path), format="wav")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
class FakeProc:
|
||||||
|
"""Stands in for demucs: writes <out>/htdemucs/<stem>/vocals.wav, then exits."""
|
||||||
|
def __init__(self, args, **kw):
|
||||||
|
self.never_ends = FakeProc.never_ends
|
||||||
|
out = Path(args[args.index("-o") + 1])
|
||||||
|
if not self.never_ends and FakeProc.write:
|
||||||
|
d = out / "htdemucs" / Path(args[-1]).stem
|
||||||
|
d.mkdir(parents=True)
|
||||||
|
wav(d / "vocals.wav")
|
||||||
|
self.returncode = None if self.never_ends else FakeProc.rc
|
||||||
|
never_ends = False
|
||||||
|
write = True
|
||||||
|
rc = 0
|
||||||
|
def poll(self):
|
||||||
|
return self.returncode
|
||||||
|
def kill(self):
|
||||||
|
self.returncode = -9
|
||||||
|
def wait(self):
|
||||||
|
return self.returncode
|
||||||
|
|
||||||
|
|
||||||
|
def env(**kw):
|
||||||
|
base = {"VOCAL_ISOLATION": "1", "VOCAL_ISOLATION_CMD": "/bin/true"}
|
||||||
|
base.update(kw)
|
||||||
|
return mock.patch.dict(os.environ, base)
|
||||||
|
|
||||||
|
|
||||||
|
class VocalIsolationTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.src = wav(Path(tempfile.mkdtemp()) / "in.wav")
|
||||||
|
FakeProc.never_ends, FakeProc.write, FakeProc.rc = False, True, 0
|
||||||
|
self.w = mock.patch.object(vi, "_weights_present", return_value=True)
|
||||||
|
self.w.start()
|
||||||
|
self.addCleanup(self.w.stop)
|
||||||
|
|
||||||
|
def test_flag_off(self):
|
||||||
|
with mock.patch.dict(os.environ, {"VOCAL_ISOLATION": ""}):
|
||||||
|
self.assertFalse(vi.available())
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src)
|
||||||
|
self.assertEqual(c.exception.status, 503)
|
||||||
|
|
||||||
|
def test_unavailable_without_weights_or_binary(self):
|
||||||
|
with env(), mock.patch.object(vi, "_weights_present", return_value=False):
|
||||||
|
self.assertFalse(vi.available())
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src)
|
||||||
|
self.assertEqual(c.exception.status, 503)
|
||||||
|
with env(VOCAL_ISOLATION_CMD=""), mock.patch("shutil.which", return_value=None):
|
||||||
|
self.assertFalse(vi.available())
|
||||||
|
|
||||||
|
def test_success_and_cleanup(self):
|
||||||
|
with env(), mock.patch("subprocess.Popen", FakeProc):
|
||||||
|
self.assertTrue(vi.available())
|
||||||
|
out = vi.isolate_vocals(self.src)
|
||||||
|
self.assertTrue(out.exists())
|
||||||
|
self.assertEqual(AudioSegment.from_file(str(out)).frame_rate, 24000)
|
||||||
|
self.assertTrue(self.src.exists()) # original untouched
|
||||||
|
self.assertEqual(list(TEMP_DIR.glob("vocal_*")), [])
|
||||||
|
self.assertEqual(list(TEMP_DIR.glob("*_vocals.*")), [])
|
||||||
|
|
||||||
|
def test_timeout_kills_and_cleans(self):
|
||||||
|
FakeProc.never_ends = True
|
||||||
|
with env(VOCAL_ISOLATION_TIMEOUT="0.3"), mock.patch("subprocess.Popen", FakeProc):
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src)
|
||||||
|
self.assertEqual(c.exception.status, 504)
|
||||||
|
self.assertEqual(list(TEMP_DIR.glob("vocal_*")), [])
|
||||||
|
|
||||||
|
def test_cancel(self):
|
||||||
|
import threading
|
||||||
|
FakeProc.never_ends = True
|
||||||
|
ev = threading.Event(); ev.set()
|
||||||
|
with env(), mock.patch("subprocess.Popen", FakeProc):
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src, ev)
|
||||||
|
self.assertEqual(c.exception.status, 499)
|
||||||
|
|
||||||
|
def test_too_long_and_failure_and_busy(self):
|
||||||
|
with env(VOCAL_ISOLATION_MAX_SECONDS="0.5"):
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src)
|
||||||
|
self.assertEqual(c.exception.status, 413)
|
||||||
|
FakeProc.write, FakeProc.rc = False, 1
|
||||||
|
with env(), mock.patch("subprocess.Popen", FakeProc):
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src)
|
||||||
|
self.assertEqual(c.exception.status, 500)
|
||||||
|
self.assertEqual(list(TEMP_DIR.glob("vocal_*")), [])
|
||||||
|
with env(), vi._busy:
|
||||||
|
with self.assertRaises(vi.VocalIsolationError) as c:
|
||||||
|
vi.isolate_vocals(self.src)
|
||||||
|
self.assertEqual(c.exception.status, 409)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
51
tests/vocal-isolation.spec.js
Normal file
51
tests/vocal-isolation.spec.js
Normal file
@ -0,0 +1,51 @@
|
|||||||
|
// @ts-check
|
||||||
|
// "Remove background music" in the URL import card (all /api routes mocked).
|
||||||
|
const { test, expect } = require('@playwright/test');
|
||||||
|
|
||||||
|
async function open(page, available) {
|
||||||
|
await page.route('**/api/**', route => route.request().method() === 'GET' ? route.fulfill({ json: {} }) : route.fulfill({ json: { ok: true } }));
|
||||||
|
await page.route('**/api/voices*', route => route.fulfill({ json: [] }));
|
||||||
|
await page.route('**/api/version', route => route.fulfill({ json: { version: 'x', features: { vocal_isolation: available } } }));
|
||||||
|
await page.route('**/api/audio/*', route => route.fulfill({ status: 404, body: '' }));
|
||||||
|
await page.route('**/static/dist/main.min.js*', r => r.abort());
|
||||||
|
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||||
|
await page.waitForFunction(() => typeof window.navTo === 'function', null, { timeout: 15_000 });
|
||||||
|
await page.waitForSelector('#app-skeleton', { state: 'detached', timeout: 15_000 });
|
||||||
|
}
|
||||||
|
|
||||||
|
test('button hidden when the server cannot isolate', async ({ page }) => {
|
||||||
|
await open(page, false);
|
||||||
|
await expect(page.locator('#lib-iso-btn')).toHaveAttribute('hidden', '');
|
||||||
|
await expect(page.locator('#yt-iso-btn')).toHaveAttribute('hidden', '');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('isolate, compare with original, and show errors', async ({ page }) => {
|
||||||
|
await open(page, true);
|
||||||
|
/** @type {any[]} */ const calls = [];
|
||||||
|
let fail = false;
|
||||||
|
await page.route('**/api/isolate-vocals', route => {
|
||||||
|
calls.push(route.request().postDataJSON());
|
||||||
|
return fail ? route.fulfill({ status: 503, json: { detail: 'Vocal isolation is turned off' } })
|
||||||
|
: route.fulfill({ json: { id: 'clean1', duration: 4 } });
|
||||||
|
});
|
||||||
|
await page.evaluate(() => window.navTo('s-voices'));
|
||||||
|
const btn = page.locator('#lib-iso-btn');
|
||||||
|
await expect(btn).not.toHaveAttribute('hidden', '');
|
||||||
|
const msg = page.locator('#lib-iso-msg');
|
||||||
|
await expect(msg).toHaveAttribute('aria-live', 'polite');
|
||||||
|
await btn.evaluate(el => el.click());
|
||||||
|
await expect(msg).toContainText('Load audio first');
|
||||||
|
await page.evaluate(() => { libAddState.id = 'orig1'; libAddState.duration = 6; });
|
||||||
|
await btn.evaluate(el => el.click());
|
||||||
|
await expect(msg).toContainText('Background music removed');
|
||||||
|
expect(calls).toEqual([{ id: 'orig1' }]);
|
||||||
|
expect(await page.evaluate(() => libAddState.id)).toBe('clean1');
|
||||||
|
const toggle = page.locator('#lib-iso-toggle');
|
||||||
|
await expect(toggle).toHaveText('Play original');
|
||||||
|
await toggle.evaluate(el => el.click());
|
||||||
|
expect(await page.evaluate(() => libAddState.id)).toBe('orig1');
|
||||||
|
await expect(toggle).toHaveText('Play without music');
|
||||||
|
fail = true;
|
||||||
|
await btn.evaluate(el => el.click());
|
||||||
|
await expect(msg).toContainText('turned off');
|
||||||
|
});
|
||||||
Loading…
Reference in New Issue
Block a user