Release v1.22.3
This commit is contained in:
parent
fe807d1682
commit
a8f8779736
15
CHANGELOG.md
15
CHANGELOG.md
@ -5,6 +5,18 @@ Follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) · versioned wi
|
||||
|
||||
---
|
||||
|
||||
## [1.22.3] — 2026-09-04
|
||||
|
||||
### Changed
|
||||
- **P0 Request Protection.** Added HTTP request-size ceilings, reduced the default upload limit from 1 GB to 256 MB,
|
||||
and added process-local mutation rate limiting with stricter limits for destructive paths.
|
||||
- **Credential Transport.** Provider API keys used for model probes are now sent in request headers instead of URL query
|
||||
strings, keeping them out of browser history and ordinary access/proxy logs.
|
||||
- **Recoverable Character and Rehearsal Changes.** Character updates/deletes and rehearsal recasts, updates, compaction
|
||||
and deletes now create atomic JSON snapshots before mutation. Missing delete targets return 404 instead of success.
|
||||
- **SSRF Protection.** User-supplied audio imports, picture URLs and generic screenplay URLs now use centralized URL
|
||||
validation, with private-network access available only through the explicit local-download setting.
|
||||
|
||||
## [1.22.2] — 2026-09-03
|
||||
|
||||
### Added
|
||||
@ -3235,7 +3247,8 @@ Initial feature-complete release.
|
||||
|
||||
---
|
||||
|
||||
[Unreleased]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.22.2...HEAD
|
||||
[Unreleased]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.22.3...HEAD
|
||||
[1.22.3]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.22.2...v1.22.3
|
||||
[1.22.2]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.22.1...v1.22.2
|
||||
[1.14.24]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.14.23...v1.14.24
|
||||
[1.14.23]: https://github.com/mARTin-B78/tts-voice-creator-clone-and-design-2/compare/v1.14.22...v1.14.23
|
||||
|
||||
109
TODO.md
109
TODO.md
@ -8,6 +8,77 @@ Priority levels:
|
||||
- **P1 — high impact:** improves memory use, responsiveness, and maintainability.
|
||||
- **P2 — quality:** improves diagnostics, polish, and long-term operations.
|
||||
|
||||
## Prioritized execution backlog
|
||||
|
||||
Effort estimates are for one developer familiar with this codebase: **S** = a few
|
||||
hours, **M** = 1–2 days, **L** = several days, **XL** = multi-week or iterative.
|
||||
Within each priority, tasks are ordered from smallest to largest effort. Complete
|
||||
the first block before investing in architecture or polish.
|
||||
|
||||
### P0 — do first
|
||||
|
||||
- [ ] **S — Stop API keys from reaching frontend JavaScript or logs.** Audit responses,
|
||||
browser storage, error paths and debug output.
|
||||
- [ ] **S — Validate uploaded filenames and paths.** Reject traversal, unsafe names and
|
||||
writes outside the intended data directories.
|
||||
- [ ] **M — Add upload, request-size, temporary-file and concurrent-job limits.** Apply
|
||||
endpoint-specific caps and clean up intermediates on failure.
|
||||
- [ ] **M — Add automatic backups before bulk deletes, voice moves and recasts.** Make
|
||||
recovery explicit and verify that backups can be restored.
|
||||
- [ ] **M — Strictly validate custom URLs and prevent SSRF.** Allow only intended
|
||||
schemes, hosts and network destinations for provider/container probes.
|
||||
- [ ] **M — Add authentication and rate limits for destructive/API endpoints.** Protect
|
||||
settings, file operations, Docker controls and bulk mutations.
|
||||
- [ ] **M — Validate character sheets and voice metadata with schemas.** Reject or repair
|
||||
malformed records at API boundaries.
|
||||
- [ ] **L — Add database migrations and prompt/generation history.** Preserve old record
|
||||
shapes and allow bad regenerations to be reverted.
|
||||
- [ ] **L — Harden container and Docker access.** Run as non-root with dropped
|
||||
capabilities/read-only filesystems where practical, and isolate Docker socket access
|
||||
behind a minimal privileged service.
|
||||
|
||||
### P1 — next, highest user impact
|
||||
|
||||
- [ ] **S — Re-measure audiobook segmentation against `cast_run3.json`.** Decide whether
|
||||
to keep, tune or revert `_audiobookMergeAdjacentDialogue` using current evidence.
|
||||
- [ ] **S — Finish the remaining Playwright smoke checks.** Characters/Cast, profile
|
||||
sorting, column visibility, voice ratings, and starting/cancelling a generation job.
|
||||
- [ ] **M — Fix the Schwester Larissa unclosed-guillemet split.** Add a regression test
|
||||
for the epigraph and verify the parser does not split mid-sentence.
|
||||
- [ ] **M — Make profile portraits and character sheets use one canonical identity.**
|
||||
Share age, hair, eyes, face, build, skin tone, marks and era; version the prompt and
|
||||
detect stale or inconsistent images.
|
||||
- [ ] **M — Add bounded duplicate-generation protection.** Ensure double-clicks and
|
||||
retried requests cannot create duplicate voice, image or casting jobs.
|
||||
- [ ] **M — Improve large-table performance.** Virtualize/paginate character and voice
|
||||
tables, update only affected rows, and disable preloading on low-memory devices.
|
||||
- [ ] **M — Add explicit cache invalidation and eviction.** Cache `/api/voices` safely
|
||||
and add LRU cleanup for audio/image entries and object URLs.
|
||||
- [ ] **M — Add constrained-device validation.** Test with Chrome CPU/memory throttling,
|
||||
a 4 GB VM/container, and measure browser, Docker and temporary-disk usage.
|
||||
- [ ] **L — Add background jobs and bounded queues.** Give voice design, transcription,
|
||||
casting and image generation IDs, progress, cancellation, timeout, retry and
|
||||
idempotency support.
|
||||
- [ ] **L — Run the Book 2 generalization regression.** Confirm speaker-attribution and
|
||||
confidence rules are not overfit to *Der Sturm*.
|
||||
- [ ] **L — Add character forms inside one character record.** Support multiple
|
||||
appearances while preserving one voice and one attribution target.
|
||||
- [ ] **L — Modularize the frontend.** Split Audiobook, Rehearser and styles, replace
|
||||
broad globals with namespaces, and consolidate the loader/module manifest.
|
||||
|
||||
### P2 — after behavior and safety stabilize
|
||||
|
||||
- [ ] **M — Add structured logs with request/job IDs.** Report provider, endpoint,
|
||||
duration and failure reason without logging prompts, keys or book contents.
|
||||
- [ ] **M — Separate provider health from configuration status.** Show actionable
|
||||
connectivity and authentication errors.
|
||||
- [ ] **M — Add visible job history and retry controls.** Expose recent failures and
|
||||
safe retry state to users.
|
||||
- [ ] **L — Add memory diagnostics.** Show browser heap where supported, rendered rows,
|
||||
cache size, active jobs, server/container memory and recent operation sizes.
|
||||
- [ ] **L — Add linting and duplicate-function checks.** Run them in CI alongside the
|
||||
existing compile, bundle and SonarQube checks.
|
||||
|
||||
## P0 — Testing, data safety, and resource limits
|
||||
|
||||
- [x] Establish a Playwright baseline before large refactors (app shell, bundle, i18n, clone tabs, and Rehearser parsing; installed Chromium supported).
|
||||
@ -204,6 +275,44 @@ LLM writes it as one of the 53 character-sheet fields, while
|
||||
facts the concept sheet uses. The deterministic one keeps both images consistent; the
|
||||
LLM-written one omits identity anchors (age, hair, build) so the model invents them.
|
||||
|
||||
### Status — partially working
|
||||
|
||||
**Working:**
|
||||
|
||||
- Character records, structured profile fields and portrait images are saved and loaded
|
||||
independently without losing the stored image data.
|
||||
- The concept-sheet path already builds from structured character facts and is the more
|
||||
consistent identity source.
|
||||
- Portrait generation now uses baseline appearance guidance, explicit head-and-shoulders
|
||||
framing, neutral backgrounds and book-period grounding.
|
||||
- Existing scene/style/background contamination in saved portrait prompts is filtered in
|
||||
the portrait path.
|
||||
|
||||
**Not working reliably:**
|
||||
|
||||
- The profile/avatar portrait and the character concept sheet can still depict different
|
||||
hair colour, age, facial features, build or overall identity.
|
||||
- Character-sheet extraction can write an LLM-authored `image_prompt` that lacks the
|
||||
identity anchors used by the deterministic concept-sheet prompt.
|
||||
- Regenerating a portrait does not guarantee that it uses exactly the same identity
|
||||
description as the concept sheet.
|
||||
- Evidence-window profile regeneration can produce behaviour or actions in `physical`
|
||||
instead of stable appearance details.
|
||||
|
||||
**Needs to be fixed:**
|
||||
|
||||
- Make one canonical identity description the input for both avatar portraits and concept
|
||||
sheets: age range, hair, eyes, face, build, skin tone, distinctive marks and clothing
|
||||
era.
|
||||
- Remove `image_prompt` from the general sheet-extraction output, or overwrite it only
|
||||
after extraction using the canonical structured fields.
|
||||
- Store the resolved identity prompt/version with each generated image so old images can
|
||||
be identified as stale and regenerated deterministically.
|
||||
- Add a consistency check or review indicator when the avatar and concept-sheet images
|
||||
were generated from different prompt versions.
|
||||
- Add an automated test covering extraction → prompt generation → avatar and concept
|
||||
sheet generation, asserting that both image requests contain the same identity anchors.
|
||||
|
||||
The app's own flow already calls `/api/character-generate-prompts` after extraction, so
|
||||
**this is mainly a process trap**: any script that regenerates sheets must call that
|
||||
endpoint too, or it leaves an LLM-authored prompt in place. Consider removing
|
||||
|
||||
@ -45,7 +45,13 @@ _TTS_CONTAINERS_RAW = os.environ.get("TTS_CONTAINER_NAMES", "") # comma-separa
|
||||
_VOICE_DESIGN_MODEL = os.environ.get("VOICE_DESIGN_MODEL", "Qwen3-TTS-12Hz-1.7B-VoiceDesign")
|
||||
_VOICE_TARGET_DBFS = float(os.environ.get("VOICE_TARGET_DBFS", "-20.0"))
|
||||
_VOICE_PEAK_DBFS = float(os.environ.get("VOICE_PEAK_DBFS", "-1.0"))
|
||||
_MAX_UPLOAD_BYTES = int(os.environ.get("MAX_UPLOAD_MB", "1024")) * 1024 * 1024
|
||||
# Keep uploads bounded by default. Individual endpoints apply smaller limits for
|
||||
# pictures and sound assets; MAX_UPLOAD_MB remains configurable for unusually
|
||||
# large source audio or PDFs without making an accidental 1 GB request the
|
||||
# default for every multipart endpoint.
|
||||
_MAX_UPLOAD_BYTES = int(os.environ.get("MAX_UPLOAD_MB", "256")) * 1024 * 1024
|
||||
_MAX_JSON_BYTES = int(os.environ.get("MAX_JSON_MB", "32")) * 1024 * 1024
|
||||
_MAX_RAW_REQUEST_BYTES = int(os.environ.get("MAX_RAW_REQUEST_MB", "256")) * 1024 * 1024
|
||||
_MAX_PICTURE_BYTES = int(os.environ.get("MAX_PICTURE_MB", "10")) * 1024 * 1024
|
||||
_MAX_SOUND_BYTES = int(os.environ.get("MAX_SOUND_MB", "25")) * 1024 * 1024
|
||||
_MAX_TTS_OUTPUT_SECONDS = float(os.environ.get("MAX_TTS_OUTPUT_SECONDS", "60"))
|
||||
|
||||
@ -9,7 +9,10 @@ Designed for easy migration to PostgreSQL / Supabase later:
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@ -138,6 +141,63 @@ def _reh_title_key(title: Any) -> str:
|
||||
return " ".join(str(title or "").split()).casefold()
|
||||
|
||||
|
||||
def _backup_character_record(record: dict, reason: str) -> Path:
|
||||
"""Write a recoverable snapshot before changing or deleting a character.
|
||||
|
||||
Character images and sheets can be large, so snapshots are individual JSON
|
||||
files rather than an ever-growing database column. The temporary file and
|
||||
replace keep a partially-written backup from looking valid after a crash.
|
||||
A failed backup deliberately aborts the mutation: destructive operations
|
||||
must not proceed when recovery cannot be prepared.
|
||||
"""
|
||||
backup_dir = CONFIG_DIR / "backups" / "characters"
|
||||
backup_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
|
||||
safe_reason = "".join(c if c.isalnum() or c in "-_" else "_" for c in str(reason or "update"))[:32]
|
||||
target = backup_dir / f"{stamp}-{safe_reason}-{uuid.uuid4().hex[:8]}.json"
|
||||
payload = {"saved_at": datetime.now(timezone.utc).isoformat(), "reason": reason, "record": record}
|
||||
fd, tmp_name = tempfile.mkstemp(prefix=f".{target.name}.", suffix=".tmp", dir=backup_dir)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=False)
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(tmp_name, target)
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_name)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
return target
|
||||
|
||||
|
||||
def _backup_rehearsal_record(record: dict, reason: str) -> Path:
|
||||
"""Snapshot a rehearsal before recast, overwrite, compaction or delete."""
|
||||
backup_dir = CONFIG_DIR / "backups" / "rehearsals"
|
||||
backup_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
|
||||
safe_reason = "".join(c if c.isalnum() or c in "-_" else "_" for c in str(reason or "update"))[:32]
|
||||
target = backup_dir / f"{stamp}-{safe_reason}-{uuid.uuid4().hex[:8]}.json"
|
||||
payload = {"saved_at": datetime.now(timezone.utc).isoformat(), "reason": reason, "record": record}
|
||||
fd, tmp_name = tempfile.mkstemp(prefix=f".{target.name}.", suffix=".tmp", dir=backup_dir)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=False)
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(tmp_name, target)
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_name)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
return target
|
||||
|
||||
|
||||
# ── Characters ────────────────────────────────────────────────────────────────
|
||||
|
||||
def char_get_all() -> list[dict]:
|
||||
@ -153,6 +213,7 @@ def char_get(char_id: str) -> dict | None:
|
||||
|
||||
|
||||
def char_put(rec: dict) -> dict:
|
||||
existing = char_get(rec.get("id", "")) if rec.get("id") else None
|
||||
image = rec.get("image")
|
||||
sheet = rec.get("sheet") or {}
|
||||
concept_art_image = sheet.get("concept_art_image")
|
||||
@ -169,12 +230,13 @@ def char_put(rec: dict) -> dict:
|
||||
# silently overwrite the real stored reference image with this
|
||||
# placeholder URL string, permanently losing it. Keep whatever is
|
||||
# already on the row for either field instead.
|
||||
existing = char_get(rec.get("id", ""))
|
||||
if isinstance(image, str) and image.startswith("/api/characters/"):
|
||||
image = existing.get("image") if existing else None
|
||||
if isinstance(concept_art_image, str) and concept_art_image.startswith("/api/characters/"):
|
||||
sheet = {**sheet, "concept_art_image": (existing.get("sheet") or {}).get("concept_art_image") if existing else None}
|
||||
rec = {**rec, "image": image, "sheet": sheet}
|
||||
if existing is not None and existing != rec:
|
||||
_backup_character_record(existing, "update")
|
||||
with _open() as conn:
|
||||
conn.execute("""
|
||||
INSERT INTO characters (id, book, name, tags, voice, image, sheet, analysis, created, updated)
|
||||
@ -200,6 +262,10 @@ def char_put(rec: dict) -> dict:
|
||||
|
||||
|
||||
def char_delete(char_id: str) -> bool:
|
||||
existing = char_get(char_id)
|
||||
if existing is None:
|
||||
return False
|
||||
_backup_character_record(existing, "delete")
|
||||
with _open() as conn:
|
||||
conn.execute("DELETE FROM characters WHERE id=?", (char_id,))
|
||||
conn.commit()
|
||||
@ -259,6 +325,9 @@ def reh_compact_titles() -> int:
|
||||
else:
|
||||
seen.add(key)
|
||||
for reh_id in delete_ids:
|
||||
old = reh_get(reh_id)
|
||||
if old is not None:
|
||||
_backup_rehearsal_record(old, "compact")
|
||||
conn.execute("DELETE FROM rehearsals WHERE id=?", (reh_id,))
|
||||
if delete_ids:
|
||||
conn.commit()
|
||||
@ -277,6 +346,7 @@ def reh_put(rec: dict) -> dict:
|
||||
# an existing rehearsal rather than creating a new library copy.
|
||||
rec = dict(rec)
|
||||
rec["created"] = existing.get("created") or rec.get("created")
|
||||
existing_by_id = reh_get(int(reh_id)) if reh_id else None
|
||||
row = {
|
||||
"title": title,
|
||||
"script": rec.get("script", ""),
|
||||
@ -293,6 +363,11 @@ def reh_put(rec: dict) -> dict:
|
||||
}
|
||||
with _open() as conn:
|
||||
if reh_id:
|
||||
if existing_by_id is not None:
|
||||
candidate = dict(rec)
|
||||
candidate["id"] = int(reh_id)
|
||||
if candidate != existing_by_id:
|
||||
_backup_rehearsal_record(existing_by_id, "update")
|
||||
row["id"] = int(reh_id)
|
||||
conn.execute("""
|
||||
INSERT INTO rehearsals
|
||||
@ -326,6 +401,10 @@ def reh_put(rec: dict) -> dict:
|
||||
|
||||
|
||||
def reh_delete(reh_id: int) -> bool:
|
||||
existing = reh_get(reh_id)
|
||||
if existing is None:
|
||||
return False
|
||||
_backup_rehearsal_record(existing, "delete")
|
||||
with _open() as conn:
|
||||
conn.execute("DELETE FROM rehearsals WHERE id=?", (reh_id,))
|
||||
conn.commit()
|
||||
|
||||
@ -117,8 +117,9 @@ async def characters_put(char_id: str, request: Request):
|
||||
|
||||
@router.delete("/api/characters/{char_id:path}")
|
||||
async def characters_delete(char_id: str):
|
||||
char_delete(char_id)
|
||||
return {"ok": True}
|
||||
if not char_delete(char_id):
|
||||
raise HTTPException(404, "Character not found")
|
||||
return {"ok": True, "backup_created": True}
|
||||
|
||||
|
||||
@router.post("/api/characters/migrate")
|
||||
|
||||
@ -19,7 +19,7 @@ from urllib.parse import quote
|
||||
import requests
|
||||
from typing import Optional
|
||||
from PIL import Image, ImageDraw, ImageOps
|
||||
from fastapi import APIRouter, File, Form, HTTPException, Request, UploadFile
|
||||
from fastapi import APIRouter, File, Form, Header, HTTPException, Request, UploadFile
|
||||
from fastapi.responses import Response, StreamingResponse
|
||||
|
||||
from core.config import _load_settings, _save_settings, _clean_preview_backend
|
||||
@ -4321,12 +4321,15 @@ async def mcp_sse(request: Request):
|
||||
# ── Conversation playground ───────────────────────────────────────────────────
|
||||
|
||||
@router.get("/api/conversation/llm-models")
|
||||
async def conversation_llm_models(url: str = "", api_key: str = ""):
|
||||
async def conversation_llm_models(
|
||||
url: str = "",
|
||||
provider_api_key: str = Header("", alias="X-Provider-API-Key"),
|
||||
):
|
||||
"""List models from a local LLM endpoint (Ollama / vLLM / LM Studio)."""
|
||||
from core.validation import _validate_http_url
|
||||
settings = _load_settings()
|
||||
base = (url or settings.get("llm_url") or "http://localhost:11434/v1").rstrip("/")
|
||||
key_to_use = api_key or settings.get("llm_api_key") or "sk-dummy-key"
|
||||
key_to_use = provider_api_key or settings.get("llm_api_key") or "sk-dummy-key"
|
||||
try:
|
||||
base = _validate_http_url(base, allow_private=True)
|
||||
r = requests.get(f"{base}/models", timeout=5, headers={"Authorization": f"Bearer {key_to_use}"})
|
||||
|
||||
@ -7,7 +7,7 @@ from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
import requests
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi import APIRouter, Header, HTTPException
|
||||
|
||||
from core.docker_client import _docker_get_json, _docker_post
|
||||
from core.validation import _normalize_service_url
|
||||
@ -120,14 +120,14 @@ async def restart_local_container(name: str):
|
||||
|
||||
|
||||
@router.get("/api/probe-url")
|
||||
async def probe_url(url: str, type: str = "", api_key: str = ""):
|
||||
async def probe_url(url: str, type: str = "", provider_api_key: str = Header("", alias="X-Provider-API-Key")):
|
||||
"""Server-side API probe — checks service-specific endpoints and validates JSON responses."""
|
||||
base = _normalize_service_url(url).rstrip("/")
|
||||
for _suffix in ("/v1", "/api/v1"):
|
||||
if base.endswith(_suffix):
|
||||
base = base[: -len(_suffix)]
|
||||
break
|
||||
key_to_use = api_key if api_key else "sk-dummy-key"
|
||||
key_to_use = provider_api_key if provider_api_key else "sk-dummy-key"
|
||||
# Anthropic doesn't speak the OpenAI-style Bearer auth every other card
|
||||
# here uses — it needs x-api-key + an anthropic-version header, so the
|
||||
# generic Bearer probe below would 401 even with a perfectly valid key.
|
||||
@ -177,7 +177,7 @@ async def probe_url(url: str, type: str = "", api_key: str = ""):
|
||||
return {"ok": False, "error": last_err}
|
||||
|
||||
@router.get("/api/engine-models")
|
||||
async def engine_models(url: str, type: str = "", api_key: str = ""):
|
||||
async def engine_models(url: str, type: str = "", provider_api_key: str = Header("", alias="X-Provider-API-Key")):
|
||||
"""Return model IDs from an OpenAI-style engine URL without exposing host-only URLs to the browser."""
|
||||
base = _normalize_service_url(url).rstrip("/")
|
||||
if not base:
|
||||
@ -192,7 +192,7 @@ async def engine_models(url: str, type: str = "", api_key: str = ""):
|
||||
last_err = "No response"
|
||||
seen: set[str] = set()
|
||||
models: list[str] = []
|
||||
key_to_use = api_key if api_key else "sk-dummy-key"
|
||||
key_to_use = provider_api_key if provider_api_key else "sk-dummy-key"
|
||||
for endpoint in candidates:
|
||||
try:
|
||||
r = requests.get(endpoint, timeout=6, headers={"Authorization": f"Bearer {key_to_use}", "User-Agent": "TTS-Voice-Creator/models"})
|
||||
|
||||
@ -12,7 +12,7 @@ from fastapi import APIRouter, File, Form, HTTPException, Request, UploadFile
|
||||
from fastapi.responses import FileResponse, Response
|
||||
|
||||
from core.config import _load_settings
|
||||
from core.constants import _VOICES_DIR_DEFAULT, _OUTPUT_DIR_DEFAULT, _VOICE_TARGET_DBFS, _VOICE_PEAK_DBFS, _BENCHMARK_SENTENCES, _MAX_TTS_OUTPUT_SECONDS, _MAX_UPLOAD_BYTES, _MAX_PICTURE_BYTES, _MAX_SOUND_BYTES
|
||||
from core.constants import _VOICES_DIR_DEFAULT, _OUTPUT_DIR_DEFAULT, _VOICE_TARGET_DBFS, _VOICE_PEAK_DBFS, _BENCHMARK_SENTENCES, _MAX_TTS_OUTPUT_SECONDS, _MAX_UPLOAD_BYTES, _MAX_PICTURE_BYTES, _MAX_SOUND_BYTES, _ALLOW_PRIVATE_DOWNLOADS
|
||||
from core.registry import _registry_put, _registry_get, TEMP_DIR
|
||||
from core.validation import _copy_limited, _safe_child_path, _validate_http_url
|
||||
from core.audio import (
|
||||
@ -910,6 +910,7 @@ async def upload_picture_url(request: Request):
|
||||
image_url = str(data.get("image_url") or "").strip()
|
||||
if not image_url or not re.match(r"^https?://", image_url, re.I):
|
||||
raise HTTPException(400, "Expected an http(s) URL")
|
||||
image_url = _validate_http_url(image_url, allow_private=_ALLOW_PRIVATE_DOWNLOADS)
|
||||
if not voice_id or not re.match(r"^[A-Za-z0-9_\-\.]+$", voice_id):
|
||||
raise HTTPException(400, "Invalid voice ID")
|
||||
|
||||
|
||||
@ -57,8 +57,9 @@ async def rehearsals_put(reh_id: int, request: Request):
|
||||
|
||||
@router.delete("/api/rehearsals/{reh_id:int}")
|
||||
async def rehearsals_delete(reh_id: int):
|
||||
reh_delete(reh_id)
|
||||
return {"ok": True}
|
||||
if not reh_delete(reh_id):
|
||||
raise HTTPException(404, "Rehearsal not found")
|
||||
return {"ok": True, "backup_created": True}
|
||||
|
||||
|
||||
@router.post("/api/rehearsals/migrate")
|
||||
|
||||
@ -139,9 +139,10 @@ _GDRIVE_MIME_EXT = {
|
||||
|
||||
def _source_import_url(value: str) -> str:
|
||||
raw = str(value or "").strip()
|
||||
if not re.match(r"^https?://", raw, re.I):
|
||||
raise HTTPException(400, "Expected an http(s) URL")
|
||||
return raw
|
||||
# All user-selected audio downloads go through this helper. Do not allow
|
||||
# arbitrary localhost/private-network targets unless the operator has
|
||||
# explicitly opted into that behavior for a local deployment.
|
||||
return _validate_http_url(raw, allow_private=_ALLOW_PRIVATE_DOWNLOADS)
|
||||
|
||||
|
||||
def _source_download_limit(name: str, fallback_mb: int) -> int:
|
||||
@ -979,7 +980,7 @@ async def fetch_web_script(request: Request):
|
||||
raise HTTPException(502, f"Fetch failed: {e}")
|
||||
fetch_url = resp.url
|
||||
else:
|
||||
fetch_url = raw_url
|
||||
fetch_url = _validate_http_url(raw_url, allow_private=False)
|
||||
headers = _SCRAPE_HEADERS
|
||||
try:
|
||||
resp = await asyncio.to_thread(
|
||||
|
||||
97
server.py
97
server.py
@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from logging.handlers import RotatingFileHandler
|
||||
from pathlib import Path
|
||||
from contextlib import asynccontextmanager
|
||||
@ -14,7 +15,7 @@ from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.gzip import GZipMiddleware
|
||||
from starlette.datastructures import MutableHeaders
|
||||
|
||||
from core.constants import STATIC_DIR, _BufferHandler
|
||||
from core.constants import STATIC_DIR, _BufferHandler, _MAX_JSON_BYTES, _MAX_UPLOAD_BYTES, _MAX_RAW_REQUEST_BYTES
|
||||
from core.config import _load_settings, _ensure_external_api_key
|
||||
from core.voice_index import refresh_voice_index_background
|
||||
from routes import admin, settings, library, stt, sources, docker, tts, conversation, reader, characters, rehearsals_db
|
||||
@ -46,6 +47,100 @@ app = FastAPI(title="TTS Voice Creator - Clone and Design", lifespan=lifespan)
|
||||
# large JS bundle and voice-list JSON. Skips already-compressed audio/images.
|
||||
app.add_middleware(GZipMiddleware, minimum_size=1024)
|
||||
|
||||
# ── Request-size guard ────────────────────────────────────────────────────────
|
||||
# FastAPI's UploadFile handlers stream individual files, but JSON and raw-body
|
||||
# endpoints otherwise accept an unbounded Content-Length before their handlers
|
||||
# get a chance to validate it. Reject oversized requests early. The existing
|
||||
# per-file _copy_limited checks remain authoritative for multipart contents.
|
||||
class RequestSizeLimitMiddleware:
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
headers = {k.decode("latin-1").lower(): v.decode("latin-1") for k, v in scope.get("headers", [])}
|
||||
raw_length = headers.get("content-length", "")
|
||||
try:
|
||||
content_length = int(raw_length) if raw_length else 0
|
||||
except ValueError:
|
||||
content_length = 0
|
||||
content_type = headers.get("content-type", "").lower()
|
||||
if content_type.startswith("application/json"):
|
||||
limit = _MAX_JSON_BYTES
|
||||
elif content_type.startswith("multipart/"):
|
||||
# Multipart framing adds a small amount around each uploaded file.
|
||||
limit = _MAX_UPLOAD_BYTES + 8 * 1024 * 1024
|
||||
else:
|
||||
limit = _MAX_RAW_REQUEST_BYTES
|
||||
if content_length > limit:
|
||||
body = b'{"detail":"Request body is too large"}'
|
||||
await send({
|
||||
"type": "http.response.start", "status": 413,
|
||||
"headers": [(b"content-type", b"application/json"), (b"content-length", str(len(body)).encode())],
|
||||
})
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
return
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
app.add_middleware(RequestSizeLimitMiddleware)
|
||||
|
||||
# ── Mutation rate limit ───────────────────────────────────────────────────────
|
||||
# This is intentionally small and process-local: it is a last-resort guard
|
||||
# against accidental loops/double-clicks, not a replacement for a reverse
|
||||
# proxy's distributed rate limiter. Long-running generation requests count once
|
||||
# when started and are not held open by this window.
|
||||
class MutationRateLimitMiddleware:
|
||||
_window_seconds = 60.0
|
||||
_general_limit = 120
|
||||
_destructive_limit = 30
|
||||
_destructive_prefixes = (
|
||||
"/api/voice", "/api/characters", "/api/rehearsals", "/api/reader/docs",
|
||||
"/api/settings", "/api/local-containers", "/api/line-audio",
|
||||
"/api/logs", "/api/tts-routing-log", "/mcp",
|
||||
)
|
||||
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
self._hits: dict[tuple[str, bool], list[float]] = {}
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] != "http" or scope.get("method", "GET") not in {"POST", "PUT", "PATCH", "DELETE"}:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
path = scope.get("path", "")
|
||||
is_destructive = path.startswith(self._destructive_prefixes)
|
||||
client = scope.get("client") or ("unknown", 0)
|
||||
key = (str(client[0]), is_destructive)
|
||||
now = time.monotonic()
|
||||
hits = [stamp for stamp in self._hits.get(key, []) if now - stamp < self._window_seconds]
|
||||
limit = self._destructive_limit if is_destructive else self._general_limit
|
||||
if len(hits) >= limit:
|
||||
retry_after = max(1, int(self._window_seconds - (now - hits[0])))
|
||||
body = b'{"detail":"Too many mutation requests; try again later"}'
|
||||
await send({
|
||||
"type": "http.response.start", "status": 429,
|
||||
"headers": [
|
||||
(b"content-type", b"application/json"),
|
||||
(b"content-length", str(len(body)).encode()),
|
||||
(b"retry-after", str(retry_after).encode()),
|
||||
],
|
||||
})
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
return
|
||||
hits.append(now)
|
||||
self._hits[key] = hits
|
||||
# Remove stale client buckets opportunistically so a long-lived process
|
||||
# does not retain one list for every historical source IP forever.
|
||||
if len(self._hits) > 2048:
|
||||
self._hits = {k: v for k, v in self._hits.items() if v and now - v[-1] < self._window_seconds}
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
app.add_middleware(MutationRateLimitMiddleware)
|
||||
|
||||
# ── In-memory log buffer handler ──────────────────────────────────────────────
|
||||
|
||||
_buf_handler = _BufferHandler()
|
||||
|
||||
@ -142,8 +142,8 @@
|
||||
const url = normalizeProbeUrl(rawUrl);
|
||||
const params = { url };
|
||||
if (type) params.type = type;
|
||||
if (apiKey) params.api_key = apiKey;
|
||||
const r = await fetch('/api/probe-url?' + new URLSearchParams(params));
|
||||
const headers = apiKey ? { 'X-Provider-API-Key': apiKey } : {};
|
||||
const r = await fetch('/api/probe-url?' + new URLSearchParams(params), { headers });
|
||||
return r.json();
|
||||
}
|
||||
|
||||
@ -309,9 +309,9 @@
|
||||
setStatus('Loading models…');
|
||||
if (refreshBtn) refreshBtn.disabled = true;
|
||||
try {
|
||||
let fetchUrl = '/api/conversation/llm-models?url=' + encodeURIComponent(url);
|
||||
if (apiKey) fetchUrl += '&api_key=' + encodeURIComponent(apiKey);
|
||||
const d = await fetch(fetchUrl).then(r => r.json());
|
||||
const fetchUrl = '/api/conversation/llm-models?url=' + encodeURIComponent(url);
|
||||
const headers = apiKey ? { 'X-Provider-API-Key': apiKey } : {};
|
||||
const d = await fetch(fetchUrl, { headers }).then(r => r.json());
|
||||
if (d.error) {
|
||||
let msg = 'Failed';
|
||||
try {
|
||||
|
||||
@ -248,9 +248,9 @@ async function _statusOpenLlmFlyup(anchor) {
|
||||
_statusFlyupList(anchor, [], null, 'Loading models…', 'llm');
|
||||
let models = [];
|
||||
try {
|
||||
let fetchUrl = '/api/conversation/llm-models?url=' + encodeURIComponent(url);
|
||||
if (apiKey) fetchUrl += '&api_key=' + encodeURIComponent(apiKey);
|
||||
const data = await fetch(fetchUrl).then(r => r.json());
|
||||
const fetchUrl = '/api/conversation/llm-models?url=' + encodeURIComponent(url);
|
||||
const headers = apiKey ? { 'X-Provider-API-Key': apiKey } : {};
|
||||
const data = await fetch(fetchUrl, { headers }).then(r => r.json());
|
||||
models = Array.isArray(data.models) ? data.models : [];
|
||||
} catch (_) {}
|
||||
if (!_statusFlyup || _statusFlyup.dataset.forKind !== 'llm') return; // closed/switched while awaiting
|
||||
@ -318,9 +318,9 @@ async function refreshStatusBarEngines({ force = false } = {}) {
|
||||
if (fresh && !force) return;
|
||||
STATUS_LLM_CACHE.pending = true;
|
||||
try {
|
||||
let fetchUrl = '/api/conversation/llm-models?url=' + encodeURIComponent(url);
|
||||
if (apiKey) fetchUrl += '&api_key=' + encodeURIComponent(apiKey);
|
||||
const data = await fetch(fetchUrl).then(r => r.json());
|
||||
const fetchUrl = '/api/conversation/llm-models?url=' + encodeURIComponent(url);
|
||||
const headers = apiKey ? { 'X-Provider-API-Key': apiKey } : {};
|
||||
const data = await fetch(fetchUrl, { headers }).then(r => r.json());
|
||||
const models = Array.isArray(data.models) ? data.models : [];
|
||||
STATUS_LLM_CACHE.url = url;
|
||||
STATUS_LLM_CACHE.model = model;
|
||||
|
||||
Loading…
Reference in New Issue
Block a user