feat(tts): separate quality voice design and base cloning - #46

Status — incremental implementation, not an audio-quality sign-off

This PR preserves the complete architecture and implementation plan for SpeechRail/Sona voice creation. Continue on feat/quality-voice-capabilities; do not merge based on unit-test success alone. Real Apple Silicon synthesis, identity evaluation and recording acceptance remain required.

Original source audit: main@9dec75bed02bddecd4be56d0b5491c4b772f0a32. Mainline loudness changes from merged PR #45 (e22738e) are incorporated; #45 is not an outstanding draft dependency.

Implemented: catalog-governed Quality VoiceDesign/Base split; Base public reference generation; independent dual-resident/concurrent TTS capability workers with cooldown group eviction; fail-closed variant handshake; signal/repetition quality checks; staged local ASR intelligibility; first-stage server reference canonicalization; and now explicit, create-only generated-reference registration through POST /v1/voices/designs (0c403ab).

Not implemented or not acoustically accepted: Sona integration with the new registration endpoint and PCM capture changes, automatic migration of old instruction voices, neural VAD/conditional denoise, independent speaker validation, reference-feature caching, professional delivery and full voice revision history. New generated-reference registration does not itself execute Base synthesis or certify identity stability; subsequent synthesis and quality validation remain separate.

1. Problem and source evidence

Sona has two distinct creation experiences: prompt-based voice design creates a new voice from a description; reference-audio cloning reproduces an authorized recorded/uploaded speaker.

The audited clone implementation invoked VoiceDesign's private ICL path. In locked MLX-Audio, Base provides the dedicated speaker encoder condition that VoiceDesign lacks. Reference acoustic codes were still present in the old path; it was not completely unconditioned. The model-role mismatch is a strong candidate for cross-text identity drift, not proof that it explains every audible defect.

The initial audit reproduced invalid output falsely passing quality checks: silence, clipping and noise-like signals could pass because generation success was treated as quality success; determinism was asserted rather than measured; only a prefix of the declared probe set was reachable. Recording gain was sensitive to pause duration, with separate client/vendor normalization. These require model-selection, reference-handling and acceptance changes, not merely stronger denoising.

2. Architecture decision and both Sona paths

Quality exclusively owns voice creation; lower tiers remain lightweight:

Quality
├── voice_design → Qwen3-TTS VoiceDesign 1.7B
└── voice_clone  → Qwen3-TTS Base 1.7B

Explicit new prompt registration:
Prompt + reference text + seed + unused target ID
 → VoiceDesign → generated reference → signal/canonicalization/ASR gates
 → new Base-bound clone profile
 → subsequent Base synthesis + separate output/identity validation

Reference registration:
Recording/upload → reference validation and conditioning
 → canonical reference → Base-bound registered voice
 → subsequent synthesis and output validation

VoiceDesign creates a voice; Base performs reference cloning and stable-reproduction experiments. Base is not a prompt-to-voice-design replacement. Sona requests business capabilities rather than encoding model repository names.

Existing metadata-only instruction voices still synthesize through VoiceDesign. Their APIs and stored assets are not silently changed. Clients must explicitly call the new generated-reference endpoint; migrating existing voices into revisioned Base-backed identities remains a separate operation. Reference clones never fall back to VoiceDesign private ICL.

Normative architecture: docs/architecture/quality-voice-capabilities.md; implemented registration contract: docs/architecture/generated-voice-registration.md.

3. Catalog, installation and capability contracts

ModelPreset includes optional tts_clone, alongside asr and primary tts:

• Quality primary/design TTS: VoiceDesign.
• Quality tts_clone: Base.
• Balanced/Light: no Base clone binding and no implicit voice-creation capability.

Base is pinned to mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit, revision 73ae2cb59832ed1eb13c249e378b854cfc643131 (ModelScope); required-file hashes are catalog governed.

Installer, model store, preflight, profile selection, catalog/build tools, system reporting, HTTP, MCP and realtime contracts are updated together. Model IDs/private paths are not a frontend requirement. Inference must not silently download missing models. The new generated-registration operation is HTTP-only in this increment; existing MCP creation tools remain compatible and do not silently call it.

4. Runtime lifecycle and identity boundary

Quality owns two independent large-TTS-model workers: VoiceDesign serves the voice_design lane and Base serves the voice_clone lane. They are not a request-level mutually exclusive slot. The router keeps the worker lifecycles separate, and each worker keeps its own request lock: different lanes can synthesize concurrently, while requests sharing one lane remain serialized.

SPEECHRAIL_WORKER_LAZY_LOAD=false starts both workers sequentially at lifecycle startup. With lazy loading enabled, the first request starts only the worker it needs; the other starts on its first request and stays resident. Switching from VoiceDesign to Base, or back, changes routing only and does not close the sibling worker, so the normal “design voice → use existing clone” flow does not repeatedly load/unload either model.

ResourceGovernor receives the internal voice_design / voice_clone lane key. Distinct Quality lanes may be admitted concurrently when total capacity permits; the same lane, unkeyed TTS, Light/Balanced TTS and unknown injected synthesizers retain conservative serialization. ASR∥TTS remains controlled separately by SPEECHRAIL_ALLOW_HEAVY_OVERLAP and the declared memory budget; Quality TTS resident bytes are counted per possible worker.

WorkerIdleEvictor treats the router as one lifecycle group. Warm standby trims both workers, and after the configured idle cooldown it closes both together; the router uses the latest child activity timestamp and each worker lock lets an active stream finish before close. After group eviction, the next request lazily restores only the selected worker. This preserves automatic memory release without reintroducing request-level capability switching.

mlx-qwen3-tts identifies the backend family; model_variant separately identifies voice_design, custom_voice or base. Composition obtains the expected variant from managed metadata or local snapshot inspection. Config and child ready handshake must agree; missing/mismatched variant is backend_identity_mismatch, never a VoiceDesign fallback. Lifecycle diagnostics report warm_capability=both plus the explicit warm_capabilities list when both workers are ready.

5. Quality acceptance — implemented checks and limits

Registration success, reference-input acceptance and synthesis-output acceptance are separate states.

Fixed probe matrix and signal/repetition checks

All six declared Chinese probe classes are covered. runs=1..3 means repetitions per probe; default is 18 TTS samples, not three samples from the list prefix. Checks include malformed/empty PCM, ineffective/silent output, clipping and per-probe repeated PCM SHA-256 comparison. Differences receive output_nondeterministic; fixed seed or a single repetition does not prove repeatability. Valid waveforms/matching hashes do not establish identity or intelligibility.

Probe work uses ResourceGovernor, a shared absolute request deadline and public AudioChunk validation. Oversized streams close before proceeding. Queue-full/timeouts remain 429/503, not a hidden quality pass.

Staged local ASR intelligibility

A. Six probes × repetitions under BATCH_TTS
 → bounded PCM, signal gate and repeat hashes
B. Close streams, release TTS reservation, evict the Quality TTS router group
 → eviction shares the original absolute deadline
C. BATCH_ASR with existing transcriber/admission
 → one representative per probe, at most six ASR calls
D. Unicode/ITN-normalized expected-versus-recognized comparison
 → minimum probe score participates in output acceptance

Expected text is never supplied as an ASR prompt. Provisional match thresholds: >=0.92 pass, 0.80..0.92 warn, <0.80 reject. Missing/failing ASR makes synthesis validation unevaluated / transcription_unavailable, never automatically passing. Validation error logging excludes vendor exception text/tracebacks and retains only stable code/request ID.

The current bounded 24 kHz → 16 kHz validation resampling is linear, not a claimed high-quality anti-aliasing implementation. Thresholds require real speech, numbers/units/dates, accents and noise calibration. ASR can err/hallucinate; it is not a universal noise detector, perceptual MOS measure or speaker verifier.

6. New explicit generated-reference registration

POST /v1/voices/designs is opt-in and requires Quality VoiceDesign/Base plus local TTS and Batch ASR. JSON fields: required id, name, instruction, reference_text; optional seed=42, language=zh. Initial language is Chinese only. IDs are 1–64 lowercase letters/digits/underscore/hyphen and cannot be system IDs/aliases. Reference text is bounded to 20–240 characters, including validation after TTS normalization; generated PCM is bounded to 2–30 seconds. Extra fields, URLs, uploaded audio and speed options are not accepted here.

Flow:

1. Reject an occupied target ID and unavailable ASR before expensive work.
2. Generate bounded VoiceDesign PCM under BATCH_TTS; explicitly close the stream on every exit.
3. Gate raw signal before gain changes so normalization cannot hide clipping; canonicalize once and gate the canonical reference.
4. Release/evict TTS under the same request deadline, then transcribe the canonical reference under BATCH_ASR/admission with an empty prompt.
5. Require the provisional 0.92 text-match threshold. Do not rewrite the expected reference text with arbitrary ASR output.
6. Only after successful validation, atomically save a new clone profile. Target-ID conflict is checked again under the registry lock before audio is written.

No temporary public voice or reference file is published while TTS/ASR is running. Existing private permissions, immutable audio path, atomic metadata commit and failure cleanup are reused. Concurrent requests cannot overwrite the winner. Existing create-clone callers keep their request/response shape, but the clone quality report is now graded on the persisted canonical reference rather than the raw upload; create-only behavior is explicit for this new operation.

201 returns voice plus synthesis_validation: unevaluated. The voice binds to Base through its clone mode, but registration does not invoke Base or execute the six-probe output matrix. Its reference report is not output-quality acceptance. Missing/failing ASR prevents registration (503), rather than publishing an unverified reference. ID conflict is 409; invalid signal/transcript is 400; malformed/oversized model output is 502; queue/mode conflict is 429; shared deadline expiration is 503.

New profiles include optional immutable VoiceCreation metadata: generated origin, method voice_design_reference_v1, VoiceDesign artifact/revision/seed, SHA-256 of instruction, actual reference text and canonical WAV, and preprocessing version energy_v1. Registry validates audio/text hashes before write and validates metadata on reload. Public responses do not expose local audio paths. These hashes provide traceability, not signatures or full revision history.

Old /v1/voices instruction creation, /v1/voices/previews and old stored records stay unchanged; /v1/voices/clone keeps its request/response shape, but its quality report is now graded on the canonicalized reference and persisted accordingly (docs/users/api-contract.md §5.7.1), and stored reports for old clones are not rewritten. This is not an in-place migration or a durable Idempotency-Key implementation: retries using the same ID receive 409 and must inspect the existing asset. Sona must explicitly integrate the new action; this increment does not modify Sona UI.

7. Reference conditioning — current implementation and complete roadmap

New reference registration canonicalizes accepted audio and saves the resulting WAV. Vendor loading uses volume_normalize=False.

Engineering initial values: mono PCM16/24 kHz; 20 ms energy windows with −45 dBFS active threshold; active target −20 dBFS; gain cap +9 dB and attenuation bound 12 dB with peak safety taking priority; peak ceiling 0.95; trim only leading/trailing low-energy areas with about 200 ms padding and minimum-duration handling. Internal pauses are not rewritten.

This is energy selection/bounded gain, not neural VAD or denoising, and parameters are not acoustically calibrated optima. Loud noise, reverb, multiple speakers and delivery cadence remain unsolved. Sona capture gain still needs coordinated removal/redesign; old references are not automatically rewritten, so end-to-end single normalization and legacy migration are not claimed complete.

Full intended pipeline:

capture/decode → actual format validation → VAD / segment selection
 → transcript verification → conditional conservative denoise
 → speech-aware single normalization → canonical reference
 → cached speaker/reference conditioning → cross-text output validation

Sona evolution: AudioWorklet → PCM/WAV with actual device/channel/rate/processing settings; WebM/Opus remains fallback. Prefer several natural short recordings and selecting a reliable reference over requiring professional narration. Distinguish raw recording, conditioned reference and actual synthesized preview.

Noise estimates need credible non-speech evidence; amplitude percentiles are not reliable SNR. Without evidence report not evaluated. Severe clipping/reverb/overlap should trigger re-recording, not promises of restoration. DeepFilterNet3 remains a conditional registration-time enhancement candidate, not mandatory processing on every request; validate local assets before invoking any loader that could download implicitly.

8. Stable voice abstraction, caching and migration

Both creation paths should converge on full VoiceProfile/VoiceRevision semantics: generated/recorded origin, provenance, canonical reference, exact transcript, preprocessing/model identity, hashes and scoped quality evidence. This increment adds generated provenance and immutable reference publication, not a complete multi-revision history or legacy migration system.

Do not silently change existing identities or synthesis meaning. Old voices require explicit revision creation, revalidation and rollback. Metadata-only save is not synthesis validation.

Future feature caches bind immutable audio/text hashes, preprocessing version, tokenizer/model revision and conditioning mode. Current waveform cache is not cached speaker embeddings/reference acoustic codes. Do not share mutable target-text decoder/KV state across requests. Multiple recordings should first improve reference selection; do not average codec tokens or concatenate arbitrary emotions by default.

9. Professional delivery and evaluation experiments

Identity-faithful cloning and professional cadence are separate objectives. Professionalized delivery seeks preserved identity with independent pace, pause and emphasis control; whole-audio speed/gain changes are not that capability.

Evaluate sequentially: A original private-ICL baseline for comparison only; B Base with identical reference to isolate model-path effects; C Base with canonical reference/accurate transcript, separately ablating cleaning/gain/transcription; D speaker-only/x-vector-style conditioning where actually supported, measuring identity/quality tradeoffs; E an expressive backend such as VoxCPM2 for offline narration. Only consider professional TTS → voice conversion if simpler separation is insufficient.

VoxCPM2/VC are research tracks, not deployed realtime capabilities. Validate actual streaming, first-audible latency, output resampling, memory and consistent RTF definitions first.

10. Acceptance, rollout and rollback

Dimensions: independent speaker similarity/cross-text identity distribution; intelligibility, omissions/repetitions/numbers/tail truncation; valid signal; credible noise/perceptual evidence; natural delivery in matched-loudness blind listening; first-audible latency, RTF, memory, cancellation and model switching.

Minimum experiments: six text classes × three repetitions per configuration, cold/warm reference cache, worker restart, short/long outputs. Calibrate independent speaker thresholds on actual data instead of assuming a universal cosine cutoff.

Implementation progression:

1. Correct Base capability and trustworthy gates — implemented software baseline, real-model calibration pending.
2. Server conditioning — initial implementation; Sona capture/gain/diagnostics still pending.
3. Explicit prompt-generated reference registration — now implemented; Sona integration and acoustic acceptance pending.
4. Reference-feature caches, full revisions and legacy migration — pending.
5. Independent identity acceptance and professional-delivery experiments — pending.

Rollback preserves old voice/reference assets. Disable an unavailable Base capability rather than silently restoring incorrect VoiceDesign private ICL; Balanced/Light remain isolated. New generated voices use new IDs, so rejecting/deleting an unsatisfactory new voice does not overwrite its predecessor. CI success does not mean audio acceptance.

11. Validation evidence and remaining checklist

Current head: 66351a6 — adds Quality dual-resident VoiceDesign/Base workers, capability-lane admission, concurrent cross-lane synthesis and cooldown group eviction on top of the earlier ModelScope Base and clone-path fixes. Earlier hashes remain for traceability.

• 36 new registration tests pass locally: new Base binding/public TTS usage, provenance reload, schema validation, tier/parameter rejection, absent/failing/mismatched ASR, silent/clipped/short/oversized/malformed output, cancellation cleanup, eviction deadline, queue rejection, concurrent target collision, failed-write cleanup and old assets preserved.
• Local Python 3.12 suite on 66351a6: full pytest passes (1739 passed, 1 skipped); Ruff, mypy, compileall, OpenAPI lint and diff checks also pass.
• Ruff, mypy (107 source files), compileall, wheel build and diff checks pass.
• Pre-rebase packaging/staging evidence (091f8cc patch SHA-256, staging run 34680935426, earlier-head CI 34680986292) validated superseded feature tips and does not substitute for the head below.
• Previous required CI for superseded head 95345b0: run 34688844618 — Ubuntu + macOS-15 / Python 3.12 success.
• Prior 5ff9ef1 passed Ubuntu/macOS CI 34677080174. Its resource-safety regressions, ASR/privacy/cleanup improvements and complete plan remain preserved.

Before merge:

• Required Ubuntu + macOS/Python 3.12 CI on current head 66351a6 (new run pending), including packaging, contracts, Ruff, mypy and coverage.
• Actual VoiceDesign → registered reference → Base synthesis on Apple Silicon, including cold/warm switching and local ASR.
• Independent speaker identity, matched-loudness listening, reference/noise comparisons.
• Real-data ASR thresholds, language coverage, numbers/units and resampling acceptance.
• Load/unload/eviction/cancellation latency and peak-memory acceptance.
• Sona generated-registration action, PCM capture/gain coordination, separate input/output quality state and diagnostic previews.
• Full revision history and explicit old-voice migration.
• Merged PR fix(tts): guard frozen clone calibration and cross-chunk peak release #45 mainline loudness changes incorporated without confusing them with clone architecture.

12. Documentation

English/Chinese README, architecture index, Quality voice architecture, clone and VoiceDesign design documents, generated-registration architecture, ASR-intelligibility contract, runtime/operations docs, OpenAPI/HTTP/MCP/realtime contracts and changelog are maintained across this PR. New generated-registration details and errors are in docs/architecture/generated-voice-registration.md and docs/users/api-contract.md. Documents distinguish implemented server/API behavior from pending client integration, migration and acoustic acceptance.