Skip to content

Commit f570ec0

Browse files
author
npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6
committed
huddle(tts): drop leading fade-in to preserve consonant onsets
Symptom (reported 2026-05-18): "the first little sound or two in a sentence is kind of getting skipped over." Affects every sentence, not just the first one, and is independent of sentence length. Root cause: `apply_fades` applied an 8 ms linear fade-in (192 samples at 24 kHz) to the start of every synthesised sentence. Probing Pocket TTS output across four prompts (Y/H/W/T onsets — see `examples/pocket_onset_probe.rs`) shows real audio energy inside the first millisecond: prompt | samples[0] | peak@1ms rms@1ms ───────────────────────────────────────────────────────────── Yep, I can hear you. | 0.00193 | 0.0331 0.0235 Hello there friend. | 0.00185 | 0.0288 0.0191 What can I help with? | 0.00180 | 0.0358 0.0242 Try this experiment now. | 0.00189 | 0.0197 0.0139 For 'Yep' and 'What' the first-1ms RMS is *equal to or greater than* the first-5ms RMS — the consonant attack peaks inside the very window the fade was nuking. A 0→1 linear ramp attenuated those onset samples by ≥6 dB over the first 4 ms, which is exactly what Tyler heard as "swallowed sounds". `samples[0]` ≈ 0.0019 (≈ −54 dBFS) is far below any audible DC-jump-click threshold, so removing the fade-in does not introduce clicks. Fade-out is retained because end-of-sentence cuts *do* create audible clicks when a non-zero waveform terminates abruptly. Changes (3 files, +101/−24): - `huddle/tts.rs`: - Rename `apply_fades` → `apply_fade_out`. Body removes the leading fade loop and operates on `&mut [f32]` instead of `&mut Vec<f32>`. - New const `FIRST_APPEND_LEAD_IN_SAMPLES = 480` (20 ms) and a single-shot `player.append(zeros)` at the `first_append` site, so the OS audio device / rodio mixer gets a quiet ramp-up window *without* scaling any real synthesis samples. Applied once per utterance — sentence boundaries continue to use `INTER_SENTENCE_SILENCE` (100 ms) and don't stack on this cushion. - New regression test `apply_fade_out_does_not_touch_leading_samples` locks in `samples[0..FADE_OUT_SAMPLES]` are byte-equal to input. Will fail loudly if anyone ever reintroduces a leading fade. - `first_append_lead_in_is_sane` pins the 20 ms × 24 kHz = 480 constant and documents why that range is reasonable. - Existing `apply_fades_*` tests renamed and updated; +2 net tests (24 → 26 in tts.rs; 317 → 319 lib-wide). - `examples/pocket_onset_probe.rs` (new, 137 lines): synthesises the four probe prompts, dumps per-prompt onset stats (samples[0], peak/RMS @ 1ms/5ms/20ms), and writes raw WAVs to /tmp for offline inspection. Documents the measurement that justifies removing the fade-in; runs against the same `/tmp/pocket-tts-bench` model directory `pocket_bench` uses. - `desktop/scripts/check-file-sizes.mjs`: bump `tts.rs` override 1130 → 1210 with updated description. Verification before push: - `cargo test --lib` (full) → 319/319 pass. - `cargo fmt --check` clean. - `cargo check` (desktop tauri crate) clean. - `pnpm check` (biome + file-sizes) clean. - Manual A/B not yet done from the worktree — Tyler will hear the result on `cargo run` after pull. Discussion: thread root c0f5988e in #sprout-desktop-lighter-tts (initial diagnosis, Max's review of approach, probe data). Signed-off-by: Tyler Longwell <[email protected]> Signed-off-by: npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6 <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
1 parent 1dbfa2c commit f570ec0

3 files changed

Lines changed: 247 additions & 25 deletions

File tree

‎desktop/scripts/check-file-sizes.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ const overrides = new Map([
7272
["src-tauri/src/huddle/pocket.rs", 620], // Pocket TTS engine wrapper + prepare_pocket_prompt (capitalize/punctuate/pad short inputs, mirror upstream pocket-tts prepare_text_prompt) + build_generation_extra (only overrides max_frames, and only for ≤4-word inputs, to bound runaway "monster breathing" generation without clipping multi-clause sentences) + 12 unit tests including regressions for the static-burst-on-multi-clause-sentence bug
7373
["src-tauri/src/huddle/preprocessing.rs", 670], // TTS text preprocessing pipeline + unified split_sentences + int_to_words 0-999999 + URL trailing punctuation preservation + 23 unit tests
7474
["src-tauri/src/huddle/relay_api.rs", 520], // audio relay recv task + per-peer frame counting for remote human TTS interrupt + NIP-98 channel member query
75-
["src-tauri/src/huddle/tts.rs", 1130], // TTS pipeline + session warmup + cancel/shutdown handling + apply_fades + normalize_for_playback (per-sentence peak normalization to -6 dBFS with MAX_GAIN cap) + 24 unit tests (18 interrupt + 6 fade/normalize)
75+
["src-tauri/src/huddle/tts.rs", 1210], // TTS pipeline + session warmup + cancel/shutdown handling + apply_fade_out (fade-out only — leading fade removed 2026-05-18 after onset-attenuation regression measured in examples/pocket_onset_probe.rs) + FIRST_APPEND_LEAD_IN_SAMPLES (20 ms zero cushion at the first-append site to give the OS audio device a quiet ramp-up window without scaling real samples) + normalize_for_playback (per-sentence peak normalization to -6 dBFS with MAX_GAIN cap) + 26 unit tests (18 interrupt + 5 fade-out + 1 first-append-lead-in + 5 normalize)
7676
["src-tauri/src/relay.rs", 510], // +4 lines for NIP-OA auth tag injection in profile sync (build_profile_event) + verification test
7777
["src-tauri/src/commands/pairing.rs", 600], // NIP-AB pairing actor: 3 Tauri commands + background WS task + NIP-42 auth + NIP-43 probe + event parsing helpers
7878
["src-tauri/src/lib.rs", 715], // +4 lines for PairingHandle managed state + 3 pairing command registrations
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
//! Onset-attenuation probe for Pocket TTS.
2+
//!
3+
//! Synthesises a handful of short sentences and dumps per-sentence onset
4+
//! statistics (samples[0], 1ms/5ms/20ms peak + RMS) so we can decide whether
5+
//! the production `apply_fades` 8 ms fade-in is masking real audio.
6+
//!
7+
//! Also writes the raw (un-faded, un-normalised) audio of each sentence to
8+
//! /tmp so they can be inspected in Audacity / aplay without rodio in the
9+
//! loop.
10+
//!
11+
//! Run with model files in /tmp/pocket-tts-bench (override with arg 1):
12+
//! cargo run --release --example pocket_onset_probe
13+
//! cargo run --release --example pocket_onset_probe /path/to/pocket-tts
14+
15+
use std::path::PathBuf;
16+
17+
use sherpa_onnx::{
18+
self, GenerationConfig, OfflineTts, OfflineTtsConfig, OfflineTtsModelConfig,
19+
OfflineTtsPocketModelConfig, Wave,
20+
};
21+
22+
const SAMPLE_RATE: u32 = 24_000;
23+
24+
/// Test prompts chosen to span different onsets:
25+
/// - palatal glide 'Y' (soft onset)
26+
/// - voiceless fricative 'H' (very soft onset)
27+
/// - labio-velar glide 'W' (medium onset)
28+
/// - voiceless stop 'T' (hard onset)
29+
const PROMPTS: &[&str] = &[
30+
"Yep, I can hear you.",
31+
"Hello there friend.",
32+
"What can I help with?",
33+
"Try this experiment now.",
34+
];
35+
36+
fn main() {
37+
let model_dir = std::env::args()
38+
.nth(1)
39+
.unwrap_or_else(|| "/tmp/pocket-tts-bench".to_string());
40+
eprintln!("Model dir: {model_dir}");
41+
42+
let dir = PathBuf::from(&model_dir);
43+
let p = |name: &str| dir.join(name).to_string_lossy().into_owned();
44+
45+
let mut cfg = OfflineTtsConfig::default();
46+
cfg.model = OfflineTtsModelConfig {
47+
pocket: OfflineTtsPocketModelConfig {
48+
lm_main: Some(p("lm_main.int8.onnx")),
49+
lm_flow: Some(p("lm_flow.int8.onnx")),
50+
encoder: Some(p("encoder.onnx")),
51+
decoder: Some(p("decoder.int8.onnx")),
52+
text_conditioner: Some(p("text_conditioner.onnx")),
53+
vocab_json: Some(p("vocab.json")),
54+
token_scores_json: Some(p("token_scores.json")),
55+
voice_embedding_cache_capacity: 16,
56+
},
57+
num_threads: 1,
58+
debug: false,
59+
..Default::default()
60+
};
61+
let engine = OfflineTts::create(&cfg).expect("engine create");
62+
63+
let voice_path = dir.join("reference_sample.wav");
64+
let wave = Wave::read(voice_path.to_str().unwrap()).expect("voice WAV");
65+
let voice_samples = wave.samples().to_vec();
66+
let voice_sr = wave.sample_rate();
67+
68+
// Warmup so we're not measuring cold-call jitter.
69+
{
70+
let cfg = GenerationConfig {
71+
speed: 1.05,
72+
num_steps: 1,
73+
silence_scale: 0.0,
74+
reference_audio: Some(voice_samples.clone()),
75+
reference_sample_rate: voice_sr,
76+
..Default::default()
77+
};
78+
let _ = engine.generate_with_config("warmup.", &cfg, None::<fn(&[f32], f32) -> bool>);
79+
}
80+
81+
println!(
82+
"{:<28} | {:>10} | {:>10} {:>10} | {:>10} {:>10} | {:>10} {:>10}",
83+
"prompt",
84+
"samples[0]",
85+
"peak@1ms",
86+
"rms@1ms",
87+
"peak@5ms",
88+
"rms@5ms",
89+
"peak@20ms",
90+
"rms@20ms"
91+
);
92+
println!("{}", "-".repeat(120));
93+
94+
for prompt in PROMPTS {
95+
// Mirror the production prompt-prep (capitalise + terminal punctuation).
96+
// These prompts already have it, so this is just to match what
97+
// sherpa-onnx sees in production.
98+
let cfg = GenerationConfig {
99+
speed: 1.05,
100+
num_steps: 1,
101+
silence_scale: 0.0,
102+
reference_audio: Some(voice_samples.clone()),
103+
reference_sample_rate: voice_sr,
104+
..Default::default()
105+
};
106+
let out = engine
107+
.generate_with_config(prompt, &cfg, None::<fn(&[f32], f32) -> bool>)
108+
.expect("synth");
109+
let samples = out.samples();
110+
111+
let n_1ms = (SAMPLE_RATE as f32 * 0.001) as usize;
112+
let n_5ms = (SAMPLE_RATE as f32 * 0.005) as usize;
113+
let n_20ms = (SAMPLE_RATE as f32 * 0.020) as usize;
114+
115+
let stats = |range: &[f32]| -> (f32, f32) {
116+
if range.is_empty() {
117+
return (0.0, 0.0);
118+
}
119+
let peak = range.iter().fold(0.0_f32, |a, &x| a.max(x.abs()));
120+
let sumsq: f32 = range.iter().map(|x| x * x).sum();
121+
let rms = (sumsq / range.len() as f32).sqrt();
122+
(peak, rms)
123+
};
124+
125+
let first = samples.first().copied().unwrap_or(0.0);
126+
let (p1, r1) = stats(&samples[..n_1ms.min(samples.len())]);
127+
let (p5, r5) = stats(&samples[..n_5ms.min(samples.len())]);
128+
let (p20, r20) = stats(&samples[..n_20ms.min(samples.len())]);
129+
130+
println!(
131+
"{:<28} | {:>10.6} | {:>10.6} {:>10.6} | {:>10.6} {:>10.6} | {:>10.6} {:>10.6}",
132+
prompt, first, p1, r1, p5, r5, p20, r20
133+
);
134+
135+
// Dump raw WAV for inspection.
136+
let safe: String = prompt
137+
.chars()
138+
.map(|c| if c.is_ascii_alphanumeric() { c } else { '_' })
139+
.collect();
140+
let out_path = format!("/tmp/pocket_onset_{}.wav", &safe[..safe.len().min(24)]);
141+
let _ = sherpa_onnx::write(&out_path, samples, SAMPLE_RATE as i32);
142+
eprintln!(
143+
" → wrote {out_path} ({} samples = {:.3} s)",
144+
samples.len(),
145+
samples.len() as f32 / SAMPLE_RATE as f32
146+
);
147+
}
148+
}

‎desktop/src-tauri/src/huddle/tts.rs‎

Lines changed: 98 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,24 @@ const TARGET_PEAK: f32 = 0.501_187_2; // 10f32.powf(-6.0 / 20.0)
7070
/// still catching pathological near-silent buffers.
7171
const MAX_GAIN: f32 = 8.0;
7272

73-
/// Fade in/out length in samples (8ms at 24kHz ≈ 192 samples).
74-
/// Eliminates clicks/pops at sentence boundaries.
75-
const FADE_SAMPLES: usize = (SAMPLE_RATE as f64 * 0.008) as usize;
73+
/// Fade-out length in samples (8 ms at 24 kHz ≈ 192 samples).
74+
///
75+
/// Applied only at the *end* of each synthesised sentence to eliminate the
76+
/// click that would otherwise occur when a non-zero waveform terminates
77+
/// abruptly. **No fade-in is applied** — see `apply_fade_out` for the
78+
/// rationale and `examples/pocket_onset_probe.rs` for the measurement that
79+
/// motivated removing the leading fade.
80+
const FADE_OUT_SAMPLES: usize = (SAMPLE_RATE as f64 * 0.008) as usize;
81+
82+
/// Length of the zero-sample cushion prepended to the very first audio
83+
/// buffer of an utterance, so the OS audio device / rodio mixer has a
84+
/// fully-quiet ramp-up window before the real onset hits.
85+
///
86+
/// Applied at the `first_append` site only — *not* per sentence — so it
87+
/// doesn't stack on top of `INTER_SENTENCE_SILENCE` at sentence boundaries.
88+
/// 20 ms ≈ 480 samples is enough to cover a CoreAudio buffer turnover
89+
/// without being audible as latency.
90+
const FIRST_APPEND_LEAD_IN_SAMPLES: usize = (SAMPLE_RATE as f64 * 0.020) as usize;
7691

7792
/// Sentence-by-sentence synthesis — keeps first-sentence latency low and lets
7893
/// playback of sentence N overlap with synthesis of sentence N+1 (see the
@@ -385,7 +400,19 @@ fn tts_worker(
385400
match engine.synth_chunk(text, "en", &style, SYNTH_STEPS, SYNTH_SPEED) {
386401
Ok(samples) if !samples.is_empty() => {
387402
let mut boosted = normalize_for_playback(samples);
388-
apply_fades(&mut boosted);
403+
// Fade-out only — fading-in would attenuate the consonant
404+
// onset (see `apply_fade_out` docstring + the
405+
// 2026-05-18 "first little sound is missing" regression).
406+
apply_fade_out(&mut boosted);
407+
if first_append {
408+
// Pre-pad the very first buffer of an utterance with
409+
// a brief silence so the OS audio device / rodio
410+
// mixer has a fully-quiet ramp-up window before the
411+
// first real sample. Applied once per utterance —
412+
// sentence boundaries use INTER_SENTENCE_SILENCE.
413+
let lead_in = vec![0.0f32; FIRST_APPEND_LEAD_IN_SAMPLES];
414+
player.append(SamplesBuffer::new(channels, rate, lead_in));
415+
}
389416
player.append(SamplesBuffer::new(channels, rate, boosted));
390417
// Insert inter-sentence silence after each synthesized chunk.
391418
player.append(SamplesBuffer::new(channels, rate, silence_buf.clone()));
@@ -474,18 +501,31 @@ fn normalize_for_playback(samples: Vec<f32>) -> Vec<f32> {
474501
.collect()
475502
}
476503

477-
/// Apply a short linear fade-in at the start and fade-out at the end of `samples`.
504+
/// Apply a short linear fade-out at the *end* of `samples`.
505+
///
506+
/// Uses `FADE_OUT_SAMPLES` (8 ms) or half the buffer length, whichever is
507+
/// smaller. Eliminates the click that occurs when a non-zero waveform
508+
/// terminates abruptly at a sentence boundary.
478509
///
479-
/// Uses `FADE_SAMPLES` (8ms) or half the buffer length, whichever is smaller.
480-
/// Eliminates clicks/pops at sentence boundaries.
481-
fn apply_fades(samples: &mut Vec<f32>) {
510+
/// # Why no fade-in
511+
///
512+
/// An earlier revision (pre 2026-05) symmetrically faded *in* over the same
513+
/// 8 ms window. That swallowed the leading consonant attack on every
514+
/// sentence — Pocket TTS produces real audio energy inside the first
515+
/// millisecond (RMS ≈ 0.02, peak ≈ 0.03 measured across four prompts in
516+
/// `examples/pocket_onset_probe.rs`), and a linear 0→1 ramp over 192 samples
517+
/// scales those onset samples by ≤50 % for the first ~4 ms. The result was
518+
/// the "first little sound or two is missing" regression heard on
519+
/// 2026-05-18.
520+
///
521+
/// The first sample of Pocket output measures ≈ 0.0018 (≈ −54 dBFS) — well
522+
/// below the threshold at which a DC-jump would be audible as a click — so
523+
/// no fade-in is needed. The OS audio device gets its quiet ramp-up window
524+
/// from `FIRST_APPEND_LEAD_IN_SAMPLES` instead, applied once per utterance
525+
/// at the `first_append` site.
526+
fn apply_fade_out(samples: &mut [f32]) {
482527
let len = samples.len();
483-
let fade = FADE_SAMPLES.min(len / 2);
484-
// Fade in: ramp from 0 → 1 over `fade` samples.
485-
for i in 0..fade {
486-
samples[i] *= i as f32 / fade as f32;
487-
}
488-
// Fade out: ramp from 1 → 0 over the last `fade` samples.
528+
let fade = FADE_OUT_SAMPLES.min(len / 2);
489529
for i in 0..fade {
490530
samples[len - 1 - i] *= i as f32 / fade as f32;
491531
}
@@ -1036,31 +1076,65 @@ mod tests {
10361076
);
10371077
}
10381078

1039-
// ── apply_fades tests ─────────────────────────────────────────────────────
1079+
// ── apply_fade_out tests ──────────────────────────────────────────────────
10401080

1081+
/// The fade-out half of the old `apply_fades`: last sample is silenced
1082+
/// and the ramp is monotonic. Mid-buffer must be untouched.
10411083
#[test]
1042-
fn apply_fades_short_buffer() {
1084+
fn apply_fade_out_short_buffer() {
10431085
let mut samples = vec![1.0f32; 10];
1044-
apply_fades(&mut samples);
1045-
assert_eq!(samples[0], 0.0);
1046-
assert_eq!(samples[9], 0.0);
1047-
assert!(samples[5] > 0.5);
1086+
apply_fade_out(&mut samples);
1087+
assert_eq!(samples[9], 0.0, "last sample should be silenced");
1088+
assert!(samples[5] > 0.5, "mid-buffer should be near-untouched");
10481089
}
10491090

1091+
/// REGRESSION (2026-05-18): the *first* samples must NOT be attenuated.
1092+
/// An earlier `apply_fades` symmetrically faded in over 8 ms which
1093+
/// swallowed the consonant onset of every sentence.
1094+
/// Lock in: samples[0..FADE_OUT_SAMPLES] are byte-equal to input.
10501095
#[test]
1051-
fn apply_fades_empty_buffer() {
1096+
fn apply_fade_out_does_not_touch_leading_samples() {
1097+
// Input long enough that fade window doesn't overlap (≫ 2× fade).
1098+
let n = FADE_OUT_SAMPLES * 4;
1099+
let input: Vec<f32> = (0..n).map(|i| 0.5 + (i as f32) * 1e-4).collect();
1100+
let mut samples = input.clone();
1101+
apply_fade_out(&mut samples);
1102+
for i in 0..FADE_OUT_SAMPLES {
1103+
assert_eq!(
1104+
samples[i], input[i],
1105+
"leading sample {i} must not be attenuated (was {} → {})",
1106+
input[i], samples[i]
1107+
);
1108+
}
1109+
// And the trailing fade still works.
1110+
assert_eq!(samples[n - 1], 0.0);
1111+
}
1112+
1113+
#[test]
1114+
fn apply_fade_out_empty_buffer() {
10521115
let mut samples: Vec<f32> = vec![];
1053-
apply_fades(&mut samples);
1116+
apply_fade_out(&mut samples);
10541117
assert!(samples.is_empty());
10551118
}
10561119

10571120
#[test]
1058-
fn apply_fades_single_sample() {
1121+
fn apply_fade_out_single_sample() {
1122+
// fade = min(FADE_OUT_SAMPLES, len/2) = 0, so nothing changes.
10591123
let mut samples = vec![1.0f32];
1060-
apply_fades(&mut samples);
1124+
apply_fade_out(&mut samples);
10611125
assert_eq!(samples[0], 1.0);
10621126
}
10631127

1128+
/// Sanity-check the first-append cushion length: 20 ms at 24 kHz must
1129+
/// land at exactly 480 samples. This is a const computation, so the
1130+
/// real value of this test is documenting *why* 20 ms was chosen — it
1131+
/// covers a typical CoreAudio buffer turnover (256–1024 samples)
1132+
/// without being audible as user-facing latency.
1133+
#[test]
1134+
fn first_append_lead_in_is_sane() {
1135+
assert_eq!(FIRST_APPEND_LEAD_IN_SAMPLES, 480, "20 ms × 24 kHz");
1136+
}
1137+
10641138
// ── normalize_for_playback tests ──────────────────────────────────────────
10651139

10661140
/// A quiet buffer (peak well under TARGET_PEAK) is scaled up to TARGET_PEAK.

0 commit comments

Comments
 (0)