Skip to content

Commit 773a2a1

Browse files
huddle(tts): prep short Pocket TTS prompts to prevent runaway generation
Pocket TTS' autoregressive LM has a stochastic sampler (temp=0.7, random seed) and a hard 500-frame ceiling (~40s of audio at 12.5Hz Mimi frame rate). On very short, unpunctuated, or lowercase inputs, the EOS logit (threshold > -4) sometimes never fires within those 500 frames, so the model produces tens of seconds of nonsensical "breathing" output. Tyler hit this on the first 'yep' utterance after the Mary swap landed: ~30s of monster noise, then subsequent utterances were fine. Non- deterministic by design — the next sampled trajectory escaped the trap. Sherpa-onnx's C++ Pocket TTS impl does not run the prompt preparation that upstream kyutai-labs/pocket-tts applies in Python. This commit mirrors that recipe locally: prepare_pocket_prompt(): 1. Collapse interior whitespace. 2. Capitalize the first letter. 3. Append '.' if no terminal punctuation. 4. If <=4 words: prepend 8 spaces; bump frames_after_eos 1 -> 3. 5. Compute adaptive max_frames from word count (saturates at 500). synth_chunk(): - Routes input through prepare_pocket_prompt before calling generate_with_config. - Plumbs frames_after_eos + max_frames via GenerationConfig.extra (HashMap<String, serde_json::Value>) — exposed by the sherpa-onnx 1.12 Rust binding as a per-call escape hatch. Source: pocket_tts.models.tts_model.prepare_text_prompt and _estimate_max_gen_len in https://github.com/kyutai-labs/pocket-tts. 12 new unit tests cover: empty input, the literal 'yep' case (must become ' Yep.'), threshold inclusivity at 4 vs 5 words, preserved existing punctuation, whitespace collapsing, no-double-capitalize, non-ASCII first letter (Cyrillic д -> Д), max_frames tightness/clamping/ monotonicity. Test results: cargo test --lib huddle::pocket -> 12/12 pass cargo test --lib -> 317/317 pass (was 305 + 12 new) cargo fmt --check -> clean cargo clippy huddle::pocket -> clean (preexisting doc-list overindent warning at line 41 not introduced here) node check-file-sizes.mjs -> clean (pocket.rs override added) Signed-off-by: Tyler Longwell <[email protected]>
1 parent 26bb3cf commit 773a2a1

2 files changed

Lines changed: 292 additions & 5 deletions

File tree

‎desktop/scripts/check-file-sizes.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ const overrides = new Map([
6969
["src-tauri/src/huddle/mod.rs", 1020], // huddle state machine + Tauri commands + sync protocol doc; state/relay/pipeline extracted + emit_huddle_state_changed wiring
7070
["src-tauri/src/huddle/models.rs", 950], // model download manager for Parakeet TDT-CTC STT + Pocket TTS with streaming downloads + SHA-256 verification + Rust-native tar extraction + version manifest + atomic swap + hot-start signaling + MODEL_LICENSE.txt sidecar (fail-closed readiness) + idempotent legacy Moonshine dir cleanup + tts_readiness_requires_license_sidecar test + Mary (VCTK p333) reference voice attribution block
7171
["src-tauri/src/huddle/stt.rs", 580], // STT pipeline + PTT edge-detection flush + PTT gating (is_speech AND ptt_active) + barge-in for VAD mode + rubato resampler + earshot VAD + sherpa-onnx transcription
72+
["src-tauri/src/huddle/pocket.rs", 560], // Pocket TTS engine wrapper + prepare_pocket_prompt (capitalize/punctuate/pad short inputs, mirror upstream pocket-tts prepare_text_prompt) + estimate_max_frames (adaptive max_frames cap to prevent runaway "monster breathing" generation when EOS logit fails to fire) + 12 unit tests
7273
["src-tauri/src/huddle/preprocessing.rs", 670], // TTS text preprocessing pipeline + unified split_sentences + int_to_words 0-999999 + URL trailing punctuation preservation + 23 unit tests
7374
["src-tauri/src/huddle/relay_api.rs", 520], // audio relay recv task + per-peer frame counting for remote human TTS interrupt + NIP-98 channel member query
7475
["src-tauri/src/huddle/tts.rs", 1130], // TTS pipeline + session warmup + cancel/shutdown handling + apply_fades + normalize_for_playback (per-sentence peak normalization to -6 dBFS with MAX_GAIN cap) + 24 unit tests (18 interrupt + 6 fade/normalize)

‎desktop/src-tauri/src/huddle/pocket.rs‎

Lines changed: 291 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
//! Kokoro engine but are unused — Pocket TTS does its own language ID from
4545
//! the input text and is not a diffusion model (consistency LM, one step).
4646
47+
use std::collections::HashMap;
4748
use std::path::{Path, PathBuf};
4849

4950
use sherpa_onnx::{GenerationConfig, OfflineTts, OfflineTtsConfig, Wave};
@@ -85,6 +86,35 @@ const SYNTH_NUM_STEPS: i32 = 1;
8586
/// helping of leading silence on every utterance.
8687
const SYNTH_SILENCE_SCALE: f32 = 0.0;
8788

89+
/// Mimi codec frame rate — the LM samples one latent per 80 ms. Used to convert
90+
/// a token-count estimate into a `max_frames` cap, mirroring upstream
91+
/// `pocket_tts.models.tts_model._estimate_max_gen_len`.
92+
const MIMI_FRAME_RATE: f32 = 12.5;
93+
94+
/// Upstream-derived "expected tokens per second of speech" for short inputs.
95+
/// Used by [`estimate_max_frames`] together with `GEN_SECONDS_PADDING` to cap
96+
/// runaway generation when the EOS logit fails to fire. Source:
97+
/// `pocket_tts.models.tts_model.TTSModel._TOKENS_PER_SECOND_ESTIMATE`.
98+
const TOKENS_PER_SECOND_ESTIMATE: f32 = 3.0;
99+
100+
/// Slack added to the token-derived gen-length estimate, in seconds. Source:
101+
/// `pocket_tts.models.tts_model.TTSModel._GEN_SECONDS_PADDING`.
102+
const GEN_SECONDS_PADDING: f32 = 2.0;
103+
104+
/// Hard ceiling on per-chunk generation length, in Mimi frames. Matches the
105+
/// sherpa-onnx upstream default (`offline-tts-pocket-impl.h:max_frames`) and
106+
/// is the worst-case bound we'll ever ask for. 500 frames = 40 s of audio.
107+
const MAX_FRAMES_HARD_CEILING: i32 = 500;
108+
109+
/// Word-count threshold (inclusive) below which we (a) pad the prompt with
110+
/// leading spaces and (b) ask for `frames_after_eos = 3` instead of 1.
111+
/// Matches upstream `pocket_tts.models.tts_model.prepare_text_prompt`.
112+
const SHORT_PROMPT_WORD_THRESHOLD: usize = 4;
113+
114+
/// Number of leading spaces prepended to short prompts. The upstream Python
115+
/// uses exactly 8 — keep parity rather than tuning blindly.
116+
const SHORT_PROMPT_PAD_SPACES: usize = 8;
117+
88118
// ── ONNX file names (five Pocket TTS sessions plus two JSON tables) ───────────
89119

90120
const FILE_LM_MAIN: &str = "lm_main.int8.onnx";
@@ -187,6 +217,122 @@ pub fn load_text_to_speech(model_dir: &str) -> Result<PocketTts, String> {
187217
Ok(PocketTts { inner })
188218
}
189219

220+
// ── Prompt preparation ────────────────────────────────────────────────────────
221+
222+
/// Result of [`prepare_pocket_prompt`]: a synthesizer-ready prompt plus the
223+
/// per-call generation hints derived from the original text.
224+
#[derive(Debug, Clone, PartialEq)]
225+
pub(crate) struct PreparedPrompt {
226+
/// Text to hand to `OfflineTts::generate_with_config`. Capitalized,
227+
/// punctuation-terminated, and (for short inputs) left-padded with spaces.
228+
pub text: String,
229+
/// Value to pass via `GenerationConfig.extra["frames_after_eos"]`.
230+
pub frames_after_eos: i32,
231+
/// Value to pass via `GenerationConfig.extra["max_frames"]`. Adaptive to
232+
/// text length — short prompts get a much tighter cap to prevent runaway
233+
/// "monster breathing" generation when the EOS logit fails to fire.
234+
pub max_frames: i32,
235+
}
236+
237+
/// Mirror of upstream `pocket_tts.models.tts_model.prepare_text_prompt` plus
238+
/// `_estimate_max_gen_len`. Sherpa-onnx's C++ Pocket TTS impl does not run
239+
/// these preparation steps, so short / unpunctuated / lowercase inputs can
240+
/// trigger up to 40 s of runaway generation when the EOS logit never crosses
241+
/// its threshold. We replicate the upstream Python recipe here:
242+
///
243+
/// 1. Collapse interior whitespace (already done by `preprocess_for_tts`, but
244+
/// cheap to re-check after sentence splitting).
245+
/// 2. Capitalize the first letter.
246+
/// 3. Append `.` if the text doesn't end in punctuation.
247+
/// 4. If fewer than five words, prepend `SHORT_PROMPT_PAD_SPACES` spaces and
248+
/// bump `frames_after_eos` from 1 → 3.
249+
/// 5. Compute an adaptive `max_frames` from the (post-padding) word count.
250+
///
251+
/// Returns `None` only if the input is empty after trimming — caller should
252+
/// skip synthesis in that case.
253+
pub(crate) fn prepare_pocket_prompt(input: &str) -> Option<PreparedPrompt> {
254+
let trimmed = input.trim();
255+
if trimmed.is_empty() {
256+
return None;
257+
}
258+
259+
// Collapse stray double-spaces / embedded newlines that may slip past
260+
// `preprocess_for_tts` when sentences are spliced back together.
261+
let mut cleaned = String::with_capacity(trimmed.len());
262+
let mut last_was_space = false;
263+
for ch in trimmed.chars() {
264+
let is_ws = ch.is_whitespace();
265+
if is_ws {
266+
if !last_was_space {
267+
cleaned.push(' ');
268+
}
269+
last_was_space = true;
270+
} else {
271+
cleaned.push(ch);
272+
last_was_space = false;
273+
}
274+
}
275+
276+
// Capitalize first character. Uses `to_uppercase` (multi-codepoint safe).
277+
let first = cleaned.chars().next().expect("cleaned non-empty above");
278+
if first.is_lowercase() {
279+
let upper: String = first.to_uppercase().collect();
280+
let mut iter = cleaned.chars();
281+
iter.next();
282+
cleaned = upper + iter.as_str();
283+
}
284+
285+
// Ensure terminal punctuation. Anything not in `.!?;:,` gets a period.
286+
// The upstream Python only checks `isalnum` → period, but for our agent
287+
// text we already may end in `!` `?` `.` etc. — treat any of those as OK.
288+
let last = cleaned
289+
.chars()
290+
.next_back()
291+
.expect("cleaned non-empty above");
292+
if !matches!(last, '.' | '!' | '?' | ';' | ':' | ',') {
293+
cleaned.push('.');
294+
}
295+
296+
// Word count of the *cleaned but not padded* text — padding is whitespace
297+
// only and would just lie to the threshold check below.
298+
let word_count = cleaned.split_whitespace().count();
299+
let is_short = word_count <= SHORT_PROMPT_WORD_THRESHOLD;
300+
301+
let final_text = if is_short {
302+
let mut padded = String::with_capacity(cleaned.len() + SHORT_PROMPT_PAD_SPACES);
303+
for _ in 0..SHORT_PROMPT_PAD_SPACES {
304+
padded.push(' ');
305+
}
306+
padded.push_str(&cleaned);
307+
padded
308+
} else {
309+
cleaned
310+
};
311+
312+
let frames_after_eos = if is_short { 3 } else { 1 };
313+
let max_frames = estimate_max_frames(word_count);
314+
315+
Some(PreparedPrompt {
316+
text: final_text,
317+
frames_after_eos,
318+
max_frames,
319+
})
320+
}
321+
322+
/// Convert a word count into a Mimi-frame cap, matching upstream
323+
/// `_estimate_max_gen_len`. We use words as a sentencepiece-token proxy: real
324+
/// SP tokenization runs ~1.2–1.5 tokens/word for English, which the
325+
/// `GEN_SECONDS_PADDING` slack absorbs. Saturates at
326+
/// `MAX_FRAMES_HARD_CEILING` so we never *raise* the upstream default.
327+
fn estimate_max_frames(word_count: usize) -> i32 {
328+
// Treat each word as ~1.3 tokens — within the slack envelope but a touch
329+
// generous so we don't truncate genuine short utterances.
330+
let approx_tokens = word_count as f32 * 1.3;
331+
let gen_len_sec = approx_tokens / TOKENS_PER_SECOND_ESTIMATE + GEN_SECONDS_PADDING;
332+
let frames = (gen_len_sec * MIMI_FRAME_RATE).ceil() as i32;
333+
frames.clamp(1, MAX_FRAMES_HARD_CEILING)
334+
}
335+
190336
impl PocketTts {
191337
/// Synthesise `text` with the given reference voice.
192338
///
@@ -202,16 +348,37 @@ impl PocketTts {
202348
_steps: usize,
203349
speed: f32,
204350
) -> Result<Vec<f32>, String> {
205-
if text.trim().is_empty() {
206-
return Ok(Vec::new());
207-
}
351+
// Mirror upstream pocket-tts prompt prep — without this short or
352+
// unpunctuated inputs can cause the LM's EOS logit to never trip,
353+
// producing up to 40 s of "monster breathing" garbage on the first
354+
// utterance. See `prepare_pocket_prompt` for the full recipe.
355+
let prepared = match prepare_pocket_prompt(text) {
356+
Some(p) => p,
357+
None => return Ok(Vec::new()),
358+
};
359+
360+
// Per-call generation hints sherpa-onnx forwards to
361+
// `offline-tts-pocket-impl.h`. `frames_after_eos` is bumped for short
362+
// prompts to give the model trailing room to gracefully decay; the
363+
// adaptive `max_frames` is the safety net that bounds runaway
364+
// generation when EOS never fires.
365+
let mut extra: HashMap<String, serde_json::Value> = HashMap::with_capacity(2);
366+
extra.insert(
367+
"frames_after_eos".to_string(),
368+
serde_json::Value::from(prepared.frames_after_eos),
369+
);
370+
extra.insert(
371+
"max_frames".to_string(),
372+
serde_json::Value::from(prepared.max_frames),
373+
);
208374

209375
let cfg = GenerationConfig {
210376
speed,
211377
num_steps: SYNTH_NUM_STEPS,
212378
silence_scale: SYNTH_SILENCE_SCALE,
213379
reference_audio: Some(style.samples.clone()),
214380
reference_sample_rate: style.sample_rate,
381+
extra: Some(extra),
215382
..Default::default()
216383
};
217384

@@ -221,11 +388,11 @@ impl PocketTts {
221388
// `generate_with_config` generic parameter.
222389
let audio = self
223390
.inner
224-
.generate_with_config(text, &cfg, None::<fn(&[f32], f32) -> bool>)
391+
.generate_with_config(&prepared.text, &cfg, None::<fn(&[f32], f32) -> bool>)
225392
.ok_or_else(|| {
226393
format!(
227394
"Pocket TTS synthesis failed for text ({} chars)",
228-
text.len()
395+
prepared.text.len()
229396
)
230397
})?;
231398

@@ -240,3 +407,122 @@ impl PocketTts {
240407
Ok(audio.samples().to_vec())
241408
}
242409
}
410+
411+
#[cfg(test)]
412+
mod tests {
413+
use super::*;
414+
415+
// ── prepare_pocket_prompt ────────────────────────────────────────────────
416+
417+
#[test]
418+
fn prepare_prompt_returns_none_for_empty_input() {
419+
assert!(prepare_pocket_prompt("").is_none());
420+
assert!(prepare_pocket_prompt(" ").is_none());
421+
assert!(prepare_pocket_prompt("\n\t ").is_none());
422+
}
423+
424+
#[test]
425+
fn prepare_prompt_pads_and_capitalizes_one_word() {
426+
// The "yep" case Tyler hit in production — bare lowercase one-word
427+
// utterance with no punctuation. Must be padded, capitalized, and
428+
// terminated.
429+
let out = prepare_pocket_prompt("yep").expect("non-empty");
430+
let pad = " ".repeat(SHORT_PROMPT_PAD_SPACES);
431+
assert_eq!(out.text, format!("{pad}Yep."));
432+
assert_eq!(out.frames_after_eos, 3);
433+
// 1 word → very low frame cap (well under the 500 hard ceiling).
434+
assert!(out.max_frames < MAX_FRAMES_HARD_CEILING);
435+
}
436+
437+
#[test]
438+
fn prepare_prompt_preserves_existing_punctuation() {
439+
let out = prepare_pocket_prompt("yes!").expect("non-empty");
440+
let pad = " ".repeat(SHORT_PROMPT_PAD_SPACES);
441+
assert_eq!(out.text, format!("{pad}Yes!")); // exclamation kept
442+
let out = prepare_pocket_prompt("really?").expect("non-empty");
443+
assert_eq!(out.text, format!("{pad}Really?"));
444+
}
445+
446+
#[test]
447+
fn prepare_prompt_threshold_is_inclusive_at_four_words() {
448+
// 4 words = short (padded); 5 words = long (not padded).
449+
let four = prepare_pocket_prompt("one two three four").expect("non-empty");
450+
assert!(
451+
four.text.starts_with(' '),
452+
"four-word input should be padded"
453+
);
454+
assert_eq!(four.frames_after_eos, 3);
455+
456+
let five = prepare_pocket_prompt("one two three four five").expect("non-empty");
457+
assert!(
458+
!five.text.starts_with(' '),
459+
"five-word input should NOT be padded"
460+
);
461+
assert_eq!(five.frames_after_eos, 1);
462+
}
463+
464+
#[test]
465+
fn prepare_prompt_does_not_pad_long_text() {
466+
let long = "This is a longer sentence that the model should handle just fine.";
467+
let out = prepare_pocket_prompt(long).expect("non-empty");
468+
assert!(!out.text.starts_with(' '));
469+
assert_eq!(out.frames_after_eos, 1);
470+
assert!(out.text.ends_with('.'));
471+
}
472+
473+
#[test]
474+
fn prepare_prompt_collapses_whitespace() {
475+
let out = prepare_pocket_prompt("Hello world\n\nfriend").expect("non-empty");
476+
// No padding (3 words → short → padded), but interior is collapsed.
477+
let pad = " ".repeat(SHORT_PROMPT_PAD_SPACES);
478+
assert_eq!(out.text, format!("{pad}Hello world friend."));
479+
}
480+
481+
#[test]
482+
fn prepare_prompt_does_not_double_capitalize_already_uppercase() {
483+
let out = prepare_pocket_prompt("HELLO there").expect("non-empty");
484+
let pad = " ".repeat(SHORT_PROMPT_PAD_SPACES);
485+
assert_eq!(out.text, format!("{pad}HELLO there."));
486+
}
487+
488+
#[test]
489+
fn prepare_prompt_handles_non_ascii_first_letter() {
490+
// Cyrillic lowercase 'д' → uppercase 'Д'. Must not panic / produce
491+
// mojibake.
492+
let out = prepare_pocket_prompt("дa").expect("non-empty");
493+
assert!(out.text.contains("Дa."));
494+
}
495+
496+
// ── estimate_max_frames ──────────────────────────────────────────────────
497+
498+
#[test]
499+
fn estimate_max_frames_is_tight_for_short_input() {
500+
// 1 word: 1 * 1.3 / 3.0 + 2.0 ≈ 2.43s ≈ 31 frames. Well below 500.
501+
let frames = estimate_max_frames(1);
502+
assert!(frames > 0);
503+
assert!(frames < 50, "got {frames}");
504+
}
505+
506+
#[test]
507+
fn estimate_max_frames_saturates_at_ceiling() {
508+
// 5000 words ≈ a runaway prompt; must clamp at the hard ceiling.
509+
assert_eq!(estimate_max_frames(5_000), MAX_FRAMES_HARD_CEILING);
510+
}
511+
512+
#[test]
513+
fn estimate_max_frames_grows_with_word_count() {
514+
let small = estimate_max_frames(2);
515+
let medium = estimate_max_frames(20);
516+
let large = estimate_max_frames(100);
517+
assert!(small < medium);
518+
assert!(medium < large);
519+
assert!(large <= MAX_FRAMES_HARD_CEILING);
520+
}
521+
522+
#[test]
523+
fn estimate_max_frames_never_zero() {
524+
// Sanity: even a 0-word prompt yields ≥1 frame so we never ask the
525+
// engine for an impossible cap.
526+
assert!(estimate_max_frames(0) >= 1);
527+
}
528+
}

0 commit comments

Comments
 (0)