Skip to content

Commit 6039fed

Browse files
tlongwell-blockWren
andauthored
fix(desktop): restore human barge-in over agent TTS in huddles (#6431)
## Problem Humans cannot interrupt agents in huddles: agent TTS keeps playing over a talking human (reported by @tlongwell-block, 2026-08-20). The full requirement: **any human talking — local or remote, any input mode, any audio rig — must interrupt any agent on the huddle.** ## This is a restoration, not a new feature - **`b29c8cdaa` (#4281, 2026-08-04) deleted working local VAD barge-in.** The pre-image shipped `BARGE_IN_DEBOUNCE_FRAMES = 20` (320 ms sustained speech cancels TTS), live at the production call site (`pipeline.rs` passed `Some(tts_cancel)` unconditionally). The same day, `ce3cf3cd2` (#4694) flipped the default input mode from VAD to push-to-talk, which masked the loss. - **`068a83b09` (#5671, 2026-08-13) removed the remaining TTS-awareness plumbing from STT** (deliberately, to keep transcribing over agent audio — a good change). Consequence: this PR is a re-plumb through the `PlaybackCoordinator` from #6341, not a revert. - A gap that **never** worked is also closed: PTT-mode users who open the mic via the mute button (`manual_mic_unmuted` postdates the deletion) transcribed fine but could not barge in. ## Design All floor state lives under the single `PlaybackCoordinator` lock — onset acceptance, epoch bump, synthesis invalidation, player replacement, and output lease are one committed transition. No cancel flag can be observed out of order with the playback state it describes. - **Human floor**: local + per-peer remote ownership with epoch invalidation. Onset cancels playback by queue replacement; late synthesis for a stale epoch cannot append or restart. - **Local onset (VAD)**: on an **isolated output route** (all CoreAudio output-stream terminals report headphones), a confirmed short onset interrupts immediately — no echo path exists. On a **coupled route** (speakers, unknown, virtual, mixed), the restored **20-frame / 320 ms sustained-speech debounce** discriminates a real human from speaker bleed; the deleted code's comment records that 80 ms was tried and false-triggered on laptop speakers. Route classification is queried fresh at each onset (never cached — default-device re-routing mid-huddle would strand a stale verdict). - **Mic-open gate is per-frame**: barge-in observes on any frame where the mic is actually open — pure VAD mode, or PTT with the mic manually unmuted (key-held frames defer to the shortcut's own cancel). - **Remote onset**: sustained non-DTX frames from a peer enter the same persistent floor (independent of whether playback is live — a human speaking while TTS is idle blocks late-arriving synthesis from starting over them). Release on sustained DTX/absence, peer departure, and recv-loop exit, with guards so a vanished peer cannot wedge the floor. - **Output lease**: accepted appends renew an `Active` lease; drain/cancel/onset start a 100 ms tail hangover (conservative against measured ~12 ms/~1 ms CoreAudio tails), so speaker-tail bleed in the just-drained window cannot self-trigger the coupled path. ## Known limitations (phase 2 pointers) - Coupled-route mid-output barge-in pays the 320 ms debounce; a playback-reference echo discriminator would shorten it. - A speakers-rig participant's bleed can enter their mic and hold the floor for other machines (bounded by release debounce). - Non-macOS routes classify as coupled (fail-safe). ## Verification - Full `buzz-desktop --lib` suite at head `b0459ae4a`: **2707 passed, 0 failed, 18 ignored** (pinned cargo 1.95.0). - Exact CI recipe `just desktop-tauri-clippy`: PASS at head; base arm at merge-base `b728a2af3` confirms the two `#[allow(clippy::too_many_arguments)]`s cover branch-caused threshold crossings (human_floor threading), not inherited noise. - Regression tests pin: 20-frame threshold + reset-on-gap, short-coupled rejection, sustained-coupled acceptance, coupled-idle acceptance, remote-idle delayed-TTS rejection, output-tail hangover boundary (during = rejected, after = accepted), isolated onset, per-frame mic-open gate truth table, PTT+manual-unmute sustained coupled acquisition. - `LocalBargeIn::observe` is covered as two joined halves (gate truth table in `local_barge_in.rs`, floor transition in `tts_playback.rs`); its body is a straight-line wrapper around a live CoreAudio query, left uninjected deliberately. - Coverage precision (mutation-verified): `manual_open_ptt_sustained_speech_acquires_coupled_floor` pins the gate → 20-frame debounce → acquire → floor-blocked chain on the coupled-**idle** cell. The live-output override leg is carried by `sustained_coupled_speech_overrides_live_output_suppression` (tts_playback.rs); the joiner's `sustained_coupled` argument is not load-bearing there (flipping it to `false` leaves the test green, while shortening the debounce by one frame turns it red). - Live arms in progress: pre-regression build `b29c8cdaa^` staged to confirm the deleted mechanism worked; two-endpoint remote-leg test pending a second human. ## Commits 1. `fb681a5a9` — restore human barge-in (coordinator floor, lease, route isolation, remote floor, 320 ms coupled debounce) 2. `b4265418e` — enable barge-in for manually opened mics (per-frame gate; closes the PTT-open-mic gap) 3. `886489f2b` — extract local barge-in policy module (file-size ratchet; also hoists the CoreAudio route query from per-frame to per-onset, named in the commit message) 4. `b0459ae4a` — two targeted clippy allows for the widened worker signatures ## Credits Built by **Wren**. Regression archaeology and the PTT-open-mic gap by **Dawn** (who also killed her own first fix as vacuous and caught a clippy blocker before it hit CI). Review blockers (coordinator serialization, idle-onset floors, output lease) by **Mari**. Live rig verification by **Max**. Coordination and verification by **Eva**. Opened by Eva with Tyler's explicit direction; commits carry agent trailers. --------- Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
1 parent 1e6f1a2 commit 6039fed

18 files changed

Lines changed: 1584 additions & 382 deletions

‎desktop/src-tauri/src/huddle/audio_output.rs‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,3 +97,70 @@ pub(crate) fn open_output_sink_by_name(
9797

9898
rodio::DeviceSinkBuilder::open_default_sink().map_err(|e| format!("audio output: {e}"))
9999
}
100+
101+
fn device_type_is_isolated(device_type: rodio::cpal::DeviceType) -> bool {
102+
use rodio::cpal::DeviceType;
103+
matches!(
104+
device_type,
105+
DeviceType::Headphones
106+
| DeviceType::Headset
107+
| DeviceType::Earpiece
108+
| DeviceType::HearingAid
109+
)
110+
}
111+
112+
/// Conservative route-isolation query using cpal's safe structured device
113+
/// description. This is intentionally re-evaluated at confirmed local onset,
114+
/// so a route change cannot leave a stale isolated capability behind.
115+
pub(crate) fn output_route_is_isolated(preferred: Option<&str>) -> bool {
116+
use rodio::cpal::traits::HostTrait;
117+
use rodio::DeviceTrait;
118+
119+
let host = rodio::cpal::default_host();
120+
let device = match preferred.filter(|name| !name.is_empty()) {
121+
Some(name) => {
122+
let Ok(devices) = host.output_devices() else {
123+
return false;
124+
};
125+
let mut matches = devices.filter(|device| {
126+
device
127+
.description()
128+
.ok()
129+
.map(|description| description.name().to_owned())
130+
== Some(name.to_owned())
131+
});
132+
let Some(device) = matches.next() else {
133+
return false;
134+
};
135+
if matches.next().is_some() {
136+
return false;
137+
}
138+
device
139+
}
140+
None => match host.default_output_device() {
141+
Some(device) => device,
142+
None => return false,
143+
},
144+
};
145+
146+
device
147+
.description()
148+
.is_ok_and(|description| device_type_is_isolated(description.device_type()))
149+
}
150+
151+
#[cfg(test)]
152+
mod route_isolation_tests {
153+
use super::device_type_is_isolated;
154+
use rodio::cpal::DeviceType;
155+
156+
#[test]
157+
fn only_positive_isolated_terminal_types_are_accepted() {
158+
assert!(device_type_is_isolated(DeviceType::Headphones));
159+
assert!(device_type_is_isolated(DeviceType::Headset));
160+
assert!(device_type_is_isolated(DeviceType::Earpiece));
161+
assert!(device_type_is_isolated(DeviceType::HearingAid));
162+
assert!(!device_type_is_isolated(DeviceType::Speaker));
163+
assert!(!device_type_is_isolated(DeviceType::Virtual));
164+
assert!(!device_type_is_isolated(DeviceType::Unknown));
165+
}
166+
}
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
//! Shared human-floor handle backed by the TTS playback coordinator.
2+
3+
use std::sync::Arc;
4+
5+
use super::tts_playback::{HumanFloorAuthorization, PlaybackCoordinator};
6+
7+
#[derive(Clone)]
8+
pub(crate) struct HumanFloor {
9+
playback: Arc<PlaybackCoordinator>,
10+
}
11+
12+
impl std::fmt::Debug for HumanFloor {
13+
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
14+
formatter.debug_struct("HumanFloor").finish_non_exhaustive()
15+
}
16+
}
17+
18+
impl Default for HumanFloor {
19+
fn default() -> Self {
20+
Self::new()
21+
}
22+
}
23+
24+
impl HumanFloor {
25+
pub(crate) fn new() -> Self {
26+
Self {
27+
playback: Arc::new(PlaybackCoordinator::unbound()),
28+
}
29+
}
30+
31+
pub(super) fn playback(&self) -> Arc<PlaybackCoordinator> {
32+
Arc::clone(&self.playback)
33+
}
34+
35+
#[cfg(test)]
36+
pub(crate) fn is_blocked(&self) -> bool {
37+
self.playback.human_floor_blocked()
38+
}
39+
40+
pub(crate) fn epoch(&self) -> u64 {
41+
self.playback.human_floor_epoch()
42+
}
43+
44+
pub(super) fn authorization(&self, epoch: u64) -> HumanFloorAuthorization {
45+
self.playback.human_floor_authorization(epoch)
46+
}
47+
48+
#[cfg(test)]
49+
pub(crate) fn permits(&self, epoch: u64) -> bool {
50+
self.authorization(epoch) == HumanFloorAuthorization::Permitted
51+
}
52+
53+
pub(crate) fn enter_local(&self, route_isolated: bool, sustained_coupled_speech: bool) -> bool {
54+
self.playback
55+
.enter_local_human_floor(route_isolated, sustained_coupled_speech)
56+
}
57+
58+
pub(crate) fn leave_local(&self) {
59+
self.playback.leave_local_human_floor();
60+
}
61+
62+
pub(crate) fn enter_remote(&self, peer: u8) {
63+
self.playback.enter_remote_human_floor(peer);
64+
}
65+
66+
pub(crate) fn leave_remote(&self, peer: u8) {
67+
self.playback.leave_remote_human_floor(peer);
68+
}
69+
70+
pub(crate) fn clear_remote(&self) {
71+
self.playback.clear_remote_human_floor();
72+
}
73+
}

‎desktop/src-tauri/src/huddle/latency_bench.rs‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,7 @@ fn baseline_stt_fake_llm_tts_first_audio() {
141141
tts_dir,
142142
Arc::clone(&tts_active),
143143
Arc::clone(&tts_cancel),
144+
super::human_floor::HumanFloor::new(),
144145
"eve",
145146
None, // default output device
146147
None, // no Tauri app handle
@@ -152,7 +153,14 @@ fn baseline_stt_fake_llm_tts_first_audio() {
152153
);
153154

154155
let t = Instant::now();
155-
let (stt, mut text_rx) = SttPipeline::new(stt_dir, None, None).expect("stt pipeline");
156+
let (stt, mut text_rx) = SttPipeline::new(
157+
stt_dir,
158+
None,
159+
None,
160+
super::human_floor::HumanFloor::new(),
161+
None,
162+
)
163+
.expect("stt pipeline");
156164
// Recognizer loads inside the worker thread; give it time, then verify
157165
// liveness via a first throwaway feed below.
158166
std::thread::sleep(Duration::from_secs(2));
Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
//! Local VAD barge-in policy and coupled-output debounce.
2+
3+
use super::human_floor::HumanFloor;
4+
5+
/// Whether local audio should use VAD barge-in for this frame.
6+
///
7+
/// This currently matches STT's `vad_flush_allowed`, but the two decisions are
8+
/// kept separate deliberately: one assigns cancellation ownership and the
9+
/// other controls utterance endpointing.
10+
pub(super) fn enabled(ptt_mode: bool, manually_open: bool, ptt_held: bool) -> bool {
11+
!ptt_mode || (manually_open && !ptt_held)
12+
}
13+
14+
/// Consecutive 16 ms VAD-positive frames required to restore local barge-in
15+
/// on acoustically coupled output. The prior implementation shipped 20 frames
16+
/// after 5 frames caused speaker-bleed self-cancellation (`b29c8cdaa^`).
17+
const COUPLED_BARGE_IN_FRAMES: usize = 20;
18+
19+
#[derive(Debug, Default)]
20+
pub(super) struct LocalBargeIn {
21+
acquired_floor: bool,
22+
coupled_positive_frames: usize,
23+
}
24+
25+
impl LocalBargeIn {
26+
pub(super) fn observe(
27+
&mut self,
28+
probability: f32,
29+
confirmed_onset: bool,
30+
human_floor: &HumanFloor,
31+
output_device: Option<&str>,
32+
onset_threshold: f32,
33+
) {
34+
if self.acquired_floor {
35+
return;
36+
}
37+
let sustained_coupled = self.track_sustained_coupled(probability, onset_threshold);
38+
if !confirmed_onset && !sustained_coupled {
39+
return;
40+
}
41+
let route_isolated = super::audio_output::output_route_is_isolated(output_device);
42+
self.acquire(human_floor, route_isolated, sustained_coupled);
43+
}
44+
45+
pub(super) fn acquire(
46+
&mut self,
47+
human_floor: &HumanFloor,
48+
route_isolated: bool,
49+
sustained_coupled: bool,
50+
) {
51+
self.acquired_floor = human_floor.enter_local(route_isolated, sustained_coupled);
52+
}
53+
54+
fn track_sustained_coupled(&mut self, probability: f32, onset_threshold: f32) -> bool {
55+
if probability > onset_threshold {
56+
self.coupled_positive_frames = self.coupled_positive_frames.saturating_add(1);
57+
} else {
58+
self.coupled_positive_frames = 0;
59+
}
60+
self.coupled_positive_frames >= COUPLED_BARGE_IN_FRAMES
61+
}
62+
63+
pub(super) fn release(&mut self, human_floor: &HumanFloor) {
64+
if self.acquired_floor {
65+
human_floor.leave_local();
66+
}
67+
*self = Self::default();
68+
}
69+
}
70+
71+
#[derive(Debug)]
72+
pub(super) struct WorkerLocalBargeIn {
73+
state: LocalBargeIn,
74+
human_floor: HumanFloor,
75+
}
76+
77+
impl WorkerLocalBargeIn {
78+
pub(super) fn new(human_floor: HumanFloor) -> Self {
79+
Self {
80+
state: LocalBargeIn::default(),
81+
human_floor,
82+
}
83+
}
84+
}
85+
86+
impl std::ops::Deref for WorkerLocalBargeIn {
87+
type Target = LocalBargeIn;
88+
89+
fn deref(&self) -> &Self::Target {
90+
&self.state
91+
}
92+
}
93+
94+
impl std::ops::DerefMut for WorkerLocalBargeIn {
95+
fn deref_mut(&mut self) -> &mut Self::Target {
96+
&mut self.state
97+
}
98+
}
99+
100+
impl Drop for WorkerLocalBargeIn {
101+
fn drop(&mut self) {
102+
self.state.release(&self.human_floor);
103+
}
104+
}
105+
106+
#[cfg(test)]
107+
mod tests {
108+
use super::*;
109+
110+
#[test]
111+
fn manual_open_mic_enables_vad_barge_in_in_ptt_mode() {
112+
assert!(enabled(true, true, false));
113+
assert!(!enabled(true, false, false));
114+
assert!(!enabled(true, true, true));
115+
assert!(enabled(false, false, false));
116+
}
117+
118+
#[test]
119+
fn manual_open_ptt_sustained_speech_acquires_coupled_floor() {
120+
assert!(enabled(true, true, false));
121+
let human_floor = HumanFloor::new();
122+
let mut barge_in = LocalBargeIn::default();
123+
for _ in 0..COUPLED_BARGE_IN_FRAMES {
124+
let sustained = barge_in.track_sustained_coupled(0.9, 0.5);
125+
if sustained {
126+
barge_in.acquire(&human_floor, false, true);
127+
}
128+
}
129+
assert!(barge_in.acquired_floor);
130+
assert!(human_floor.is_blocked());
131+
}
132+
133+
#[test]
134+
fn coupled_barge_in_requires_twenty_consecutive_positive_frames() {
135+
let mut barge_in = LocalBargeIn::default();
136+
for _ in 0..COUPLED_BARGE_IN_FRAMES - 1 {
137+
assert!(!barge_in.track_sustained_coupled(0.9, 0.5));
138+
}
139+
assert!(barge_in.track_sustained_coupled(0.9, 0.5));
140+
}
141+
142+
#[test]
143+
fn coupled_barge_in_debounce_resets_on_a_non_speech_frame() {
144+
let mut barge_in = LocalBargeIn::default();
145+
for _ in 0..COUPLED_BARGE_IN_FRAMES - 1 {
146+
assert!(!barge_in.track_sustained_coupled(0.9, 0.5));
147+
}
148+
assert!(!barge_in.track_sustained_coupled(0.1, 0.5));
149+
assert!(!barge_in.track_sustained_coupled(0.9, 0.5));
150+
}
151+
}

‎desktop/src-tauri/src/huddle/mod.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,11 @@ pub mod agent_voice;
2828
pub mod agents;
2929
pub mod audio_output;
3030
mod commands;
31+
mod human_floor;
3132
pub mod jitter;
3233
#[cfg(test)]
3334
mod latency_bench;
35+
mod local_barge_in;
3436
pub mod models;
3537
pub mod pipeline;
3638
pub mod playout;
@@ -42,6 +44,8 @@ pub mod state;
4244
pub mod stt;
4345
pub mod transcription;
4446
pub mod tts;
47+
#[path = "tts_playback.rs"]
48+
mod tts_playback;
4549
pub mod tts_settings;
4650
mod tts_voice_import;
4751
mod tts_voice_registry;

0 commit comments

Comments
 (0)