Skip to content

Commit b17c077

Browse files
authored
fix(buzz-acp): bound busy-owner hold to prevent cross-channel starvation (#7337)
## Problem #6732 added a busy-owner hold to the ACP harness: when a scope's recorded session owner (`session_owners`) is checked out on **any** turn, `dispatch_pending` holds the scope's batch instead of dispatching it. The hold was added to keep one provider session per thread — but it is unconditional: it applies to `Conversation` scopes too, and it has no time bound. Under the default `session_policy=channel`, every channel collapses to a single `Conversation` scope, so once two channels' sessions land on the same worker (pass 2 of `try_claim` picks the first idle worker by index, so this happens quickly after any restart), channel A's mention starves behind channel B's in-flight turn — for up to the full `max_turn_duration` (7200s by default) — while other workers sit idle. The only signal is a DEBUG-level log, and the 👀 seen-reaction is added at queue admission *before* the hold decision, so the user sees the agent acknowledge the mention and then nothing. Observed in production on the first day of the v0.5.22 rollout: three separate incidents where a mention got 👀 but no turn started until an unrelated channel's turn ended on the shared worker (in the worst case the blocking turn sat in a single tool call for 6+ minutes). ## Fix One new seam, `AgentPool::hold_decision`, replaces the raw `should_hold_for_busy_owner` check in `dispatch_pending` (the predicate itself is unchanged and remains the inner check): - **`Conversation` scopes never hold.** Channel-policy channels and all DMs dispatch immediately; a busy owner means forking onto an idle worker, exactly the pre-#6732 behavior. This removes the cross-channel head-of-line blocking entirely for the default policy. - **`Thread` scopes hold for a bounded window.** `HOLD_BUSY_OWNER_TIMEOUT` (10s) is measured from the first time the batch is held (`held_since` stamp); once elapsed, the batch stops holding and forks a fresh session on an idle worker, rebuilding thread context from the relay. This preserves #6732's session-continuity intent for the momentary-busy case while capping the worst-case wait. No new timer is needed: held batches are requeued with preserved timestamps and re-evaluated on every dispatch trigger (turn end, relay event, 30s maintenance tick), so the effective worst-case re-check gap on a fully silent system is one maintenance tick. - **Holds are observable.** Holding logs at INFO and a hold expiry logs at WARN (previously DEBUG-only), and both emit observer-feed events (`busy_owner_hold`, `busy_owner_hold_forked`) with the scope, owner index, and held duration. `held_since` is derived state and is cleared on every removal path: dispatch/fork (inside `hold_decision`), `invalidate_channel_sessions`, `invalidate_scope_session`, and `switch_idle_agent_model`. ## Accepted trade-offs - A fork after an expired hold leaves the old owner's now-orphaned thread session in its session map until natural rotation/invalidation — benign, and identical to pre-#6732 fork semantics (`loadSession: false`; sessions are worker-pinned, so migration is not an option). - Under sustained pool exhaustion the hold stamp is cleared on the fork attempt and re-stamped next cycle, so the bound is effectively "timeout after a worker frees up," not absolute wall clock. ## Tests - New table test `hold_decision_covers_variant_session_busy_and_timeout` over the full input space (scope variant × idle-session presence × owner busyness × elapsed vs. window). The `Conversation` + busy-owner row is the cross-channel regression guard; the past-window row guards the bound. Both were mutation-checked: removing the variant gate or the timeout branch fails the suite. - `busy_session_owner_holds_batch_instead_of_forking_session` extended with the Hold → ForkAfterHold transition, the `Conversation` dispatch guard, and `held_since` pruning on channel invalidation. - Scope-invalidation and idle-model-switch tests extended to cover `held_since` cleanup alongside the existing `session_owners` assertions. Signed-off-by: Will Pfleger <[email protected]>
1 parent e7e2993 commit b17c077

2 files changed

Lines changed: 436 additions & 35 deletions

File tree

‎crates/buzz-acp/src/lib.rs‎

Lines changed: 163 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -3049,9 +3049,13 @@ async fn tokio_main() -> Result<()> {
30493049
// called on relay events or pool results, neither of which
30503050
// arrive when the channel is silent.
30513051
if queue.has_flushable_work() {
3052-
for (scope, thread_tags) in
3053-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3054-
{
3052+
for (scope, thread_tags) in dispatch_pending(
3053+
&mut pool,
3054+
&mut queue,
3055+
&ctx,
3056+
&mut last_activity,
3057+
observer.as_ref(),
3058+
) {
30553059
typing_channels.insert(scope, thread_tags);
30563060
}
30573061
}
@@ -3101,9 +3105,13 @@ async fn tokio_main() -> Result<()> {
31013105
// this, batches requeued during crash recovery sit idle until the
31023106
// next relay event arrives — which can be minutes on quiet channels.
31033107
if respawn_collected {
3104-
for (scope, thread_tags) in
3105-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3106-
{
3108+
for (scope, thread_tags) in dispatch_pending(
3109+
&mut pool,
3110+
&mut queue,
3111+
&ctx,
3112+
&mut last_activity,
3113+
observer.as_ref(),
3114+
) {
31073115
typing_channels.insert(scope, thread_tags);
31083116
}
31093117
}
@@ -3541,7 +3549,7 @@ async fn tokio_main() -> Result<()> {
35413549
);
35423550
if pool_ready {
35433551
for (scope, thread_tags) in
3544-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3552+
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity, observer.as_ref())
35453553
{
35463554
typing_channels.insert(scope, thread_tags);
35473555
}
@@ -3641,7 +3649,7 @@ async fn tokio_main() -> Result<()> {
36413649
} else if queue.has_flushable_work() {
36423650
tracing::debug!("heartbeat_skipped_events");
36433651
for (scope, thread_tags) in
3644-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3652+
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity, observer.as_ref())
36453653
{
36463654
typing_channels.insert(scope, thread_tags);
36473655
}
@@ -3742,9 +3750,13 @@ async fn tokio_main() -> Result<()> {
37423750
{
37433751
break;
37443752
}
3745-
for (scope, thread_tags) in
3746-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3747-
{
3753+
for (scope, thread_tags) in dispatch_pending(
3754+
&mut pool,
3755+
&mut queue,
3756+
&ctx,
3757+
&mut last_activity,
3758+
observer.as_ref(),
3759+
) {
37483760
typing_channels.insert(scope, thread_tags);
37493761
}
37503762
}
@@ -3767,9 +3779,13 @@ async fn tokio_main() -> Result<()> {
37673779
tracing::error!("all agents dead — exiting");
37683780
break;
37693781
}
3770-
for (scope, thread_tags) in
3771-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3772-
{
3782+
for (scope, thread_tags) in dispatch_pending(
3783+
&mut pool,
3784+
&mut queue,
3785+
&ctx,
3786+
&mut last_activity,
3787+
observer.as_ref(),
3788+
) {
37733789
typing_channels.insert(scope, thread_tags);
37743790
}
37753791
}
@@ -3921,9 +3937,13 @@ async fn tokio_main() -> Result<()> {
39213937
// tear down the in-flight task; on its completion the
39223938
// queue drains. We still try here in case the in-flight
39233939
// task has already returned.
3924-
for (scope, thread_tags) in
3925-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3926-
{
3940+
for (scope, thread_tags) in dispatch_pending(
3941+
&mut pool,
3942+
&mut queue,
3943+
&ctx,
3944+
&mut last_activity,
3945+
observer.as_ref(),
3946+
) {
39273947
typing_channels.insert(scope, thread_tags);
39283948
}
39293949
}
@@ -3949,9 +3969,13 @@ async fn tokio_main() -> Result<()> {
39493969
"ready",
39503970
None,
39513971
);
3952-
for (scope, thread_tags) in
3953-
dispatch_pending(&mut pool, &mut queue, &ctx, &mut last_activity)
3954-
{
3972+
for (scope, thread_tags) in dispatch_pending(
3973+
&mut pool,
3974+
&mut queue,
3975+
&ctx,
3976+
&mut last_activity,
3977+
observer.as_ref(),
3978+
) {
39553979
typing_channels.insert(scope, thread_tags);
39563980
}
39573981
}
@@ -4340,6 +4364,7 @@ fn dispatch_pending(
43404364
queue: &mut EventQueue,
43414365
ctx: &Arc<PromptContext>,
43424366
last_activity: &mut tokio::time::Instant,
4367+
observer: Option<&observer::ObserverHandle>,
43434368
) -> Vec<(scope::SessionScope, ThreadTags)> {
43444369
// Keyed by the exact session scope, not the channel: two threads dispatching
43454370
// concurrently in one channel get distinct typing entries so completing one
@@ -4348,27 +4373,82 @@ fn dispatch_pending(
43484373
// Batches held back this cycle because the worker that owns their thread's
43494374
// session is busy. They stay flushed-out of the queue (in-flight) until we
43504375
// release them at the end so `flush_next` cannot re-pick them mid-loop;
4351-
// releasing requeues them so the next dispatch (when the owner returns)
4352-
// reuses that exact session instead of forking a duplicate.
4376+
// releasing requeues them so the next dispatch (when the owner returns, or
4377+
// once the bounded hold expires) reuses that exact session or forks a fresh
4378+
// one instead of starving.
43534379
let mut held: Vec<FlushBatch> = Vec::new();
4380+
// One clock read for the whole cycle so every batch's bounded-hold window is
4381+
// measured against the same instant.
4382+
let now = std::time::Instant::now();
43544383
loop {
43554384
let batch = match queue.flush_next() {
43564385
Some(b) => b,
43574386
None => break,
43584387
};
43594388
let channel_id = batch.channel_id;
43604389
let scope = batch.scope.clone();
4361-
// Authoritative affinity: if the worker that owns this thread's session
4362-
// is checked out (busy on another turn), hold the batch rather than let
4363-
// an idle worker open a second session for the same thread.
4364-
if pool.should_hold_for_busy_owner(&scope) {
4365-
tracing::debug!(
4366-
channel = %channel_id,
4367-
scope = %scope.telemetry_label(),
4368-
"holding batch — session owner busy; awaiting its return to avoid duplicate session"
4369-
);
4370-
held.push(batch);
4371-
continue;
4390+
// Authoritative affinity, variant-gated and bounded: only a `Thread`
4391+
// scope whose session owner is checked out (busy on another turn) is
4392+
// held, and only until `HOLD_BUSY_OWNER_TIMEOUT` elapses. `Conversation`
4393+
// scopes never hold — a busy owner there forks onto another idle worker,
4394+
// so an active channel cannot starve a sibling channel on a shared
4395+
// worker. A held thread that outwaits the window forks a fresh session
4396+
// rather than starve behind an unbounded turn.
4397+
match pool.hold_decision(&scope, now, pool::HOLD_BUSY_OWNER_TIMEOUT) {
4398+
pool::HoldDecision::Hold {
4399+
held_for,
4400+
owner_index,
4401+
} => {
4402+
tracing::info!(
4403+
channel = %channel_id,
4404+
scope = %scope.telemetry_label(),
4405+
owner_index,
4406+
held_for_secs = held_for.as_secs_f64(),
4407+
"busy-owner hold — thread session owner busy; awaiting its return"
4408+
);
4409+
if let Some(observer) = observer {
4410+
observer.emit(
4411+
"busy_owner_hold",
4412+
None,
4413+
&observer::context_for(Some(channel_id), None, None),
4414+
serde_json::json!({
4415+
"scope": scope.telemetry_label(),
4416+
"ownerIndex": owner_index,
4417+
"heldForSecs": held_for.as_secs_f64(),
4418+
"timeoutSecs": pool::HOLD_BUSY_OWNER_TIMEOUT.as_secs_f64(),
4419+
}),
4420+
);
4421+
}
4422+
held.push(batch);
4423+
continue;
4424+
}
4425+
pool::HoldDecision::ForkAfterHold {
4426+
held_for,
4427+
owner_index,
4428+
} => {
4429+
tracing::warn!(
4430+
channel = %channel_id,
4431+
scope = %scope.telemetry_label(),
4432+
owner_index,
4433+
held_for_secs = held_for.as_secs_f64(),
4434+
"busy-owner hold expired — forking fresh session on an idle worker"
4435+
);
4436+
if let Some(observer) = observer {
4437+
observer.emit(
4438+
"busy_owner_hold_forked",
4439+
None,
4440+
&observer::context_for(Some(channel_id), None, None),
4441+
serde_json::json!({
4442+
"scope": scope.telemetry_label(),
4443+
"ownerIndex": owner_index,
4444+
"heldForSecs": held_for.as_secs_f64(),
4445+
}),
4446+
);
4447+
}
4448+
// Fall through to try_claim below (fork); record_scope_owner
4449+
// reassigns ownership to the new worker automatically.
4450+
}
4451+
pool::HoldDecision::Dispatch => {}
43724452
}
43734453
let typing_scope = batch
43744454
.events
@@ -6117,13 +6197,61 @@ mod owner_control_command_tests {
61176197
// A brand-new thread with no recorded owner is never held.
61186198
assert!(!pool.should_hold_for_busy_owner(&thread_scope(ch, &"d".repeat(64))));
61196199

6120-
// Channel-wide session invalidation prunes the directory so a stale
6121-
// owner can never strand a held batch.
6200+
// The bounded hold decision stamps A's first-held time, then forks once
6201+
// the window elapses rather than starving behind the busy owner.
6202+
let now = std::time::Instant::now();
6203+
assert!(
6204+
matches!(
6205+
pool.hold_decision(&ta, now, pool::HOLD_BUSY_OWNER_TIMEOUT),
6206+
pool::HoldDecision::Hold { .. }
6207+
),
6208+
"busy owner within window => hold"
6209+
);
6210+
assert!(pool.held_since_contains(&ta), "hold stamps first-held time");
6211+
assert!(
6212+
matches!(
6213+
pool.hold_decision(
6214+
&ta,
6215+
now + pool::HOLD_BUSY_OWNER_TIMEOUT,
6216+
pool::HOLD_BUSY_OWNER_TIMEOUT
6217+
),
6218+
pool::HoldDecision::ForkAfterHold { .. }
6219+
),
6220+
"elapsed window => fork on an idle worker"
6221+
);
6222+
assert!(
6223+
!pool.held_since_contains(&ta),
6224+
"fork clears the first-held stamp"
6225+
);
6226+
6227+
// A conversation scope never holds even with a busy recorded owner —
6228+
// this is the cross-channel head-of-line-blocking regression guard.
6229+
let cs = scope::SessionScope::Conversation { channel_id: ch };
6230+
pool.record_scope_owner(cs.clone(), 0);
6231+
assert_eq!(
6232+
pool.hold_decision(&cs, now, pool::HOLD_BUSY_OWNER_TIMEOUT),
6233+
pool::HoldDecision::Dispatch,
6234+
"conversation scope forks a busy owner rather than holding"
6235+
);
6236+
6237+
// Re-stamp A's hold so channel invalidation has an entry to prune.
6238+
assert!(matches!(
6239+
pool.hold_decision(&ta, now, pool::HOLD_BUSY_OWNER_TIMEOUT),
6240+
pool::HoldDecision::Hold { .. }
6241+
));
6242+
assert!(pool.held_since_contains(&ta));
6243+
6244+
// Channel-wide session invalidation prunes the owner directory and the
6245+
// hold stamps so a stale owner can never strand a held batch.
61226246
pool.invalidate_channel_sessions(ch);
61236247
assert!(
61246248
!pool.should_hold_for_busy_owner(&ta),
61256249
"owner directory pruned on channel invalidation"
61266250
);
6251+
assert!(
6252+
!pool.held_since_contains(&ta),
6253+
"hold stamps pruned on channel invalidation"
6254+
);
61276255
}
61286256

61296257
#[test]

0 commit comments

Comments
 (0)