Repository navigation
Commit b17c077
authored
fix(buzz-acp): bound busy-owner hold to prevent cross-channel starvation (#7337)
## Problem
#6732 added a busy-owner hold to the ACP harness: when a scope's
recorded session owner (`session_owners`) is checked out on **any**
turn, `dispatch_pending` holds the scope's batch instead of dispatching
it. The hold was added to keep one provider session per thread — but it
is unconditional: it applies to `Conversation` scopes too, and it has no
time bound.
Under the default `session_policy=channel`, every channel collapses to a
single `Conversation` scope, so once two channels' sessions land on the
same worker (pass 2 of `try_claim` picks the first idle worker by index,
so this happens quickly after any restart), channel A's mention starves
behind channel B's in-flight turn — for up to the full
`max_turn_duration` (7200s by default) — while other workers sit idle.
The only signal is a DEBUG-level log, and the 👀 seen-reaction is added
at queue admission *before* the hold decision, so the user sees the
agent acknowledge the mention and then nothing.
Observed in production on the first day of the v0.5.22 rollout: three
separate incidents where a mention got 👀 but no turn started until an
unrelated channel's turn ended on the shared worker (in the worst case
the blocking turn sat in a single tool call for 6+ minutes).
## Fix
One new seam, `AgentPool::hold_decision`, replaces the raw
`should_hold_for_busy_owner` check in `dispatch_pending` (the predicate
itself is unchanged and remains the inner check):
- **`Conversation` scopes never hold.** Channel-policy channels and all
DMs dispatch immediately; a busy owner means forking onto an idle
worker, exactly the pre-#6732 behavior. This removes the cross-channel
head-of-line blocking entirely for the default policy.
- **`Thread` scopes hold for a bounded window.**
`HOLD_BUSY_OWNER_TIMEOUT` (10s) is measured from the first time the
batch is held (`held_since` stamp); once elapsed, the batch stops
holding and forks a fresh session on an idle worker, rebuilding thread
context from the relay. This preserves #6732's session-continuity intent
for the momentary-busy case while capping the worst-case wait. No new
timer is needed: held batches are requeued with preserved timestamps and
re-evaluated on every dispatch trigger (turn end, relay event, 30s
maintenance tick), so the effective worst-case re-check gap on a fully
silent system is one maintenance tick.
- **Holds are observable.** Holding logs at INFO and a hold expiry logs
at WARN (previously DEBUG-only), and both emit observer-feed events
(`busy_owner_hold`, `busy_owner_hold_forked`) with the scope, owner
index, and held duration.
`held_since` is derived state and is cleared on every removal path:
dispatch/fork (inside `hold_decision`), `invalidate_channel_sessions`,
`invalidate_scope_session`, and `switch_idle_agent_model`.
## Accepted trade-offs
- A fork after an expired hold leaves the old owner's now-orphaned
thread session in its session map until natural rotation/invalidation —
benign, and identical to pre-#6732 fork semantics (`loadSession: false`;
sessions are worker-pinned, so migration is not an option).
- Under sustained pool exhaustion the hold stamp is cleared on the fork
attempt and re-stamped next cycle, so the bound is effectively "timeout
after a worker frees up," not absolute wall clock.
## Tests
- New table test `hold_decision_covers_variant_session_busy_and_timeout`
over the full input space (scope variant × idle-session presence × owner
busyness × elapsed vs. window). The `Conversation` + busy-owner row is
the cross-channel regression guard; the past-window row guards the
bound. Both were mutation-checked: removing the variant gate or the
timeout branch fails the suite.
- `busy_session_owner_holds_batch_instead_of_forking_session` extended
with the Hold → ForkAfterHold transition, the `Conversation` dispatch
guard, and `held_since` pruning on channel invalidation.
- Scope-invalidation and idle-model-switch tests extended to cover
`held_since` cleanup alongside the existing `session_owners` assertions.
Signed-off-by: Will Pfleger <[email protected]>1 parent e7e2993 commit b17c077
2 files changed
Lines changed: 436 additions & 35 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3049 | 3049 | | |
3050 | 3050 | | |
3051 | 3051 | | |
3052 | | - | |
3053 | | - | |
3054 | | - | |
| 3052 | + | |
| 3053 | + | |
| 3054 | + | |
| 3055 | + | |
| 3056 | + | |
| 3057 | + | |
| 3058 | + | |
3055 | 3059 | | |
3056 | 3060 | | |
3057 | 3061 | | |
| |||
3101 | 3105 | | |
3102 | 3106 | | |
3103 | 3107 | | |
3104 | | - | |
3105 | | - | |
3106 | | - | |
| 3108 | + | |
| 3109 | + | |
| 3110 | + | |
| 3111 | + | |
| 3112 | + | |
| 3113 | + | |
| 3114 | + | |
3107 | 3115 | | |
3108 | 3116 | | |
3109 | 3117 | | |
| |||
3541 | 3549 | | |
3542 | 3550 | | |
3543 | 3551 | | |
3544 | | - | |
| 3552 | + | |
3545 | 3553 | | |
3546 | 3554 | | |
3547 | 3555 | | |
| |||
3641 | 3649 | | |
3642 | 3650 | | |
3643 | 3651 | | |
3644 | | - | |
| 3652 | + | |
3645 | 3653 | | |
3646 | 3654 | | |
3647 | 3655 | | |
| |||
3742 | 3750 | | |
3743 | 3751 | | |
3744 | 3752 | | |
3745 | | - | |
3746 | | - | |
3747 | | - | |
| 3753 | + | |
| 3754 | + | |
| 3755 | + | |
| 3756 | + | |
| 3757 | + | |
| 3758 | + | |
| 3759 | + | |
3748 | 3760 | | |
3749 | 3761 | | |
3750 | 3762 | | |
| |||
3767 | 3779 | | |
3768 | 3780 | | |
3769 | 3781 | | |
3770 | | - | |
3771 | | - | |
3772 | | - | |
| 3782 | + | |
| 3783 | + | |
| 3784 | + | |
| 3785 | + | |
| 3786 | + | |
| 3787 | + | |
| 3788 | + | |
3773 | 3789 | | |
3774 | 3790 | | |
3775 | 3791 | | |
| |||
3921 | 3937 | | |
3922 | 3938 | | |
3923 | 3939 | | |
3924 | | - | |
3925 | | - | |
3926 | | - | |
| 3940 | + | |
| 3941 | + | |
| 3942 | + | |
| 3943 | + | |
| 3944 | + | |
| 3945 | + | |
| 3946 | + | |
3927 | 3947 | | |
3928 | 3948 | | |
3929 | 3949 | | |
| |||
3949 | 3969 | | |
3950 | 3970 | | |
3951 | 3971 | | |
3952 | | - | |
3953 | | - | |
3954 | | - | |
| 3972 | + | |
| 3973 | + | |
| 3974 | + | |
| 3975 | + | |
| 3976 | + | |
| 3977 | + | |
| 3978 | + | |
3955 | 3979 | | |
3956 | 3980 | | |
3957 | 3981 | | |
| |||
4340 | 4364 | | |
4341 | 4365 | | |
4342 | 4366 | | |
| 4367 | + | |
4343 | 4368 | | |
4344 | 4369 | | |
4345 | 4370 | | |
| |||
4348 | 4373 | | |
4349 | 4374 | | |
4350 | 4375 | | |
4351 | | - | |
4352 | | - | |
| 4376 | + | |
| 4377 | + | |
| 4378 | + | |
4353 | 4379 | | |
| 4380 | + | |
| 4381 | + | |
| 4382 | + | |
4354 | 4383 | | |
4355 | 4384 | | |
4356 | 4385 | | |
4357 | 4386 | | |
4358 | 4387 | | |
4359 | 4388 | | |
4360 | 4389 | | |
4361 | | - | |
4362 | | - | |
4363 | | - | |
4364 | | - | |
4365 | | - | |
4366 | | - | |
4367 | | - | |
4368 | | - | |
4369 | | - | |
4370 | | - | |
4371 | | - | |
| 4390 | + | |
| 4391 | + | |
| 4392 | + | |
| 4393 | + | |
| 4394 | + | |
| 4395 | + | |
| 4396 | + | |
| 4397 | + | |
| 4398 | + | |
| 4399 | + | |
| 4400 | + | |
| 4401 | + | |
| 4402 | + | |
| 4403 | + | |
| 4404 | + | |
| 4405 | + | |
| 4406 | + | |
| 4407 | + | |
| 4408 | + | |
| 4409 | + | |
| 4410 | + | |
| 4411 | + | |
| 4412 | + | |
| 4413 | + | |
| 4414 | + | |
| 4415 | + | |
| 4416 | + | |
| 4417 | + | |
| 4418 | + | |
| 4419 | + | |
| 4420 | + | |
| 4421 | + | |
| 4422 | + | |
| 4423 | + | |
| 4424 | + | |
| 4425 | + | |
| 4426 | + | |
| 4427 | + | |
| 4428 | + | |
| 4429 | + | |
| 4430 | + | |
| 4431 | + | |
| 4432 | + | |
| 4433 | + | |
| 4434 | + | |
| 4435 | + | |
| 4436 | + | |
| 4437 | + | |
| 4438 | + | |
| 4439 | + | |
| 4440 | + | |
| 4441 | + | |
| 4442 | + | |
| 4443 | + | |
| 4444 | + | |
| 4445 | + | |
| 4446 | + | |
| 4447 | + | |
| 4448 | + | |
| 4449 | + | |
| 4450 | + | |
| 4451 | + | |
4372 | 4452 | | |
4373 | 4453 | | |
4374 | 4454 | | |
| |||
6117 | 6197 | | |
6118 | 6198 | | |
6119 | 6199 | | |
6120 | | - | |
6121 | | - | |
| 6200 | + | |
| 6201 | + | |
| 6202 | + | |
| 6203 | + | |
| 6204 | + | |
| 6205 | + | |
| 6206 | + | |
| 6207 | + | |
| 6208 | + | |
| 6209 | + | |
| 6210 | + | |
| 6211 | + | |
| 6212 | + | |
| 6213 | + | |
| 6214 | + | |
| 6215 | + | |
| 6216 | + | |
| 6217 | + | |
| 6218 | + | |
| 6219 | + | |
| 6220 | + | |
| 6221 | + | |
| 6222 | + | |
| 6223 | + | |
| 6224 | + | |
| 6225 | + | |
| 6226 | + | |
| 6227 | + | |
| 6228 | + | |
| 6229 | + | |
| 6230 | + | |
| 6231 | + | |
| 6232 | + | |
| 6233 | + | |
| 6234 | + | |
| 6235 | + | |
| 6236 | + | |
| 6237 | + | |
| 6238 | + | |
| 6239 | + | |
| 6240 | + | |
| 6241 | + | |
| 6242 | + | |
| 6243 | + | |
| 6244 | + | |
| 6245 | + | |
6122 | 6246 | | |
6123 | 6247 | | |
6124 | 6248 | | |
6125 | 6249 | | |
6126 | 6250 | | |
| 6251 | + | |
| 6252 | + | |
| 6253 | + | |
| 6254 | + | |
6127 | 6255 | | |
6128 | 6256 | | |
6129 | 6257 | | |
| |||
0 commit comments