Skip to content

Commit 62486fd

Browse files
wesbillmanMongo
andcommitted
fix(buzz-acp): reap respawn JoinSet, gate re-sleep on real in-flight
The idle-pool-sleep teardown gate used `!respawn_tasks.is_empty()` as a respawn busy bit, but completed respawn tasks are never joined: their payloads arrive out-of-band via `respawn_rx` and the only JoinSet op is `.shutdown()` at teardown. Tokio retains finished tasks in a JoinSet until `join_next`, so after the first slot refill or crash recovery `!respawn_tasks.is_empty()` stays true forever — `idle_pool_sleep_due` could then never pass and a woken lazy pool would never re-sleep, defeating the feature. Fix uses the authoritative signal that already exists: `any_respawn_in_flight(&crash_history)`, backed by each slot's `respawn_in_flight` flag (set at all three spawn sites, cleared when the `respawn_rx` payload is received, and guaranteed to clear even on panic via RespawnGuard). The gate call site now reads that instead of JoinSet occupancy, keeping `!wake_tasks.is_empty()` (which self-drains). Also add a non-blocking JoinSet reaper (`respawn_tasks.join_next().now_or_never()`) after the `respawn_rx` drain, so completed handles are released and the set does not grow without bound — same pattern already used for `pool.join_set`. Regression: `respawn_in_flight_signal_gates_then_clears_for_sleep` proves the busy->eligible transition through the real predicate, and `completed_respawn_tasks_are_reaped_from_the_joinset` proves the reaper drains finished tasks. Co-authored-by: Mongo <5c25403eab7271f9f94ddd4f2b270e8cac2c92e2c830c51877cca6ec974ffb3f@buzz.block.builderlab.xyz> Signed-off-by: Wes <[email protected]>
1 parent edac9dc commit 62486fd

1 file changed

Lines changed: 86 additions & 1 deletion

File tree

‎crates/buzz-acp/src/lib.rs‎

Lines changed: 86 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1637,6 +1637,79 @@ mod idle_pool_sleep_tests {
16371637
false
16381638
));
16391639
}
1640+
1641+
fn slot(respawn_in_flight: bool) -> SlotCircuit {
1642+
SlotCircuit {
1643+
crash_times: Vec::new(),
1644+
open_until: None,
1645+
respawn_in_flight,
1646+
}
1647+
}
1648+
1649+
// The call-site signal for the `wake_or_respawn_in_flight` gate is
1650+
// `any_respawn_in_flight(&crash_history)`, NOT `!respawn_tasks.is_empty()`.
1651+
// Regression for the PR #5682 review blocker: completed respawn tasks are
1652+
// never joined from the `respawn_tasks` JoinSet (their payloads arrive
1653+
// out-of-band via `respawn_rx`), so `!is_empty()` stays true forever after
1654+
// the first refill/crash recovery and the pool could never re-sleep. The
1655+
// authoritative signal clears per-slot when the payload is received.
1656+
#[test]
1657+
fn respawn_in_flight_signal_gates_then_clears_for_sleep() {
1658+
let (last, now, bound) = ready_after_bound();
1659+
1660+
// A respawn in flight for any slot defers sleep.
1661+
let busy = [slot(false), slot(true), slot(false)];
1662+
assert!(any_respawn_in_flight(&busy));
1663+
assert!(!idle_pool_sleep_due(
1664+
true,
1665+
last,
1666+
now,
1667+
bound,
1668+
false,
1669+
false,
1670+
false,
1671+
any_respawn_in_flight(&busy),
1672+
));
1673+
1674+
// Once the respawn completes (payload received → flag cleared), the
1675+
// signal goes false and the otherwise-quiet pool becomes sleep-eligible
1676+
// — even though a naive `!JoinSet.is_empty()` would still be stuck true.
1677+
let quiet = [slot(false), slot(false), slot(false)];
1678+
assert!(!any_respawn_in_flight(&quiet));
1679+
assert!(idle_pool_sleep_due(
1680+
true,
1681+
last,
1682+
now,
1683+
bound,
1684+
false,
1685+
false,
1686+
false,
1687+
any_respawn_in_flight(&quiet),
1688+
));
1689+
}
1690+
1691+
// The reaper (`respawn_tasks.join_next().now_or_never()` loop) must drain
1692+
// completed handles so the JoinSet does not grow without bound and so
1693+
// `!respawn_tasks.is_empty()` cannot become a permanent busy bit if anyone
1694+
// ever reintroduces it as the gate signal.
1695+
#[tokio::test]
1696+
async fn completed_respawn_tasks_are_reaped_from_the_joinset() {
1697+
let mut respawn_tasks: tokio::task::JoinSet<()> = tokio::task::JoinSet::new();
1698+
respawn_tasks.spawn(async {});
1699+
respawn_tasks.spawn(async {});
1700+
// Let both tasks run to completion.
1701+
tokio::task::yield_now().await;
1702+
tokio::time::sleep(Duration::from_millis(10)).await;
1703+
1704+
// The reaper drains finished handles non-blockingly.
1705+
while respawn_tasks.join_next().now_or_never().flatten().is_some() {}
1706+
1707+
assert!(
1708+
respawn_tasks.is_empty(),
1709+
"completed respawn tasks must be reaped so the set does not wedge \
1710+
the idle-sleep gate or grow unbounded"
1711+
);
1712+
}
16401713
}
16411714

16421715
pub fn run() -> Result<()> {
@@ -2255,6 +2328,17 @@ async fn tokio_main() -> Result<()> {
22552328
}
22562329
}
22572330
}
2331+
// Reap completed respawn handles from the JoinSet. Payloads are
2332+
// delivered out-of-band through `respawn_rx` (drained above), so the
2333+
// JoinSet is never joined by the normal flow — Tokio retains finished
2334+
// tasks until `join_next`, so without this the set grows on every
2335+
// refill/crash recovery and `!respawn_tasks.is_empty()` would stay true
2336+
// forever. Non-blocking (`now_or_never`), same pattern as
2337+
// `drain_ready_join_results` for `pool.join_set`. The authoritative
2338+
// in-flight signal is `any_respawn_in_flight(&crash_history)` (each
2339+
// slot's `respawn_in_flight` is cleared when its payload is received),
2340+
// not JoinSet occupancy.
2341+
while respawn_tasks.join_next().now_or_never().flatten().is_some() {}
22582342
// Flush requeued events that were waiting for a live agent. Without
22592343
// this, batches requeued during crash recovery sit idle until the
22602344
// next relay event arrives — which can be minutes on quiet channels.
@@ -2769,7 +2853,8 @@ async fn tokio_main() -> Result<()> {
27692853
queue.has_in_flight() || heartbeat_in_flight,
27702854
!pool.join_set.is_empty(),
27712855
queue.has_undispatched_work(),
2772-
!wake_tasks.is_empty() || !respawn_tasks.is_empty(),
2856+
!wake_tasks.is_empty()
2857+
|| any_respawn_in_flight(&crash_history),
27732858
) {
27742859
tracing::info!(
27752860
idle_pool_sleep_seconds = config.idle_pool_sleep_secs,

0 commit comments

Comments
 (0)