fix(agents): wake top-level requester when its last parallel child settles#99396
Conversation
|
Codex review: needs maintainer review before merge. Reviewed July 16, 2026, 4:06 PM ET / 20:06 UTC. Summary PR surface: Source +543, Tests +1027. Total +1570 across 14 files. Reproducibility: yes. The linked production incident and the branch's redacted live fan-out runs provide a high-confidence path: a top-level requester spawns parallel completion-required children, yields, and receives no further user input while completion announcements fail or are mis-tracked. Review metrics: 2 noteworthy metrics.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Persist the requester-settle obligation in lifecycle-owned SQLite state, or retain delete-mode registry rows until the wake reaches a terminal outcome, then add a restart-between-siblings regression and refresh exact-head validation. Do we have a high-confidence way to reproduce the issue? Yes. The linked production incident and the branch's redacted live fan-out runs provide a high-confidence path: a top-level requester spawns parallel completion-required children, yields, and receives no further user input while completion announcements fail or are mis-tracked. Is this the best way to solve the issue? Unclear pending one lifecycle decision. A runtime-level terminal-drain wake is the correct boundary, but the best implementation must either survive restart or receive explicit owner approval for a process-local limitation. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 54eb03fcf06c. Label changesLabel justifications:
Evidence reviewedPR surface: Source +543, Tests +1027. Total +1570 across 14 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (13 earlier review cycles; latest 8 shown)
|
2849035 to
698e241
Compare
698e241 to
d4940ed
Compare
|
@clawsweeper re-review Both rank-up moves from the previous review are addressed:
Correction from an earlier version of this comment: the parent's "still waiting for honeybee-facts" line in the proof run was an accurate status at that moment (honeybee was still running), not a live reproduction of the incident's mis-tracking — the Evidence section now states this precisely. |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Pushed a second commit and stronger evidence:
|
46ed666 to
6ed62bf
Compare
|
Follow-up on the two reds from the previous run, both resolved on head
Same validation set as before rerun locally at the new head — all green. |
6ed62bf to
7ef9bf5
Compare
|
Full CI-equivalent chain rerun locally at the pushed head — tsgo:prod, lint, format:check, check:docs, tsgo, tsgo:test, max-lines-ratchet, deadcode-exports, deadcode-unused-files, plus focused suites (settle-wake 21, lifecycle+queries+registry 226, retry-grace e2e 9) — all green. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Round-9 review notes the previous root-work blocker is fixed on The scenario is real, and the review itself marks the gating next step as a lifecycle-owner decision: whether requester-settle obligations must be restart-durable, or process-local recovery is an accepted bound. Both implementation paths are clear from this side — persisting the settle obligation in the lifecycle-owned store, or deferring delete-mode row retirement until the wake reaches a terminal outcome (which would let the existing registry persistence carry the wave across restarts) — and I'm happy to implement whichever model the owner picks, with the restart-between-siblings regression the review asks for. Until that call is made I'll hold the branch as-is: CI is green at the current head and the in-process behavior is covered by the existing suites. |
…ttles [AI] A top-level session (normal chat/dashboard, not itself a subagent) that spawns parallel subagents and waits via sessions_yield never receives the "all descendants settled -> synthesize" wake: wakeSubagentRunAfterDescendants only targets orchestrators with a subagent-registry run record. The parent only sees passive per-child announces, commonly mis-tracks the outstanding set on the final completion turn (or never hears results whose announce gave up), and then parks until a human sends a message. Fix: when a child reaches a terminal settle (announce delivered, give-up, or delivery suspended) and its requester has no more descendants awaiting settle, deliver a one-shot "all spawned subagents settled - synthesize and deliver now" wake to the parked top-level requester through the existing announce delivery pipeline (active requesters get it steered into the live turn; parked ones get a new origin-routed turn). - hasDescendantRunAwaitingSettle: early-exit drain check where a suspended final delivery counts as settled (suspension is terminal for automatic retries). - maybeWakeRequesterAfterAllChildrenSettled (subagent-announce.ts): scopes the batch to the settling child's parallel wave (the connected component of overlapping run lifetimes), skips nested/cron requesters (owned by the descendant-settle wake), skips single delivered completions and fire-and-forget children, and dedupes concurrent last-sibling settles via a batch-stable announce idempotency key (requester-settle:<requester>:<sorted runIds>). - Trigger fires from completeCleanupBookkeeping (all cleanup-terminal paths) and suspendPendingFinalDelivery, and is skipped by the suspended-delivery discard sweep, so the wake is outcome-independent: a child whose announce gave up with NO_REPLY still counts toward - and can trigger - the drain. Companion to the nested-orchestrator wake; covers the top-level case it left open. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Rukuut98qvDwzSwMp3u4kN
…es [AI] The settle wake is the only event that ever fires after a fan-out drains, so a wake turn lost to a transient infra failure re-parked the requester permanently. Observed live: the wake turn survived dispatch, reconciled all investigators, and then died to a provider stream stall (LLM idle timeout) mid-synthesis — with nothing left to retry it. Bounded recovery: up to 3 attempts with 30s/120s backoff. Each retry uses a fresh `:retry-N` idempotency suffix because the gateway dedupe caches terminal run outcomes per key, so re-dispatching the same key would no-op. A legitimately silent wake reply already classifies as delivered and never retries; terminal failures and an abandoned requester stop immediately. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Rukuut98qvDwzSwMp3u4kN
…tle wake [AI] cleanup="delete" retires a child's registry row in the same funnel pass that schedules the settle wake, so a pure delete-mode fan-out could never rebuild its drained wave from live rows: empty batch, no wake, requester parked — the exact incident class this wake exists to close. Retiring cleanup paths (delete branch, reconciled-killed tombstone) now pass settledRowRetired, and the wake ledgers the in-hand record before its first await: concurrent last-sibling settles all see the same ledgered rows by the time any batch computation runs, keeping batch membership and the idempotency key stable — the guarantee keep-mode gets from rows persisting in the registry. Ledgered rows merge into the batch candidates (a live registry row wins by runId); entries clear on delivered/terminal wakes and the nested/requester-gone exits, with a 24h TTL prune as the memory backstop. In-memory only: a restart degrades a mid-wave delete fan-out to waking with the surviving rows.
… them [AI] The delivered-finalize path clears completion.resultText (delete mode) and the frozen delivery payload before cleanup bookkeeping runs, so the requester-settle wake ledgered a row whose findings had already been emptied: a cleanup="delete" fan-out woke its requester with "(no output)" for successfully delivered children. Capture an immutable snapshot of the entry (completion + delivery payload) at the top of finalizeSubagentCleanup and in the resumed-announce give-up path — before any clearing — and pass it through completeCleanupBookkeeping to the settle wake for the retired-row branches. Keep-mode is unchanged: its registry rows keep their result text and win over ledgered copies. Regression: a lifecycle-driven test drives the real delivered delete-mode cleanup (completeSubagentRun -> announce -> finalize -> funnel) and asserts the wake's settledEntry still carries the child's result text after the live entry was cleared and the row retired.
The zero-required and single-delivered guard exits decline to wake a wave that has already fully drained. Ledgered delete-mode rows in such a batch can never join a later wake (a child spawned after the drain cannot lifetime-overlap rows that already ended, and any still-running overlap would have failed the drain gate before these exits), so holding their child-result snapshots for the 24h TTL was pure retention: high-volume delete-mode fan-outs could pin large result texts in process memory. Release the batch at both exits. No behavior change: waves with pending overlapping children return earlier at the drain gate, which still keeps their rows.
Upstream flipped session reads to the sqlite-backed session accessor (openclaw#98236), which bypasses the loadSessionStore mock this e2e used for its in-memory session fixture — requester entry lookups came back empty and the settle wake exited at the usable-session guard. Serve loadSessionEntry from the same fixture; everything else in the accessor stays real.
…[AI] Two pieces of fallout from rebasing across upstream tooling changes: - The max-lines lint budget (openclaw#107315) has no grandfathered suppression for subagent-announce.ts, and the settle wake + retired-row ledger pushed it past the cap. The wake is a coherent unit, so it moves to subagent-announce.requester-settle-wake.ts (matching its test file) with its own registry-runtime seam; the registry's lazy dep and the lifecycle type alias point at the new module. announce.ts drops back well under the budget with no suppression needed. - The announce read-path refactor rewired the runtime barrels onto subagent-registry-announce-read.js, orphaning the hasDescendantRunAwaitingSettle re-export on subagent-registry.ts, which the deadcode-exports gate now flags. Removed; the retry-grace e2e reads the announce-read implementation directly.
…work [AI] The settle wake was launched as a detached promise from cleanup bookkeeping, so registry cleanup or shutdown could reach quiescence before the wake admitted its gateway turn and the last-child completion could still be lost during restart or teardown. Route the wake through runWithGatewayIndependentRootWorkContinuation: a live cleanup parent reserves the root synchronously, and restart drain now waits for the in-flight wake. Adds a deterministic quiescence-race regression.
Store requester-settle wake obligations on subagent run rows, replay them after restart, and retire cleanup rows only after a durable outcome. Persist admitted attempts and retry deadlines, coalesce live restores, drain disconnected waves, and let the sweeper recover stranded processors. Co-authored-by: smthfoxy <[email protected]>
Keep the bulk-cancellation regression focused on recovery after the injected persistence failure instead of assuming an exact write count; requester-settle bookkeeping now adds a legitimate durable write. Co-authored-by: smthfoxy <[email protected]>
Use a hoisted function declaration for the lifecycle scheduler so lint accepts the intentional callback cycle without a mutable binding. Co-authored-by: smthfoxy <[email protected]>
Co-authored-by: smthfoxy <[email protected]>
Co-authored-by: smthfoxy <[email protected]>
Co-authored-by: smthfoxy <[email protected]>
Co-authored-by: smthfoxy <[email protected]>
Co-authored-by: smthfoxy <[email protected]>
|
Merged via squash.
|
Closes #99395
What Problem This Solves
Fixes an issue where a top-level session that spawned parallel subagents could remain parked after the final child settled, requiring another user message before it synthesized and delivered the combined answer. The failure also affected delete-mode children and completion announcements that exhausted their own delivery retries.
Why This Change Was Made
The top-level requester now receives the existing settle wake when its required child wave drains. The wake obligation is stored on canonical
subagent_runsrows in the shared SQLite state, including frozen batch membership, admitted attempt, retry deadline, last error, and deferred row-retirement intent. Restore replays pending work and reuses the admitted idempotency key after a crash; the registry sweep recovers any durable marker left without a live processor.Nested orchestrators retain their descendant-settle path. Cron, fire-and-forget, and single already-delivered completions remain explicit no-wake cases. No fallback store or doctor migration is added because this is new additive state.
User Impact
Parallel subagent workflows can finish and deliver their consolidated answer without a manual nudge, including across gateway restarts and transient wake-delivery failures. Delete cleanup still removes child rows, but only after the persisted settle obligation reaches delivered, terminal, abandoned, retry-exhausted, or explicit no-wake disposition.
Evidence
node scripts/run-vitest.mjs src/agents/subagent-announce.requester-settle-wake.test.ts src/agents/subagent-registry-lifecycle.test.ts src/agents/subagent-registry.store.sqlite.test.ts src/agents/subagent-registry.test.ts— 4 files, 236 passed.node scripts/run-vitest.mjs src/agents/subagent-registry-queries.test.ts src/agents/subagent-registry.lifecycle-retry-grace.e2e.test.ts src/state/openclaw-state-db.test.ts— 3 files, 107 passed, 1 skipped.node scripts/generate-kysely-types.mjs --verify— passed.git diff --checkandnode scripts/check-max-lines-ratchet.mjs— passed.Release note: fixed top-level sessions parking after parallel subagent fan-outs; requester-settle delivery now survives restarts and bounded retries.
Contributor credit retained for @smthfoxy.