Skip to content

fix(agents): wake top-level requester when its last parallel child settles#99396

Merged
steipete merged 16 commits into
openclaw:mainfrom
smthfoxy:fix/toplevel-requester-settle-wake
Jul 17, 2026
Merged

fix(agents): wake top-level requester when its last parallel child settles#99396
steipete merged 16 commits into
openclaw:mainfrom
smthfoxy:fix/toplevel-requester-settle-wake

Conversation

@smthfoxy

@smthfoxy smthfoxy commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Closes #99395

What Problem This Solves

Fixes an issue where a top-level session that spawned parallel subagents could remain parked after the final child settled, requiring another user message before it synthesized and delivered the combined answer. The failure also affected delete-mode children and completion announcements that exhausted their own delivery retries.

Why This Change Was Made

The top-level requester now receives the existing settle wake when its required child wave drains. The wake obligation is stored on canonical subagent_runs rows in the shared SQLite state, including frozen batch membership, admitted attempt, retry deadline, last error, and deferred row-retirement intent. Restore replays pending work and reuses the admitted idempotency key after a crash; the registry sweep recovers any durable marker left without a live processor.

Nested orchestrators retain their descendant-settle path. Cron, fire-and-forget, and single already-delivered completions remain explicit no-wake cases. No fallback store or doctor migration is added because this is new additive state.

User Impact

Parallel subagent workflows can finish and deliver their consolidated answer without a manual nudge, including across gateway restarts and transient wake-delivery failures. Delete cleanup still removes child rows, but only after the persisted settle obligation reaches delivered, terminal, abandoned, retry-exhausted, or explicit no-wake disposition.

Evidence

  • node scripts/run-vitest.mjs src/agents/subagent-announce.requester-settle-wake.test.ts src/agents/subagent-registry-lifecycle.test.ts src/agents/subagent-registry.store.sqlite.test.ts src/agents/subagent-registry.test.ts — 4 files, 236 passed.
  • node scripts/run-vitest.mjs src/agents/subagent-registry-queries.test.ts src/agents/subagent-registry.lifecycle-retry-grace.e2e.test.ts src/state/openclaw-state-db.test.ts — 3 files, 107 passed, 1 skipped.
  • node scripts/generate-kysely-types.mjs --verify — passed.
  • git diff --check and node scripts/check-max-lines-ratchet.mjs — passed.
  • Fresh maintainer autoreview — clean.

Release note: fixed top-level sessions parking after parallel subagent fan-outs; requester-settle delivery now survives restarts and bounded retries.

Contributor credit retained for @smthfoxy.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: L triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 3, 2026
@clawsweeper

clawsweeper Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 16, 2026, 4:06 PM ET / 20:06 UTC.

Summary
Adds terminal drain detection, wave-scoped requester wakes with bounded retries, retired child-result retention, root-work accounting, and regression coverage for top-level parallel subagent requesters.

PR surface: Source +543, Tests +1027. Total +1570 across 14 files.

Reproducibility: yes. The linked production incident and the branch's redacted live fan-out runs provide a high-confidence path: a top-level requester spawns parallel completion-required children, yields, and receives no further user input while completion announcements fail or are mis-tracked.

Review metrics: 2 noteworthy metrics.

  • Wake retry policy: 3 attempts with 30s and 120s backoffs. Retries improve recovery from provider stalls but can overlap when a gateway RPC times out before the underlying turn finishes.
  • Durable settle records: 0 persisted; 1 process-local retired-row ledger. The unresolved restart behavior depends directly on whether wave membership survives process replacement.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #99395
Summary: This PR is the candidate fix for the canonical top-level requester stall report; related yielded-parent work covers adjacent lifecycle projections or nested orchestrators rather than the same remaining top-level durability decision.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Obtain a lifecycle-owner decision on restart durability.
  • [P2] If durability is required, add the restart-between-siblings regression and implement the selected persistent-state model.
  • Refresh the branch against current main and rerun exact-head lifecycle validation.

Risk before merge

  • [P1] A gateway restart after delete-mode siblings are retired but before the final delivered child settles can erase the in-memory wave ledger; the remaining child may then hit the single-delivered no-wake guard and leave the requester parked.
  • [P1] Fresh retry idempotency suffixes can admit a later wake while an earlier timed-out gateway call is still completing; the prompt-level NO_REPLY escape hatch bounds but does not deterministically eliminate duplicate synthesis turns.
  • [P1] The branch is behind current main, so the final durability choice still needs a refreshed merge result and exact-head lifecycle validation.

Maintainer options:

  1. Make the settle obligation restart-durable (recommended)
    Persist the obligation or retain delete-mode rows through terminal wake completion, then prove the restart window with a focused regression.
  2. Approve the process-local boundary
    A lifecycle owner may explicitly accept that a narrow restart window can still park the requester, followed by a current-main rebase and exact-head validation.
  3. Pause pending lifecycle ownership
    Keep the PR open without further contributor churn until an owner selects the durability model.

Next step before merge

  • [P2] A lifecycle owner must choose and approve the restart-durability contract; no safe automated repair should guess between persistent obligations and delayed row retirement.

Maintainer decision needed

  • Question: Must top-level requester-settle obligations survive a gateway restart during a delete-mode parallel wave, or is process-local recovery an accepted lifecycle boundary?
  • Rationale: The branch deliberately uses an in-memory ledger, and choosing persistence versus delayed row retirement changes lifecycle ownership, cleanup timing, and restart semantics; automation should not choose that contract.
  • Likely owner: masatohoshino — Their recent work owns the durable subagent cleanup and reconciliation funnel where this restart contract is decided.
  • Options:
    • Require restart durability (recommended): Persist the pending settle obligation or defer row retirement until terminal wake completion, with a restart-between-siblings regression before merge.
    • Accept process-local recovery: Explicitly approve the restart window as a bounded limitation, document the operational consequence, and merge after a current-main refresh and exact-head validation.

Security
Cleared: The diff changes internal agent lifecycle and retry behavior without adding dependencies, secret access, external downloads, workflow permissions, or a new trust boundary.

Review details

Best possible solution:

Persist the requester-settle obligation in lifecycle-owned SQLite state, or retain delete-mode registry rows until the wake reaches a terminal outcome, then add a restart-between-siblings regression and refresh exact-head validation.

Do we have a high-confidence way to reproduce the issue?

Yes. The linked production incident and the branch's redacted live fan-out runs provide a high-confidence path: a top-level requester spawns parallel completion-required children, yields, and receives no further user input while completion announcements fail or are mis-tracked.

Is this the best way to solve the issue?

Unclear pending one lifecycle decision. A runtime-level terminal-drain wake is the correct boundary, but the best implementation must either survive restart or receive explicit owner approval for a process-local limitation.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 54eb03fcf06c.

Label changes

Label justifications:

  • P1: The PR addresses a real broken parallel-agent workflow where final user-visible synthesis can remain indefinitely parked.
  • merge-risk: 🚨 message-delivery: The new wake and retry path can determine whether a consolidated answer is missing, duplicated, or delivered after child completion.
  • merge-risk: 🚨 session-state: Wave membership, terminal settle state, and requester wake eligibility can diverge across a gateway restart.
  • merge-risk: 🚨 availability: The patch adds tracked, potentially long-running retry turns that participate in gateway restart draining and shutdown quiescence.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (logs): The PR body provides redacted after-fix live gateway logs for three-child and eight-investigator fan-outs, showing autonomous final delivery after drain and retry recovery after a failed wake turn.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides redacted after-fix live gateway logs for three-child and eight-investigator fan-outs, showing autonomous final delivery after drain and retry recovery after a failed wake turn.
Evidence reviewed

PR surface:

Source +543, Tests +1027. Total +1570 across 14 files.

View PR surface stats
Area Files Added Removed Net
Source 9 553 10 +543
Tests 5 1041 14 +1027
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 14 1594 24 +1570

What I checked:

  • Current-main gap: Current main does not contain the branch-only requester-settle-wake module, while the earlier merged nested-orchestrator wake only covers requesters represented by subagent registry runs. (src/agents/subagent-registry-lifecycle.ts, 54eb03fcf06c)
  • Process-local durability boundary: The PR stores retired delete-mode child rows in an in-memory ledger; the contributor and prior review agree that a restart between sibling retirement and final settle can erase the wave history needed to trigger the wake. (src/agents/subagent-announce.requester-settle-wake.ts, 7ef9bf5a581a)
  • Root-work blocker fixed: The current head routes requester settle wakes through independent gateway root-work continuation so restart draining cannot pass an already scheduled in-process wake. (src/agents/subagent-registry-lifecycle.ts, 7ef9bf5a581a)
  • Regression coverage: The branch adds focused coverage for wave membership, concurrency, retries, cleanup modes, result snapshots, ledger release, and gateway root-work quiescence. (src/agents/subagent-announce.requester-settle-wake.test.ts, 7ef9bf5a581a)
  • Real behavior proof: Two redacted live gateway runs show a top-level requester delivering its consolidated answer without a post-yield user message, including an eight-investigator run with completion announce exhaustion and a failed first wake attempt. (7ef9bf5a581a)
  • Exact-head validation: The supplied check inventory reports the current head's CI gate, agent test shards, lint, production/test types, dependency checks, QA smoke, security scans, and build artifacts as successful. (7ef9bf5a581a)

Likely related people:

  • masatohoshino: Authored the recent durable subagent kill/completion reconciliation that substantially shaped the cleanup and lifecycle funnel this PR extends. (role: recent lifecycle contributor; confidence: high; commits: 153ee2abbaff; files: src/agents/subagent-registry-lifecycle.ts, src/agents/subagent-registry.ts)
  • galiniliev: Introduced the merged descendant-settle wake for yielded nested orchestrators, the closest existing behavior to this top-level requester wake. (role: introduced related behavior; confidence: high; commits: 6883c6c070f6; files: src/agents/subagent-registry-lifecycle.ts, src/agents/subagent-registry.ts)
  • jalehman: Led the session and transcript SQLite storage flip, making them relevant if requester-settle obligations become durable state. (role: adjacent storage owner; confidence: medium; commits: 0a8e3604ba24; files: src/config/sessions/session-accessor.ts, src/commands/doctor-session-sqlite.ts)
  • kklouzal: Authored the merged yielded-parent terminal-state contract across agent, gateway, and UI lifecycle projections. (role: recent yielded-lifecycle contributor; confidence: medium; commits: 2333bd59dbeb; files: src/agents/agent-lifecycle-parent-state.ts, src/gateway/session-lifecycle-state.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (13 earlier review cycles; latest 8 shown)
  • reviewed 2026-07-10T10:25:33.588Z sha 19454cd :: needs changes before merge. :: [P2] Release ledger entries when no wake is needed
  • reviewed 2026-07-10T12:39:13.665Z sha 951393f :: needs maintainer review before merge. :: none
  • reviewed 2026-07-15T08:26:35.463Z sha 3619144 :: found issues before merge. :: [P1] Register the settle wake as independent root work
  • reviewed 2026-07-15T09:38:23.655Z sha 3619144 :: needs changes before merge. :: [P1] Register the settle wake as independent root work
  • reviewed 2026-07-15T23:57:01.926Z sha 7ef9bf5 :: found issues before merge. :: [P1] Preserve delete-mode wave state across restart
  • reviewed 2026-07-16T00:39:21.148Z sha 7ef9bf5 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-16T02:08:56.250Z sha 7ef9bf5 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-16T04:14:44.618Z sha 7ef9bf5 :: needs maintainer review before merge. :: none

@smthfoxy
smthfoxy force-pushed the fix/toplevel-requester-settle-wake branch from 2849035 to 698e241 Compare July 3, 2026 06:49
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 3, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 3, 2026
@smthfoxy
smthfoxy force-pushed the fix/toplevel-requester-settle-wake branch from 698e241 to d4940ed Compare July 3, 2026 08:09
@smthfoxy

smthfoxy commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Both rank-up moves from the previous review are addressed:

  1. Wave scoping fixed — the batch is now the full connected overlap component seeded at the settling run (transitive, not direct-overlap-only), with a staggered A–B–C regression (includes the whole connected drained wave for a staggered fan-out). Any last-settler computes the same component, so the idempotency key is batch-stable.
  2. After-fix real-run proof added to the PR body (Evidence section) — a real gateway built from this branch, one user message, three parallel subagents + sessions_yield: consolidated synthesis delivered ~2 s after the last child settled with 0 post-yield user messages, and the announce:requester-settle:<requester>:<3 sorted runIds> wake fired at drain carrying all three children's findings (correctly absorbed by the NO_REPLY escape hatch since the synthesis had already landed on the final announce turn).

Correction from an earlier version of this comment: the parent's "still waiting for honeybee-facts" line in the proof run was an accurate status at that moment (honeybee was still running), not a live reproduction of the incident's mis-tracking — the Evidence section now states this precisely.

@clawsweeper

clawsweeper Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 3, 2026
@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 3, 2026
@smthfoxy

smthfoxy commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a second commit and stronger evidence:

  • 59765c581d fix(agents): retry the requester settle wake on transient turn failures — a real fan-out run showed the one-shot wake turn dying to a provider stream stall after the drain, which would re-park the requester with nothing left to fire. The wake now retries up to 3 times (30 s/120 s backoff) under fresh :retry-N idempotency suffixes; silent replies still classify as delivered and never retry.
  • Evidence: the production failure class, rescued end-to-end — a deep-research-style 8-investigator fan-out (two batches, one spawn failure, staggered multi-minute children) drove the parent to NO_REPLY essentially every completion event: eight announce retry-limit give-ups, i.e. the exact parked-forever trajectory from Top-level parent that spawns parallel subagents + sessions_yield never auto-synthesizes — parks until a user nudge #99395. At drain the wake fired; its first turn died to a provider stall; the retry turn ran 4.7 minutes and delivered the full ~48k-char consolidated report — zero post-yield user messages. Full timeline in the PR body's Evidence section.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 3, 2026
@smthfoxy
smthfoxy force-pushed the fix/toplevel-requester-settle-wake branch from 46ed666 to 6ed62bf Compare July 15, 2026 23:37
@smthfoxy

Copy link
Copy Markdown
Contributor Author

Follow-up on the two reds from the previous run, both resolved on head 6ed62bf056c:

  • checks-node-compact-large-6 (subagent-registry.test.ts > keeps killed session timing root-admitted): a timing sensitivity the root-work fix exposed — that suite runs the real settle wake through the default lazy dep, and the newly tracked wake root could still be draining when the 1s waitForFast count-zero assertion fired on a loaded runner. The registry seam suite now injects a resolved maybeWakeRequesterAfterAllChildrenSettled mock like the other announce deps (wake behavior keeps its own dedicated suites), making the drain assertions deterministic. Folded into the root-work commit.
  • check-dependencies (extensions/vllm/register.runtime.ts unused): main-side — refactor(deadcode): tighten provider extension roots #108497 tightened provider extension roots minutes before the merge ref was cut, and the orphaned shim was removed upstream by refactor(vllm): remove unused registration shim #108500. Reproduced locally only with pristine upstream knip config; this branch doesn't touch vllm. The new merge ref contains the fix.

Same validation set as before rerun locally at the new head — all green.

@smthfoxy
smthfoxy force-pushed the fix/toplevel-requester-settle-wake branch from 6ed62bf to 7ef9bf5 Compare July 15, 2026 23:49
@smthfoxy

Copy link
Copy Markdown
Contributor Author

7ef9bf5a581 fixes the two reds from the previous head — both were mine, introduced by the last amend and not covered before pushing:

  • check-test-types: the new subagent-registry.test-helpers deps type needed the maybeWakeRequesterAfterAllChildrenSettled field this PR adds to the registry deps surface (one-line type addition).
  • check-lint: a stray line wrap in the injected wake mock, fixed by oxfmt.

Full CI-equivalent chain rerun locally at the pushed head — tsgo:prod, lint, format:check, check:docs, tsgo, tsgo:test, max-lines-ratchet, deadcode-exports, deadcode-unused-files, plus focused suites (settle-wake 21, lifecycle+queries+registry 226, retry-grace e2e 9) — all green.

@smthfoxy

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@smthfoxy

Copy link
Copy Markdown
Contributor Author

Round-9 review notes the previous root-work blocker is fixed on 7ef9bf5a581 (thanks for confirming), and raises one late P1: the retired-row ledger is process-local, so a restart landing between sibling retirements and the final child's delivered settle can erase wave history and hit the single-delivered no-wake guard.

The scenario is real, and the review itself marks the gating next step as a lifecycle-owner decision: whether requester-settle obligations must be restart-durable, or process-local recovery is an accepted bound. Both implementation paths are clear from this side — persisting the settle obligation in the lifecycle-owned store, or deferring delete-mode row retirement until the wake reaches a terminal outcome (which would let the existing registry persistence carry the wave across restarts) — and I'm happy to implement whichever model the owner picks, with the restart-between-siblings regression the review asks for.

Until that call is made I'll hold the branch as-is: CI is green at the current head and the in-process behavior is covered by the existing suites.

@steipete steipete self-assigned this Jul 16, 2026
smthfoxy and others added 16 commits July 17, 2026 01:24
…ttles [AI]

A top-level session (normal chat/dashboard, not itself a subagent) that
spawns parallel subagents and waits via sessions_yield never receives the
"all descendants settled -> synthesize" wake: wakeSubagentRunAfterDescendants
only targets orchestrators with a subagent-registry run record. The parent
only sees passive per-child announces, commonly mis-tracks the outstanding
set on the final completion turn (or never hears results whose announce gave
up), and then parks until a human sends a message.

Fix: when a child reaches a terminal settle (announce delivered, give-up, or
delivery suspended) and its requester has no more descendants awaiting
settle, deliver a one-shot "all spawned subagents settled - synthesize and
deliver now" wake to the parked top-level requester through the existing
announce delivery pipeline (active requesters get it steered into the live
turn; parked ones get a new origin-routed turn).

- hasDescendantRunAwaitingSettle: early-exit drain check where a suspended final
  delivery counts as settled (suspension is terminal for automatic retries).
- maybeWakeRequesterAfterAllChildrenSettled (subagent-announce.ts): scopes
  the batch to the settling child's parallel wave (the connected component of overlapping run
  lifetimes), skips nested/cron requesters (owned by the descendant-settle
  wake), skips single delivered completions and fire-and-forget children,
  and dedupes concurrent last-sibling settles via a batch-stable
  announce idempotency key (requester-settle:<requester>:<sorted runIds>).
- Trigger fires from completeCleanupBookkeeping (all cleanup-terminal paths)
  and suspendPendingFinalDelivery, and is skipped by the suspended-delivery
  discard sweep, so the wake is outcome-independent: a child whose announce
  gave up with NO_REPLY still counts toward - and can trigger - the drain.

Companion to the nested-orchestrator wake; covers the top-level case it
left open.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rukuut98qvDwzSwMp3u4kN
…es [AI]

The settle wake is the only event that ever fires after a fan-out drains, so
a wake turn lost to a transient infra failure re-parked the requester
permanently. Observed live: the wake turn survived dispatch, reconciled all
investigators, and then died to a provider stream stall (LLM idle timeout)
mid-synthesis — with nothing left to retry it.

Bounded recovery: up to 3 attempts with 30s/120s backoff. Each retry uses a
fresh `:retry-N` idempotency suffix because the gateway dedupe caches
terminal run outcomes per key, so re-dispatching the same key would no-op. A
legitimately silent wake reply already classifies as delivered and never
retries; terminal failures and an abandoned requester stop immediately.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rukuut98qvDwzSwMp3u4kN
…tle wake [AI]

cleanup="delete" retires a child's registry row in the same funnel pass
that schedules the settle wake, so a pure delete-mode fan-out could never
rebuild its drained wave from live rows: empty batch, no wake, requester
parked — the exact incident class this wake exists to close.

Retiring cleanup paths (delete branch, reconciled-killed tombstone) now
pass settledRowRetired, and the wake ledgers the in-hand record before
its first await: concurrent last-sibling settles all see the same
ledgered rows by the time any batch computation runs, keeping batch
membership and the idempotency key stable — the guarantee keep-mode gets
from rows persisting in the registry. Ledgered rows merge into the batch
candidates (a live registry row wins by runId); entries clear on
delivered/terminal wakes and the nested/requester-gone exits, with a 24h
TTL prune as the memory backstop. In-memory only: a restart degrades a
mid-wave delete fan-out to waking with the surviving rows.
… them [AI]

The delivered-finalize path clears completion.resultText (delete mode) and
the frozen delivery payload before cleanup bookkeeping runs, so the
requester-settle wake ledgered a row whose findings had already been
emptied: a cleanup="delete" fan-out woke its requester with "(no output)"
for successfully delivered children.

Capture an immutable snapshot of the entry (completion + delivery payload)
at the top of finalizeSubagentCleanup and in the resumed-announce give-up
path — before any clearing — and pass it through completeCleanupBookkeeping
to the settle wake for the retired-row branches. Keep-mode is unchanged:
its registry rows keep their result text and win over ledgered copies.

Regression: a lifecycle-driven test drives the real delivered delete-mode
cleanup (completeSubagentRun -> announce -> finalize -> funnel) and asserts
the wake's settledEntry still carries the child's result text after the
live entry was cleared and the row retired.
The zero-required and single-delivered guard exits decline to wake a wave
that has already fully drained. Ledgered delete-mode rows in such a batch
can never join a later wake (a child spawned after the drain cannot
lifetime-overlap rows that already ended, and any still-running overlap
would have failed the drain gate before these exits), so holding their
child-result snapshots for the 24h TTL was pure retention: high-volume
delete-mode fan-outs could pin large result texts in process memory.

Release the batch at both exits. No behavior change: waves with pending
overlapping children return earlier at the drain gate, which still keeps
their rows.
Upstream flipped session reads to the sqlite-backed session accessor
(openclaw#98236), which bypasses the loadSessionStore mock this e2e used for its
in-memory session fixture — requester entry lookups came back empty and
the settle wake exited at the usable-session guard. Serve loadSessionEntry
from the same fixture; everything else in the accessor stays real.
…[AI]

Two pieces of fallout from rebasing across upstream tooling changes:

- The max-lines lint budget (openclaw#107315) has no grandfathered suppression for
  subagent-announce.ts, and the settle wake + retired-row ledger pushed it
  past the cap. The wake is a coherent unit, so it moves to
  subagent-announce.requester-settle-wake.ts (matching its test file) with
  its own registry-runtime seam; the registry's lazy dep and the lifecycle
  type alias point at the new module. announce.ts drops back well under
  the budget with no suppression needed.

- The announce read-path refactor rewired the runtime barrels onto
  subagent-registry-announce-read.js, orphaning the
  hasDescendantRunAwaitingSettle re-export on subagent-registry.ts, which
  the deadcode-exports gate now flags. Removed; the retry-grace e2e reads
  the announce-read implementation directly.
…work [AI]

The settle wake was launched as a detached promise from cleanup
bookkeeping, so registry cleanup or shutdown could reach quiescence
before the wake admitted its gateway turn and the last-child completion
could still be lost during restart or teardown. Route the wake through
runWithGatewayIndependentRootWorkContinuation: a live cleanup parent
reserves the root synchronously, and restart drain now waits for the
in-flight wake. Adds a deterministic quiescence-race regression.
Store requester-settle wake obligations on subagent run rows, replay them after restart, and retire cleanup rows only after a durable outcome. Persist admitted attempts and retry deadlines, coalesce live restores, drain disconnected waves, and let the sweeper recover stranded processors.

Co-authored-by: smthfoxy <[email protected]>
Keep the bulk-cancellation regression focused on recovery after the injected persistence failure instead of assuming an exact write count; requester-settle bookkeeping now adds a legitimate durable write.

Co-authored-by: smthfoxy <[email protected]>
Use a hoisted function declaration for the lifecycle scheduler so lint accepts the intentional callback cycle without a mutable binding.

Co-authored-by: smthfoxy <[email protected]>
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XL status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Top-level parent that spawns parallel subagents + sessions_yield never auto-synthesizes — parks until a user nudge

2 participants