Implement durable system-event fallback for subagent completion when handoff is still pending#90740
Closed
bradfreels wants to merge 1 commit into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace the give-up branch in
subagent-announce-delivery.ts(when the direct subagent-announce handoff firescompletion_handoff_pending) with a durable fallback that enqueues the completion trigger into the requester's in-process system-event inbox via the existingenqueueSystemEventprimitive (src/infra/system-events.ts). The original{delivered:false, error:"completion agent handoff is still pending"}shape is preserved as a try/catch safety net if the enqueue itself fails.Motivation
Today's
openclaw tasks audit --jsonon a self-hosted gateway (srv1355327, OpenClaw2026.5.26) showed 34delivery_failed+ 37stale_blockedrows, all with error"completion agent handoff is still pending". RCA traced every one to the same code path: when the parent session is active but non-streaming (yielded, queued behind work, mid compaction-prep), the gateway accepts the agent-run dispatch but returns a non-terminal status. The existing give-up logic exhausted its retries and discarded the child's completion content — even though the content was preserved on disk in the child's worktree the whole time.The new durable-fallback path uses the same primitive (
enqueueSystemEvent) that's already used for:So no new infrastructure surface is introduced. The fallback is keyed
subagent-announce:${params.directIdempotencyKey}, idempotent on(text, contextKey, deliveryContext), and drained by the parent on next turn-start via the standarddrainSystemEventsconsumer.Changes
6 files, +365/-9:
src/agents/subagent-announce-delivery.ts— replace the give-up return with durable-queue enqueue + try/catch safety netsrc/agents/subagent-announce-dispatch.ts— extendSubagentDeliveryPathunion from"steered" | "direct" | "none"to include"durable_queue"src/plugin-sdk/agent-harness-task-runtime.ts— extendisDurableAgentHarnessCompletionDeliveryto treatdurable_queuepath as durable (so monitor retry does not re-fire)src/agents/subagent-announce-delivery.test.ts— 5 new unit tests covering the durable-queue branch + counterfactual probessrc/agents/embedded-agent-runner/sessions-yield.orchestration.test.ts— 1 new integration test reproducing the original bug + verifying the fix end-to-endsrc/plugin-sdk/agent-harness-task-runtime.test.ts— 1 new test forisDurableAgentHarnessCompletionDelivery(durable_queue)Architectural invariant
The fix is self-contained at runtime: no code path reads
warden-status,workspace/memory, or.openclaw/ledger paths to make a delivery decision. Status files remain observability-only, not load-bearing. Verified via grep gate in CI patch.Tests
Local on commit
49e43d03(6.2 lane), re-run on rebasedc9dc427a:subagent-announce-delivery.test.ts: 102/102 pass (97 pre-existing + 5 new)sessions-yield.orchestration.test.ts: 5/5 pass (4 pre-existing + 1 new)agent-harness-task-runtime.test.ts: 7/7 pass (6 pre-existing + 1 new)src/secrets/config-io.ts:12is unchanged from main.)Independent review
A second-pass review on the 6.2 lane found all 11 checklist items + all 6 counterfactual probes green. Notable items probed:
directIdempotencyKeycollision risk: not exploitable in practice —enqueueSystemEventdedupes on the full(text, contextKey, deliveryContext)triple, notcontextKeyalone, so worst case is degraded observability not lost delivery.enqueueSystemEventreturning success and parent's nextdrainSystemEventscall: thin silent-loss window exists at parent process death; recommended as follow-up enhancement (TTL/staleness sweep overpeekSystemEventEntries) but not a regression of current behavior since the give-up branch already lost deliveries during this window.reasonfield) exactly preserved in the catch fallback — no behavior delta for callers that match onerrortext only.Compatibility
{delivered:true, path:"durable_queue"}), not a change to existing failure shapes.enqueueSystemEventitself throws (try/catch safety net).Related
Reported as a recurring class of zombie task-runs in self-hosted deployments. Operator-side watchdog cron
b03ce892-ed27-4b1b-a3c5-8abde6c0660aexists as a downstream mitigation; can be retired once this fix ships in the next release.