Skip to content

Implement durable system-event fallback for subagent completion when handoff is still pending#90740

Closed
bradfreels wants to merge 1 commit into
openclaw:mainfrom
bradfreels:clawflow/2026-06-05-durable-subagent-completion-rebased-on-36d9241c
Closed

Implement durable system-event fallback for subagent completion when handoff is still pending#90740
bradfreels wants to merge 1 commit into
openclaw:mainfrom
bradfreels:clawflow/2026-06-05-durable-subagent-completion-rebased-on-36d9241c

Conversation

@bradfreels

Copy link
Copy Markdown

Summary

Replace the give-up branch in subagent-announce-delivery.ts (when the direct subagent-announce handoff fires completion_handoff_pending) with a durable fallback that enqueues the completion trigger into the requester's in-process system-event inbox via the existing enqueueSystemEvent primitive (src/infra/system-events.ts). The original {delivered:false, error:"completion agent handoff is still pending"} shape is preserved as a try/catch safety net if the enqueue itself fails.

Motivation

Today's openclaw tasks audit --json on a self-hosted gateway (srv1355327, OpenClaw 2026.5.26) showed 34 delivery_failed + 37 stale_blocked rows, all with error "completion agent handoff is still pending". RCA traced every one to the same code path: when the parent session is active but non-streaming (yielded, queued behind work, mid compaction-prep), the gateway accepts the agent-run dispatch but returns a non-terminal status. The existing give-up logic exhausted its retries and discarded the child's completion content — even though the content was preserved on disk in the child's worktree the whole time.

The new durable-fallback path uses the same primitive (enqueueSystemEvent) that's already used for:

  • Cron job firings
  • Hook events
  • Restart-sentinel deliveries
  • ACP child-spawn announcements
  • Cross-session monitor events

So no new infrastructure surface is introduced. The fallback is keyed subagent-announce:${params.directIdempotencyKey}, idempotent on (text, contextKey, deliveryContext), and drained by the parent on next turn-start via the standard drainSystemEvents consumer.

Changes

6 files, +365/-9:

  • src/agents/subagent-announce-delivery.ts — replace the give-up return with durable-queue enqueue + try/catch safety net
  • src/agents/subagent-announce-dispatch.ts — extend SubagentDeliveryPath union from "steered" | "direct" | "none" to include "durable_queue"
  • src/plugin-sdk/agent-harness-task-runtime.ts — extend isDurableAgentHarnessCompletionDelivery to treat durable_queue path as durable (so monitor retry does not re-fire)
  • src/agents/subagent-announce-delivery.test.ts — 5 new unit tests covering the durable-queue branch + counterfactual probes
  • src/agents/embedded-agent-runner/sessions-yield.orchestration.test.ts — 1 new integration test reproducing the original bug + verifying the fix end-to-end
  • src/plugin-sdk/agent-harness-task-runtime.test.ts — 1 new test for isDurableAgentHarnessCompletionDelivery(durable_queue)

Architectural invariant

The fix is self-contained at runtime: no code path reads warden-status, workspace/memory, or .openclaw/ ledger paths to make a delivery decision. Status files remain observability-only, not load-bearing. Verified via grep gate in CI patch.

Tests

Local on commit 49e43d03 (6.2 lane), re-run on rebased c9dc427a:

  • subagent-announce-delivery.test.ts: 102/102 pass (97 pre-existing + 5 new)
  • sessions-yield.orchestration.test.ts: 5/5 pass (4 pre-existing + 1 new)
  • agent-harness-task-runtime.test.ts: 7/7 pass (6 pre-existing + 1 new)
  • Typecheck: 0 errors in all 6 touched files. (Pre-existing TS4058 on src/secrets/config-io.ts:12 is unchanged from main.)

Independent review

A second-pass review on the 6.2 lane found all 11 checklist items + all 6 counterfactual probes green. Notable items probed:

  1. Empty directIdempotencyKey collision risk: not exploitable in practice — enqueueSystemEvent dedupes on the full (text, contextKey, deliveryContext) triple, not contextKey alone, so worst case is degraded observability not lost delivery.
  2. Crash window between enqueueSystemEvent returning success and parent's next drainSystemEvents call: thin silent-loss window exists at parent process death; recommended as follow-up enhancement (TTL/staleness sweep over peekSystemEventEntries) but not a regression of current behavior since the give-up branch already lost deliveries during this window.
  3. Existing 5.26 give-up shape (no reason field) exactly preserved in the catch fallback — no behavior delta for callers that match on error text only.

Compatibility

  • Backward-compatible: the durable-queue path is a NEW positive-delivery shape ({delivered:true, path:"durable_queue"}), not a change to existing failure shapes.
  • Existing retry/give-up bookkeeping continues to fire if enqueueSystemEvent itself throws (try/catch safety net).
  • Migration: no schema/state migration. The system-event inbox is already populated and drained by every running gateway.

Related

Reported as a recurring class of zombie task-runs in self-hosted deployments. Operator-side watchdog cron b03ce892-ed27-4b1b-a3c5-8abde6c0660a exists as a downstream mitigation; can be retired once this fix ships in the next release.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P1 High-priority user-facing bug, regression, or broken workflow. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. size: M status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant