Skip to content

fix(announce): durable in-process queue fallback for direct-pending handoffs#90144

Closed
bradfreels wants to merge 1 commit into
openclaw:mainfrom
bradfreels:morpheus/announce-durable-queue-fallback-v2
Closed

fix(announce): durable in-process queue fallback for direct-pending handoffs#90144
bradfreels wants to merge 1 commit into
openclaw:mainfrom
bradfreels:morpheus/announce-durable-queue-fallback-v2

Conversation

@bradfreels

Copy link
Copy Markdown

Summary

When deliverSubagentAnnouncement direct-dispatch returns a non-terminal status (accepted/started/in_flight) and the requester parent is not actively streaming (yielded, queued, compaction-prep, or tool-result truncated), today's behaviour returns

{ delivered: false, reason: "completion_handoff_pending", error: "completion agent handoff is still pending" }

— which surfaces as the dominant signature in openclaw tasks audit --json delivery_failed findings: the requester parent never receives a child-completion event that already arrived in the gateway. From inside the requester, the symptom is a subagent that appears active (waiting on N children) with pendingDescendants > 0 indefinitely after the envelope's endedAt already fired, because the completion event can't deliver back.

This PR routes those cases into the requester's existing in-process durable system-event inbox instead, keyed by the announce idempotency key. The inbox is the same primitive used today by jobs, hooks, restart sentinels, ACP child spawns, monitor events, and the group/channel completion path in task-registry. It is idempotent on (text, contextKey, deliveryContext), bounded at 20 events per session, and drained on the parent's next turn-start.

What changed

  • SubagentDeliveryPath union in subagent-announce-dispatch.ts extended with a new "durable_queue" literal (with doc comment).
  • subagent-announce-delivery.ts: the directAnnounceStillPending give-up branch (when expectsCompletionMessage && expectedMediaUrls.length === 0 && !requiresMessageToolDelivery) now calls enqueueSystemEvent with contextKey: "subagent-announce:${params.directIdempotencyKey}" and deliveryContext: requesterSessionOrigin ?? completionExternalFallbackOrigin ?? directOrigin, returning { delivered: true, path: "durable_queue" }. Existing directIdempotencyKey is reused — no new param threading.
  • Safety net: if enqueueSystemEvent itself throws, the original give-up payload ({ delivered: false, path: "direct", reason: "completion_handoff_pending", error: ... }) is returned so today's retry/give-up bookkeeping remains the floor.
  • isDurableAgentHarnessCompletionDelivery in plugin-sdk/agent-harness-task-runtime.ts now treats "durable_queue" as durable, preventing the Codex native-subagent monitor from looping retry against an inbox that already idempotently holds the event.

Tail-risk: SubagentDeliveryPath consumers

Three real consumers; all safe:

Consumer Site Disposition
subagent-registry-lifecycle.ts if (delivery.path === "none") gated on !delivered Safe — durable_queue returns delivered:true
subagent-announce.ts if (!delivery.delivered && delivery.path === "direct" && delivery.error) Safe — gated on !delivered
plugin-sdk/agent-harness-task-runtime.ts (isDurableAgentHarnessCompletionDelivery) Returned false for any non-steered/direct path Fixed — extended + new test

Tests

  • 5 unit tests in subagent-announce-delivery.test.ts — covers the new branch firing, idempotency on duplicate fires, no-fallback-on-active-stream (steered path wins), no-fallback-with-media (existing direct/media path preserved), and the result-envelope shape downstream consumers see.
  • 1 integration test in embedded-agent-runner/sessions-yield.orchestration.test.ts — reproduces the failure class end-to-end: yielded parent → runEmbeddedAgent returns stopReason:"end_turn"isEmbeddedAgentRunActive(parentSessionId) returns falsedeliverSubagentAnnouncement with stubbed callGateway returning accepted → asserts path:"durable_queue", inbox holds the trigger, drainSystemEvents returns it, post-drain inbox empty.
  • 1 unit test in agent-harness-task-runtime.test.ts — asserts durable_queue is treated as durable and that delivered:false still fails fast.

Targeted suite (5 files): 150/150 pass.

src/agents/subagent-announce-delivery.test.ts          97/97  (92 baseline + 5 new)
src/agents/subagent-announce-dispatch.test.ts          11/11  (no regression)
src/agents/subagent-registry-lifecycle.test.ts         30/30  (no regression)
src/agents/embedded-agent-runner/sessions-yield.orchestration.test.ts
                                                         5/5  (4 baseline + 1 new integration)
src/plugin-sdk/agent-harness-task-runtime.test.ts        7/7  (6 baseline + 1 new)

Full unit lane (pnpm test:unit): 10071 passed across 1055 files. The 5 failures observed are in src/commands/status.summary.runtime.test.ts (3) and src/agents/tools/subagents-tool.test.ts (2) — both of which fail identically on a clean unpatched main HEAD (verified via stash + re-run). Cause appears to be a test-isolation issue where those tests load the host's ~/.openclaw/openclaw.json rather than a sandboxed config; not related to this patch.

pnpm tsgo:core, pnpm tsgo:test:src, pnpm exec oxfmt --check (6 files), pnpm exec oxlint (6 files): all clean.

Not run

  • pnpm test:e2e, pnpm test:live, pnpm test:docker:* — require live provider credentials and/or docker daemon access. The not_streaming failure class is exercised deterministically by the integration repro; patch surface is one branch in one direct-dispatch function plus one downstream consumer. Happy to run any of these on request from a maintainer who has the live secrets.

Out of scope (deliberate)

  • The "recovery pointer" enrichment for delivery_failed audit findings (so the Brad-side watchdog could attempt automated retry) is held for a follow-up PR after this core fix lands. Filed mentally; not in this diff.

Origin / context

I wrote and operated a Brad-side audit watchdog (alert-only, 3-minute cadence) that has been catching every fresh delivery_failed incident on this side over the past day. The watchdog will stay alert-only and be retired after this fix is deployed and proven stable across multiple subagent flows. This PR is the upstream-side root-cause fix corresponding to the watchdog's mitigation surface.

…andoffs

When subagent direct-announce dispatch returns a non-terminal status
("accepted"/"started"/"in_flight") and the requester parent is not
streaming (yielded, queued, compaction-prep, or tool-result truncated),
the previous behaviour returned {delivered:false, error:"completion
agent handoff is still pending"}, surfacing as the dominant
`delivery_failed` audit-finding signature.

Route those cases into the requester's in-process durable system-event
inbox instead, keyed by the announce idempotency key. The inbox is
idempotent on (text, contextKey, deliveryContext), bounded at 20 events
per session, and drained on the parent's next turn-start (the same path
already used by jobs, hooks, restart sentinels, ACP child spawns,
monitor events, and the group/channel completion path in task-registry).

The `SubagentDeliveryPath` union gains a new `"durable_queue""` literal
so consumers can route on it. The harness-task-runtime durability check
(`isDurableAgentHarnessCompletionDelivery`) is extended to recognise the
new path, preventing Codex native-subagent monitor retry from churning
duplicates against an already-committed inbox enqueue.

If `enqueueSystemEvent` itself throws, the original give-up payload
(structured `reason: "completion_handoff_pending"` + error string) is
returned as a safety net so existing retry/give-up bookkeeping remains
the floor.

Tests:
- 5 unit tests in subagent-announce-delivery.test.ts covering the new
  branch, idempotency on duplicate fires, no-fallback-on-active-stream,
  no-fallback-with-media (preserve existing path), and result-envelope
  shape.
- 1 integration test in embedded-agent-runner/sessions-yield.orchestration
  reproducing the full failure class end-to-end: parent yield ->
  isEmbeddedAgentRunActive(false) -> announce with stubbed callGateway
  returning `accepted` -> assert path:"durable_queue" -> drain returns
  the trigger.
- 1 unit test in agent-harness-task-runtime.test.ts asserting
  durable_queue is treated as durable and that delivered:false still
  fails fast.

Targeted suite (5 files): 150/150 pass.
Full unit lane: 10071 pass; the 5 pre-existing failures on main
(`status.summary.runtime`, `subagents-tool` config-isolation issues)
reproduce with no patch applied and are not regressions from this
change.
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 4, 2026
@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Current main already fixes the direct-pending subagent completion handoff through the merged narrower fix in #94349, so this branch’s durable-queue implementation is now superseded and should not remain a landing candidate.

Root-cause cluster
Relationship: superseded
Canonical: #94349
Summary: The current PR and the merged canonical PR target the same pending direct-announce completion-loss branch; the merged PR is the current-main implementation.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the merged narrow pending-status fix on main and leave any stronger durable-queue delivery contract to a separate maintainer-sponsored design if needed.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: v2026.6.8 had the plain completion completion_handoff_pending branch, and current main now has a regression test showing accepted is credited as direct delivery. I did not run a fresh live repro in this read-only cleanup review.

Is this the best way to solve the issue?

No for this branch as submitted. The narrower merged current-main fix solves the reported handoff loss without adding the broader durable-queue/SDK surface that previously carried session-state and compatibility risk.

Security review:

Security review cleared: The PR diff only changes internal TypeScript agent delivery logic and tests; it does not touch dependencies, workflows, credentials, packaging, or external code execution.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • sallyom: Authored and merged the focused current-main fix that removed the pending completion handoff failure branch in the same delivery module. (role: merged superseding fix author; confidence: high; commits: 95c87e31e295, 0647bf44bdc2; files: src/agents/subagent-announce-delivery.ts, src/agents/subagent-announce-delivery.test.ts)
  • steipete: Git history shows foundational work unifying the subagent announce delivery pipeline, which is the owner boundary for this behavior. (role: feature-history owner; confidence: high; commits: 4258a3307f5a; files: src/agents/subagent-announce-delivery.ts, src/agents/subagent-announce-dispatch.ts)
  • bryanpearson: Introduced Codex/native subagent completion delivery through the generic plugin harness runtime, adjacent to the SDK durability helper this PR changes. (role: adjacent harness contributor; confidence: medium; commits: f9d35dc68180; files: src/plugin-sdk/agent-harness-task-runtime.ts, extensions/codex/src/app-server/native-subagent-monitor.ts)
  • maxpetrusenko: Authored a completion announce delivery target inheritance fix in the same delivery module, relevant to handoff target/origin behavior. (role: adjacent delivery contributor; confidence: medium; commits: 8262078ee50c; files: src/agents/subagent-announce-delivery.ts)

Codex review notes: model internal, reasoning high; reviewed against d4833e27c775; fix evidence: commit 95c87e31e295, main fix timestamp 2026-06-18T23:38:11Z.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 4, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jun 14, 2026
@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant