Skip to content

Codex native subagent completion delivery retries forever (no attempt cap, no deadline) on a permanently non-durable delivery #97593

Description

@yetval

Summary

When a Codex native subagent (spawn_agent) completes, the native-subagent monitor delivers the completion to the parent via runtime.deliverAgentHarnessTaskCompletion. If that delivery is deterministically non-durable (for example the parent's external-channel send permanently fails, or the parent reply is silent / NO_REPLY for a subagent completion), the monitor reschedules the delivery with no maximum attempt count and no overall deadline. It re-fires forever, and each retry re-invokes deliverAgentHarnessTaskCompletion, which in production wakes and re-runs the parent agent.

Environment

  • Commit: 19cac35 (origin/main at time of filing)
  • Surface: Codex harness, native multi-agent (spawn_agent) completion delivery, extensions/codex/src/app-server/native-subagent-monitor.ts

Steps to reproduce

  1. Run a Codex native multi-agent session where a parent spawns a subagent via spawn_agent.
  2. Arrange for the parent completion delivery to be permanently non-durable, for example: the parent's external channel is down so the send keeps failing, or the parent session is message_tool_only and the completion reply is silent (subagent completions are excluded from acceptsIntentionalSilentCompletion, so a silent parent yields delivered: false).
  3. Let the subagent finish so the monitor delivers the terminal completion to the parent.
  4. Observe the delivery is rescheduled indefinitely. Each retry re-invokes deliverAgentHarnessTaskCompletion, re-running the parent agent, with the delay clamped at the last schedule entry (300000 ms) so it re-fires every 5 minutes forever.

Expected

The retry loop should stop after a small bounded number of attempts and/or an overall deadline. On give-up the completion should be marked failed or abandoned instead of being rescheduled, so a permanently non-durable delivery cannot drive an unbounded series of parent re-runs.

Actual

scheduleCompletionDeliveryRetry clamps only the delay, never the count, and there is no overall deadline anywhere in the file. completionDeliveryAttempt is reset to 0 only on success, used solely as a clamped delay index, and incremented, but never compared to a maximum. A permanently non-durable delivery therefore reschedules forever, re-invoking deliverAgentHarnessTaskCompletion on every tick.

Root cause

extensions/codex/src/app-server/native-subagent-monitor.ts

deliverPendingCompletion reschedules on every non-durable delivery, and the finally block clears deliveringCompletionKey so the next tick is not deduped:

if (isDurableAgentHarnessCompletionDelivery(delivery)) {
  state.deliveredCompletionKeys.add(completionKey);
  childState.pendingCompletion = undefined;
  childState.pendingCompletionEventAt = undefined;
  childState.completionDeliveryAttempt = 0;
  ...
  return;
}
const error = delivery.error ?? "completion delivery did not produce a parent response";
this.markCompletionDeliveryPending(completion, error);
this.scheduleCompletionDeliveryRetry(childState);

scheduleCompletionDeliveryRetry clamps the delay index but never bounds the attempt count and sets no deadline:

const attempt = childState.completionDeliveryAttempt;
const delayMs =
  this.completionDeliveryRetryDelaysMs[
    Math.min(attempt, this.completionDeliveryRetryDelaysMs.length - 1)
  ];
childState.completionDeliveryAttempt += 1;
childState.completionDeliveryTimer = setTimeout(() => {
  childState.completionDeliveryTimer = undefined;
  const state = this.parentStates.get(childState.parentThreadId);
  if (!state) {
    return;
  }
  void this.deliverPendingCompletion(state, childState);
}, delayMs);

The default schedule is [5000, 15000, 30000, 60000, 120000, 300000], and Math.min(attempt, len - 1) pins the delay at the last entry, so for a permanently non-durable delivery the retry re-fires every 5 minutes indefinitely.

A non-durable delivery is the natural terminal state here: isDurableAgentHarnessCompletionDelivery returns false whenever delivered is false, and acceptsIntentionalSilentCompletion is false for subagent completions, so a silent parent reply produces delivered: false permanently.

Sibling surfaces

The equivalent completion-delivery loop on the OpenClaw sessions_spawn announce path (sessionKey agent:main:subagent:...) is bounded: #88523 added a retry cap, #90178 / #95080 added give-up / expiry plus blocked-parent-wake handling, and #19552 / #17737 were the original loop reports there. The Codex native-subagent monitor under extensions/codex is a separate, later implementation (introduced via #83445) and has neither a cap nor a deadline. This is distinct from #96518 (opencode-go watchdog drop), #92238 (codex exec-unavailable), and #92088 (update-handoff recovery). A search for scheduleCompletionDeliveryRetry, completionDeliveryAttempt, and "completion delivery cap" returns no existing issue or PR covering this location; the open PR #96662 touches this file but does not change the retry logic.

Real behavior proof

Behavior addressed: Codex native-subagent completion delivery retries with no attempt cap and no deadline, re-running the parent agent forever on a permanently non-durable delivery.
Real environment tested: the real CodexNativeSubagentMonitor from extensions/codex/src/app-server/native-subagent-monitor.ts driven end to end (register parent, deliver a terminal native subagent completion notification, advance the clock), with only the injected runtime delivery result stubbed at its legitimate seam to return a permanently non-durable result (delivered: false); the rescheduling under observation is production code, at commit 19cac35.
Exact steps or command run after this patch: construct the monitor with the real runtime seam, register a parent, push a terminal native subagent completion, then advance time across 200 retry ticks and count deliverAgentHarnessTaskCompletion invocations.
Evidence after fix:

# OBSERVED (origin/main 19cac35a06ed, permanently non-durable delivery injected at the runtime seam)
RED: completion delivery retried 201 times in 200 ticks with no cap/deadline (expected <= 12). Unbounded crash-loop.

# Mechanism observed
- completionDeliveryAttempt only indexes a clamped delay array; it is never compared to a maximum
- scheduleCompletionDeliveryRetry re-arms a setTimeout every tick; the finally block clears deliveringCompletionKey so the next tick is never deduped
- at the default schedule the clamped delay settles at 300000 ms, so delivery re-fires every 5 minutes indefinitely

# EXPECTED (bounded retry on identical inputs)
delivery stops after a small attempt cap and/or an overall deadline, then marks the completion failed or abandoned instead of rescheduling

Observed result after fix: the monitor re-invokes the parent-waking deliverAgentHarnessTaskCompletion 201 times over 200 ticks with no cap and no deadline; a permanently non-durable delivery reschedules forever and never gives up.
What was not tested: the live external-channel send-failure path against a real provider/channel was not exercised; the non-durable delivery was injected at the runtime seam rather than by knocking down a real channel. The suggested remediation (a bounded attempt cap and/or deadline mirroring the deadline-bounded announce-path retry, with a failed/abandoned terminal state on give-up) was not implemented or measured here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automation

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions