fix: forward pending timeout snapshot in waitForAgentJob fallback timer by Pick-cat · Pull Request #89367 · openclaw/openclaw

Pick-cat · 2026-06-02T07:10:08Z

What Problem This Solves

Fixes an issue where sub-agent runs killed by their hard timeout would record status: timeout in the registry but the waiting parent session never received the notification. The parent's agent.wait outer fallback timer only checked getPendingAgentRunError and skipped getPendingAgentRunTimeout, so it resolved null and agent.wait downgraded the outcome to a generic soft/gateway_draining timeout, losing the hard-phase context.

Why This Change Was Made

The waitForAgentJob outer fallback timer now also forwards a pending hard-timeout snapshot — but it does not rederive the hard/soft decision locally. It reuses the existing terminalOutcomeFromSnapshot() helper (which runs the shared buildAgentRunTerminalOutcome classifier) and forwards only when reason === "hard_timeout".

This honors the single canonical contract in src/agents/agent-run-terminal-outcome.ts (per the root rule: "Agent run terminal state: normalize/merge via agent-run-terminal-outcome.ts; do not rederive timeout/cancel precedence in projections"). In particular the canonical classifier excludes:

restart-cancelled snapshots (stopReason === "restart") even when they reached provider-started with a timeout status — these are cancelled, not hard timeouts, and must stay correctable via retry grace;
soft queue / gateway_draining phases.

An earlier revision hand-rolled the gate as isHardAgentRunTimeoutPhase(phase) || (status === "timeout" && providerStarted === true), which dropped the !restartCancelled guard and would forward a restart-cancelled snapshot as a false hard timeout. Reusing the shared classifier fixes that and shrinks the diff: agent-run-terminal-outcome.ts is now unchanged from main, and the production change is confined to src/gateway/server-methods/agent-job.ts (+17/-1).

User Impact

Parent sessions correctly receive hard-timeout completion events for timed-out sub-agents (instead of a generic gateway_draining timeout), while restart-cancelled and soft-phase runs are left to retry grace instead of being mislabeled as hard timeouts.

Evidence

All proofs re-run on head 9c074e57180 (Node v22.22.0, Linux).

1. E2E — real `waitForAgentJob` + real `emitAgentEvent`, incl. restart-cancelled regression

$ node scripts/run-vitest.mjs src/gateway/server-methods/agent-job.subagent-timeout-notify.e2e.test.ts --run
 Test Files  1 passed (1) / Tests  4 passed (4)

The 4 cases drive the real in-process agent event bus and the real waitForAgentJob: the three hard-timeout phases (preflight / provider / post_turn) forward the precise snapshot, and a new case asserts a restart-cancelled provider-started timeout is NOT forwarded as a hard timeout.

2. Negative control — the regression test guards the exact bug

Reverting only the production change in agent-job.ts to the earlier hand-rolled gate (tests kept) fails exactly the new case, proving it guards the !restartCancelled contract:

$ node scripts/run-vitest.mjs src/gateway/server-methods/agent-job.subagent-timeout-notify.e2e.test.ts --run
 × does not forward a restart-cancelled provider-started timeout as a hard timeout
   AssertionError: expected { …(9) } to be null
 Test Files  1 failed (1) / Tests  1 failed | 3 passed (4)

3. Contract + server-methods lanes

$ node scripts/run-vitest.mjs src/agents/agent-run-terminal-outcome.test.ts --run
 Test Files  1 passed (1) / Tests  14 passed (14)

$ node scripts/run-vitest.mjs src/gateway/server-methods/server-methods.test.ts --run
 Test Files  1 passed (1) / Tests  161 passed (161)

$ node scripts/run-oxlint.mjs src/gateway/server-methods/agent-job.ts src/gateway/server-methods/agent-job.subagent-timeout-notify.e2e.test.ts
 exit 0 (clean)

Not tested here

A live external Gateway with a real model/tool turn was not captured (no live model in this environment). This change is confined to the wait-layer snapshot-forwarding decision, which now defers entirely to the shared buildAgentRunTerminalOutcome classifier; the canonical classifier and its restart-cancelled / soft-phase behavior are covered by the contract tests above plus the real-event-bus e2e and its negative control.

Related: #89095 (this PR covers the wait-layer fallback slice; the broader no-deliverable-payload parent-wake path is tracked separately).

clawsweeper · 2026-06-02T07:11:49Z

Codex review: stale review; fresh review needed.

Summary
The latest durable ClawSweeper review was for head b98d86483180106942141d1119f811e2ddd954e8, but the PR head is now 1e417faaa5406a980c6a002711121672d7bdad62. Its old verdict and PR readiness labels are no longer current.

Next step
Run or wait for a fresh ClawSweeper review on the current PR head.

Review history (2 earlier review cycles)

reviewed 2026-07-03T04:40:53.522Z sha 90313a1 :: needs maintainer review before merge. :: none
reviewed 2026-07-03T04:56:51.873Z sha 90313a1 :: needs maintainer review before merge. :: none

Pick-cat · 2026-06-02T08:17:28Z

Addressed P1 finding: the outer timer now only forwards pending timeout snapshots that carry an endedAt field (hard timeouts). Transient aborts without endedAt remain correctable via the retry grace. Added regression test 'does not return a transient abort snapshot before the timeout grace expires'.

@clawsweeper re-review

clawsweeper · 2026-06-02T08:17:31Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26807480779
Updated: 2026-06-02T08:23:40.005Z

Pick-cat · 2026-06-02T08:57:35Z

Changed gate from endedAt to timeoutPhase (provider-level hard-timeout attribution). Aborts with endedAt but without timeoutPhase remain correctable. Added regression: abort with endedAt recovers via later start+success before grace expires.

@clawsweeper re-review

clawsweeper · 2026-06-02T08:57:38Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26809483306
Updated: 2026-06-02T09:04:38.949Z

Pick-cat · 2026-06-02T09:18:47Z

Replaced ad-hoc timeoutPhase string check with shared isHardAgentRunTimeoutPhase() predicate (only preflight / provider / post_turn are hard). Soft phases like queue and gateway_draining remain correctable. Updated regression test to use timeoutPhase: "queue".

@clawsweeper re-review

clawsweeper · 2026-06-02T09:18:50Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26810530052
Updated: 2026-06-02T09:25:54.395Z

clawsweeper · 2026-06-26T12:04:59Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Pick-cat · 2026-07-03T04:20:18Z

Rebased on latest main (conflicts resolved, 163/163 tests pass).

Real behavior proof added — standalone .mts script, real timers (no fake timers), real waitForAgentJob + emitAgentEvent event bus:

Evidence after fix:

[case 1] hard timeout (provider phase) — snapshot forwarded
  ok: result is not null (snapshot forwarded)
  ok: status is timeout
  ok: timeoutPhase is provider
  ok: resolved within timeout window
  info: elapsed=298 ms, status=timeout, phase=provider
[case 2] soft timeout (queue phase) — stays correctable
  ok: result is null (queue is soft/correctable)
[case 3] normal completion — snapshot passes through
  ok: status is ok
[case 4] hard timeout (preflight phase) — snapshot forwarded
  ok: result is not null (preflight hard timeout forwarded)
  ok: status is timeout
  ok: timeoutPhase is preflight

ALL PROOF ASSERTIONS: 10 passed, 0 failed

Negative control (main code, same script):

[case 1] hard timeout (provider phase) — snapshot forwarded
  FAIL: result is not null (snapshot forwarded)
  FAIL: status is timeout
  FAIL: timeoutPhase is provider
  info: elapsed=299 ms, status=undefined, phase=undefined
[case 4] hard timeout (preflight phase) — snapshot forwarded
  FAIL: result is not null (preflight hard timeout forwarded)
  FAIL: status is timeout
  FAIL: timeoutPhase is preflight
  info: status=undefined, phase=undefined

ALL PROOF ASSERTIONS: 4 passed, 6 failed
exit=1

Without the fix, hard-timeout snapshots are silently dropped (result=null). With the fix, they are correctly forwarded.

@clawsweeper re-review

clawsweeper · 2026-07-03T04:20:21Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

steipete · 2026-07-07T09:59:39Z

Land-ready maintainer pass complete for the wait-layer slice of #89095.

Work done: kept the canonical hard-timeout fallback, removed duplicate/overstated parent-notification E2E coverage, replaced it with one focused wait-race suite, and added the maintainer changelog entry thanking @Pick-cat.
Review fix: exact-head proof exposed current-main restart cancellation as an immediate error rather than null; fresh autoreview also found that ignoreCachedSnapshot could consume stale shared pending state. The final head preserves only pending state observed by a fresh waiter while normal waits still recover shared hard-timeout snapshots.
Focused proof: sanitized AWS Crabbox run run_0ff829c7b3c0 on patch-identical pre-rebase head b98d86483180106942141d1119f811e2ddd954e8; /usr/local/bin/pnpm test src/gateway/server-methods/agent-job.timeout-fallback.test.ts src/agents/agent-run-terminal-outcome.test.ts passed 30 assertions across the gateway-methods, gateway-client, and unit-fast shards.
Broad proof: exact-head CI run https://github.com/openclaw/openclaw/actions/runs/28857841517 completed successfully on final prepared head 1e417faaa5406a980c6a002711121672d7bdad62; build, lint, types, contracts, compact Node shards, and QA smoke passed.
Review: final post-rebase branch autoreview reported no accepted/actionable findings.
Before/after: the outer wait timer previously returned null while a hard timeout was pending; it now returns the canonical phase/timestamps, while queue/draining waits remain correctable and fresh waits cannot inherit a prior attempt's pending state.
Known gap: this does not solve no-deliverable-payload announce drops. Keep Sub-agent run timeout does not notify parent session (completion event silently dropped) #89095 open; fix(subagents): wake yielded parent on no-deliverable-payload completions (closes part of #89095) #96189 is the complementary delivery-layer work. No live model/gateway parent-notification proof was run here.

…b fallback timer (fixes openclaw#89095)

…ation (openclaw#89095)

…ntract Also treat provider-started timeout snapshots as hard timeouts in the waitForAgentJob fallback, mirroring buildAgentRunTerminalOutcome. Co-Authored-By: Claude Opus 4.7 <[email protected]>

…allback

steipete · 2026-07-07T10:11:17Z

Merged via squash.

Prepared head SHA: 1e417faaa5406a980c6a002711121672d7bdad62
Landed commit: 61dce7cb6e888cb5a0e0624e8d2a56b0fa42a8e9

openclaw-barnacle Bot added gateway Gateway runtime size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 2, 2026

Pick-cat force-pushed the fix/issue-89095-subagent-timeout-notify branch from 1a08775 to 7724129 Compare June 2, 2026 07:52

openclaw-barnacle Bot removed the size: XS label Jun 2, 2026

openclaw-barnacle Bot added the size: S label Jun 2, 2026

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 2, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 2, 2026

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 2, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 2, 2026

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 2, 2026

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 29, 2026

clawsweeper Bot mentioned this pull request Jul 1, 2026

fix(subagents): wake yielded parent on no-deliverable-payload completions (closes part of #89095) #96189

Open

Pick-cat force-pushed the fix/issue-89095-subagent-timeout-notify branch from ff7fe92 to 90313a1 Compare July 3, 2026 04:19

steipete self-assigned this Jul 7, 2026

steipete force-pushed the fix/issue-89095-subagent-timeout-notify branch from 90313a1 to b806399 Compare July 7, 2026 09:43

openclaw-barnacle Bot added size: S and removed size: M labels Jul 7, 2026

steipete force-pushed the fix/issue-89095-subagent-timeout-notify branch from b806399 to 91e34b7 Compare July 7, 2026 09:44

Pick-cat and others added 10 commits July 7, 2026 06:00

fix(agents): export isHardAgentRunTimeoutPhase for wait-layer reuse

f98b4f2

fix(gateway): forward pending hard-timeout snapshot in waitForAgentJo…

f877445

…b fallback timer (fixes openclaw#89095)

test(gateway): add e2e proof for subagent hard-timeout parent notific…

3efa6df

…ation (openclaw#89095)

fix(gateway): match fallback hard-timeout gate to terminal-outcome co…

997e433

…ntract Also treat provider-started timeout snapshots as hard timeouts in the waitForAgentJob fallback, mirroring buildAgentRunTerminalOutcome. Co-Authored-By: Claude Opus 4.7 <[email protected]>

fix(gateway): use shared hard-timeout classifier in waitForAgentJob f…

c3be11e

…allback

chore: amend author email

37677ab

chore: rebase on main, resolve conflicts

4c06770

test(gateway): focus timeout fallback coverage

0a2c911

docs(changelog): note agent wait timeout fix

3e3736c

fix(gateway): isolate fresh wait terminal state

1e417fa

github-actions Bot mentioned this pull request Jul 7, 2026

📡 Upstream Digest — 2026-07-07 10:48 UTC curtismercier/openclaw-mods#1229

Open

Uh oh!

Uh oh!

Conversation

Pick-cat commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

1. E2E — real waitForAgentJob + real emitAgentEvent, incl. restart-cancelled regression

2. Negative control — the regression test guards the exact bug

3. Contract + server-methods lanes

Not tested here

Uh oh!

clawsweeper Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pick-cat commented Jun 2, 2026

Uh oh!

clawsweeper Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pick-cat commented Jun 2, 2026

Uh oh!

clawsweeper Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pick-cat commented Jun 2, 2026

Uh oh!

clawsweeper Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pick-cat commented Jul 3, 2026

Uh oh!

clawsweeper Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pick-cat commented Jun 2, 2026 •

edited

Loading

1. E2E — real `waitForAgentJob` + real `emitAgentEvent`, incl. restart-cancelled regression

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

clawsweeper Bot commented Jun 2, 2026 •

edited

Loading

clawsweeper Bot commented Jun 26, 2026 •

edited

Loading

clawsweeper Bot commented Jul 3, 2026 •

edited

Loading

steipete commented Jul 7, 2026 •

edited

Loading