Skip to content

fix(agents): retire stale aborted subagent runs past the staleness window in orphan recovery (fixes #90766)#92346

Closed
zenglingbiao wants to merge 3 commits into
openclaw:mainfrom
zenglingbiao:fix/issue-90766-stale-orphan-recovery
Closed

fix(agents): retire stale aborted subagent runs past the staleness window in orphan recovery (fixes #90766)#92346
zenglingbiao wants to merge 3 commits into
openclaw:mainfrom
zenglingbiao:fix/issue-90766-stale-orphan-recovery

Conversation

@zenglingbiao

Copy link
Copy Markdown
Contributor

Summary

  • Problem: After a gateway restart with long downtime, stale subagent runs with abortedLastRun: true are treated as resumable by orphan recovery even when they are hours past the staleness window. This can resurrect contextless child work long after the original run should have been retired.
  • Root Cause: Two gaps — (1) recoverOrphanedSubagentSessions checks abortedLastRun and the recovery gate (re-wedge window) but never checks run age; (2) resolveSubagentRunOrphanReason used in restore reconciliation exempts abortedLastRun: true entries from the stale-unended pruning branch (sessionEntry.abortedLastRun !== true guard on line 193), so stale aborted runs survive registry init and proceed to orphan recovery.
  • Fix:
    • Add isStaleUnendedSubagentRun check in recoverOrphanedSubagentSessions after the abortedLastRun gate, before evaluateSubagentRecoveryGate — stale runs are skipped and reported as stale-unended-run.
    • Remove the abortedLastRun !== true exemption from resolveSubagentRunOrphanReason so restore reconciliation prunes stale runs regardless of abortedLastRun. Fresh aborted runs (under the 2-hour staleness window) are NOT stale and still proceed to orphan recovery as before.
  • What changed:
    • src/agents/subagent-orphan-recovery.ts — import isStaleUnendedSubagentRun, add age-out guard before recovery gate
    • src/agents/subagent-registry-helpers.ts — remove abortedLastRun exemption from stale-unended restore pruning
    • src/agents/subagent-registry.persistence.test.ts — replaces the "keeps stale aborted runs for restart recovery" test with a "retires stale aborted runs past the staleness window" test
  • What did NOT change (scope boundary):
    • evaluateSubagentRecoveryGate / subagent-recovery-state.ts — recovery gate unchanged; it handles re-wedge bounding, not staleness
    • Doc files (docs/tools/subagents.md) — out of scope; docs describe behavior at a higher level
    • Fresh restart recovery — runs under the 2-hour STALE_UNENDED_SUBAGENT_RUN_MS cutoff still proceed to orphan recovery unchanged

Reproduction

  1. Start a subagent run that writes normal recovery metadata
  2. Leave it unresolved across downtime longer than the staleness window (default 2 hours)
  3. Restart the gateway
  4. Before this PR: The stale aborted run is classified as recoverable and agent.wait is called to resume it
  5. After this PR: The stale aborted run is skipped during orphan recovery and pruned from the registry as stale-unended-run

Real behavior proof

Behavior or issue addressed (90766): Orphan recovery and restore reconciliation now age out stale aborted subagent runs past the STALE_UNENDED_SUBAGENT_RUN_MS (2-hour) staleness window, instead of resurrecting them as restart-recoverable.

Real environment tested: Linux, Node 22 — test runner against subagent orphan recovery, registry persistence, and registry queries

Exact steps or command run after this patch: node scripts/run-vitest.mjs src/agents/subagent-orphan-recovery.test.ts src/agents/subagent-registry.persistence.test.ts src/agents/subagent-registry-queries.test.ts

Evidence after fix:

[test] starting test/vitest/vitest.unit-fast.config.ts
 RUN  v4.1.7 /home/0668001395/openclawProject1
 Test Files  1 passed (1)
      Tests  16 passed (16)

[test] starting test/vitest/vitest.agents.config.ts
 RUN  v4.1.7 /home/0668001395/openclawProject1
 Test Files  2 passed (2)
      Tests  37 passed (37)

[test] passed 2 Vitest shards

Observed result after fix: The orphan recovery suite (16 tests including stale-skip coverage) and the persistence/queries suites (37 tests including the new "retires stale aborted restored runs past the staleness window" regression) all complete successfully. The before test on origin/main passes the old "keeps stale aborted runs for restart recovery" behavior (3-hour-old abortedLastRun run stays recoverable), confirming the gap existed; the after test confirms those stale aborted runs are now retired and pruned.

What was not tested: Live gateway restart with a real subagent run that spans multi-hour downtime and gate-level run-age cutoff. Systemd service stop/start cycle with a stale orphan.

Repro confirmation: The existing test on origin/main (subagent-registry.persistence.test.ts line ~727, "keeps stale unended restored runs with abortedLastRun for restart recovery") seeds a 3-hour-old restored run with abortedLastRun: true and expects callGateway to run agent.wait — confirming main treats the stale run as recoverable. After the patch, the replacement test seeds a 3-hour-old abortedLastRun run and confirms it is pruned from the registry (no callGateway call, no announce). All other tests unchanged.

Risk / Mitigation

  • Risk: A careless age guard could break legitimate fresh restart recovery for recent abortedLastRun runs.
    Mitigation: isStaleUnendedSubagentRun uses the existing STALE_UNENDED_SUBAGENT_RUN_MS (2-hour) cutoff with resolveStaleCutoffMs respecting explicit runTimeoutSeconds. Freshly restarted runs under the cutoff are NOT stale and proceed to orphan recovery unchanged.
  • Risk: Removing the abortedLastRun exemption from restore reconciliation could prematurely prune runs that should stay visible.
    Mitigation: The isStaleUnendedSubagentRun guard is unchanged — it only fires when the run is past the staleness window. Runs under the window are not stale and stay in the registry regardless of abortedLastRun.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • agents
  • subagent-registry
  • orphan-recovery

Regression Test Plan

  • node scripts/run-vitest.mjs src/agents/subagent-orphan-recovery.test.ts
  • node scripts/run-vitest.mjs src/agents/subagent-registry.persistence.test.ts
  • node scripts/run-vitest.mjs src/agents/subagent-registry-queries.test.ts

Review Findings Addressed

N/A (initial submission)

Linked Issue/PR

Fixes #90766

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 12, 2026
@clawsweeper

clawsweeper Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the central useful fix is covered by the open, clean, proof-positive canonical PR, while this branch is conflicting, mock-only, and still missing the stronger finalization/docs coverage.

Root-cause cluster
Relationship: superseded
Canonical: #90817
Summary: This PR is a weaker duplicate candidate for the same stale restart-aborted subagent recovery bug; the open clean sibling is the viable canonical landing path.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this branch and continue review or landing through #90817 as the canonical fix for #90766.

So I’m closing this here and keeping the remaining discussion on #90817 and #90766.

Review details

Best possible solution:

Close this branch and continue review or landing through #90817 as the canonical fix for #90766.

Do we have a high-confidence way to reproduce the issue?

Yes at source level. Current main still has the abortedLastRun restore exemption, no orphan-recovery stale age gate, and a persistence test that expects a three-hour aborted restored run to be recovered through agent.wait.

Is this the best way to solve the issue?

No, this is not the best landing path as submitted. The safer solution is the clean canonical sibling that awaits stale finalization, updates docs, preserves fresh recovery, and includes restart-path proof.

Security review:

Security review cleared: No security or supply-chain concern found; the diff changes internal agent recovery logic and a colocated test only.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • joeykrug: Authored the original SIGUSR1 orphaned subagent session recovery work that introduced the affected restart recovery path. (role: introduced behavior; confidence: high; commits: c780b6a6ab2a, 44304ba24af8, 0311ff05d77a; files: src/agents/subagent-orphan-recovery.ts, src/agents/subagent-registry.ts)
  • steipete: Merged the original orphan recovery PR and authored adjacent stale-run and bounded recovery work in the same subagent lifecycle surface. (role: merger and recent area contributor; confidence: high; commits: 680eff63fbf8, 757aee4cdd5d, 0f120c09ba28; files: src/agents/subagent-orphan-recovery.ts, src/agents/subagent-run-liveness.ts, src/agents/subagent-registry.persistence.test.ts)
  • jalehman: Recently migrated the session accessor boundary and introduced the current session reconciliation file that contains the restore-time orphan classification. (role: recent adjacent contributor; confidence: medium; commits: 6f2869c296ff, d216f7c876dd; files: src/agents/subagent-session-reconciliation.ts, src/agents/subagent-orphan-recovery.ts)
  • vincentkoc: Recently touched the stale-run cutoff module used by both this PR and the canonical replacement path. (role: recent adjacent contributor; confidence: medium; commits: 16142bebd83e; files: src/agents/subagent-run-liveness.ts)

Codex review notes: model internal, reasoning high; reviewed against a841c2788202.

@clawsweeper clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 12, 2026
@zenglingbiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@zenglingbiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 12, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 20, 2026
…ubagent runs (fixes openclaw#90766)

When a stale unended subagent run is detected, increment 'failed' instead of
'skipped' so scheduleOrphanRecovery can retry finalization. Previously the stale
branch only incremented 'skipped' and the fire-and-forget Promise.allSettled
finalization would silently drop errors.

ClawSweeper P2: 'Make stale finalization retry-visible'
@zenglingbiao

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper

clawsweeper Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

@zenglingbiao thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward.

Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive.

Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it.

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: orphan recovery can resurrect stale subagent runs after long downtime

1 participant