fix(diagnostics): clear embedded-run activity when recovery declares lane idle#88820
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 1, 2026, 1:07 AM ET / 05:07 UTC. Summary PR surface: Source +325, Tests +606. Total +931 across 6 files. Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Review detailsBest possible solution: Retry the Codex review after fixing the execution failure. Do we have a high-confidence way to reproduce the issue? Unclear. The review failed before ClawSweeper could establish a reproduction path. Is this the best way to solve the issue? Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction. AGENTS.md: unclear because the file could not be read completely. Codex review notes: model gpt-5.5, reasoning high; reviewed against 4e57546a8752. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +325, Tests +606. Total +931 across 6 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
37ee982 to
75a0bc7
Compare
00623aa to
c0a4394
Compare
steipete
left a comment
There was a problem hiding this comment.
Thanks for the focused fix. I think this needs one more cleanup in the same helper before it reliably satisfies the recovery invariant.
Findings:
- [P1] Clear run-scoped tool/model activity when the idle transition is authoritative.
clearDiagnosticEmbeddedRunsForSessioncurrently only emptiesactiveEmbeddedRuns, but the bypassed teardown path it is replacing would normally go throughmarkDiagnosticEmbeddedRunEnded, which also clearsactiveToolsandactiveModelCallsunless explicitly told not to. If the aborted embedded run had a stale tool/model marker, this patch changes the lane fromidle + hasActiveEmbeddedRuntoidle + orphaned tool/model activity;isIdleQueuedRecoverableSessionStallintentionally treats that state as recoverable when queueDepth remains positive, so the liveness sweep can still re-trigger recovery instead of converging. Please make the authoritative idle cleanup clear the same run activity that normal embedded-run teardown clears, and add a regression case with a stale tool or model marker plus queued work.src/logging/diagnostic-run-activity.ts:300,src/logging/diagnostic.ts:553
Best-fix verdict: too narrow as written. The right layer is the recovery coordinator’s guarded idle branch, but the activity-store helper should reconcile the whole terminal embedded-run activity state, not only the owner set.
Alternatives considered: moving this into WhatsApp/channel code is worse because the invariant is shared diagnostics/session recovery; clearing only the owner set is insufficient because existing orphaned-activity recovery explicitly requeues stale tool/model markers; changing the idle-queued classifier would be broader and would weaken the existing orphan cleanup from #87028.
Code read: src/logging/diagnostic-session-recovery-coordinator.ts, src/logging/diagnostic-run-activity.ts, src/logging/diagnostic.ts, src/logging/diagnostic-session-attention.ts, src/logging/diagnostic-stuck-session-recovery.runtime.ts, src/agents/embedded-agent-runner/runs.ts, src/auto-reply/reply/reply-run-registry.ts, linked issue #88660.
Remaining uncertainty: I did not run the PR tests locally because I reviewed the fetched PR head without switching branches. CI currently has one unrelated-looking check-dependencies failure (ui/package.json: three unused); all diagnostics/code checks I inspected are passing.
76a6524 to
f337f84
Compare
f337f84 to
a326686
Compare
steipete
left a comment
There was a problem hiding this comment.
Approved after the cleanup follow-up.
What changed since my requested-changes review:
- The authoritative recovery-idle cleanup now clears embedded-run owners plus stale tool/model child activity, matching terminal embedded-run teardown semantics.
- Added regressions for stale tool and model markers, plus the existing newer-run generation guard case.
- Rebased onto current
origin/mainand added a separate CI repair for current cron schema assertions plus regenerated Codex prompt snapshots, because those checks were already failing on the updated base.
Local proof run on head a326686afaa61fcef7d941ee2d44cb86f6adee88:
pnpm exec oxfmt --check src/logging/diagnostic-run-activity.ts src/logging/diagnostic-session-recovery-coordinator.ts src/logging/diagnostic.test.ts src/agents/tools/cron-tool.schema.test.tspnpm test src/logging/diagnostic.test.ts -- --reporter=verbose-> 58 passedpnpm test src/agents/tools/cron-tool.schema.test.ts -- --reporter=verbose-> 15 passedpnpm prompt:snapshots:check-> current, 7 files.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --prompt ...-> clean, no accepted/actionable findings
Known proof gap: no live WhatsApp gateway recovery cycle was driven; this is covered at the deterministic diagnostics coordinator/activity-store layer.
a326686 to
9b3f6ad
Compare
|
Land-ready proof for the refreshed branch at What changed after review:
Local proof run on the final stack:
Known proof gap: no live WhatsApp gateway recovery cycle was run; this is covered with focused diagnostic recovery regression tests in |
54a224d to
2c4adf1
Compare
|
Land-ready proof for current head 2c4adf1. What changed since the earlier review:
Local proof on the final rebased branch:
Autoreview result: clean, no accepted/actionable findings reported. Known gap: no live WhatsApp gateway recovery cycle was run; proof is focused unit/static coverage for the diagnostic reconciliation path touched here. |
…lane idle Stuck-session recovery transitions a lane to idle via the recovery coordinator, but only mutated the session-state store. When an aborted embedded run was removed without markDiagnosticEmbeddedRunEnded, the activity store kept hasActiveEmbeddedRun set, so the liveness sweep reported idle/embedded_run and isIdleQueuedRecoverableSessionStall re-triggered recovery indefinitely. Reconcile the activity store from the authoritative idle declaration by clearing the session's embedded-run owners. The existing generation guard already excludes any newer run that re-armed activity, so a live requeued run is preserved.
… cleanup clearDiagnosticEmbeddedRunActivityForSession (renamed from clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and model markers alongside the embedded-run owners, matching the default markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the lane as idle + orphaned tool/model activity, which isIdleQueuedRecoverableSessionStall still treats as recoverable while work is queued, so the liveness sweep kept re-triggering recovery instead of converging. Adds regression cases with stale tool and model markers plus queued work.
2c4adf1 to
fbb17fd
Compare
|
Maintainer update at What changed after review:
Proof run locally on the PR branch:
Note: after the clean autoreview pass, |
…lane idle (openclaw#88820) * fix(diagnostics): clear embedded-run activity when recovery declares lane idle Stuck-session recovery transitions a lane to idle via the recovery coordinator, but only mutated the session-state store. When an aborted embedded run was removed without markDiagnosticEmbeddedRunEnded, the activity store kept hasActiveEmbeddedRun set, so the liveness sweep reported idle/embedded_run and isIdleQueuedRecoverableSessionStall re-triggered recovery indefinitely. Reconcile the activity store from the authoritative idle declaration by clearing the session's embedded-run owners. The existing generation guard already excludes any newer run that re-armed activity, so a live requeued run is preserved. * fix(diagnostics): reconcile tool/model activity on authoritative idle cleanup clearDiagnosticEmbeddedRunActivityForSession (renamed from clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and model markers alongside the embedded-run owners, matching the default markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the lane as idle + orphaned tool/model activity, which isIdleQueuedRecoverableSessionStall still treats as recoverable while work is queued, so the liveness sweep kept re-triggering recovery instead of converging. Adds regression cases with stale tool and model markers plus queued work. * test(phone-control): align service mocks with keyed store API * fix(diagnostics): preserve rearmed recovery activity * fix(diagnostics): clear recovered owner markers * fix(diagnostics): clear recovered embedded work keys * fix(diagnostics): ignore stale same-key recovery owners * fix(diagnostics): preserve same-session recovery rearm * fix(diagnostics): ignore stale queued activity starts * fix(diagnostics): record recovery cutoffs for empty activity * fix(diagnostics): preserve fresh recovery markers * fix(diagnostics): prune stale activity before fresh recovery block --------- Co-authored-by: Peter Steinberger <[email protected]>
…lane idle (openclaw#88820) * fix(diagnostics): clear embedded-run activity when recovery declares lane idle Stuck-session recovery transitions a lane to idle via the recovery coordinator, but only mutated the session-state store. When an aborted embedded run was removed without markDiagnosticEmbeddedRunEnded, the activity store kept hasActiveEmbeddedRun set, so the liveness sweep reported idle/embedded_run and isIdleQueuedRecoverableSessionStall re-triggered recovery indefinitely. Reconcile the activity store from the authoritative idle declaration by clearing the session's embedded-run owners. The existing generation guard already excludes any newer run that re-armed activity, so a live requeued run is preserved. * fix(diagnostics): reconcile tool/model activity on authoritative idle cleanup clearDiagnosticEmbeddedRunActivityForSession (renamed from clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and model markers alongside the embedded-run owners, matching the default markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the lane as idle + orphaned tool/model activity, which isIdleQueuedRecoverableSessionStall still treats as recoverable while work is queued, so the liveness sweep kept re-triggering recovery instead of converging. Adds regression cases with stale tool and model markers plus queued work. * test(phone-control): align service mocks with keyed store API * fix(diagnostics): preserve rearmed recovery activity * fix(diagnostics): clear recovered owner markers * fix(diagnostics): clear recovered embedded work keys * fix(diagnostics): ignore stale same-key recovery owners * fix(diagnostics): preserve same-session recovery rearm * fix(diagnostics): ignore stale queued activity starts * fix(diagnostics): record recovery cutoffs for empty activity * fix(diagnostics): preserve fresh recovery markers * fix(diagnostics): prune stale activity before fresh recovery block --------- Co-authored-by: Peter Steinberger <[email protected]>
…lane idle (openclaw#88820) * fix(diagnostics): clear embedded-run activity when recovery declares lane idle Stuck-session recovery transitions a lane to idle via the recovery coordinator, but only mutated the session-state store. When an aborted embedded run was removed without markDiagnosticEmbeddedRunEnded, the activity store kept hasActiveEmbeddedRun set, so the liveness sweep reported idle/embedded_run and isIdleQueuedRecoverableSessionStall re-triggered recovery indefinitely. Reconcile the activity store from the authoritative idle declaration by clearing the session's embedded-run owners. The existing generation guard already excludes any newer run that re-armed activity, so a live requeued run is preserved. * fix(diagnostics): reconcile tool/model activity on authoritative idle cleanup clearDiagnosticEmbeddedRunActivityForSession (renamed from clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and model markers alongside the embedded-run owners, matching the default markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the lane as idle + orphaned tool/model activity, which isIdleQueuedRecoverableSessionStall still treats as recoverable while work is queued, so the liveness sweep kept re-triggering recovery instead of converging. Adds regression cases with stale tool and model markers plus queued work. * test(phone-control): align service mocks with keyed store API * fix(diagnostics): preserve rearmed recovery activity * fix(diagnostics): clear recovered owner markers * fix(diagnostics): clear recovered embedded work keys * fix(diagnostics): ignore stale same-key recovery owners * fix(diagnostics): preserve same-session recovery rearm * fix(diagnostics): ignore stale queued activity starts * fix(diagnostics): record recovery cutoffs for empty activity * fix(diagnostics): preserve fresh recovery markers * fix(diagnostics): prune stale activity before fresh recovery block --------- Co-authored-by: Peter Steinberger <[email protected]>
Summary
idlebut diagnostics keep reportingidle/embedded_run,q=1and re-emitsession.stalledforactive_work_without_progress. The lane looks healthy at a high level while the direct session lane is not actually clean, so queued work stalls and recovery re-fires indefinitely.idle/processing, queue depth) and the run-activity store (activeEmbeddedRunsowners plusactiveTools/activeModelCalls, surfaced asactiveWorkKind/hasActiveEmbeddedRun).applyRecoveryOutcomeToDiagnosticStateinsrc/logging/diagnostic-session-recovery-coordinator.tsdeclares the laneidleand reconciles only the session-state store. The run-activity store is cleared solely bymarkDiagnosticEmbeddedRunEnded(which clears the owner AND its tool/model markers by default), and the embedded-run teardown can bypass it (handle replaced/mismatched, or force-clear finding no live handle). When that happens the coordinator flips the lane toidlebut the run activity survives, so the liveness sweep formatsidle+ activity together andisIdleQueuedRecoverableSessionStallkeeps re-triggering recovery without ever converging. The decisive evidence in the report islast=embedded_run:started(never:ended) on an idle lane.clearDiagnosticEmbeddedRunActivityForSession({ sessionId, sessionKey })clears the session'sactiveEmbeddedRunsowners AND theiractiveTools/activeModelCallsmarkers and stamps terminal progress, matching the defaultmarkDiagnosticEmbeddedRunEndedteardown; it is a no-op when there is nothing to clear.session.stateevent.idle + hasActiveEmbeddedRuntoidle + orphaned tool/model activity, whichisIdleQueuedRecoverableSessionStall'shasOrphanedActivitybranch (added by fix(diagnostics): recover orphaned session activity and yield event loop during lock contention #87028) still treats as recoverable whilequeueDepth > 0— so the sweep would keep re-triggering recovery.processing), so only leaked/terminal activity is dropped — a legitimately requeued run is preserved.src/logging/diagnostic-run-activity.ts— addclearDiagnosticEmbeddedRunActivityForSession, which reconciles a session's terminal embedded-run activity (owners + tool/model markers).src/logging/diagnostic-session-recovery-coordinator.ts— call it from the authoritative idle branch ofapplyRecoveryOutcomeToDiagnosticState.diagnostic.test.ts(stuck session recovery activity reconciliationdescribe): embedded-owner reconcile, stale-tool-marker convergence, stale-model-marker convergence, and newer-run-preserved guard cases.expectedStateidle re-recovery path, and thepreserveQueuedIdleWorkqueue reconciliation are unchanged; the clear runs only after those already decided to flip the lane idle.isIdleQueuedRecoverableSessionStall) and the embedded-run teardown paths (clearActiveEmbeddedRun/forceClearEmbeddedAgentRun) are unchanged.docs/reference/config).extensions/*api/runtime-api, registry/loader).Reproduction
markDiagnosticEmbeddedRunEnded(handle replaced/mismatched, or force-clear finds no live handle).abortedoutcome and declares the laneidle.idle+ activity andisIdleQueuedRecoverableSessionStallre-fires recovery forever;last=embedded_run:startednever advances to:ended.activeWorkKind, the lane reads cleanly idle, and recovery converges.Real behavior proof
Behavior addressed (#88660): after
abort_embedded_runrecovery declares a lane idle, the lane is no longer classifiable asidle/embedded_run(oridle + orphaned tool/model activity); the whole terminal embedded-run activity is reconciled atomically with the idle transition so the liveness sweep stops re-triggering recovery.Real environment tested (Linux, Node 22 — Vitest against the production recovery coordinator, run-activity store, and heartbeat liveness loop):
requestStuckSessionRecovery/applyRecoveryOutcomeToDiagnosticStatedriving the realdiagnostic-session-stateanddiagnostic-run-activitystores, plus an end-to-endstartDiagnosticHeartbeatsweep exercising the realisIdleQueuedRecoverableSessionStalldispatch.Exact steps or command run after this patch:
pnpm test src/logging/diagnostic.test.ts; full related suitespnpm test src/logging/diagnostic-stuck-session-recovery.runtime.test.ts src/logging/diagnostic-stuck-session-recovery.integration.test.ts src/logging/diagnostic-session-attention.test.ts src/agents/embedded-agent-runner/runs.test.ts;pnpm tsgo;pnpm exec oxfmt --checkandnode scripts/run-oxlint.mjson the changed files.Evidence after fix (Vitest output for the touched test file):
The reconciliation describe covers four cases:
clears the embedded-run activity flag when recovery declares the lane idle,clears a stale tool marker left by the aborted run so a queued idle lane converges,clears a stale model marker left by the aborted run so a queued idle lane converges, andpreserves an active flag for a newer run that re-armed work mid-recovery.Observed result after fix: after the coordinator declares the lane
idle,getDiagnosticSessionActivitySnapshotreturns noactiveWorkKindand nohasActiveEmbeddedRun— including when the aborted run had left a stale tool or model marker. An end-to-end heartbeat sweep over that reconciled state does not re-dispatch recovery; with an owner-only clear the same sweep still re-dispatches (orphanedtool_call/model_callkeeps the lane recoverable), confirming the broader clear is what makes the lane converge. When a newer run re-arms activity mid-recovery, the generation guard bails: the lane staysprocessingand keeps its activity (live run preserved).What was not tested: the live WhatsApp-gateway abort/recovery cycle end-to-end on the reporter's deployment. The coordinator idle declaration, the activity reconciliation, and the liveness re-dispatch are covered deterministically against the production functions; a full gateway round-trip was not driven.
Repro confirmation: each reconcile case fails on an owner-only-clear tree (the activity snapshot still reports
embedded_run/tool_call/model_call) and passes with the fix, so the tests cover the production change.Risk / Mitigation
processing, so the guard bails before the clear — only leaked/terminal activity is dropped.markDiagnosticEmbeddedRunEndedteardown clears by default, and is gated behind the same guarded idle branch; it is a no-op when there is nothing to clear.Change Type (select all)
Scope (select all touched areas)
Linked Issue/PR
Fixes #88660