Skip to content

fix(diagnostics): clear embedded-run activity when recovery declares lane idle#88820

Merged
steipete merged 12 commits into
openclaw:mainfrom
openperf:fix/88660-recovery-idle-embedded-run
Jun 1, 2026
Merged

fix(diagnostics): clear embedded-run activity when recovery declares lane idle#88820
steipete merged 12 commits into
openclaw:mainfrom
openperf:fix/88660-recovery-idle-embedded-run

Conversation

@openperf

@openperf openperf commented May 31, 2026

Copy link
Copy Markdown
Member

Summary

  • Problem: Issue WhatsApp lane remains idle/embedded_run after abort_embedded_run recovery #88660 reports that after stuck-session recovery aborts a channel lane's embedded run, the lane settles to idle but diagnostics keep reporting idle/embedded_run,q=1 and re-emit session.stalled for active_work_without_progress. The lane looks healthy at a high level while the direct session lane is not actually clean, so queued work stalls and recovery re-fires indefinitely.
  • Root Cause: Diagnostics keep two independent per-session stores: the session-state store (idle/processing, queue depth) and the run-activity store (activeEmbeddedRuns owners plus activeTools/activeModelCalls, surfaced as activeWorkKind/hasActiveEmbeddedRun). applyRecoveryOutcomeToDiagnosticState in src/logging/diagnostic-session-recovery-coordinator.ts declares the lane idle and reconciles only the session-state store. The run-activity store is cleared solely by markDiagnosticEmbeddedRunEnded (which clears the owner AND its tool/model markers by default), and the embedded-run teardown can bypass it (handle replaced/mismatched, or force-clear finding no live handle). When that happens the coordinator flips the lane to idle but the run activity survives, so the liveness sweep formats idle + activity together and isIdleQueuedRecoverableSessionStall keeps re-triggering recovery without ever converging. The decisive evidence in the report is last=embedded_run:started (never :ended) on an idle lane.
  • Fix: Make the idle declaration authoritative over both stores, reconciling the whole terminal embedded-run activity (not just the owner set).
    • clearDiagnosticEmbeddedRunActivityForSession({ sessionId, sessionKey }) clears the session's activeEmbeddedRuns owners AND their activeTools/activeModelCalls markers and stamps terminal progress, matching the default markDiagnosticEmbeddedRunEnded teardown; it is a no-op when there is nothing to clear.
    • The coordinator calls it inside the guard-passed idle branch, immediately after the queue-depth reconciliation and before the session.state event.
    • Clearing only the owner set is insufficient: a stale tool/model marker would change the lane from idle + hasActiveEmbeddedRun to idle + orphaned tool/model activity, which isIdleQueuedRecoverableSessionStall's hasOrphanedActivity branch (added by fix(diagnostics): recover orphaned session activity and yield event loop during lock contention #87028) still treats as recoverable while queueDepth > 0 — so the sweep would keep re-triggering recovery.
    • The existing generation guard already bails when a newer run re-armed activity (a live run bumps the generation or keeps state processing), so only leaked/terminal activity is dropped — a legitimately requeued run is preserved.
  • What changed:
    • src/logging/diagnostic-run-activity.ts — add clearDiagnosticEmbeddedRunActivityForSession, which reconciles a session's terminal embedded-run activity (owners + tool/model markers).
    • src/logging/diagnostic-session-recovery-coordinator.ts — call it from the authoritative idle branch of applyRecoveryOutcomeToDiagnosticState.
    • Tests added to diagnostic.test.ts (stuck session recovery activity reconciliation describe): embedded-owner reconcile, stale-tool-marker convergence, stale-model-marker convergence, and newer-run-preserved guard cases.
  • What did NOT change (scope boundary):
    • The generation/ownership guards, the expectedState idle re-recovery path, and the preserveQueuedIdleWork queue reconciliation are unchanged; the clear runs only after those already decided to flip the lane idle.
    • The idle-queued classifier (isIdleQueuedRecoverableSessionStall) and the embedded-run teardown paths (clearActiveEmbeddedRun / forceClearEmbeddedAgentRun) are unchanged.
    • Config surface unchanged (no schema, defaults, doctor migrations, or docs/reference/config).
    • Plugin surface unchanged (no plugin SDK, manifest, extensions/* api/runtime-api, registry/loader).
    • Gateway protocol unchanged.

Reproduction

  1. A channel lane (e.g. WhatsApp direct) hits a stalled embedded run with queued work; the run leaves an embedded owner and possibly a tool/model marker.
  2. Stuck-session recovery aborts the run; its teardown removes the handle without reaching markDiagnosticEmbeddedRunEnded (handle replaced/mismatched, or force-clear finds no live handle).
  3. The recovery coordinator applies the aborted outcome and declares the lane idle.
  4. Before this PR: the run activity survives in the activity store, so the liveness sweep reports idle + activity and isIdleQueuedRecoverableSessionStall re-fires recovery forever; last=embedded_run:started never advances to :ended.
  5. After this PR: the idle declaration clears the session's embedded-run owners and their tool/model markers, the activity snapshot reports no activeWorkKind, the lane reads cleanly idle, and recovery converges.

Real behavior proof

Behavior addressed (#88660): after abort_embedded_run recovery declares a lane idle, the lane is no longer classifiable as idle/embedded_run (or idle + orphaned tool/model activity); the whole terminal embedded-run activity is reconciled atomically with the idle transition so the liveness sweep stops re-triggering recovery.

Real environment tested (Linux, Node 22 — Vitest against the production recovery coordinator, run-activity store, and heartbeat liveness loop): requestStuckSessionRecovery / applyRecoveryOutcomeToDiagnosticState driving the real diagnostic-session-state and diagnostic-run-activity stores, plus an end-to-end startDiagnosticHeartbeat sweep exercising the real isIdleQueuedRecoverableSessionStall dispatch.

Exact steps or command run after this patch: pnpm test src/logging/diagnostic.test.ts; full related suites pnpm test src/logging/diagnostic-stuck-session-recovery.runtime.test.ts src/logging/diagnostic-stuck-session-recovery.integration.test.ts src/logging/diagnostic-session-attention.test.ts src/agents/embedded-agent-runner/runs.test.ts; pnpm tsgo; pnpm exec oxfmt --check and node scripts/run-oxlint.mjs on the changed files.

Evidence after fix (Vitest output for the touched test file):

 diagnostic.test.ts   Tests  58 passed (58)

The reconciliation describe covers four cases: clears the embedded-run activity flag when recovery declares the lane idle, clears a stale tool marker left by the aborted run so a queued idle lane converges, clears a stale model marker left by the aborted run so a queued idle lane converges, and preserves an active flag for a newer run that re-armed work mid-recovery.

Observed result after fix: after the coordinator declares the lane idle, getDiagnosticSessionActivitySnapshot returns no activeWorkKind and no hasActiveEmbeddedRun — including when the aborted run had left a stale tool or model marker. An end-to-end heartbeat sweep over that reconciled state does not re-dispatch recovery; with an owner-only clear the same sweep still re-dispatches (orphaned tool_call/model_call keeps the lane recoverable), confirming the broader clear is what makes the lane converge. When a newer run re-arms activity mid-recovery, the generation guard bails: the lane stays processing and keeps its activity (live run preserved).

What was not tested: the live WhatsApp-gateway abort/recovery cycle end-to-end on the reporter's deployment. The coordinator idle declaration, the activity reconciliation, and the liveness re-dispatch are covered deterministically against the production functions; a full gateway round-trip was not driven.

Repro confirmation: each reconcile case fails on an owner-only-clear tree (the activity snapshot still reports embedded_run / tool_call / model_call) and passes with the fix, so the tests cover the production change.

Risk / Mitigation

  • Risk: A live run's activity could be cleared. Mitigation: the clear runs only on the authoritative idle transition that already passed the generation guard; a live run bumps the generation or keeps state processing, so the guard bails before the clear — only leaked/terminal activity is dropped.
  • Risk: Over-broad reconciliation of tool/model markers. Mitigation: this matches exactly what the normal markDiagnosticEmbeddedRunEnded teardown clears by default, and is gated behind the same guarded idle branch; it is a no-op when there is nothing to clear.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Diagnostics / logging
  • Sessions / storage

Linked Issue/PR

Fixes #88660

@clawsweeper

clawsweeper Bot commented May 31, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 1, 2026, 1:07 AM ET / 05:07 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +325, Tests +606. Total +931 across 6 files.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Merge readiness
Overall: 🌊 off-meta tidepool
Proof: 🌊 off-meta tidepool
Patch quality: 🌊 off-meta tidepool
Result: rating does not apply to this item.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 4e57546a8752.

Label changes

Label changes:

  • add rating: 🌊 off-meta tidepool: Overall readiness is 🌊 off-meta tidepool; proof is 🌊 off-meta tidepool and patch quality is 🌊 off-meta tidepool.
  • remove P1: Current review triage priority is none.
  • remove rating: 🐚 platinum hermit: Current PR rating is rating: 🌊 off-meta tidepool, so this older rating label is no longer current.
  • remove merge-risk: 🚨 session-state: Current PR review selected no merge-risk labels.
  • remove status: 👀 ready for maintainer look: Current PR status no longer selects a status label.

Label justifications:

  • rating: 🌊 off-meta tidepool: Overall readiness is 🌊 off-meta tidepool; proof is 🌊 off-meta tidepool and patch quality is 🌊 off-meta tidepool.
Evidence reviewed

PR surface:

Source +325, Tests +606. Total +931 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 4 341 16 +325
Tests 2 608 2 +606
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 6 949 18 +931

What I checked:

  • failure reason: codex execution failed.
  • codex failure detail: Codex review failed for this PR with exit 1.
  • codex stdout: Per-item Codex failure; continuing with the rest of the shard.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@openperf
openperf force-pushed the fix/88660-recovery-idle-embedded-run branch from 37ee982 to 75a0bc7 Compare May 31, 2026 23:35
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P1 High-priority user-facing bug, regression, or broken workflow. labels May 31, 2026
@openperf
openperf force-pushed the fix/88660-recovery-idle-embedded-run branch 2 times, most recently from 00623aa to c0a4394 Compare June 1, 2026 00:43
@steipete steipete self-assigned this Jun 1, 2026

@steipete steipete left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused fix. I think this needs one more cleanup in the same helper before it reliably satisfies the recovery invariant.

Findings:

  • [P1] Clear run-scoped tool/model activity when the idle transition is authoritative. clearDiagnosticEmbeddedRunsForSession currently only empties activeEmbeddedRuns, but the bypassed teardown path it is replacing would normally go through markDiagnosticEmbeddedRunEnded, which also clears activeTools and activeModelCalls unless explicitly told not to. If the aborted embedded run had a stale tool/model marker, this patch changes the lane from idle + hasActiveEmbeddedRun to idle + orphaned tool/model activity; isIdleQueuedRecoverableSessionStall intentionally treats that state as recoverable when queueDepth remains positive, so the liveness sweep can still re-trigger recovery instead of converging. Please make the authoritative idle cleanup clear the same run activity that normal embedded-run teardown clears, and add a regression case with a stale tool or model marker plus queued work. src/logging/diagnostic-run-activity.ts:300, src/logging/diagnostic.ts:553

Best-fix verdict: too narrow as written. The right layer is the recovery coordinator’s guarded idle branch, but the activity-store helper should reconcile the whole terminal embedded-run activity state, not only the owner set.

Alternatives considered: moving this into WhatsApp/channel code is worse because the invariant is shared diagnostics/session recovery; clearing only the owner set is insufficient because existing orphaned-activity recovery explicitly requeues stale tool/model markers; changing the idle-queued classifier would be broader and would weaken the existing orphan cleanup from #87028.

Code read: src/logging/diagnostic-session-recovery-coordinator.ts, src/logging/diagnostic-run-activity.ts, src/logging/diagnostic.ts, src/logging/diagnostic-session-attention.ts, src/logging/diagnostic-stuck-session-recovery.runtime.ts, src/agents/embedded-agent-runner/runs.ts, src/auto-reply/reply/reply-run-registry.ts, linked issue #88660.

Remaining uncertainty: I did not run the PR tests locally because I reviewed the fetched PR head without switching branches. CI currently has one unrelated-looking check-dependencies failure (ui/package.json: three unused); all diagnostics/code checks I inspected are passing.

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. label Jun 1, 2026
@openperf
openperf force-pushed the fix/88660-recovery-idle-embedded-run branch from 76a6524 to f337f84 Compare June 1, 2026 02:22
@steipete
steipete force-pushed the fix/88660-recovery-idle-embedded-run branch from f337f84 to a326686 Compare June 1, 2026 02:36
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: L and removed size: M labels Jun 1, 2026

@steipete steipete left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after the cleanup follow-up.

What changed since my requested-changes review:

  • The authoritative recovery-idle cleanup now clears embedded-run owners plus stale tool/model child activity, matching terminal embedded-run teardown semantics.
  • Added regressions for stale tool and model markers, plus the existing newer-run generation guard case.
  • Rebased onto current origin/main and added a separate CI repair for current cron schema assertions plus regenerated Codex prompt snapshots, because those checks were already failing on the updated base.

Local proof run on head a326686afaa61fcef7d941ee2d44cb86f6adee88:

  • pnpm exec oxfmt --check src/logging/diagnostic-run-activity.ts src/logging/diagnostic-session-recovery-coordinator.ts src/logging/diagnostic.test.ts src/agents/tools/cron-tool.schema.test.ts
  • pnpm test src/logging/diagnostic.test.ts -- --reporter=verbose -> 58 passed
  • pnpm test src/agents/tools/cron-tool.schema.test.ts -- --reporter=verbose -> 15 passed
  • pnpm prompt:snapshots:check -> current, 7 files
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --prompt ... -> clean, no accepted/actionable findings

Known proof gap: no live WhatsApp gateway recovery cycle was driven; this is covered at the deterministic diagnostics coordinator/activity-store layer.

@steipete
steipete force-pushed the fix/88660-recovery-idle-embedded-run branch from a326686 to 9b3f6ad Compare June 1, 2026 02:54
@steipete

steipete commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Land-ready proof for the refreshed branch at 9b3f6adf2752534bdae5d2db749971c343d0351b.

What changed after review:

  • Rebased the contributor fix onto current origin/main.
  • Kept the diagnostics recovery fix that clears stale embedded-run, tool-call, and model-call activity only after authoritative idle recovery for the same generation.
  • Refreshed the cron tool schema regression/snapshots for the current nullable schema shape.
  • Added a narrow phone-control CI repair so the touched plugin test mocks satisfy the current keyed-store SDK contract and runtime config readers accept readonly snapshots without casts.

Local proof run on the final stack:

  • pnpm exec oxfmt --check src/logging/diagnostic-run-activity.ts src/logging/diagnostic-session-recovery-coordinator.ts src/logging/diagnostic.test.ts src/agents/tools/cron-tool.schema.test.ts extensions/phone-control/index.ts extensions/phone-control/index.test.ts
  • node scripts/run-oxlint.mjs src/logging/diagnostic-run-activity.ts src/logging/diagnostic-session-recovery-coordinator.ts src/logging/diagnostic.test.ts src/agents/tools/cron-tool.schema.test.ts extensions/phone-control/index.ts extensions/phone-control/index.test.ts
  • pnpm test src/logging/diagnostic.test.ts src/agents/tools/cron-tool.schema.test.ts extensions/phone-control/index.test.ts -- --reporter=verbose (58 diagnostics, 15 cron schema, 14 phone-control tests passed)
  • pnpm check:test-types
  • pnpm prompt:snapshots:check
  • git diff --check origin/main..HEAD
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --prompt "Final autoreview for PR #88820 after the last rebase onto current origin/main. Upstream changed only iOS files; verify no new actionable blocker in the diagnostics recovery, cron schema snapshot, or phone-control CI repair stack." -> clean, no accepted/actionable findings

Known proof gap: no live WhatsApp gateway recovery cycle was run; this is covered with focused diagnostic recovery regression tests in src/logging/diagnostic.test.ts.

@openclaw-barnacle openclaw-barnacle Bot added the gateway Gateway runtime label Jun 1, 2026
@steipete
steipete force-pushed the fix/88660-recovery-idle-embedded-run branch from 54a224d to 2c4adf1 Compare June 1, 2026 03:55
@steipete

steipete commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Land-ready proof for current head 2c4adf1.

What changed since the earlier review:

  • Rebased on current origin/main.
  • Kept the contributor diagnostic recovery fix and tightened it so recovery clears only activity owned by the recovered run.
  • Added owner metadata for embedded runs, tool markers, and model-call markers so custom reply work keys are cleared for the recovered owner while fresh embedded/tool owners under the same session key are preserved.
  • Kept the phone-control keyed-store type/test fix needed by current test types.

Local proof on the final rebased branch:

  • pnpm test src/logging/diagnostic.test.ts extensions/phone-control/index.test.ts -- --reporter=verbose
  • pnpm check:test-types
  • pnpm exec oxfmt --check src/logging/diagnostic-run-activity.ts src/logging/diagnostic-session-recovery-coordinator.ts src/logging/diagnostic.test.ts extensions/phone-control/index.ts extensions/phone-control/index.test.ts
  • git diff --check origin/main...HEAD
  • pnpm check:architecture
  • pnpm lint --threads=8
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --prompt "Final autoreview rerun for PR fix(diagnostics): clear embedded-run activity when recovery declares lane idle #88820 after fixes: recovery clears recovered owner embedded work keys including custom reply work keys, clears recovered owner tool/model markers, and preserves fresh embedded/tool owners under the same session key. Focus on actionable merge blockers only."

Autoreview result: clean, no accepted/actionable findings reported.

Known gap: no live WhatsApp gateway recovery cycle was run; proof is focused unit/static coverage for the diagnostic reconciliation path touched here.

@openclaw-barnacle openclaw-barnacle Bot removed gateway Gateway runtime agents Agent runtime and tooling labels Jun 1, 2026
openperf and others added 3 commits June 1, 2026 06:02
…lane idle

Stuck-session recovery transitions a lane to idle via the recovery
coordinator, but only mutated the session-state store. When an aborted
embedded run was removed without markDiagnosticEmbeddedRunEnded, the
activity store kept hasActiveEmbeddedRun set, so the liveness sweep
reported idle/embedded_run and isIdleQueuedRecoverableSessionStall
re-triggered recovery indefinitely.

Reconcile the activity store from the authoritative idle declaration by
clearing the session's embedded-run owners. The existing generation
guard already excludes any newer run that re-armed activity, so a live
requeued run is preserved.
… cleanup

clearDiagnosticEmbeddedRunActivityForSession (renamed from
clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and
model markers alongside the embedded-run owners, matching the default
markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the
lane as idle + orphaned tool/model activity, which
isIdleQueuedRecoverableSessionStall still treats as recoverable while work is
queued, so the liveness sweep kept re-triggering recovery instead of converging.
Adds regression cases with stale tool and model markers plus queued work.
@steipete
steipete force-pushed the fix/88660-recovery-idle-embedded-run branch from 2c4adf1 to fbb17fd Compare June 1, 2026 05:03
@steipete

steipete commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Maintainer update at fbb17fdc98ad3f89068f7f8c7a8623cc17403b62.

What changed after review:

  • Recovery now clears/reconciles embedded-run, tool, and model-call activity when a mutating stuck-session recovery declares a lane idle.
  • It records diagnostic event cutoffs for recovered owners before empty-activity returns, so stale queued tool.execution.started / model.call.started events cannot recreate orphaned work after recovery.
  • It preserves activity started after recovery begins, including same-session rearmed embedded/tool work, while pruning cutoff-old same-key owners/markers before returning blocked for a fresh owner.
  • Phone-control keyed-store test mocks remain aligned with the current store API.

Proof run locally on the PR branch:

  • pnpm test src/logging/diagnostic.test.ts extensions/phone-control/index.test.ts -- --reporter=verbose (68 logging tests, 14 phone-control tests)
  • pnpm check:test-types
  • pnpm lint --threads=8
  • pnpm check:architecture
  • git diff --check origin/main...HEAD
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --prompt "..." -> clean: no accepted/actionable findings reported

Note: after the clean autoreview pass, origin/main advanced by one unrelated installer-test commit; I did a clean rebase and reran the focused diagnostics/phone-control test plus diff check on the pushed SHA.

@clawsweeper clawsweeper Bot removed the rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. label Jun 1, 2026
@steipete
steipete merged commit c0195f7 into openclaw:main Jun 1, 2026
158 checks passed
@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 1, 2026
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 1, 2026
…lane idle (openclaw#88820)

* fix(diagnostics): clear embedded-run activity when recovery declares lane idle

Stuck-session recovery transitions a lane to idle via the recovery
coordinator, but only mutated the session-state store. When an aborted
embedded run was removed without markDiagnosticEmbeddedRunEnded, the
activity store kept hasActiveEmbeddedRun set, so the liveness sweep
reported idle/embedded_run and isIdleQueuedRecoverableSessionStall
re-triggered recovery indefinitely.

Reconcile the activity store from the authoritative idle declaration by
clearing the session's embedded-run owners. The existing generation
guard already excludes any newer run that re-armed activity, so a live
requeued run is preserved.

* fix(diagnostics): reconcile tool/model activity on authoritative idle cleanup

clearDiagnosticEmbeddedRunActivityForSession (renamed from
clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and
model markers alongside the embedded-run owners, matching the default
markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the
lane as idle + orphaned tool/model activity, which
isIdleQueuedRecoverableSessionStall still treats as recoverable while work is
queued, so the liveness sweep kept re-triggering recovery instead of converging.
Adds regression cases with stale tool and model markers plus queued work.

* test(phone-control): align service mocks with keyed store API

* fix(diagnostics): preserve rearmed recovery activity

* fix(diagnostics): clear recovered owner markers

* fix(diagnostics): clear recovered embedded work keys

* fix(diagnostics): ignore stale same-key recovery owners

* fix(diagnostics): preserve same-session recovery rearm

* fix(diagnostics): ignore stale queued activity starts

* fix(diagnostics): record recovery cutoffs for empty activity

* fix(diagnostics): preserve fresh recovery markers

* fix(diagnostics): prune stale activity before fresh recovery block

---------

Co-authored-by: Peter Steinberger <[email protected]>
SYU8384 pushed a commit to SYU8384/openclaw that referenced this pull request Jun 3, 2026
…lane idle (openclaw#88820)

* fix(diagnostics): clear embedded-run activity when recovery declares lane idle

Stuck-session recovery transitions a lane to idle via the recovery
coordinator, but only mutated the session-state store. When an aborted
embedded run was removed without markDiagnosticEmbeddedRunEnded, the
activity store kept hasActiveEmbeddedRun set, so the liveness sweep
reported idle/embedded_run and isIdleQueuedRecoverableSessionStall
re-triggered recovery indefinitely.

Reconcile the activity store from the authoritative idle declaration by
clearing the session's embedded-run owners. The existing generation
guard already excludes any newer run that re-armed activity, so a live
requeued run is preserved.

* fix(diagnostics): reconcile tool/model activity on authoritative idle cleanup

clearDiagnosticEmbeddedRunActivityForSession (renamed from
clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and
model markers alongside the embedded-run owners, matching the default
markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the
lane as idle + orphaned tool/model activity, which
isIdleQueuedRecoverableSessionStall still treats as recoverable while work is
queued, so the liveness sweep kept re-triggering recovery instead of converging.
Adds regression cases with stale tool and model markers plus queued work.

* test(phone-control): align service mocks with keyed store API

* fix(diagnostics): preserve rearmed recovery activity

* fix(diagnostics): clear recovered owner markers

* fix(diagnostics): clear recovered embedded work keys

* fix(diagnostics): ignore stale same-key recovery owners

* fix(diagnostics): preserve same-session recovery rearm

* fix(diagnostics): ignore stale queued activity starts

* fix(diagnostics): record recovery cutoffs for empty activity

* fix(diagnostics): preserve fresh recovery markers

* fix(diagnostics): prune stale activity before fresh recovery block

---------

Co-authored-by: Peter Steinberger <[email protected]>
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…lane idle (openclaw#88820)

* fix(diagnostics): clear embedded-run activity when recovery declares lane idle

Stuck-session recovery transitions a lane to idle via the recovery
coordinator, but only mutated the session-state store. When an aborted
embedded run was removed without markDiagnosticEmbeddedRunEnded, the
activity store kept hasActiveEmbeddedRun set, so the liveness sweep
reported idle/embedded_run and isIdleQueuedRecoverableSessionStall
re-triggered recovery indefinitely.

Reconcile the activity store from the authoritative idle declaration by
clearing the session's embedded-run owners. The existing generation
guard already excludes any newer run that re-armed activity, so a live
requeued run is preserved.

* fix(diagnostics): reconcile tool/model activity on authoritative idle cleanup

clearDiagnosticEmbeddedRunActivityForSession (renamed from
clearDiagnosticEmbeddedRunsForSession) now clears the aborted run's tool and
model markers alongside the embedded-run owners, matching the default
markDiagnosticEmbeddedRunEnded teardown. Clearing only the owner set left the
lane as idle + orphaned tool/model activity, which
isIdleQueuedRecoverableSessionStall still treats as recoverable while work is
queued, so the liveness sweep kept re-triggering recovery instead of converging.
Adds regression cases with stale tool and model markers plus queued work.

* test(phone-control): align service mocks with keyed store API

* fix(diagnostics): preserve rearmed recovery activity

* fix(diagnostics): clear recovered owner markers

* fix(diagnostics): clear recovered embedded work keys

* fix(diagnostics): ignore stale same-key recovery owners

* fix(diagnostics): preserve same-session recovery rearm

* fix(diagnostics): ignore stale queued activity starts

* fix(diagnostics): record recovery cutoffs for empty activity

* fix(diagnostics): preserve fresh recovery markers

* fix(diagnostics): prune stale activity before fresh recovery block

---------

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: phone-control merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WhatsApp lane remains idle/embedded_run after abort_embedded_run recovery

2 participants