Skip to content

fix(diagnostics): reclaim wedged session lanes with a stale leaked active run#86056

Merged
steipete merged 2 commits into
openclaw:mainfrom
openperf:fix/85639-stuck-recovery-liveness-gate
May 25, 2026
Merged

fix(diagnostics): reclaim wedged session lanes with a stale leaked active run#86056
steipete merged 2 commits into
openclaw:mainfrom
openperf:fix/85639-stuck-recovery-liveness-gate

Conversation

@openperf

@openperf openperf commented May 24, 2026

Copy link
Copy Markdown
Member

Summary

  • Problem: A group session lane can wedge permanently — messages queue (queueDepth=1) but never dispatch. Stuck-session recovery detects the wedge but skips with reason=active_reply_work action=keep_lane, repeating every ~2 minutes (the reporter saw 23+ days). The bot goes silent with no user-visible error. (Group session lane wedges indefinitely: stuck-recovery skips with active_reply_work false positive #85639)
  • Root Cause: A multi-layer state divergence.
    • An embedded agent run dies abnormally (child crash / parent doesn't observe the death cleanly), so ACTIVE_EMBEDDED_RUNS keeps a leaked handle and isEmbeddedPiRunActive(...) stays true, while the separate diagnostic run-activity tracking already shows no active run.
    • The diagnostic heartbeat therefore classifies the lane as stale_session_state (queued_work_without_active_run), which is recoveryEligible without allowActiveAbort (that grant is reserved for session.stalled classifications).
    • recoverStuckDiagnosticSession then reads the leaked isEmbeddedPiRunActive flag and, because allowActiveAbort !== true, skips with active_reply_work — a tautology: it keeps the lane because the lane looks active.
    • The existing age-based escape doesn't fire: ageMs is the session's last-activity age, which resets every time a new message queues, so it never crosses a timeout (the reporter's age stayed ~124s for weeks). The existing progress-based abort (lastProgressAgeMs) only runs for session.stalled classifications, not stale_session_state.
  • Fix: Make the active-run skip a liveness check instead of a blind flag read. Before keeping the lane for active_embedded_run / active_reply_work, consult the run's real forward-progress age (getDiagnosticSessionActivitySnapshot(...).lastProgressAgeMs, which is touched by tool/model/chunk events, not by incoming queued messages). If a run flagged active has made no forward progress for ≥ a staleness window and there is queued work, treat it as a leaked/dead handle and reclaim it (abort + drain + force-clear) instead of skipping. A genuinely progressing run (recent lastProgressAgeMs) is still kept.
  • What changed:
    • src/logging/diagnostic-stuck-session-recovery.runtime.ts: add isActiveRunProgressStale (queued work + stale lastProgressAgeMs); both the active_embedded_run and active_reply_work skip branches reclaim instead of skipping when the active run's progress is stale.
    • src/logging/diagnostic-session-recovery.ts + src/logging/diagnostic.ts: thread the resolved diagnostics.stuckSessionAbortMs into the recovery request (staleActiveProgressAbortMs) so the reclaim honors an operator-raised abort threshold; it falls back to the MIN_STALLED_EMBEDDED_RUN_ABORT_MS (5 min) floor when not supplied, matching the existing stalled-run abort policy.
    • src/logging/diagnostic-stuck-session-recovery.runtime.test.ts: regression coverage.
  • What did NOT change (scope boundary):
    • The allowActiveAbort path, the diagnostic classifier, the heartbeat, and the run registries are unchanged. The reclaim is additive: it only fires when an active flag is paired with stale forward progress and queued work, so existing skip/abort behavior is otherwise identical.
    • No config keys, protocol, or message-delivery behavior changed.

Reproduction

  1. A run starts on a group session lane, then dies abnormally so its ACTIVE_EMBEDDED_RUNS handle is never cleared (diagnostic activity already shows no active run).
  2. New messages keep queuing on the lane.
  • Before: every heartbeat classifies stale_session_state, recovery reads the leaked active flag and skips with active_reply_work — the lane never frees (permanent wedge).
  • After: once the leaked run has made no forward progress past the staleness window with queued work waiting, recovery reclaims the lane (abort + force-clear) and queued work dispatches.

Real behavior proof

  • Behavior addressed: when a session is stuck with queued work and a leaked "active" run flag whose forward progress is stale, recovery must reclaim the lane rather than skip with active_reply_work; a still-progressing run must not be reclaimed.
  • Real environment tested: the real recoverStuckDiagnosticSession decision path exercised under Vitest (Linux, Node 22). The run/abort registry and lane seams are mocked exactly as the existing 15-case suite in this file does; the recovery logic, the new liveness gate, and lastProgressAgeMs thresholding are real.
  • Exact steps or command run after this patch: pnpm test src/logging/diagnostic-stuck-session-recovery.runtime.test.ts.
  • Evidence after fix (verbatim test output):
 Test Files  1 passed (1)
      Tests  19 passed (19)
  • Before evidence (same tests with the production fix reverted, on base c91c3c6e5a): the new staleness-reclaim case fails because the unpatched recovery skips (the wedge); the two control cases (still-progressing, no queued work) pass because the unpatched path also keeps the lane:
 × reclaims stale leaked reply work with queued work and no forward progress (#85639)
 Test Files  1 failed (1)
      Tests  1 failed | 2 passed | 15 skipped (18)
  • Real-component proof (no mocked seams): a standalone harness (tsx /tmp/qmd85639/repro.mts, Linux/Node 22) drives the real recoverStuckDiagnosticSession, the real embedded-run registry (setActiveEmbeddedRun leaks a handle to simulate abnormal death), and the real diagnostic run-activity tracker; the clock is advanced so the run's real lastProgressAgeMs crosses the window. FRESH (recent progress) keeps the lane; STALE (10 min, no progress, queued work) reclaims it (real abort + force-clear), after which the handle is gone (active=false):
=== #85639 real-component recovery repro (real registry + activity + recovery) ===
--- FRESH: leaked active handle but recent forward progress ---
[repro] FRESH: lastProgress=0s active=true -> status=skipped action=observe_only reason=active_embedded_run
--- STALE: leaked active handle, no forward progress for 10m + queued work ---
[repro] STALE: lastProgress=600s active=false -> status=aborted action=abort_embedded_run reason=-

(The STALE run also emitted the real gateway logs stuck session recovery reclaiming stale active run … and … action=abort_embedded_run aborted=true forceCleared=true.)

  • Observed result after fix: the stale leaked active run is reclaimed (abort + drain + force-clear, logged reclaiming stale active …); the still-progressing run and the no-queued-work cases still keep the lane (no false reclaim).
  • What was not tested: a full multi-process live gateway wedge over days was not run. The bug is in-memory recovery decision logic; the real-component harness above exercises the actual recovery + registry + activity tracker end to end, and the unit tests cover the decision branches. The staleness window honors diagnostics.stuckSessionAbortMs (default ≥ 5 min) so a live-but-slow run is not reclaimed.

Risk / Mitigation

  • Risk: reclaiming could abort a genuinely active long-running run.
  • Mitigation: reclaim only fires when forward progress (lastProgressAgeMs, which is not refreshed by incoming queued messages) is stale for ≥ the resolved diagnostics.stuckSessionAbortMs (operator-configurable; default ≥ 5 min) and there is queued work; a run making any recent progress is kept. An operator who raises the threshold to protect slow active work is honored (covered by a raised-threshold test: stale at 10 min is kept, at 25 min is reclaimed). This mirrors the existing blocked_tool_call abort which already triggers at the same lastProgressAgeMs/threshold scale. Without queued work, nothing is reclaimed.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Gateway / orchestration
  • Sessions / storage

Linked Issue/PR

Fixes #85639

@openclaw-barnacle openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. labels May 24, 2026
@clawsweeper

clawsweeper Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed May 25, 2026, 9:15 AM ET / 13:15 UTC.

Summary
The PR adds stale-progress liveness checks to stuck diagnostic session recovery, threads the resolved stuck-session abort threshold into recovery requests, adds regression tests, and records a changelog entry.

PR surface: Source +74, Tests +130, Docs +1. Total +205 across 5 files.

Reproducibility: Do we have a high-confidence way to reproduce the issue? Yes at source and harness level: current main keeps active_reply_work without a progress check, and the PR body includes before/after focused output plus a real-component recovery transcript.

Review metrics: 1 noteworthy metric.

  • Recovery Branch Changes: 2 skip branches changed to conditional reclaim. Both active embedded-run and active reply-work recovery branches now affect session and message-delivery behavior when progress is stale.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • Merging changes current recovery behavior from preserving apparently active work to aborting or force-clearing it when queued work waits and no forward progress is observed past the configured threshold; a genuinely live but silent long-running run could be stopped.
  • The proof is strong for the in-memory recovery decision path, but it is not a full multi-process gateway or live Telegram wedge over days, so maintainers may still want broader live proof before accepting the recovery-policy tradeoff.

Maintainer options:

  1. Land With Recovery Policy Accepted (recommended)
    Maintainers can land this focused fix if they accept that stale-progress queued lanes may now abort and force-clear an apparently active run after the configured threshold.
  2. Request Broader Gateway Proof
    Maintainers can ask for a packaged or live gateway scenario if they want more confidence that slow but healthy runs are not stopped by the new liveness gate.
  3. Hold For Recovery Redesign
    If aborting leaked-looking active runs is not acceptable, hold this PR and redesign the linked issue around a different recovery policy.

Next step before merge
No automated repair is needed; a maintainer should accept or reject the recovery-policy risk and wait for required checks before landing or holding the PR.

Security
Cleared: The diff touches in-process diagnostic recovery, tests, and changelog only; I found no new dependency, workflow, secret, package, or external code-execution surface.

Review details

Best possible solution:

Land the focused liveness-gated recovery after maintainer acceptance of the abort-threshold policy, keeping the existing diagnostics threshold and regression coverage.

Do we have a high-confidence way to reproduce the issue?

Do we have a high-confidence way to reproduce the issue? Yes at source and harness level: current main keeps active_reply_work without a progress check, and the PR body includes before/after focused output plus a real-component recovery transcript.

Is this the best way to solve the issue?

Is this the best way to solve the issue? Yes, subject to maintainer policy acceptance: it reuses the existing diagnostic activity signal and diagnostics.stuckSessionAbortMs instead of adding a new hard timeout or config surface.

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 8fe4f34af224.

Label changes

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies after-fix terminal output from focused tests and a real-component harness using real recovery, registry, and activity tracking, with before-fix failing output for the stale-reclaim case.

Label justifications:

  • P1: The linked bug can leave a real group session lane wedged for days with queued messages never dispatching.
  • merge-risk: 🚨 compatibility: Existing setups with apparently active but silent long-running runs may now be abort-drained after the configured stale-progress threshold.
  • merge-risk: 🚨 session-state: The new recovery path can abort, force-clear, and update diagnostic session lane state.
  • merge-risk: 🚨 message-delivery: The touched path directly controls whether queued channel messages stay stuck or are released after recovery.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body supplies after-fix terminal output from focused tests and a real-component harness using real recovery, registry, and activity tracking, with before-fix failing output for the stale-reclaim case.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies after-fix terminal output from focused tests and a real-component harness using real recovery, registry, and activity tracking, with before-fix failing output for the stale-reclaim case.
Evidence reviewed

PR surface:

Source +74, Tests +130, Docs +1. Total +205 across 5 files.

View PR surface stats
Area Files Added Removed Net
Source 3 76 2 +74
Tests 1 130 0 +130
Docs 1 1 0 +1
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 5 207 2 +205

What I checked:

Likely related people:

  • steipete: History shows repeated commits introducing and refining stale session lane recovery, stuck diagnostic session recovery, and the related diagnostics threshold behavior. (role: feature owner and recent recovery contributor; confidence: high; commits: c500e8704f4e, 761e668acf8a, e30be460e1a3; files: src/logging/diagnostic-stuck-session-recovery.runtime.ts, src/logging/diagnostic.ts, src/logging/diagnostic-session-recovery.ts)
  • joshavant: Merged work in the same diagnostic recovery area added stale embedded tool-call recovery and touched the central diagnostic heartbeat/recovery path. (role: recent adjacent contributor; confidence: medium; commits: a1e208ee26b9; files: src/logging/diagnostic.ts, src/logging/diagnostic-stuck-session-recovery.runtime.test.ts)
  • FullerStackDev: Current line blame for the logging recovery files points through a broad recent migration commit that carried these files into the current main tree, though the behavior predates that migration. (role: recent area migration contributor; confidence: low; commits: f036bac14423; files: src/logging/diagnostic-stuck-session-recovery.runtime.ts, src/logging/diagnostic.ts, src/logging/diagnostic-run-activity.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels May 24, 2026
@clawsweeper

clawsweeper Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

✨ Hatched: 🥚 common Clockwork Proofling

Hatch command

Comment @clawsweeper hatch when this PR is hatchable.

Hatchability rules:

  • Merged PRs are hatchable.
  • Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
  • Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

Rarity: 🥚 common.
Trait: collects tiny proofs.
Image traits: location artifact grotto; accessory commit compass; palette pearl, teal, and neon green; mood sleepy but ready; pose curling around a status light; shell polished stone shell; lighting gentle morning glow; background small review tokens.
Share on X: post this hatch
Copy: My PR egg hatched a 🥚 common Clockwork Proofling in ClawSweeper.

What is this egg doing here?
  • Eggs appear after the PR passes real-behavior proof. It is here for vibes, not verdicts: it does not change labels, ratings, merge decisions, or automation.
  • The shell reacts to review momentum: open follow-up work warms it up, re-review makes it wobble, and a clean final review lets it hatch.
  • Hatchability usually comes from sufficient real-behavior proof, no blocking P0/P1/P2 findings, no security attention needed, and clean correctness. A merged PR is already final, so merge makes the egg hatchable independently.
  • The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
  • Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

@openperf
openperf force-pushed the fix/85639-stuck-recovery-liveness-gate branch from 55f4171 to 87df9d4 Compare May 24, 2026 13:36
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels May 24, 2026
@openperf
openperf force-pushed the fix/85639-stuck-recovery-liveness-gate branch 2 times, most recently from c0db2f3 to 9cfee88 Compare May 24, 2026 13:59
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 24, 2026
@openperf
openperf force-pushed the fix/85639-stuck-recovery-liveness-gate branch from 9cfee88 to e9ad954 Compare May 24, 2026 14:11
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 24, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels May 24, 2026
@openperf
openperf force-pushed the fix/85639-stuck-recovery-liveness-gate branch from e9ad954 to a81d041 Compare May 24, 2026 14:39
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 24, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 24, 2026
@openperf
openperf force-pushed the fix/85639-stuck-recovery-liveness-gate branch from a81d041 to 42709dc Compare May 25, 2026 06:46
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 25, 2026
@openperf
openperf force-pushed the fix/85639-stuck-recovery-liveness-gate branch from 42709dc to 97d0d34 Compare May 25, 2026 06:57
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 25, 2026
@steipete steipete self-assigned this May 25, 2026
openperf and others added 2 commits May 25, 2026 14:04
…tive run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639
@steipete
steipete force-pushed the fix/85639-stuck-recovery-liveness-gate branch from 4c1d613 to 09084c5 Compare May 25, 2026 13:08
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 25, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 25, 2026
@steipete

Copy link
Copy Markdown
Contributor

Verification for PR #86056 on head 09084c5e1da459f83245a54b5db1f9d98670fc34:

Behavior addressed: stale active-run and reply-work bookkeeping no longer leaves queued session work wedged when there is no forward progress.
Real environment tested: local focused Vitest/format/autoreview plus GitHub CI on the PR head.
Exact steps or command run after this patch:

node scripts/run-vitest.mjs src/logging/diagnostic-stuck-session-recovery.runtime.test.ts
git diff --check origin/main...HEAD
./node_modules/.bin/oxfmt --check --threads=1 src/logging/diagnostic-stuck-session-recovery.runtime.test.ts
/Users/steipete/Projects/agent-scripts/skills/autoreview/scripts/autoreview --mode branch --base origin/main

Evidence after fix: focused diagnostics regression suite passed with 20 tests; diff check and oxfmt check passed; autoreview reported no accepted/actionable findings; CI run 26402142228 passed after rerunning the unrelated checks-node-agentic-agents timeout; real behavior proof 26402140563, CodeQL High 26402142229, and Critical Quality 26402142262 passed.
Observed result after fix: active reply-work and active embedded-run stale lanes are both reclaimed when queued work is blocked with no session progress.
What was not tested: no multi-day live gateway wedge reproduction; coverage is the focused runtime recovery harness plus PR real-behavior proof.

@steipete
steipete merged commit 6f76d9f into openclaw:main May 25, 2026
169 of 171 checks passed
steipete added a commit that referenced this pull request May 25, 2026
…tive run (#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes #85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 26, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
SYU8384 pushed a commit to SYU8384/openclaw that referenced this pull request Jun 3, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
sablehead pushed a commit to sablehead/openclaw that referenced this pull request Jun 10, 2026
…tive run (openclaw#86056)

* fix(diagnostics): reclaim wedged session lanes with a stale leaked active run

A group session lane could wedge permanently (openclaw#85639): an embedded run that dies
abnormally leaves a stale ACTIVE_EMBEDDED_RUNS handle, so the diagnostic heartbeat
classifies the lane stale_session_state (recoveryEligible without allowActiveAbort)
while stuck-session recovery reads the leaked isEmbeddedPiRunActive flag and skips
with active_reply_work — a tautology that keeps the lane forever. The age-based
escape never fires because ageMs (last-activity) resets on every incoming queued
message.

Make the active-run skip a liveness check: before keeping the lane, consult the
run's real forward-progress age (lastProgressAgeMs, not refreshed by incoming
messages). If a run flagged active has made no forward progress past the resolved
diagnostics.stuckSessionAbortMs threshold (threaded through the recovery request;
falls back to a 5-minute floor) with queued work waiting, treat it as a
leaked/dead handle and reclaim it (abort + drain + force-clear) instead of
skipping. A genuinely progressing run, or one within an operator-raised
threshold, is kept.

Fixes openclaw#85639

* test(diagnostics): cover stale active run recovery

---------

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Group session lane wedges indefinitely: stuck-recovery skips with active_reply_work false positive

2 participants