Skip to content

fix(diagnostics): evict orphaned tool/model activity on owner-less run end#90750

Open
849261680 wants to merge 4 commits into
openclaw:mainfrom
849261680:fix/87310-diagnostic-stale-tool-activity
Open

fix(diagnostics): evict orphaned tool/model activity on owner-less run end#90750
849261680 wants to merge 4 commits into
openclaw:mainfrom
849261680:fix/87310-diagnostic-stale-tool-activity

Conversation

@849261680

@849261680 849261680 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Stale native tool_call diagnostic activity could survive a clean run teardown and re-block later turns on the same sessionKey as blocked_tool_call (#87310).

markDiagnosticEmbeddedRunEnded({ clearRunActivity: false }) — used by the reply-run wrapper so it does not clobber a still-draining inner embedded run — removed only the embedded-run marker and left tool/model markers untouched. When a native tool never emitted its matching tool.execution.completed event (e.g. the embedded Codex app-server tool whose owner is torn down first), that marker leaked into the per-session activity (keyed by sessionKey). A later turn then saw activeWorkKind=tool_call with a very large activeToolAgeMs, was classified blocked_tool_call, and produced repeated session.stalled diagnostics — and recovery could even abort the fresh, healthy run.

Fix

  • src/logging/diagnostic-run-activity.ts: when an embedded/reply run ends and no embedded-run owner remains for the session, leftover tool/model markers are now evicted (they can never report completion). A still-active inner run keeps its markers, preserving the original reason clearRunActivity: false existed.
  • New structured session.activity.evicted event (reason=orphaned_no_owner, evictedTools/evictedModelCalls) so operators can distinguish recovered stale state from a real active tool (issue items 3 & 5). Recorded in the stability ring buffer and exported as the openclaw.session.activity.evicted (OTel) / openclaw_session_activity_evicted_total (Prometheus) counter.
  • Docs updated in docs/gateway/opentelemetry.md and docs/gateway/prometheus.md.

Recovery-driven cleanup (clearDiagnosticEmbeddedRunActivityForSession) is unchanged and remains observable via the existing session.recovery.completed event; this change targets the non-recovery clean-teardown path that previously evicted nothing and leaked silently.

Real behavior proof (runtime diagnostics)

  • Behavior addressed: Orphaned native tool_call (and model_call) diagnostic activity surviving a clean reply-run completion and re-blocking the next turn on the same sessionKey as blocked_tool_call.
  • Real environment tested: An in-process runtime harness exercising the real diagnostic runtime — startDiagnosticHeartbeat (30s warn / 60s abort) wired to the real recoverStuckDiagnosticSession from diagnostic-stuck-session-recovery.runtime.ts and the real reply-run-registry — on macOS, Node 22.22.2. The diagnosticLogger.warn output below is the actual emitted runtime log, not a mocked assertion.
  • Exact steps or command run after this patch: Drove the real heartbeat: turn 1 opened a reply operation on agent:main:slack:channel:REDACTED, started a native bash tool that never emits tool.execution.completed, then completed the reply run and marked the session idle; turn 2 queued a new message on the same session key and went processing; advanced 8 heartbeat cycles past the 60s abort threshold and captured diagnosticLogger.warn. The same harness was run once with the patch reverted to capture the "before" lines.
  • Observed result after fix: The reason=blocked_tool_call classification=blocked_tool_call line no longer appears for the follow-up turn; the orphaned bash marker is evicted at reply completion and the follow-up turn is handled as a recoverable stale lane (release_lane), so the session converges instead of repeatedly re-blocking.
  • Evidence after fix: Redacted runtime diagnostic logger output captured from the real diagnostic heartbeat + stuck-session recovery runtime (synthetic session ids):
BEFORE (patch reverted) — turn 2 is misclassified off the orphaned tool from turn 1:
stalled session: sessionId=SESSION-2 sessionKey=agent:main:slack:channel:REDACTED state=processing age=60s queueDepth=1 reason=blocked_tool_call classification=blocked_tool_call activeWorkKind=tool_call lastProgress=embedded_run:ended lastProgressAge=60s activeTool=bash activeToolCallId=T1 activeToolAge=60s terminalProgressStale=true recovery=checking

AFTER (this patch) — the orphaned bash marker is evicted on reply completion; turn 2 is a recoverable stale lane, never blocked_tool_call:
stuck session: sessionId=SESSION-2 sessionKey=agent:main:slack:channel:REDACTED state=processing age=60s queueDepth=1 reason=queued_work_without_active_run classification=stale_session_state lastProgress=embedded_run:ended lastProgressAge=60s terminalProgressStale=true recovery=checking
stuck session recovery: sessionId=SESSION-2 sessionKey=agent:main:slack:channel:REDACTED age=60s action=release_lane aborted=false drained=true released=0
stuck session recovery outcome: status=released action=release_lane sessionId=SESSION-2 sessionKey=agent:main:slack:channel:REDACTED lane=session:agent:main:slack:channel:REDACTED released=0
  • What was not tested: A live external systemd gateway + Slack socket-mode deployment was not stood up; the timing-sensitive hung-native-tool condition is non-deterministic in production and was reproduced deterministically through the real in-process runtime path. OTel/Prometheus exporter wiring is covered by exporter tests rather than a scraped live collector.

Verification

Suites run after the patch: diagnostic, reply-run-registry, diagnostic-stability, diagnostic-stuck-session-recovery.runtime/.integration, diagnostic-session-attention, diagnostics-otel, diagnostics-prometheus — all green. tsgo core + core-test + extensions + extensions-test clean; oxlint/oxfmt clean. New regression tests:

  • reply-run-registry: evicts an orphaned native tool when a reply run completes without its completion.
  • diagnostic: evicts orphaned tool/model markers when the last owner ends without clearing run activity; keeps tool markers when an inner embedded run is still active.
  • diagnostics-otel / diagnostics-prometheus: export the session.activity.evicted counter by total markers without leaking raw ids.

Closes #87310

@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. docs Improvements or additions to documentation gateway Gateway runtime size: M labels Jun 5, 2026
@clawsweeper

clawsweeper Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 30, 2026, 1:34 PM ET / 17:34 UTC.

Summary
The PR evicts orphaned diagnostic tool/model markers when the final embedded owner ends, adds a structured eviction event with OTel/Prometheus counters and docs, and adds regression tests.

PR surface: Source +135, Tests +224, Docs +11. Total +370 across 11 files.

Reproducibility: yes. at source level and through supplied runtime logs. Current main has the clearRunActivity:false reply-run teardown path, and the PR body shows before/after diagnostic heartbeat output for the stale blocked_tool_call scenario.

Review metrics: 2 noteworthy metrics.

  • Operator telemetry added: 1 diagnostic event, 2 exporter counters. The new event and metric names become operator-visible diagnostics that should be accepted before merge.
  • Owner-less cleanup path: 1 clearRunActivity:false teardown path changed. This is the runtime boundary that decides whether stale or genuine tool/model activity survives after reply-run teardown.

Stored data model
Persistent data-model change detected: serialized state: extensions/diagnostics-otel/src/service.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #87310
Summary: This PR is the focused candidate fix for the canonical stale diagnostic tool/model marker report; related model-call and broader gateway recovery items overlap but do not replace it.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] The owner-less teardown boundary is session-state sensitive: if embedded-run ownership is misidentified, genuine tool/model markers could be evicted or stale markers could still affect recovery decisions.
  • [P1] The supplied proof exercises the real in-process diagnostic heartbeat and recovery runtime, but it does not include a live external systemd gateway plus Slack socket-mode artifact for the production deployment shape.
  • [P1] The new session.activity.evicted event and OTel/Prometheus metric names are operator-visible telemetry surface that maintainers should accept before merge.

Maintainer options:

  1. Accept boundary and merge (recommended)
    A maintainer can accept the in-process runtime proof plus regression/exporter coverage and merge once current-head checks are green.
  2. Request live gateway proof
    Maintainers can ask for a redacted live Slack/systemd gateway repro artifact if they want production-shape evidence before accepting the cleanup boundary.
  3. Pause for broader recovery design
    If maintainers want one larger stale-lane recovery redesign instead of this focused marker eviction, leave the PR open or close it in favor of a canonical design item.

Next step before merge

  • [P2] The PR has no concrete automated repair target; maintainers should review the session-state boundary and telemetry names, then merge or request live gateway proof.

Security
Cleared: No concrete security or supply-chain regression was found; the telemetry additions avoid exporting raw session ids or keys in exporter tests.

Review details

Best possible solution:

Land the focused cleanup if maintainers accept the owner-less activity boundary and telemetry names, while keeping broader stale lane and model-call recovery reports separate.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level and through supplied runtime logs. Current main has the clearRunActivity:false reply-run teardown path, and the PR body shows before/after diagnostic heartbeat output for the stale blocked_tool_call scenario.

Is this the best way to solve the issue?

Yes, this is the best current fix shape: it repairs the diagnostic activity owner boundary rather than adding a channel-specific workaround. Maintainer review should focus on whether the owner-less cleanup condition and telemetry names are acceptable.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 5c4e478df4ef.

Label changes

Label justifications:

  • P1: The PR targets a stale diagnostic activity bug that can keep later session work classified as blocked and delay or suppress channel replies.
  • merge-risk: 🚨 session-state: Merging changes how per-session diagnostic tool/model activity is retained or evicted during reply-run teardown.
  • merge-risk: 🚨 message-delivery: Incorrect cleanup could either leave later turns blocked behind stale activity or evict active-work evidence used to protect message flow.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): The PR body includes redacted before/after runtime diagnostic logs from the real diagnostic heartbeat and stuck-session recovery runtime, which is sufficient for this non-visual diagnostic change.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes redacted before/after runtime diagnostic logs from the real diagnostic heartbeat and stuck-session recovery runtime, which is sufficient for this non-visual diagnostic change.
Evidence reviewed

PR surface:

Source +135, Tests +224, Docs +11. Total +370 across 11 files.

View PR surface stats
Area Files Added Removed Net
Source 5 138 3 +135
Tests 4 225 1 +224
Docs 2 11 0 +11
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 11 374 4 +370

What I checked:

  • Current-main cleanup gap: Current main still clears active tool/model markers in markDiagnosticEmbeddedRunEnded only when clearRunActivity is not false, so the reply-run opt-out path can leave markers behind. (src/logging/diagnostic-run-activity.ts:324, 5c4e478df4ef)
  • Affected reply-run caller: Reply-run teardown calls markDiagnosticEmbeddedRunEnded with clearRunActivity:false, matching the owner-less path described by the linked issue and PR body. (src/auto-reply/reply/reply-run-registry.ts:362, 5c4e478df4ef)
  • PR cleanup implementation: At the PR head, markDiagnosticEmbeddedRunEnded evicts leftover tool/model activity only when clearRunActivity is false and no embedded-run owner remains, then fences late async start events with the existing recovered-owner cutoff mechanism. (src/logging/diagnostic-run-activity.ts:327, 83c3212755ca)
  • Regression coverage: The PR adds reply-run tests for synchronous orphan eviction and a queued tool start that drains after owner-less completion, plus diagnostic tests for preserving markers while an inner embedded run remains active. (src/auto-reply/reply/reply-run-registry.test.ts:101, 83c3212755ca)
  • Exporter wiring: The PR records the new eviction event as an OTel counter and a Prometheus counter while using only the reason label, not raw session ids or keys. (extensions/diagnostics-otel/src/service.ts:1428, 83c3212755ca)
  • Related issue and proof context: The linked canonical issue reports stale activeWorkKind=tool_call activity poisoning later turns as blocked_tool_call, and the PR body supplies redacted before/after runtime diagnostic logs from the real heartbeat and stuck-session recovery path.

Likely related people:

  • obviyus: GitHub commit metadata shows obviyus authored the reply-run diagnostic tracking commit for the lifecycle path that calls markDiagnosticEmbeddedRunEnded with clearRunActivity:false, and also committed adjacent native-tool diagnostic simplification. (role: introduced adjacent reply-run diagnostics; confidence: high; commits: 3d3a2399b569, e36cc6b1114b; files: src/auto-reply/reply/reply-run-registry.ts, src/logging/diagnostic-run-activity.ts, extensions/codex/src/app-server/event-projector.ts)
  • openperf: Authored the recent recovery cleanup that reconciles embedded-run activity when recovery declares a lane idle, directly adjacent to this activity-store cleanup boundary. (role: recent diagnostic recovery contributor; confidence: high; commits: 80fce89f71d2; files: src/logging/diagnostic-run-activity.ts, src/logging/diagnostic.test.ts)
  • steipete: Authored and committed adjacent recovered-owner cutoff work in the diagnostic activity store, including the mechanism this PR reuses for late async start fencing. (role: recent adjacent contributor and committer; confidence: medium; commits: d46c48954ab8; files: src/logging/diagnostic-run-activity.ts, src/logging/diagnostic.test.ts)
  • keshavbotagent: Authored the merged commit that made Codex native tools visible to diagnostics, creating the marker class involved in the stale tool-call report. (role: introduced adjacent native-tool diagnostics; confidence: medium; commits: 8cc148b7d088; files: extensions/codex/src/app-server/event-projector.ts, src/logging/diagnostic-run-activity.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-06-21T19:00:34.365Z sha 83c3212 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. labels Jun 5, 2026
@openclaw-barnacle openclaw-barnacle Bot added the extensions: diagnostics-otel Extension: diagnostics-otel label Jun 5, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 5, 2026
@openclaw-barnacle openclaw-barnacle Bot added extensions: diagnostics-prometheus proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 5, 2026
@849261680

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

The previous review evaluated an earlier head (7 files) and flagged a "no-dedicated-metric OTel design". That has since been addressed at the current head a02a88e8172b:

  • session.activity.evicted is now exported as a dedicated counter in both exporters — openclaw.session.activity.evicted (diagnostics-otel) and openclaw_session_activity_evicted_total (diagnostics-prometheus), incremented by total evicted markers with a reason attribute and no raw session ids — with exporter tests and metric docs on both pages.
  • The PR body now includes redacted before/after runtime diagnostic-logger output from the real diagnostic heartbeat + recoverStuckDiagnosticSession runtime: the blocked_tool_call line is present before the patch and gone after (the orphaned bash marker is evicted at reply completion, follow-up turn is a recoverable release_lane). The Real behavior proof CI gate is green (proof: supplied).

A fully live external Slack/systemd gateway repro of the timing-sensitive hung-native-tool condition is noted under "What was not tested"; the runtime path is exercised deterministically in-process.

@clawsweeper

clawsweeper Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 5, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 6, 2026
849261680 added 4 commits June 6, 2026 13:47
…n end

Stale native tool_call activity could survive a clean run teardown and
re-block later turns on the same sessionKey as blocked_tool_call.

markDiagnosticEmbeddedRunEnded({ clearRunActivity: false }) (used by the
reply-run wrapper so it does not clobber a still-draining inner run) removed
only the embedded-run marker and left tool/model markers untouched. When a
native tool never emitted its completion event, that marker leaked: a future
turn on the same session key saw activeWorkKind=tool_call with a very large
age and was misclassified as blocked_tool_call, repeatedly triggering stalled
diagnostics and even aborting fresh work.

Now, once no embedded-run owner remains for the session, leftover tool/model
markers are evicted (they can never report completion). A still-active inner
run keeps its markers. Eviction emits a structured session.activity.evicted
event (reason=orphaned_no_owner, evictedTools/evictedModelCalls) so operators
can distinguish recovered stale state from a real active tool.

Recovery-driven cleanup is unchanged and remains observable via
session.recovery.completed.

Closes openclaw#87310
The OTel exporter event switch is exhaustiveness-checked (oxlint). Add an
explicit no-op case for the new session.activity.evicted event, matching the
session.long_running / session.stalled precedent (eviction is captured by
stability records and recovery counters; no dedicated OTel metric).
…unter

Address the docs/exporter mismatch: the OpenTelemetry page advertised the new
session.activity.evicted event, but the exporters did not emit it. Wire both
exporters to emit a counter incremented by the total evicted markers
(openclaw.session.activity.evicted / openclaw_session_activity_evicted_total,
attr reason), document the metrics on the OTel and Prometheus pages, and add
exporter tests. No raw session ids/keys are exported.
…down

Address review: tool/model start events are async-queued, so a start emitted
before an owner-less reply-run teardown could drain after the eviction and
re-arm an owner-less marker, restoring the blocked_tool_call leak. The
owner-less eviction now records a start-event sequence cutoff for the session
owner refs at the current event sequence (the same mechanism stuck-session
recovery uses via recoveredOwnerStartEventCutoffs), so a late-draining start is
ignored instead of recreating the marker. Adds an async-drain regression test.
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 6, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 6, 2026
@849261680
849261680 force-pushed the fix/87310-diagnostic-stale-tool-activity branch from 3563d76 to 83c3212 Compare June 6, 2026 05:56
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 6, 2026
@849261680

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Addressed both findings from the last review and rebased onto current main (head 83c3212755, now mergeable):

  • [P1] late queued starts: owner-less reply-run teardown now records a start-event sequence cutoff for the session owner refs at the current event sequence (the same recoveredOwnerStartEventCutoffs mechanism stuck-session recovery uses), so a tool.execution.started / model.call.started emitted before teardown but drained afterward is ignored instead of re-arming an owner-less marker. Added an async-drain regression test (waitForDiagnosticEventsDrained) that fails without the cutoff.
  • [P2] dirty branch: rebased onto current upstream main; PR is mergeable again.

All affected suites + tsgo core/extensions/test configs + oxlint/oxfmt are green on the rebased base.

@clawsweeper

clawsweeper Bot commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 6, 2026
@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jun 19, 2026
zhangqueping added a commit to zhangqueping/openclaw that referenced this pull request Jun 25, 2026
…session visibility

Introduce hasActiveModelCall to DiagnosticSessionActivitySnapshot,
derived from the existing activeModelCalls tracking, so that model
calls in CLI harness sessions are surfaced to diagnostics.

Use hasActiveModelCall for classification (line 83 of
diagnostic-session-attention.ts) so CLI sessions with hung model
calls are visible — previously only embedded-run sessions were
classified because the gate required hasActiveEmbeddedRun.

Keep hasActiveEmbeddedRun for the active-abort recovery gate
(diagnostic.ts line 547) — this preserves the live-owner boundary:
only sessions with an active embedded run can be force-recovered.
CLI session recovery will follow after ownerless-marker cleanup
(openclaw#90750) provides a reliable liveness signal for non-embedded
model calls.

No threshold, warn-multiplier, or policy defaults are changed.

Ref: openclaw#94650
zhangqueping added a commit to zhangqueping/openclaw that referenced this pull request Jun 28, 2026
…tiveEmbeddedRun

Option C: hasActiveModelCall is only used when hasActiveEmbeddedRun is
also true.  This preserves the ownerless stale-marker invariant —
non-embedded model calls (CLI harness, Codex-backed providers) with
hasActiveModelCall but without hasActiveEmbeddedRun stay on the
stalled_agent_run path and are NOT force-recovered.

Two layers updated:
1. classifySessionAttention (line 83): requires hasActiveEmbeddedRun
   + hasActiveModelCall, so ownerless model calls fall through to
   the generic stalled classification.
2. isStalledModelCallRecoveryEligible (line 555): requires
   hasActiveEmbeddedRun + hasActiveModelCall, so CLI sessions are
   not force-recovered.

Both gates will be relaxed for non-embedded sessions once openclaw#90750
provides reliable ownerless-marker cleanup.

Also fixes the snapshot flag comment in diagnostic-run-activity.ts
to match the classifier-only contract.
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 21, 2026
@steipete steipete self-assigned this Jul 21, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Jul 22, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This assigned pull request has been automatically marked as stale after being open for 27 days.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 22, 2026
@clawsweeper

clawsweeper Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(diagnostics): evict orphaned tool/model activity on owner-less run end This is item 1/1 in the current shard. Shard 11/22.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation extensions: diagnostics-otel Extension: diagnostics-otel extensions: diagnostics-prometheus gateway Gateway runtime merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: M stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale diagnostic tool_call activity can survive recovery/reset and re-block sessions as blocked_tool_call

2 participants