Skip to content

fix(agents): make run liveness evidence-based#101985

Closed
obviyus wants to merge 4 commits into
fix/reply-run-terminal-releasefrom
fix/truthful-run-liveness
Closed

fix(agents): make run liveness evidence-based#101985
obviyus wants to merge 4 commits into
fix/reply-run-terminal-releasefrom
fix/truthful-run-liveness

Conversation

@obviyus

@obviyus obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Track B (sub-tracks B1/B4/B5) of #101863 — the "truthful liveness" half of the run-lifecycle reliability design. Stacked on #101910 (Track A) and uses its primitives (expireStaleReplyOperation, run_stalled, activity stamps).

Three verified defects:

  1. claude-cli backend: active-tool heartbeat masks wedged subprocesses, defeats stuck-session detection #96168 — fake CLI liveness. startClaudeLiveActiveToolHeartbeat emitted cli_live:tool_running on a 10s setInterval while a tool was merely marked active, resetting the shared lastProgressAt clock with zero evidence the CLI child was alive. The blocked_tool_call classifier — which correctly requires BOTH tool age AND progress age to be stale — could structurally never fire for a wedged CLI tool, and neither could any stuck-recovery abort gate.
  2. Stuck-session recovery aborts long-but-active agent runs at warnMs×3 (~6min) with misleading "Reply operation aborted by user" reason #88870 — watchdog kills blamed on the user + quiet tools killed. Stuck-recovery aborts of reply-owned runs routed through operation.abortByUser(), so watchdog kills surfaced as aborted_by_user. And any truthful-evidence fix would expose quiet-but-alive tools to the generic ~6-min abort threshold. Stuck-session liveness monitor misclassifies healthy processing runs as stale during event-loop stalls #101670 — staleness ages are raw Date.now() deltas, so an event-loop stall inflates every age at the next heartbeat tick and trips spurious recovery.
  3. Live E2E gap from fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes #101910 — steer capture accepts messages into a run based only on streaming/stopped state; a wedged run swallows the very human message that would trigger stale takeover at admission.

Why This Change Was Made

  • B1: deleted the timer heartbeat entirely (mechanism, constant, lifecycle wiring). Real frames — tool start/result, terminal result, any stdout frame — were already stamping progress; they are now the only stamps. To keep truthful evidence from re-exposing quiet tools, blocked_tool_call recovery eligibility gains a floor: max(stuckSessionAbortMs, BLOCKED_TOOL_CALL_ABORT_FLOOR_MS = 15 min). Early warnings (naming the tool) keep current thresholds; the CLI byte-level no-output watchdog (evidence-based, ≤600s for truly-silent children) is untouched.
  • B4: stuck-recovery expiry of reply-owned runs now routes through expireStaleReplyOperation(…, "stuck_recovery") → terminal result {failed, run_stalled} instead of aborted_by_user. The diagnostic heartbeat records its own tick time; a tick delayed >3× the interval defers all recovery decisions to the next on-time tick (a late tick means the process stalled, not the sessions).
  • B5: both steer-capture gates now refuse evidence-stale runs — queueReplyRunMessage when the operation's lastActivityAtMs exceeds the stale-takeover window, prepareEmbeddedAgentQueueMessage when diagnostic lastProgressAgeMs exceeds the paired constant (new closed failure reason stale_run; consumers fall through like no_active_run). Refused messages take the normal followup-admission path where Track A's takeover can reclaim. Draining already-captured entries from dead handles stays Track C (durability).

User Impact

  • A wedged CLI tool is now detectable: stuck-session classification and recovery work as designed instead of being permanently disarmed by a fake heartbeat.
  • Legit quiet long tools are safer than before: recovery aborts need 15 min of zero evidence (previously ~6 min wall-clock whenever the heuristics did fire).
  • Watchdog recoveries stop reading as "aborted by user".
  • A gateway event-loop stall no longer mass-aborts healthy sessions on the next heartbeat tick.
  • Messages sent to a dead-looking session reach admission (and Track A reclaim) instead of being swallowed into the wedged run's queue.

Evidence

  • The defeat mechanism, thresholds, and every stamp site were mapped from source before design (see the Track B design comment on Design: run lifecycle reliability — bounded liveness, abort⇒release, crash resume, completion drive #101863 for the full file:line mechanism maps).
  • Tests: wedged-tool staleness → classification → floor-gated recovery eligibility; real frames keep runs live; recovery expiry yields run_stalled (user aborts still aborted_by_user); delayed-tick guard (fake timers); both steer gates (stale refused → fall-through, fresh accepted, no-snapshot skip); stale_run consumer fall-through.
  • Validation: focused suites + import sweep over every test touching the changed modules + tsgo core/core-test lanes green on Testbox; re-run green after review edits.
  • Net LOC: a whole liveness mechanism deleted; source diff dominated by the two guards and gates.
  • Live behavior proof posted as a PR comment: fake-heartbeat string absent from dist; reason=stale_run steer refusal on a 16-min-stale lane (steer-accepted unconditionally on main); channel-turn takeover 34 ms after a rescue message on a 10-min wedge with the turn re-running on a fresh model call; recovery-attribution race found live and fixed (expire-ordering on the fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes #101910 base + live-handle recovery leg here), both with regression tests.

Part of #101863 (Tracks B1/B4/B5). Fixes #96168; addresses the abort-attribution and quiet-tool halves of #88870 and the heartbeat half of #101670. Stacked on #101910 — do not merge before it.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: M maintainer Maintainer-authored PR labels Jul 8, 2026
@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 8:02 AM ET / 12:02 UTC.

Summary
The branch removes Claude live active-tool timer heartbeats, adds evidence-stale stale_run refusal and stuck-recovery attribution paths with a quiet-tool floor, and updates agent/reply/diagnostic regression tests.

PR surface: Source +87, Tests +395. Total +482 across 14 files.

Reproducibility: yes. Current main source still emits synthetic cli_live:tool_running progress from an active-tool interval, while the PR head removes that timer and the PR discussion provides live output for stale refusal, takeover, and recovery attribution.

Review metrics: 1 noteworthy metric.

  • Queue failure reasons: 1 added (stale_run). The new closed queue outcome affects whether active-run handoffs are steered, retried through admission, or delivered directly before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #96168
Summary: This PR is a strong candidate fix for the canonical Claude CLI fake-liveness issue and a partial Track B implementation for the broader run-lifecycle reliability cluster.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Risk before merge

Maintainer options:

  1. Fix And Land Through The Integration Stack (recommended)
    Resolve the open integration blockers, re-review the exact fix(agents): run-lifecycle reliability — bounded release, evidence-based liveness, watchdog semantics #102160 head, and let that atomic PR land this Track B work with its coupled dependencies.
  2. Accept Direct Track B Runtime Risk
    Maintainers can land this branch directly after Track A if they explicitly accept the standalone stale-run and quiet-tool-floor behavior change.
  3. Keep As Review Record Only
    Pause this PR as a per-track record and close it only after the integration branch lands or maintainers reject the runtime policy.

Next step before merge

  • [P2] Human maintainer review is required because this protected draft changes session-state and message-delivery defaults and is now planned to land through the open integration PR, not because of a narrow repairable defect on this branch.

Maintainer decision needed

Security
Cleared: The diff changes runtime liveness logic and tests only; no dependency, workflow, secret, install, package, or supply-chain surface change was found.

Review details

Best possible solution:

Keep this per-track PR open as the Track B review record, then land or close it through #102160 only after the integration branch is fixed, re-reviewed, and accepted by maintainers.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main source still emits synthetic cli_live:tool_running progress from an active-tool interval, while the PR head removes that timer and the PR discussion provides live output for stale refusal, takeover, and recovery attribution.

Is this the best way to solve the issue?

Yes, with a merge-sequencing caveat. Removing synthetic liveness and routing stale runs through the reply-run and embedded-run owner paths is the maintainable fix for this track, but it should land through the accepted Track A or integration path rather than independently.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 83ebbcb3acf7.

Label changes

Label justifications:

  • P1: The PR targets active agent liveness and message handoff failures where wedged runs can mask recovery or swallow rescue messages.
  • merge-risk: 🚨 compatibility: The branch changes default stale-run and quiet-tool behavior that existing long-running setups may observe during upgrade.
  • merge-risk: 🚨 session-state: The patch changes when reply-owned and embedded runs are considered evidence-stale and eligible for takeover or recovery.
  • merge-risk: 🚨 message-delivery: The new stale_run refusal path changes whether active-run messages are steered, retried through admission, or delivered directly.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR discussion includes live gateway output showing fake-heartbeat absence, stale steer refusal, admission takeover, and corrected run_stalled attribution after the fix.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR discussion includes live gateway output showing fake-heartbeat absence, stale steer refusal, admission takeover, and corrected run_stalled attribution after the fix.
Evidence reviewed

PR surface:

Source +87, Tests +395. Total +482 across 14 files.

View PR surface stats
Area Files Added Removed Net
Source 8 133 46 +87
Tests 6 400 5 +395
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 14 533 51 +482

What I checked:

Likely related people:

  • obviyus: They authored this Track B branch, the linked run-lifecycle design, the Track A base, and the open atomic integration PR that now owns the proposed landing path. (role: lifecycle-stack proposer and integration owner; confidence: high; commits: 348c468d335b, aa874c4e538a, 99ab256c1214; files: src/auto-reply/reply/reply-run-registry.ts, src/agents/embedded-agent-runner/runs.ts, src/logging/diagnostic.ts)
  • joshavant: Merged PR Fix Claude live tool progress for watchdog recovery #87546 added the Claude live active-tool heartbeat behavior that this PR replaces with evidence-based liveness plus a floor for valid quiet tools. (role: introduced shipped heartbeat behavior; confidence: high; commits: 4c3a0292ff9e, 980a643df26b, 5b73eeb9cdcf; files: src/agents/cli-runner/claude-live-session.ts, src/agents/cli-runner.spawn.test.ts)
  • vincentkoc: Current-main blame and recent commits around the central Claude live/reply-run/diagnostic files map to Vincent Koc commits, making this a useful routing signal for current runtime ownership. (role: recent area contributor and carry-forward signal; confidence: medium; commits: 3bbc2732c0f7, e085fa1a3ffd; files: src/agents/cli-runner/claude-live-session.ts, src/auto-reply/reply/reply-run-registry.ts, src/logging/diagnostic.ts)
  • harjothkhara: They authored open fix: stop Claude tool heartbeats from hiding stuck sessions #96198 for the same Claude CLI heartbeat masking issue; that PR is useful context for the narrower fix this branch supersedes or folds into the broader Track B shape. (role: adjacent candidate-fix author; confidence: medium; commits: 93054bc5a8d7; files: src/logging/diagnostic-run-activity.ts, src/logging/diagnostic.test.ts, src/agents/cli-runner.spawn.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (12 earlier review cycles; latest 8 shown)
  • reviewed 2026-07-08T08:49:31.924Z sha 1eb76b6 :: found issues before merge. :: [P3] Remove the branch-only Track B brief
  • reviewed 2026-07-08T10:06:52.905Z sha 4ff1246 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T10:17:59.962Z sha 0f817a6 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T10:33:33.633Z sha a98b238 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T11:11:19.261Z sha 5b4ae6c :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T11:19:27.129Z sha 5b4ae6c :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T11:31:17.428Z sha a075205 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T11:40:40.318Z sha a075205 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 8, 2026
@obviyus

obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

P1 fixed in f880b20: stale_run steer refusals from the active-wake path now report the requester as not-active, so non-completion announce dispatch falls through to direct delivery instead of dropping the handoff (subagent-announce-delivery.ts, with a regression test asserting the steer-primary → direct-primary phase sequence for a stale-but-registered requester).

Live behavior proof is running now on this exact branch (same harness as the #101910 and #101986 E2E comments): wedged run → 10-min stale window → channel msg refused by steer capture (reason=stale_run) → admission takeover → new turn; plus a default-config stuck-recovery abort asserting run_stalled attribution with no aborted_by_user. Evidence will be posted here and the PR body updated when it completes (~40 min).

On sequencing: agreed and stated in the PR body — this is stacked on #101910 (its base branch) and must not merge independently.

@obviyus

obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Live behavior proof (requested by review) — and one more bug it caught, now fixed

Same harness as the #101910 / #101986 E2E comments: local gateway from this branch's dist, black-hole model endpoint holding a genuine wedge open (explicit provider timeoutSeconds: 36000), qa-channel synthetic transport for real channel-inbound turns.

Artifact check — fake heartbeat gone

grep -r 'cli_live:tool_running' dist | wc -l0.

Proof 1 — stale steer refusal + takeover: PASS (flips the exact failure from the #101910 E2E)

  • Steer refusal: on a 16-minute-stale lane, an injected message was refused by steer capture — queue message failed: ... reason=stale_run + rejected steering injection: queue_message_failed reason=stale_run. On main this exact probe is steer-accepted unconditionally (the "wedged run swallows its own rescue message" hole).
  • Takeover: a channel DM injected 10m35s into the wedge produced reply run stale takeover: forced release sessionKey=agent:main:qa-channel:... reason=no_activity 34 ms after injection; the wedged fetch aborted (elapsedMs=635096) and the rescue message's turn started ~2.4s later with a fresh model request.
  • Surface note: channel visible turns reclaim at admission, before the steer layer runs, so refusal and takeover appear on different probe surfaces rather than one chain — both behaviors independently proven.
  • Harness correction worth recording: qa-channel serializes inbound per account, which is what actually parked msg2 in the fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes #101910 E2E's "Test B2" — the steer-capture interpretation there was partially the harness, not only the runtime. The refusal path is now proven directly regardless.

Proof 2 — recovery attribution: FAILED live, root-caused, fixed

Default stuck recovery aborted the wedge at 374s and freed the lane, but the operation still terminated as aborted_by_user, not run_stalled. Two-part root cause, both fixed:

  1. Expire ordering race (Track A code): expireStaleReplyOperation cancelled the backend before setting its result; the cancel synchronously re-enters the run loop's abort handler → abortByUser() stamps aborted_by_user → expire's if (!result) skips run_stalled. Fixed on the base branch (fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes #101910, 1ca9288cf2f): result is stamped before backend cancel, with a reentrancy regression test (backend whose cancel calls abortByUser() must still end run_stalled).
  2. Live-handle recovery leg (this branch): abortAndDrainEmbeddedAgentRun only routed through the expiry when no embedded handle existed — the handle-present case (the common field shape) went down the plain abort path. Fixed in 1eb76b6c090: stuck recovery expires the reply operation first regardless of handle state, then aborts/drains the handle; new test covers the live-handle case with a re-entering abort handler.

Also in this push: the review's P1 (stale_run → subagent announce dropped) fixed with a phase-sequence regression test — a stale requester now falls through to direct delivery.

Validation after fixes: registry/admission/runs/recovery/announce suites green, tsgo core + core-test lanes exit 0, branch rebased onto the updated #101910 base. Full timestamped evidence in the harness report (request ledger, log lines); happy to attach the raw report if wanted.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 8, 2026
@obviyus

obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@obviyus
obviyus force-pushed the fix/truthful-run-liveness branch from 1eb76b6 to b14ff32 Compare July 8, 2026 09:45
@obviyus
obviyus force-pushed the fix/reply-run-terminal-release branch from 36536d8 to ef2678f Compare July 8, 2026 09:46
@obviyus
obviyus force-pushed the fix/truthful-run-liveness branch from b14ff32 to 4ff1246 Compare July 8, 2026 09:47
@clawsweeper clawsweeper Bot removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jul 8, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 8, 2026
@obviyus
obviyus force-pushed the fix/truthful-run-liveness branch from 4ff1246 to 0f817a6 Compare July 8, 2026 10:12
@obviyus
obviyus force-pushed the fix/reply-run-terminal-release branch from ef2678f to c40447e Compare July 8, 2026 10:22
@obviyus
obviyus force-pushed the fix/truthful-run-liveness branch from 0f817a6 to a98b238 Compare July 8, 2026 10:23
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 8, 2026
@obviyus
obviyus force-pushed the fix/truthful-run-liveness branch 2 times, most recently from 5b4ae6c to a075205 Compare July 8, 2026 11:24
@obviyus
obviyus force-pushed the fix/reply-run-terminal-release branch from ae9081d to aa874c4 Compare July 8, 2026 11:45
@obviyus

obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Landing plan update: this track lands atomically with its siblings via the integration PR #102160 (all commits preserved, rebase merge) — sequential landing would put untested intermediate states on main (e.g. the stale takeover without the quiet-tool floor). This PR stays open as the per-track review record and closes when #102160 lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant