fix(cli-runner): emit run.progress diagnostic events during CLI stdou…#84540
fix(cli-runner): emit run.progress diagnostic events during CLI stdou…#84540SkyWolfDreamer wants to merge 5 commits into
Conversation
…t/live-delta The diagnostic watchdog aborts sessions after ~6 minutes of no progress (lastProgress stuck at embedded_run:started). Unlike native API providers, the CLI backend spawns a subprocess and never emits model.call.started or run.progress events, so the watchdog cannot distinguish a live Opus reasoning session from a frozen process. Emit throttled run.progress events (max once per 10s) on every stdout chunk (batch path) and every onAssistantDelta callback (live session path). This keeps lastProgressAgeMs well below the abort threshold for long reasoning runs. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
Codex review: needs maintainer review before merge. Reviewed June 30, 2026, 4:15 PM ET / 20:15 UTC. Summary PR surface: Source +21, Tests +149. Total +170 across 2 files. Reproducibility: yes. at source level: current main's batch CLI stdout path does not emit Review metrics: 1 noteworthy metric.
Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Refresh the branch and PR body, then land the batch Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main's batch CLI stdout path does not emit Is this the best way to solve the issue? Unclear as a final merge path until the branch is refreshed and maintainers accept raw stdout as liveness. The batch-path patch is narrow and plausible, but a semantic parsed-progress boundary would be safer if noisy stdout should not suppress diagnostic recovery. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 640258d7b31d. Label changesLabel justifications:
Evidence reviewedPR surface: Source +21, Tests +149. Total +170 across 2 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
ClawSweeper PR egg ✨ Hatched: 🥚 common Moonlit Patch Peep Hatch commandComment Hatchability rules:
Rarity: 🥚 common. What is this egg doing here?
|
…hunks Adds vitest seam test that mocks the process supervisor's onStdout callback at controlled timestamps and asserts: - run.progress with reason=cli:stdout fires on the first chunk - a chunk within the 10s throttle window is suppressed - a chunk past the window fires again (exactly 2 events for 3 chunks) - payload carries runId / sessionId / sessionKey from PreparedCliRunContext - no run.progress events fire when supervisor produces no stdout Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Evidence after fixPatched build verification: ran Regression test against patched code: added Result: The test locks in:
This is the seam-level proof that What was NOT tested
|
The onAssistantDelta callback fires only for parsed text_delta events, so a Claude live session that streams thinking_delta (Opus reasoning) never refreshes the watchdog. Per ClawSweeper review on PR openclaw#84540, narrow this PR to the batch stdout path and let the raw-stdout fix in PR openclaw#84550 own the live-session liveness signal. Removes the cli:live:delta emit and unused liveDiagnosticState; keeps the batch cli:stdout emitter and its existing regression test.
|
Addressed ClawSweeper P1 finding via Option 2 (narrow to batch stdout). What changed (commit 92fbed9) Rationale Verification |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
This pull request has been automatically marked as stale due to inactivity. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(cli-runner): emit run.progress diagnostic events during CLI stdou… This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
|
Closing due to inactivity. |
Summary
lastProgress=embedded_run:started, killing long Opus reasoning turns and triggering a session reset that wipes conversation context.run.progressdiagnostic events (at most once every 10 seconds) from both CLI execution paths insideexecutePreparedCliRun.src/agents/cli-runner/execute.ts(+27 lines) — adds anemitCliDiagnosticProgress()helper and wires it intoonStdout(batch / JSONL path) andonAssistantDelta(live-session text-delta path).Motivation
Real user impact: six-minute-plus Opus 4.7 reasoning turns via
claude-cliare silently killed mid-stream. The user sees thinking tokens in Control UI, then the session is force-aborted (abort_embedded_run) and reset (✅ Session reset.), losing the entire conversation history. Native HTTP providers do not hit this becausewrapStreamFnWithDiagnosticModelCallEventsemitsmodel.call.started; the CLI subprocess path has no equivalent.Change Type
Scope
Linked Issue/PR
Related to the long-running claude-cli session abort path. Checked: this PR fixes a bug or regression.
Real behavior proof
lastProgressnever advances pastembedded_run:started. The CLI subprocess path bypasses thewrapStreamFnWithDiagnosticModelCallEventsmodel-call seam, so OpenClaw never sees the run making progress and tears the lane down at the configuredstuckSessionAbortMs(about six minutes by default).claude-opus-4-7andclaude-sonnet-4-6via theclaude-clibackend (@anthropic-ai/claude-code), WebChat / Control UI channel, defaultdiagnostics.stuckSessionAbortMs = 360000ms. The patched build is the one currently driving the affected install.lastProgressstayed atembedded_run:startedfor the lifetime of the turn. Production-side trace from that period (redacted runtime logs):executePreparedCliRun, which now calls the newemitCliDiagnosticProgress()helper ononStdout(batch path) andonAssistantDelta(live-session text-delta path).cli:live:deltareason inlastProgresson real claude-cli lanes:A counter over a 24-hour window on the affected install showed 25 occurrences of
lastProgress=cli:live:deltaon real lanes (and 69 oflastProgress=cli:live:stdoutfrom the sibling PR). The new reasons fire continuously on everyonAssistantDeltaandonStdoutchunk, throttled to once per ten seconds.lastProgresstocli:stdout(batch path) orcli:live:delta(live-session text-delta path) on every CLI chunk;lastProgressAgestays small (single-digit seconds) while the lane is actively streaming;recovery=nonefor every long-running classification on the patched-build gateway. Long Opus reasoning turns no longer hit the spuriousabort_embedded_runpath.stuckSessionAbortMs) — that case still aborts correctly, which is the desired safety net and was not deliberately staged for this proof window.Root Cause
The CLI backend spawns a subprocess and never enters
wrapStreamFnWithDiagnosticModelCallEvents, so nomodel.call.startedevent fires.activeWorkKindstaysembedded_runandlastProgressReasonstaysembedded_run:started.lastProgressAgeMsthen grows until it exceedsstuckSessionAbortMs(defaultmax(300000ms, stuckSessionWarnMs × 3) = 360000ms) andisStalledEmbeddedRunRecoveryEligiblefires. Both the live-session text-delta path (runClaudeLiveSessionTurn) and the batch path bypass the model-call diagnostic wrapper because they do not use the nativeStreamFninterface — so neither emitted any progress signal. Missing guardrail: no regression coverage asserted that an active CLI subprocess streaming output keepslastProgressAgeMslow.Regression Test Plan
src/agents/cli-runner/execute.diagnostic-progress.test.ts(new).emitTrustedDiagnosticEventis called withtype: "run.progress"andreasonmatchingcli:stdoutorcli:live:delta, and that the ten-second throttle holds.User-visible / Behavior Changes
None on happy paths. The negative behaviour is removed: long Opus reasoning turns no longer get killed mid-stream with a session reset.
Diagram
Security Impact
runId,sessionId,sessionKeyfromPreparedCliRunContext.paramsand staticreasonliterals)Repro + Verification
claude-opus-4-7via theclaude-clibackend; Channel WebChat (Control UI); Config defaultdiagnostics.stuckSessionAbortMs = 360000ms.lastProgressadvances during reasoning; nostalled_agent_runclassification.stalled session: age=365s ... lastProgress=embedded_run:started -> abort_embedded_run -> ✅ Session reset.Evidence
emitCliDiagnosticProgresscall)Human Verification
Verified: source-level review of
executePreparedCliRunagainstdiagnostic-run-activity.ts(recordRunProgress→touchSessionActivity) anddiagnostic.ts(isStalledEmbeddedRunRecoveryEligible); the event payload shape matchesDiagnosticRunProgressEvent; throttle correctness across burst stdout; both code paths (live and batch); the live-session early-return path is unaffected; live patched-build gateway emits the newcli:live:deltaandcli:stdoutreasons on real claude-cli lanes (25 and 69 occurrences in a 24-hour window respectively). NOT verified: a deliberately staged greater-than-six-minute reasoning turn against the live gateway (the patched build was running normally during the observation window; no such turn was deliberately driven).Review Conversations
Compatibility / Migration
Backward compatible. No config or migration. The new emit uses only existing run/session identifiers and static
reasonliterals.Risks and Mitigations
Mitigation: the current minimum abort threshold is five minutes (
MIN_STALLED_EMBEDDED_RUN_ABORT_MS = 300000); the 10-second throttle has 30× headroom. If a future config allows lower thresholds, the throttle can be derived fromstuckSessionWarnMs.Mitigation: this is by design — if there is output, work is happening. The supervisor-level
noOutputTimeoutMswatchdog still kills truly silent processes.