fix(agents): run-lifecycle reliability β bounded release, evidence-based liveness, watchdog semantics#102160
Conversation
|
ClawSweeper review: did not complete due to Codex infrastructure failure. Reviewed July 9, 2026, 3:43 AM ET / 07:43 UTC. Summary PR surface: Source +354, Tests +691, Docs 0. Total +1045 across 46 files. Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path. Review metrics: none identified. Stored data model Merge readiness This is a ClawSweeper/Codex infrastructure failure, not a PR readiness or patch-quality verdict. Risk before merge
Maintainer options:
Next step before merge
Review detailsBest possible solution: Retry the Codex review after fixing the execution failure. Do we have a high-confidence way to reproduce the issue? Unclear. The review failed before ClawSweeper could establish a reproduction path. Is this the best way to solve the issue? Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction. AGENTS.md: unclear because the file could not be read completely. Codex review notes: model internal, reasoning high; reviewed against f0cc5e500f66. Label changesLabel changes:
Evidence reviewedPR surface: Source +354, Tests +691, Docs 0. Total +1045 across 46 files. View PR surface stats
What I checked:
Likely related people:
How this review workflow works
Review history (6 earlier review cycles)
|
99ab256 to
ce3339d
Compare
|
Both code findings fixed and rebuilt into this branch (
On the maintainer-acceptance item: the upgrade-behavior decisions (10-minute takeover default, |
|
@clawsweeper autoreview |
|
π¦π I queued a lightweight read-only assist pass. It will post a separate answer comment and will not edit the durable ClawSweeper review comment or trigger close, merge, repair, label, or branch changes. Request: autoreview |
|
ClawSweeper assist: Existing autoreview has already run on #102160 at head Evidence:
Suggested next action: For a fresh full correctness pass on the current PR, use Source: #102160 (comment) |
|
Soak finding (HamVerBot, 2026-07-08 17:21 UTC): a live Claude CLI turn was killed with Analysis: not a regression from this branch β the live-session no-output watchdog resets only on real subprocess stdout/stderr ( Fix in f493f8b: unified the two watchdog arming sites into Out of scope sibling: the generic (non-live) CLI runner delegates its no-output kill to the process supervisor ( Validation: |
Partial prerequisite backport from 969efda for PR openclaw#102160. (cherry picked from commit 868027e)
Backport the applicable run-lifecycle stack from #102160 onto the 2026.6.33 extended-stable line based on 2026.6.11. This includes bounded reply-operation recovery, finite LLM silence guards under unlimited run budgets, Claude CLI quiet-tool handling, and Codex app-server suspect-client retirement. Preserve the 6.33 diagnostics steering adaptation and Codex 0.139 fixture baseline.
What Problem This Solves
The run-lifecycle reliability stack from #101863: agent turns that hang forever, sessions that die holding their lane until a gateway restart, watchdog kills blamed on the user, and rescue messages swallowed by the very runs they should rescue β the top-reported reliability class (canonical open issues: #92270, #97538, #100872, #89766, #96168, #89742, #85826, #88870, #101670, and the steer-capture gap proven live in #101910's E2E).
This PR lands the four verified track branches atomically as one linear 15-commit stack (rebase merge, all commits preserved):
run_stalledtimeoutSeconds: 0, creation deadlineAtomic landing matters: the tracks protect each other (e.g. A's 10-minute stale takeover is only safe for quiet embedded tools with B1's blocked-tool floor). Sequential landing would put intermediate states on
mainthat no gate ever tested.Why This Change Was Made
Full design and per-track rationale: #101863 (invariants + mechanism maps + two comparative studies). Per-PR bodies carry the detailed why; one paragraph each:
run_stalled.aborted_by_user. A delayed diagnostic tick (event-loop stall) defers recovery instead of mass-aborting. Steer capture refuses evidence-dead runs so rescue messages reach admission; quiet tool phases keep a 15-minute floor across takeover and steer gates.timeoutSeconds: 0means unlimited (schema-accepted) instead of a secret 1-second budget; self-hosted providers get 300s mid-stream tolerance ([Bug]: Agent stall detector hard-coded 120s threshold kills legitimate long model calls on local vLLMΒ #85826) across default, agent-budget, and cron paths; stream creation is bounded even where local idle is opted out.openai/codexsource in fix(codex): ungate app-server terminal liveness and fan out poisoned-client retirementΒ #102097); it now fires on true 30-minute silence, and terminal-kind kills retire the shared client for all co-leased sessions into a retryable path β scoped so 60s per-turn hiccups keep graceful semantics.User Impact
/stopon a wedged turn frees the session within 60 seconds; a new message reclaims a silently dead run instead of queueing forever.run_stalled), not "aborted by user".timeoutSeconds: 0) is safe by default: unlimited budget, finite stall detection.Evidence
reason=stale_run) and channel takeover 34ms after a rescue message on a 10-minute wedge; the previously-lethaltimeoutSeconds: 0config now fails loudly at the ~300s watchdog; the E2E also found three bugs (config schema gate, headers-never-arrive wedge, attribution race) that are fixed in this stack.run_stalledattribution, 10-minute takeover default, B3 semantics, B2 terminal-only fan-out). The two removed-then-restored SDK exports ship as@deprecatedper SDK policy.Closes #101910, closes #101985, closes #101986, closes #102097 (superseded by this atomic landing; they remain the per-track review records). Fixes #96168; fixes #85826. Part of #101863 β the remaining linked issues (#92270, #97538, #100872, #89766, #89742 et al.) will be verified against live reports and closed with proof after this lands.