Skip to content

fix(agents): run-lifecycle reliability β€” bounded release, evidence-based liveness, watchdog semantics#102160

Merged
obviyus merged 20 commits into
mainfrom
integration/run-lifecycle
Jul 9, 2026
Merged

fix(agents): run-lifecycle reliability β€” bounded release, evidence-based liveness, watchdog semantics#102160
obviyus merged 20 commits into
mainfrom
integration/run-lifecycle

Conversation

@obviyus

@obviyus obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

The run-lifecycle reliability stack from #101863: agent turns that hang forever, sessions that die holding their lane until a gateway restart, watchdog kills blamed on the user, and rescue messages swallowed by the very runs they should rescue β€” the top-reported reliability class (canonical open issues: #92270, #97538, #100872, #89766, #96168, #89742, #85826, #88870, #101670, and the steer-capture gap proven live in #101910's E2E).

This PR lands the four verified track branches atomically as one linear 15-commit stack (rebase merge, all commits preserved):

Track PR (review record) Commits
A β€” terminal settle, stale takeover, run_stalled #101910 5
B1 β€” evidence-based liveness, honest recovery attribution, steer gates #101985 4
B3 β€” idle-watchdog semantics, timeoutSeconds: 0, creation deadline #101986 4
B2 β€” app-server terminal-watch ungating, poisoned-client fan-out #102097 2

Atomic landing matters: the tracks protect each other (e.g. A's 10-minute stale takeover is only safe for quiet embedded tools with B1's blocked-tool floor). Sequential landing would put intermediate states on main that no gate ever tested.

Why This Change Was Made

Full design and per-track rationale: #101863 (invariants + mechanism maps + two comparative studies). Per-PR bodies carry the detailed why; one paragraph each:

  • A: a wedged turn held its session lane forever (abort recorded, release never ran; admission had no staleness takeover). Now: terminal results force-release within 60s regardless of the owner; operations carry real-event activity evidence; visible turns reclaim terminal-unreleased or evidence-dead runs via one primitive with the closed code run_stalled.
  • B1: the CLI timer heartbeat faked liveness (claude-cli backend: active-tool heartbeat masks wedged subprocesses, defeats stuck-session detectionΒ #96168) β€” deleted; only real frames stamp. Recovery kills stop reading as aborted_by_user. A delayed diagnostic tick (event-loop stall) defers recovery instead of mass-aborting. Steer capture refuses evidence-dead runs so rescue messages reach admission; quiet tool phases keep a 15-minute floor across takeover and steer gates.
  • B3: an unlimited run budget silently disabled all stream liveness β€” budget and liveness are now independent; timeoutSeconds: 0 means unlimited (schema-accepted) instead of a secret 1-second budget; self-hosted providers get 300s mid-stream tolerance ([Bug]: Agent stall detector hard-coded 120s threshold kills legitimate long model calls on local vLLMΒ #85826) across default, agent-budget, and cron paths; stream creation is bounded even where local idle is opted out.
  • B2: the app-server terminal watch was gated off exactly when a poisoned client wedged mid-request (protocol proof from sibling openai/codex source in fix(codex): ungate app-server terminal liveness and fan out poisoned-client retirementΒ #102097); it now fires on true 30-minute silence, and terminal-kind kills retire the shared client for all co-leased sessions into a retryable path β€” scoped so 60s per-turn hiccups keep graceful semantics.

User Impact

  • /stop on a wedged turn frees the session within 60 seconds; a new message reclaims a silently dead run instead of queueing forever.
  • Wedged CLI tools become detectable; healthy quiet tools get more protection than before (15-minute floor vs the old ~6-minute wall-clock kills).
  • Watchdog recoveries are labeled honestly (run_stalled), not "aborted by user".
  • The long-task config (timeoutSeconds: 0) is safe by default: unlimited budget, finite stall detection.
  • One dead Codex process no longer wedges every session sharing it.

Evidence

  • Integration verification (this exact tree): three adversarial cross-PR review agents (14 real findings, all fixed in the stack β€” including the unanimous takeover/floor threshold inversion and a suppressed-run staleness bug), changed-lane checks + typecheck lanes green, full test suite green on the stable main base; residual full-suite failures on the newest main base were each reproduced on clean main (browser/anthropic/nvidia/qa-lab/discord β€” upstream, unrelated).
  • Live E2E (harness comments on fix(auto-reply): bound terminal reply-run settle and reclaim stale lanesΒ #101910, fix(agents): keep finite LLM idle watchdog under unlimited run budgetsΒ #101986, fix(agents): make run liveness evidence-basedΒ #101985): wedged-lane abortβ†’release in ~107ms with the next turn admitted; stale steer refusal (reason=stale_run) and channel takeover 34ms after a rescue message on a 10-minute wedge; the previously-lethal timeoutSeconds: 0 config now fails loudly at the ~300s watchdog; the E2E also found three bugs (config schema gate, headers-never-arrive wedge, attribution race) that are fixed in this stack.
  • Reviews: ClawSweeper reviewed all four PRs (and re-reviewed A and B1 post-fix); every P1 is fixed or policy-accepted with the decision recorded on the PR (run_stalled attribution, 10-minute takeover default, B3 semantics, B2 terminal-only fan-out). The two removed-then-restored SDK exports ship as @deprecated per SDK policy.
  • Per-track unit/behavior tests throughout (registry settle/reentrancy/takeover, steer gates, floor interactions, resolver matrices, app-server watch/fan-out/pending-acquire).

Closes #101910, closes #101985, closes #101986, closes #102097 (superseded by this atomic landing; they remain the per-track review records). Fixes #96168; fixes #85826. Part of #101863 β€” the remaining linked issues (#92270, #97538, #100872, #89766, #89742 et al.) will be verified against live reports and closed with proof after this lands.

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper review: did not complete due to Codex infrastructure failure. Reviewed July 9, 2026, 3:43 AM ET / 07:43 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +354, Tests +691, Docs 0. Total +1045 across 46 files.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/agents/cli-runner.spawn.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Not assessed.
Failure reason: timeout.

This is a ClawSweeper/Codex infrastructure failure, not a PR readiness or patch-quality verdict.
Keep any merge decision on the normal maintainer review path until ClawSweeper can complete a fresh review.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against f0cc5e500f66.

Label changes

Label changes:

  • remove proof: sufficient: Current real behavior proof status is not_applicable, not sufficient.
  • remove P1: Current review triage priority is none.
  • remove rating: 🐚 platinum hermit: Current review failed before PR readiness was assessed, so no rating label should remain.
  • remove merge-risk: 🚨 compatibility: Current PR review selected no merge-risk labels.
  • remove merge-risk: 🚨 session-state: Current PR review selected no merge-risk labels.
  • remove merge-risk: 🚨 message-delivery: Current PR review selected no merge-risk labels.
  • remove status: πŸ‘€ ready for maintainer look: Current PR status no longer selects a status label.
Evidence reviewed

PR surface:

Source +354, Tests +691, Docs 0. Total +1045 across 46 files.

View PR surface stats
Area Files Added Removed Net
Source 23 554 200 +354
Tests 22 1405 714 +691
Docs 1 7 7 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 46 1966 921 +1045

What I checked:

  • failure reason: timeout.
  • codex failure detail: Codex review failed for this PR: Codex process timed out after 1199999ms.
  • codex stderr: No stderr captured.
  • codex stdout: bad22b9c31e576392:src/config/schema.labels.ts:751: "agents.list.*.heartbeat.timeoutSeconds": "Heartbeat Timeout (Seconds)",\nc04278466edd8ca7af25001bad22b9c31e576392:src/config/schema.labels.ts:930: "hooks.mappings[].timeoutSeconds": "Hook Mapping Timeout (sec)",\nc04278466edd8ca7af25001bad22b9c31e576392:src/config/schema.labels.ts:1071: "agents.list[].heartbeat.timeoutSeconds": "Agent Heartbeat Timeout (Seconds)",\nc04278466edd8ca7af25001bad22b9c31e576392:src/config/schema.test.ts:513: const tags = deriveTagsForPath("tools.web.fetch.timeoutSeconds");\nc04278466edd8ca7af25001bad22b9c31e576392:src/config/schema.test.ts:529: timeoutSeconds: 15,\nc04278466edd8ca7af25001bad22b9c31e576392:src/config/schema.test.ts:542: timeoutSeconds: 15,\nc04278466edd8ca7af25001bad22b9c31e576392:src/config/schema.test.ts:941: "tools.web.fetch.timeoutSeconds": {},\nc04278466edd8ca7af25001bad22b9c31e576392:.
  • process error code: ETIMEDOUT.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (6 earlier review cycles)
  • reviewed 2026-07-08T11:51:33.966Z sha 99ab256 :: needs changes before merge. :: [P2] Apply self-hosted defaults before explicit run caps | [P2] Honor suspect close for detached shared leases
  • reviewed 2026-07-08T12:00:33.242Z sha 99ab256 :: needs changes before merge. :: [P2] Apply self-hosted defaults before explicit run caps | [P2] Honor suspect close for detached shared leases
  • reviewed 2026-07-08T12:19:30.646Z sha ce3339d :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T12:31:15.027Z sha ce3339d :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T02:04:05.891Z sha f493f8b :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T07:07:47.161Z sha 264cfcd :: needs maintainer review before merge. :: none

@obviyus obviyus self-assigned this Jul 8, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 8, 2026
@obviyus
obviyus force-pushed the integration/run-lifecycle branch from 99ab256 to ce3339d Compare July 8, 2026 12:12
@obviyus

obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Both code findings fixed and rebuilt into this branch (ce3339dd05a):

  1. Self-hosted explicit run-timeout cap (d2a3cc9ba76 on fix(agents): keep finite LLM idle watchdog under unlimited run budgetsΒ #101986): the explicit non-cron run-budget branch now goes through the same provider-class ceiling as the agent-budget branch β€” one shared clampToClassIdleCeiling helper (local: budget-bound with no class ceiling; self-hosted: 300s; cloud: 120s), replacing the cloud-only implicit clamp entirely. Resolver table tests added for budgets above and below the tiers.
  2. Detached-but-leased suspect close (c66a2e58956 on fix(codex): ungate app-server terminal liveness and fan out poisoned-client retirementΒ #102097): failActiveLeases now closes the physical client even when the entry was already gracefully detached from the shared map (the one-shot-cleanup-then-terminal-kill ordering), with a WeakSet guard so a repeat suspect retire honestly reports closed: false. Regression test covers graceful-retire β†’ suspect-retire.

On the maintainer-acceptance item: the upgrade-behavior decisions (10-minute takeover default, run_stalled attribution, B3 timeout semantics, B2 terminal-only fan-out) are each recorded explicitly by @obviyus on the per-track PRs (#101910, #101986, #102097). This PR stays draft while the stack soaks on a production bot for several days; undraft follows the soak verdict plus a final rebase + full gate.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: πŸ‘€ ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 8, 2026
@obviyus

obviyus commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper autoreview

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

πŸ¦žπŸ‘€
ClawSweeper assist is taking a look at your question.

I queued a lightweight read-only assist pass. It will post a separate answer comment and will not edit the durable ClawSweeper review comment or trigger close, merge, repair, label, or branch changes.

Request: autoreview

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper assist: Existing autoreview has already run on #102160 at head ce3339dd05a810e3e60146680404816c177221f2. Current result is: no findings, but maintainer review is still required before merge because this is a draft XL run-lifecycle compatibility/session-state/message-delivery change.

Evidence:

  • Previous ClawSweeper review: β€œneeds maintainer review before merge,” reviewed July 8, 2026 12:31 UTC, SHA ce3339dd05a810e3e60146680404816c177221f2.
  • Earlier findings were fixed after force-push ce3339dd05a: self-hosted explicit timeout cap and detached-but-leased suspect close.
  • Labels now include proof: sufficient, rating: 🐚 platinum hermit, and status: πŸ‘€ ready for maintainer look.
  • PR is still draft, mergeable state is behind, with 46 files changed and 17 commits.

Suggested next action: For a fresh full correctness pass on the current PR, use @clawsweeper review.


Source: #102160 (comment)
Assist reasoning: low.

@obviyus

obviyus commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Soak finding (HamVerBot, 2026-07-08 17:21 UTC): a live Claude CLI turn was killed with CLI produced no output for 180s and was terminated while a quiet Bash tool was running (diagnostics: blocked_tool_call activeTool=Bash activeToolAge=163s recovery=none).

Analysis: not a regression from this branch β€” the live-session no-output watchdog resets only on real subprocess stdout/stderr (handleClaudeStdout), and the deleted cli_live:tool_running heartbeat never fed it; main kills this scenario identically. But it is the same inconsistency this stack fixes elsewhere: embedded runs honor BLOCKED_TOOL_CALL_ABORT_FLOOR_MS (15 min) for quiet tool calls, while the claude-cli live watchdog killed them at the 180s base window even though turn.activeTools proves a tool is outstanding.

Fix in f493f8b: unified the two watchdog arming sites into armNoOutputTimer; while observed tool calls are outstanding, the quiet window extends to max(noOutputTimeoutMs, BLOCKED_TOOL_CALL_ABORT_FLOOR_MS); the retryable cli_no_output_timeout code is preserved exactly for turns that never produced output; the kill message now reports actual quiet time. New test covers no-kill-at-base + kill-at-floor + timed_out tool provenance (replaces the test that pinned the old mid-tool kill).

Out of scope sibling: the generic (non-live) CLI runner delegates its no-output kill to the process supervisor (src/process/supervisor/supervisor.ts), which is agent-agnostic and has no tool visibility; extending it needs a supervisor API seam β€” follow-up if it ever bites in practice.

Validation: cli-runner.spawn.test.ts + cli-runner.reliability.test.ts green (148/148), tsgo core + core-test lanes green, oxlint/oxfmt clean.

This was referenced Jul 9, 2026
RomneyDa added a commit that referenced this pull request Jul 10, 2026
Partial prerequisite backport from 969efda for PR #102160.
wheakerd pushed a commit to wheakerd/clawdbot that referenced this pull request Jul 15, 2026
Partial prerequisite backport from 969efda for PR openclaw#102160.

(cherry picked from commit 868027e)
RomneyDa added a commit that referenced this pull request Jul 18, 2026
Backport the applicable run-lifecycle stack from #102160 onto the 2026.6.33 extended-stable line based on 2026.6.11. This includes bounded reply-operation recovery, finite LLM silence guards under unlimited run budgets, Claude CLI quiet-tool handling, and Codex app-server suspect-client retirement. Preserve the 6.33 diagnostics steering adaptation and Codex 0.139 fixture baseline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation extensions: codex maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. size: XL

Projects

None yet

1 participant