fix(cron): prevent lane timeout during long tool execution by ajwan8998 · Pull Request #94082 · openclaw/openclaw

ajwan8998 · 2026-06-17T09:50:31Z

Summary

Keep a native embedded run lane alive only after its own timeout watchdog is armed, so a long-running tool does not lose its lane to a stale queue deadline.
Preserve native timeout recovery, but release the lane after one 30-second grace when a timed-out or explicitly cancelled backend ignores cancellation.
Leave plugin-owned transport progress behavior unchanged.

Real behavior proof

Behavior addressed: A native embedded attempt can legitimately run a tool longer than the queue liveness window; an ignored native timeout, /stop, or restart cancellation must still free the global lane promptly.

Real environment tested: AWS Crabbox Linux (cbx_a2c6b2574248), Node 24/Pnpm project test environment.

Exact steps or command run after this patch:

corepack pnpm test src/process/command-queue.test.ts src/cron/isolated-agent/run.meta-error-status.test.ts src/cron/service/timer.regression.test.ts src/agents/embedded-agent-runner/run.compaction-loop-guard.test.ts

Evidence after fix: AWS Crabbox run run_080e5fa4432f passed 68 cron tests, 32 process queue tests, and 10 embedded-agent guard tests. The guard coverage exercises a live native heartbeat, an ignored native timeout, and an ignored explicit cancellation.

Observed result after fix: Native tool activity can retain its lane past the ordinary liveness window, while timeout and cancellation paths release a stuck lane after one bounded grace period.

What was not tested: A live external tool process; the focused runner and queue tests cover the timeout, cancellation, and liveness contracts without a provider or channel dependency.

Closes #94033

clawsweeper · 2026-06-17T10:52:51Z

Codex review: needs changes before merge. Reviewed June 18, 2026, 12:14 PM ET / 16:14 UTC.

Summary
The PR adds native embedded-run lane heartbeat callbacks, shared command-queue abort/release timeout controls, and focused tests for long-running tools plus ignored timeout/cancellation paths.

PR surface: Source +130, Tests +263. Total +393 across 7 files.

Reproducibility: yes. Source inspection shows current main uses a sliding command-lane timeout tied to laneTaskProgressAtMs while the awaited embedded attempt has no whole-attempt heartbeat, matching the linked cron long-tool timeout report.

Review metrics: 1 noteworthy metric.

Command-queue timeout controls: 3 added. The PR adds abort signal, abort grace, and release signal controls to a shared lane API, so maintainers should review behavior outside the cron report before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #94033
Summary: This PR is a candidate fix for the isolated cron long-tool lane-timeout bug; several sibling PRs target the same root cause with narrower heartbeat-only approaches.

Members:

canonical: Bug: Cron isolated agent timeout during long tool execution #94033 - The PR body closes this issue, and the issue describes the exact isolated cron long-tool lane-timeout failure.
same_root_cause: fix(cron): add lane progress heartbeat during embedded agent tool execution (fixes #94033) #94090 - This open PR changes the same embedded-runner backend attempt call site for the same linked issue.
same_root_cause: fix(cron): prevent lane timeout expiry during long tool execution in isolated cron jobs #94193 - This open PR is another heartbeat branch for the same embedded-runner lane timeout bug.
same_root_cause: fix(agents): keep lane-task timeout alive during long-running tools (fixes #94033) #94465 - This closed PR targeted the same issue and its author later commented that this PR's three-phase timeout control design is the preferred approach.
partial_overlap: fix(agents): keep lane timeout alive during long tool execution #94060 - This open PR references the same issue but appears to target a narrower or stale runner path rather than the current embedded-runner path implicated by the report.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🐚 platinum hermit
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Stop the lane heartbeat when the timeout-release grace fires and add focused coverage for that cleanup path.

Risk before merge

[P2] The timeout-release timer can free a stuck native lane while leaving its heartbeat interval alive if the backend ignores cancellation and the awaited attempt never settles.
[P2] The PR changes the shared command-lane timeout contract, so queued work can start while an ignored backend is still unwinding outside active lane accounting.
[P1] The maintainer proof override clears the contributor proof gate, but the PR body still says no live external tool process was tested.

Maintainer options:

Stop Heartbeat Before Release (recommended)
Have the release timer stop the native lane progress heartbeat before aborting taskTimeoutReleaseSignal, and add focused coverage for the ignored-timeout cleanup path.
Accept Bounded Lane Release
After the heartbeat cleanup is fixed, maintainers can intentionally accept the shared command-queue release behavior as the desired availability tradeoff.
Narrow Back To Runner Heartbeat
If maintainers do not want generic queue abort/release controls, pause this branch and replace it with a smaller embedded-runner-owned heartbeat fix.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Stop the native lane progress heartbeat when the timeout-release grace fires, before freeing the command lane, and add focused coverage that a timed-out ignored attempt does not leave the heartbeat active after release.

Next step before merge

[P2] A narrow mechanical repair can stop the heartbeat on timeout release and add focused coverage without changing the product direction.

Security
Cleared: The diff does not touch dependencies, CI, package metadata, secrets handling, downloaded artifacts, or external code execution sources.

Review findings

[P2] Stop the heartbeat before releasing a stuck timed-out lane — src/agents/embedded-agent-runner/run.ts:1901-1902

Review details

Best possible solution:

Stop the heartbeat before timeout release frees the lane, then keep the bounded native-lane liveness design if maintainers accept the shared queue abort/release tradeoff.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main uses a sliding command-lane timeout tied to laneTaskProgressAtMs while the awaited embedded attempt has no whole-attempt heartbeat, matching the linked cron long-tool timeout report.

Is this the best way to solve the issue?

No, not yet. The branch targets the right failure mode, but the timeout-release path should stop the heartbeat before freeing the lane; the broader shared queue semantics remain a maintainer availability decision.

Full review comments:

[P2] Stop the heartbeat before releasing a stuck timed-out lane — src/agents/embedded-agent-runner/run.ts:1901-1902
When a native attempt times out and then ignores cancellation, this timer only aborts laneTaskReleaseController. The queue releases the lane while rawAttempt can remain pending, so the .finally() that clears progressInterval never runs and one heartbeat interval can stay alive for each stuck timed-out run. Stop the heartbeat in the release timer before freeing the lane.
Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.88

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 508594a620f2.

Label changes

Label justifications:

P1: The linked issue describes a current released isolated-cron workflow failing during long-running real work before the configured timeout behavior can complete.
merge-risk: 🚨 availability: The diff changes active lane timeout and release behavior, which can affect whether queued work starts while an ignored backend is still unwinding.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Override: A maintainer applied proof: override for this PR.

Evidence reviewed

PR surface:

Source +130, Tests +263. Total +393 across 7 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	5	138	8	+130
Tests	2	263	0	+263
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	7	401	8	+393

Acceptance criteria:

[P1] node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run.compaction-loop-guard.test.ts.
[P1] node scripts/run-vitest.mjs src/process/command-queue.test.ts.
[P1] node scripts/run-vitest.mjs src/cron/isolated-agent/run.meta-error-status.test.ts src/cron/service/timer.regression.test.ts.

What I checked:

Current main lane timeout wiring: Current main derives the embedded global lane timeout from the run timeout and exposes only taskTimeoutProgressAtMs from laneTaskProgressAtMs, so the queue timeout depends on explicit progress updates. (src/agents/embedded-agent-runner/run.ts:639, 508594a620f2)
Current main sliding timeout behavior: Current main races the active task against a progress-window timeout and rejects with CommandLaneTaskTimeoutError when the progress timestamp is stale. (src/process/command-queue.ts:273, 508594a620f2)
PR timeout-release path: On the PR head, armAttemptTimeoutRelease starts a timer whose callback only aborts laneTaskReleaseController; it does not stop the lane progress heartbeat before freeing the queue lane. (src/agents/embedded-agent-runner/run.ts:1901, dc3f6629d282)
PR attempt timeout callback: The attempt timeout callback calls params.onAttemptTimeout and aborts the attempt-local runAbortController, while the heartbeat in run.ts is tied to attemptAbortController or the run.ts finally path. (src/agents/embedded-agent-runner/run/attempt.ts:3393, dc3f6629d282)
PR queue release semantics: The PR command queue rejects and releases the active lane immediately when taskTimeoutReleaseSignal aborts, even if the original task promise remains unsettled. (src/process/command-queue.ts:331, dc3f6629d282)
Linked issue and proof context: The linked issue reports released isolated cron runs failing with CommandLaneTaskTimeoutError during long tool execution; the PR body includes a Crabbox test run and has a maintainer-applied proof override, but it still says no live external tool process was tested. (dc3f6629d282)

Likely related people:

vincentkoc: Blame in the current checkout attributes the central lane timeout/progress wiring and command-queue timeout path to Vincent Koc's current-main snapshot, and the live PR timeline shows vincentkoc force-pushed the current head, assigned the PR to himself, and applied proof: override. (role: recent area contributor and branch adopter; confidence: high; commits: 9592a83a2924, dc3f6629d282, c1df7aa08be2; files: src/agents/embedded-agent-runner/run.ts, src/process/command-queue.ts)
steipete: Live commit history shows repeated nearby cron, embedded-lane, and command-queue timeout work, including progress-aware run lane and cron lane timeout preservation changes. (role: adjacent cron and queue contributor; confidence: medium; commits: 21c5f8dc6df1, 41859bb3fc93, 470098bd26f3; files: src/process/command-queue.ts, src/agents/embedded-agent-runner/run.ts, src/cron/isolated-agent/run.ts)
openperf: Recent command-queue history includes openperf-authored work on releasing wedged lanes and legacy command-queue singleton recovery, which is close to the active-lane release behavior changed here. (role: adjacent recovery contributor; confidence: medium; commits: f194be19e261, e777a2b230f1; files: src/process/command-queue.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

wangmiao0668000666 · 2026-06-18T15:55:09Z

I just closed my own attempt (#94465) after studying this implementation. The three-phase timeout control design is exactly what's needed - particularly the bounded heartbeat (delayed start + abort-signal integration) and the grace-period release mechanism.

The test coverage for ignored timeout/cancellation scenarios is especially valuable and something my own PR was missing. This is a textbook example of how to properly handle liveness mechanisms with failure-path bounding.

Thanks for the comprehensive solution and for working through the design with the review process. Looking forward to seeing this land!

The lane timeout sliding window expires during long-running tool execution (e.g. exec commands >5min) because noteLaneTaskProgress() is never called during tool execution. Add a periodic 30s interval that calls noteLaneTaskProgress() while the embedded attempt runs, keeping the lane alive until the attempt completes. Closes: openclaw#94033 Co-Authored-By: Claude <[email protected]>

Co-Authored-By: Claude <[email protected]>

openclaw-barnacle Bot added agents Agent runtime and tooling size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 17, 2026

clawsweeper Bot mentioned this pull request Jun 17, 2026

fix(cron): prevent lane timeout expiry during long tool execution in isolated cron jobs #94193

Closed

This was referenced Jun 18, 2026

Bug: Cron isolated agent timeout during long tool execution #94033

Closed

fix(agents): keep lane-task timeout alive during long-running tools (fixes #94033) #94465

Closed

openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 18, 2026

ajwan8998 force-pushed the fix/issue-94033-cron-lane-timeout branch from 8110741 to 646d968 Compare June 18, 2026 08:57

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 18, 2026

vincentkoc force-pushed the fix/issue-94033-cron-lane-timeout branch from 646d968 to 3b04ea6 Compare June 18, 2026 12:06

vincentkoc self-assigned this Jun 18, 2026

openclaw-barnacle Bot added size: M triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. and removed size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 18, 2026

vincentkoc added the proof: override Maintainer override for the external PR real behavior proof gate. label Jun 18, 2026

openclaw-barnacle Bot removed the triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. label Jun 18, 2026

clawsweeper Bot mentioned this pull request Jun 18, 2026

fix(cron): add lane progress heartbeat during embedded agent tool execution (fixes #94033) #94090

Closed

4 tasks

ajwan8998 and others added 5 commits June 19, 2026 00:03

chore: retry CI

fd3a501

Co-Authored-By: Claude <[email protected]>

chore: retry

0ced1d3

Co-Authored-By: Claude <[email protected]>

chore: retry

0aa68ba

Co-Authored-By: Claude <[email protected]>

fix(cron): release cancelled embedded lanes

dc3f662

vincentkoc force-pushed the fix/issue-94033-cron-lane-timeout branch from 3b04ea6 to dc3f662 Compare June 18, 2026 16:05

vincentkoc merged commit fa4f1ab into openclaw:main Jun 18, 2026
27 of 28 checks passed

hannesrudolph mentioned this pull request Jun 18, 2026

fix(codex): release timed-out app-server lanes #94685

Closed

clawsweeper Bot mentioned this pull request Jun 19, 2026

fix(codex): release timed-out app-server lanes #94833

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(cron): prevent lane timeout during long tool execution#94082

fix(cron): prevent lane timeout during long tool execution#94082
vincentkoc merged 5 commits into
openclaw:mainfrom
ajwan8998:fix/issue-94033-cron-lane-timeout

ajwan8998 commented Jun 17, 2026 •

edited by vincentkoc

Loading

Uh oh!

clawsweeper Bot commented Jun 17, 2026 •

edited

Loading

Uh oh!

wangmiao0668000666 commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

ajwan8998 commented Jun 17, 2026 • edited by vincentkoc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Real behavior proof

Uh oh!

clawsweeper Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangmiao0668000666 commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ajwan8998 commented Jun 17, 2026 •

edited by vincentkoc

Loading

clawsweeper Bot commented Jun 17, 2026 •

edited

Loading