fix: let isolated cron setup use the job timeout budget by aaroneden · Pull Request #96219 · openclaw/openclaw

aaroneden · 2026-06-23T23:33:26Z

What Problem This Solves

OpenClaw cron isolated agentTurn jobs could fail before the runner ever started because the setup watchdog had a fixed 60-second ceiling, independent of the job's much larger timeout budget. Aaron's live Route Meeting Transcripts (Daily) cron (3549b060-2ab7-440b-aca5-7c8c9871e0d3) showed this exact failure pattern: repeated runs ended after about 60 seconds with cron: isolated agent setup timed out before runner start, no session id, no model/provider, and no token usage.

Root cause

The isolated cron agent setup watchdog had a hardcoded 60-second ceiling. Installed-code evidence from the failing environment pointed to server-cron-B_E4z-Px.js:493 (CRON_AGENT_SETUP_WATCHDOG_MS = 6e4) plus server-cron-B_E4z-Px.js:566-569, where the setup watchdog fired while still waiting for the runner.

Fix

Derive the isolated-agent setup watchdog from the job timeout with bounded limits: keep the existing 60-second minimum for short jobs, cap setup at 10 minutes, and otherwise allow setup to use half of the job timeout budget. Timeout diagnostics still use setupTimeoutErrorMessage(...), and setup timeout telemetry reports the derived watchdog timeout.

Sibling occurrences

Searched for CRON_AGENT_SETUP_WATCHDOG_MS, setupTimeoutErrorMessage, waiting_for_runner, and resolveCronAgentSetupWatchdogMs. The only cron setup watchdog path is src/cron/service/agent-watchdog.ts; it is fixed here. Other hits are tests and retry-hint text, intentionally unchanged.

Evidence

Added a regression test for an isolated cron agentTurn whose runner starts after the previous 60-second setup ceiling but before the job timeout.

RED on unmodified origin/main: node scripts/test-projects.mjs src/cron/service/agent-watchdog.test.ts failed with cron: isolated agent setup timed out before runner start.
GREEN after fix: same test passed, 3 tests.
Cron service suite: node scripts/test-projects.mjs src/cron/service passed, 15 files / 151 tests.
Changed tests: pnpm test:changed passed, 2 files / 9 tests.
Lint/type/check: pnpm lint passed; pnpm check passed.
Format: global pnpm format:check currently reports unrelated baseline formatting issues across 218 existing files; targeted pnpm exec oxfmt --check src/cron/service/agent-watchdog.test.ts src/cron/service/agent-watchdog.ts src/cron/service/timer.ts passed, and git diff --check origin/main passed.
Live recovery proof: patched the same behavior into Aaron's installed OpenClaw bundle, restarted Gateway, and force-ran cron 3549b060-2ab7-440b-aca5-7c8c9871e0d3; it completed ok in 128,683 ms with session 30d3924b-6479-4bf7-a922-d6d6c3e27afd, model gpt-5.5, provider openai. Scheduler readback now shows lastRunStatus=ok, lastStatus=ok, last_error=null, and consecutive_errors=0.

clawsweeper · 2026-06-24T02:55:11Z

Codex review: needs maintainer review before merge. Reviewed June 30, 2026, 11:25 PM ET / 03:25 UTC.

Summary
The PR derives the isolated cron pre-runner setup watchdog from the cron job timeout budget, reports that derived timeout in telemetry, and adds watchdog regression coverage.

PR surface: Source +7, Tests +17. Total +24 across 3 files.

Reproducibility: yes. Source inspection shows current main always arms a fixed 60-second setup watchdog before runner start while isolated agentTurn jobs can have much larger timeout budgets; the PR body also supplies red/green and live recovery proof.

Review metrics: 1 noteworthy metric.

Implicit setup watchdog behavior: 1 changed: fixed 60s -> 60s floor, half job timeout, 10m cap. This changes default cron availability behavior for long isolated agentTurn jobs, so maintainers should explicitly accept the longer stuck-setup window before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Risk before merge

[P1] Merging changes the default isolated setup wait from a fixed 60 seconds to as much as 10 minutes, so genuinely stuck runner setup can hold cron capacity longer before cleanup and notification.
[P1] The RapidFix comment and related Codex/configurable-watchdog threads describe post-start deadlocks, Codex initialize attribution, or operator tuning; this PR should not be treated as closing those separate problems.

Maintainer options:

Accept the bounded longer setup window (recommended)
Merge after maintainer acceptance that isolated cron runner setup can occupy a cron slot for up to 10 minutes when startup is genuinely stuck.
Tighten the cap before merge
Ask for a lower cap or additional runtime coverage if maintainers want stronger protection against stuck setup occupying cron capacity.
Defer to configurability work
Pause this PR if maintainers decide the canonical fix must be an operator-tunable watchdog instead of a derived default.

Next step before merge

No automated repair is needed; the next action is maintainer review and acceptance of the bounded longer setup-watchdog tradeoff.

Security
Cleared: The diff only changes cron watchdog timing logic and tests; it does not touch dependencies, secrets, CI, packaging, lockfiles, permissions, or code-execution supply-chain surfaces.

Review details

Best possible solution:

Land the bounded derived-watchdog fix if maintainers accept longer stuck-setup occupancy, while tracking post-start deadlocks, Codex initialize diagnostics, and configurability in their separate items.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main always arms a fixed 60-second setup watchdog before runner start while isolated agentTurn jobs can have much larger timeout budgets; the PR body also supplies red/green and live recovery proof.

Is this the best way to solve the issue?

Yes for the narrow pre-runner setup-timeout bug. The fix sits at the watchdog owner, reuses the existing job timeout budget, avoids a new config surface, and leaves broader Codex and post-start cleanup behavior to separate work.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against fa3c9de45965.

Label changes

Label justifications:

P2: The PR addresses a normal-priority isolated cron runtime bug with limited blast radius to pre-runner setup timing.
merge-risk: 🚨 availability: Extending the setup watchdog can keep cron capacity occupied longer when isolated runner setup is truly stuck.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix live output from a patched installed OpenClaw bundle where the affected cron completed successfully, plus red/green targeted tests and broader validation.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live output from a patched installed OpenClaw bundle where the affected cron completed successfully, plus red/green targeted tests and broader validation.

Evidence reviewed

PR surface:

Source +7, Tests +17. Total +24 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	14	7	+7
Tests	1	17	0	+17
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	31	7	+24

What I checked:

Repository policy read: Root AGENTS.md was read fully and applied: PR review must inspect current source, callers, tests, history, proof, and merge risk rather than deciding from the diff alone. (AGENTS.md:1, fa3c9de45965)
Current main still has fixed setup watchdog: Current main arms isolated pre-runner setup with CRON_AGENT_SETUP_WATCHDOG_MS, so setup still times out at the fixed 60-second ceiling before this PR. (src/cron/service/agent-watchdog.ts:145, fa3c9de45965)
Agent-turn timeout budget can be much larger: Current main gives agentTurn cron jobs a 60-minute default timeout, proving the fixed 60-second setup watchdog can be far below the configured job timeout budget. (src/cron/service/timeout-policy.ts:16, fa3c9de45965)
PR diff uses bounded derived setup timeout: The PR computes setupTimeoutMs once from jobTimeoutMs, uses it for the setup timer, exposes it on the watchdog handle, and reports it in isolated setup-timeout telemetry. (src/cron/service/agent-watchdog.ts:145, ea14857772d2)
Existing caller/callee path checked: executeJobCoreWithTimeout creates the watchdog for isolated agentTurn runs and wires noteRunnerStarted through runIsolatedAgentJob; runCronIsolatedAgentTurn only calls the start callback after isolated setup reaches the runner boundary. (src/cron/service/timer.ts:236, fa3c9de45965)
Current tests cover setup-timeout behavior: Adjacent timer regression tests currently assert 60-second setup timeout notification and cleanup paths, so this PR's new focused watchdog test is in the right test surface. (src/cron/service/timer.regression.test.ts:2728, fa3c9de45965)

Likely related people:

ghitafilali: Authored the merged isolated cron setup-timeout restart work that added the current agent-watchdog file and setup-timeout regression surface. (role: introduced setup-timeout behavior; confidence: high; commits: b2c5e790b441; files: src/cron/service/agent-watchdog.ts, src/cron/service/agent-watchdog.test.ts, src/cron/service/timer.regression.test.ts)
849261680: Authored the recent merged change that altered setup-timeout handling to avoid gateway restart and touched the timer regression surface for this behavior. (role: recent area contributor; confidence: high; commits: 19707cce1d73; files: src/cron/service/timer.ts, src/cron/service/timer.regression.test.ts)
steipete: Authored the merged cron service timer-helper split that established agent-watchdog.ts as the current owner boundary for setup and execution watchdog behavior. (role: owner-adjacent refactor author; confidence: medium; commits: 9d31cbbd6ad8; files: src/cron/service/agent-watchdog.ts, src/cron/service/timer.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

nessim-liamani · 2026-06-24T20:03:27Z

Same family of issue observed here (RapidFix.be) but with a post-start failure mode, not a pre-start timeout.

Our pattern: The isolated session does start — it claims the queue item (sets status='processing'), runs for 60-108 seconds, but then the model call fails or returns empty content. The session hangs with "⚠️ Agent couldn't generate a response. Note: some tool actions may have already been executed." The revision is left permanently stuck in "processing" and the cron runner retains "already-running" state, blocking all future runs until a disable/enable cycle.

Two root causes we found (local LLM context):

Model context overflow — our local llama-server (Qwen3-8B Q5_K_M, GTX 1060 6GB) at -c 4096 was exceeded by the isolated session's light-context bootstrap, causing HTTP 500 → silent fallback to paid API
Thinking-mode leak — Qwen3-8B build b9638 outputs all text as reasoning_content with empty content unless --reasoning off is set

Fixes applied locally: -c 4608 --parallel 1, --reasoning off, gate script hardened with || true.

The 60-second setup watchdog (this PR's fix) would help when the model is slow to respond, but once the session has started and made a model call, a different cleanup mechanism is needed to prevent orphaned "processing" states and "already-running" deadlocks.

openclaw-barnacle · 2026-07-16T04:32:04Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

openclaw-barnacle · 2026-07-23T04:34:31Z

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

fix: let isolated cron setup use job timeout budget

ea14857

openclaw-barnacle Bot added size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 23, 2026

This was referenced Jun 24, 2026

fix(cron): preserve provider/model on isolated-run timeout row #95943

Merged

Fix isolated cron cold runner setup timeout #86893

Closed

nessim-liamani mentioned this pull request Jun 24, 2026

cron: silent model fallback to paid APIs during agentTurn jobs — no alert, no audit, no cost visibility #96548

Closed

openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 16, 2026

openclaw-barnacle Bot closed this Jul 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix: let isolated cron setup use the job timeout budget#96219

fix: let isolated cron setup use the job timeout budget#96219
aaroneden wants to merge 1 commit into
openclaw:mainfrom
aaroneden:fix/cron-isolated-setup-watchdog-timeout

aaroneden commented Jun 23, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

nessim-liamani commented Jun 24, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 16, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

aaroneden commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Root cause

Fix

Sibling occurrences

Evidence

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessim-liamani commented Jun 24, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 16, 2026

Uh oh!

openclaw-barnacle Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aaroneden commented Jun 23, 2026 •

edited

Loading

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading