Fix isolated cron cold runner setup timeout by dredozubov · Pull Request #86893 · openclaw/openclaw

dredozubov · 2026-05-26T12:41:56Z

Summary

let long isolated cron jobs wait up to 5 minutes for cold Codex runner setup
keep the existing 60s setup watchdog floor for short jobs
add regression coverage for the long-job watchdog window

Tests

node scripts/run-vitest.mjs src/cron/service/timer.regression.test.ts

Real behavior proof

Behavior addressed: isolated cron agent-turn setup no longer aborts at the old fixed 60s runner-start watchdog when the job has a longer timeout budget; the run stays alive past 60s and then reaches the new bounded setup timeout.

Real environment tested: local OpenClaw cron runtime on PR head 2c969858b6389c7e3e056c1a5e51ab894868bc6b, macOS, Node 26.0.0, isolated temp cron store, no external credentials or network services.

Exact steps or command run after this patch: ran a real-time node --import tsx --input-type=module script from the PR checkout that instantiated CronService, created an isolated agentTurn cron job with timeoutSeconds: 130, started a manual run, withheld the runner-start signal, sampled state after 61s, then let cron reach the derived setup bound.

Evidence after fix: copied terminal output from the real-time cron runtime run:

{
  "openclawHead": "2c969858b6389c7e3e056c1a5e51ab894868bc6b",
  "scenario": "isolated cron agent-turn waits before runner start",
  "configuredTimeoutSeconds": 130,
  "expectedPatchedSetupBoundMs": 65000,
  "after61s": {
    "elapsedMs": 61035,
    "runnerHookEnteredMs": 41,
    "abortObserved": false,
    "cleanupCalls": 0
  },
  "final": {
    "elapsedMs": 65067,
    "abortObservedAtMs": 65044,
    "cleanupCalls": 1,
    "cleanupTimeoutMs": 130000,
    "runResult": {
      "ok": true,
      "ran": true
    },
    "lastRunStatus": "error",
    "lastStatus": "error",
    "lastError": "cron: isolated agent setup timed out before runner start",
    "persistedJobs": 1
  },
  "events": [
    {
      "action": "started",
      "elapsedMs": 34,
      "jobId": "2ddaca0a-a12e-4b7c-b7d9-2ab0746ee121"
    },
    {
      "elapsedMs": 65061,
      "level": "warn",
      "message": "cron: job run returned error status",
      "payload": {
        "jobId": "2ddaca0a-a12e-4b7c-b7d9-2ab0746ee121",
        "jobName": "PR 86893 live setup-window proof",
        "error": "cron: isolated agent setup timed out before runner start",
        "diagnosticsSummary": "cron: isolated agent setup timed out before runner start"
      }
    },
    {
      "action": "finished",
      "elapsedMs": 65062,
      "jobId": "2ddaca0a-a12e-4b7c-b7d9-2ab0746ee121",
      "status": "error",
      "error": "cron: isolated agent setup timed out before runner start",
      "durationMs": 65014
    }
  ]
}

Observed result after fix: at 61.035s cron had not aborted and cleanup had not run; at about 65.044s cron aborted the pre-runner setup phase and recorded cron: isolated agent setup timed out before runner start, matching the patched derived bound for a 130s isolated job instead of the previous fixed 60s abort.

What was not tested: a full external Codex provider cold start with live model credentials was not run; this proof isolates the cron watchdog path before runner start, which is the changed runtime behavior.

clawsweeper · 2026-05-26T12:43:58Z

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close this PR as superseded: the useful cron setup-timeout idea is now covered by a newer clean replacement that implements the change in the current watchdog owner with telemetry and proof, while this branch is conflicting and targets the old timer.ts owner.

Root-cause cluster
Relationship: superseded
Canonical: #96219
Summary: The current PR and the replacement both address the fixed isolated cron setup watchdog; the replacement is the clean canonical landing path.

Members:

canonical: fix: let isolated cron setup use the job timeout budget #96219 - It implements the same job-timeout-derived setup watchdog in the current owner, reports the resolved timeout, has sufficient proof, and is cleanly mergeable.
partial_overlap: feat(cron): make isolated-agent setup watchdog configurable #88396 - It discusses a configurable setup watchdog, which overlaps the timeout policy but adds a separate config-surface decision.
partial_overlap: fix(codex): bound shared app-server startup waits #89442 - It addresses Codex initialize lifecycle timeouts that can surface through the same cron setup-timeout symptom but does not replace the generic setup-budget fix.
adjacent_distinct: fix: restart gateway after isolated cron setup timeout #89055 - It merged gateway restart recovery after setup timeout, but does not change the setup watchdog budget.
adjacent_distinct: fix(cron): retry isolated setup timeouts #94588 - It merged retry classification for isolated setup timeouts, which complements but does not replace the setup-window change.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this branch and let maintainers review or land #96219 as the clean canonical implementation if they accept the availability tradeoff.

So I’m closing this here and keeping the remaining discussion on #96219.

Review details

Best possible solution:

Close this branch and let maintainers review or land #96219 as the clean canonical implementation if they accept the availability tradeoff.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main source still uses a fixed 60s pre-runner setup watchdog while isolated agentTurn jobs can have much larger timeout budgets; the PR body also supplies after-fix runtime output.

Is this the best way to solve the issue?

No. The idea is valid, but this branch is no longer the best implementation path because #96219 applies the same fix in the current watchdog owner and carries telemetry/proof.

Security review:

Security review cleared: The diff only changes cron timeout logic and tests; no dependency, secret, CI, package, or supply-chain surface was introduced.

AGENTS.md: found and applied where relevant.

What I checked:

Current main setup watchdog owner: Current main owns the pre-runner setup timer in src/cron/service/agent-watchdog.ts and still arms it with the fixed CRON_AGENT_SETUP_WATCHDOG_MS value. (src/cron/service/agent-watchdog.ts:141, eac1d3349c29)
This PR targets stale runtime owner: The PR diff changes createCronAgentWatchdog inside src/cron/service/timer.ts, but current main imports that watchdog from agent-watchdog.ts; live GitHub also reports this PR as CONFLICTING / DIRTY. (src/cron/service/timer.ts:303, e864cf5b23a7)
Superseding PR implements the central fix: The newer replacement PR computes setupTimeoutMs from jobTimeoutMs, uses that value for the pre-runner setup timer, and keeps the 60s floor with a bounded cap. (src/cron/service/agent-watchdog.ts:74, ea14857772d2)
Superseding PR carries telemetry: The replacement reports watchdog.setupTimeoutMs() in isolated setup-timeout telemetry instead of the stale fixed constant. (src/cron/service/timer.ts:277, ea14857772d2)
Superseding PR is viable: GitHub reports the replacement PR as clean and maintainer-editable, and its ClawSweeper review records sufficient live behavior proof with no contributor-facing blocker. (ea14857772d2)
Canonical-search pass: Search for cron setup-timeout work found the newer exact replacement at fix: let isolated cron setup use the job timeout budget #96219 plus adjacent configurable, Codex-initialize, restart, and retry work; the replacement is the safest canonical landing path for this branch's central change.

Likely related people:

ghitafilali: Authored the merged setup-timeout gateway restart work that added the current restart/telemetry consumer around this failure class. (role: recent adjacent contributor; confidence: medium; commits: b2c5e790b441; files: src/cron/service/agent-watchdog.ts, src/cron/service/timer.ts, src/gateway/server-cron.ts)
steipete: Recent path history shows the split of cron timer helpers into agent-watchdog.ts, which is the owner this PR now needs to target. (role: current watchdog module refactor contributor; confidence: medium; commits: 9d31cbbd6ad8, 1e7510ae103e; files: src/cron/service/agent-watchdog.ts, src/cron/service/timer.ts)
vincentkoc: Recent main history includes cron timer and regression-test maintenance near the affected service files. (role: recent cron service contributor; confidence: medium; commits: 44ae2fd93601, efbefceb0e2e; files: src/cron/service/timer.ts, src/cron/service/timer.regression.test.ts)
aaroneden: Authored the clean replacement PR and is credited in the merged setup-timeout retry work that touches the same failure family. (role: canonical replacement author and adjacent retry contributor; confidence: medium; commits: ea14857772d2, 73aab309894d; files: src/cron/service/agent-watchdog.ts, src/cron/service/timer.ts, src/cron/retry-hint.ts)

Codex review notes: model internal, reasoning high; reviewed against eac1d3349c29.

clawsweeper · 2026-05-26T12:49:06Z

ClawSweeper PR egg: ✨ hatched ✨ glimmer Cosmic Shellbean. Rarity: ✨ glimmer. Trait: polishes edge cases.

Details

Share on X: post this hatch
Copy: My PR egg hatched a ✨ glimmer Cosmic Shellbean in ClawSweeper.
Hatchability:

Merged PRs are hatchable.
Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

About:

Eggs appear after real-behavior proof passes. They are collectible flavor only.
Review momentum changes the shell state: follow-up work warms it, re-review makes it wobble, and a clean final review lets it hatch.
The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

dredozubov · 2026-05-28T06:57:21Z

@clawsweeper re-review

clawsweeper · 2026-05-28T06:57:24Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26559789885
Updated: 2026-05-28T07:04:06.531Z

clawsweeper · 2026-06-24T20:34:13Z

ClawSweeper applied the proposed close for this PR.

Action: closed this PR.
Close reason: duplicate or superseded.
Evidence: durable ClawSweeper review.
Coverage proof: PR B carries PR A's concrete remaining behavior and review concerns in the current implementation path, while any leftover differences such as exact branch history or incidental test details can be handled on PR B. Covering PR: fix: let isolated cron setup use the job timeout budget #96219.

clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels May 26, 2026

openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 26, 2026

clawsweeper Bot added P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels May 26, 2026

Joshtt23 mentioned this pull request May 26, 2026

fix(runtime): isolate workers and bound cron top-off #86955

Closed

RomneyDa mentioned this pull request May 28, 2026

test(cron): speed up isolated fallback tests #87520

Merged

dredozubov force-pushed the codex/cron-codex-isolation branch from 634e73c to 2c96985 Compare May 28, 2026 06:37

openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 28, 2026

fix(cron): allow cold isolated runner setup

e864cf5

RomneyDa added the dependency-guard-backfill label May 29, 2026

dredozubov force-pushed the codex/cron-codex-isolation branch from 2c96985 to e864cf5 Compare May 29, 2026 04:52

clawsweeper Bot removed the status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. label May 29, 2026

This was referenced May 30, 2026

feat(cron): make isolated-agent setup watchdog configurable #88396

Closed

fix: restart gateway after isolated cron setup timeout #89055

Merged

mbelinky mentioned this pull request Jun 3, 2026

feat(cron): support command jobs #89712

Merged

clawsweeper Bot mentioned this pull request Jun 15, 2026

fix(cron): classify spaced 'timed out' failures as retryable timeout #91515

Closed

clawsweeper Bot mentioned this pull request Jun 21, 2026

fix(codex): bound shared app-server startup waits #89442

Merged

clawsweeper Bot closed this Jun 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix isolated cron cold runner setup timeout#86893

Fix isolated cron cold runner setup timeout#86893
dredozubov wants to merge 1 commit into
openclaw:mainfrom
dredozubov:codex/cron-codex-isolation

dredozubov commented May 26, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented May 26, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented May 26, 2026 •

edited

Loading

Uh oh!

dredozubov commented May 28, 2026

Uh oh!

clawsweeper Bot commented May 28, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

dredozubov commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Real behavior proof

Uh oh!

clawsweeper Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dredozubov commented May 28, 2026

Uh oh!

clawsweeper Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dredozubov commented May 26, 2026 •

edited

Loading

clawsweeper Bot commented May 26, 2026 •

edited

Loading

clawsweeper Bot commented May 26, 2026 •

edited

Loading

clawsweeper Bot commented May 28, 2026 •

edited

Loading