fix(cron): avoid gateway restart on setup timeout by 849261680 · Pull Request #96396 · openclaw/openclaw

849261680 · 2026-06-24T12:11:20Z

What Problem This Solves

An isolated cron setup timeout currently escalates a tick-level failure into a full Gateway restart. Current main, v2026.6.9, and latest stable v2026.6.10 all still have onIsolatedAgentSetupTimeout calling requestSafeGatewayRestart({ delayMs: 0 }) from src/gateway/server-cron.ts, and maybeNotifyIsolatedAgentSetupTimeout marks state.restartRecoveryPending = true in src/cron/service/timer.ts.

Under event-loop saturation, that restart does not fix the saturated workload. It can instead kill unrelated in-flight embedded runs and freeze cron scheduling behind restart recovery.

Why This Change Was Made

This changes the setup-timeout response to stay at the cron-job boundary:

src/gateway/server-cron.ts logs the isolated setup timeout but no longer requests a Gateway restart.
src/cron/service/timer.ts no longer flips restartRecoveryPending when the setup-timeout notification fires.
The existing run-result path still records the job error and applies cron error backoff.
Manual setup-timeout notification state was renamed to remove the stale restart semantics.

This is intentionally narrower than adaptive setup timeouts or a new corruption detector. It removes the destructive unconditional restart path while preserving the existing timeout detection, cleanup, error recording, and backoff behavior.

User Impact

When one isolated cron job times out during setup, OpenClaw now backs off that job without restarting the Gateway. In-flight non-cron embedded work is preserved, and cron scheduling is not frozen behind restart recovery just because the timeout notification was sent.

Evidence

Current/shipped behavior checked:

gh release list --repo openclaw/openclaw --limit 5
git fetch upstream tag v2026.6.9 tag v2026.6.10
git grep -n "requestSafeGatewayRestart" upstream/main -- src/gateway/server-cron.ts src/cron/service/timer.ts
git grep -n "restartRecoveryPending = true" upstream/main -- src/cron/service/timer.ts
git grep -n "requestSafeGatewayRestart" v2026.6.9 -- src/gateway/server-cron.ts
git grep -n "restartRecoveryPending = true" v2026.6.9 -- src/cron/service/timer.ts
git grep -n "requestSafeGatewayRestart" v2026.6.10 -- src/gateway/server-cron.ts
git grep -n "restartRecoveryPending = true" v2026.6.10 -- src/cron/service/timer.ts

Verification run in the dedicated openclaw-95784 worktree:

pnpm install --frozen-lockfile
OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=.vitest-cache-95784-active OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=300000 node scripts/run-vitest.mjs src/cron/active-jobs-manual-run.test.ts
OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=.vitest-cache-95784-gateway OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=300000 node scripts/run-vitest.mjs src/gateway/server-cron.test.ts
OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=.vitest-cache-95784-cron OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=300000 node scripts/run-vitest.mjs src/cron/service/timer.regression.test.ts
node_modules/.bin/oxfmt --check --threads=1 src/cron/active-jobs-manual-run.test.ts src/cron/service.test-harness.ts src/cron/service/ops.ts src/cron/service/state.ts src/cron/service/timer.ts src/cron/service/timer.regression.test.ts src/gateway/server-cron.ts src/gateway/server-cron.test.ts
node_modules/.bin/oxlint src/cron/active-jobs-manual-run.test.ts src/cron/service.test-harness.ts src/cron/service/ops.ts src/cron/service/state.ts src/cron/service/timer.ts src/cron/service/timer.regression.test.ts src/gateway/server-cron.ts src/gateway/server-cron.test.ts --threads=1 --report-unused-disable-directives-severity error
git diff --check

Observed results:

src/cron/active-jobs-manual-run.test.ts: 1 file passed, 6 tests passed.
src/gateway/server-cron.test.ts: 2 files passed, 48 tests passed.
src/cron/service/timer.regression.test.ts: 1 file passed, 64 tests passed.
oxfmt --check: all 8 matched files use the correct format.
direct oxlint: passed with no diagnostics.
git diff --check: passed.

What was not tested: a full live systemd --user Gateway under production event-loop saturation. The changed behavior is source-level deterministic: the unconditional restart call and restart-recovery state flip are removed, and the focused tests cover the Gateway handler, manual timeout notification, scheduler re-arm, and scheduled-batch continuation paths.

Additional Live Runtime Proof After ClawSweeper Review

ClawSweeper requested real behavior proof beyond focused tests. I added a local live runtime proof using a real Node process, the actual CronService runtime, and the real 60-second setup watchdog. This is not a Vitest run and does not use fake timers.

Command shape:

node --import tsx /tmp/openclaw-95784-live-cron-proof.mjs

The proof script creates a temporary cron service state, registers a SIGUSR1 listener, runs an isolated agent cron job whose runner intentionally never reaches onExecutionStarted, waits for the real setup watchdog to fire, then reads back the job runtime state.

Observed output, with only the temp directory elided:

{
  "proof": "issue-95784 live CronService setup-timeout no-restart proof",
  "pid": 57726,
  "elapsedMs": 60161,
  "runResult": {
    "ok": true,
    "ran": true
  },
  "setupTimeoutNotification": {
    "jobId": "bda19f66-56aa-418e-82a3-5a32b8ec7025",
    "error": "cron: isolated agent setup timed out before runner start",
    "timeoutMs": 60000,
    "atMs": 60160
  },
  "sigusr1Count": 0,
  "handoffExists": false,
  "jobState": {
    "lastRunStatus": "error",
    "lastStatus": "error",
    "lastDurationMs": 60026,
    "lastError": "cron: isolated agent setup timed out before runner start",
    "lastErrorReason": "timeout",
    "consecutiveErrors": 1,
    "consecutiveSkipped": 0
  },
  "warningMessages": [
    "cron: job run returned error status",
    "cron: isolated agent setup timed out before runner start; backing off job without gateway restart"
  ]
}

Proof result: after the real setup watchdog fired, the cron job was recorded as an error with backoff state (consecutiveErrors: 1), no SIGUSR1 restart was emitted (sigusr1Count: 0), no supervisor restart handoff was written (handoffExists: false), and the process stayed alive long enough to read back the job state.

clawsweeper · 2026-06-24T12:12:16Z

Codex review: needs maintainer review before merge. Reviewed June 24, 2026, 8:36 AM ET / 12:36 UTC.

Summary
This PR removes the isolated cron setup-timeout Gateway restart and restart-recovery pause, renames the manual timeout notification flag, and updates cron/Gateway tests for job-local backoff.

PR surface: Source -11, Tests -5. Total -16 across 8 files.

Reproducibility: yes. for the source-level path: current main and v2026.6.10 call requestSafeGatewayRestart and set restartRecoveryPending when an isolated setup timeout is notified. The PR body adds terminal runtime proof for the after-fix no-restart/backoff path, but I did not reproduce the full systemd saturation incident.

Review metrics: 1 noteworthy metric.

Recovery policy change: 1 restart trigger removed, 1 scheduler pause removed. This changes latest-release failure handling for isolated cron setup timeouts, so maintainers should notice the compatibility and availability tradeoff before merge.

Stored data model
Persistent data-model change detected: serialized state: src/cron/service/state.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #95784
Summary: This PR is a candidate fix for the canonical open report about isolated cron setup timeouts causing destructive Gateway restarts; related items cover the earlier opposite policy or complementary recovery/event-loop surfaces.

Members:

canonical: [Bug]: onIsolatedAgentSetupTimeout triggers full-process gateway restart on event-loop saturation — destroys in-flight work and re-enters the same saturation on next tick (2026.6.9) #95784 - The PR body closes this issue and implements its central no-restart request.
partial_overlap: fix: restart gateway after isolated cron setup timeout #89055 - This merged PR introduced the automatic setup-timeout restart policy that the current PR proposes to reverse.
partial_overlap: Fix/95784 cron setup timeout no restart #95813 - This was an earlier closed-unmerged attempt at similar no-restart behavior and is not a safe canonical replacement.
partial_overlap: [Bug]: Main-session restart-recovery has no cross-boot retry budget — wedged sessions death-loop the gateway across reboots (2026.6.9) #95750 - This covers the complementary cross-boot restart-recovery retry problem after restarts, not the cron timeout trigger itself.
partial_overlap: [Bug]: Isolated agent runs stall in runtime-plugins phase before execution start — recurring across hourly crons on 2026.5.22, no named-plugin diagnostic #87327 - This reports isolated cron stalls in runtime-plugins, the symptom that can trigger the setup-timeout path.
1 more in the report.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Get maintainer confirmation that job-local backoff should replace the shipped automatic restart policy, or revise with a stronger escalation predicate.

Risk before merge

[P2] Merging changes shipped v2026.6.10 behavior from automatic Gateway restart plus scheduler recovery pause to job-local error/backoff, so true stale or corrupt Gateway setup-timeout cases may repeat as cron failures until a stronger escalation path exists.
[P1] The added runtime proof demonstrates no-restart/backoff behavior in a real Node CronService path, but it does not reproduce the full reported systemd event-loop saturation incident.

Maintainer options:

Confirm Job-Local Default
Maintainers can accept the behavior change explicitly and merge the no-restart default with the current focused tests and terminal runtime proof.
Add Escalation Predicate First
Before merge, require a stronger repeated-failure or idle-loop corruption predicate that can still request a non-destructive Gateway restart when job-local backoff is not enough.
Pause For Recovery Policy
If the shipped automatic restart path should not be removed yet, pause or close this PR until the restart/backoff policy is decided with the related recovery issues.

Next step before merge

[P2] Maintainers need to decide the recovery-policy tradeoff before merge; there is no narrow mechanical repair left for ClawSweeper after proof was added.

Security
Cleared: The diff removes a restart call and renames in-memory cron state/tests; it adds no dependency, workflow, secret, permission, or executable supply-chain surface.

Review details

Best possible solution:

Land the job-local backoff default only with maintainer-approved escalation semantics for corruption-class setup failures, while keeping the focused no-restart and scheduler rearm regression coverage.

Do we have a high-confidence way to reproduce the issue?

Yes for the source-level path: current main and v2026.6.10 call requestSafeGatewayRestart and set restartRecoveryPending when an isolated setup timeout is notified. The PR body adds terminal runtime proof for the after-fix no-restart/backoff path, but I did not reproduce the full systemd saturation incident.

Is this the best way to solve the issue?

Mostly yes for the narrow no-restart default: the patch removes the direct process-restart side effect at the cron/Gateway boundary and keeps existing job error backoff. The remaining question is product policy for when, if ever, setup timeouts should escalate to Gateway restart.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e4763b0631bf.

Label changes

Label changes:

add proof: sufficient: Contributor real behavior proof is sufficient. The PR body now includes terminal runtime proof from a real Node CronService path using the actual 60-second setup watchdog, showing no SIGUSR1, no supervisor handoff, and job error/backoff state after timeout.
add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body now includes terminal runtime proof from a real Node CronService path using the actual 60-second setup watchdog, showing no SIGUSR1, no supervisor handoff, and job error/backoff state after timeout.
remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P1: The PR targets a linked Gateway restart-loop and session-loss bug affecting cron, embedded-run, and channel workflows under event-loop saturation.
merge-risk: 🚨 compatibility: Merging changes shipped setup-timeout recovery behavior from automatic Gateway restart and restart-recovery pause to job-local backoff.
merge-risk: 🚨 availability: If setup timeouts indicate a genuinely stale or corrupt Gateway, removing the automatic restart path could leave cron workflows failing without central recovery.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body now includes terminal runtime proof from a real Node CronService path using the actual 60-second setup watchdog, showing no SIGUSR1, no supervisor handoff, and job error/backoff state after timeout.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body now includes terminal runtime proof from a real Node CronService path using the actual 60-second setup watchdog, showing no SIGUSR1, no supervisor handoff, and job error/backoff state after timeout.

Evidence reviewed

PR surface:

Source -11, Tests -5. Total -16 across 8 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	5	7	18	-11
Tests	3	15	20	-5
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	8	22	38	-16

What I checked:

Repository policy read: Read the full root AGENTS.md and scoped Gateway AGENTS.md; the review applied the high-confidence whole-path, Gateway, proof, and compatibility-risk guidance. (AGENTS.md:1, e4763b0631bf)
Scoped Gateway policy read: Scoped Gateway guidance was read because the PR touches src/gateway/server-cron.ts. (src/gateway/AGENTS.md:1, e4763b0631bf)
Current main restart hook: Current main calls requestSafeGatewayRestart from onIsolatedAgentSetupTimeout with reason cron.isolated_agent_setup_timeout, delayMs 0, and preservePendingEmitHooks true. (src/gateway/server-cron.ts:549, e4763b0631bf)
Current main scheduler pause: Current main sets restartRecoveryPending after a setup-timeout notification, causing armTimer and timer ticks to skip while restart recovery is pending. (src/cron/service/timer.ts:330, e4763b0631bf)
Latest release behavior: Release v2026.6.10 contains the same setup-timeout restart hook and restartRecoveryPending flip, so the PR changes shipped behavior. (src/gateway/server-cron.ts:549, aa69b12d0086)
PR head removes Gateway restart: At the PR head, the setup-timeout handler only logs the timeout and no longer imports or calls requestSafeGatewayRestart. (src/gateway/server-cron.ts:548, 1ec0749e6af6)

Likely related people:

ghitafilali: Authored the merged setup-timeout restart work that added the current Gateway restart hook and cron restart-recovery handling. (role: introduced behavior; confidence: high; commits: 13d06b5d6f9b, b2c5e790b441; files: src/gateway/server-cron.ts, src/cron/service/timer.ts, src/cron/service/ops.ts)
vincentkoc: GitHub PR metadata shows vincentkoc merged the setup-timeout restart PR, and recent path history includes adjacent cron timeout/delivery changes plus the v2026.6.10 release containing the behavior. (role: merger and recent release contributor; confidence: high; commits: b2c5e790b441, efbefceb0e2e, 44ae2fd93601; files: src/gateway/server-cron.ts, src/cron/service/timer.ts, CHANGELOG.md)
shakkernerd: Recent GitHub path history on cron timer and timer regression tests includes multiple timed cron cancellation fixes adjacent to setup-timeout and backoff behavior. (role: adjacent cron cancellation contributor; confidence: medium; commits: 9082233a433c, 24196e05f59b, 372f85d368d8; files: src/cron/service/timer.ts, src/cron/service/timer.regression.test.ts)
openperf: Recent restart coordinator work changed coalesced Gateway restart continuation behavior, which is part of the recovery surface affected by removing this automatic restart call. (role: adjacent restart coordinator contributor; confidence: medium; commits: afcbdd7416ac; files: src/infra/restart-coordinator.ts, src/infra/restart.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

849261680 · 2026-06-24T12:26:32Z

@clawsweeper re-review

Added the requested live runtime proof to the PR body under "Additional Live Runtime Proof After ClawSweeper Review". The proof uses a real Node process, actual CronService runtime code, and the real 60s setup watchdog; observed sigusr1Count: 0, handoffExists: false, lastRunStatus: "error", and consecutiveErrors: 1 after setup timeout.

clawsweeper · 2026-06-24T12:26:35Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

velvet-shark · 2026-06-25T16:11:40Z

Merged via squash.

Prepared head SHA: cc2ac4ea017afbbef839dc6b5712f89a4fbbe2bc
Landed commit: 19707cce1d7336e2fe5e8431b0d79240692ebb96

* fix(cron): avoid gateway restart on setup timeout * fix(cron): avoid gateway restart on setup timeout --------- Co-authored-by: Radek Sienkiewicz <[email protected]>

* fix(cron): avoid gateway restart on setup timeout * fix(cron): avoid gateway restart on setup timeout --------- Co-authored-by: Radek Sienkiewicz <[email protected]> (cherry picked from commit 19707cc)

fix(cron): avoid gateway restart on setup timeout

1ec0749

openclaw-barnacle Bot added gateway Gateway runtime size: S labels Jun 24, 2026

velvet-shark self-assigned this Jun 25, 2026

fix(cron): avoid gateway restart on setup timeout

cc2ac4e

velvet-shark requested a review from a team as a code owner June 25, 2026 16:10

openclaw-barnacle Bot added extensions: lmstudio extensions: firecrawl extensions: ollama extensions: openshell extensions: perplexity extensions: xai extensions: parallel channel: raft Channel integration: Raft size: XL and removed size: S labels Jun 25, 2026

velvet-shark merged commit 19707cc into openclaw:main Jun 25, 2026
322 of 331 checks passed

github-actions Bot mentioned this pull request Jun 25, 2026

📡 Upstream Digest — 2026-06-25 18:04 UTC curtismercier/openclaw-mods#1151

Open

alexzhu0 mentioned this pull request Jul 15, 2026

Make runtime plugin startup stalls name in-flight plugins #88681

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(cron): avoid gateway restart on setup timeout#96396

fix(cron): avoid gateway restart on setup timeout#96396
velvet-shark merged 2 commits into
openclaw:mainfrom
849261680:fix/95784-cron-timeout-no-restart

849261680 commented Jun 24, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

849261680 commented Jun 24, 2026

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

velvet-shark commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

849261680 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Additional Live Runtime Proof After ClawSweeper Review

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

849261680 commented Jun 24, 2026

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

velvet-shark commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

849261680 commented Jun 24, 2026 •

edited

Loading

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading