Skip to content

fix: restart gateway after isolated cron setup timeout#89055

Merged
vincentkoc merged 1 commit into
openclaw:mainfrom
ghitafilali:codex/openclaw-gateway-isolated-runner-health
Jun 14, 2026
Merged

fix: restart gateway after isolated cron setup timeout#89055
vincentkoc merged 1 commit into
openclaw:mainfrom
ghitafilali:codex/openclaw-gateway-isolated-runner-health

Conversation

@ghitafilali

@ghitafilali ghitafilali commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • classify isolated cron setup timeouts as a gateway-health failure class
  • notify the gateway cron service when setup stalls before the isolated runner starts
  • request a safe gateway restart through the existing restart coordinator so affected recurring jobs can recover centrally
  • add cron-service and gateway regression coverage for the restart path

Root Cause

Cron jobs that use isolated agent turns can fail before the runner starts with cron: isolated agent setup timed out before runner start. Today that error is recorded and alerted, but it is not treated as a platform-level gateway/isolated-runner health failure. Individual cron workflows can self-repair around it, but the shared failure mode keeps recurring across unrelated jobs.

Impact

When this failure class occurs, OpenClaw now asks the existing safe-restart coordinator to restart only after the normal preflight drain says it is safe. This should reduce repeated user-visible cron failure cards without interrupting active work.

Real behavior proof

  • Behavior or issue addressed: Recurring isolated-agent cron jobs were repeatedly failing before runner start while the gateway was stale/hot, producing cron: isolated agent setup timed out before runner start cards instead of recovering centrally.
  • Real environment tested: Real local macOS OpenClaw setup, LaunchAgent-managed local gateway on loopback, released OpenClaw 2026.5.28, with active background workload.
  • Exact steps or command run after this patch: Verified the PR head 354de1a3b99027027b303c29b8852ec8fef9091d; ran the PR-head proof command pnpm test src/gateway/server-cron.test.ts src/cron/service/timer.regression.test.ts src/infra/restart-coordinator.test.ts src/infra/infra-runtime.test.ts --reporter verbose; ran the production build command node scripts/build-all.mjs; then ran the built-runtime proof command node --import tsx /tmp/openclaw-pr-89055-setup-timeout-proof.mjs. I also checked the real recovered machine with openclaw gateway status, openclaw status --json, and ps -p 93467 -o pid,pcpu,pmem,rss,etime,command.
  • PR-build restart proof: node scripts/build-all.mjs passed on PR head 354de1a3b99027027b303c29b8852ec8fef9091d. The built-runtime proof then ran a controlled setup stall through the real cron timeout path and the real safe restart scheduler. Its output showed productionBuildCompleted=true, distContainsGatewayHook=true, elapsedMs=60282, result.error="cron: isolated agent setup timed out before runner start", restartRequests[0].status="scheduled", restartRequests[0].preflightSafe=true, restartRequests[0].restartCoalesced=false, restartRequests[0].restartMode="emit", restartRequests[0].restartReason="cron.isolated_agent_setup_timeout", sigusr1Events[0].authorized=true, sigusr1Events[0].recovered=true, and all four assertions true: setup timeout result, safe restart scheduled, gateway run loop recovered, and production dist contains the gateway hook.
  • Evidence after fix: Copied live output from the recovered real machine showed openclaw gateway status with Runtime: running (pid 93467, state active), Connectivity probe: ok, Capability: admin-capable, and Gateway version: 2026.5.28. Copied live openclaw status --json output showed gateway.reachable=true, gateway.connectLatencyMs=45, queuedSystemEvents=[], and tasks.byStatus.queued=0. Copied live ps output showed gateway pid 93467 around 1.0% CPU, 3.2% memory, RSS 534560 KB. Before recovery, the same setup had a stale gateway timeout and a gateway node process around 109% CPU with roughly 2.8GB RSS. Supplemental PR-head terminal output also showed ✓ |gateway| ... requests a safe gateway restart when isolated cron setup times out, ✓ |infra| ... coalesces duplicate scheduled restarts into a single pending timer, and ✓ |infra| ... defers SIGUSR1 until deferral check returns 0.
  • Observed result after fix: The real gateway recovered and stayed reachable instead of continuing the setup-timeout loop. Normal OpenClaw work resumed through the restored gateway: representative background work and inference processes were running successfully after the gateway recovered. On PR head, the selected proof run passed gateway 22/22, infra 34/34, and cron 43/43, proving this patch requests the same safe restart path for the setup-timeout class while the existing coordinator coalesces/defers repeated restart requests.
  • What was not tested: The exact PR build was not installed into a live released OpenClaw package; the built-runtime proof exercises the PR-head cron timeout path and real safe restart scheduler in-process, while the live machine proof covers the operational recovery path on the released install.

Testing

  • pnpm test src/gateway/server-cron.test.ts src/cron/service/timer.regression.test.ts src/infra/restart-coordinator.test.ts src/infra/infra-runtime.test.ts --reporter verbose
  • pnpm test src/cron/service/timer.regression.test.ts src/gateway/server-cron.test.ts
  • pnpm tsgo:core
  • git diff --check

Note: pnpm check:changed was also attempted locally before this PR, but the local Blacksmith Testbox wrapper failed its own --version/--help sanity check before project checks could run.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 1, 2026
@clawsweeper

clawsweeper Bot commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 14, 2026, 6:25 AM ET / 10:25 UTC.

Summary
The PR detects isolated cron setup timeouts, asks the existing safe Gateway restart coordinator to recover, adds cron active-run drain/reset handling during restart, and expands cron/gateway regression coverage.

PR surface: Source +876, Tests +1377. Total +2253 across 28 files.

Reproducibility: yes. for the source-level path: current main has the isolated cron setup-timeout watchdog/error path and no restart hook. I did not run a live repro in this read-only review, and the live proof in the PR body is stale relative to current head.

Review metrics: 2 noteworthy metrics.

  • Automatic restart trigger: 1 added. A cron setup-timeout path can now request Gateway restart without an operator command, so maintainers should review availability behavior before merge.
  • Current-head proof: 0 matching proof entries; body names 354de1a, head is 13d06b5. The supplied proof does not cover the refreshed lifecycle-reset diff now under review.

Stored data model
Persistent data-model change detected: serialized state: src/cron/service/state.ts, unknown-data-model-change: src/cron/service/state.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Add redacted current-head real or built-runtime proof for head 13d06b5 showing setup-timeout restart scheduling and Gateway recovery.
  • Show the refreshed cron lifecycle reset path: active cron task abort, active marker drain/reset, and scheduler re-admission after restart.
  • Get explicit maintainer acceptance of the automatic safe-restart policy for this timeout class.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR includes copied live/build output, but it names old head 354de1a and does not prove current head 13d06b5 after the lifecycle-reset refresh; update the PR body with redacted current-head terminal/live output.

Risk before merge

  • [P1] The PR body's real behavior proof is tied to old head 354de1a, while the current head is 13d06b5 and includes broader cron lifecycle drain/reset behavior that is not demonstrated in the PR body.
  • [P2] Merging adds a non-operator path where one isolated cron setup-timeout class can request a Gateway restart and pause cron scheduling while recovery is pending; maintainers need to accept that availability tradeoff.
  • [P1] The restart coordinator and in-process lifecycle reset paths are broad runtime surfaces, so current-head packaged or built-runtime proof would materially reduce restart-loop and scheduler-regression uncertainty.

Maintainer options:

  1. Require Current-Head Restart Proof (recommended)
    Before merge, require redacted terminal/live or built-runtime proof on head 13d06b5 showing setup-timeout restart scheduling, Gateway recovery, and cron drain/reset re-admission.
  2. Accept The Automatic Recovery Policy
    Maintainers can explicitly accept that this setup-timeout class is severe enough to request the existing safe restart coordinator once current-head proof is refreshed.
  3. Narrow Or Pause Recovery
    If one setup timeout is too broad a restart signal, pause or rework the PR around a repeat threshold, health check, opt-in strict mode, or the existing alert-only behavior.

Next step before merge

  • [P1] Human maintainer review is needed for current-head proof and the automatic Gateway restart policy; there is no narrow mechanical repair for ClawSweeper to apply safely.

Security
Cleared: No concrete security or supply-chain concern was found; the diff adds no dependencies, workflows, secrets handling, or new third-party code execution surface.

Review details

Best possible solution:

Merge only after maintainers accept automatic safe restart for this failure class and current-head real or built-runtime proof demonstrates setup-timeout restart plus cron lifecycle drain/reset recovery.

Do we have a high-confidence way to reproduce the issue?

Yes for the source-level path: current main has the isolated cron setup-timeout watchdog/error path and no restart hook. I did not run a live repro in this read-only review, and the live proof in the PR body is stale relative to current head.

Is this the best way to solve the issue?

Mostly yes on code shape: the PR uses a cron-owned callback and the existing safe restart coordinator instead of inventing a parallel restart path. The remaining question is policy and proof, not a clearly better implementation layer.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 37e3e895b020.

Label changes

Label changes:

  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.

Label justifications:

  • P1: The PR targets a recurring Gateway cron setup-timeout failure affecting scheduled agent workflows in real user-style operation.
  • merge-risk: 🚨 availability: The diff can cause automatic Gateway restart scheduling and cron admission pauses from a cron timeout path, which can affect process availability.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR includes copied live/build output, but it names old head 354de1a and does not prove current head 13d06b5 after the lifecycle-reset refresh; update the PR body with redacted current-head terminal/live output.
Evidence reviewed

PR surface:

Source +876, Tests +1377. Total +2253 across 28 files.

View PR surface stats
Area Files Added Removed Net
Source 19 984 108 +876
Tests 9 1383 6 +1377
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 28 2367 114 +2253

What I checked:

  • Repository policy applied: Read the full root AGENTS.md plus scoped agent, embedded-runner, and gateway guides; the review applied the availability-sensitive Gateway, whole-path review, and real-behavior-proof guidance. (AGENTS.md:1, 37e3e895b020)
  • Current main behavior: Current main has the cron setup watchdog returning a setup-timeout error from executeJobCoreWithTimeout without any Gateway restart callback, so the central behavior remains unsolved on main. (src/cron/service/timer.ts:181, 37e3e895b020)
  • PR setup-timeout classification: The PR head records isolated setup timeouts, suppresses lane-wait false positives, and carries an isolatedAgentSetupTimeout signal into cron finalization. (src/cron/service/timer.ts:276, 13d06b5d6f9b)
  • PR Gateway restart hook: Gateway cron wires the new setup-timeout callback to requestSafeGatewayRestart with reason cron.isolated_agent_setup_timeout and logs restart preflight details. (src/gateway/server-cron.ts:549, 13d06b5d6f9b)
  • Restart safety path: The safe restart coordinator counts active cron runs as restart blockers, while Gateway pre-restart deferral includes cron active count at PR head. (src/infra/restart-coordinator.ts:87, 13d06b5d6f9b)
  • Restart lifecycle reset: The PR head aborts/drains active cron task runs, waits for active cron markers, retires cron task tracking, and resets cron active jobs during in-process Gateway restart iteration. (src/cli/gateway-cli/run-loop.ts:802, 13d06b5d6f9b)

Likely related people:

  • shakkernerd: Recent path history shows multiple cron timeout/cancellation commits adjacent to the new cron task drain and settlement tracking behavior. (role: recent cron cancellation contributor; confidence: medium; commits: 9082233a433c, 24196e05f59b, 372f85d368d8; files: src/cron/service/timer.ts, src/tasks/cron-task-cancel.ts)
  • joshavant: Recent current-main history touched the same cron timer and gateway cron surfaces for disabled heartbeat one-shot retry behavior. (role: recent cron timer contributor; confidence: medium; commits: e728957989f6; files: src/cron/service/timer.ts, src/gateway/server-cron.ts)
  • openperf: Recent restart-coordinator history added session-routing guard behavior around coalesced Gateway restart continuations, adjacent to this PR's safe restart scheduling path. (role: restart coordinator contributor; confidence: medium; commits: afcbdd7416ac; files: src/infra/restart-coordinator.ts, src/infra/restart.ts)
  • ooiuuii: Recent Gateway restart lifecycle history changed active run ownership and cron lifecycle preservation across restart, adjacent to this PR's drain/reset additions. (role: restart lifecycle adjacent contributor; confidence: medium; commits: d20fdf3b3842; files: src/cli/gateway-cli/run-loop.ts, src/cron/active-jobs.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 1, 2026
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 1, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 1, 2026
@ghitafilali
ghitafilali force-pushed the codex/openclaw-gateway-isolated-runner-health branch from b370977 to 853a75b Compare June 1, 2026 10:56
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 1, 2026
@ghitafilali

ghitafilali commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review\n\nAdded PR-head proof to the Real behavior proof section: verbose gateway/cron/infra test output now shows the setup-timeout callback requests safe restart, and the existing restart layer coalesces/defers repeated restart requests.

@clawsweeper

clawsweeper Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 1, 2026
@ghitafilali

ghitafilali commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

I updated the PR body with stronger PR-build proof in the Real behavior proof section.

New evidence added:

  • node scripts/build-all.mjs passed on PR head 354de1a3b99027027b303c29b8852ec8fef9091d.
  • node --import tsx /tmp/openclaw-pr-89055-setup-timeout-proof.mjs ran a controlled setup stall through the real cron timeout path and the real safe restart scheduler.
  • The proof output showed productionBuildCompleted=true, distContainsGatewayHook=true, setup timeout after 60282ms, restart status=scheduled, preflightSafe=true, restart reason cron.isolated_agent_setup_timeout, SIGUSR1 authorized=true, SIGUSR1 recovered=true, and all assertions true.
  • The PR-head selected proof still passes gateway 22/22, infra 34/34, and cron 43/43, including the coalescing/deferral restart coordinator checks.

Please re-evaluate proof quality/rank with this added built-runtime evidence. The remaining policy question is maintainer acceptance of the automatic safe Gateway restart tradeoff.

@clawsweeper

clawsweeper Bot commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jun 1, 2026
@mbelinky mbelinky mentioned this pull request Jun 3, 2026
Merged
@vincentkoc vincentkoc self-assigned this Jun 14, 2026
@vincentkoc
vincentkoc force-pushed the codex/openclaw-gateway-isolated-runner-health branch from 354de1a to 7231c3f Compare June 14, 2026 09:50
@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes agents Agent runtime and tooling size: XL and removed size: S labels Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 14, 2026
@vincentkoc
vincentkoc force-pushed the codex/openclaw-gateway-isolated-runner-health branch from 7231c3f to 13d06b5 Compare June 14, 2026 10:18
@vincentkoc

vincentkoc commented Jun 14, 2026

Copy link
Copy Markdown
Member

Maintainer fixes pushed to this PR branch.

Final head SHA: 13d06b5

Verification:

  • Focused cron/tasks/CLI regression suite: node scripts/run-vitest.mjs src/cron/service/agent-watchdog.test.ts src/cli/gateway-cli/run-loop.test.ts src/cron/service/timer.regression.test.ts src/cron/service/ops.regression.test.ts src/cron/active-jobs-manual-run.test.ts src/tasks/cron-task-cancel.test.ts --maxWorkers=1
  • Gateway filesystem/session regression suite: node scripts/run-vitest.mjs src/gateway/session-utils.fs.test.ts --maxWorkers=1
  • Test typecheck: node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.test.src.json --pretty false
  • Core lint shards: node scripts/run-oxlint-shards.mjs --only=core --split-core
  • Whitespace: git diff --check
  • Autoreview: clean after fixes.
  • Changed gate: Blacksmith Testbox tbx_01kv2srgbex71w9ce5rwv2wtr4, run https://github.com/openclaw/openclaw/actions/runs/27495583749, exit 0.
  • GitHub PR checks on the final SHA: clean, 135 success / 1 neutral / 22 skipped / 0 failures.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 14, 2026
@vincentkoc
vincentkoc merged commit b2c5e79 into openclaw:main Jun 14, 2026
160 checks passed
This was referenced Jun 14, 2026
Closed
Open
Closed
Closed
This was referenced Jun 22, 2026
Closed
Closed
Open
Closed
This was referenced Jun 23, 2026
Closed
Merged
@steipete steipete mentioned this pull request Jul 18, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

agents Agent runtime and tooling cli CLI command changes gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P1 High-priority user-facing bug, regression, or broken workflow. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ghitafilali @vincentkoc