Skip to content

Fix/95784 cron setup timeout no restart#95813

Closed
hanZeng-08 wants to merge 2 commits into
openclaw:mainfrom
hanZeng-08:fix/95784-cron-setup-timeout-no-restart
Closed

Fix/95784 cron setup timeout no restart#95813
hanZeng-08 wants to merge 2 commits into
openclaw:mainfrom
hanZeng-08:fix/95784-cron-setup-timeout-no-restart

Conversation

@hanZeng-08

@hanZeng-08 hanZeng-08 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Fixes #95784

Summary

Problem: When an isolated cron agent failed to complete its setup phase within the configured timeout, onIsolatedAgentSetupTimeout unconditionally called requestSafeGatewayRestart with delayMs: 0. This destroyed in-flight embedded runs and caused a restart-loop under event-loop saturation, because the root cause was event-loop starvation rather than gateway state corruption. After restart, the same workload re-saturated the loop and the next cron tick re-wedged the gateway.

Solution: Remove the gateway restart request from the setup-timeout handler. Let the cron job return an error status, which triggers the existing error-backoff path in applyJobResult. The cron service keeps scheduling; only the failing job is delayed with exponential backoff, preserving all other in-flight work and avoiding the restart amplification loop.

What changed:

  • src/gateway/server-cron.ts: onIsolatedAgentSetupTimeout now logs a warning only, no longer calls requestSafeGatewayRestart
  • src/cron/service/timer.ts: maybeNotifyIsolatedAgentSetupTimeout no longer sets state.restartRecoveryPending, so the cron scheduler is not frozen and the timer continues to re-arm
  • src/gateway/server-cron.test.ts: Updated to assert no-restart behavior and job error/backoff state
  • src/cron/service/timer.regression.test.ts: Updated to assert continued scheduling after setup timeout without gateway restart

What did NOT change: The cron watchdog CRON_AGENT_SETUP_WATCHDOG_MS (60s default), the setup-timeout detection logic, the cleanup path for timed-out runs, or the error-backoff schedule in applyJobResult. The onIsolatedAgentSetupTimeout callback still fires for observability, but its default action is now a no-op instead of a restart request.

Fixes #95784

Real behavior proof

Behavior addressed: Isolated cron agent setup timeout no longer triggers a full gateway restart; instead the job errors and backs off, while the scheduler keeps running other jobs.

Real environment tested: Linux, Node v22.22.3, OpenClaw worktree SHA c1d18d3

Exact steps or command run after this patch:

 pnpm test src/gateway/server-cron.test.ts --run
 pnpm test src/cron/service/timer.regression.test.ts --run

Evidence after fix:

[test] starting test/vitest/vitest.gateway.config.ts
 ✓ |gateway-server| ../../src/gateway/server-cron.test.ts (24 tests) 772ms
 ✓ |gateway-methods| ../../src/gateway/server-cron.test.ts (24 tests) 644ms
 Test Files  2 passed (2)
      Tests  48 passed (48)

[test] starting test/vitest/vitest.cron.config.ts
 ✓ |cron| src/cron/service/timer.regression.test.ts (64 tests) 2654ms
 Test Files  1 passed (1)
      Tests  64 passed (64)

All 112 tests pass. Key regression assertions:

  • does not restart gateway on isolated cron setup timeout; job errors and backs off instead — verifies requestSafeGatewayRestart is NOT called, job gets lastRunStatus: "error", and consecutiveErrors >= 1
  • suppresses scheduled rearm after manual setup-timeout restart request — renamed and updated to assert restartRecoveryPending: false, timer is NOT null, and scheduled jobs are NOT blocked
  • continues scheduled batch after manual setup-timeout without requesting gateway restart — verifies that a setup timeout on one job does not freeze the scheduler; the next scheduled job still runs and completes with status: "ok"

Observed result after fix: After an isolated cron setup timeout, the cron job is marked as error with exponential backoff, but the gateway keeps running, the timer re-arms, and other scheduled jobs proceed normally. No requestSafeGatewayRestart call is made.

What was not tested: Full gateway startup with systemd Restart=always and live event-loop saturation under production workload. The fix is verified through unit and regression tests; the behavior change is deterministic because the restart call was unconditional and is now removed.

Regression Test Plan

Coverage level: Gateway server tests + Cron timer regression tests

Target test files:

  • src/gateway/server-cron.test.ts — 24 tests, 2 test projects (gateway-server, gateway-methods)
  • src/cron/service/timer.regression.test.ts — 64 tests

Scenario locked in:

  1. Isolated cron setup timeout does NOT trigger requestSafeGatewayRestart
  2. The failing job gets lastRunStatus: "error" and consecutiveErrors >= 1, triggering backoff
  3. The cron scheduler timer is NOT null after setup timeout (restartRecoveryPending stays false)
  4. Concurrent scheduled jobs continue to be admitted and run after a setup timeout on another job

Why this is the smallest reliable guardrail: The tests cover the exact two boundaries that were changed:

  • The gateway handler (server-cron.ts) no longer calls restart
  • The cron timer (timer.ts) no longer freezes the scheduler
    Plus the end-to-end scheduling behavior that was previously blocked by restartRecoveryPending.

Root Cause

Root cause: onIsolatedAgentSetupTimeout in server-cron.ts unconditionally called requestSafeGatewayRestart({ delayMs: 0, preservePendingEmitHooks: true }) on every isolated cron setup timeout. This treated a single tick-level timeout as a process-level failure, even though the dominant failure mode was event-loop saturation (process-wide workload pressure, not gateway state corruption).

Missing detection / guardrail: The code did not distinguish between:

  • "gateway state is corrupt" (rare, warrants restart)
  • "event loop is saturated by concurrent embedded runs" (common, restart makes it worse)

There was no signal disambiguation, no adaptive timeout based on event-loop health, and no degradation path that preserved in-flight work. The requestSafeGatewayRestart call with delayMs: 0 skipped the active-work drain and killed embedded runs that were not related to the cron job.


@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: XS labels Jun 22, 2026
vincentkoc and others added 2 commits June 23, 2026 18:29
When an isolated cron agent setup timed out, onIsolatedAgentSetupTimeout
unconditionally called requestSafeGatewayRestart with delayMs: 0, destroying
in-flight embedded runs and causing a restart-loop under event-loop saturation.

This change removes the gateway restart request from the handler and lets the
cron job return an error status, which triggers the existing error-backoff path
in applyJobResult. The cron service keeps scheduling; only the failing job
is delayed, preserving all other in-flight work.

- server-cron.ts: onIsolatedAgentSetupTimeout now logs a warning only
- timer.ts: maybeNotifyIsolatedAgentSetupTimeout no longer sets
  restartRecoveryPending, so the cron scheduler is not frozen
- Tests updated to assert no-restart behavior and continued scheduling

Fixes openclaw#95784
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime size: XS

Projects

None yet

2 participants