Fix/95784 cron setup timeout no restart#95813
Closed
hanZeng-08 wants to merge 2 commits into
Closed
Conversation
When an isolated cron agent setup timed out, onIsolatedAgentSetupTimeout unconditionally called requestSafeGatewayRestart with delayMs: 0, destroying in-flight embedded runs and causing a restart-loop under event-loop saturation. This change removes the gateway restart request from the handler and lets the cron job return an error status, which triggers the existing error-backoff path in applyJobResult. The cron service keeps scheduling; only the failing job is delayed, preserving all other in-flight work. - server-cron.ts: onIsolatedAgentSetupTimeout now logs a warning only - timer.ts: maybeNotifyIsolatedAgentSetupTimeout no longer sets restartRecoveryPending, so the cron scheduler is not frozen - Tests updated to assert no-restart behavior and continued scheduling Fixes openclaw#95784
hanZeng-08
force-pushed
the
fix/95784-cron-setup-timeout-no-restart
branch
from
June 23, 2026 10:31
c1d18d3 to
bd634fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #95784
Summary
Problem: When an isolated cron agent failed to complete its setup phase within the configured timeout,
onIsolatedAgentSetupTimeoutunconditionally calledrequestSafeGatewayRestartwithdelayMs: 0. This destroyed in-flight embedded runs and caused a restart-loop under event-loop saturation, because the root cause was event-loop starvation rather than gateway state corruption. After restart, the same workload re-saturated the loop and the next cron tick re-wedged the gateway.Solution: Remove the gateway restart request from the setup-timeout handler. Let the cron job return an error status, which triggers the existing error-backoff path in
applyJobResult. The cron service keeps scheduling; only the failing job is delayed with exponential backoff, preserving all other in-flight work and avoiding the restart amplification loop.What changed:
src/gateway/server-cron.ts:onIsolatedAgentSetupTimeoutnow logs a warning only, no longer callsrequestSafeGatewayRestartsrc/cron/service/timer.ts:maybeNotifyIsolatedAgentSetupTimeoutno longer setsstate.restartRecoveryPending, so the cron scheduler is not frozen and the timer continues to re-armsrc/gateway/server-cron.test.ts: Updated to assert no-restart behavior and job error/backoff statesrc/cron/service/timer.regression.test.ts: Updated to assert continued scheduling after setup timeout without gateway restartWhat did NOT change: The cron watchdog
CRON_AGENT_SETUP_WATCHDOG_MS(60s default), the setup-timeout detection logic, the cleanup path for timed-out runs, or the error-backoff schedule inapplyJobResult. TheonIsolatedAgentSetupTimeoutcallback still fires for observability, but its default action is now a no-op instead of a restart request.Fixes #95784
Real behavior proof
Behavior addressed: Isolated cron agent setup timeout no longer triggers a full gateway restart; instead the job errors and backs off, while the scheduler keeps running other jobs.
Real environment tested: Linux, Node v22.22.3, OpenClaw worktree SHA c1d18d3
Exact steps or command run after this patch:
Evidence after fix:
All 112 tests pass. Key regression assertions:
does not restart gateway on isolated cron setup timeout; job errors and backs off instead— verifiesrequestSafeGatewayRestartis NOT called, job getslastRunStatus: "error", andconsecutiveErrors >= 1suppresses scheduled rearm after manual setup-timeout restart request— renamed and updated to assertrestartRecoveryPending: false,timeris NOT null, and scheduled jobs are NOT blockedcontinues scheduled batch after manual setup-timeout without requesting gateway restart— verifies that a setup timeout on one job does not freeze the scheduler; the next scheduled job still runs and completes withstatus: "ok"Observed result after fix: After an isolated cron setup timeout, the cron job is marked as error with exponential backoff, but the gateway keeps running, the timer re-arms, and other scheduled jobs proceed normally. No
requestSafeGatewayRestartcall is made.What was not tested: Full gateway startup with systemd
Restart=alwaysand live event-loop saturation under production workload. The fix is verified through unit and regression tests; the behavior change is deterministic because the restart call was unconditional and is now removed.Regression Test Plan
Coverage level: Gateway server tests + Cron timer regression tests
Target test files:
src/gateway/server-cron.test.ts— 24 tests, 2 test projects (gateway-server, gateway-methods)src/cron/service/timer.regression.test.ts— 64 testsScenario locked in:
requestSafeGatewayRestartlastRunStatus: "error"andconsecutiveErrors >= 1, triggering backoffrestartRecoveryPendingstays false)Why this is the smallest reliable guardrail: The tests cover the exact two boundaries that were changed:
server-cron.ts) no longer calls restarttimer.ts) no longer freezes the schedulerPlus the end-to-end scheduling behavior that was previously blocked by
restartRecoveryPending.Root Cause
Root cause:
onIsolatedAgentSetupTimeoutinserver-cron.tsunconditionally calledrequestSafeGatewayRestart({ delayMs: 0, preservePendingEmitHooks: true })on every isolated cron setup timeout. This treated a single tick-level timeout as a process-level failure, even though the dominant failure mode was event-loop saturation (process-wide workload pressure, not gateway state corruption).Missing detection / guardrail: The code did not distinguish between:
There was no signal disambiguation, no adaptive timeout based on event-loop health, and no degradation path that preserved in-flight work. The
requestSafeGatewayRestartcall withdelayMs: 0skipped the active-work drain and killed embedded runs that were not related to the cron job.