fix(gateway): preserve restart drain for active runs#71465
Conversation
🔒 Aisle Security AnalysisWe found 3 potential security issue(s) in this PR:
1. 🟡 Indefinite restart drain can block gateway restarts (DoS / patch prevention)
DescriptionThe gateway shutdown/restart loop now allows restart drains to wait indefinitely when
This creates an availability risk where an external actor who can cause long-running/stuck tasks (e.g., via requests that trigger work on the command queue or embedded runs) can indefinitely delay restarts/reloads, potentially blocking security updates or recovery actions. Vulnerable code (restart drain with undefined timeout, no watchdog during drain): const restartDrainTimeoutMs = isRestart ? resolveRestartDrainTimeoutMs() : 0;
...
if (activeTasks > 0 || activeRuns > 0) {
const [tasksDrain, runsDrain] = await Promise.all([
activeTasks > 0 ? waitForActiveTasks(restartDrainTimeoutMs) : Promise.resolve({ drained: true }),
activeRuns > 0 ? waitForActiveEmbeddedRuns(restartDrainTimeoutMs) : Promise.resolve({ drained: true }),
]);
}
...
// For indefinite restart, watchdog is only armed *after* draining
armCloseForceExitTimerForIndefiniteRestart();
await server?.close(...);RecommendationReintroduce a bounded default for restart draining and/or ensure a watchdog is armed before entering the drain. Options:
const timeoutMs = loadConfig().gateway?.reload?.deferralTimeoutMs;
const restartDrainTimeoutMs =
typeof timeoutMs === "number" && Number.isFinite(timeoutMs) && timeoutMs >= 0
? timeoutMs
: DEFAULT_RESTART_DRAIN_TIMEOUT_MS;
// Always arm a force-exit watchdog for restart, even if drain timeout is undefined.
armForceExitTimer(SUPERVISOR_STOP_TIMEOUT_MS - 5_000);Also consider making drain behavior resilient to untrusted/remote-triggerable work by:
2. 🟡 Restart deferral can wait indefinitely, preventing restart signal emission (DoS)
Description
This can be abused (or triggered accidentally) to indefinitely block restarts initiated via the restart coordinator (e.g., config watcher or RPC path), impacting availability and the ability to apply updates. Vulnerable code: const maxWaitMs =
typeof opts.maxWaitMs === "number" && Number.isFinite(opts.maxWaitMs) && opts.maxWaitMs > 0
? Math.max(pollMs, Math.floor(opts.maxWaitMs))
: undefined;
...
if (maxWaitMs !== undefined && elapsedMs >= maxWaitMs) {
...
void emitPreparedGatewayRestart(opts.emitHooks);
}RecommendationMaintain a finite default maximum wait for restart deferral, even when For example, restore the previous default (5 minutes) unless explicitly configured: const DEFAULT_DEFERRAL_MAX_WAIT_MS = 300_000;
const raw = opts.maxWaitMs;
const maxWaitMs =
typeof raw === "number" && Number.isFinite(raw) && raw >= 0
? Math.max(pollMs, Math.floor(raw))
: DEFAULT_DEFERRAL_MAX_WAIT_MS;If supporting indefinite deferral is necessary, add a separate hard-stop watchdog and/or an operator override (force restart) to prevent untrusted workloads from blocking restarts indefinitely. 3. 🟡 Untrusted state-dir override allows tampering with gateway restart intent file
DescriptionThe gateway restart/stop decision on SIGTERM is now influenced by a local file ( Because
Vulnerable code (intent path derived from env; no ownership/permission checks; hardlink prevents clearing): function resolveGatewayRestartIntentPath(env: NodeJS.ProcessEnv = process.env): string {
return path.join(resolveStateDir(env), GATEWAY_RESTART_INTENT_FILENAME);
}
const stat = fs.lstatSync(intentPath);
if (!stat.isFile() || stat.size > GATEWAY_RESTART_INTENT_MAX_BYTES) {
return false;
}
raw = fs.readFileSync(intentPath, "utf8");
...
clearGatewayRestartIntentSync(env);const stat = fs.lstatSync(intentPath);
if (!stat.isFile() || stat.nlink > 1) {
return false;
}
fs.unlinkSync(intentPath);This is primarily a local integrity issue whose impact depends on how RecommendationHarden the trust boundary around the restart-intent control file.
Example hardening (POSIX-only checks guarded for portability): const stat = fs.lstatSync(intentPath);
if (!stat.isFile() || stat.size > MAX_BYTES) return false;
if (typeof process.getuid === 'function') {
if (stat.uid !== process.getuid()) return false;
if ((stat.mode & 0o077) !== 0) return false;
}Also consider opening/reading using Analyzed PR: #71465 at commit Last updated on: 2026-04-25T08:36:15Z |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 74e88c3b1a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const wroteRestartIntent = params.serviceNoun === "Gateway"; | ||
| if (wroteRestartIntent) { | ||
| writeGatewayRestartIntentSync(); | ||
| } |
There was a problem hiding this comment.
Write restart intent using the managed service env
runServiceRestart writes the restart-intent marker with the caller's process.env, but the Gateway process reads it from its own environment (consumeGatewayRestartIntentSync() in the run loop). If the managed service uses a different OPENCLAW_STATE_DIR/home/profile than the shell that runs openclaw gateway restart, the marker is written to the wrong state directory, SIGTERM is handled as a normal stop, and in-flight work is not drained during service-manager restarts. This breaks the new restart-drain preservation path for non-default env deployments.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR changes the default behavior for restart-required config reloads from a 5-minute force timeout to indefinitely waiting for active operations to drain, and adds a short-lived restart intent file mechanism so that Confidence Score: 4/5Safe to merge; logic is correct and well-covered by tests, with only minor style and edge-case hardening suggestions. No P0 or P1 issues found. The two P2 findings are: (1) a naming/intent-tracking nit in lifecycle-core.ts, and (2) an edge case in consumeGatewayRestartIntentSync where a non-ENOENT read failure silently destroys the intent file via the finally block. Both are low-impact in practice. src/infra/restart.ts (consumeGatewayRestartIntentSync finally block), src/cli/daemon-cli/lifecycle-core.ts (wroteRestartIntent flag naming)
|
| let raw: string; | ||
| try { | ||
| raw = fs.readFileSync(intentPath, "utf8"); | ||
| } catch { | ||
| return false; | ||
| } finally { | ||
| clearGatewayRestartIntentSync(env); | ||
| } |
There was a problem hiding this comment.
finally deletes the intent file even on read failures
clearGatewayRestartIntentSync is called unconditionally in finally, including when readFileSync throws (e.g., a transient permission error where the file exists but is momentarily unreadable). In that case the function returns false — treating the SIGTERM as a stop — but the intent file is silently deleted, so a retry on the next poll cannot recover it. For ENOENT this is harmless, but for other readFileSync errors it could cause a restart intent to be permanently lost.
A simpler alternative that preserves "consume on success" while avoiding accidental deletion on read failures:
let raw: string;
try {
raw = fs.readFileSync(intentPath, "utf8");
} catch {
return false;
}
clearGatewayRestartIntentSync(env);Prompt To Fix With AI
This is a comment left during a code review.
Path: src/infra/restart.ts
Line: 133-140
Comment:
**`finally` deletes the intent file even on read failures**
`clearGatewayRestartIntentSync` is called unconditionally in `finally`, including when `readFileSync` throws (e.g., a transient permission error where the file exists but is momentarily unreadable). In that case the function returns `false` — treating the SIGTERM as a stop — but the intent file is silently deleted, so a retry on the next poll cannot recover it. For ENOENT this is harmless, but for other `readFileSync` errors it could cause a restart intent to be permanently lost.
A simpler alternative that preserves "consume on success" while avoiding accidental deletion on read failures:
```ts
let raw: string;
try {
raw = fs.readFileSync(intentPath, "utf8");
} catch {
return false;
}
clearGatewayRestartIntentSync(env);
```
How can I resolve this? If you propose a fix, please make it concise.092c5c4 to
10df5c5
Compare
10df5c5 to
fe56249
Compare
Summary
openclaw gateway restartgoes through launchd/systemd/scheduled-task SIGTERM by writing a short-lived restart intent before invoking the service managergateway.reload.deferralTimeoutMsas an opt-in force timeout, with periodic still-pending warnings while restart is deferredFixes #65485.
Related to #57556 and #57425.
Thanks @rijhsinghani for narrowing the SIGTERM restart path and @lmdeagles for the earlier graceful-restart work in #57556.
Tests
pnpm config:schema:checkpnpm config:docs:checkpnpm test:serial src/infra/infra-runtime.test.ts src/infra/restart.deferral-timeout.test.ts src/cli/gateway-cli/run-loop.test.ts src/cli/daemon-cli/lifecycle-core.test.ts src/gateway/config-reload.test.ts src/gateway/server-methods/config.shared-auth.test.ts src/gateway/server-restart-deferral.test.tsOPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=600000 pnpm check:changed