Skip to content

fix(gateway): preserve restart drain for active runs#71465

Merged
vincentkoc merged 1 commit into
mainfrom
fix/gateway-restart-drain-c117686
Apr 25, 2026
Merged

fix(gateway): preserve restart drain for active runs#71465
vincentkoc merged 1 commit into
mainfrom
fix/gateway-restart-drain-c117686

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

  • wait indefinitely by default before restart-required config reloads while active operations, replies, embedded runs, or task runs are still draining
  • preserve the same drain path when openclaw gateway restart goes through launchd/systemd/scheduled-task SIGTERM by writing a short-lived restart intent before invoking the service manager
  • keep explicit positive gateway.reload.deferralTimeoutMs as an opt-in force timeout, with periodic still-pending warnings while restart is deferred

Fixes #65485.
Related to #57556 and #57425.

Thanks @rijhsinghani for narrowing the SIGTERM restart path and @lmdeagles for the earlier graceful-restart work in #57556.

Tests

  • pnpm config:schema:check
  • pnpm config:docs:check
  • pnpm test:serial src/infra/infra-runtime.test.ts src/infra/restart.deferral-timeout.test.ts src/cli/gateway-cli/run-loop.test.ts src/cli/daemon-cli/lifecycle-core.test.ts src/gateway/config-reload.test.ts src/gateway/server-methods/config.shared-auth.test.ts src/gateway/server-restart-deferral.test.ts
  • OPENCLAW_VITEST_NO_OUTPUT_TIMEOUT_MS=600000 pnpm check:changed

@vincentkoc vincentkoc self-assigned this Apr 25, 2026
@vincentkoc
vincentkoc marked this pull request as ready for review April 25, 2026 07:32
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime cli CLI command changes agents Agent runtime and tooling labels Apr 25, 2026
@aisle-research-bot

aisle-research-bot Bot commented Apr 25, 2026

Copy link
Copy Markdown

🔒 Aisle Security Analysis

We found 3 potential security issue(s) in this PR:

# Severity Title
1 🟡 Medium Indefinite restart drain can block gateway restarts (DoS / patch prevention)
2 🟡 Medium Restart deferral can wait indefinitely, preventing restart signal emission (DoS)
3 🟡 Medium Untrusted state-dir override allows tampering with gateway restart intent file
1. 🟡 Indefinite restart drain can block gateway restarts (DoS / patch prevention)
Property Value
Severity Medium
CWE CWE-400
Location src/cli/gateway-cli/run-loop.ts:129-245

Description

The gateway shutdown/restart loop now allows restart drains to wait indefinitely when gateway.reload.deferralTimeoutMs is absent or non-positive.

  • resolveRestartDrainTimeoutMs() returns undefined unless a positive timeout is explicitly configured.
  • When the timeout is undefined, the code calls waitForActiveTasks(undefined) and waitForActiveEmbeddedRuns(undefined), both of which can wait indefinitely for in-flight work captured at call time.
  • The forceExitTimer watchdog is not armed during the drain phase for indefinite restarts (it is only armed later, right before server.close()), so a stuck/long-running task or embedded run can prevent the restart forever.

This creates an availability risk where an external actor who can cause long-running/stuck tasks (e.g., via requests that trigger work on the command queue or embedded runs) can indefinitely delay restarts/reloads, potentially blocking security updates or recovery actions.

Vulnerable code (restart drain with undefined timeout, no watchdog during drain):

const restartDrainTimeoutMs = isRestart ? resolveRestartDrainTimeoutMs() : 0;
...
if (activeTasks > 0 || activeRuns > 0) {
  const [tasksDrain, runsDrain] = await Promise.all([
    activeTasks > 0 ? waitForActiveTasks(restartDrainTimeoutMs) : Promise.resolve({ drained: true }),
    activeRuns > 0 ? waitForActiveEmbeddedRuns(restartDrainTimeoutMs) : Promise.resolve({ drained: true }),
  ]);
}
...
​// For indefinite restart, watchdog is only armed *after* draining
armCloseForceExitTimerForIndefiniteRestart();
await server?.close(...);

Recommendation

Reintroduce a bounded default for restart draining and/or ensure a watchdog is armed before entering the drain.

Options:

  1. Keep the previous safe default (e.g., 5 minutes) when config is absent:
const timeoutMs = loadConfig().gateway?.reload?.deferralTimeoutMs;
const restartDrainTimeoutMs =
  typeof timeoutMs === "number" && Number.isFinite(timeoutMs) && timeoutMs >= 0
    ? timeoutMs
    : DEFAULT_RESTART_DRAIN_TIMEOUT_MS;
  1. If supporting indefinite drain is desired, still arm a hard upper bound watchdog during the drain (distinct from the server.close watchdog), so a stuck task cannot block restarts forever:
// Always arm a force-exit watchdog for restart, even if drain timeout is undefined.
armForceExitTimer(SUPERVISOR_STOP_TIMEOUT_MS - 5_000);

Also consider making drain behavior resilient to untrusted/remote-triggerable work by:

  • restricting which tasks/runs can block restart
  • adding cancellation/abort mechanisms for tasks
  • emitting restart anyway after a maximum elapsed time
2. 🟡 Restart deferral can wait indefinitely, preventing restart signal emission (DoS)
Property Value
Severity Medium
CWE CWE-400
Location src/infra/restart.ts:378-437

Description

deferGatewayRestartUntilIdle() changed from a bounded wait (previously default 5 minutes) to an indefinite wait unless a positive maxWaitMs is explicitly provided.

  • maxWaitMs is set to undefined when opts.maxWaitMs is absent or non-positive.
  • The polling loop only triggers onTimeout and emits the restart signal when maxWaitMs is defined.
  • As a result, if pending work never drains (e.g., stuck tasks, hung embedded runs, or a workload that keeps pending > 0), the restart signal may never be emitted.

This can be abused (or triggered accidentally) to indefinitely block restarts initiated via the restart coordinator (e.g., config watcher or RPC path), impacting availability and the ability to apply updates.

Vulnerable code:

const maxWaitMs =
  typeof opts.maxWaitMs === "number" && Number.isFinite(opts.maxWaitMs) && opts.maxWaitMs > 0
    ? Math.max(pollMs, Math.floor(opts.maxWaitMs))
    : undefined;
...
if (maxWaitMs !== undefined && elapsedMs >= maxWaitMs) {
  ...
  void emitPreparedGatewayRestart(opts.emitHooks);
}

Recommendation

Maintain a finite default maximum wait for restart deferral, even when maxWaitMs is not provided, to ensure the system eventually makes progress.

For example, restore the previous default (5 minutes) unless explicitly configured:

const DEFAULT_DEFERRAL_MAX_WAIT_MS = 300_000;
const raw = opts.maxWaitMs;
const maxWaitMs =
  typeof raw === "number" && Number.isFinite(raw) && raw >= 0
    ? Math.max(pollMs, Math.floor(raw))
    : DEFAULT_DEFERRAL_MAX_WAIT_MS;

If supporting indefinite deferral is necessary, add a separate hard-stop watchdog and/or an operator override (force restart) to prevent untrusted workloads from blocking restarts indefinitely.

3. 🟡 Untrusted state-dir override allows tampering with gateway restart intent file
Property Value
Severity Medium
CWE CWE-732
Location src/infra/restart.ts:82-93

Description

The gateway restart/stop decision on SIGTERM is now influenced by a local file (gateway-restart-intent.json) stored under the state directory, which is resolved from OPENCLAW_STATE_DIR.

Because resolveStateDir(env) trusts env.OPENCLAW_STATE_DIR and consumeGatewayRestartIntentSync() does not validate file/directory ownership or permissions, an attacker who can write to the resolved state directory can:

  • Force behavior changes/DoS: make a SIGTERM be treated as a "restart" (draining) rather than a "stop" by pre-placing a valid intent file.
  • Persistently influence behavior: clearGatewayRestartIntentSync() refuses to unlink files with nlink > 1, so a hard-linked intent file cannot be removed by the consumer and can continue to affect subsequent SIGTERM handling.
  • In environments where the gateway/CLI is launched with elevated privileges and attacker-controlled environment (e.g., sudo -E, misconfigured service env), OPENCLAW_STATE_DIR could redirect writes to unexpected locations.

Vulnerable code (intent path derived from env; no ownership/permission checks; hardlink prevents clearing):

function resolveGatewayRestartIntentPath(env: NodeJS.ProcessEnv = process.env): string {
  return path.join(resolveStateDir(env), GATEWAY_RESTART_INTENT_FILENAME);
}

const stat = fs.lstatSync(intentPath);
if (!stat.isFile() || stat.size > GATEWAY_RESTART_INTENT_MAX_BYTES) {
  return false;
}
raw = fs.readFileSync(intentPath, "utf8");
...
clearGatewayRestartIntentSync(env);
const stat = fs.lstatSync(intentPath);
if (!stat.isFile() || stat.nlink > 1) {
  return false;
}
fs.unlinkSync(intentPath);

This is primarily a local integrity issue whose impact depends on how OPENCLAW_STATE_DIR is configured and whether the state dir is writable by other users (or the process runs with elevated privileges under an attacker-influenced environment).

Recommendation

Harden the trust boundary around the restart-intent control file.

  1. Constrain/validate the state directory when running in privileged contexts (or always):
  • Refuse to honor OPENCLAW_STATE_DIR when process.getuid?.() === 0 (or when running as a system service), or only allow it to point under an expected base directory.
  • Ensure the state directory is not group/world-writable and is owned by the current uid.
  1. Validate file ownership & mode before trusting content:
  • Require stat.uid === process.getuid() (when available)
  • Require (stat.mode & 0o077) === 0 (no group/world access)
  1. Make clearing robust:
  • Consider unlinking regardless of nlink (or at least treat nlink>1 as suspicious and ignore without using it).

Example hardening (POSIX-only checks guarded for portability):

const stat = fs.lstatSync(intentPath);
if (!stat.isFile() || stat.size > MAX_BYTES) return false;
if (typeof process.getuid === 'function') {
  if (stat.uid !== process.getuid()) return false;
  if ((stat.mode & 0o077) !== 0) return false;
}

Also consider opening/reading using openSync and (where supported) O_NOFOLLOW to reduce filesystem trickery risk.


Analyzed PR: #71465 at commit fe56249

Last updated on: 2026-04-25T08:36:15Z

@openclaw-barnacle openclaw-barnacle Bot added size: M maintainer Maintainer-authored PR labels Apr 25, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74e88c3b1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/daemon-cli/lifecycle-core.ts Outdated
Comment on lines +465 to +468
const wroteRestartIntent = params.serviceNoun === "Gateway";
if (wroteRestartIntent) {
writeGatewayRestartIntentSync();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Write restart intent using the managed service env

runServiceRestart writes the restart-intent marker with the caller's process.env, but the Gateway process reads it from its own environment (consumeGatewayRestartIntentSync() in the run loop). If the managed service uses a different OPENCLAW_STATE_DIR/home/profile than the shell that runs openclaw gateway restart, the marker is written to the wrong state directory, SIGTERM is handled as a normal stop, and in-flight work is not drained during service-manager restarts. This breaks the new restart-drain preservation path for non-default env deployments.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR changes the default behavior for restart-required config reloads from a 5-minute force timeout to indefinitely waiting for active operations to drain, and adds a short-lived restart intent file mechanism so that gateway restart-triggered SIGTERMs are recognized as restarts (not stops) by the run loop. The implementation is well-tested across all changed modules, and the onStillPending hook provides periodic log warnings to prevent silent indefinite hangs from going unnoticed.

Confidence Score: 4/5

Safe to merge; logic is correct and well-covered by tests, with only minor style and edge-case hardening suggestions.

No P0 or P1 issues found. The two P2 findings are: (1) a naming/intent-tracking nit in lifecycle-core.ts, and (2) an edge case in consumeGatewayRestartIntentSync where a non-ENOENT read failure silently destroys the intent file via the finally block. Both are low-impact in practice.

src/infra/restart.ts (consumeGatewayRestartIntentSync finally block), src/cli/daemon-cli/lifecycle-core.ts (wroteRestartIntent flag naming)

Comments Outside Diff (1)

  1. src/cli/daemon-cli/lifecycle-core.ts, line 172-183 (link)

    P2 wroteRestartIntent flag set before the actual write

    wroteRestartIntent is set to true based on params.serviceNoun === "Gateway" before writeGatewayRestartIntentSync() is called. Because writeGatewayRestartIntentSync silently swallows its own errors (it logs a warning internally but never throws), if the write fails the flag is still true. On a subsequent service.restart failure, clearGatewayRestartIntentSync is called to clean up a file that was never written — a no-op in practice but potentially misleading. Consider capturing the return value of writeGatewayRestartIntentSync or renaming to shouldWriteRestartIntent to reflect that it's a precondition, not a past-tense record of success.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: src/cli/daemon-cli/lifecycle-core.ts
    Line: 172-183
    
    Comment:
    **`wroteRestartIntent` flag set before the actual write**
    
    `wroteRestartIntent` is set to `true` based on `params.serviceNoun === "Gateway"` before `writeGatewayRestartIntentSync()` is called. Because `writeGatewayRestartIntentSync` silently swallows its own errors (it logs a warning internally but never throws), if the write fails the flag is still `true`. On a subsequent `service.restart` failure, `clearGatewayRestartIntentSync` is called to clean up a file that was never written — a no-op in practice but potentially misleading. Consider capturing the return value of `writeGatewayRestartIntentSync` or renaming to `shouldWriteRestartIntent` to reflect that it's a precondition, not a past-tense record of success.
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/cli/daemon-cli/lifecycle-core.ts
Line: 172-183

Comment:
**`wroteRestartIntent` flag set before the actual write**

`wroteRestartIntent` is set to `true` based on `params.serviceNoun === "Gateway"` before `writeGatewayRestartIntentSync()` is called. Because `writeGatewayRestartIntentSync` silently swallows its own errors (it logs a warning internally but never throws), if the write fails the flag is still `true`. On a subsequent `service.restart` failure, `clearGatewayRestartIntentSync` is called to clean up a file that was never written — a no-op in practice but potentially misleading. Consider capturing the return value of `writeGatewayRestartIntentSync` or renaming to `shouldWriteRestartIntent` to reflect that it's a precondition, not a past-tense record of success.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/infra/restart.ts
Line: 133-140

Comment:
**`finally` deletes the intent file even on read failures**

`clearGatewayRestartIntentSync` is called unconditionally in `finally`, including when `readFileSync` throws (e.g., a transient permission error where the file exists but is momentarily unreadable). In that case the function returns `false` — treating the SIGTERM as a stop — but the intent file is silently deleted, so a retry on the next poll cannot recover it. For ENOENT this is harmless, but for other `readFileSync` errors it could cause a restart intent to be permanently lost.

A simpler alternative that preserves "consume on success" while avoiding accidental deletion on read failures:

```ts
let raw: string;
try {
  raw = fs.readFileSync(intentPath, "utf8");
} catch {
  return false;
}
clearGatewayRestartIntentSync(env);
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(gateway): preserve restart drain for..." | Re-trigger Greptile

Comment thread src/infra/restart.ts
Comment on lines +133 to +140
let raw: string;
try {
raw = fs.readFileSync(intentPath, "utf8");
} catch {
return false;
} finally {
clearGatewayRestartIntentSync(env);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 finally deletes the intent file even on read failures

clearGatewayRestartIntentSync is called unconditionally in finally, including when readFileSync throws (e.g., a transient permission error where the file exists but is momentarily unreadable). In that case the function returns false — treating the SIGTERM as a stop — but the intent file is silently deleted, so a retry on the next poll cannot recover it. For ENOENT this is harmless, but for other readFileSync errors it could cause a restart intent to be permanently lost.

A simpler alternative that preserves "consume on success" while avoiding accidental deletion on read failures:

let raw: string;
try {
  raw = fs.readFileSync(intentPath, "utf8");
} catch {
  return false;
}
clearGatewayRestartIntentSync(env);
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/infra/restart.ts
Line: 133-140

Comment:
**`finally` deletes the intent file even on read failures**

`clearGatewayRestartIntentSync` is called unconditionally in `finally`, including when `readFileSync` throws (e.g., a transient permission error where the file exists but is momentarily unreadable). In that case the function returns `false` — treating the SIGTERM as a stop — but the intent file is silently deleted, so a retry on the next poll cannot recover it. For ENOENT this is harmless, but for other `readFileSync` errors it could cause a restart intent to be permanently lost.

A simpler alternative that preserves "consume on success" while avoiding accidental deletion on read failures:

```ts
let raw: string;
try {
  raw = fs.readFileSync(intentPath, "utf8");
} catch {
  return false;
}
clearGatewayRestartIntentSync(env);
```

How can I resolve this? If you propose a fix, please make it concise.

@vincentkoc
vincentkoc force-pushed the fix/gateway-restart-drain-c117686 branch 3 times, most recently from 092c5c4 to 10df5c5 Compare April 25, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling cli CLI command changes docs Improvements or additions to documentation gateway Gateway runtime maintainer Maintainer-authored PR size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway SIGTERM-restarts kill in-flight agent runs on non-critical config changes

1 participant