feat: graceful gateway restart with session recovery#57556
Conversation
Adds a 5-part restart protection system that eliminates silent work loss during gateway restarts. 1. Pre-restart manifest — before drain, enumerate active sessions, cron runs, and subagents; write restart-manifest.json to state dir 2. Post-restart recovery — on startup, read manifest, inject system events into interrupted sessions, re-queue cron, notify parents about killed subagents, delete manifest 3. Readiness gate — CLI and gateway tool warn when active work exists; requires --force / force:true to proceed 4. Drain-aware message queuing — messages arriving during drain are appended to manifest for post-restart replay 5. Configuration — gateway.restart.* config surface (6 keys, all opt-out with safe defaults) New files: src/gateway/restart-recovery.ts — core recovery module (~500 lines) src/agents/tools/gateway-tool.restart.test.ts — readiness gate tests Modified: 16 files across gateway tool, CLI lifecycle, run loop, dispatch, config schema, server startup, and restart infrastructure. 19 tests passing. Tested on macOS (Darwin arm64, LaunchAgent). Closes openclaw#57425
Greptile SummaryThis PR introduces a well-structured five-part restart recovery system for OpenClaw gateways: a pre-restart manifest, post-restart session recovery, a readiness gate, drain-aware message queuing, and matching configuration schema. The overall design is sound and addresses a genuine reliability gap for multi-agent deployments. Two issues were found during review:
Confidence Score: 4/5Safe to merge after fixing the manifestWriteChain chain-poisoning bug; all other findings are low-probability edge cases or style. One clear P1 defect: a single failed writeManifestFile permanently poisons the module-level manifestWriteChain, causing all subsequent drain-window messages to be silently dropped for the process lifetime. This directly undermines the drain-queue feature that is a stated goal of the PR. The fix is a one-liner. The P2 async-SIGTERM race is low probability but worth documenting. Everything else — config schema, readiness gate, tests, intent file lifecycle — looks solid. src/gateway/restart-recovery.ts (queueDrainRestartMessage chain poisoning), src/cli/gateway-cli/run-loop.ts (async SIGTERM handler race)
|
| Filename | Overview |
|---|---|
| src/gateway/restart-recovery.ts | New core module (498 lines). Contains a P1 chain-poisoning bug in queueDrainRestartMessage: a single write failure permanently rejects the module-level manifestWriteChain, silently dropping all subsequent drain messages for the process lifetime. |
| src/cli/gateway-cli/run-loop.ts | SIGTERM handler made async to check restart intent before deciding action. Introduces a race window where a concurrent signal can hijack the shutdown path, causing the manifest to be skipped. Logic for service-restart action (write manifest, drain, then stop) is otherwise correct. |
| src/agents/tools/gateway-tool.ts | Adds readiness-gate check and force parameter to the restart action. Implementation is clean and matches test coverage. |
| src/cli/daemon-cli/lifecycle.ts | Adds readiness gate check at the top of runDaemonRestart with JSON and plain-text output paths; correctly bypassed by --force. No issues found. |
| src/infra/restart.ts | Adds primeGatewayRestartReason / consumeGatewayRestartReason for passing the human-readable restart reason into the manifest. Simple, well-tested. |
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/gateway/restart-recovery.ts
Line: 436-444
Comment:
**`manifestWriteChain` is permanently poisoned on any write failure**
`manifestWriteChain` is a module-level singleton promise. If `writeManifestFile` throws (e.g. disk full, permission denied), `manifestWriteChain` is left as a rejected promise. Every subsequent call to `queueDrainRestartMessage` chains a `.then()` off a rejected promise — which is a no-op and produces a new rejected promise — so **all further drain messages are silently dropped** for the lifetime of the process.
The fix is to catch errors inside the chained operation so the chain always resolves:
```suggestion
manifestWriteChain = manifestWriteChain
.then(async () => {
const manifest = await readManifestFile(restartConfig.manifestPath);
if (!manifest) {
return;
}
manifest.queuedMessages.push(queued);
await writeManifestFile(restartConfig.manifestPath, manifest);
queuedToManifest = true;
})
.catch(() => {
// Prevent a write failure from permanently rejecting the chain.
});
```
With this change, `await manifestWriteChain` no longer rejects — but since `queueDrainRestartMessage` is already called with `.catch(() => false)` at the dispatch site, that's fine.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: src/cli/gateway-cli/run-loop.ts
Line: 192-201
Comment:
**Async SIGTERM handler races with concurrent signals**
`onSigterm` now defers the `request()` call until an async file-read completes. During that brief window, `shuttingDown` is still `false`, so a concurrent SIGINT (or a second SIGTERM from the supervisor retrying) will call `request("stop", ...)`, set `shuttingDown = true`, and start a clean shutdown — **without writing the restart manifest**. When the original SIGTERM's async resolves and calls `request("service-restart", ...)`, it is silently ignored by the `if (shuttingDown) return` guard.
The intent file has already been consumed (unlinked) by `consumeGatewayRestartIntent`, so on the next startup `clearGatewayRestartIntent` is a no-op and `recoverGatewayRestartManifest` finds no manifest, meaning session recovery is silently skipped for that restart.
This is unlikely in normal operation, but launchd/systemd will sometimes escalate with a second SIGTERM if the process hasn't exited quickly. Consider atomically setting a "SIGTERM in-flight" flag on entry and short-circuiting any concurrent signal path that would turn a service-restart into a plain stop.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "feat: graceful gateway restart with sess..." | Re-trigger Greptile
| manifestWriteChain = manifestWriteChain.then(async () => { | ||
| const manifest = await readManifestFile(restartConfig.manifestPath); | ||
| if (!manifest) { | ||
| return; | ||
| } | ||
| manifest.queuedMessages.push(queued); | ||
| await writeManifestFile(restartConfig.manifestPath, manifest); | ||
| queuedToManifest = true; | ||
| }); |
There was a problem hiding this comment.
manifestWriteChain is permanently poisoned on any write failure
manifestWriteChain is a module-level singleton promise. If writeManifestFile throws (e.g. disk full, permission denied), manifestWriteChain is left as a rejected promise. Every subsequent call to queueDrainRestartMessage chains a .then() off a rejected promise — which is a no-op and produces a new rejected promise — so all further drain messages are silently dropped for the lifetime of the process.
The fix is to catch errors inside the chained operation so the chain always resolves:
| manifestWriteChain = manifestWriteChain.then(async () => { | |
| const manifest = await readManifestFile(restartConfig.manifestPath); | |
| if (!manifest) { | |
| return; | |
| } | |
| manifest.queuedMessages.push(queued); | |
| await writeManifestFile(restartConfig.manifestPath, manifest); | |
| queuedToManifest = true; | |
| }); | |
| manifestWriteChain = manifestWriteChain | |
| .then(async () => { | |
| const manifest = await readManifestFile(restartConfig.manifestPath); | |
| if (!manifest) { | |
| return; | |
| } | |
| manifest.queuedMessages.push(queued); | |
| await writeManifestFile(restartConfig.manifestPath, manifest); | |
| queuedToManifest = true; | |
| }) | |
| .catch(() => { | |
| // Prevent a write failure from permanently rejecting the chain. | |
| }); |
With this change, await manifestWriteChain no longer rejects — but since queueDrainRestartMessage is already called with .catch(() => false) at the dispatch site, that's fine.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/restart-recovery.ts
Line: 436-444
Comment:
**`manifestWriteChain` is permanently poisoned on any write failure**
`manifestWriteChain` is a module-level singleton promise. If `writeManifestFile` throws (e.g. disk full, permission denied), `manifestWriteChain` is left as a rejected promise. Every subsequent call to `queueDrainRestartMessage` chains a `.then()` off a rejected promise — which is a no-op and produces a new rejected promise — so **all further drain messages are silently dropped** for the lifetime of the process.
The fix is to catch errors inside the chained operation so the chain always resolves:
```suggestion
manifestWriteChain = manifestWriteChain
.then(async () => {
const manifest = await readManifestFile(restartConfig.manifestPath);
if (!manifest) {
return;
}
manifest.queuedMessages.push(queued);
await writeManifestFile(restartConfig.manifestPath, manifest);
queuedToManifest = true;
})
.catch(() => {
// Prevent a write failure from permanently rejecting the chain.
});
```
With this change, `await manifestWriteChain` no longer rejects — but since `queueDrainRestartMessage` is already called with `.catch(() => false)` at the dispatch site, that's fine.
How can I resolve this? If you propose a fix, please make it concise.| const onSigterm = () => { | ||
| gatewayLog.info("signal SIGTERM received"); | ||
| request("stop", "SIGTERM"); | ||
| void consumeGatewayRestartIntent() | ||
| .then((intent) => { | ||
| request(intent ? "service-restart" : "stop", intent?.reason ?? "SIGTERM"); | ||
| }) | ||
| .catch(() => { | ||
| request("stop", "SIGTERM"); | ||
| }); | ||
| }; |
There was a problem hiding this comment.
Async SIGTERM handler races with concurrent signals
onSigterm now defers the request() call until an async file-read completes. During that brief window, shuttingDown is still false, so a concurrent SIGINT (or a second SIGTERM from the supervisor retrying) will call request("stop", ...), set shuttingDown = true, and start a clean shutdown — without writing the restart manifest. When the original SIGTERM's async resolves and calls request("service-restart", ...), it is silently ignored by the if (shuttingDown) return guard.
The intent file has already been consumed (unlinked) by consumeGatewayRestartIntent, so on the next startup clearGatewayRestartIntent is a no-op and recoverGatewayRestartManifest finds no manifest, meaning session recovery is silently skipped for that restart.
This is unlikely in normal operation, but launchd/systemd will sometimes escalate with a second SIGTERM if the process hasn't exited quickly. Consider atomically setting a "SIGTERM in-flight" flag on entry and short-circuiting any concurrent signal path that would turn a service-restart into a plain stop.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/cli/gateway-cli/run-loop.ts
Line: 192-201
Comment:
**Async SIGTERM handler races with concurrent signals**
`onSigterm` now defers the `request()` call until an async file-read completes. During that brief window, `shuttingDown` is still `false`, so a concurrent SIGINT (or a second SIGTERM from the supervisor retrying) will call `request("stop", ...)`, set `shuttingDown = true`, and start a clean shutdown — **without writing the restart manifest**. When the original SIGTERM's async resolves and calls `request("service-restart", ...)`, it is silently ignored by the `if (shuttingDown) return` guard.
The intent file has already been consumed (unlinked) by `consumeGatewayRestartIntent`, so on the next startup `clearGatewayRestartIntent` is a no-op and `recoverGatewayRestartManifest` finds no manifest, meaning session recovery is silently skipped for that restart.
This is unlikely in normal operation, but launchd/systemd will sometimes escalate with a second SIGTERM if the process hasn't exited quickly. Consider atomically setting a "SIGTERM in-flight" flag on entry and short-circuiting any concurrent signal path that would turn a service-restart into a plain stop.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 696b9b9d61
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (!opts.force) { | ||
| const readiness = evaluateGatewayRestartReadinessGate(); |
There was a problem hiding this comment.
Guard readiness gate against invalid config
This new early gate runs evaluateGatewayRestartReadinessGate() before the existing runServiceRestart preflight validation, and that helper defaults to loadConfig(). If openclaw.json is malformed or schema-invalid, restart now throws out of runDaemonRestart instead of reaching the established getConfigValidationError path that prints actionable restart/doctor guidance. In practice, openclaw gateway restart can crash with a raw config parse error rather than the expected structured failure output.
Useful? React with 👍 / 👎.
| const blocked = | ||
| restartConfig.readinessGate && readiness.totalActive > restartConfig.readinessGateThreshold; |
There was a problem hiding this comment.
Block restarts when active count meets threshold
The readiness gate uses totalActive > readinessGateThreshold, which allows restart without --force when active work equals the configured threshold. With readinessGateThreshold: 1, a single running session/cron job still bypasses the gate and gets interrupted, which conflicts with the new config semantics (“minimum active work count that triggers the gate”). The comparison should be inclusive (>=) or the setting/help text should be redefined.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0f2602848d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const restartConfig = resolveGatewayRestartConfig(cfg); | ||
| const readiness = inspectGatewayRestartReadiness(cfg); | ||
| const blocked = | ||
| restartConfig.readinessGate && readiness.totalActive >= restartConfig.readinessGateThreshold; |
There was a problem hiding this comment.
Prevent default readiness gate from blocking idle restarts
The new gate condition uses readiness.totalActive >= restartConfig.readinessGateThreshold, and the same module defaults readinessGateThreshold to 0, so a gateway with no active sessions/cron runs still evaluates as blocked (0 >= 0). That means openclaw gateway restart and tool-based restarts now require --force/force: true even when nothing would be interrupted, which contradicts the intended “block when active work exists” behavior and makes default restart flows fail unexpectedly.
Useful? React with 👍 / 👎.
|
Thanks @lmdeagles. I landed #71465 as the canonical fix for this restart-drain cluster. It preserves active-run draining for restart-required reloads, carries service-manager SIGTERM restarts through a restart intent, updates the config docs/changelog, and includes focused coverage for the drain and SIGTERM handoff paths. That covers the in-flight-kill part of this PR, so I am closing this to avoid two competing restart implementations. The broader session recovery/interruption-propagation scope remains tracked in #57425. |
|
I'm honored @vincentkoc ! My very first PR! |
Problem
When the gateway restarts — whether from
openclaw gateway restart, a config change, SIGUSR1, or a crash — all in-flight work is silently killed. There is no mechanism for sessions to know they were interrupted, no way for parent sessions to learn their subagents died, and no recovery path other than waiting for the next heartbeat or manual intervention.This is the single biggest reliability gap in multi-agent OpenClaw deployments.
Real-world impact
Running 7 agents across Discord + iMessage + cron, a single gateway restart can:
The blast radius scales with agent count. Community reports: #51917, #30043, #4410, #43311.
Solution
Five-part restart protection system:
1. Pre-restart session manifest
Before drain, enumerate active sessions, cron runs, and subagents. Write
restart-manifest.jsonto the state directory with full context (session keys, status, channels, last message previews, active subagents).2. Post-restart session recovery
On startup, read manifest and:
[System]events into interrupted sessions so agents know they were restarted3. Readiness gate
Both CLI (
openclaw gateway restart) and the agent gateway tool check active sessions before proceeding. Returns a warning with session details. Requires--force/force: trueto override.4. Drain-aware message queuing
Messages arriving during the drain window catch
GatewayDrainingErrorand are appended to the manifest for post-restart replay, instead of being silently dropped.5. Configuration
{ "gateway": { "restart": { "sessionRecovery": true, "cronRetryOnInterrupt": true, "readinessGate": true, "readinessGateThreshold": 0, "drainQueueMessages": true, "manifestPath": "restart-manifest.json" } } }All settings default to enabled with safe values.
What this does NOT solve
Files changed
src/gateway/restart-recovery.ts(~500 lines) — core modulesrc/agents/tools/gateway-tool.restart.test.ts— readiness gate tests18 files changed, 854 insertions, 6 deletions. 19 tests passing.
Testing
OPENCLAW_HOME=/tmp/openclaw-test, port 18790) — manifest write on SIGUSR1 confirmed workingPrior art
Environment
Closes #57425