Skip to content

feat: graceful gateway restart with session recovery#57556

Closed
lmdeagles wants to merge 6 commits into
openclaw:mainfrom
lmdeagles:feature/graceful-restart-recovery
Closed

feat: graceful gateway restart with session recovery#57556
lmdeagles wants to merge 6 commits into
openclaw:mainfrom
lmdeagles:feature/graceful-restart-recovery

Conversation

@lmdeagles

Copy link
Copy Markdown

Problem

When the gateway restarts — whether from openclaw gateway restart, a config change, SIGUSR1, or a crash — all in-flight work is silently killed. There is no mechanism for sessions to know they were interrupted, no way for parent sessions to learn their subagents died, and no recovery path other than waiting for the next heartbeat or manual intervention.

This is the single biggest reliability gap in multi-agent OpenClaw deployments.

Real-world impact

Running 7 agents across Discord + iMessage + cron, a single gateway restart can:

  • Kill 3-4 active conversations simultaneously
  • Orphan subagents mid-task (research, code generation, file operations)
  • Drop cron jobs that were minutes into expensive multi-tool workflows
  • Leave group chat messages permanently unanswered

The blast radius scales with agent count. Community reports: #51917, #30043, #4410, #43311.

Solution

Five-part restart protection system:

1. Pre-restart session manifest

Before drain, enumerate active sessions, cron runs, and subagents. Write restart-manifest.json to the state directory with full context (session keys, status, channels, last message previews, active subagents).

2. Post-restart session recovery

On startup, read manifest and:

  • Inject [System] events into interrupted sessions so agents know they were restarted
  • Re-queue interrupted cron runs (if configured)
  • Notify parent sessions about killed subagents
  • Replay messages that arrived during drain

3. Readiness gate

Both CLI (openclaw gateway restart) and the agent gateway tool check active sessions before proceeding. Returns a warning with session details. Requires --force / force: true to override.

4. Drain-aware message queuing

Messages arriving during the drain window catch GatewayDrainingError and are appended to the manifest for post-restart replay, instead of being silently dropped.

5. Configuration

{
  "gateway": {
    "restart": {
      "sessionRecovery": true,
      "cronRetryOnInterrupt": true,
      "readinessGate": true,
      "readinessGateThreshold": 0,
      "drainQueueMessages": true,
      "manifestPath": "restart-manifest.json"
    }
  }
}

All settings default to enabled with safe values.

What this does NOT solve

  • Crash recovery — no pre-crash manifest possible; requires periodic snapshots (separate issue)
  • Subagent state resume — agents decide their own recovery strategy
  • Idempotency — recovery is a safety net, not a substitute for good workflow design

Files changed

  • New: src/gateway/restart-recovery.ts (~500 lines) — core module
  • New: src/agents/tools/gateway-tool.restart.test.ts — readiness gate tests
  • Modified: 16 files across gateway tool, CLI lifecycle, run loop, dispatch, config schema, server startup, restart infrastructure

18 files changed, 854 insertions, 6 deletions. 19 tests passing.

Testing

  • Unit tests cover readiness gate blocking, force-restart override, mock typing across all integration points
  • Built from source and ran a second isolated gateway instance (OPENCLAW_HOME=/tmp/openclaw-test, port 18790) — manifest write on SIGUSR1 confirmed working
  • Full integration test with active sessions pending (WS protocol v3 device pairing makes automated session creation non-trivial from outside the gateway)

Prior art

Environment

  • OpenClaw 2026.3.28-3.30
  • macOS (Darwin 25.3.0, arm64), LaunchAgent
  • 7 agents, 3 Discord bots, BlueBubbles iMessage, 12 cron jobs

Closes #57425

Adds a 5-part restart protection system that eliminates silent work loss
during gateway restarts.

1. Pre-restart manifest — before drain, enumerate active sessions, cron
   runs, and subagents; write restart-manifest.json to state dir
2. Post-restart recovery — on startup, read manifest, inject system events
   into interrupted sessions, re-queue cron, notify parents about killed
   subagents, delete manifest
3. Readiness gate — CLI and gateway tool warn when active work exists;
   requires --force / force:true to proceed
4. Drain-aware message queuing — messages arriving during drain are
   appended to manifest for post-restart replay
5. Configuration — gateway.restart.* config surface (6 keys, all opt-out
   with safe defaults)

New files:
  src/gateway/restart-recovery.ts — core recovery module (~500 lines)
  src/agents/tools/gateway-tool.restart.test.ts — readiness gate tests

Modified: 16 files across gateway tool, CLI lifecycle, run loop,
dispatch, config schema, server startup, and restart infrastructure.

19 tests passing. Tested on macOS (Darwin arm64, LaunchAgent).

Closes openclaw#57425
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime cli CLI command changes agents Agent runtime and tooling size: L labels Mar 30, 2026
@greptile-apps

greptile-apps Bot commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a well-structured five-part restart recovery system for OpenClaw gateways: a pre-restart manifest, post-restart session recovery, a readiness gate, drain-aware message queuing, and matching configuration schema. The overall design is sound and addresses a genuine reliability gap for multi-agent deployments.

Two issues were found during review:

  • P1 — manifestWriteChain chain poisoning (restart-recovery.ts:436): The module-level serial-write chain used to safely append drain messages to the manifest has no error recovery. If writeManifestFile throws (disk full, permission error, etc.), manifestWriteChain is left as a permanently rejected promise for the life of the process. Every subsequent call to queueDrainRestartMessage chains off a rejected promise (the .then() callback is never invoked), so all further drain-window messages are silently not queued. This is fixable with a single .catch(() => {}) appended to the chain assignment.

  • P2 — Async SIGTERM handler races with concurrent signals (run-loop.ts:192): Making onSigterm async (to check the restart-intent file) means shuttingDown remains false during the brief file-read. A concurrent SIGINT or a supervisor-retried SIGTERM will set shuttingDown = true and start a plain stop, discarding the service-restart path entirely. The intent file is already consumed by this point, so recoverGatewayRestartManifest on the next startup finds nothing and silently skips recovery. This is unlikely but worth hardening against supervisor escalation behavior.

Confidence Score: 4/5

Safe to merge after fixing the manifestWriteChain chain-poisoning bug; all other findings are low-probability edge cases or style.

One clear P1 defect: a single failed writeManifestFile permanently poisons the module-level manifestWriteChain, causing all subsequent drain-window messages to be silently dropped for the process lifetime. This directly undermines the drain-queue feature that is a stated goal of the PR. The fix is a one-liner. The P2 async-SIGTERM race is low probability but worth documenting. Everything else — config schema, readiness gate, tests, intent file lifecycle — looks solid.

src/gateway/restart-recovery.ts (queueDrainRestartMessage chain poisoning), src/cli/gateway-cli/run-loop.ts (async SIGTERM handler race)

Important Files Changed

Filename Overview
src/gateway/restart-recovery.ts New core module (498 lines). Contains a P1 chain-poisoning bug in queueDrainRestartMessage: a single write failure permanently rejects the module-level manifestWriteChain, silently dropping all subsequent drain messages for the process lifetime.
src/cli/gateway-cli/run-loop.ts SIGTERM handler made async to check restart intent before deciding action. Introduces a race window where a concurrent signal can hijack the shutdown path, causing the manifest to be skipped. Logic for service-restart action (write manifest, drain, then stop) is otherwise correct.
src/agents/tools/gateway-tool.ts Adds readiness-gate check and force parameter to the restart action. Implementation is clean and matches test coverage.
src/cli/daemon-cli/lifecycle.ts Adds readiness gate check at the top of runDaemonRestart with JSON and plain-text output paths; correctly bypassed by --force. No issues found.
src/infra/restart.ts Adds primeGatewayRestartReason / consumeGatewayRestartReason for passing the human-readable restart reason into the manifest. Simple, well-tested.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/gateway/restart-recovery.ts
Line: 436-444

Comment:
**`manifestWriteChain` is permanently poisoned on any write failure**

`manifestWriteChain` is a module-level singleton promise. If `writeManifestFile` throws (e.g. disk full, permission denied), `manifestWriteChain` is left as a rejected promise. Every subsequent call to `queueDrainRestartMessage` chains a `.then()` off a rejected promise — which is a no-op and produces a new rejected promise — so **all further drain messages are silently dropped** for the lifetime of the process.

The fix is to catch errors inside the chained operation so the chain always resolves:

```suggestion
  manifestWriteChain = manifestWriteChain
    .then(async () => {
      const manifest = await readManifestFile(restartConfig.manifestPath);
      if (!manifest) {
        return;
      }
      manifest.queuedMessages.push(queued);
      await writeManifestFile(restartConfig.manifestPath, manifest);
      queuedToManifest = true;
    })
    .catch(() => {
      // Prevent a write failure from permanently rejecting the chain.
    });
```

With this change, `await manifestWriteChain` no longer rejects — but since `queueDrainRestartMessage` is already called with `.catch(() => false)` at the dispatch site, that's fine.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/cli/gateway-cli/run-loop.ts
Line: 192-201

Comment:
**Async SIGTERM handler races with concurrent signals**

`onSigterm` now defers the `request()` call until an async file-read completes. During that brief window, `shuttingDown` is still `false`, so a concurrent SIGINT (or a second SIGTERM from the supervisor retrying) will call `request("stop", ...)`, set `shuttingDown = true`, and start a clean shutdown — **without writing the restart manifest**. When the original SIGTERM's async resolves and calls `request("service-restart", ...)`, it is silently ignored by the `if (shuttingDown) return` guard.

The intent file has already been consumed (unlinked) by `consumeGatewayRestartIntent`, so on the next startup `clearGatewayRestartIntent` is a no-op and `recoverGatewayRestartManifest` finds no manifest, meaning session recovery is silently skipped for that restart.

This is unlikely in normal operation, but launchd/systemd will sometimes escalate with a second SIGTERM if the process hasn't exited quickly. Consider atomically setting a "SIGTERM in-flight" flag on entry and short-circuiting any concurrent signal path that would turn a service-restart into a plain stop.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "feat: graceful gateway restart with sess..." | Re-trigger Greptile

Comment thread src/gateway/restart-recovery.ts Outdated
Comment on lines +436 to +444
manifestWriteChain = manifestWriteChain.then(async () => {
const manifest = await readManifestFile(restartConfig.manifestPath);
if (!manifest) {
return;
}
manifest.queuedMessages.push(queued);
await writeManifestFile(restartConfig.manifestPath, manifest);
queuedToManifest = true;
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 manifestWriteChain is permanently poisoned on any write failure

manifestWriteChain is a module-level singleton promise. If writeManifestFile throws (e.g. disk full, permission denied), manifestWriteChain is left as a rejected promise. Every subsequent call to queueDrainRestartMessage chains a .then() off a rejected promise — which is a no-op and produces a new rejected promise — so all further drain messages are silently dropped for the lifetime of the process.

The fix is to catch errors inside the chained operation so the chain always resolves:

Suggested change
manifestWriteChain = manifestWriteChain.then(async () => {
const manifest = await readManifestFile(restartConfig.manifestPath);
if (!manifest) {
return;
}
manifest.queuedMessages.push(queued);
await writeManifestFile(restartConfig.manifestPath, manifest);
queuedToManifest = true;
});
manifestWriteChain = manifestWriteChain
.then(async () => {
const manifest = await readManifestFile(restartConfig.manifestPath);
if (!manifest) {
return;
}
manifest.queuedMessages.push(queued);
await writeManifestFile(restartConfig.manifestPath, manifest);
queuedToManifest = true;
})
.catch(() => {
// Prevent a write failure from permanently rejecting the chain.
});

With this change, await manifestWriteChain no longer rejects — but since queueDrainRestartMessage is already called with .catch(() => false) at the dispatch site, that's fine.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/restart-recovery.ts
Line: 436-444

Comment:
**`manifestWriteChain` is permanently poisoned on any write failure**

`manifestWriteChain` is a module-level singleton promise. If `writeManifestFile` throws (e.g. disk full, permission denied), `manifestWriteChain` is left as a rejected promise. Every subsequent call to `queueDrainRestartMessage` chains a `.then()` off a rejected promise — which is a no-op and produces a new rejected promise — so **all further drain messages are silently dropped** for the lifetime of the process.

The fix is to catch errors inside the chained operation so the chain always resolves:

```suggestion
  manifestWriteChain = manifestWriteChain
    .then(async () => {
      const manifest = await readManifestFile(restartConfig.manifestPath);
      if (!manifest) {
        return;
      }
      manifest.queuedMessages.push(queued);
      await writeManifestFile(restartConfig.manifestPath, manifest);
      queuedToManifest = true;
    })
    .catch(() => {
      // Prevent a write failure from permanently rejecting the chain.
    });
```

With this change, `await manifestWriteChain` no longer rejects — but since `queueDrainRestartMessage` is already called with `.catch(() => false)` at the dispatch site, that's fine.

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines 192 to 201
const onSigterm = () => {
gatewayLog.info("signal SIGTERM received");
request("stop", "SIGTERM");
void consumeGatewayRestartIntent()
.then((intent) => {
request(intent ? "service-restart" : "stop", intent?.reason ?? "SIGTERM");
})
.catch(() => {
request("stop", "SIGTERM");
});
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Async SIGTERM handler races with concurrent signals

onSigterm now defers the request() call until an async file-read completes. During that brief window, shuttingDown is still false, so a concurrent SIGINT (or a second SIGTERM from the supervisor retrying) will call request("stop", ...), set shuttingDown = true, and start a clean shutdown — without writing the restart manifest. When the original SIGTERM's async resolves and calls request("service-restart", ...), it is silently ignored by the if (shuttingDown) return guard.

The intent file has already been consumed (unlinked) by consumeGatewayRestartIntent, so on the next startup clearGatewayRestartIntent is a no-op and recoverGatewayRestartManifest finds no manifest, meaning session recovery is silently skipped for that restart.

This is unlikely in normal operation, but launchd/systemd will sometimes escalate with a second SIGTERM if the process hasn't exited quickly. Consider atomically setting a "SIGTERM in-flight" flag on entry and short-circuiting any concurrent signal path that would turn a service-restart into a plain stop.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/cli/gateway-cli/run-loop.ts
Line: 192-201

Comment:
**Async SIGTERM handler races with concurrent signals**

`onSigterm` now defers the `request()` call until an async file-read completes. During that brief window, `shuttingDown` is still `false`, so a concurrent SIGINT (or a second SIGTERM from the supervisor retrying) will call `request("stop", ...)`, set `shuttingDown = true`, and start a clean shutdown — **without writing the restart manifest**. When the original SIGTERM's async resolves and calls `request("service-restart", ...)`, it is silently ignored by the `if (shuttingDown) return` guard.

The intent file has already been consumed (unlinked) by `consumeGatewayRestartIntent`, so on the next startup `clearGatewayRestartIntent` is a no-op and `recoverGatewayRestartManifest` finds no manifest, meaning session recovery is silently skipped for that restart.

This is unlikely in normal operation, but launchd/systemd will sometimes escalate with a second SIGTERM if the process hasn't exited quickly. Consider atomically setting a "SIGTERM in-flight" flag on entry and short-circuiting any concurrent signal path that would turn a service-restart into a plain stop.

How can I resolve this? If you propose a fix, please make it concise.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 696b9b9d61

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/cli/daemon-cli/lifecycle.ts Outdated
Comment on lines +153 to +154
if (!opts.force) {
const readiness = evaluateGatewayRestartReadinessGate();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard readiness gate against invalid config

This new early gate runs evaluateGatewayRestartReadinessGate() before the existing runServiceRestart preflight validation, and that helper defaults to loadConfig(). If openclaw.json is malformed or schema-invalid, restart now throws out of runDaemonRestart instead of reaching the established getConfigValidationError path that prints actionable restart/doctor guidance. In practice, openclaw gateway restart can crash with a raw config parse error rather than the expected structured failure output.

Useful? React with 👍 / 👎.

Comment thread src/gateway/restart-recovery.ts Outdated
Comment on lines +249 to +250
const blocked =
restartConfig.readinessGate && readiness.totalActive > restartConfig.readinessGateThreshold;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Block restarts when active count meets threshold

The readiness gate uses totalActive > readinessGateThreshold, which allows restart without --force when active work equals the configured threshold. With readinessGateThreshold: 1, a single running session/cron job still bypasses the gate and gets interrupted, which conflicts with the new config semantics (“minimum active work count that triggers the gate”). The comparison should be inclusive (>=) or the setting/help text should be redefined.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0f2602848d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

const restartConfig = resolveGatewayRestartConfig(cfg);
const readiness = inspectGatewayRestartReadiness(cfg);
const blocked =
restartConfig.readinessGate && readiness.totalActive >= restartConfig.readinessGateThreshold;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent default readiness gate from blocking idle restarts

The new gate condition uses readiness.totalActive >= restartConfig.readinessGateThreshold, and the same module defaults readinessGateThreshold to 0, so a gateway with no active sessions/cron runs still evaluates as blocked (0 >= 0). That means openclaw gateway restart and tool-based restarts now require --force/force: true even when nothing would be interrupted, which contradicts the intended “block when active work exists” behavior and makes default restart flows fail unexpectedly.

Useful? React with 👍 / 👎.

@vincentkoc

Copy link
Copy Markdown
Member

Thanks @lmdeagles. I landed #71465 as the canonical fix for this restart-drain cluster. It preserves active-run draining for restart-required reloads, carries service-manager SIGTERM restarts through a restart intent, updates the config docs/changelog, and includes focused coverage for the drain and SIGTERM handoff paths.

That covers the in-flight-kill part of this PR, so I am closing this to avoid two competing restart implementations. The broader session recovery/interruption-propagation scope remains tracked in #57425.

@vincentkoc vincentkoc closed this Apr 25, 2026
@lmdeagles

Copy link
Copy Markdown
Author

I'm honored @vincentkoc ! My very first PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling cli CLI command changes gateway Gateway runtime size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Graceful Gateway Restart with Session Recovery

2 participants