Skip to content

Graceful gateway shutdown: drain active sessions before restart #32961

Description

@immeasurablematt

Problem

When the gateway restarts (openclaw gateway restart), all in-flight agent sessions are killed immediately. Any agent mid-task — writing a Linear comment, running a multi-step cron job, composing a Discord message — gets hard-killed with no chance to finish or checkpoint. This causes:

  • Consecutive error counters climbing on cron jobs that were healthy but happened to be running during restart
  • Partial writes — agents mid-way through multi-step operations (e.g., moved a Linear issue to "In Review" but didn't post the summary to Discord)
  • Lost work — agents doing long-running tasks (research, audits) lose all progress
  • User friction — operators have to manually re-trigger interrupted work or wait for the next cron cycle

This is especially painful for setups with many cron jobs on tight intervals, where any restart is likely to interrupt something.

Proposed Solution: Graceful Drain

Add a drain phase between "stop requested" and "process killed," similar to how Kubernetes, systemd, and PM2 handle graceful shutdown.

Flow

openclaw gateway restart
  │
  ├─ 1. Stop accepting new sessions/cron triggers
  │
  ├─ 2. Signal all active sessions: "shutdown imminent"
  │     └─ Agents receive a system message or event they can react to
  │
  ├─ 3. Wait for active sessions to reach idle state
  │     └─ With a configurable timeout (e.g., `gracefulTimeoutMs: 60000`)
  │
  ├─ 4. Force-kill any sessions still active after timeout
  │
  └─ 5. Shutdown and restart

Configuration

{
  "gateway": {
    "gracefulShutdown": {
      "enabled": true,
      "drainTimeoutMs": 60000,
      "notifyAgents": true
    }
  }
}

Agent-Side Hooks (optional, additive)

If notifyAgents is true, active sessions receive a system event (e.g., gateway:shutdown) that plugin hooks or the agent itself can react to — save state, finish the current step, or acknowledge readiness to shut down.

This could integrate with the existing plugin/hook system:

api.on("gateway:shutdown", async ({ session, deadline }) => {
  // Agent has until `deadline` to finish
  // Return { ready: true } to signal early completion
});

CLI Behavior

# Current (hard kill)
openclaw gateway restart

# With this feature
openclaw gateway restart              # drain + restart (default)
openclaw gateway restart --force      # hard kill (current behavior)
openclaw gateway restart --drain 120  # custom drain timeout in seconds

Alternatives Considered

  • Checkpoint/resume: Agents serialize in-flight state and resume after restart. Much more complex, requires agent-side state management. Drain is simpler and covers 90% of cases.
  • Cron-aware restart windows: Only restart when no cron jobs are scheduled within N minutes. Fragile with many interval-based jobs and doesn't help ad-hoc sessions.
  • Retry-on-restart: Gateway automatically re-sends the last message to interrupted sessions. Risky for non-idempotent operations (double-posting, duplicate Linear updates).

Impact

Any multi-agent setup with cron jobs benefits. The more agents and jobs, the bigger the impact. A typical setup with 6 agents and 25+ cron jobs has a near-100% chance of interrupting something on every restart.

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions