Skip to content

Gateway restart drops queued and in-flight inbound messages #51620

Description

@tomsalphaclawbot

Problem

When the gateway restarts (via openclaw gateway restart, config change, SIGUSR1, or LaunchAgent restart), inbound messages that haven't been fully processed are silently lost. This includes:

  1. Followup queue items (FollowupQueueState.items) — these are in-memory only (resolveGlobalSingleton) and are discarded on process exit.
  2. Messages arriving during the drain window — once markGatewayDraining() is called, new enqueueCommandInLane() calls reject with GatewayDrainingError, and channel plugins don't persist these rejected messages for replay.
  3. Messages arriving while the gateway is down — channel-dependent; Slack Socket Mode may redeliver some, but there's no guarantee across all providers.

The existing restart sentinel system (restart-sentinel.json) only persists a single notification about the restart itself (status, reason, delivery context), not the pending inbound message queue.

Expected Behavior

Before shutting down for restart, the gateway should:

  1. Persist any pending followup queue items (FOLLOWUP_QUEUES) to disk (e.g. state/pending-messages.json)
  2. After restart, on startup, read and replay persisted pending messages through normal dispatch

This ensures that user messages sent during a restart window are not silently dropped.

Current Behavior

  • chatRunState.clear() is called in server-close.ts, wiping in-memory state
  • clearFollowupQueue() / queue GC removes pending items
  • GatewayDrainingError rejects new enqueues but the rejected message is not persisted
  • User experiences: message sent → no response → must manually repeat after restart

Reproduction

  1. Send a message in any channel (Slack, Telegram, etc.)
  2. Immediately trigger openclaw gateway restart before the reply is delivered
  3. Observe the message gets no response after restart

Proposed Approach

Phase 1: Persist followup queues across restart

  • In the close handler (before chatRunState.clear()), serialize FOLLOWUP_QUEUES entries to state/pending-messages.json
  • On startup (in server-restart-sentinel.ts or adjacent), read the file and re-enqueue items via enqueueFollowupRun()
  • Delete the file after successful replay

Phase 2: Handle drain-window arrivals

  • When GatewayDrainingError is thrown, persist the rejected inbound context (channel, sender, message text, thread info) to the same pending file
  • Replay these on next startup

Phase 3: Channel-level message acknowledgment

  • For channels that support it (Slack Socket Mode, etc.), defer message acknowledgment until the message is either processed or persisted, so the platform can redeliver on restart

Impact

This particularly affects agent-initiated restarts (config changes, updates) where the agent triggers a restart while actively in a conversation — the user's most recent message(s) in the thread disappear silently.

References

  • src/process/command-queue.tsGatewayDrainingError, markGatewayDraining(), resetAllLanes()
  • src/auto-reply/reply/queue/state.tsFOLLOWUP_QUEUES (in-memory only)
  • src/auto-reply/reply/queue/enqueue.tsenqueueFollowupRun()
  • src/gateway/server-close.tscreateGatewayCloseHandler()
  • src/cli/gateway-cli/run-loop.ts — drain + restart flow
  • src/infra/restart-sentinel.ts — existing restart notification persistence
  • src/gateway/server-restart-sentinel.ts — post-restart notification delivery

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions