-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Gateway restart drops queued and in-flight inbound messages #51620
Copy link
Copy link
Open
Open
Copy link
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Problem
When the gateway restarts (via
openclaw gateway restart, config change, SIGUSR1, or LaunchAgent restart), inbound messages that haven't been fully processed are silently lost. This includes:FollowupQueueState.items) — these are in-memory only (resolveGlobalSingleton) and are discarded on process exit.markGatewayDraining()is called, newenqueueCommandInLane()calls reject withGatewayDrainingError, and channel plugins don't persist these rejected messages for replay.The existing restart sentinel system (
restart-sentinel.json) only persists a single notification about the restart itself (status, reason, delivery context), not the pending inbound message queue.Expected Behavior
Before shutting down for restart, the gateway should:
FOLLOWUP_QUEUES) to disk (e.g.state/pending-messages.json)This ensures that user messages sent during a restart window are not silently dropped.
Current Behavior
chatRunState.clear()is called inserver-close.ts, wiping in-memory stateclearFollowupQueue()/ queue GC removes pending itemsGatewayDrainingErrorrejects new enqueues but the rejected message is not persistedReproduction
openclaw gateway restartbefore the reply is deliveredProposed Approach
Phase 1: Persist followup queues across restart
chatRunState.clear()), serializeFOLLOWUP_QUEUESentries tostate/pending-messages.jsonserver-restart-sentinel.tsor adjacent), read the file and re-enqueue items viaenqueueFollowupRun()Phase 2: Handle drain-window arrivals
GatewayDrainingErroris thrown, persist the rejected inbound context (channel, sender, message text, thread info) to the same pending filePhase 3: Channel-level message acknowledgment
Impact
This particularly affects agent-initiated restarts (config changes, updates) where the agent triggers a restart while actively in a conversation — the user's most recent message(s) in the thread disappear silently.
References
src/process/command-queue.ts—GatewayDrainingError,markGatewayDraining(),resetAllLanes()src/auto-reply/reply/queue/state.ts—FOLLOWUP_QUEUES(in-memory only)src/auto-reply/reply/queue/enqueue.ts—enqueueFollowupRun()src/gateway/server-close.ts—createGatewayCloseHandler()src/cli/gateway-cli/run-loop.ts— drain + restart flowsrc/infra/restart-sentinel.ts— existing restart notification persistencesrc/gateway/server-restart-sentinel.ts— post-restart notification delivery