fix: persist pending followup queues across gateway restart#1178
Open
BingqingLyu wants to merge 2 commits into
Open
fix: persist pending followup queues across gateway restart#1178BingqingLyu wants to merge 2 commits into
BingqingLyu wants to merge 2 commits into
Conversation
Before this change, in-memory followup queue items (FollowupQueueState) were silently lost when the gateway restarted. Messages arriving during the drain window hit GatewayDrainingError and were dropped with no persistence or replay mechanism. This adds: 1. Queue persistence (persist.ts): Before shutdown, serialize non-empty followup queues to state/pending-messages.json. Strips non-portable fields (config, skillsSnapshot) and includes a version marker. 2. Post-restart replay (server-pending-messages.ts): On startup, read the persisted file, inject the missed messages as system events into their respective sessions, and delete the file. Includes a 5-minute staleness guard to avoid replaying ancient messages. 3. Close handler integration: persistFollowupQueues() is called before chatRunState.clear() in server-close.ts so queue state is captured while still available. 4. Startup integration: replayPersistedPendingMessages() runs 1.5s after startup (after channels initialize) alongside the existing restart sentinel wake. Tests: 16 new tests covering persist, consume, round-trip, staleness, corruption handling, and system event injection. Closes openclaw#51620
- test/restart-persistence.test.ts: 3 integration tests that exercise the full persist → consume → replay pipeline with realistic multi-session multi-channel scenarios, staleness guard, and corruption handling - test/prove-restart-persistence.sh: standalone bash/node script that proves the mechanism end-to-end against the compiled build, with human-readable step-by-step output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes openclaw#51620 — gateway restart drops queued and in-flight inbound messages.
Changes
New:
src/auto-reply/reply/queue/persist.tsSerializes all non-empty
FOLLOWUP_QUEUESentries tostate/pending-messages.jsonbefore shutdown. Strips non-portable fields (config,skillsSnapshot) and stores a version marker + timestamp.New:
src/gateway/server-pending-messages.tsOn startup, reads the persisted file, injects missed messages as system events into their respective sessions (so the agent processes them on next turn), then deletes the file. Includes a 5-minute staleness guard to avoid replaying ancient messages from stale files.
Modified:
src/gateway/server-close.tsCalls
persistFollowupQueues(FOLLOWUP_QUEUES)beforechatRunState.clear()so queue state is captured while still available. Best-effort — failure doesn't block shutdown.Modified:
src/gateway/server-startup.tsCalls
replayPersistedPendingMessages()1.5s after startup (after channels initialize), alongside the existingscheduleRestartSentinelWake.Tests
16 new tests covering:
persistFollowupQueues: empty queues, non-empty queues, config stripping, multiple queuesconsumePersistedQueues: no file, read+delete, staleness guard (>5min), corrupt file, wrong versionreplayPersistedPendingMessages: no entries, empty entries, system event injection, multiple queues, prompt truncation, empty items skipLimitations / Follow-up
This PR covers Phase 1: persisting in-memory followup queue items across restart.
Still not covered (follow-up work):
GatewayDrainingError) — these are rejected at the channel level before reaching the queueTesting
All 16 tests pass.
Upstream issue: openclaw#51620