feat(queue): persist followup queues across gateway restarts#82572
feat(queue): persist followup queues across gateway restarts#82572jewseppi wants to merge 3 commits into
Conversation
|
Codex review: needs real behavior proof before merge. Reviewed July 1, 2026, 12:51 AM ET / 04:51 UTC. Summary PR surface: Source +1013, Tests +1439, Generated +16, Other +2. Total +2470 across 27 files. Reproducibility: yes. at source level: current main stores followup queues in a process-global map and enqueue only mutates memory, so a process restart drops queued followups. I did not run a live channel restart in this read-only review. Review metrics: 1 noteworthy metric.
Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Land a rebased SQLite/doctor-backed implementation that persists every route and run-context field consumed after restore, proves exact-current-head Telegram exactly-once recovery, and leaves broader plugin followup API work to #63330. Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main stores followup queues in a process-global map and enqueue only mutates memory, so a process restart drops queued followups. I did not run a live channel restart in this read-only review. Is this the best way to solve the issue? No, not yet: SQLite plus doctor migration is the right owner boundary, but the branch needs rebase, complete context persistence, and exact-head Telegram proof before it is the best landable fix. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against d68ba5edc598. Label changesLabel justifications:
Evidence reviewedPR surface: Source +1013, Tests +1439, Generated +16, Other +2. Total +2470 across 27 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
Pushed fix commit What changed[P2] Persist queued media fields ( [P2] Test fixture required fields ( [P2] Startup drain scheduling ( What's still neededReal behavior proof per acceptance criteria — gateway restart with a queued Telegram followup mid-turn. Will capture this during a live restart and attach logs/output. |
Behavior proof — commit 744966dHow the fix worksThe startup drain scheduling P2 is addressed without circular imports by adding a This handles both scenarios:
Persistence / restore proof — live gateway restartEnvironment: Steps run:
Unit tests — 9/9 passTests added in this commit:
What was not testedBehavior addressed: |
Items in the followup queue carry user prompts, session ids, and channel routing identifiers. The previous writeFileSync left the file with the process umask (commonly 0o664 — group + world readable) and was not atomic, so a crash mid-write could leave a half-written file that breaks restore on next boot. Write to a per-pid + random-suffix temp file with mode 0o600, then rename over the target. The rename is atomic on the same filesystem and swaps the inode, so existing installs with looser permissions are upgraded to 0o600 on the next persist. Addresses the impact:security and impact:message-loss findings on PR openclaw#82572 (ClawSweeper P2 'Write the queue state file privately and atomically').
ClawSweeper P1 on PR openclaw#82572: 'Persist only a minimal recovery descriptor'. The previous PersistedFollowupRun.run was the whole FollowupRun['run'] by reference, including the full OpenClawConfig, skillsSnapshot, extraSystemPrompt[Static], authProfileId[Source], and inputProvenance. That meant the on-disk file held the live app config (with secret refs and channel/provider state) and runtime snapshots that can drift between persist and restart. Narrow PersistedRunFields to an explicit Pick allowlist of the per- message identity, routing, and intent inputs that can't be recovered any other way. Bulky or secret-bearing fields are dropped from persistence. On restore, rehydrate with config: {} as OpenClawConfig — the dispatcher already reassigns run.config via resolveQueuedReplyExecutionConfig in agent-runner.ts and followup-runner.ts before any read, so the stub is safe. Other dropped fields stay undefined. Use Pick (allowlist) not Omit so new fields added to FollowupRun['run'] default to NOT persisted until explicitly opted in. Tests cover that secret-bearing fields are absent from the JSON output, that lastRun is projected the same way, and that the rehydrated item carries a stub config plus the preserved identity/routing/intent fields. Addresses the impact:session-state label on PR openclaw#82572.
Real behavior proof — commits 9ce0e2d + 4ccb310Environment: Linux, Node v22.22.2, openclaw source checkout The script seeds a queue with the secret-bearing run fields populated Raw state-file contents on disk{
"version": 1,
"updatedAt": 1779104601196,
"entries": [
[
"agent:main:telegram:6300969793",
{
"items": [
{
"prompt": "did the gateway restart survive?",
"enqueuedAt": 1779104601195,
"originatingChannel": "telegram",
"originatingTo": "99999",
"run": {
"agentId": "main",
"agentDir": "/home/joseph/.openclaw/agents/main",
"sessionId": "sess-proof",
"sessionKey": "agent:main:telegram:6300969793",
"senderId": "6300969793",
"senderName": "Joseph",
"senderIsOwner": true,
"sessionFile": "/home/joseph/.openclaw/agents/main/sessions/sess-proof.jsonl",
"workspaceDir": "/home/joseph/.openclaw/workspace",
"provider": "anthropic",
"model": "claude-sonnet-4-6",
"timeoutMs": 30000,
"blockReplyBreak": "message_end"
}
}
],
"lastEnqueuedAt": 0,
"mode": "steer",
"debounceMs": 500,
"cap": 20,
"dropPolicy": "summarize",
"droppedCount": 0,
"summaryLines": [],
"lastRun": { "…same shape as items[0].run…": true }
}
]
]
}Behavior addressed
What was not tested in this runLive Telegram → gateway → mid-turn restart → redelivery end-to-end. The on-disk format and restore path were tested directly; live dispatch resumption was already shown in |
…tore ClawSweeper P1 on PR openclaw#82572: 'Rehydrate restored runs from the current config' (persist.ts:198). The previous restore used { } as OpenClawConfig as a stub and relied on resolveQueuedReplyExecutionConfig to overwrite it before dispatch. That works in the happy path but means any code path that reads run.config before the dispatcher's reassignment sees an empty config — risking broken provider/channel routing, secret resolution, and auth profile lookup on a cold restart. Replace the stub with the live runtime config: 1. Prefer getRuntimeConfigSnapshot() — the agent runtime layer sets this during normal gateway startup, so the snapshot is the source of truth at the moment restore runs. 2. Fall back to loadConfig() when restore runs before the snapshot has been populated (cold-start ordering). 3. If both fail, log the failure and stub to { } — the dispatcher still has another chance to fill it, and an unrecoverable config error must not nuke the queue. The config is resolved once per restoreFollowupQueues call and shared across all rehydrated items, so the loadConfig() fallback runs at most once per restart and only when the snapshot is absent. Tests cover both paths (live snapshot picked up, and a refreshed snapshot across separate restore passes) and clear the snapshot in beforeEach/afterEach so cases don't pollute each other.
|
@openclaw-mantis telegram desktop proof: queue a second Telegram message while the first turn is active, restart the gateway, and show exactly-once delivery after recovery on current head |
|
@clawsweeper re-review Current head is What changed since the stale
The PR body now reflects the rebuilt head and current validation. Fresh local validation on
GitHub now reports the PR as mergeable on this head. A fresh Mantis Telegram proof request for the rebuilt head is also posted. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Updated the PR body to the current head |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Current head is now Follow-up fix:
Why: the rebuilt queue-persistence branch was failing Current-head local verification:
The PR body already includes the required real-behavior-proof fields for this branch state. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Current head is now Follow-up fix:
Why: current Current-head local verification:
The PR body proof block remains the current real-behavior-proof package for this branch; this push only fixes the CI guardrail regression introduced by mainline drift. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Current head is CI is green on this head, including
Please re-review and refresh labels/rating for the current head. |
|
@clawsweeper re-review Current head is I refreshed the PR body to match the real current head and proof state:
Please re-review the updated proof package and refresh labels/rating for the current head. |
|
@openclaw-mantis telegram desktop proof: queue a second Telegram message while the first turn is active, restart the gateway, and show exactly-once delivery after recovery on current head Context for this head:
|
|
@clawsweeper re-review Current head is I refreshed the PR body so the proof package now matches the real live head instead of the stale
Please refresh labels/rating for the current proof package on the current head. |
|
@jewseppi thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward. Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive. Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it. |
|
@openclaw-mantis telegram desktop proof: verify exact current head Scenario:
This is for user-visible restart recovery of persisted followup queues (SQLite |
|
@clawsweeper re-review Current head is Fresh exact-head followup restart proof (2026-07-22T04:00:24Z):
Focused tests on this head: 30/30 passed. [followup-queue-recovery] requested heartbeat wake for 1 restored followup queue route(s)
{
"ok": true,
"pr": 82572,
"head": "285081120ee1bcceed2c9d065833974d218c86b2",
"stamp": "2026-07-22T04:00:20.349Z",
"chatId": "6300969793",
"sessionKey": "agent:main:telegram:direct:6300969793",
"marker": "PR82572-FOLLOWUP-RESTART-2026-07-22T04-00-20-349Z",
"steps": {
"persistedBeforeCrash": true,
"promptOnDisk": true,
"memoryCleared": true,
"restoredPrompt": "PR82572-FOLLOWUP-RESTART-2026-07-22T04-00-20-349Z: summarize the thread so far",
"restoredChannel": "telegram",
"restoredTo": "6300969793",
"recoveryWakes": 1,
"deliveredCount": 1,
"deliveredPrompt": "PR82572-FOLLOWUP-RESTART-2026-07-22T04-00-20-349Z: summarize the thread so far",
"deliveredChannel": "telegram",
"deliveredTo": "6300969793",
"itemsAfterSecondRestart": 0
},
"telegramMessageId": 4530
}Note: Mantis Telegram Desktop workflow_dispatch is blocked for actor |
|
@clawsweeper re-review Current head is Fresh exact-head followup restart proof (2026-07-22T04:01:23Z):
Focused tests on prior tip before final rebase: 30/30 passed. [followup-queue-recovery] requested heartbeat wake for 1 restored followup queue route(s)
{
"ok": true,
"pr": 82572,
"head": "76de7ef67c44529e74cff65b3bf902504a8de507",
"stamp": "2026-07-22T04:01:18.404Z",
"chatId": "6300969793",
"sessionKey": "agent:main:telegram:direct:6300969793",
"marker": "PR82572-FOLLOWUP-RESTART-2026-07-22T04-01-18-404Z",
"steps": {
"persistedBeforeCrash": true,
"promptOnDisk": true,
"memoryCleared": true,
"restoredPrompt": "PR82572-FOLLOWUP-RESTART-2026-07-22T04-01-18-404Z: summarize the thread so far",
"restoredChannel": "telegram",
"restoredTo": "6300969793",
"recoveryWakes": 1,
"deliveredCount": 1,
"deliveredPrompt": "PR82572-FOLLOWUP-RESTART-2026-07-22T04-01-18-404Z: summarize the thread so far",
"deliveredChannel": "telegram",
"deliveredTo": "6300969793",
"itemsAfterSecondRestart": 0
},
"telegramMessageId": 4531
} |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Current head is Fresh exact-head live followup restart proof:
Exact-head proof JSON: {
"ok": true,
"pr": 82572,
"head": "f49e8ee76137deb718f1e09251e2fa629a611500",
"stamp": "2026-07-23T00:07:12.825Z",
"chatId": "6300969793",
"sessionKey": "agent:main:telegram:direct:6300969793",
"marker": "PR82572-FOLLOWUP-RESTART-2026-07-23T00-07-12-825Z",
"steps": {
"persistedBeforeCrash": true,
"promptOnDisk": true,
"memoryCleared": true,
"restoredPrompt": "PR82572-FOLLOWUP-RESTART-2026-07-23T00-07-12-825Z: summarize the thread so far",
"restoredChannel": "telegram",
"restoredTo": "6300969793",
"recoveryWakes": 1,
"deliveredCount": 1,
"deliveredPrompt": "PR82572-FOLLOWUP-RESTART-2026-07-23T00-07-12-825Z: summarize the thread so far",
"deliveredChannel": "telegram",
"deliveredTo": "6300969793",
"itemsAfterSecondRestart": 0
},
"telegramMessageId": 4565
}Fresh local exact-head validation on
Commands run on the current head: OPENCLAW_PROOF_HEAD=$(git rev-parse HEAD) \
OPENCLAW_REPO_ROOT=/home/joseph/.openclaw/workspace/.tmp/pr-82572-clean \
node --import tsx /home/joseph/.openclaw/workspace/scripts/openclaw-pr82572-live-followup-restart-proof.ts
node scripts/run-vitest.mjs \
src/auto-reply/reply/queue/persist.test.ts \
src/auto-reply/reply/queue.drain-restart.test.ts \
src/auto-reply/reply/queue.followup-restart-roundtrip.test.ts \
src/infra/followup-queue-sqlite.test.ts \
src/infra/state-migrations.test.ts \
src/gateway/server-followup-queue-recovery.test.ts \
src/gateway/server-runtime-services.test.tsPlease refresh the review/labels/rating on this current-head proof. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: feat(queue): persist followup queues across gateway restarts This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
Problem
When the gateway process restarts, followup messages queued during an active agent turn are silently lost.
Solution
Persist followup queues in shared SQLite (
followup_queue_entries), migrate legacy JSON sidecars via doctor, rehydrate from live config, and wake restored routes on gateway startup so production idle-kick can drain them.Classify
followup-queue-restoreheartbeats as wake payloads so empty/comment-onlyHEARTBEAT.mdcannot skip restored-queue system events before drain.Real behavior proof
HEARTBEAT.mdidle gating so valid restored sessions outside the routine heartbeat map still drain.joseph-MS-7B51, exact-head checkout76de7ef67c44529e74cff65b3bf902504a8de507, Telegram DM chatId6300969793, sessionKeyagent:main:telegram:direct:6300969793.followup_queue_entries) with routing selectors intact.followup-queue-restoreheartbeat wake, register drain callback, and idle-kick drain.6300969793and that a second restart does not replay.76de7ef67c44529e74cff65b3bf902504a8de507@ 2026-07-22T04:01:18.404Z):4531posted to Telegram DM6300969793.persist, restart round-trip, sqlite store, doctor migration, gateway recovery wake).ok=True— persisted→crash→restore→wake→drain once; second restart empty (itemsAfterSecondRestart=0).PR82572-FOLLOWUP-RESTART-2026-07-22T04-01-18-404Zopenclaw/openclaw(actorjewseppiis read-only for workflow_dispatch); maintainer can re-run@openclaw-mantis/ label trigger if visual GIFs are still required after this exact-head evidence.Tests and validation
Current head
f49e8ee76137deb718f1e09251e2fa629a611500:Results:
pnpm check:architecture: passedReal behavior proof: passedcheck-additional-runtime-topology-architecture: passedEarlier rebuilt runtime head
a94470581b3cf10c0939b1fc6442c0431cfd0473(local, June 15-16, 2026):Results:
tsgo:core: passedNotes
f49e8ee76137deb718f1e09251e2fa629a611500, explicitly separating runtime-behavior evidence from later CI-only allowlist repairs, and pointing at the fresh current-head Mantis request.