Skip to content

[Bug] Gateway reply-queue state not flushed on restart — stale queued replies drain to wrong Matrix room after restart #91613

Description

@substrateindependent

Summary

Gateway reply-queue state is not flushed on restart. Queued outbound Matrix replies targeting a specific room persist across a gateway process restart and drain to their destination after the gateway comes back up — even when the originating session was isolated and completed long before the restart.

Version

2026.4.25 — reproduced on two separate Mac mini fleet agents (Charlie and Johnny), both running LaunchAgent-managed gateways.

Observed behavior

2026-06-08, ~07:00–07:56 CDT

Charlie (Mac mini, LaunchAgent):

  • 07:00:02 — Morning Briefing cron session starts (session f8986989, targeting Glenn's DM room)
  • 07:14:46 — long-running session warning, age=131s
  • 07:15:06UND_ERR_SOCKET fetch failure mid-model-call (Anthropic provider)
  • 07:16 — gateway self-restarts (process exits and launchd relaunches)
  • 07:34:01 — liveness warning: active + queued sessions both keyed to @johnny DM room (wrong room — Charlie's session was targeting Glenn's DM)
  • 07:34:47–07:34:58ReplyRunAlreadyActiveError loop; queued reply drains into Johnny's DM with Charlie's Morning Briefing tool narration as payload (memory searches, Linear issue lookups, CFO briefing reasoning)

Johnny (Mac mini, LaunchAgent):

  • Same symptom independently: post-restart boot session narration (⏰ Cron, 🧩 Linear Get Issue, 📊 Session Status, 🛠️ exec, 📖 Read, 📝 Edit) leaked into a different agent's DM room through multiple restart cycles between 07:40–07:55 CDT.
  • Second restart (launchctl kickstart) at 07:56 CDT finally stabilized after the queued boot session drained.

Key diagnostic finding

After both restarts, sessions.json showed 0 pending sessions. The reply-queue state is stored separately from the sessions index — meaning the standard sessions-store check does not reveal pending outbound message queue entries. The reply queue appears to persist in a separate store that survives process restart.

Why sessionTarget: isolated does NOT mitigate this

All fleet cron jobs use sessionTarget: isolated. Both affected cron sessions were isolated. The reply-queue corruption still occurred because isolated governs per-run context (not outbound message queue state). These are orthogonal.

Impact / severity

  • Any agent can have outbound messages queued to a wrong room after a network error or restart
  • Content leaking can include tool narration, reasoning, memory search results, Linear data — any assistant output from the corrupted session
  • In this incident: CFO briefing context (sensitive business reasoning) delivered to a peer agent's DM
  • Blast radius scales with session content: a session with GWS auth, customer PII, or credentials could produce a serious information exposure

Proposed fixes (for upstream evaluation)

  1. Flush pending outbound queue on gateway startup — discard any queued replies that predate the current boot. Safest default; may lose legitimate in-flight replies during brief restarts, but correct behavior.
  2. Per-room session lock — enforce that a queued reply can only drain if the room key matches the currently active session key. sessionScope: per-room partially addresses this at config level but should be enforced at the transport layer.
  3. TTL on queued replies — expire queued replies after N minutes (e.g., 5 min). Replies surviving a restart are almost certainly stale.

Current workaround

sessionScope: per-room on the matrix channel config prevents wrong-room drain even if the queue survives restart (config-level mitigation, not a transport-layer fix).

Steps to reproduce

  1. Run a cron/isolated session on an agent with Matrix configured
  2. Mid-session, trigger a network error (UND_ERR_SOCKET or similar) that causes the gateway to self-restart
  3. After gateway restart, observe sessions.json shows 0 pending sessions
  4. Wait: queued reply from the pre-restart session drains to its (potentially wrong) destination room
  5. Check logs: ReplyRunAlreadyActiveError loop confirms the stale reply is retrying

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions