-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Slack Socket Mode: event loop starvation causes pong timeouts and silent message loss #58519
Copy link
Copy link
Closed as not planned
Closed as not planned
Copy link
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Problem
Slack Socket Mode WebSocket pong responses are missed when the Node.js event loop is busy processing agent turns. We empirically observe 4+ consecutive pong timeouts (5000ms deadline) during heavy turns, which causes Slack to drop the WebSocket connection. Messages sent by the gateway during or just after a dead socket window are silently lost — the gateway logs a successful delivery, but Slack never receives it.
What we observe
[WARN] socket-mode:SlackWebSocket A pong wasn't received from the server before the timeout of 5000ms!— we see runs of 4+ sequential timeouts during heavy processingdelivered replyfor messages it sent to a stale/dying socket. The user never receives them. From the user's perspective, the bot goes silent.Gateway diagnostic logs show the stuck session pattern alongside the pong timeouts:
Root cause
All processing — agent turns, tool dispatch, sub-agent orchestration, JSON parsing, context assembly, and WebSocket keepalive — runs on the same single Node.js event loop thread. When heavy turns monopolize the event loop, the Slack SDK's ping/pong handler can't fire within its 5000ms deadline.
This isn't a Slack server-side issue (as in #14248) — it's the gateway's event loop being too busy to service the pings at all.
Suggested fix
Move Slack Socket Mode WebSocket keepalive (ping/pong) to a
worker_threadsthread. The keepalive handler doesn't need access to session state or agent context — it just needs to respond "pong" to Slack's "ping." This is a small, isolated piece of work that would prevent connection drops regardless of how busy the main thread is.Alternatively (or additionally):
clientPingTimeoutper Expose clientPingTimeout for Slack Socket Mode configuration #14248 — a higher timeout would reduce false positives, though it doesn't prevent genuine event loop starvation from causing dropssetImmediate()yields in long-running synchronous processing paths (context assembly, transcript serialization) to give the event loop breathing roomRelated
clientPingTimeoutfor Slack Socket Mode configuration