-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Gateway becomes slow or times out under multi-session / multi-agent load #92057
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Summary
On a local OpenClaw installation, the gateway becomes very slow and can appear stuck when the system has many sessions and multiple agents running. The symptoms are most visible during multi-session / multi-agent execution: inbound messages are delayed, gateway RPCs time out, and some Codex/OpenClaw dynamic tool calls can hang or time out.
This has reproduced more than once on the same machine. A previous local audit on 2026-06-07 pointed to gateway event-loop stalls caused or amplified by heavy log/history/session IO. The issue is currently observable again.
Current observed behavior
As of 2026-06-11 08:xx CST on the affected machine:
openclaw gateway status --jsontook about 11.09s.127.0.0.1:18789, but RPC probing fails:rpc.ok: falseerror: "timeout"url: "ws://127.0.0.1:18789"openclaw gateway stability --jsonfails with:GatewayTransportError: gateway timeout after 10000ms/opt/homebrew/opt/node@22/bin/node /opt/homebrew/lib/node_modules/openclaw/dist/index.js gateway --port 18789openclaw sessions list --jsonstill returns, but the environment has a large session store.Local environment
2026.6.5openclaw/dist/index.js gateway --port 18789v22.22.010.9.426.5.1/ Darwin25.5.0/ arm64@openclaw/codex 2026.5.27@tencent-weixin/openclaw-weixin 2.4.318789Session / data scale on the affected machine
The affected installation uses multiple agents and a large local session history:
~/.openclaw/agents/*/sessions: about4820jsonl files.881MB.198MB,136MB,108MB,88MB,57MB.10MBto26MBeach.This looks like the kind of workload that can accumulate naturally when OpenClaw is used as a long-running multi-agent coordinator.
Earlier local audit evidence from 2026-06-07
A previous investigation of a similar gateway hang on the same machine found:
logs.taillatency up to42946ms.eventLoopDelayP99Ms=11467.2.eventLoopUtilization=1.1006during or after stalls.codex dynamic tool call timed outafter120000ms.~/.openclaw/logs/gateway.log:32M,532900lines.~/.openclaw/logs/gateway.err.log:11M,104560lines.Rotating the oversized gateway logs improved
logs.tailand general gateway responsiveness at that time, but the broader multi-session / multi-agent degradation appears to recur with large session/history state.Expected behavior
The gateway should remain responsive under a large but realistic local multi-agent workload. Expensive operations such as log tailing, session listing, history reads, or transcript scans should not block the event loop long enough to cause WebSocket/RPC timeouts or message delivery delays.
Actual behavior
Under the described workload, the gateway can become slow enough that:
gateway statusreports a running process but RPC timeout.gateway stabilitytimes out.Suspected contributing factors
Based on local evidence, possible contributors include:
Suggested hardening
Possible fixes or mitigations:
logs.tailuses bounded reverse-tail reads and never scans full large logs on the gateway event loop.sessions.list,chat.history,logs.tail, dynamic tool dispatch, and WebSocket handshake.Privacy note
I am not attaching raw local logs because they may contain private conversation/session data. The figures above are sanitized operational metrics and command results from the affected machine.