Skip to content

Gateway becomes slow or times out under multi-session / multi-agent load #92057

Description

@xiaopings

Summary

On a local OpenClaw installation, the gateway becomes very slow and can appear stuck when the system has many sessions and multiple agents running. The symptoms are most visible during multi-session / multi-agent execution: inbound messages are delayed, gateway RPCs time out, and some Codex/OpenClaw dynamic tool calls can hang or time out.

This has reproduced more than once on the same machine. A previous local audit on 2026-06-07 pointed to gateway event-loop stalls caused or amplified by heavy log/history/session IO. The issue is currently observable again.

Current observed behavior

As of 2026-06-11 08:xx CST on the affected machine:

  • openclaw gateway status --json took about 11.09s.
  • The gateway process is running and listening on 127.0.0.1:18789, but RPC probing fails:
    • rpc.ok: false
    • error: "timeout"
    • url: "ws://127.0.0.1:18789"
  • openclaw gateway stability --json fails with:
    • GatewayTransportError: gateway timeout after 10000ms
  • Process snapshot showed the gateway process consuming close to a full CPU core:
    • /opt/homebrew/opt/node@22/bin/node /opt/homebrew/lib/node_modules/openclaw/dist/index.js gateway --port 18789
    • uptime around 1h20m+
    • CPU around 97% at the time of capture
  • openclaw sessions list --json still returns, but the environment has a large session store.

Local environment

  • OpenClaw CLI: 2026.6.5
  • Gateway command: openclaw/dist/index.js gateway --port 18789
  • Node: v22.22.0
  • npm: 10.9.4
  • macOS: 26.5.1 / Darwin 25.5.0 / arm64
  • OpenClaw Codex package: @openclaw/codex 2026.5.27
  • Weixin gateway plugin: @tencent-weixin/openclaw-weixin 2.4.3
  • Gateway bind mode: loopback, port 18789

Session / data scale on the affected machine

The affected installation uses multiple agents and a large local session history:

  • ~/.openclaw/agents/*/sessions: about 4820 jsonl files.
  • Main agent sessions directory: about 881MB.
  • Other agent sessions directories include examples around 198MB, 136MB, 108MB, 88MB, 57MB.
  • Largest archived trajectory/checkpoint jsonl files observed: roughly 10MB to 26MB each.

This looks like the kind of workload that can accumulate naturally when OpenClaw is used as a long-running multi-agent coordinator.

Earlier local audit evidence from 2026-06-07

A previous investigation of a similar gateway hang on the same machine found:

  • logs.tail latency up to 42946ms.
  • eventLoopDelayP99Ms=11467.2.
  • eventLoopUtilization=1.
  • WebSocket handshakes closing with code 1006 during or after stalls.
  • codex dynamic tool call timed out after 120000ms.
  • Persistent logs had grown large before rotation:
    • ~/.openclaw/logs/gateway.log: 32M, 532900 lines.
    • ~/.openclaw/logs/gateway.err.log: 11M, 104560 lines.

Rotating the oversized gateway logs improved logs.tail and general gateway responsiveness at that time, but the broader multi-session / multi-agent degradation appears to recur with large session/history state.

Expected behavior

The gateway should remain responsive under a large but realistic local multi-agent workload. Expensive operations such as log tailing, session listing, history reads, or transcript scans should not block the event loop long enough to cause WebSocket/RPC timeouts or message delivery delays.

Actual behavior

Under the described workload, the gateway can become slow enough that:

  • gateway status reports a running process but RPC timeout.
  • gateway stability times out.
  • Message delivery and multi-agent execution feel delayed or stuck.
  • Gateway CPU can sit near a full core while the local client observes timeouts.

Suspected contributing factors

Based on local evidence, possible contributors include:

  • Synchronous or unbounded file IO in gateway request paths, especially around log tail/history/session operations.
  • Full-file scans for log tailing or session history where bounded reverse reads or indexed summaries would be safer.
  • Large session/trajectory/checkpoint stores increasing the cost of session enumeration or history lookups.
  • Long-running model/tool calls amplifying gateway backpressure when the event loop is already delayed.

Suggested hardening

Possible fixes or mitigations:

  • Ensure logs.tail uses bounded reverse-tail reads and never scans full large logs on the gateway event loop.
  • Move expensive session/history/log IO off the main gateway event loop or make it streaming/backpressure-aware.
  • Add explicit latency metrics for gateway handlers such as sessions.list, chat.history, logs.tail, dynamic tool dispatch, and WebSocket handshake.
  • Add guardrails for large local session stores, for example indexed metadata, pagination by default, and cleanup recommendations when thresholds are exceeded.
  • Consider a built-in maintenance command or background compaction/rotation policy for long-running multi-agent installs.

Privacy note

I am not attaching raw local logs because they may contain private conversation/session data. The figures above are sanitized operational metrics and command results from the affected machine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions