Skip to content

Bug: Gateway OOM crash from Discord EventQueue backlog (RSS grows to 4GB) #30212

Description

@lemodigital

Problem

Gateway process RSS grows continuously when Discord EventQueue has sustained backlog, eventually hitting the Node.js heap limit (~4GB) and crashing with FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory.

Environment

  • OpenClaw: 2026.2.26
  • macOS 26.3 (arm64)
  • Node.js: v22.22.0
  • 7 Discord bot accounts, 30+ channels/threads

Evidence

OOM Crash (Feb 25)

[2662:0x7bd40c000] 4828960 ms: Mark-Compact 4022.6 (4123.9) -> 3988.4 (4112.5) MB
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Memory Growth Pattern (Mar 1)

  • 11:01 AM: Gateway restart, RSS ~200MB
  • 11:56 AM: RSS 537MB
  • 12:04 PM: RSS 730MB
  • 12:10 PM: RSS 1058MB (proactive restart at this point)
  • Growth rate: ~100MB/min during heavy EventQueue backlog

EventQueue Backlog

  • 17,480+ slow listener events in ~1 hour
  • DiscordReactionListener: 1-12 seconds per event
  • DiscordMessageListener: 30-87 seconds per event
  • Rate: ~70 slow events/minute sustained

Preceding Symptoms (from error log)

[discord] Request queue is full (1000 / 1000)
[discord] Slow listener detected: DiscordMessageListener took 1497.2 seconds
[diagnostic] lane wait exceeded: lane=session:agent:circe:discord:channel:... waitedMs=1826140 queueAhead=62

Impact

  • Gateway crashed and was down for 5h17m (LaunchAgent KeepAlive did not recover quickly)
  • 3 weekly cron jobs (soul-evolution) were skipped
  • All Discord-based agent communication was interrupted

Root Cause Analysis

The EventQueue processes Discord events (reactions, messages) synchronously. When LLM responses take >30s, the listener times out but the event remains in the queue. With 7 bot accounts receiving events across 30+ channels, the backlog grows faster than it drains. Each queued event holds references that prevent GC, causing RSS to grow linearly until OOM.

Related Issues

Suggested Fix

  1. Short-term: Merge PR fix(discord): expose EventQueue listenerTimeout as configurable option #28945 (configurable EventQueue timeout)
  2. Medium-term: Add EventQueue backpressure — drop/skip events when queue exceeds threshold
  3. Long-term: Process Discord events in a worker thread to prevent event loop blocking

Workaround

Periodic openclaw gateway restart (every 4-6 hours) to clear the EventQueue and reset RSS.

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions