Skip to content

v2026.4.2 gateway memory leak: RSS grows to 3-4GB within 1-2 hours, silent OOM crash #62717

Description

@bryangauvin

Bug Report: Gateway Memory Leak Causing Silent OOM Crashes

Summary

The OpenClaw gateway (v2026.4.2) exhibits a persistent memory leak where RSS grows from ~560MB at startup to 3-4GB within 1-2 hours, eventually crashing silently. The process disappears without logging any error, SIGTERM, or heap-related message. launchd relaunches it automatically, but active agent runs are killed with process_lost.

Environment

  • Platform: macOS arm64, Mac Mini, 16GB RAM
  • Node: v22.22.1 with --max-old-space-size=6144
  • OpenClaw: v2026.4.2 (d74a122)
  • Config: 12 agents, 19 enabled cron jobs, Discord + iMessage channels, maxConcurrent=2
  • Session files: Pruned to 10-20 per agent, sessions.json ~10MB total

Observed Behavior

Gateway restarts observed today (April 7): 6 restarts in 12 hours

Memory growth pattern:

  • Startup: ~560MB RSS (3.5% of 16GB)
  • After 20 min: ~3.6GB (22%)
  • After 1-2 hours: ~4.5GB → silent crash
  • No SIGTERM, no heap error, no log entry before crash

This pattern persists even after:

  1. Pruning session files to 10 per agent
  2. Cleaning sessions.json indexes (10.5MB total)
  3. Removing stale .deleted/.tmp files
  4. Setting --max-old-space-size=6144

Root Cause Analysis

The memory leak appears to be in the gateway's runtime state accumulation, not in session file loading. With only 10-20 session files per agent (~270MB total on disk, ~30MB in sessions.json indexes), the gateway still grows to 3-4GB RSS. This suggests:

  1. Agent run state (embedded runs, tool call history) is not being released after runs complete
  2. The cron scheduler accumulates job state across runs
  3. WebSocket connection state may not be properly GC'd after connections close

Impact

  • Active agent runs killed with process_lost every 1-2 hours
  • Tasks must be retried, wasting LLM tokens and time
  • The Paperclip integration sees repeated failures for long-running tasks (>15 min runs impossible if gateway crashes every 60-90 min)

Workarounds Applied

  1. --max-old-space-size=6144 in launchd plist (delays crash but doesn't prevent it)
  2. Automated session pruning every 2 hours (reduces startup memory, doesn't fix runtime leak)
  3. ThrottleInterval=10 in launchd (prevents restart loops)

Suggested Investigation

  • Profile heap growth using --inspect flag during a 30-minute run
  • Check if WeakRef or finalizers are used for run/session state
  • Check if the embedded agent runner holds references to completed runs
  • Check cron job scheduler state accumulation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions