Skip to content

CRITICAL: Gateway process massive virtual memory leak (22GB+) - regenerates immediately after restart #6413

Description

@unchainedio

🐛 Bug Description

The openclaw-gateway process has a severe virtual memory leak causing VSZ to balloon to 22+ GB within minutes of starting. This happens immediately and consistently on every restart, even with minimal activity.

📊 Environment

Component Version/Details
OpenClaw 2026.1.30
OS Ubuntu 24.04 LTS
Kernel Linux 6.8.0-90-generic x64
Node.js v22.22.0
Installation npm global (pnpm)
RAM 3.7 GB
Swap 4 GB (added as mitigation)

🔴 Critical Symptoms

Memory Usage Pattern

PID       %MEM    VSZ        RSS    COMMAND
648981    10.7%   22734080   418688 openclaw-gateway
  • VSZ (Virtual Memory): 22.7 GB and growing
  • RSS (Physical Memory): ~400-500 MB (normal)
  • Growth Rate: Reaches 22GB within 1-2 minutes of restart
  • Persistence: Returns immediately after every restart

Memory Mapping Analysis

  • Two 8 GB anonymous mappings
  • Multiple 2 GB anonymous mappings
  • Numerous 64 MB mappings
  • Pattern suggests mmap() overcommit or heap fragmentation

🔄 Reproduction Steps

  1. Start OpenClaw: openclaw gateway start
  2. Monitor: watch -n 5 'ps aux | grep openclaw-gateway'
  3. Observe: VSZ grows to 22GB within 60-120 seconds
  4. Restart: openclaw gateway restart
  5. Result: Memory returns to 22GB immediately

✅ Tested Mitigations (Failed)

Attempt Result
Restart gateway ❌ Leak returns immediately
Clear sub-agents ❌ Not related to sub-agents
Update to latest (2026.1.30) ❌ Already on latest, no fix
Check session files ❌ Session files only ~2MB
Check for zombie processes ❌ None found
4GB swap added ✅ Prevents OOM crashes only

🛡️ Current Workaround

Auto-heal system implemented:

  • Monitors gateway VSZ every 5 minutes
  • Triggers restart at 15GB threshold
  • Saves conversation context before restart
  • Commits config changes to git
  • System stable but restart-heavy (~every 10-15 min)

📈 Impact Assessment

  • Severity: 🔴 CRITICAL
  • System Stability: At risk without swap
  • User Experience: Disrupted by frequent restarts
  • Data Integrity: No data loss (context preserved)
  • Workaround Effectiveness: 70% (stable but noisy)

🔍 Root Cause Hypothesis

  1. Memory-mapped file leak — mmap() regions not being released
  2. Node.js heap fragmentation — V8 heap growing unbounded
  3. WebSocket connection leak — Connection objects accumulating
  4. Event loop accumulation — Callbacks/events not being cleaned up

📎 Additional Context

  • Issue started: ~2026-02-01 14:00 UTC
  • First noticed: During dashboard/Tailscale configuration
  • Not related to: Session file bloat, sub-agents, cron jobs
  • Process count: Normal (~143 total system processes)
  • No orphaned Node.js processes detected

🙋 Request

URGENT: Please investigate and release a patch. Happy to provide:

  • Heap dumps
  • strace logs
  • Additional diagnostics
  • Testing of fixes

Reported by: @unchainedio
Affected system: Production deployment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions