Skip to content

[Bug]: Active-memory plugin embedded session leak → OOM (embedded abort settle timed out) #85644

Description

@paluj85-cloud

Summary

Active-memory plugin spawns an embedded sub-agent for every eligible message. When the embedded sub-agent times out with "embedded abort settle timed out", the embedded session is not cleaned up — the runningAtMs flag remains set, the session object leaks, and RSS grows monotonically until the gateway crashes with OOM (~1.9 GB RSS, crash cycle every ~11 hours).

Symptoms (from production instance)

  • RSS grows from 318 MB → 610 MB in 3 minutes after gateway restart
  • Full crash cycle: ~1.9 GB RSS → OOM → crash
  • Journal pattern:
    embedded_run_failover_decision: surface_error, timeout
    embedded abort settle timed out
    
  • After disabling active-memory: RSS stable at ~400 MB, no crashes

Root Cause

  1. Active-memory calls runEmbeddedPiAgent inside before_prompt_build
  2. When the embedded agent times out (timeoutMs exceeded), the error path does not properly clean up the embedded session
  3. The runningAtMs flag in the session lock remains set
  4. Each subsequent message spawns a NEW embedded session without freeing the previous one
  5. Memory accumulates with each timed-out embedded run → unbounded RSS growth → OOM

Workaround

Disable active-memory plugin:

{
  "plugins.entries.active-memory": {
    "enabled": false,
    "_bug": "B1 — session leak after embedded abort settle timeout"
  }
}

Memory tools (memory_search, memory_get) continue to work independently of the plugin. The plugin itself is the only component that leaks.

Evidence

Our gateway ran for 5 days with active-memory enabled and crashed every ~11 hours with OOM (1.9 GB RSS on a 2 GB system, 4 GB swap exhausted). After disabling active-memory, the gateway has been stable for 48+ hours with RSS at ~400 MB.

Multiple restart cycles confirmed the correlation:

  • Restart → RSS climbs 318→610 MB within 3 min
  • 11h later → 1.9 GB → crash
  • After plugin disabled → stable at ~400 MB

Related Issues

None of the open issues describe the session leak → OOM crash cycle specifically. The common thread is that active-memory embedded runs are fragile across multiple failure modes, but the memory leak after timeout is not yet tracked.

Suggested Fix

  1. Session cleanup on timeout: When "embedded abort settle timed out" occurs, ensure the embedded session is terminated and its memory freed
  2. Reset runningAtMs flag: The session lock should be released when the embedded run times out
  3. Add leak detection: Monitor embedded session count per agent; if sessions accumulate without completion, log a warning
  4. Consider non-blocking mode: Per Reliability: active-memory blocks replies and QMD boot initialization can overload multi-agent gateways #72015 suggestion, make active-memory non-blocking by default (opportunistic recall that does not block user replies)

Environment

  • OpenClaw: 2026.5.x on Linux (Debian 12, arm64)
  • Node v22
  • Gateway running under systemd
  • RSS monitored via ps + journalctl OOM killer logs

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Emergency: data loss, security bypass, crash loop, or unusable core runtime.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions