You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Active-memory plugin spawns an embedded sub-agent for every eligible message. When the embedded sub-agent times out with "embedded abort settle timed out", the embedded session is not cleaned up — the runningAtMs flag remains set, the session object leaks, and RSS grows monotonically until the gateway crashes with OOM (~1.9 GB RSS, crash cycle every ~11 hours).
Symptoms (from production instance)
RSS grows from 318 MB → 610 MB in 3 minutes after gateway restart
Full crash cycle: ~1.9 GB RSS → OOM → crash
Journal pattern:
embedded_run_failover_decision: surface_error, timeout
embedded abort settle timed out
After disabling active-memory: RSS stable at ~400 MB, no crashes
Memory tools (memory_search, memory_get) continue to work independently of the plugin. The plugin itself is the only component that leaks.
Evidence
Our gateway ran for 5 days with active-memory enabled and crashed every ~11 hours with OOM (1.9 GB RSS on a 2 GB system, 4 GB swap exhausted). After disabling active-memory, the gateway has been stable for 48+ hours with RSS at ~400 MB.
Multiple restart cycles confirmed the correlation:
None of the open issues describe the session leak → OOM crash cycle specifically. The common thread is that active-memory embedded runs are fragile across multiple failure modes, but the memory leak after timeout is not yet tracked.
Suggested Fix
Session cleanup on timeout: When "embedded abort settle timed out" occurs, ensure the embedded session is terminated and its memory freed
Reset runningAtMs flag: The session lock should be released when the embedded run times out
Add leak detection: Monitor embedded session count per agent; if sessions accumulate without completion, log a warning
Summary
Active-memory plugin spawns an embedded sub-agent for every eligible message. When the embedded sub-agent times out with
"embedded abort settle timed out", the embedded session is not cleaned up — therunningAtMsflag remains set, the session object leaks, and RSS grows monotonically until the gateway crashes with OOM (~1.9 GB RSS, crash cycle every ~11 hours).Symptoms (from production instance)
Root Cause
runEmbeddedPiAgentinsidebefore_prompt_buildtimeoutMsexceeded), the error path does not properly clean up the embedded sessionrunningAtMsflag in the session lock remains setWorkaround
Disable active-memory plugin:
{ "plugins.entries.active-memory": { "enabled": false, "_bug": "B1 — session leak after embedded abort settle timeout" } }Memory tools (
memory_search,memory_get) continue to work independently of the plugin. The plugin itself is the only component that leaks.Evidence
Our gateway ran for 5 days with active-memory enabled and crashed every ~11 hours with OOM (1.9 GB RSS on a 2 GB system, 4 GB swap exhausted). After disabling active-memory, the gateway has been stable for 48+ hours with RSS at ~400 MB.
Multiple restart cycles confirmed the correlation:
Related Issues
None of the open issues describe the session leak → OOM crash cycle specifically. The common thread is that active-memory embedded runs are fragile across multiple failure modes, but the memory leak after timeout is not yet tracked.
Suggested Fix
"embedded abort settle timed out"occurs, ensure the embedded session is terminated and its memory freedrunningAtMsflag: The session lock should be released when the embedded run times outEnvironment
ps+ journalctl OOM killer logs