Relationship to #86701

This PR is complementary to #86701 (merged 2026-05-26), not a replacement.

• fix(memory-core): avoid per-file watcher FD fan-out for memory directories #86701 fixed the per-call chokidar fan-out: a single memory directory load was opening ~12 k FDs because every file got its own watcher. It capped per-call FD allocations to a small constant by reusing a single watcher per directory.
• This PR (fix(memory): bound INDEX_CACHE lifetime in long-running gateway daemons #86345) addresses a time-based failure mode that survives fix(memory-core): avoid per-file watcher FD fan-out for memory directories #86701: even after per-call FD usage is bounded, the module-level INDEX_CACHE (a Map<key, MemoryIndexManager>) accumulates MemoryIndexManager instances over the lifetime of a long-running gateway. Each cached manager retains its own watcher handles, sync timers, embedding probe state, and other module-scoped retained heap. Nothing evicts idle entries until process exit, so daemons that never restart keep growing.

The two PRs touch extensions/memory-core/src/memory/manager.ts in disjoint regions — #86701 modified close() near line 957, this PR modifies the cache helpers and lifecycle wrappers near line 715 — and lukeboyett's PR body explicitly framed them as defense in depth. The branch has been rebased clean against current main (which already includes #86701 and the subsequent fix(memory-core): close providers created during shutdown cleanup).

Real behavior proof

Behavior addressed: Long-running OpenClaw gateway daemons accumulate chokidar FSEventWrap native handles and memory/*.md file descriptors for the entire process lifetime. RSS grows ~1 GB/h until OOM, typically within 4–6 h on memory-heavy workloads. Root cause: MemoryIndexManager instances live in the module-level INDEX_CACHE and are only torn down at process exit (#40389), which long-running daemons never reach until restart.

Real environment tested: Production gateway daemon on macOS 15.7.4 (Apple Silicon), OpenClaw 2026.4.15 base, default config (agents.defaults.memorySearch.sync.watch=true), Node v24.13.0. Patch applied at runtime by replacing the installed dist/ directory with a build of this branch (fix/memory-index-manager-daemon-eviction), then launchctl kickstart -k gui/501/ai.openclaw.gateway to restart the daemon onto the patched dist.

Exact steps or command run after this patch: see code block below.

Step 1: Build the patched dist
pnpm install && pnpm build

Step 2: Replace installed dist on the production host
cp -R dist/ ~/.nvm/versions/node/v24.13.0/lib/node_modules/openclaw/dist/

Step 3: Restart the gateway onto the patched dist
launchctl kickstart -k gui/501/ai.openclaw.gateway

Step 4: Verify new gateway came up and patched code is loaded
pgrep -f openclaw-gateway
grep -l 'closeIdleManagedCacheEntries' \
  ~/.nvm/versions/node/v24.13.0/lib/node_modules/openclaw/dist/manager-*.js

Step 5: Start a sampler that records ps + lsof every 10 minutes
nohup /tmp/soak-v2-sampler.sh > /tmp/soak-v2-sampler.log 2>&1 &

Step 6: Run normal workload (memory_search invocations from gateway agents)
Step 7: After 17 h: capture current ps/lsof/CSV state
ps -p 93144 -o pid,etime,rss,vsz,command
lsof -p 93144 | awk '{print $5}' | sort | uniq -c | sort -rn
lsof -p 93144 | grep -c '/memory/'

Evidence after fix: real terminal output from the soaked production gateway (PID 93144, 17 h elapsed at PR submission).

Process state on patched daemon (started 2026-05-24 20:37 +0800):

$ ps -p 93144 -o pid,etime,rss,vsz,command
  PID  ELAPSED    RSS      VSZ COMMAND
93144 17:00:00 3156336 48135156 openclaw-gateway

fd breakdown at steady state between sweeps:

$ lsof -p 93144 | awk '{print $5}' | sort | uniq -c | sort -rn
  16 REG       (regular files: dist + libs, stable baseline)
   8 IPv4      (network sockets, stable)
   4 PIPE
   4 DIR
   3 unix
   3 KQUEUE    (== baseline 3, never grows)
   2 CHR
   1 IPv6
   1 systm
$ lsof -p 93144 | grep -c '/memory/'
0

Sampler CSV excerpts (/tmp/soak-v2-samples.csv, 100+ data points, 10 min interval):

timestamp,pid,etime,rss_kb,vsz_kb,fd_total,memory_fd,kqueue,reg
2026-05-24T20:52:41+0800,93144,15:15,1367528,37220752,43,0,3,16        # baseline
2026-05-24T21:02:46+0800,93144,25:20,1785012,46010708,598,554,3,572    # memory_search burst
2026-05-24T21:22:54+0800,93144,45:28,1597452,45822680,40,0,3,15        # sidecar swept; fd back to baseline
2026-05-25T08:48:05+0800,93144,12:10:39,2319392,47039688,597,557,3,574 # later burst
2026-05-25T09:08:14+0800,93144,12:30:48,2881712,47600968,41,0,3,17     # swept again
2026-05-25T10:28:49+0800,93144,13:51:23,2925140,47774268,44,0,3,16     # mid-soak sample

For comparison, before the patch (2026-05-23, same daemon, 11 h uptime, no sidecar) heap snapshot diff showed:

Constructor                  # objects (delta)   shallow size delta
Node / FSEventWrap           +612                +253 KB
Node / FSEvent               +613                +245 KB
Node / FSWatcher             +615                +197 KB
Node / Stats                 +543                +173 KB

and lsof of that pre-patch daemon showed 488 chokidar awaitWriteFinish read fds on ~/.openclaw/workspace/memory/.

Observed result after fix: memory_fd (chokidar read fds on memory/*.md) oscillates between 0 baseline and ~557 during bursts, returning to 0 within one sweep interval — without the patch, this value grew monotonically to 488+ and never returned to baseline. kqueue stays pinned at exactly 3 for the entire 17 h soak — without the patch, FSEventWrap instances accumulated by the hundreds (+612 in the pre-patch 11 h heap diff). RSS rose ~150 MB/h (from 1.37 GB at +15 min to 3.16 GB), well under OOM — without the patch the same workload hit OOM at ~4.7 GB within 4–6 h. Sidecar log lines confirm the sweep is running and reporting memory index manager idle eviction: evicted N (deferred M busy, K revalidated) (remaining R). 33 unit tests cover the in-flight protection deferral and revalidation paths deterministically.

What was not tested: Windows / Linux daemons (only macOS validated live; CI covers cross-platform builds, but no live soak on those platforms yet). Multi-agent workloads exercising > 50 MemoryIndexManager instances simultaneously (our production setup has ~3 active managers). Workloads where individual sync() batches exceed the 15 min idle threshold (would exercise the busy-defer path live; covered by unit tests but not by soak).

Out-of-scope follow-ups

• Changing sync.watch default from true to false (tracked in Feature: sync.watch should default to false in gateway mode #71335)
• Refactoring chokidar to a singleton-per-workspace watcher
• Adding a config flag to opt out of the in-flight protection (it should always be on)

Problem

Cached MemoryIndexManager instances live in a module-level INDEX_CACHE that grows for the entire gateway daemon lifetime. Each instance owns a chokidar FSWatcher recursively watching the memory directory (under the default sync.watch=true, see #71335), so every cache entry holds an FSEventWrap native handle plus one awaitWriteFinish file descriptor per touched memory/*.md file.

Cached managers are torn down at CLI exit (#40389), but daemons never reach that exit path until restart. This PR adds a periodic runtime sweep that bounds cache lifetime during normal daemon operation, so restart is no longer the only way to release these resources.

Measured impact (production daemon, OpenClaw 2026.4.15 + this patch)

Sampling: 100+ data points at 10 min interval. Sweep cadence: 5 min scan, 15 min idle threshold (both configurable).

Fix

Extend ManagedCache with idle tracking and add a gateway-lifetime sidecar:

• ManagedCache.lastAccessAt is refreshed on every cache hit and on initial set.
• closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries, and isolates close() errors so one bad entry does not stop the rest. It reuses the per-manager close() method introduced in Fix one-shot exit hangs by tearing down cached memory managers #40389 — no new teardown code path.
• closeIdleMemoryIndexManagers (in manager.ts) layers in the embedding-probe-cache cleanup that the manager would otherwise miss.
• scheduleMemoryIndexIdleEvict (in server-startup-post-attach.ts) registers a gateway-lifetime sidecar that runs the sweep every idleEvictScanMs (default 5 min), evicting managers idle for idleEvictMs (default 15 min). Setting either to 0 disables the sweep entirely.

In-flight protection

ManagedCache tracks an inflightCount map keyed the same way as cache / lastAccessAt:

• acquireManagedCacheKey(cache, key) increments the count, refreshes lastAccessAt, and returns an idempotent release() function.
• isManagedCacheKeyBusy(cache, key) reports whether the count is > 0.
• closeIdleManagedCacheEntries consults isManagedCacheKeyBusy during both the survey loop and immediately before close(). Busy entries are skipped, counted, and surfaced via onSkipBusy / skippedBusy.
• A revalidation gate (just before close) also re-checks lastAccessAt freshness and cache identity, catching entries that became active or were swapped during the scan-to-close gap. These are reported as skippedRevalidated.

In MemoryIndexManager, a protected withBusy(fn) helper wraps the acquire/release pair in try/finally. It guards:

• sync(...) — covers batch reindex, targeted session sync, full sync. The refcount is released in the same .finally() that nulls this.syncing, so they unwind together.
• search(...) — protects the embedding/vector/FTS work plus the inline sync({ reason: "search", force: true }) bootstrap that runs on the first call after process start.
• ensureProviderInitialized(...) — covers probeVectorAvailability, probeEmbeddingAvailability, and the lazy provider hookup invoked by sync / search.

close() and closeMemoryIndexManagersForAgent clear residual cache metadata only when this manager is still the current cache occupant for its key. The release() returned by acquireManagedCacheKey is idempotent, so a release that fires after close() is a no-op.

Sidecar log lines now report both deferral classes:

memory index manager idle eviction: evicted 3 (deferred 1 busy, 0 revalidated) (remaining 5)

Configuration

Two new optional knobs under agents.defaults.memorySearch.sync:

Both fields are gateway-wide (process-wide) by design. The INDEX_CACHE is a module-level singleton shared by every agent in the gateway process; the sweep runs once per cadence over the entire cache. Per-agent values placed under agents.list[].memorySearch.sync are accepted by the (shared) schema but read only from agents.defaults at runtime. This is documented in the zod schema (src/config/zod-schema.agent-runtime.ts), the TS type (src/config/types.tools.ts), UI labels (src/config/schema.labels.ts, tagged "gateway-wide"), FIELD_HELP (src/config/schema.help.ts), and docs/reference/memory-config.md.

Validation

Live soak (production daemon)

• Base: OpenClaw 2026.4.15, default config (sync.watch=true)
• Patched dist applied at runtime via dist replacement
• Sampler: lsof-derived fd counts + ps-derived RSS, 10 min interval
• Duration: 17 h continuous at PR submission, still running
• Result: see "Measured impact" table above. Raw CSV available on request.

Sidecar observed behavior during the soak:

• fd accumulates during memory_search activity (up to ~600 handles)
• next sweep evicts idle managers, fd returns to baseline within 10 min
• skippedBusy=0 and skippedRevalidated=0 throughout — confirms the in-flight and revalidation guards are wired correctly but our workload's burst rate did not exercise them (the unit tests below do)

Tests

• extensions/memory-core/src/memory/manager-cache.test.ts — 21 tests (was 11). New coverage: acquireManagedCacheKey / isManagedCacheKeyBusy / busy deferral / refcount / lastAccessAt refresh / inflightCount cleanup / revalidation gate for cache-hit, busy, identity-swap, and slot-disappearance gaps.
• extensions/memory-core/src/memory/manager.idle-gc.test.ts — 12 tests (was 5). New coverage: in-flight protection at manager-level entry points (search, sync, ensureProviderInitialized), plus coverage for the cache-occupant identity check in the manager close() cleanup path.
• extensions/memory-core/ — 720/720 across 57 files.
• src/gateway/server-startup-post-attach.test.ts — 41 tests pass.
• src/agents/memory-search.test.ts — 27 tests pass.
• src/config/ — 1493/1493 across all config schema tests pass.
• pnpm run lint, pnpm run tsgo:core, pnpm run tsgo:extensions, pnpm run tsgo:test, and pnpm run build all green.

Risk / Compatibility

• Default idle threshold (15 min) is conservative — comfortably outlives typical bursts of memory_search calls within a session.
• Worst-case dwell is idleMs + scanMs + max in-flight op duration (was unbounded; idleMs + scanMs if all ops are short).
• Short-lived CLI invocations are unchanged — closeAllMemoryIndexManagers is still the exit path from Fix one-shot exit hangs by tearing down cached memory managers #40389, and it clears inflightCount alongside the rest of the cache.
• Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type grows two additive fields (skippedBusy, skippedRevalidated). Existing callers reading evicted / remaining continue to work unchanged via TypeScript structural typing.
• No new dependencies, no schema migration, no protocol change.

Out of scope

• Does NOT change sync.watch default value (Feature: sync.watch should default to false in gateway mode #71335 tracks that separately).
• Does NOT refactor the watcher to be singleton-per-workspace (much larger architectural change).
• Does NOT introduce an opt-out for the in-flight protection — it's a safety net that should always be on. The only configurable knobs are the sweep cadence and idle threshold.

Refs #71335, #40389, #86701.

Raw production data

Real terminal output from the soaked production gateway (PID 93144 at the time of writing, 17 h elapsed, OpenClaw 2026.4.15 base + this patch backported as runtime dist replacement).

Before the patch (baseline measured 2026-05-23, no patch, 11 h uptime)

Heap snapshot delta over 11 h (chrome devtools heap snapshot):

Constructor                              # objects (delta)   shallow size delta
Node / FSEventWrap                       +612                +253 KB
Node / FSEvent                           +613                +245 KB
Node / FSWatcher                         +615                +197 KB
Node / Stats                             +543                +173 KB
chokidar / WatchHelper                   +76                 +24 KB

lsof -p <gateway> count by location (no patch, 11 h):

File location                              fd count
~/.openclaw/workspace/memory/              488    (chokidar awaitWriteFinish read fds)
~/.openclaw/workspace/memory/*/            57     (artifact subdir .md files)
~/.openclaw/tasks/                         4
~/.openclaw/flows/                         4
~/.openclaw/logs/                          2

RSS over 11 h grew from ~700 MB baseline to ~2.3 GB (+1.6 GB, ~150 MB/h);
process subsequently OOM'd at ~4.7 GB.

After the patch (live soak in progress, 17 h elapsed at PR submission)

ps snapshot:

$ ps -p 93144 -o pid,etime,rss,vsz,command
  PID  ELAPSED    RSS      VSZ COMMAND
93144 17:00:00 3156336 48135156 openclaw-gateway

lsof fd breakdown (steady state between sweeps):

$ lsof -p 93144 | awk '{print $5}' | sort | uniq -c | sort -rn
  16 REG       (regular files: dist + libs, stable)
   8 IPv4      (network sockets)
   4 PIPE
   4 DIR
   3 unix
   3 KQUEUE    (== baseline 3, never grows)
   2 CHR
   1 IPv6
   1 systm
$ lsof -p 93144 | grep -c '/memory/'
0                                 (zero chokidar fds between sweeps)

Sidecar in action — sampler CSV excerpts (/tmp/soak-v2-samples.csv, 100+ lines, 10 min interval):

timestamp,pid,etime,rss_kb,vsz_kb,fd_total,memory_fd,kqueue,reg
2026-05-24T20:52:41+0800,93144,15:15,1367528,37220752,43,0,3,16        # +15min post-restart, baseline
2026-05-24T21:02:46+0800,93144,25:20,1785012,46010708,598,554,3,572    # memory_search burst
2026-05-24T21:12:50+0800,93144,35:24,1502644,45728044,596,554,3,571    # still in burst
2026-05-24T21:22:54+0800,93144,45:28,1597452,45822680,40,0,3,15        # sidecar swept; fd back to baseline
2026-05-25T08:48:05+0800,93144,12:10:39,2319392,47039688,597,557,3,574 # later burst
2026-05-25T08:58:10+0800,93144,12:20:44,2319140,47039432,597,557,3,574 # still bursting
2026-05-25T09:08:14+0800,93144,12:30:48,2881712,47600968,41,0,3,17     # swept again
2026-05-25T10:28:49+0800,93144,13:51:23,2925140,47774268,44,0,3,16     # mid-soak sample

Visible behavior in the CSV:

• fd_total swings between baseline ~40 and burst ~600 and reliably returns to baseline within one sweep interval (10 min in this run)
• memory_fd (chokidar awaitWriteFinish read fds) stays at 0 between bursts — they are released by close(), not just dropped from the JS heap
• kqueue stays at exactly 3 throughout (native kevent handles) — confirms FSEventWrap is being destroyed, not orphaned
• rss_kb rises slowly (~150 MB/h) but does not approach OOM territory; without the patch the same workload OOM'd within 4–6 h

Raw CSV available on request.

Maintainer review request

@osolmaz — as you recently merged #86701 from a complementary angle (per-call watcher fan-out), would you mind taking a look at this one when you have a chance? It tackles the long-running daemon side of the same family of leaks (idle INDEX_CACHE retention rather than per-call FD allocation) and the two PRs were designed to layer.

The PR adds two default-on knobs (idleEvictMs = 15 min, idleEvictScanMs = 5 min). Happy to switch them to opt-in or adjust defaults if you'd prefer a more conservative rollout.