Skip to content

Fix one-shot exit hangs by tearing down cached memory managers#40389

Merged
frankekn merged 5 commits into
openclaw:mainfrom
Julbarth:fix/one-shot-memory-teardown-clean
Mar 9, 2026
Merged

Fix one-shot exit hangs by tearing down cached memory managers#40389
frankekn merged 5 commits into
openclaw:mainfrom
Julbarth:fix/one-shot-memory-teardown-clean

Conversation

@Julbarth

@Julbarth Julbarth commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Describe the problem and fix in 2–5 bullets:

  • Problem: One-shot CLI runs can emit successful completion events but still hang on exit.
  • Why it matters: Bench/automation workflows can time out even when the task itself completed.
  • What changed: Added explicit global teardown for cached memory search/index managers and called teardown in CLI finally; also drained in-flight index manager creations during teardown.
  • What did NOT change (scope boundary): No model routing/tool behavior changes, no plugin/channel behavior changes, no secret/token handling changes.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

User-visible / Behavior Changes

One-shot CLI commands now exit cleanly after completion in scenarios where cached memory-manager watcher handles previously kept the process alive.

Security Impact (required)

  • New permissions/capabilities? (No)
  • Secrets/tokens handling changed? (No)
  • New/changed network calls? (No)
  • Command/tool execution surface changed? (No)
  • Data access scope changed? (No)
  • If any Yes, explain risk + mitigation:

Repro + Verification

Environment

  • OS: macOS 14.0 (local repro); CI includes Linux/Windows jobs
  • Runtime/container: Node.js + pnpm
  • Model/provider: N/A (provider-independent; reproduced with successful one-shot completion)
  • Integration/channel (if any): CLI one-shot run
  • Relevant config (redacted): default memory settings (memory search enabled)

Steps

  1. Run one-shot CLI command on pre-fix build with default memory enabled.
  2. Observe completion events emitted successfully.
  3. Process remains alive instead of exiting.

Expected

  • Process exits after successful completion.

Actual

  • Process can remain alive due to lingering memory-manager resources (watcher handles), with race risk from in-flight manager creation.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios:
    • Reproduced one-shot hang before fix.
    • Confirmed clean exit after fix.
    • Ran targeted tests:
      • pnpm exec vitest run src/cli/run-main.exit.test.ts
      • pnpm exec vitest run src/memory/search-manager.test.ts
      • pnpm exec vitest run src/memory/manager.get-concurrency.test.ts
  • Edge cases checked:
    • Teardown called through lazy runtime boundary (no eager manager import path).
    • Global teardown drains pending manager creations.
    • QMD manager cache teardown still closes and resets as expected.
  • What you did not verify:
    • Long-duration stress testing across all providers/channels.
    • Full production workload profiling.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

  • Backward compatible? (Yes)
  • Config/env changes? (No)
  • Migration needed? (No)
  • If yes, exact upgrade steps:

Failure Recovery (if this breaks)

  • How to disable/revert this change quickly: Revert this PR commit set.
  • Files/config to restore: src/cli/run-main.ts, src/cli/run-main.exit.test.ts, src/memory/manager.ts, src/memory/manager-runtime.ts, src/memory/search-manager.ts, src/memory/search-manager.test.ts, src/memory/manager.get-concurrency.test.ts, src/memory/index.ts
  • Known bad symptoms reviewers should watch for: one-shot process not exiting, or teardown-time errors in memory shutdown path.

Risks and Mitigations

List only real risks for this PR. Add/remove entries as needed. If none, write None.

  • Risk: Teardown during in-flight creation could cause lifecycle races.
    • Mitigation: Explicitly await pending creations before closing cached managers.
  • Risk: Accidental eager loading of memory manager module on non-memory paths.
    • Mitigation: Teardown path uses lazy runtime import boundary; tests cover behavior.

@greptile-apps

greptile-apps Bot commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes one-shot CLI exit hangs caused by lingering FSWatcher handles inside cached MemoryIndexManager instances. The fix adds explicit global teardown functions (closeAllMemorySearchManagers / closeAllMemoryIndexManagers) and calls them in the CLI finally block, ensuring all watcher handles are released after the program finishes.

Key changes:

  • src/cli/run-main.ts: Wraps the main CLI body in try/finally; calls closeCliMemoryManagers() (lazy dynamic import) on every exit path, including the tryRouteCli early-return path.
  • src/memory/manager.ts: closeAllMemoryIndexManagers() drains INDEX_CACHE_PENDING via Promise.allSettled before clearing INDEX_CACHE and closing each manager — correctly handles in-flight creations.
  • src/memory/search-manager.ts: closeAllMemorySearchManagers() closes QMD (FallbackMemoryManager) entries first, then delegates to closeAllMemoryIndexManagers(). The implementation unconditionally calls loadManagerRuntime() even when memory was never accessed, which loads the manager module purely as a teardown artifact and breaks the lazy-import boundary.
  • src/memory/search-manager.test.ts: Correctly migrates mockPrimary, fallbackManager, and related mocks to vi.hoisted() (required because vi.mock() factories are hoisted above module scope) and fixes the mocked module path from ./manager.js./manager-runtime.js to match the actual import in production code.
  • src/memory/manager.get-concurrency.test.ts: New test verifies that closeAllMemoryIndexManagers drains a slow in-flight get() and that a subsequent get() returns a fresh (non-cached) manager instance.

Confidence Score: 4/5

  • Safe to merge; fixes a real hang with correct lifecycle ordering. One minor issue with lazy-loading boundary breaks when teardown loads manager-runtime unconditionally.
  • The PR correctly implements global teardown with proper draining of in-flight manager creations before closure. Tests cover critical scenarios. Teardown errors are silently swallowed, preventing regressions on non-memory paths. The core logic for fixing the hang is sound. However, closeAllMemorySearchManagers unconditionally imports manager-runtime.js even when it was never loaded during execution, breaking the lazy boundary property on CLI paths that never accessed memory. This is a correctness issue with an easy fix but doesn't impact functional safety.
  • src/memory/search-manager.ts — the unconditional loadManagerRuntime() call should be guarded to preserve lazy-loading boundary.

Last reviewed commit: cec15df

Comment thread src/memory/search-manager.ts Outdated
@Julbarth
Julbarth force-pushed the fix/one-shot-memory-teardown-clean branch 2 times, most recently from 0bc26dd to 524f80e Compare March 9, 2026 01:37
@Julbarth

Julbarth commented Mar 9, 2026

Copy link
Copy Markdown
Contributor Author

All feedback addressed, CI green, ready for merge.

@frankekn frankekn self-assigned this Mar 9, 2026
@frankekn
frankekn force-pushed the fix/one-shot-memory-teardown-clean branch 2 times, most recently from 141558b to f07f95a Compare March 9, 2026 15:53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f07f95a2a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/run-main.ts Outdated
@@ -13,6 +13,15 @@ import { applyCliProfileEnv, parseCliProfileArgs } from "./profile.js";
import { tryRouteCli } from "./route.js";
import { normalizeWindowsArgv } from "./windows-argv.js";

async function closeCliMemoryManagers(): Promise<void> {
try {
const { closeAllMemorySearchManagers } = await import("../memory/index.js");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid eager memory-module import during CLI teardown

runCli now always executes closeCliMemoryManagers() in finally, and that helper imports ../memory/index.js on every command path. Because src/memory/index.ts statically re-exports MemoryIndexManager from manager.ts, even routed/fast-path commands that never used memory still load the full memory stack at shutdown, which defeats route-first startup goals and adds avoidable module-load overhead (or failures in stripped runtimes). Use a teardown import that stays on the lazy runtime boundary instead of the memory barrel export.

Useful? React with 👍 / 👎.

@frankekn
frankekn force-pushed the fix/one-shot-memory-teardown-clean branch 3 times, most recently from 8bdc1f2 to 6b02314 Compare March 9, 2026 23:28
@frankekn
frankekn force-pushed the fix/one-shot-memory-teardown-clean branch from 6b02314 to 0e600e8 Compare March 9, 2026 23:34
@frankekn
frankekn merged commit c0cba7f into openclaw:main Mar 9, 2026
25 of 26 checks passed
@frankekn

frankekn commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

Thanks @Julbarth!

jenawant pushed a commit to jenawant/openclaw that referenced this pull request Mar 10, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
aiwatching pushed a commit to aiwatching/openclaw that referenced this pull request Mar 10, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
Moshiii pushed a commit to Moshiii/openclaw that referenced this pull request Mar 11, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
Moshiii pushed a commit to Moshiii/openclaw that referenced this pull request Mar 11, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
Ruijie-Ysp pushed a commit to Ruijie-Ysp/clawdbot that referenced this pull request Mar 12, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
V-Gutierrez pushed a commit to V-Gutierrez/openclaw-vendor that referenced this pull request Mar 17, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
alexey-pelykh pushed a commit to remoteclaw/remoteclaw that referenced this pull request Mar 22, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn

(cherry picked from commit c0cba7f)
alexey-pelykh pushed a commit to remoteclaw/remoteclaw that referenced this pull request Mar 22, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn

(cherry picked from commit c0cba7f)
lovewanwan pushed a commit to lovewanwan/openclaw that referenced this pull request Apr 28, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…law#40389)

Merged via squash.

Prepared head SHA: 0e600e8
Co-authored-by: Julbarth <[email protected]>
Co-authored-by: frankekn <[email protected]>
Reviewed-by: @frankekn
quengh added a commit to quengh/openclaw that referenced this pull request May 25, 2026
Cached MemoryIndexManager instances live in a module-level INDEX_CACHE
that grows for the entire gateway daemon lifetime. Each instance owns a
chokidar FSWatcher recursively watching the memory directory (under the
default sync.watch=true), so every cache entry holds an FSEventWrap
native handle plus one awaitWriteFinish file descriptor per touched
memory/*.md file.

Cached managers are torn down at CLI exit (openclaw#40389), but daemons never
reach that exit path until restart. A production daemon (OpenClaw
2026.4.15) measured +1.6 GB RSS over 11 h and 612 accumulated
FSEventWrap handles before crashing.

This change adds a periodic runtime sweep that closes idle managers
during normal daemon operation, so restart is no longer the only way
to release these resources.

Approach
--------

ManagedCache:
- Track lastAccessAt per cache key (refreshed on hit and on initial set).
- closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries,
  and isolates close() errors so one bad entry does not stop the rest.
  Reuses the per-manager close() introduced in openclaw#40389 — no new teardown
  code path.

In-flight protection (always-on safety net):
- inflightCount map keyed the same way as cache / lastAccessAt.
- acquireManagedCacheKey() increments the count, refreshes lastAccessAt,
  and returns an idempotent release() function.
- isManagedCacheKeyBusy() reports whether the count is > 0.
- closeIdleManagedCacheEntries consults isManagedCacheKeyBusy in both
  the survey loop and immediately before close(); busy entries are
  skipped, counted, and surfaced via onSkipBusy / skippedBusy.
- A revalidation gate (just before close) also re-checks lastAccessAt
  freshness and cache identity, catching entries that became active or
  were swapped during the scan-to-close gap. Reported as
  skippedRevalidated.

MemoryIndexManager:
- Protected withBusy(fn) helper wraps acquire/release in try/finally.
- Guards sync(), search(), and ensureProviderInitialized() so the busy
  ref is held across provider init as well as the manager work itself.
- close() and closeMemoryIndexManagersForAgent clear any residual
  inflightCount entry alongside the cache slot. release() is idempotent
  so a release that fires after close() is a no-op.

Gateway sidecar:
- scheduleMemoryIndexIdleEvict registers a gateway-lifetime sidecar
  that calls closeIdleMemorySearchManagers every idleEvictScanMs
  (default 5 min), evicting managers idle for idleEvictMs (default
  15 min). Setting either to 0 disables the sweep entirely.

Configuration
-------------

Two new optional fields under agents.defaults.memorySearch.sync:
- idleEvictMs (default 900000): idle TTL before a manager is eligible
  for eviction. 0 disables.
- idleEvictScanMs (default 300000): sweep cadence. 0 disables.

Documented in the config schema (zod), the MemorySearchConfig type,
and the UI field labels map.

Validation
----------

Live soak on the production daemon (OpenClaw 2026.4.15 + this patch
backported, 13 h+ continuous, sampling every 10 min):

| Metric          | Without patch (11 h)    | With patch (13 h soak)        |
|-----------------|-------------------------|-------------------------------|
| FSEventWrap     | 612 native handles      | bounded, returns to ~3 baseline |
| memory_fd       | 488 (chokidar holds)    | 0 between sweeps              |
| RSS growth rate | ~1 GB/h                 | ~150 MB/h (~7x improvement)   |
| Outcome at 13 h | OOM within 4 to 6 h     | 3.25 GB stable, no OOM        |

Sidecar observed behavior:
- fd accumulates during memory_search activity (up to ~600 handles)
- next sweep evicts idle managers, fd returns to baseline within 10 min
- skippedBusy and skippedRevalidated stayed at 0 throughout the soak —
  the workload burst rate did not exercise the deferral paths, but the
  unit tests cover them deterministically

Tests
-----

- extensions/memory-core/src/memory/manager-cache.test.ts: 21 tests
  (was 11). New coverage: acquireManagedCacheKey, isManagedCacheKeyBusy,
  busy deferral, refcount, lastAccessAt refresh, inflightCount cleanup,
  revalidation gate for cache-hit / busy / identity-swap / slot-
  disappearance gaps.
- extensions/memory-core/src/memory/manager.idle-gc.test.ts: 10 tests
  (was 5). New coverage: in-flight protection at manager-level entry
  points (search, sync, ensureProviderInitialized).
- extensions/memory-core/: 718/718 across 57 files.
- src/gateway/server-startup-post-attach.test.ts: 41 tests pass.
- src/agents/memory-search.test.ts: 27 tests pass.
- pnpm run lint, tsgo:core, tsgo:extensions, tsgo:test, build all green.

Risk / Compatibility
--------------------

- Default idle threshold (15 min) is conservative — comfortably outlives
  typical bursts of memory_search calls within a session.
- Worst-case dwell is idleMs + scanMs + max in-flight op duration
  (was unbounded; idleMs + scanMs if all ops are short).
- Short-lived CLI invocations are unchanged: closeAllMemoryIndexManagers
  remains the exit path from openclaw#40389, and it clears inflightCount
  alongside the rest of the cache.
- Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type
  grows two additive fields (skippedBusy, skippedRevalidated). Existing
  callers reading evicted / remaining continue to work unchanged via
  TypeScript structural typing.
- No new dependencies, no schema migration, no protocol change.

Out of scope
------------

- Does NOT change sync.watch default value (openclaw#71335 tracks that
  separately).
- Does NOT refactor the watcher to be singleton-per-workspace (much
  larger architectural change).
- Does NOT introduce an opt-out for the in-flight protection — it's a
  safety net that should always be on. The only configurable knobs are
  the sweep cadence and idle threshold.

Refs openclaw#71335, openclaw#40389.
quengh added a commit to quengh/openclaw that referenced this pull request May 25, 2026
Cached MemoryIndexManager instances live in a module-level INDEX_CACHE
that grows for the entire gateway daemon lifetime. Each instance owns a
chokidar FSWatcher recursively watching the memory directory (under the
default sync.watch=true), so every cache entry holds an FSEventWrap
native handle plus one awaitWriteFinish file descriptor per touched
memory/*.md file.

Cached managers are torn down at CLI exit (openclaw#40389), but daemons never
reach that exit path until restart. A production daemon (OpenClaw
2026.4.15) measured +1.6 GB RSS over 11 h and 612 accumulated
FSEventWrap handles before OOM.

This change adds a periodic runtime sweep that closes idle managers
during normal daemon operation, so restart is no longer the only way
to release these resources.

Approach
--------

ManagedCache:
- Track lastAccessAt per cache key (refreshed on hit and on initial set).
- closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries,
  and isolates close() errors so one bad entry does not stop the rest.
  Reuses the per-manager close() introduced in openclaw#40389 - no new teardown
  code path.

In-flight protection (always-on safety net):
- inflightCount map keyed the same way as cache / lastAccessAt.
- acquireManagedCacheKey() increments the count, refreshes lastAccessAt,
  and returns an idempotent release() function.
- isManagedCacheKeyBusy() reports whether the count is > 0.
- closeIdleManagedCacheEntries consults isManagedCacheKeyBusy in both
  the survey loop and immediately before close(); busy entries are
  skipped, counted, and surfaced via onSkipBusy / skippedBusy.
- A revalidation gate (just before close) also re-checks lastAccessAt
  freshness and cache identity, catching entries that became active or
  were swapped during the scan-to-close gap. Reported as
  skippedRevalidated.
- Cache metadata cleanup in MemoryIndexManager.close() and
  closeMemoryIndexManagersForAgent is now scoped to the cache identity
  check that previously guarded only the INDEX_CACHE delete, so a
  replacement manager installed in the same key during the close()
  await window keeps its own lastAccessAt and inflightCount entries.

MemoryIndexManager:
- Protected withBusy(fn) helper wraps acquire/release in try/finally.
- Guards sync(), search(), and ensureProviderInitialized() so the busy
  ref is held across provider init as well as the manager work itself.
- close() and closeMemoryIndexManagersForAgent clear residual cache
  metadata only when this instance is still the cache occupant.
  release() is idempotent so a release that fires after close() is a
  no-op.

Gateway sidecar:
- scheduleMemoryIndexIdleEvict registers a gateway-lifetime sidecar
  that calls closeIdleMemorySearchManagers every idleEvictScanMs
  (default 5 min), evicting managers idle for idleEvictMs (default
  15 min). Setting either to 0 disables the sweep entirely.
- Resolved from agents.defaults.memorySearch.sync only. The cache is a
  process-wide singleton and the sweep runs once per cadence over the
  entire cache; there is no per-agent sweep timer, and no coherent
  per-agent answer when two agents share a workspace and disagree on
  the threshold. Per-agent values under agents.list[].memorySearch.sync
  are accepted by the (shared) schema but ignored at runtime; this is
  documented on the TS doc, zod comments, UI labels, FIELD_HELP, and
  in docs/reference/memory-config.md.

Configuration
-------------

Two new optional fields under agents.defaults.memorySearch.sync:
- idleEvictMs (default 900000): idle TTL before a manager is eligible
  for eviction. 0 disables.
- idleEvictScanMs (default 300000): sweep cadence. 0 disables.

Documented in:
- src/config/zod-schema.agent-runtime.ts (schema with scope contract)
- src/config/types.tools.ts (MemorySearchConfig TS doc)
- src/config/schema.labels.ts (UI labels, tagged "gateway-wide")
- src/config/schema.help.ts (FIELD_HELP user-facing copy)
- docs/reference/memory-config.md (reference docs with ParamField blocks)

Validation
----------

Live soak on the production daemon (OpenClaw 2026.4.15 + this patch
backported, 17 h+ continuous at PR submission, sampling every 10 min):

| Metric          | Without patch (11 h)    | With patch (17 h soak)        |
|-----------------|-------------------------|-------------------------------|
| FSEventWrap     | 612 native handles      | bounded, returns to ~3 baseline |
| memory_fd       | 488 (chokidar holds)    | 0 between sweeps              |
| RSS growth rate | ~1 GB/h                 | ~150 MB/h (~7x improvement)   |
| Outcome         | OOM within 4 to 6 h     | 3.16 GB stable, no OOM        |

Sidecar observed behavior:
- fd accumulates during memory_search activity (up to ~600 handles)
- next sweep evicts idle managers, fd returns to baseline within 10 min
- skippedBusy and skippedRevalidated stayed at 0 throughout the soak -
  the workload burst rate did not exercise the deferral paths, but the
  unit tests cover them deterministically

Tests
-----

- extensions/memory-core/src/memory/manager-cache.test.ts: 21 tests
  (was 11). New coverage: acquireManagedCacheKey, isManagedCacheKeyBusy,
  busy deferral, refcount, lastAccessAt refresh, inflightCount cleanup,
  revalidation gate for cache-hit / busy / identity-swap / slot-
  disappearance gaps.
- extensions/memory-core/src/memory/manager.idle-gc.test.ts: 12 tests
  (was 5). New coverage: in-flight protection at manager-level entry
  points (search, sync, ensureProviderInitialized), plus cache-occupant identity-check
  coverage for the close() metadata-cleanup path.
- extensions/memory-core/: 720/720 across 57 files.
- src/gateway/server-startup-post-attach.test.ts: 41 tests pass.
- src/agents/memory-search.test.ts: 27 tests pass.
- src/config/: 1493/1493 across all config schema tests pass.
- pnpm run lint, tsgo:core, tsgo:extensions, tsgo:test, build all green.

Risk / Compatibility
--------------------

- Default idle threshold (15 min) is conservative - comfortably outlives
  typical bursts of memory_search calls within a session.
- Worst-case dwell is idleMs + scanMs + max in-flight op duration
  (was unbounded; idleMs + scanMs if all ops are short).
- Short-lived CLI invocations are unchanged: closeAllMemoryIndexManagers
  remains the exit path from openclaw#40389, and it clears inflightCount
  alongside the rest of the cache.
- Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type
  grows two additive fields (skippedBusy, skippedRevalidated). Existing
  callers reading evicted / remaining continue to work unchanged via
  TypeScript structural typing.
- No new dependencies, no schema migration, no protocol change.

Out of scope
------------

- Does NOT change sync.watch default value (openclaw#71335 tracks that
  separately).
- Does NOT refactor the watcher to be singleton-per-workspace (much
  larger architectural change).
- Does NOT introduce an opt-out for the in-flight protection - it's a
  safety net that should always be on. The only configurable knobs are
  the sweep cadence and idle threshold.

Refs openclaw#71335, openclaw#40389.
quengh added a commit to quengh/openclaw that referenced this pull request May 27, 2026
Cached MemoryIndexManager instances live in a module-level INDEX_CACHE
that grows for the entire gateway daemon lifetime. Each instance owns a
chokidar FSWatcher recursively watching the memory directory (under the
default sync.watch=true), so every cache entry holds an FSEventWrap
native handle plus one awaitWriteFinish file descriptor per touched
memory/*.md file.

Cached managers are torn down at CLI exit (openclaw#40389), but daemons never
reach that exit path until restart. A production daemon (OpenClaw
2026.4.15) measured +1.6 GB RSS over 11 h and 612 accumulated
FSEventWrap handles before OOM.

This change adds a periodic runtime sweep that closes idle managers
during normal daemon operation, so restart is no longer the only way
to release these resources.

Approach
--------

ManagedCache:
- Track lastAccessAt per cache key (refreshed on hit and on initial set).
- closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries,
  and isolates close() errors so one bad entry does not stop the rest.
  Reuses the per-manager close() introduced in openclaw#40389 - no new teardown
  code path.

In-flight protection (always-on safety net):
- inflightCount map keyed the same way as cache / lastAccessAt.
- acquireManagedCacheKey() increments the count, refreshes lastAccessAt,
  and returns an idempotent release() function.
- isManagedCacheKeyBusy() reports whether the count is > 0.
- closeIdleManagedCacheEntries consults isManagedCacheKeyBusy in both
  the survey loop and immediately before close(); busy entries are
  skipped, counted, and surfaced via onSkipBusy / skippedBusy.
- A revalidation gate (just before close) also re-checks lastAccessAt
  freshness and cache identity, catching entries that became active or
  were swapped during the scan-to-close gap. Reported as
  skippedRevalidated.
- Cache metadata cleanup in MemoryIndexManager.close() and
  closeMemoryIndexManagersForAgent is now scoped to the cache identity
  check that previously guarded only the INDEX_CACHE delete, so a
  replacement manager installed in the same key during the close()
  await window keeps its own lastAccessAt and inflightCount entries.

MemoryIndexManager:
- Protected withBusy(fn) helper wraps acquire/release in try/finally.
- Guards sync(), search(), and ensureProviderInitialized() so the busy
  ref is held across provider init as well as the manager work itself.
- close() and closeMemoryIndexManagersForAgent clear residual cache
  metadata only when this instance is still the cache occupant.
  release() is idempotent so a release that fires after close() is a
  no-op.

Gateway sidecar:
- scheduleMemoryIndexIdleEvict registers a gateway-lifetime sidecar
  that calls closeIdleMemorySearchManagers every idleEvictScanMs
  (default 5 min), evicting managers idle for idleEvictMs (default
  15 min). Setting either to 0 disables the sweep entirely.
- Resolved from agents.defaults.memorySearch.sync only. The cache is a
  process-wide singleton and the sweep runs once per cadence over the
  entire cache; there is no per-agent sweep timer, and no coherent
  per-agent answer when two agents share a workspace and disagree on
  the threshold. Per-agent values under agents.list[].memorySearch.sync
  are accepted by the (shared) schema but ignored at runtime; this is
  documented on the TS doc, zod comments, UI labels, FIELD_HELP, and
  in docs/reference/memory-config.md.

Configuration
-------------

Two new optional fields under agents.defaults.memorySearch.sync:
- idleEvictMs (default 900000): idle TTL before a manager is eligible
  for eviction. 0 disables.
- idleEvictScanMs (default 300000): sweep cadence. 0 disables.

Documented in:
- src/config/zod-schema.agent-runtime.ts (schema with scope contract)
- src/config/types.tools.ts (MemorySearchConfig TS doc)
- src/config/schema.labels.ts (UI labels, tagged "gateway-wide")
- src/config/schema.help.ts (FIELD_HELP user-facing copy)
- docs/reference/memory-config.md (reference docs with ParamField blocks)

Validation
----------

Live soak on the production daemon (OpenClaw 2026.4.15 + this patch
backported, 17 h+ continuous at PR submission, sampling every 10 min):

| Metric          | Without patch (11 h)    | With patch (17 h soak)        |
|-----------------|-------------------------|-------------------------------|
| FSEventWrap     | 612 native handles      | bounded, returns to ~3 baseline |
| memory_fd       | 488 (chokidar holds)    | 0 between sweeps              |
| RSS growth rate | ~1 GB/h                 | ~150 MB/h (~7x improvement)   |
| Outcome         | OOM within 4 to 6 h     | 3.16 GB stable, no OOM        |

Sidecar observed behavior:
- fd accumulates during memory_search activity (up to ~600 handles)
- next sweep evicts idle managers, fd returns to baseline within 10 min
- skippedBusy and skippedRevalidated stayed at 0 throughout the soak -
  the workload burst rate did not exercise the deferral paths, but the
  unit tests cover them deterministically

Tests
-----

- extensions/memory-core/src/memory/manager-cache.test.ts: 21 tests
  (was 11). New coverage: acquireManagedCacheKey, isManagedCacheKeyBusy,
  busy deferral, refcount, lastAccessAt refresh, inflightCount cleanup,
  revalidation gate for cache-hit / busy / identity-swap / slot-
  disappearance gaps.
- extensions/memory-core/src/memory/manager.idle-gc.test.ts: 12 tests
  (was 5). New coverage: in-flight protection at manager-level entry
  points (search, sync, ensureProviderInitialized), plus cache-occupant identity-check
  coverage for the close() metadata-cleanup path.
- extensions/memory-core/: 720/720 across 57 files.
- src/gateway/server-startup-post-attach.test.ts: 41 tests pass.
- src/agents/memory-search.test.ts: 27 tests pass.
- src/config/: 1493/1493 across all config schema tests pass.
- pnpm run lint, tsgo:core, tsgo:extensions, tsgo:test, build all green.

Risk / Compatibility
--------------------

- Default idle threshold (15 min) is conservative - comfortably outlives
  typical bursts of memory_search calls within a session.
- Worst-case dwell is idleMs + scanMs + max in-flight op duration
  (was unbounded; idleMs + scanMs if all ops are short).
- Short-lived CLI invocations are unchanged: closeAllMemoryIndexManagers
  remains the exit path from openclaw#40389, and it clears inflightCount
  alongside the rest of the cache.
- Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type
  grows two additive fields (skippedBusy, skippedRevalidated). Existing
  callers reading evicted / remaining continue to work unchanged via
  TypeScript structural typing.
- No new dependencies, no schema migration, no protocol change.

Out of scope
------------

- Does NOT change sync.watch default value (openclaw#71335 tracks that
  separately).
- Does NOT refactor the watcher to be singleton-per-workspace (much
  larger architectural change).
- Does NOT introduce an opt-out for the in-flight protection - it's a
  safety net that should always be on. The only configurable knobs are
  the sweep cadence and idle threshold.

Refs openclaw#71335, openclaw#40389.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli CLI command changes size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: One-shot CLI can hang after completion due to cached memory manager FSWatcher handles

2 participants