Fix one-shot exit hangs by tearing down cached memory managers#40389
Conversation
Greptile SummaryThis PR fixes one-shot CLI exit hangs caused by lingering Key changes:
Confidence Score: 4/5
Last reviewed commit: cec15df |
0bc26dd to
524f80e
Compare
|
All feedback addressed, CI green, ready for merge. |
141558b to
f07f95a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f07f95a2a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -13,6 +13,15 @@ import { applyCliProfileEnv, parseCliProfileArgs } from "./profile.js"; | |||
| import { tryRouteCli } from "./route.js"; | |||
| import { normalizeWindowsArgv } from "./windows-argv.js"; | |||
|
|
|||
| async function closeCliMemoryManagers(): Promise<void> { | |||
| try { | |||
| const { closeAllMemorySearchManagers } = await import("../memory/index.js"); | |||
There was a problem hiding this comment.
Avoid eager memory-module import during CLI teardown
runCli now always executes closeCliMemoryManagers() in finally, and that helper imports ../memory/index.js on every command path. Because src/memory/index.ts statically re-exports MemoryIndexManager from manager.ts, even routed/fast-path commands that never used memory still load the full memory stack at shutdown, which defeats route-first startup goals and adds avoidable module-load overhead (or failures in stripped runtimes). Use a teardown import that stays on the lazy runtime boundary instead of the memory barrel export.
Useful? React with 👍 / 👎.
8bdc1f2 to
6b02314
Compare
6b02314 to
0e600e8
Compare
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn (cherry picked from commit c0cba7f)
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn (cherry picked from commit c0cba7f)
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
…law#40389) Merged via squash. Prepared head SHA: 0e600e8 Co-authored-by: Julbarth <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn
Cached MemoryIndexManager instances live in a module-level INDEX_CACHE that grows for the entire gateway daemon lifetime. Each instance owns a chokidar FSWatcher recursively watching the memory directory (under the default sync.watch=true), so every cache entry holds an FSEventWrap native handle plus one awaitWriteFinish file descriptor per touched memory/*.md file. Cached managers are torn down at CLI exit (openclaw#40389), but daemons never reach that exit path until restart. A production daemon (OpenClaw 2026.4.15) measured +1.6 GB RSS over 11 h and 612 accumulated FSEventWrap handles before crashing. This change adds a periodic runtime sweep that closes idle managers during normal daemon operation, so restart is no longer the only way to release these resources. Approach -------- ManagedCache: - Track lastAccessAt per cache key (refreshed on hit and on initial set). - closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries, and isolates close() errors so one bad entry does not stop the rest. Reuses the per-manager close() introduced in openclaw#40389 — no new teardown code path. In-flight protection (always-on safety net): - inflightCount map keyed the same way as cache / lastAccessAt. - acquireManagedCacheKey() increments the count, refreshes lastAccessAt, and returns an idempotent release() function. - isManagedCacheKeyBusy() reports whether the count is > 0. - closeIdleManagedCacheEntries consults isManagedCacheKeyBusy in both the survey loop and immediately before close(); busy entries are skipped, counted, and surfaced via onSkipBusy / skippedBusy. - A revalidation gate (just before close) also re-checks lastAccessAt freshness and cache identity, catching entries that became active or were swapped during the scan-to-close gap. Reported as skippedRevalidated. MemoryIndexManager: - Protected withBusy(fn) helper wraps acquire/release in try/finally. - Guards sync(), search(), and ensureProviderInitialized() so the busy ref is held across provider init as well as the manager work itself. - close() and closeMemoryIndexManagersForAgent clear any residual inflightCount entry alongside the cache slot. release() is idempotent so a release that fires after close() is a no-op. Gateway sidecar: - scheduleMemoryIndexIdleEvict registers a gateway-lifetime sidecar that calls closeIdleMemorySearchManagers every idleEvictScanMs (default 5 min), evicting managers idle for idleEvictMs (default 15 min). Setting either to 0 disables the sweep entirely. Configuration ------------- Two new optional fields under agents.defaults.memorySearch.sync: - idleEvictMs (default 900000): idle TTL before a manager is eligible for eviction. 0 disables. - idleEvictScanMs (default 300000): sweep cadence. 0 disables. Documented in the config schema (zod), the MemorySearchConfig type, and the UI field labels map. Validation ---------- Live soak on the production daemon (OpenClaw 2026.4.15 + this patch backported, 13 h+ continuous, sampling every 10 min): | Metric | Without patch (11 h) | With patch (13 h soak) | |-----------------|-------------------------|-------------------------------| | FSEventWrap | 612 native handles | bounded, returns to ~3 baseline | | memory_fd | 488 (chokidar holds) | 0 between sweeps | | RSS growth rate | ~1 GB/h | ~150 MB/h (~7x improvement) | | Outcome at 13 h | OOM within 4 to 6 h | 3.25 GB stable, no OOM | Sidecar observed behavior: - fd accumulates during memory_search activity (up to ~600 handles) - next sweep evicts idle managers, fd returns to baseline within 10 min - skippedBusy and skippedRevalidated stayed at 0 throughout the soak — the workload burst rate did not exercise the deferral paths, but the unit tests cover them deterministically Tests ----- - extensions/memory-core/src/memory/manager-cache.test.ts: 21 tests (was 11). New coverage: acquireManagedCacheKey, isManagedCacheKeyBusy, busy deferral, refcount, lastAccessAt refresh, inflightCount cleanup, revalidation gate for cache-hit / busy / identity-swap / slot- disappearance gaps. - extensions/memory-core/src/memory/manager.idle-gc.test.ts: 10 tests (was 5). New coverage: in-flight protection at manager-level entry points (search, sync, ensureProviderInitialized). - extensions/memory-core/: 718/718 across 57 files. - src/gateway/server-startup-post-attach.test.ts: 41 tests pass. - src/agents/memory-search.test.ts: 27 tests pass. - pnpm run lint, tsgo:core, tsgo:extensions, tsgo:test, build all green. Risk / Compatibility -------------------- - Default idle threshold (15 min) is conservative — comfortably outlives typical bursts of memory_search calls within a session. - Worst-case dwell is idleMs + scanMs + max in-flight op duration (was unbounded; idleMs + scanMs if all ops are short). - Short-lived CLI invocations are unchanged: closeAllMemoryIndexManagers remains the exit path from openclaw#40389, and it clears inflightCount alongside the rest of the cache. - Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type grows two additive fields (skippedBusy, skippedRevalidated). Existing callers reading evicted / remaining continue to work unchanged via TypeScript structural typing. - No new dependencies, no schema migration, no protocol change. Out of scope ------------ - Does NOT change sync.watch default value (openclaw#71335 tracks that separately). - Does NOT refactor the watcher to be singleton-per-workspace (much larger architectural change). - Does NOT introduce an opt-out for the in-flight protection — it's a safety net that should always be on. The only configurable knobs are the sweep cadence and idle threshold. Refs openclaw#71335, openclaw#40389.
Cached MemoryIndexManager instances live in a module-level INDEX_CACHE that grows for the entire gateway daemon lifetime. Each instance owns a chokidar FSWatcher recursively watching the memory directory (under the default sync.watch=true), so every cache entry holds an FSEventWrap native handle plus one awaitWriteFinish file descriptor per touched memory/*.md file. Cached managers are torn down at CLI exit (openclaw#40389), but daemons never reach that exit path until restart. A production daemon (OpenClaw 2026.4.15) measured +1.6 GB RSS over 11 h and 612 accumulated FSEventWrap handles before OOM. This change adds a periodic runtime sweep that closes idle managers during normal daemon operation, so restart is no longer the only way to release these resources. Approach -------- ManagedCache: - Track lastAccessAt per cache key (refreshed on hit and on initial set). - closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries, and isolates close() errors so one bad entry does not stop the rest. Reuses the per-manager close() introduced in openclaw#40389 - no new teardown code path. In-flight protection (always-on safety net): - inflightCount map keyed the same way as cache / lastAccessAt. - acquireManagedCacheKey() increments the count, refreshes lastAccessAt, and returns an idempotent release() function. - isManagedCacheKeyBusy() reports whether the count is > 0. - closeIdleManagedCacheEntries consults isManagedCacheKeyBusy in both the survey loop and immediately before close(); busy entries are skipped, counted, and surfaced via onSkipBusy / skippedBusy. - A revalidation gate (just before close) also re-checks lastAccessAt freshness and cache identity, catching entries that became active or were swapped during the scan-to-close gap. Reported as skippedRevalidated. - Cache metadata cleanup in MemoryIndexManager.close() and closeMemoryIndexManagersForAgent is now scoped to the cache identity check that previously guarded only the INDEX_CACHE delete, so a replacement manager installed in the same key during the close() await window keeps its own lastAccessAt and inflightCount entries. MemoryIndexManager: - Protected withBusy(fn) helper wraps acquire/release in try/finally. - Guards sync(), search(), and ensureProviderInitialized() so the busy ref is held across provider init as well as the manager work itself. - close() and closeMemoryIndexManagersForAgent clear residual cache metadata only when this instance is still the cache occupant. release() is idempotent so a release that fires after close() is a no-op. Gateway sidecar: - scheduleMemoryIndexIdleEvict registers a gateway-lifetime sidecar that calls closeIdleMemorySearchManagers every idleEvictScanMs (default 5 min), evicting managers idle for idleEvictMs (default 15 min). Setting either to 0 disables the sweep entirely. - Resolved from agents.defaults.memorySearch.sync only. The cache is a process-wide singleton and the sweep runs once per cadence over the entire cache; there is no per-agent sweep timer, and no coherent per-agent answer when two agents share a workspace and disagree on the threshold. Per-agent values under agents.list[].memorySearch.sync are accepted by the (shared) schema but ignored at runtime; this is documented on the TS doc, zod comments, UI labels, FIELD_HELP, and in docs/reference/memory-config.md. Configuration ------------- Two new optional fields under agents.defaults.memorySearch.sync: - idleEvictMs (default 900000): idle TTL before a manager is eligible for eviction. 0 disables. - idleEvictScanMs (default 300000): sweep cadence. 0 disables. Documented in: - src/config/zod-schema.agent-runtime.ts (schema with scope contract) - src/config/types.tools.ts (MemorySearchConfig TS doc) - src/config/schema.labels.ts (UI labels, tagged "gateway-wide") - src/config/schema.help.ts (FIELD_HELP user-facing copy) - docs/reference/memory-config.md (reference docs with ParamField blocks) Validation ---------- Live soak on the production daemon (OpenClaw 2026.4.15 + this patch backported, 17 h+ continuous at PR submission, sampling every 10 min): | Metric | Without patch (11 h) | With patch (17 h soak) | |-----------------|-------------------------|-------------------------------| | FSEventWrap | 612 native handles | bounded, returns to ~3 baseline | | memory_fd | 488 (chokidar holds) | 0 between sweeps | | RSS growth rate | ~1 GB/h | ~150 MB/h (~7x improvement) | | Outcome | OOM within 4 to 6 h | 3.16 GB stable, no OOM | Sidecar observed behavior: - fd accumulates during memory_search activity (up to ~600 handles) - next sweep evicts idle managers, fd returns to baseline within 10 min - skippedBusy and skippedRevalidated stayed at 0 throughout the soak - the workload burst rate did not exercise the deferral paths, but the unit tests cover them deterministically Tests ----- - extensions/memory-core/src/memory/manager-cache.test.ts: 21 tests (was 11). New coverage: acquireManagedCacheKey, isManagedCacheKeyBusy, busy deferral, refcount, lastAccessAt refresh, inflightCount cleanup, revalidation gate for cache-hit / busy / identity-swap / slot- disappearance gaps. - extensions/memory-core/src/memory/manager.idle-gc.test.ts: 12 tests (was 5). New coverage: in-flight protection at manager-level entry points (search, sync, ensureProviderInitialized), plus cache-occupant identity-check coverage for the close() metadata-cleanup path. - extensions/memory-core/: 720/720 across 57 files. - src/gateway/server-startup-post-attach.test.ts: 41 tests pass. - src/agents/memory-search.test.ts: 27 tests pass. - src/config/: 1493/1493 across all config schema tests pass. - pnpm run lint, tsgo:core, tsgo:extensions, tsgo:test, build all green. Risk / Compatibility -------------------- - Default idle threshold (15 min) is conservative - comfortably outlives typical bursts of memory_search calls within a session. - Worst-case dwell is idleMs + scanMs + max in-flight op duration (was unbounded; idleMs + scanMs if all ops are short). - Short-lived CLI invocations are unchanged: closeAllMemoryIndexManagers remains the exit path from openclaw#40389, and it clears inflightCount alongside the rest of the cache. - Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type grows two additive fields (skippedBusy, skippedRevalidated). Existing callers reading evicted / remaining continue to work unchanged via TypeScript structural typing. - No new dependencies, no schema migration, no protocol change. Out of scope ------------ - Does NOT change sync.watch default value (openclaw#71335 tracks that separately). - Does NOT refactor the watcher to be singleton-per-workspace (much larger architectural change). - Does NOT introduce an opt-out for the in-flight protection - it's a safety net that should always be on. The only configurable knobs are the sweep cadence and idle threshold. Refs openclaw#71335, openclaw#40389.
Cached MemoryIndexManager instances live in a module-level INDEX_CACHE that grows for the entire gateway daemon lifetime. Each instance owns a chokidar FSWatcher recursively watching the memory directory (under the default sync.watch=true), so every cache entry holds an FSEventWrap native handle plus one awaitWriteFinish file descriptor per touched memory/*.md file. Cached managers are torn down at CLI exit (openclaw#40389), but daemons never reach that exit path until restart. A production daemon (OpenClaw 2026.4.15) measured +1.6 GB RSS over 11 h and 612 accumulated FSEventWrap handles before OOM. This change adds a periodic runtime sweep that closes idle managers during normal daemon operation, so restart is no longer the only way to release these resources. Approach -------- ManagedCache: - Track lastAccessAt per cache key (refreshed on hit and on initial set). - closeIdleManagedCacheEntries walks lastAccessAt, closes stale entries, and isolates close() errors so one bad entry does not stop the rest. Reuses the per-manager close() introduced in openclaw#40389 - no new teardown code path. In-flight protection (always-on safety net): - inflightCount map keyed the same way as cache / lastAccessAt. - acquireManagedCacheKey() increments the count, refreshes lastAccessAt, and returns an idempotent release() function. - isManagedCacheKeyBusy() reports whether the count is > 0. - closeIdleManagedCacheEntries consults isManagedCacheKeyBusy in both the survey loop and immediately before close(); busy entries are skipped, counted, and surfaced via onSkipBusy / skippedBusy. - A revalidation gate (just before close) also re-checks lastAccessAt freshness and cache identity, catching entries that became active or were swapped during the scan-to-close gap. Reported as skippedRevalidated. - Cache metadata cleanup in MemoryIndexManager.close() and closeMemoryIndexManagersForAgent is now scoped to the cache identity check that previously guarded only the INDEX_CACHE delete, so a replacement manager installed in the same key during the close() await window keeps its own lastAccessAt and inflightCount entries. MemoryIndexManager: - Protected withBusy(fn) helper wraps acquire/release in try/finally. - Guards sync(), search(), and ensureProviderInitialized() so the busy ref is held across provider init as well as the manager work itself. - close() and closeMemoryIndexManagersForAgent clear residual cache metadata only when this instance is still the cache occupant. release() is idempotent so a release that fires after close() is a no-op. Gateway sidecar: - scheduleMemoryIndexIdleEvict registers a gateway-lifetime sidecar that calls closeIdleMemorySearchManagers every idleEvictScanMs (default 5 min), evicting managers idle for idleEvictMs (default 15 min). Setting either to 0 disables the sweep entirely. - Resolved from agents.defaults.memorySearch.sync only. The cache is a process-wide singleton and the sweep runs once per cadence over the entire cache; there is no per-agent sweep timer, and no coherent per-agent answer when two agents share a workspace and disagree on the threshold. Per-agent values under agents.list[].memorySearch.sync are accepted by the (shared) schema but ignored at runtime; this is documented on the TS doc, zod comments, UI labels, FIELD_HELP, and in docs/reference/memory-config.md. Configuration ------------- Two new optional fields under agents.defaults.memorySearch.sync: - idleEvictMs (default 900000): idle TTL before a manager is eligible for eviction. 0 disables. - idleEvictScanMs (default 300000): sweep cadence. 0 disables. Documented in: - src/config/zod-schema.agent-runtime.ts (schema with scope contract) - src/config/types.tools.ts (MemorySearchConfig TS doc) - src/config/schema.labels.ts (UI labels, tagged "gateway-wide") - src/config/schema.help.ts (FIELD_HELP user-facing copy) - docs/reference/memory-config.md (reference docs with ParamField blocks) Validation ---------- Live soak on the production daemon (OpenClaw 2026.4.15 + this patch backported, 17 h+ continuous at PR submission, sampling every 10 min): | Metric | Without patch (11 h) | With patch (17 h soak) | |-----------------|-------------------------|-------------------------------| | FSEventWrap | 612 native handles | bounded, returns to ~3 baseline | | memory_fd | 488 (chokidar holds) | 0 between sweeps | | RSS growth rate | ~1 GB/h | ~150 MB/h (~7x improvement) | | Outcome | OOM within 4 to 6 h | 3.16 GB stable, no OOM | Sidecar observed behavior: - fd accumulates during memory_search activity (up to ~600 handles) - next sweep evicts idle managers, fd returns to baseline within 10 min - skippedBusy and skippedRevalidated stayed at 0 throughout the soak - the workload burst rate did not exercise the deferral paths, but the unit tests cover them deterministically Tests ----- - extensions/memory-core/src/memory/manager-cache.test.ts: 21 tests (was 11). New coverage: acquireManagedCacheKey, isManagedCacheKeyBusy, busy deferral, refcount, lastAccessAt refresh, inflightCount cleanup, revalidation gate for cache-hit / busy / identity-swap / slot- disappearance gaps. - extensions/memory-core/src/memory/manager.idle-gc.test.ts: 12 tests (was 5). New coverage: in-flight protection at manager-level entry points (search, sync, ensureProviderInitialized), plus cache-occupant identity-check coverage for the close() metadata-cleanup path. - extensions/memory-core/: 720/720 across 57 files. - src/gateway/server-startup-post-attach.test.ts: 41 tests pass. - src/agents/memory-search.test.ts: 27 tests pass. - src/config/: 1493/1493 across all config schema tests pass. - pnpm run lint, tsgo:core, tsgo:extensions, tsgo:test, build all green. Risk / Compatibility -------------------- - Default idle threshold (15 min) is conservative - comfortably outlives typical bursts of memory_search calls within a session. - Worst-case dwell is idleMs + scanMs + max in-flight op duration (was unbounded; idleMs + scanMs if all ops are short). - Short-lived CLI invocations are unchanged: closeAllMemoryIndexManagers remains the exit path from openclaw#40389, and it clears inflightCount alongside the rest of the cache. - Public MemoryPluginRuntime.closeIdleMemorySearchManagers return type grows two additive fields (skippedBusy, skippedRevalidated). Existing callers reading evicted / remaining continue to work unchanged via TypeScript structural typing. - No new dependencies, no schema migration, no protocol change. Out of scope ------------ - Does NOT change sync.watch default value (openclaw#71335 tracks that separately). - Does NOT refactor the watcher to be singleton-per-workspace (much larger architectural change). - Does NOT introduce an opt-out for the in-flight protection - it's a safety net that should always be on. The only configurable knobs are the sweep cadence and idle threshold. Refs openclaw#71335, openclaw#40389.
Summary
Describe the problem and fix in 2–5 bullets:
finally; also drained in-flight index manager creations during teardown.Change Type (select all)
Scope (select all touched areas)
Linked Issue/PR
User-visible / Behavior Changes
One-shot CLI commands now exit cleanly after completion in scenarios where cached memory-manager watcher handles previously kept the process alive.
Security Impact (required)
No)No)No)No)No)Yes, explain risk + mitigation:Repro + Verification
Environment
Steps
Expected
Actual
Evidence
Attach at least one:
Human Verification (required)
What you personally verified (not just CI), and how:
pnpm exec vitest run src/cli/run-main.exit.test.tspnpm exec vitest run src/memory/search-manager.test.tspnpm exec vitest run src/memory/manager.get-concurrency.test.tsReview Conversations
If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.
Compatibility / Migration
Yes)No)No)Failure Recovery (if this breaks)
src/cli/run-main.ts,src/cli/run-main.exit.test.ts,src/memory/manager.ts,src/memory/manager-runtime.ts,src/memory/search-manager.ts,src/memory/search-manager.test.ts,src/memory/manager.get-concurrency.test.ts,src/memory/index.tsRisks and Mitigations
List only real risks for this PR. Add/remove entries as needed. If none, write
None.