fix(memory): stop watcher write-polling fd pressure by frankekn · Pull Request #81802 · openclaw/openclaw

frankekn · 2026-05-14T12:05:01Z

Summary

remove chokidar awaitWriteFinish polling from built-in memory and QMD watchers
record only changed watcher paths, then settle those paths by size/mtime before watch-triggered sync
keep the existing debounced watch sync so bursts still batch before re-indexing
update watcher tests and changelog for the FD-pressure fix

Why

Issue #78224 gives the concrete reproduction for the same failure class behind #77327: large Markdown extraPaths trees with memorySearch.sync.watch: true build up thousands of regular-file descriptors, then gateway child-process spawns fail with EBADF.

The linked PR #78029 only changes bounded QMD read-stream cleanup. I could not reproduce a leak from that narrow path on Node v25.9.0, so this PR targets the watcher polling surface called out by the large-tree report instead.

The follow-up keeps chokidar from polling the whole tree while still avoiding obvious partial-write indexing: the watcher queue compares only paths that emitted events, and reschedules the watch sync if a changed file is still moving.

Fixes #77327.
Fixes #78224.
Alternative to #78029.

Verification

/opt/homebrew/bin/node v25.9.0 standalone bounded QMD read loop, 1,000 iterations: target file FDs stayed at 0
/opt/homebrew/bin/node v25.9.0 standalone chokidar watch over 500 wiki Markdown files + 500 SKILL.md files: target REG FDs stayed at 0 before/after close
OPENCLAW_TEST_HEAVY_CHECK_LOCK_HELD=1 OPENCLAW_TEST_WORKERS=1 pnpm test extensions/memory-core/src/memory/manager.watcher-config.test.ts extensions/memory-core/src/memory/qmd-manager.test.ts
pnpm exec oxfmt --check CHANGELOG.md extensions/memory-core/src/memory/watch-settle.ts extensions/memory-core/src/memory/manager-sync-ops.ts extensions/memory-core/src/memory/qmd-manager.ts extensions/memory-core/src/memory/manager.watcher-config.test.ts extensions/memory-core/src/memory/qmd-manager.test.ts
git diff --check
OPENCLAW_TEST_HEAVY_CHECK_LOCK_HELD=1 OPENCLAW_TEST_WORKERS=1 pnpm check:changed

frankekn · 2026-05-14T12:05:10Z

@codex review

clawsweeper · 2026-05-14T12:08:34Z

Codex review: needs maintainer review before merge.

Summary
The PR removes chokidar awaitWriteFinish from memory-core built-in and QMD watchers, adds per-event file-stat settling before debounced sync, updates watcher tests, and adds a changelog entry.

Reproducibility: yes. The large memorySearch.sync.watch plus Markdown extraPaths reproduction in #78224 maps directly to current main's chokidar watcher using awaitWriteFinish, though I did not run the multi-hour FD repro in this read-only review.

Real behavior proof
Not applicable: Not applicable for a maintainer/MEMBER PR; the PR body still includes copied after-fix FD-count checks from real Node watcher runs.

Next step before merge
Protected maintainer-labeled member PR with no discrete automated repair finding; the next action is maintainer review, CI/Testbox validation, and landing decision.

Security
Cleared: The diff touches memory watcher logic, tests, and changelog only; it does not change dependencies, CI, permissions, secrets handling, or install/build execution paths.

Review details

Best possible solution:

Land the narrow watcher-polling removal after maintainer CI/Testbox proof confirms large watched Markdown trees no longer grow REG file descriptors and delayed sync still indexes settled content.

Do we have a high-confidence way to reproduce the issue?

Yes. The large memorySearch.sync.watch plus Markdown extraPaths reproduction in #78224 maps directly to current main's chokidar watcher using awaitWriteFinish, though I did not run the multi-hour FD repro in this read-only review.

Is this the best way to solve the issue?

Yes. Removing chokidar's whole-tree write polling while settling only changed event paths is the narrowest maintainable fix for the identified FD pressure and addresses the earlier chunked-write concern without adding a new config surface.

Acceptance criteria:

OPENCLAW_TEST_HEAVY_CHECK_LOCK_HELD=1 OPENCLAW_TEST_WORKERS=1 pnpm test extensions/memory-core/src/memory/manager.watcher-config.test.ts extensions/memory-core/src/memory/qmd-manager.test.ts
pnpm exec oxfmt --check CHANGELOG.md extensions/memory-core/src/memory/watch-settle.ts extensions/memory-core/src/memory/manager-sync-ops.ts extensions/memory-core/src/memory/qmd-manager.ts extensions/memory-core/src/memory/manager.watcher-config.test.ts extensions/memory-core/src/memory/qmd-manager.test.ts
git diff --check
OPENCLAW_TEST_HEAVY_CHECK_LOCK_HELD=1 OPENCLAW_TEST_WORKERS=1 pnpm check:changed

What I checked:

Live protected PR state: The live pull API shows this PR is open at head ae4e97c, author association is MEMBER, and labels include maintainer plus extensions: memory-core, so cleanup should not auto-close it. (ae4e97cc1340)
Current main built-in watcher behavior: Current main still watches memory roots and extraPaths with chokidar awaitWriteFinish using watchDebounceMs and a 100ms poll interval. (extensions/memory-core/src/memory/manager-sync-ops.ts:449, 83d7ab0d362f)
Current main QMD watcher behavior: Current main also enables chokidar awaitWriteFinish in the QMD watcher before scheduling debounced watch sync. (extensions/memory-core/src/memory/qmd-manager.ts:1560, 83d7ab0d362f)
PR built-in watcher change: The PR head removes awaitWriteFinish from the built-in memory watcher and records changed event paths before scheduling the existing watch sync. (extensions/memory-core/src/memory/manager-sync-ops.ts:456, ae4e97cc1340)
PR QMD watcher change: The PR head removes awaitWriteFinish from QMD and gates watch-triggered sync on the same settle helper before running sync({ reason: "watch" }). (extensions/memory-core/src/memory/qmd-manager.ts:1566, ae4e97cc1340)
PR settle helper: The added helper snapshots only changed event paths, rechecks missing-baseline paths after 100ms, and asks callers to reschedule when size/mtime is still moving. (extensions/memory-core/src/memory/watch-settle.ts:69, ae4e97cc1340)

Likely related people:

frankekn: Author of this PR and prior memory-core history includes b186d98, which recorded @frankekn as author/reviewer for memory-core registration work; frankekn is also assigned on the related QMD FD PR. (role: current fix author and prior memory-core contributor; confidence: high; commits: 3bb6d0ea1330, ae4e97cc1340, b186d9847c03; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/qmd-manager.ts, extensions/memory-core/src/memory/watch-settle.ts)
steipete: GitHub file history shows recent broad memory/QMD maintenance and refactors touching the same memory-core files, including 694ca50 and ededff4. (role: recent area contributor; confidence: medium; commits: 694ca50e9775, ededff4bc3fd, dffa88f39615; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/qmd-manager.ts)
jasonxargs-boop: Commit 2204753 changed memory-core chokidar watching to watch memory directories directly and ignore non-Markdown churn, which is adjacent to this PR's watcher pressure fix. (role: watcher path contributor; confidence: medium; commits: 2204753b6214; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager.watcher-config.test.ts)
vincentkoc: The shallow checkout blame attributes the current watcher option lines to c70adb8, and commit 2204753 lists Vincent Koc as co-author on memory-core watcher changes; precision is limited by the grafted checkout. (role: adjacent watcher contributor; confidence: low; commits: c70adb8528, 2204753b6214; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/qmd-manager.ts)

Remaining risk / open question:

The multi-hour large-tree FD-growth reproduction and CI/Testbox lanes were not run in this read-only review.
Two earlier bot review threads were opened on the first commit; the second commit appears to address the chunked-write concern, but maintainers may still need to resolve those conversations.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 83d7ab0d362f.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3bb6d0ea13

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

frankekn · 2026-05-14T15:13:22Z

@codex review

chatgpt-codex-connector · 2026-05-14T15:20:10Z

Codex Review: Didn't find any major issues. Can't wait for the next one!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

hxy91819

LGTM.

This PR resolves issues in scenarios with a large number of MD documents: the original implementation, designed to detect when file writing was complete, continuously held FDs for every file in the watch tree to perform size polling, which led to FD exhaustion. The fix removes this mechanism, switching to performing temporary stat checks (settle) only on files that have triggered events after a debounce. This reduces FD consumption from O(N) to 0. The only trade-off is that index refreshing might be delayed by one cycle in extreme cases, but the benefits are very significant.

frankekn · 2026-05-15T10:12:36Z

Merged via squash.

Prepared head SHA: 623874619b8b32815095a25a901e3ecc2c8b301f
Merge commit: b04e42812eca1752ef5e8213346ae5cda7779d9b

Thanks @frankekn!

@frankekn

Merged via squash. Prepared head SHA: 6238746 Co-authored-by: frankekn <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn

…ories Closes openclaw#86613. A single authorized POST /tools/invoke memory_search call against a workspace with multi-thousand .md files under <workspace>/memory/ opens one read-only FD per file (~12,400 in our captures), never released until process exit. Root cause: chokidar v5 calls fs.watch(path) per filesystem node (handler.js:126), no recursive option. On macOS each per-file fs.watch opens a kqueue VNODE FD that lsof reports as a regular-file REG FD. So one watcher per .md file = one FD per file = 12k+ FDs. Switch directory watch paths to Node >= 22 native recursive fs.watch(dir, { recursive: true }) — one watcher per directory via FSEvents on macOS / inotify recursive on Linux / ReadDirectoryChangesW on Windows. Individual file paths (MEMORY.md, file-typed extraPaths) continue through chokidar. Same dirty-event semantics, different backend, small constant FD profile regardless of tree size. Regression introduced by openclaw#64711 (jasonxargs-boop, 2026-04-13) which changed the chokidar watch target from glob memory/**/*.md to bare directory memory/, making the recursive walk implicit. PR openclaw#81802 (frankekn, 2026-05-15) removed awaitWriteFinish and added watch-settle on the same path; that reduced write-polling pressure but did not address the per-node fs.watch allocation (verification used 1,000 files, well under the storm-producing scale). Lab verification on clean upstream-main + synthetic 12,391-file workspace: 0 → 1 REG FD (MEMORY.md only; 12,391 file events covered by one recursive kqueue/FSEvents watcher), flat plateau, second memory_search returns 200 OK with zero additional FDs. Complementary to openclaw#86345 which attacks the same failure class by bounding INDEX_CACHE lifetime; this PR is non-overlapping in manager-sync-ops.ts and touches a disjoint region of manager.ts. Changes: - manager-sync-ops.ts: split watchPaths by dir/file; native recursive fs.watch for dirs with shouldIgnoreMemoryWatchPath filter and per-event lstat; chokidar retained for files. New TEST_MEMORY_NATIVE_WATCH_FACTORY_KEY for test injection. Native creation failure falls back into chokidar set. Null filename (Node platform caveat) triggers broad markDirty. Runtime error closes + removes + marks dirty. Idempotence guard expanded. Symlink policy preserved (extraPaths-only skip). - manager.ts: close() tears down nativeMemoryWatchers in addition to chokidar watcher. - manager.watcher-config.test.ts: nativeWatchMock paired with new factory key; assertions split between chokidar (files) and native (dirs); new cases for rename/change dispatch, null filename, native creation failure fallback, runtime error close, ensureWatcher re-entrancy guard. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

@frankekn

Merged via squash. Prepared head SHA: 6238746 Co-authored-by: frankekn <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn

@frankekn

Merged via squash. Prepared head SHA: 6238746 Co-authored-by: frankekn <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn

@frankekn

Merged via squash. Prepared head SHA: 6238746 Co-authored-by: frankekn <[email protected]> Co-authored-by: frankekn <[email protected]> Reviewed-by: @frankekn

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: XS maintainer Maintainer-authored PR labels May 14, 2026

chatgpt-codex-connector Bot reviewed May 14, 2026

View reviewed changes

Comment thread extensions/memory-core/src/memory/manager-sync-ops.ts

Comment thread extensions/memory-core/src/memory/qmd-manager.ts

openclaw-barnacle Bot added size: M and removed size: XS labels May 14, 2026

frankekn self-assigned this May 14, 2026

hxy91819 approved these changes May 15, 2026

View reviewed changes

frankekn force-pushed the fix/memory-watch-fd-pressure branch from ae4e97c to f1ceab5 Compare May 15, 2026 10:05

frankekn added 4 commits May 15, 2026 18:05

fix(memory): stop watcher write-polling fd pressure

3779b11

fix(memory): settle changed watch files

38f4fef

docs: move memory watcher changelog entry

ac4fd63

docs: credit memory watcher contributor

6238746

frankekn force-pushed the fix/memory-watch-fd-pressure branch from f1ceab5 to 6238746 Compare May 15, 2026 10:11

frankekn merged commit b04e428 into main May 15, 2026
99 checks passed

frankekn deleted the fix/memory-watch-fd-pressure branch May 15, 2026 10:12

github-actions Bot mentioned this pull request May 15, 2026

📡 Upstream Digest — 2026-05-15 10:13 UTC curtismercier/openclaw-mods#860

Open

zknicker mentioned this pull request May 15, 2026

fix(codex): preserve Discord sender metadata in Gateway IPC chat.history #82184

Merged

This was referenced May 15, 2026

Feature: sync.watch should default to false in gateway mode #71335

Open

Destroy bounded QMD read streams #78029

Closed

RomneyDa mentioned this pull request May 23, 2026

[Bug]: Transient Telegram pairing-store read failures can trigger bogus pairing prompts #85577

Closed

This was referenced May 24, 2026

docs(memory): align descriptors and docs with recursive memory/**/*.md #85899

Closed

fix(memory): complete recursive memory search follow-up #72748

Closed

lukeboyett mentioned this pull request May 25, 2026

[Bug]: Every memory_search call leaks ~N FDs (one per .md file in workspace memory tree) on macOS; long-lived gateways degrade toward FD exhaustion #86613

Closed

lukeboyett mentioned this pull request May 26, 2026

fix(memory-core): avoid per-file watcher FD fan-out for memory directories #86701

Merged

18 tasks

RomneyDa mentioned this pull request Jun 1, 2026

fix(memory): warn on gateway watcher FD risk #89185

Merged

clawsweeper Bot mentioned this pull request Jun 19, 2026

feat(watch): replace chokidar dep with in-repo chokidar-slim + async add API #89972

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory): stop watcher write-polling fd pressure#81802

fix(memory): stop watcher write-polling fd pressure#81802
frankekn merged 4 commits into
mainfrom
fix/memory-watch-fd-pressure

frankekn commented May 14, 2026 •

edited

Loading

Uh oh!

frankekn commented May 14, 2026

Uh oh!

clawsweeper Bot commented May 14, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

frankekn commented May 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 14, 2026

Uh oh!

hxy91819 left a comment

Uh oh!

Uh oh!

frankekn commented May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

frankekn commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Verification

Uh oh!

frankekn commented May 14, 2026

Uh oh!

clawsweeper Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

frankekn commented May 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented May 14, 2026

Uh oh!

hxy91819 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

frankekn commented May 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

frankekn commented May 14, 2026 •

edited

Loading

clawsweeper Bot commented May 14, 2026 •

edited

Loading