fix(memory): add watcher error handling and non-zero default intervalMinutes#42042
fix(memory): add watcher error handling and non-zero default intervalMinutes#42042nxmxbbd wants to merge 5 commits into
Conversation
Greptile SummaryThis PR fixes a silent-watcher-death scenario where the chokidar file watcher could stop firing events after long gateway uptime, leaving the memory index stale indefinitely. Three targeted changes address the root cause and add defense-in-depth:
The logic is sound: Minor note: the watcher error handler uses Confidence Score: 4/5
Last reviewed commit: 58527b1 |
0239bea to
a147f2e
Compare
a147f2e to
f607e30
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f607e306fb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -239,7 +239,7 @@ function mergeConfig( | |||
| overrides?.sync?.watchDebounceMs ?? | |||
| defaults?.sync?.watchDebounceMs ?? | |||
| DEFAULT_WATCH_DEBOUNCE_MS, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 15, | |||
There was a problem hiding this comment.
Preserve manual-only sync configs when defaulting intervalMinutes
Setting intervalMinutes to 15 here changes existing configs that already turned off the other automatic sync triggers (onSessionStart, onSearch, and watch) into background reindexing configs. MemoryManagerSyncOps.ensureIntervalSync() still schedules sync({ reason: "interval" }) regardless of those flags, so deployments that disabled auto-sync to avoid periodic CPU/embedding traffic will now start scanning every 15 minutes unless they discover and add a new intervalMinutes: 0 override.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Thanks for flagging. This is intentional --- the whole point of this PR is to provide a safety net when the chokidar watcher silently dies.
A few notes:
- The scenario (all three sync triggers disabled + no explicit
intervalMinutes) is an extreme edge case. Users who go that far in disabling sync are almost certainly also settingintervalMinutes: 0explicitly. - Even when the interval fires,
syncMemoryFilescompares file hashes and skips unchanged files (line 740), so there is no re-embedding cost --- just a lightweight file listing + hash check. - Users who truly want zero background sync can still set
intervalMinutes: 0to opt out.
Adding conditional logic (e.g. only default to 15 when watch: true) would undermine the safety net --- if watch is disabled precisely because chokidar is broken, those users lose the fallback too.
f607e30 to
1fc0af6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1fc0af6c73
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -239,7 +239,7 @@ function mergeConfig( | |||
| overrides?.sync?.watchDebounceMs ?? | |||
| defaults?.sync?.watchDebounceMs ?? | |||
| DEFAULT_WATCH_DEBOUNCE_MS, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 15, | |||
There was a problem hiding this comment.
Avoid default interval rescans for multimodal memory files
When intervalMinutes is omitted, this now enables a 15-minute background sync for every memory index. In that sync path, ensureIntervalSync() marks the manager dirty and syncMemoryFiles() rebuilds buildFileEntry() for every memory asset before it can decide the hash is unchanged (src/memory/manager-sync-ops.ts:715-749). For multimodal memory, buildFileEntry() rereads the entire file to compute dataHash (src/memory/internal.ts:205-243), so a workspace with several 5-10 MB images/audio notes will now incur recurring full-blob IO and RSS spikes every 15 minutes even when nothing changed. That is a meaningful performance regression for users who only took the default.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Thanks for flagging. The IO pattern you describe is accurate, but it is pre-existing behavior --- every sync trigger (watch, onSessionStart, onSearch) goes through the same syncMemoryFiles → buildFileEntry → readFile path. This PR does not change how sync works; it only adds an interval trigger as a fallback for when the watcher silently dies.
A stat-based fast path in buildFileEntry (skip readFile when mtime+size are unchanged) would be a worthwhile optimization, but that applies to all sync triggers equally and is out of scope for this fix.
Also worth noting: multimodal memory is opt-in and off by default, so the default-config user this comment is concerned about would not have large binary files in their memory directory in the first place.
1fc0af6 to
0608836
Compare
d0e8837 to
0eb69f9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0eb69f9b7f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -239,7 +239,7 @@ function mergeConfig( | |||
| overrides?.sync?.watchDebounceMs ?? | |||
| defaults?.sync?.watchDebounceMs ?? | |||
| DEFAULT_WATCH_DEBOUNCE_MS, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 15, | |||
There was a problem hiding this comment.
Avoid enabling 15-minute rescans on permanently cached managers
This default is broader than a one-shot safety net because builtin managers are cached globally in src/memory/manager.ts:164-198 and most long-lived callsites never close them (for example src/agents/tools/memory-tool.ts:96-127 and src/agents/pi-embedded-runner/compact.ts:323-334). In a multi-agent gateway—or after any config change that produces a new cache key—each touched manager will now keep its own 15-minute interval sync loop forever, so background workspace scans accumulate across dormant managers instead of only the active one.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Thanks for the detailed analysis. The manager cache accumulation you describe is technically accurate — cached managers aren't evicted and each keeps its own interval timer.
However, this is a pre-existing property of the manager cache, not something introduced by this PR. Any intervalMinutes > 0 value (whether user-configured or defaulted) behaves identically. The fix here is adding a cache eviction / idle-cleanup policy to the manager cache itself, which is an independent concern and shouldn't block a safety-net default for silent watcher death.
In practice: default users have 1 manager (single agent, single workspace, same settings = same cache key). Even in multi-agent gateways, the number of distinct cache keys is small, and the interval sync is lightweight (stat + hash compare, unchanged files skipped).
0f58389 to
d71d266
Compare
CI noteCurrent CI failures ( |
e912434 to
1595a1b
Compare
1595a1b to
e7d30e1
Compare
e7d30e1 to
98361af
Compare
CI note (rebase 2026-04-01)Rebased onto
All reproduce on upstream's own latest CI run on |
98361af to
a1cbf70
Compare
CI note (rebase 2026-04-06)Rebased onto
Our changed files ( |
|
@obviyus Related to the above --- adds error recovery for the memory file watcher (chokidar silently dies, memory search stops working). All bot feedback addressed. |
91a64cc to
58f2c1c
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58f2c1cc6d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -233,7 +233,7 @@ function mergeConfig( | |||
| overrides?.sync?.watchDebounceMs ?? | |||
| defaults?.sync?.watchDebounceMs ?? | |||
| DEFAULT_WATCH_DEBOUNCE_MS, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 15, | |||
There was a problem hiding this comment.
Keep sync resolver default aligned with interval fallback
Changing mergeConfig() to default intervalMinutes to 15 here, while resolveSyncConfig() (used by resolveMemorySearchSyncConfig) still defaults to 0, introduces two conflicting defaults for the same setting. That divergence means callers using the sync-only resolver can still behave as if interval fallback is disabled even when the main memory config resolver enables it, which undermines the safety-net behavior this commit is introducing. Please centralize or mirror the default so both resolvers produce the same value when intervalMinutes is omitted.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 29a4a1ab — resolveSyncConfig now defaults intervalMinutes to 15, matching mergeConfig. Good catch.
58f2c1c to
47c486f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47c486f260
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -233,7 +233,7 @@ function mergeConfig( | |||
| overrides?.sync?.watchDebounceMs ?? | |||
| defaults?.sync?.watchDebounceMs ?? | |||
| DEFAULT_WATCH_DEBOUNCE_MS, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 15, | |||
There was a problem hiding this comment.
Keep sync defaults consistent across memory config resolvers
This change updates mergeConfig() to default sync.intervalMinutes to 15, but resolveSyncConfig() (used by resolveMemorySearchSyncConfig) still defaults it to 0, so two public resolvers now disagree for the same omitted config. Callers that depend on the sync-only resolver (for example extensions/memory-core/src/memory/qmd-manager.ts) will continue behaving as if interval fallback is disabled, which creates inconsistent runtime behavior and makes the new default safety net apply only on one code path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same fix as above — aligned in 29a4a1ab.
cb80de9 to
fb66698
Compare
|
Fixed in 9a4f6edbd6 - updated extractMemoryErrorReason to use formatErrorMessage() for consistent error handling pattern with isMemoryReadonlyDbError function. |
145351b to
4341d80
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4341d800c7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| @@ -358,7 +358,7 @@ function resolveSyncConfig( | |||
| overrides?.sync?.watchDebounceMs ?? | |||
| defaults?.sync?.watchDebounceMs ?? | |||
| DEFAULT_WATCH_DEBOUNCE_MS, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 0, | |||
| intervalMinutes: overrides?.sync?.intervalMinutes ?? defaults?.sync?.intervalMinutes ?? 15, | |||
There was a problem hiding this comment.
Skip interval fallback when memory source is disabled
Changing the omitted sync.intervalMinutes default to 15 enables periodic sync for configurations that only index sessions (sources: ["sessions"]) even though there is no file watcher to recover. ensureIntervalSync() still schedules sync({ reason: "interval" }) regardless of sources, so these setups now run background sync cycles (provider/vector preflight + metadata checks) every 15 minutes when they previously stayed idle unless session triggers fired. This is an unintended workload increase for session-only deployments and can produce periodic noise despite no memory-file source being enabled.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0032db90ab - added check to skip interval timer setup when memory files are not in configured sources. Session-only deployments now avoid the unnecessary background sync cycles.
Adds a test that simulates a dead watcher scenario: after dirty is cleared by a completed sync, the interval callback must set dirty=true before calling sync() so that syncMemoryFiles actually runs. Without this, interval sync is a no-op when the watcher has silently died. Uses fake timers to advance past the interval tick and a sync spy to confirm both the dirty flag and the sync invocation.
…rorReason Use formatErrorMessage() instead of instanceof Error check to align with the error handling pattern used elsewhere in the file (isMemoryReadonlyDbError)
0032db9 to
ccf479c
Compare
|
Superseded by #65755 — clean squash onto latest main with identical changes. |
Summary
Fixes #40088 — the chokidar file watcher can silently stop firing events after long gateway uptime, leaving the memory index stale with no fallback mechanism.
Changes
1. Default
intervalMinutesfrom 0 → 15File:
src/agents/memory-search.tsWhen the watcher fails silently,
intervalMinutes: 0(the current default) means there is no periodic fallback. Changing to 15 ensures all users have a safety net. Users who explicitly setintervalMinutes: 0still get no interval sync (opt-out respected).2. Mark dirty before interval sync
File:
src/memory/manager-sync-ops.ts(ensureIntervalSync)The interval timer calls
this.sync({ reason: "interval" })withoutforce. InsiderunSync,shouldSyncMemorydepends onthis.dirty, which is only set by the watcher'smarkDirtycallback. If the watcher has died silently,dirtystaysfalseand the interval sync skips memory files entirely — making it a no-op.Setting
this.dirty = truebefore each interval tick ensuressyncMemoryFilesactually runs, listing files and comparing hashes. Unchanged files are skipped (hash match at line 740), so the overhead is negligible (~50ms for a typical workspace).Why
dirty = trueinstead offorce: true? Theforceflag inrunSynctriggersneedsFullReindex(line 1000), which does a full atomic reindex — temp DB swap, re-embed everything. That is way too heavy for a periodic safety net. Settingdirty = trueonly makesshouldSyncMemorypass the gate (line 1029), thensyncMemoryFilesdoes a lightweight hash comparison and skips unchanged files.3. Add
watcher.on('error')handlerFile:
src/memory/manager-sync-ops.ts(ensureWatcher)chokidar's
FSWatcherextends Node'sEventEmitter. Without anerrorlistener, errors likeENOSPC(inotify limit reached) propagate as unhandled exceptions — hitting the globaluncaughtExceptionhandler which callsprocess.exit(1), crashing the gateway. The new handler logs the error via the memory subsystem logger.Tests
memory-search.test.ts: DefaultintervalMinutesis 15; explicit overrides (including 0) respectedmanager.watcher-config.test.ts: Watcher registers anerrorevent handler; interval sync marks dirty when watcher has stopped firing (fake timer test with sync spy)All existing memory tests pass.
Alternatives considered
sync({ force: true })in interval callbackforcetriggersneedsFullReindex→ full atomic reindex (temp DB swap + re-embed everything). Too heavy for a 15-minute safety net.watch: truewatchis disabled because chokidar is broken, those users lose the fallback too.Bot review notes
Codex raised three P2 concerns across three review rounds, all addressed:
intervalMinutes) is extreme; those users can setintervalMinutes: 0. Even when the interval fires, unchanged files are skipped via hash comparison.buildFileEntrywould help but is out of scope. Multimodal memory is opt-in and off by default.