Watcher can repeatedly re-index dirty repos and cause large disk write amplification #937
Description
Summary
We observed background watcher processes causing very large sustained disk writes during normal MCP usage. On a macOS workstation using Codex with several indexed repos, host write counters exceeded 1 TB/day.
The issue appears to be write amplification from the watcher/indexing path, not application logs.
What we observed
- Multiple processes were active for different repos.
- Some processes wrote hundreds of MB within a short sampling window while idle from the user's perspective.
- Codex SQLite TRACE/DEBUG/INFO logging was ruled out: log inserts were blocked and the log WAL stopped growing.
- Re-indexing repeated even without commits, because the worktree stayed dirty/untracked.
Suspected cause
The watcher treats a dirty/untracked worktree as changed on every poll. If the dirty state persists, each poll can trigger another re-index.
The incremental indexing path reduces some parsing work, but persistence still appears to rewrite/replace the project SQLite DB and may also rewrite the compressed artifact. For larger repos this can turn each watcher poll into hundreds of MB of writes.
This is amplified when multiple MCP clients/sessions run in the same repo or overlapping repos, since each process can independently trigger indexing and there does not appear to be a cross-process index lock/deduplication guard.
Expected behavior
The watcher should not repeatedly re-index the same unchanged dirty state.
Possible mitigations:
- Trigger background watcher re-index only on Git HEAD changes, or
- Track a dirty-state signature and re-index only when that signature changes, and
- Add a cross-process lock/deduplication guard so multiple MCP processes cannot rebuild the same repo index concurrently.
Longer term, incremental indexing should ideally avoid near-full DB/artifact rewrites when only a small set of files changed.
Related issues
This seems related to several existing reports, but combines them into one disk-write-amplification failure mode:
- watcher: continuous auto-sync re-indexing churn on Windows (real driver of #832) #841
- Incremental reindex costs ≈ a full index, especially after artifact bootstrap #867
- Add config key to disable file watcher #335
- auto_index uses tens of GB of memory in a non-git folder containing code #713
- Fast artifact export raw-copies a live WAL-mode DB — torn graph.db.zst that import then installs unchecked #895
- Installing a fresh DB file leaves the old -wal/-shm behind — SQLite replays the stale WAL over the new file #897
- Windows v0.8.1: stale graph + Permission denied on delete_project after orphan cbm process (reconfirm #277) #914
Activity
DeusData commentedon Jul 7, 2026
Thanks for the write-amplification report. Triage: high-priority watcher/performance bug under #593.
Persistent dirty or untracked state should not cause repeated full re-index/write cycles while the user is idle. A fix should reproduce a dirty-repo polling loop and prove the watcher coalesces or suppresses repeat work without missing real file changes.
Artemonim commentedon Jul 8, 2026
I disabled the auto-update of the MCP itself, since on the contrary it works too rarely for me while the agent is running, and I built the index rebuild into my local CI, which the agents themselves call when they finish part of the work.
DeusData commentedon Jul 12, 2026
Fixed on main by 9789a7c (merged in 94be485, PR #1045), shipping in v0.9.1.
What changed, mapped to your asks:
git status --porcelain -uall -zentries plus each listed file's (size, mtime_ns). A persistently dirty/untracked tree reindexes once per distinct state — idle polls over the same dirty tree no longer trigger anything, killing the repeated full reindex → DB/artifact rewrite cycle. Editing a dirty file again, adding/removing files, or reverting to clean each trigger exactly one reindex, so no real change is missed.The near-full DB/artifact rewrite per incremental run (your "longer term" point) is tracked in #867, where a contributor fix stack (#868–#870) is in review.
Thanks for the write-counter evidence — it made the amplification loop unambiguous.
fix(watcher): reindex dirty repos once per distinct state, not per poll
fix(watcher): reindex dirty repos once per distinct state, not per poll
fix(watcher): stop nested non-git dirs and monorepo siblings retrigge…
fix: address QA round 1
fix(watcher): back off re-forking a hard-failing index worker
test(watcher): wait for the 14th reindex attempt in the sustained-fai…
test(watcher): pin the pending-free drain on a held state, not a git …
fix(watcher): back off re-forking a hard-failing index worker
fix: address QA round 1
test(watcher): pin the pending-free drain on a held state, not a git …
test(watcher): wait for the 14th reindex attempt in the sustained-fai…
test(watcher): pin the pending-free drain on a held state, not a git …
test(watcher): wait for the 14th reindex attempt in the sustained-fai…
feat(watcher): two-round confirmation + cooldown + index_status fresh…
feat(watcher): opt-in polling of non-git project roots (#1948)
test(watcher): poll-only-watched-projects waits for the watched reind…