Repository navigation
v1.2.6: Watcher notification on an idle Windows checkout clears ready during periodic content verification (unchanged-file filter gaps) #221
Description
Summary
On an idle Windows checkout, a managed view lost its ready flag during a periodic content verification, although as far as I can tell no file's content, version or visibility had changed.
- Between the samples taken 30.8 s and 32.2 s into the pass, four
Modifynotifications arrived. The unchanged-file filter dropped three. - The watcher passed the fourth to
ViewSlot::invalidate. The input epoch advanced from 48 to 49, and the sample at 32.2 s reportedready: false. The view kept serving its last publication.
I think the fourth notification hit one of two gaps in WorktreeView::unchanged_file (worktrees.rs:555-571):
- Whitelisted dot-files. The filter requires the view's recorded hidden flag to equal the raw
is_hidden. The recorded flag isfalsefor a dot-file that an ignore-file whitelist admits, butis_hiddenistrue, so the filter never matches such a file. This checkout has at least one: a nested.gitignorethat contains*followed by!.gitignore. - Files the walk didn't list. Only files the walk listed have evidence, so the filter never matches any other file. This checkout has 3,893 tracked files that ignore rules match.
The daemon doesn't record which paths it invalidates, and it has since exited, so I can't tell which case applied. In an earlier run on the same checkout and binary, the filter dropped all 8 notifications of a burst between 41.9 s and 44.9 s into a content pass, and the view stayed ready.
Environment
- tgrep v1.2.6, one managed shared daemon. A single host process (the Copilot runtime) held the view's only lease.
- Windows 11, 16 logical cores.
- A large private repository with about 299k tracked files in about 41k directories. The last reconcile discovered 293,288 files. The overlay had 3,893 tombstones; all of them are tracked files that are present on disk and matched by ignore rules.
- A ReFS Dev Drive that isn't marked trusted. Defender real-time protection is on, and an endpoint management agent periodically walks the tree. This is the same host as Managed shared mode on a large Windows repo: 25–32 min cold readiness, watcher input churn, scheduler stall, query cliffs (v1.2.3) #210.
- The reconciliation policy is in the excerpt below.
Observations
Both runs used the same checkout and tgrep binary. The harness polled views.status throughout: 94 min in run 1 and 69 min in run 2. Times are relative to the start of the content pass, computed from the running work item's elapsed_ms. Line numbers refer to the v1.2.6 tag.
| Run 1 (earlier) | Run 2 (this failure) | |
|---|---|---|
| Notifications before the pass | None in the preceding 49 min. Earlier, only 4 Git-metadata notifications, which the watcher ignores | None in the 68 min observed before the pass |
| Burst, between the samples at | 41.9 s and 44.9 s | 30.8 s and 32.2 s |
events / by_kind.modify |
+8 / +8 | +4 / +4 |
unchanged_paths_filtered |
+8 | +3 |
invalidated_paths / epoch_advances |
+0 / +0 | +1 / +1 |
| Other watcher counters | Unchanged | Unchanged |
| Result | ready stayed true; the pass finished unchanged after about 266 s |
ready: false at the next sample (input_epoch 49, authoritative input_epoch 48); the harness stopped there |
In run 2's last sample, last_verification still describes the previous metadata pass. content_age_ms exceeds content_delay_ms, so the content pass was due, and the only running work item is that background verification.
After run 2, git status --ignored was empty. No file or directory in the checkout had a modification or creation time in the 98 minutes before the failure, which covers the whole observed window, and none had the Windows hidden attribute.
Run 2's last views.status sample (IDs, paths, hashes and the operation ID omitted)
{
"ready": false,
"serving": true,
"closing": false,
"error": null,
"input_epoch": 49,
"authoritative": {
"active": true,
"committed": true,
"input_epoch": 48,
"reconciled_epoch": 50,
"pending": null
},
"leases": 1,
"queries": 0,
"queued_work": [
{
"class": "background",
"state": "running",
"elapsed_ms": 32165
}
],
"last_verification": {
"age_ms": 152391,
"content_age_ms": 4768537,
"content_attempt_age_ms": 4768537,
"content_delay_ms": 4715836,
"content_retry_delay_ms": 4715836,
"metadata_delay_ms": 162631,
"outcome": "unchanged",
"tier": "metadata"
},
"watch_mode": "native",
"watch_count": 2,
"watch_fallback": null,
"watch_events": {
"by_kind": {
"access": 0,
"create": 4,
"error": 0,
"modify": 83,
"other": 0,
"remove": 0
},
"epoch_advances": 49,
"events": 87,
"full_invalidations": 0,
"git_metadata_ignored": 35,
"invalidated_paths": 49,
"tree_invalidations": 0,
"unchanged_paths_filtered": 3
},
"overlay": {
"tombstones": 3893
},
"last_reconcile": {
"full": false,
"epoch": 50,
"files_discovered": 293288,
"metadata_reused": 293288,
"base_reused": 287435,
"overlay_reused": 1,
"content_reads_avoided": 293288,
"files_read": 0,
"provisional_files": 0,
"peak_workers": 12,
"peak_private_bytes": 2025108392,
"timings": {
"discovery_us": 7582536,
"admitted_us": 8559562
}
},
"reconciliation_policy": {
"change_journal": "not-integrated",
"configured_watch_mode": "auto",
"content_interval_ms": 1800000,
"full_repair_triggers": [
"first-attachment-or-restore",
"explicit-full-refresh",
"candidate-read-error",
"watcher-error-or-overflow",
"git-or-ignore-change",
"periodic-verification"
],
"full_verification_reason": "periodic-native-verification",
"incremental_coverage": "unavailable",
"interval_ms": 120000,
"metadata_interval_ms": 120000,
"native_missed_changes": "repaired-by-periodic-verification",
"native_periodic_full_verification": true,
"native_registration_mode": "recursive-roots",
"native_verification_availability": "serving-until-difference",
"native_watch_budget_per_view": 256,
"periodic_full_verification": true,
"poll_interval_ms": 120000,
"verification_idle_cost_multiplier": 19
}
}How I narrowed it down
- notify 7.0.0 sends one event per path (windows.rs:359). A rename inside the watched tree produces two
Modify(Name)events, one per endpoint (windows.rs:370-382), andFILE_ACTION_MODIFIEDbecomesModify(Any)(windows.rs:393-396). - In both runs
eventsequalsunchanged_paths_filtered + invalidated_paths, and no other watcher counter moved. So the invalidated path wasn't:- a directory that was still readable, the checkout root or a path outside it, which set
tree; - Git metadata or a path under
.gitor.tgrep; - a created or removed file;
- an endpoint of a rename inside the checkout, which would have invalidated two paths.
- a directory that was still readable, the checkout root or a path outside it, which set
- A rename with only one endpoint inside the checkout can't be ruled out from the counters, but it seems unlikely.
ViewSlot::unchanged_filealso returnsfalsewhile a replacement publication is being prepared (views.rs:2149-2158). That's unlikely here: the running work item was the periodic verification (a backgroundreconcilewithverify: true), and the other three notifications in the same burst were filtered.
Code path
watch_eventtreatsModify(Any | Data | Metadata | Other)as a content candidate. The comment there notes that Windows reports attribute-only changes asModify(Any)(managed.rs:1123-1133).- The watcher stats each candidate. A directory sets
tree, and a file is dropped ifViewSlot::unchanged_filereturnstrue. Any other path is pushed and passed toViewSlot::invalidate(managed.rs:1182-1211). WorktreeView::unchanged_file(worktrees.rs:555-571) requires a regular file with a trusted version, a recorded hidden flag equal to the rawis_hidden, and evidence with an equal version.- The walk records every hidden entry (walker.rs:526-546).
finish_visibility(walker.rs:549-575) then removes the entries an ignore-file whitelist admits (visibility.rs:77-91; the comment at visibility.rs:69 reads "Ignore-file whitelists can admit a hidden entry even without --hidden"). The view keeps that map (worktrees.rs:814), andPathVisibility::recordapplies the same rule (visibility.rs:60-61) throughis_whitelisted(gitignore.rs:243-248). is_hiddenistruefor a leading.or the hidden attribute (visibility.rs:110-122). So for a dot-file that a whitelist admits,records_hiddenreturnsfalsewhileis_hiddenreturnstrue.- Evidence is created only for files the walk listed (worktrees.rs:738-775, worktrees.rs:1756-1760). A file that ignore rules exclude has none.
- The walk records every hidden entry (walker.rs:526-546).
ViewSlot::invalidate(views.rs:2174-2176) callsinvalidate_inner(views.rs:2183-2230), which reachesWorktreeView::invalidate_path(worktrees.rs:594-629). A path with a.git,.gitignore,.ignoreor.gitattributescomponent also setscontrol.tree(worktrees.rs:615-620). So a whitelisted.gitignorewould also force a rewalk, which the watcher'stree_invalidationscounter doesn't count.readyrequires the publication'sinput_epochto equal the slot's epoch (views.rs:1924-1930), so one invalidation clears it. The view keeps serving because managed views favor availability and allow stale reads (managed.rs:2656-2658).- The comment on
verify_controlled(worktrees.rs:957-971) says the pass verifies the view "while it keeps serving its published state", and: "A concurrent invalidation leaves its hints queued; an otherwise unchanged pass returnsVerification::Supersededand retains verified evidence for the next pass." Soreadymost likely staysfalseuntil the pass ends and a later refresh applies the queued hint.
What I couldn't verify
Both bursts came 30–45 s into a content pass, after at least 49 quiet minutes, so the pass may have triggered them. I couldn't confirm that. In #210 (section 2, same host), an external tree walk produced attribute-only notifications for unmodified files, while plain reads produced none. If the pass did trigger them, case 1 is the better fit: the pass reads the files the walk listed, and case 2 covers only files it didn't list.
Expected
The comment at managed.rs:1182-1184 says: "Drop notifications that left a file's trusted metadata equal to the indexed evidence, so external walkers and attribute churn do not trigger passes." That should also hold for whitelisted dot-files and for files the walk didn't list. Real changes must still invalidate: a version change, a hidden-attribute change that changes visibility, and edits to ignore or attributes files, even ignored ones.
Impact
- No incorrect results: the view kept serving its last publication.
readymost likely staysfalseuntil the pass ends, about 4 more minutes here. That's an estimate from run 1's 266 s pass, not a measurement.- Hosts that gate on
readystop using the index meanwhile. The Copilot runtime falls back to scanning. - If the path was an ignore or attributes file, the invalidation also forces a rewalk. Discovery took 7.6 s on this checkout.
Possible fixes
- Compare like with like in
unchanged_file: apply the whitelist when computing the expected hidden flag, or keep the raw hidden bit with the evidence. - Drop candidates for regular files that the view's ignore rules exclude. Check the matcher rather than missing evidence, because a new file has no evidence either. Ignore and attributes files keep their rewalk. Alternatively, keep metadata-only evidence for files the walk skips.
- Report why a candidate wasn't filtered in
views.status, for example as counters for missing evidence, a hidden mismatch, a version mismatch or a preparing slot, and optionally the last few invalidated paths. That would have identified the path here. - Add a Windows regression test with a subdirectory
.gitignorecontaining*and!.gitignore, and a force-added file that ignore rules match. Toggleattrib +Randattrib -Ron each, then assert thatunchanged_paths_filteredrises,invalidated_pathsdoesn't, and the view stays ready.
Questions
- Is comparing the whitelist-adjusted hidden flag with the raw
is_hiddenintended? - Would dropping notifications for files the view's ignore rules exclude be acceptable, given that ignore and attributes files keep their rewalk?
I haven't run a standalone reproduction yet; I'll add one here once the host is free. Sanitized excerpts of the views.status samples from both runs are available on request.
Related: #210.
Activity
shengyfu commentedon Nov 1, 2026
Standalone reproduction
Both cases reproduce on a synthetic repository on the same host as runs 1 and 2, with the tgrep v1.2.6 release binary:
readyatfalseuntil the pass ended, and the pass completed assuperseded.Method
The fixture is a new Git repository with
core.autocrlf=false, no system Git configuration, and a scratch global configuration that only turns off automaticgcand maintenance. It has 30,000 (Phase A) or 120,000 (Phase B) bulk text files of 8 KiB underbulk/, 200 per directory, plus these files, all tracked:control.txt.editorconfigroot = true.gitignore*.logwl/.gitignore!.keep.txtbulk/d000/f00007.txtwl/.keep.txtwl/.gitignoresub/.gitignore*, then!.gitignoresub/inner.txtsub/.gitignoreign/tracked.log*.logThe view listed 30,006 or 120,006 files: the bulk files and the six other files marked yes.
os.chmod(path, S_IREAD), a 0.2 s pause, thenos.chmod(path, S_IREAD | S_IWRITE). This sets and clearsFILE_ATTRIBUTE_READONLY, likeattrib +Randattrib -R. The script checks that the mtime and size are unchanged and that neither the hidden nor the read-only attribute is left set.tgrep serve --shareddaemon with a policy that matches the Copilot runtime's defaults (workers1,queue_items256). The view is attached the way the runtime attaches one:owners.prepare, atgrep shared owner-hold --claimchild,owners.register, thenviews.attachwith contentraw-git-blob-auto-v1, coveragetracked-regular-files-v1and content-only eligibility.tgrep shared manage <repo>/.git views.statusprocess. The median spacing was 0.26–0.27 s at idle and 0.56 s during a content pass.--poll-interval 2): wait for a periodic content verification (a runningreconcilewithverify: truethat isn't metadata-only) and time it without toggling. During each of the next two passes, togglecontrol.txt5 s in and the target 8 s in, then sample until the operation has finished and the view is ready and idle.Phase A: at idle, 30,006 listed files
The attach took 8.7 s, with native watching. The first full content pass read all 30,006 listed files and produced no notifications. All 18 toggles matched the expectation, none overlapped a verification, and the view kept serving throughout.
events+2 andunchanged_paths_filtered+2, nothing else.readynever dropped, and no operation ran.events+2,invalidated_paths+2 andepoch_advances+2, with no tree or full invalidation. Each queued one hintedreconcile(verify: false,full: false), which restoredreadyabout 1 s after the toggle.files_read/metadata_reusedwl/.keep.txtsub/.gitignoreign/tracked.logsub/inner.txtTimes are seconds after the toggle returned: the first and last not-ready samples, then the next ready sample. Every hinted reconcile reported
files_discovered30,006 andhint_lookups90,009, withtimings.admitted_usbetween 0.825 s and 0.892 s.Phase B: during a content pass, 120,006 listed files
The attach took 25.5 s, with native watching. Times are from the start of the content pass, computed from the
elapsed_msthatviews.statusreported for it inqueued_work. Toggle times are when the toggle returned.ign/tracked.logwl/.keep.txtcontrol.txttoggleverified: true,changed: false,full: true,files_read120,006verification: "superseded",verified: null,full: false,files_read0,metadata_reused120,006verification: "superseded",verified: null,full: false,files_read1,metadata_reused120,005timings.admitted_usreadystayedfalsein every sample from the drop until the operation finished: 12.2 s (case 2) and 12.6 s (case 1) after the target toggle.last_verification.outcomeatsuperseded. The next content pass started 394 s after case 2's pass ended, which matches thecontent_delay_msreported after that pass (393,961 ms).What this shows
readyuntil a refresh applies the hint. At idle that refresh is a separate hinted reconcile, about 1 s here. During a content pass,readystaysfalseuntil the pass ends. With run 1's pass of about 266 s, that would be about 234 s in run 2 and about 70 s in run 3 (below); both are estimates.readystayedfalseuntil the pass ended, but no later refresh was needed. The superseded operation applied the queued hint itself: its receipt describes that hinted step (full: false;files_read1 forwl/.keep.txt, the hinted path), and the view was ready in the sample where the operation finished.verified: null, but the scheduler handlessupersededlikeunchanged(managed.rs:2201-2206). With run 2'scontent_delay_msof 4,715,836 ms, the next content verification on the real checkout would come about 79 min later..editorconfig,.gitignore,wl/.gitignore), two of them ignore files, were filtered, and the two whitelisted dot-files weren't. So the mismatch comes from the whitelist adjustment, not from hidden files or ignore files in general.files_read0), yet each keptreadyatfalseuntil it finished.files_discovered30,006 andhint_lookups90,009. Thesub/.gitignorerefreshes took 0.842 s and 0.853 s (timings.admitted_us), within the 0.825–0.892 s of the other targets.Modifynotifications.Run 3
A third run on the same checkout and binary failed the same way. Its content pass started after 61.7 minutes without notifications. Between the samples at 195.3 s and 196.3 s into the pass, 36
Modifynotifications arrived:unchanged_paths_filtered+31,invalidated_paths+5 andepoch_advances+5 (input epoch 16 to 21; authoritative 16). The tree, full and Git-metadata counters didn't move. The view reportedready: falseand kept serving. The harness stopped at that sample, so as in run 2 there's no receipt.Still unverified
Separate observation: notifications during the first pass after a checkout
The 120k fixture was committed with
git fast-importand checked out withgit reset --hard, like a fresh clone. During the first full content pass after the attach, which took about 45 s, 237,540Modifynotifications arrived. None were filtered (unchanged_paths_filteredstayed 0). A hinted reconcile then re-read 120,000 of the 120,006 listed files (21.4 s,hint_lookups720,009). The view first reportedready65 s afterviews.attachreturned.Later content passes over the same files produced no notifications besides the toggles. On the 30k fixture, where
git addhad read every file after the script wrote it, the first pass produced none either. I haven't identified the source. If it's general, a host that attaches to a fresh clone or worktree reads every file twice before the view is first ready. Would you like a separate issue for this?Environment
I can attach the script (standard-library Python only) if it would help.
Filter no-op notifications for whitelisted dot-files and excluded tra…