Filter no-op notifications for whitelisted dot-files and excluded tracked files; bump to 1.2.7 - #222

Fixes #221.

Since v1.2.4 (#211), the managed watcher drops file notifications that leave a file's trusted metadata and visibility unchanged, so external walkers and attribute churn don't trigger passes. #221 shows two kinds of files that the filter could never prove unchanged, so metadata-only notifications for them on an idle checkout still cleared a managed view's ready during periodic content verification. Every notification for them invalidated the path and cleared ready until the next pass:

1. Dot-files that an ignore-file whitelist admits (wl/.keep.txt, sub/.gitignore in the issue's fixture). The filter compared the raw hidden status of the notified file with the recorded hidden flag. The recorded flag is whitelist-adjusted (a whitelisted dot-file is recorded as visible), so the two never matched.
2. Tracked files that ignore rules exclude from the walk (sub/inner.txt, ign/tracked.log). The walk doesn't list them, so the view had no evidence to compare, and missing evidence always invalidates.

Fix

Case 1: compare raw with raw. The metadata walk now also reports the listed non-directory paths whose hidden status a whitelist cleared. The view keeps that set with its published state, so the filter reconstructs the raw recorded status (records_hidden || whitelisted) and compares it with the raw status of the notified file. A whitelisted dot-file is dropped only when its trusted version and raw hidden status both equal the published evidence. A verification that finds the published state current keeps only the entries that its own walk still reports as hidden and whitelisted (see Review follow-ups).

Case 2: version-checked records for excluded tracked files. Each pass now also records metadata (trusted FileVersion and raw hidden status) for tracked regular files that the walk didn't list. A notification for such a file is dropped only when its current trusted version and raw hidden status equal the record.

• Candidates are regular files in the base generation (the pinned commit's tree) that the published state doesn't list. A candidate gets a record only when the ignore rules the pass's walk applied exclude it or a parent directory, so a tracked file that no rule excludes, whether restored after the walk or skipped by the walk for another reason, gets none. Only rules that the walk applies the same way count as proof, so neither the global excludes file nor the repository's info/exclude does, and no file below a nested repository gets a record (see Review follow-ups). Tracked files that only those two files exclude get no record and keep invalidating, as in v1.2.6. Ignore and attributes files (.gitignore, .ignore, p4ignore.ini, .gitattributes, compared ASCII case-insensitively) never get a record, so notifications for them invalidate exactly as before, including their tree check where v1.2.6 requested one.
• Untracked ignored files (build output, for example) get no record and still invalidate.
• Records are rebuilt by every pass, which already rewalks the whole tree. Only existing paths are stat'ed: the builder lists each candidate directory once, skips the subtree of a missing directory, and skips names that aren't present, so deleted files and sparse or partial checkouts don't cost a failed stat each.
• Both new sets are counted in the view's retained memory, and the builder charges its working memory like the rest of the pass.
• A pass that publishes a new state publishes the records its walk produced along with it. A verification pass that finds the published state current updates the existing records in place, as it already does for listed evidence. It refreshes the records it finds again and drops the rest: files that are gone, are no longer regular files or have no trusted version. It adds none, so a file excluded since the last publication keeps invalidating until a pass publishes.
• A provisional first publication and a restored view carry no records, so notifications for those files invalidate as in v1.2.6 until a pass publishes records.

When the filter can't prove that nothing changed, it invalidates, as in v1.2.6. Missing evidence never justifies a drop, because a new file has none either. A file notification still invalidates when the path is missing, its version is untrusted, there is no evidence or record for it, its version or raw hidden status changed, or the view is closing, preparing or has no published state. Directory notifications still request a tree check, and creates, removes, renames and edits to ignore or attributes files invalidate as before.

Diagnostics (additive)

views.status → watch_events gains:

• unchanged_excluded_filtered: the part of the existing unchanged_paths_filtered that was proven unchanged by an excluded-file record;
• unfiltered: why file notifications were kept, as counts: missing, untrusted, no_evidence, version_changed, hidden_changed, unavailable.

last_reconcile gains excluded_tracked (records built by the pass) and timings.excluded_records_us.

Compatibility

All RPC and views.status changes are additive; no field was removed, renamed or retyped, so clients that read only authoritative, ready, closing and error are unaffected. Nothing persisted changes: the new sets live only in memory with the published view, so checkpoints, catalogs and other on-disk formats are unchanged.

Pre-existing behavior this PR doesn't change

• Rule edits racing a pass. Listed evidence is the version read with the file's content, after the walk parsed the ignore rules. v1.2.6 already drops equal-version notifications for hidden-recorded control files such as the root .gitignore. If an edit lands between the rule parse and the content read and its notification arrives after publication, the published rules stay stale until the next pass, which rewalks membership. Case 1 extends this to whitelisted control files such as sub/.gitignore; excluding them would leave the issue's sub/.gitignore case unfixed. Excluded-file records never include ignore or attributes files.
• p4ignore.ini. The managed watcher invalidates the root p4ignore.ini without requesting a tree check, while the legacy watcher treats it as an ignore-rules file. Unchanged here.
• Hard links. A write through one name is reported only for that name; other names of the same file aren't invalidated, with or without this PR.
• An in-tree global excludes file isn't treated as a rules file by the managed watcher. Unchanged here.
• Version evidence. On Windows a FileVersion is the size and the modification and creation times. A replacement with identical values (for example through file-name tunneling) is indistinguishable, as it already is for listed files in v1.2.6.
• Nested Git repositories created or removed later. The watcher drops notifications under .git, so creating or removing a nested Git repository below the root can go unnotified, although it changes which of the outer repository's rules the walk applies below it. The published state can then be stale until another pass runs. Records add one consequence: a record made before the change can drop a no-op notification for a file that the next walk lists, where v1.2.6 invalidated.
• Global excludes and the working directory. The metadata walk never sets the walker's working directory, so it matches global exclude patterns relative to the daemon's working directory rather than the view root. An anchored global pattern (a leading or interior /) therefore matches the paths Git would only when the daemon runs in the view's root; elsewhere it's anchored at the daemon's working directory. If the working directory can't be read, the walk applies no global rules at all. Records no longer depend on global rules.
• info/exclude behind a .git file. The walk applies the repository's info/exclude behind a .git file only when Git is required and the pointer, resolved against the daemon's working directory, leads to a Git directory with a commondir file. It therefore leaves the file out where Git applies it: in submodules and --separate-git-dir checkouts, behind any .git file with --no-require-git, and in linked worktrees with relative pointers unless the pointer also resolves from the daemon's working directory. Records no longer depend on info/exclude.
• Whitelists from those two files. The point-query matcher's whitelist check, which decides whether a hidden file counts as visible, still reads info/exclude and the global excludes file, roots global patterns at the view root, and finds info/exclude in the layouts above. Indexed queries without --hidden and a non-indexed scan can therefore disagree about a hidden file that only one of those files whitelists. This affects visibility only, not staleness: the whitelist applies only to paths the walk listed, and a notification is dropped only when the file's raw hidden status and version equal the published evidence.

Answers to the issue's questions

1. Is comparing the whitelist-adjusted hidden flag with the raw is_hidden intended? No. The recorded flag is whitelist-adjusted because queries use it for visibility. The filter needs to know whether the file's own hidden status changed, so it now compares the raw status with the raw status reconstructed from the published state.
2. Would dropping notifications for files the view's ignore rules exclude be acceptable, given that ignore and attributes files keep their rewalk? This PR takes a narrower route. A matcher-based drop would need a matcher kept consistent with the published state that agrees with the walk in every case (parent-directory exclusion, negations, .ignore precedence, info/exclude and global excludes, and the case-insensitive tracked-file exemption); any disagreement would drop a real membership change. Instead, excluded tracked files get the same version-checked evidence as listed files, so a drop needs the same proof. The record builder does use the matcher the walk built, but only as an extra condition for building a record, never to drop a notification. Untracked ignored files still invalidate; if they matter in practice, a matcher-based drop for them can follow separately.

Tests

Tests of the fix. Each fails on v1.2.6 product code (checked by restoring only the v1.2.6 product code under these tests, and separately by reverting just the comparison and the record lookup inside the new code):

• worktrees::tests::unchanged_file_drops_whitelisted_hidden_files (wl/.keep.txt, sub/.gitignore): on v1.2.6 fails at wl/.keep.txt.
• worktrees::tests::unchanged_file_drops_tracked_files_that_ignore_rules_exclude (sub/inner.txt, ign/tracked.log): on v1.2.6 fails at sub/inner.txt.
• worktrees::tests::excluded_records_classify_notifications_and_skip_absent_paths: uses the new reason API, so it can't compile on v1.2.6; with the comparison and lookup reverted it fails (HiddenChanged instead of Unchanged). It also checks that absent tracked paths and missing directories produce no records.
• serve::shared::managed::scheduling_tests::nested_git_metadata_events_use_the_most_specific_root_in_either_order, extended with no-op notifications for the four targets: on v1.2.6, two no-op notifications for wl/.keep.txt advance the view's epoch twice instead of not at all. It also checks that an edit to ign/tracked.log, a new file and a missing path still invalidate.

Real changes still invalidate:

• worktrees::tests::unchanged_file_keeps_new_files_rule_edits_and_visibility_changes: content edits to each target, an edit to a .gitignore that its own rule ignores (still requests a tree check), a new unignored file, and on Windows the hidden attribute set on a listed non-dot file. Passes on v1.2.6 and with the fix.
• The existing worktrees::tests::unchanged_file_requires_equal_trusted_evidence_and_hidden_status is unchanged and passes.

Diagnostics:

• serve::shared::managed::scheduling_tests::watch_counters_classify_filtered_and_kept_paths: counts each reason a notification was dropped or kept. It tests the counters' bookkeeping rather than the fix, so it also passes with the comparison and lookup reverted.

Memory accounting (first review follow-up):

• worktrees::tests::excluded_records_charge_scratch_before_allocating_and_release_it: runs the record builder with a real permit and memory charge over 200 excluded tracked files, each in its own directory. It checks that the caller's charge ends at its entry value plus the records, that the permit's peak covers the directory stack and the candidate list, and that the scratch is released. As added in the third commit, it fails against the second commit's worktrees.rs at the first check (58,296 bytes instead of 55,000, because the listed-path set and the candidate list stay on the caller's charge); with that check disabled, it fails at the peak check. The fourth commit changes only how the test gets the listed paths: from a walk instead of a literal list.

Exclusion proof (second review follow-up):

• worktrees::tests::excluded_records_require_the_walks_rules_to_exclude_a_file: walks a checkout under a permit, then restores a tracked file that none of the walk's rules exclude, then adds a rule that would exclude it. Only the file that the walk's rules excluded gets a record, both with the matcher the walk built and with one rebuilt from the rule files the walk read, and the walk's input charges are released on return. Against the third commit's product code, with the test adapted to that commit's signatures, it fails: the restored file gets a record. If the rebuilt matcher reads the rule files again instead of the walk's snapshots, it fails with the rebuilt matcher.

Whitelist pruning (fifth commit):

• worktrees::tests::unchanged_verification_prunes_whitelisted_entries_its_walk_no_longer_reports (Windows): without notifications, a file loses its hidden attribute and the info/exclude whitelist that admitted it, and verification finds the published state current. Setting the attribute again must then be reported as a hidden-status change. Without the pruning, it fails: the notification is classified Unchanged instead of HiddenChanged.
• worktrees::tests::unchanged_file_drops_whitelisted_hidden_files now also checks that an unchanged verification keeps the entries its walk still reports. If verification cleared the set instead of pruning it, both tests fail.

Rules the walk applies (sixth commit). The unit tests remove tracked files before the walk and restore them before the records are built:

• worktrees::tests::excluded_records_skip_nested_repositories: a root *.log rule excludes tracked files in a directory holding a .git directory, one holding a .git file, one holding a .jj directory, and a control directory with none, checked with the walk's matcher and with a rebuilt one. The test first checks the walk's premise: it lists the boundary directories' kept.log, which stays present and which the root rule excludes. Only the control directory's files get records, and the caller's charge ends at its entry value plus those records. In each boundary directory, -a.log and -d/x.log list before the marker on NTFS (observed order: -a.log, -d, .git or .jj, kept.log, x.log), so a record and a queued subdirectory are taken before the boundary is found and must be rolled back. Against the fifth commit's worktrees.rs, it fails with 9 extra records; with only the rollback disabled, it fails with the 6 taken before the marker. A full pass with a root .ignore rule, which the walk applies below nested repositories too, then checks that excluded_tracked counts only the control directory's records.
• worktrees::tests::excluded_records_never_take_global_excludes_as_proof: the walk's matcher is replaced by one whose only proof for three files is a global pattern (anchored, with an interior /, and unanchored), plus a control that .gitignore excludes. Only the control gets a record; with the gate's full rule set, all four do.
• shared_daemon test managed::global_excludes_never_prove_excluded_file_records: a daemon with an isolated home whose core.excludesFile holds the same three patterns, and whose working directory is outside the view. With all files present, its walk lists the two anchored files and excludes the unanchored one, and excluded_tracked is 1, for the .gitignore control. With the gate's full rule set, it is 2: the unanchored file gets a record too.
• worktrees::tests::excluded_records_never_take_the_repository_exclude_file_as_proof: info/exclude excludes a tracked file behind a .git directory, behind a .git file that points to a separate Git directory (as git init --separate-git-dir writes it), and in a linked worktree whose .git file holds a relative pointer. The test first checks the walk's premise: only the .git directory layout's walk applies the file. Then the file gets no record with either matcher, while a file that .gitignore excludes does. The directory layout pins the conservative rule: there the walk applies info/exclude too, so a record would be safe, but the gate still doesn't take the file as proof. With info/exclude accepted as proof again, each layout fails on its own; in the separate and relative layouts, the record goes to a file that the next walk lists.

Validation (Windows x64)

Run on this branch's head, 9a95c5f. For the shared_daemon and qualification runs, TGREP_V1_BINARY and TGREP_V1_READER point at a scripts/qualification/prepare_legacy.py build, as in CI.

The scripts/agent unit tests need fcntl, so CI runs them only off Windows; they weren't run here.

Cost on a large tree

Measured on Windows x64 with release builds and temporary benchmarks that aren't part of this PR. The synthetic Git repository has:

• 100,001 listed files in 2,000 directories;
• 4,000 tracked .log files that a *.log rule excludes, four in each of 1,000 directories;
• 2,500 tracked paths that are absent from the checkout: a deleted 2,000-file subtree and 500 deleted files.

The pass table was measured on the first commit, with a benchmark that drives WorktreeView passes directly and without a permit. Every pass reported excluded_tracked: 4000.

• Building the records added 156–168 ms per pass on this tree; discovery_us includes it. That is about 15% of a metadata verification and 0.5% of a warm content pass. The first review fix adds 14–23 ms to that, and the second 0–11 ms; see the benchmark items below.
• Finding the candidates hashes every listed path and checks every tracked entry, so every pass pays for that scan, plus one listing of the root, even when no tracked file is excluded. A standalone replica of the scan takes about 15 ms over this tree's paths and about 80 ms over 287,000 synthetic paths, near the size of the checkout in v1.2.6: Watcher notification on an idle Windows checkout clears ready during periodic content verification (unchanged-file filter gaps) #221. Since the first review fix, each tracked entry is checked twice: once to count the candidates before their list is allocated, and once to collect them.
• The rest, about 145 ms here, is listing directories and recording the 4,000 files. The builder lists each directory that contains a candidate, directly or deeper down, once: 1,542 directories here, including 500 leaf directories whose only candidate is a deleted file. It skips a missing subtree as a whole and never stats a missing name, so the deleted 2,000-file subtree costs no listing or stat.
• Retained memory is charged at the path length plus 256 bytes per record, about 1.1 MB for these 4,000 records. While the builder runs, its scratch is charged as well: here 4.8 MB for the set of 100,001 listed paths, released once the candidates are collected, and 104 KB for the 6,500 candidates. Since the second review fix, the walk's charge for the rule-file contents it read and the patterns it parsed stays held until the records are built, instead of being released when the walk ends, so the builder's scratch now adds to it. That charge grows with the size and number of rule files, not with the number of files; here it covers one single-pattern .gitignore.
• For scale, the checkout in v1.2.6: Watcher notification on an idle Windows checkout clears ready during periodic content verification (unchanged-file filter gaps) #221 has 3,893 such files among 293,288 discovered files, and its discovery took 7.6 s.
• For the first review fix, a second benchmark runs only the record builder on this tree: 21 rounds per process, three processes per build, alternating between the second commit, the third, and the third without its counting pass. The median time went from 164 ms to 178 ms without a permit, as in the pass table, and from 182 ms to 205 ms with a permit and memory charge, as managed passes run. Without the counting pass, the third commit took 161 ms and 180 ms, so the counting pass accounts for the increase; with a permit, it also checks for cancellation at every entry. The permit's peak fell from 6.00 MB to 4.90 MB, because the listed-path set's charge is now released before the records are charged.
• For the second review fix, the same benchmark ran twice more on a regenerated tree of the same shape, comparing the third and fourth commits. The second run also alternated with the fourth commit without its exclusion check. Each run alternates the builds, so compare within a run; the same build's median moved by up to 10 ms between runs. The fourth commit's median was 0–7 ms higher without a permit (178 → 185 ms in the first run, 188 ms for both in the second) and 6–11 ms higher with one (203 → 214 ms, then 212 → 218 ms). Without the check, it took 186 ms and 211 ms, within 2 ms of the third commit, so the point queries against the walk's matcher, one per candidate found on disk, account for the increase. The permit's peak stayed at 4.90 MB; the benchmark holds no input control, so it doesn't include the walk's rule-file charge.
• The fifth and sixth commits weren't benchmarked. The fifth adds a set filter to verifications that find the published state current. The sixth compares each name the builder lists below the root with .git and .jj, ignoring ASCII case, stops listing a directory at the marker, and matches candidates against fewer rules.

Version

The second commit bumps the version to 1.2.7, as the 1.2.6 bump did: Cargo.toml, Cargo.lock, fuzz/Cargo.lock, and the two examples in CONTRIBUTING.md.

Review follow-ups

The third commit, cd26f6a, addresses a review comment: the record builder now charges its scratch (the listed-path set, the candidate list and the directory stack) to the pass's private-memory reservation before allocating it, and releases it when it drops, so the caller's charge grows only by the records. Before, the set and the list were charged to the caller's charge, which held both until publication resized it or the pass ended; the list was charged after it was allocated, and by length; and the stack and the per-directory name lists weren't charged. The records are unchanged: each directory entry's name is now looked up by binary search in the directory's sorted candidates, which replaces those lists.

The fourth commit, b9da528, addresses a second review comment: the record builder recorded every present tracked file that the walk hadn't listed, as if ignore rules excluded it. A tracked file restored after the walk is unlisted too, so it could get a record; if its create notification was missed, a later notification that left it unchanged was dropped, although the published state didn't contain the file. A candidate now gets a record only when the ignore rules the walk applied exclude it or a parent directory. The metadata walk returns the point-query matcher it already builds for visibility; when it built none, one is built from the walk's rule files once a candidate is found on disk. Managed views always walk under a permit, so both matchers parse the rule-file contents the walk read through its input control: nothing is read again, and a rule changed since the walk proves nothing. A rebuilt matcher leaves out the case-insensitive tracked-file layer, which only adds exclusions. Files the walk skipped only for other reasons get no record, so their notifications keep invalidating.

The fifth commit, 3204323, addresses a later review's note on stale whitelist state. A verification that found the published state current refreshed the evidence but kept the set of whitelisted hidden paths as published. On Windows, a file can lose both its hidden attribute and the whitelist that admitted it without a change to its visibility or its file version, which has no change time. If neither change was notified, verification kept the stale entry; the notification for setting the attribute again then looked unchanged and was dropped, so the file stayed visible to queries without --hidden until a later pass or verification. v1.2.6 invalidated in that case. Elsewhere the raw hidden status depends only on the name, so changing it is a rename, which verification sees as a membership change. Unchanged verification now keeps only the entries that its own walk still reports as hidden and whitelisted. It prunes the set rather than replacing it, as it does the records, because retained view memory is charged for the published set. A whitelisted file hidden since the last publication therefore has no entry and keeps invalidating until a pass publishes.

The sixth commit, 9a95c5f, addresses the latest review's note on mismatched nested Git ignore boundaries, and two related gaps found while auditing it. A record lets the watcher drop a no-op notification, so the matcher must not exclude a file that the walk would list. It could in three cases:

• Nested repositories. Below a nested repository, the walk stops applying the outer repository's .gitignore and info/exclude rules, while the matcher still applies them. A tracked file there that was absent during the walk and restored before the records were built could get a record although the next walk lists it; if its create notification was missed, a later notification that left it unchanged was dropped, although the published state didn't contain the file. Records now skip every directory below the root that holds an entry named .git or .jj, of any type and in any ASCII case, or whose listing fails partway. The walk treats a directory as another repository only when a lookup of .git or .jj there finds an entry, and never with --no-require-git. Unless the file system equates another name with those beyond ASCII case, this skips at least every subtree the walk treats as one, and skipping only removes records. A listing can return the marker after other entries, so the records already taken from that directory are released with their charges, and its queued subdirectories are dropped.
• Global excludes. The walk matches global patterns relative to the daemon's working directory, while the matcher roots them at the view root, so an anchored global pattern could exclude a file in the matcher that the walk lists.
• info/exclude behind a .git file. The walk applies it only when Git is required and the pointer, resolved against the daemon's working directory, leads to a Git directory with a commondir file. The matcher resolves a relative pointer against the directory holding .git, and takes a Git directory without commondir, such as a submodule's or a --separate-git-dir checkout's, as its own common directory, so it applied info/exclude where the walk didn't.

Neither exclude file proves exclusion any more. Both have the lowest precedence of the standard rules, and the p4ignore and case filters don't read the standard decision, so leaving them out of the proof only removes exclusions and stays safe however the walk resolves them. When Git ignores case, the case filter still matches info/exclude's rules, but it never hides a tracked file, and the walk applies the same filter. This narrows case 2: tracked files that only info/exclude or the global excludes file excludes get no record, and their no-op notifications keep invalidating, as in v1.2.6. Outside nested repositories, records proven by .gitignore, .ignore or the p4ignore filter are unchanged, as are the walk, visibility and whitelists. The walk's own handling of both files is unchanged too (see Pre-existing behavior above).