Skip to content

Commit 9a95c5f

Browse files
shengyfuCopilot
andcommitted
Prove excluded-file records only with rules the walk applies
An excluded-file record lets the watcher drop an unchanged notification for a tracked file the walk did not list. A record is made only when the pass's matcher proves the file excluded, so the matcher must not exclude a file the walk would list. It could in three cases. Below a nested repository, the walk stops applying the outer repository's `.gitignore` and `info/exclude` rules, while the matcher still applies them. A tracked file there that was absent during the walk and restored before the records were built could get a record although the next walk lists it. If its create notification was missed, a later notification that left it unchanged was dropped, although the published state did not contain the file. Records now skip every directory below the root that holds an entry named `.git` or `.jj`, of any type and in any ASCII case, or whose listing fails partway. The walk treats a directory as another repository only when a lookup of `.git` or `.jj` there finds an entry, and never with `--no-require-git`. Unless the file system equates another name with those beyond ASCII case, this skips at least every subtree the walk treats as one, and skipping only removes records. A listing can return the marker after other entries, so the records and queued subdirectories already taken from that directory are then released. Neither exclude file proves exclusion any more. The metadata walk never sets the walker's working directory, so it matches global patterns relative to the process's working directory, while the matcher roots them at the view root. An anchored global pattern could therefore exclude a file in the matcher that the walk lists. Behind a `.git` file, the walk applies the repository's `info/exclude` only when Git is required and the pointer, resolved against the process's working directory, leads to a Git directory with a `commondir` file. The matcher resolves a relative pointer against the directory holding `.git`, and takes a Git directory without `commondir`, such as a submodule's or a `--separate-git-dir` checkout's, as its own common directory, so it applied `info/exclude` where the walk did not. Both files have the lowest precedence of the standard rules, and the p4ignore and case filters don't read the standard decision, so leaving them out of the proof only removes exclusions and stays safe however the walk resolves them. When Git ignores case, the case filter still matches `info/exclude`'s rules, but it never hides a tracked file, and the walk applies the same filter. Tracked files that only an exclude file excludes get no record and keep invalidating, as in 1.2.6. Outside nested repositories, records proven by `.gitignore`, `.ignore` or the p4ignore filter are unchanged, as are the walk, visibility and whitelists. The tests remove tracked files before the walk and restore them before the records are built, with the walk's matcher and with a rebuilt one, an injected matcher whose only proof is global, a daemon whose global excludes file holds anchored and unanchored patterns, and `info/exclude` behind a `.git` directory, a separate Git directory and a linked worktree's relative pointer. Co-authored-by: Copilot App <[email protected]>
1 parent 3204323 commit 9a95c5f

5 files changed

Lines changed: 421 additions & 20 deletions

File tree

‎SHARED_WORKTREE_INDEXES.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1597,6 +1597,31 @@ trusted metadata. A tracked file the walk did not list gets a record only when
15971597
the ignore rules the walk applied, parsed from the rule-file contents it read,
15981598
exclude the file or a parent directory; a tracked file that no rule excludes,
15991599
whether restored after the walk or skipped only for another reason, gets none.
1600+
Neither the global excludes file nor the repository's `info/exclude` counts as
1601+
that proof, because the walk can list a file that one of them excludes from the
1602+
root. The walk matches global patterns relative to the daemon's working
1603+
directory rather than the view root, so an anchored global pattern can match a
1604+
file from the root that the walk lists. Behind a `.git` file, the walk applies
1605+
`info/exclude` only when Git is required and the pointer, resolved against the
1606+
daemon's working directory, leads to a Git directory with a `commondir` file. An
1607+
absolute linked-worktree pointer does; a submodule's or a `--separate-git-dir`
1608+
checkout's Git directory has no `commondir`, and a relative pointer resolves
1609+
from the wrong directory. These two files have the lowest precedence of the
1610+
standard rules, so leaving them out only removes exclusions: a tracked file that
1611+
only they exclude gets no record, and its no-op notifications keep invalidating,
1612+
as before 1.2.7. When Git ignores case, the case filter still matches
1613+
`info/exclude`'s rules, but it never hides a tracked file, and the walk applies
1614+
the same filter. Nor does a file below a nested repository get a record,
1615+
because the walk stops applying the outer repository's `.gitignore` and
1616+
`info/exclude` rules there, while the matcher still applies them. Any
1617+
directory entry below the root named `.git` or `.jj` in any ASCII case and of
1618+
any type marks such a repository, as does a listing error, whether or not Git
1619+
rules require a repository. The walk treats a directory as another repository
1620+
only when a lookup of `.git` or `.jj` in it finds an entry, and never with
1621+
`--no-require-git`. Unless the file system equates another name with those
1622+
beyond ASCII case, this skips at least every subtree the walk treats as one. A
1623+
listing can return the marker after other entries, so the records and
1624+
subdirectories already taken from that directory are then released.
16001625
It stats only excluded tracked files that are present, listing each directory
16011626
once; absent files and missing directories cost no failed stat. The charges for
16021627
the rule-file snapshots and parsed patterns the walk read are held until the

‎tgrep-cli/tests/shared_daemon/managed.rs‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1160,6 +1160,79 @@ fn assert_scan_parity(daemon: &Daemon, root: &Path, home: &Path) {
11601160
}
11611161
}
11621162

1163+
#[test]
1164+
fn global_excludes_never_prove_excluded_file_records() {
1165+
let fixture = Fixture::new();
1166+
let home = fixture.temp.path().join("home");
1167+
fs::create_dir(&home).unwrap();
1168+
fs::write(
1169+
home.join(".gitconfig"),
1170+
"[core]\nexcludesFile = ~/global-ignore\n",
1171+
)
1172+
.unwrap();
1173+
fs::write(
1174+
home.join("global-ignore"),
1175+
"/anchored.glob\nsub/interior.glob\n*.globany\n",
1176+
)
1177+
.unwrap();
1178+
fs::write(fixture.a.join(".gitignore"), "*.ctl\n").unwrap();
1179+
fs::create_dir(fixture.a.join("sub")).unwrap();
1180+
for path in [
1181+
"anchored.glob",
1182+
"control.ctl",
1183+
"sub/any.globany",
1184+
"sub/interior.glob",
1185+
] {
1186+
fs::write(fixture.a.join(path), "shared_term excluded\n").unwrap();
1187+
}
1188+
git(&fixture.a, &["add", "--force", "."]);
1189+
git(&fixture.a, &["commit", "-qm", "excluded tracked files"]);
1190+
let revision = git(&fixture.a, &["rev-parse", "HEAD"]);
1191+
// The walk matches global patterns from the daemon's working directory,
1192+
// which it inherits from this process, so it lists the anchored files.
1193+
assert!(!std::env::current_dir().unwrap().starts_with(&fixture.a));
1194+
let policy_path = fixture.storage.join("policy.json");
1195+
fs::write(&policy_path, serde_json::to_vec(&policy()).unwrap()).unwrap();
1196+
let mut daemon = Daemon::start_with_home(
1197+
&fixture.a,
1198+
&fixture.storage,
1199+
&[
1200+
"--shared-policy",
1201+
policy_path.to_str().unwrap(),
1202+
"--no-watch",
1203+
],
1204+
Some(&home),
1205+
);
1206+
let (claim, guard) = owner(&daemon);
1207+
let mut input = attach_input(&fixture, &claim);
1208+
input["request"]["revision"] = json!(revision);
1209+
let operation = daemon.rpc("views.attach", input);
1210+
let view = completed(&daemon, &operation)["result"]["current"].clone();
1211+
verified_attach(&daemon, &operation);
1212+
let result = search(&daemon, &fixture.a).to_string();
1213+
for (name, listed) in [
1214+
("anchored.glob", true),
1215+
("interior.glob", true),
1216+
("any.globany", false),
1217+
("control.ctl", false),
1218+
] {
1219+
assert_eq!(result.contains(name), listed, "{name}: {result}");
1220+
}
1221+
let status = daemon.rpc("views.status", json!({"id":view["id"]}));
1222+
assert_eq!(status["ready"], true);
1223+
// Only `.gitignore` proves an unlisted file excluded: the walk excludes
1224+
// `sub/any.globany` too, but by a global pattern.
1225+
assert_eq!(status["last_reconcile"]["excluded_tracked"], 1, "{status}");
1226+
let detached = daemon.rpc(
1227+
"views.detach",
1228+
json!({"owner":claim.owner,"lease":"managed-client"}),
1229+
);
1230+
assert_eq!(detached["lease_released"], true);
1231+
daemon.rpc("owners.release", json!({"claim":claim}));
1232+
drop(guard);
1233+
stop(&mut daemon);
1234+
}
1235+
11631236
#[test]
11641237
fn bounded_ignore_inputs_preserve_scan_parity_and_leave_sibling_views_serviceable() {
11651238
let fixture = Fixture::new();

‎tgrep-core/src/gitignore.rs‎

Lines changed: 40 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -221,7 +221,21 @@ impl IgnoreMatcher {
221221
}
222222

223223
pub fn is_ignored(&self, path: &Path, is_dir: bool) -> bool {
224-
if self.standard_decision(path, is_dir, true) == Some(true) {
224+
self.ignored(path, is_dir, true)
225+
}
226+
227+
/// Like [`is_ignored`](Self::is_ignored), but the standard rules leave out
228+
/// the repository's `info/exclude` file and the global excludes file. They
229+
/// have the lowest precedence there, and the p4ignore and case filters
230+
/// don't read the standard decision, so leaving them out can only turn an
231+
/// ignored path into one that isn't. When Git ignores case, the case filter
232+
/// still matches `info/exclude`'s rules, but it never hides a tracked file.
233+
pub fn is_ignored_without_exclude_files(&self, path: &Path, is_dir: bool) -> bool {
234+
self.ignored(path, is_dir, false)
235+
}
236+
237+
fn ignored(&self, path: &Path, is_dir: bool, excludes: bool) -> bool {
238+
if self.standard_decision(path, is_dir, true, excludes) == Some(true) {
225239
return true;
226240
}
227241
if self.local_is_filter
@@ -244,10 +258,16 @@ impl IgnoreMatcher {
244258
/// through the walker's hidden filter. Parent-directory matches are not
245259
/// inherited here: the walker makes the hidden decision for each entry.
246260
pub fn is_whitelisted(&self, path: &Path, is_dir: bool) -> bool {
247-
self.standard_decision(path, is_dir, false) == Some(false)
261+
self.standard_decision(path, is_dir, false, true) == Some(false)
248262
}
249263

250-
fn standard_decision(&self, path: &Path, is_dir: bool, parents: bool) -> Option<bool> {
264+
fn standard_decision(
265+
&self,
266+
path: &Path,
267+
is_dir: bool,
268+
parents: bool,
269+
excludes: bool,
270+
) -> Option<bool> {
251271
let matched = |matcher: &Gitignore, path: &Path| {
252272
let result = if parents {
253273
matcher.matched_path_or_any_parents(path, is_dir)
@@ -306,6 +326,7 @@ impl IgnoreMatcher {
306326
}
307327

308328
if standard_decision.is_none()
329+
&& excludes
309330
&& let Some((prefix, matcher)) = &self.repo_exclude
310331
{
311332
let path = if prefix.is_empty() {
@@ -316,7 +337,7 @@ impl IgnoreMatcher {
316337
standard_decision = matched(matcher, Path::new(&path));
317338
}
318339

319-
if standard_decision.is_none() {
340+
if excludes && standard_decision.is_none() {
320341
standard_decision = matched(&self.global, path);
321342
}
322343
standard_decision
@@ -861,13 +882,21 @@ fn git_repo_root(root: &Path) -> Option<&Path> {
861882

862883
/// The `info/exclude` file whose rules apply to `root`, if there is one.
863884
///
864-
/// Not `.git/info/exclude`. In a linked worktree or a submodule `.git` is a
865-
/// file holding a `gitdir:` pointer, and that directory in turn holds a
866-
/// `commondir` naming the repository every worktree shares — which is where the
867-
/// one `info/exclude` lives. `WalkBuilder` resolves that chain, so a matcher
868-
/// that stopped at the literal path enforced different rules than the walk in
869-
/// exactly the layouts where the two differ, and the watcher would index a file
870-
/// the next stale check evicts.
885+
/// Not `.git/info/exclude`. In a linked worktree `.git` is a file holding a
886+
/// `gitdir:` pointer, and that directory in turn holds a `commondir` naming the
887+
/// repository every worktree shares — which is where the one `info/exclude`
888+
/// lives. When Git is required, `WalkBuilder` follows that chain through an
889+
/// absolute pointer, so a matcher that stopped at the literal path enforced
890+
/// different rules than the walk in exactly the layouts where the two differ,
891+
/// and the watcher would index a file the next stale check evicts.
892+
///
893+
/// This function follows Git further than the walk does. It resolves a
894+
/// relative pointer against the directory holding `.git` (`read_git_dir_using`)
895+
/// where the walk resolves it against the process's working directory, and it
896+
/// takes a Git directory without `commondir`, such as a submodule's or a
897+
/// `--separate-git-dir` checkout's, as its own common directory
898+
/// (`common_git_dir`) where the walk applies no `info/exclude`. Excluded-file
899+
/// records therefore never take this file as proof.
871900
pub fn repo_exclude_path(root: &Path) -> Option<PathBuf> {
872901
let path = repo_exclude_candidate(root)?;
873902
path.is_file().then_some(path)

‎tgrep-core/src/worktrees.rs‎

Lines changed: 49 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1442,7 +1442,19 @@ impl WorktreeView {
14421442
/// first candidate is found. Under a permit, both parse the rule-file
14431443
/// contents the walk read; without one, they read the rule files again.
14441444
/// Tracked files the walk skipped only for other reasons, such as excluded
1445-
/// directories, get no record.
1445+
/// directories, get no record. Nor does anything below a possible nested
1446+
/// repository: a directory other than the root with an entry named `.git`
1447+
/// or `.jj`, in any ASCII case and of any type, or whose listing fails
1448+
/// partway. Below a nested repository, the walk stops applying the outer
1449+
/// repository's `.gitignore` and `info/exclude` rules, while the matcher
1450+
/// still applies them. Neither exclude file proves exclusion either, as
1451+
/// the walk can list a file that one excludes from the root: it matches
1452+
/// global excludes' anchored patterns relative to the process's working
1453+
/// directory, and it skips the repository's `info/exclude` behind a `.git`
1454+
/// file whose Git directory, resolved against that working directory, has
1455+
/// no `commondir` file, and behind any `.git` file when Git is not
1456+
/// required. A tracked file that only an exclude file excludes therefore
1457+
/// gets no record, and its notifications keep invalidating.
14461458
///
14471459
/// Tracked paths are matched by exact spelling while listing only the
14481460
/// directories that contain candidates, one listing at a time. A
@@ -1454,7 +1466,8 @@ impl WorktreeView {
14541466
/// Scratch buffers are charged to the permit before they are allocated,
14551467
/// a growing buffer together with the old storage it still holds, and
14561468
/// released when they are dropped. The walk's rules keep their own charge
1457-
/// until they are dropped on return. `memory` grows only by the records.
1469+
/// until they are dropped on return. `memory` grows only by the records,
1470+
/// and shrinks again for those removed below a nested repository.
14581471
fn excluded_records(
14591472
&self,
14601473
walk: &walker::MetaWalkResult,
@@ -1528,19 +1541,33 @@ impl WorktreeView {
15281541
let Ok(entries) = fs::read_dir(&directory) else {
15291542
continue;
15301543
};
1544+
let subdirectories = pending.len();
1545+
let mut nested = false;
15311546
for entry in entries {
15321547
check()?;
15331548
let Ok(entry) = entry else {
1534-
continue;
1549+
if prefix == 0 {
1550+
continue;
1551+
}
1552+
// It could be the entry that marks a nested repository.
1553+
nested = true;
1554+
break;
15351555
};
15361556
let name = entry.file_name();
15371557
let Some(name) = name.to_str() else {
15381558
continue;
15391559
};
1560+
let name = name.as_bytes();
1561+
// Any type, in any ASCII case.
1562+
if prefix != 0
1563+
&& (name.eq_ignore_ascii_case(b".git") || name.eq_ignore_ascii_case(b".jj"))
1564+
{
1565+
nested = true;
1566+
break;
1567+
}
15401568
let Ok(kind) = entry.file_type() else {
15411569
continue;
15421570
};
1543-
let name = name.as_bytes();
15441571
if kind.is_dir() {
15451572
let first = range.partition_point(|path| {
15461573
path.as_bytes()[prefix..].iter().lt(name.iter().chain(b"/"))
@@ -1572,11 +1599,9 @@ impl WorktreeView {
15721599
rules.matcher = self.walk_matcher(walk, rules.inputs.as_ref(), permit)?;
15731600
proven = true;
15741601
}
1575-
if !rules
1576-
.matcher
1577-
.as_ref()
1578-
.is_some_and(|matcher| matcher.is_ignored(Path::new(path), false))
1579-
{
1602+
if !rules.matcher.as_ref().is_some_and(|matcher| {
1603+
matcher.is_ignored_without_exclude_files(Path::new(path), false)
1604+
}) {
15801605
continue;
15811606
}
15821607
let Ok(metadata) = entry.metadata() else {
@@ -1599,6 +1624,21 @@ impl WorktreeView {
15991624
},
16001625
);
16011626
}
1627+
if nested {
1628+
pending.truncate(subdirectories);
1629+
let mut released = 0;
1630+
for path in range {
1631+
if !path.as_bytes()[prefix..].contains(&b'/') && records.remove(*path).is_some()
1632+
{
1633+
released += path.len() as u64 + EXCLUDED_RECORD_BYTES;
1634+
}
1635+
}
1636+
if let Some(memory) = memory.as_deref_mut().filter(|_| released > 0) {
1637+
memory
1638+
.resize(memory.bytes() - released)
1639+
.map_err(crate::Error::from)?;
1640+
}
1641+
}
16021642
}
16031643
Ok(records)
16041644
}

0 commit comments

Comments
 (0)