Skip to content

Commit b9da528

Browse files
shengyfuCopilot
andcommitted
Require ignore rules to prove exclusion before recording
excluded_records recorded every present tracked file that the walk had not listed, as if ignore rules excluded it. A tracked file restored after the walk is unlisted too, so it could get a record. If its create notification was missed, a later Modify that left it unchanged was dropped as unchanged, although the published state did not contain it. A candidate now gets a record only when the ignore rules the walk applied exclude it or a parent directory: - The metadata walk returns the point-query matcher it already builds for visibility, and a controlled walk keeps its input control alive with it. - When the walk built no matcher, one is built from the walk's rule files, only once a candidate is found on disk. - Under a permit, as managed views always walk, both matchers parse the rule-file contents the walk read through its input control, so nothing is read again and a rule changed since the walk proves nothing. A rebuilt matcher leaves out the case-insensitive tracked-file layer, which only adds exclusions. The input control's charge for those snapshots and patterns is now held until the records are built, then released. Tracked files that the walk skipped only for other reasons get no record, so their notifications keep invalidating. Co-authored-by: Copilot App <[email protected]>
1 parent cd26f6a commit b9da528

4 files changed

Lines changed: 191 additions & 26 deletions

File tree

‎SHARED_WORKTREE_INDEXES.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1593,8 +1593,14 @@ duplicate-write churn). For a listed file, that is its trusted read evidence and
15931593
the hidden status the walk observed before whitelists applied: a dot-file that
15941594
an ignore-file whitelist admits is visible to queries but still hidden by name.
15951595
For a regular tracked file that ignore rules exclude, each pass records its
1596-
trusted metadata. It stats only excluded tracked files that are present, listing
1597-
each directory once; absent files and missing directories cost no failed stat.
1596+
trusted metadata. A tracked file the walk did not list gets a record only when
1597+
the ignore rules the walk applied, parsed from the rule-file contents it read,
1598+
exclude the file or a parent directory; a file restored after the walk, or one
1599+
skipped only for another reason, gets none. It stats only excluded tracked files
1600+
that are present, listing each directory once; absent files and missing
1601+
directories cost no failed stat. The charges for the rule-file snapshots and
1602+
parsed patterns the walk read are held until the records are built, then
1603+
released.
15981604
`last_reconcile.excluded_tracked` counts these records, which count toward
15991605
retained view memory, and `last_reconcile.timings.excluded_records_us` reports
16001606
their cost within discovery. Metadata verification refreshes them in place.

‎tgrep-core/src/walker.rs‎

Lines changed: 63 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -499,7 +499,7 @@ pub fn walk_dir_with_ignorecase(
499499

500500
let gitignore_files = gitignore_files.into_inner().unwrap();
501501
let ignore_files = ignore_files.into_inner().unwrap();
502-
let (visibility, _) = finish_visibility(
502+
let (visibility, _, _) = finish_visibility(
503503
&root,
504504
opts.no_ignore,
505505
opts.no_require_git,
@@ -556,8 +556,13 @@ fn finish_visibility(
556556
ignorecase: Option<std::sync::Arc<crate::gitignore::CaseInsensitiveIgnore>>,
557557
read_control: Option<&std::sync::Arc<dyn ignore::gitignore::FileReadControl>>,
558558
permit: Option<&std::sync::Arc<crate::managed::WorkPermit>>,
559-
) -> crate::managed::Result<(crate::visibility::PathVisibility, Vec<String>)> {
559+
) -> crate::managed::Result<(
560+
crate::visibility::PathVisibility,
561+
Vec<String>,
562+
Option<crate::gitignore::IgnoreMatcher>,
563+
)> {
560564
let mut whitelisted = Vec::new();
565+
let mut rules = None;
561566
if !no_ignore && !visibility.is_empty() {
562567
let matcher = crate::gitignore::matcher_from_ignore_paths_controlled(
563568
root,
@@ -568,8 +573,21 @@ fn finish_visibility(
568573
read_control,
569574
);
570575
whitelisted = visibility.apply_ignore_rules_controlled(matcher.as_ref(), permit)?;
576+
rules = matcher;
571577
}
572-
Ok((visibility, whitelisted))
578+
Ok((visibility, whitelisted, rules))
579+
}
580+
581+
/// Ignore rules a metadata walk applied, kept for point queries after it.
582+
///
583+
/// A controlled walk's input control holds the charge for the rules it read
584+
/// and serves the walk's snapshots of them, so building a matcher from it
585+
/// reads nothing from disk. Dropping this releases that charge.
586+
pub(crate) struct WalkRules {
587+
/// The point-query matcher the walk built for visibility, if it needed
588+
/// one. Declared first so it drops before the charge.
589+
pub(crate) matcher: Option<crate::gitignore::IgnoreMatcher>,
590+
pub(crate) inputs: Option<std::sync::Arc<crate::managed::inputs::InputControl>>,
573591
}
574592

575593
/// Build a point-query ignore matcher from `.gitignore` and `.ignore` files
@@ -706,17 +724,36 @@ pub fn walk_file_metadata_with_ignorecase(
706724
opts: &MetaWalkOptions,
707725
ignorecase: Option<std::sync::Arc<crate::gitignore::CaseInsensitiveIgnore>>,
708726
) -> MetaWalkResult {
709-
walk_metadata_inner(root, opts, ignorecase, None, None)
710-
.expect("an uncontrolled metadata walk has no work-budget failure")
711-
.0
727+
walk_file_metadata_with_rules(root, opts, ignorecase).0
728+
}
729+
730+
/// [`walk_file_metadata_with_ignorecase`], also returning the rules it applied.
731+
pub(crate) fn walk_file_metadata_with_rules(
732+
root: &Path,
733+
opts: &MetaWalkOptions,
734+
ignorecase: Option<std::sync::Arc<crate::gitignore::CaseInsensitiveIgnore>>,
735+
) -> (MetaWalkResult, WalkRules) {
736+
let (walk, _, matcher) = walk_metadata_inner(root, opts, ignorecase, None, None)
737+
.expect("an uncontrolled metadata walk has no work-budget failure");
738+
(
739+
walk,
740+
WalkRules {
741+
matcher,
742+
inputs: None,
743+
},
744+
)
712745
}
713746

714747
pub(crate) fn walk_file_metadata_controlled(
715748
root: &Path,
716749
opts: &MetaWalkOptions,
717750
repository: &crate::generations::Repository,
718751
permit: &std::sync::Arc<crate::managed::WorkPermit>,
719-
) -> crate::managed::Result<(MetaWalkResult, crate::managed::work::MemoryCharge)> {
752+
) -> crate::managed::Result<(
753+
MetaWalkResult,
754+
crate::managed::work::MemoryCharge,
755+
WalkRules,
756+
)> {
720757
let inputs = crate::managed::inputs::InputControl::new(permit)?;
721758
let control: std::sync::Arc<dyn ignore::gitignore::FileReadControl> = inputs.clone();
722759
let result = (|| {
@@ -728,23 +765,37 @@ pub(crate) fn walk_file_metadata_controlled(
728765
)?
729766
.map(std::sync::Arc::new)
730767
};
731-
let (walk, charge) =
768+
let (walk, charge, matcher) =
732769
walk_metadata_inner(root, opts, ignorecase, Some(permit), Some(control))?;
733770
Ok((
734771
walk,
735772
charge.expect("controlled walk owns its memory charge"),
773+
matcher,
736774
))
737775
})();
738-
inputs.finish(result)
776+
let (walk, charge, matcher) = inputs.finish(result)?;
777+
Ok((
778+
walk,
779+
charge,
780+
WalkRules {
781+
matcher,
782+
inputs: Some(inputs),
783+
},
784+
))
739785
}
740786

787+
#[allow(clippy::type_complexity)]
741788
fn walk_metadata_inner(
742789
root: &Path,
743790
opts: &MetaWalkOptions,
744791
ignorecase: Option<std::sync::Arc<crate::gitignore::CaseInsensitiveIgnore>>,
745792
permit: Option<&std::sync::Arc<crate::managed::WorkPermit>>,
746793
read_control: Option<std::sync::Arc<dyn ignore::gitignore::FileReadControl>>,
747-
) -> crate::managed::Result<(MetaWalkResult, Option<crate::managed::work::MemoryCharge>)> {
794+
) -> crate::managed::Result<(
795+
MetaWalkResult,
796+
Option<crate::managed::work::MemoryCharge>,
797+
Option<crate::gitignore::IgnoreMatcher>,
798+
)> {
748799
let memory = std::sync::Arc::new(std::sync::Mutex::new(
749800
permit.map(|permit| permit.memory(0)).transpose()?,
750801
));
@@ -995,7 +1046,7 @@ fn walk_metadata_inner(
9951046
let memory = memory.lock().expect("walk memory lock").take();
9961047
let gitignore_files = gitignore_files.into_inner().unwrap();
9971048
let ignore_files = ignore_files.into_inner().unwrap();
998-
let (visibility, whitelisted_hidden) = finish_visibility(
1049+
let (visibility, whitelisted_hidden, matcher) = finish_visibility(
9991050
&root,
10001051
opts.no_ignore,
10011052
opts.no_require_git,
@@ -1022,6 +1073,7 @@ fn walk_metadata_inner(
10221073
skipped_error: skipped_error.into_inner(),
10231074
},
10241075
memory,
1076+
matcher,
10251077
))
10261078
}
10271079

‎tgrep-core/src/worktrees.rs‎

Lines changed: 73 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -789,7 +789,7 @@ impl WorktreeView {
789789
}
790790
let mut options = self.options.walk.clone();
791791
options.max_file_size = None;
792-
let (walk, mut memory) =
792+
let (walk, mut memory, _) =
793793
walker::walk_file_metadata_controlled(&self.root, &options, &repository, permit)
794794
.map_err(crate::Error::from)?;
795795
if walk.skipped_error != 0 {
@@ -1425,6 +1425,16 @@ impl WorktreeView {
14251425
/// them unchanged can be dropped. Ignore and attributes files get no
14261426
/// record: their notifications keep invalidating membership.
14271427
///
1428+
/// Not being listed is no proof of exclusion: a file restored after the
1429+
/// walk is not listed either. A candidate found on disk is recorded only
1430+
/// when the walk's own ignore rules ignore the file or one of its parent
1431+
/// directories. The point-query matcher the walk built for visibility is
1432+
/// reused; otherwise one is built from the walk's rule files when the
1433+
/// first candidate is found. Under a permit, both parse the rule-file
1434+
/// contents the walk read; without one, they read the rule files again.
1435+
/// Tracked files the walk skipped only for other reasons, such as excluded
1436+
/// directories, get no record.
1437+
///
14281438
/// Tracked paths are matched by exact spelling while listing only the
14291439
/// directories that contain candidates, one listing at a time. A
14301440
/// directory that cannot be listed (absent in a sparse or partial
@@ -1434,13 +1444,19 @@ impl WorktreeView {
14341444
///
14351445
/// Scratch buffers are charged to the permit before they are allocated,
14361446
/// a growing buffer together with the old storage it still holds, and
1437-
/// released when they are dropped. `memory` grows only by the records.
1447+
/// released when they are dropped. The walk's rules keep their own charge
1448+
/// until they are dropped on return. `memory` grows only by the records.
14381449
fn excluded_records(
14391450
&self,
1440-
listed: &[String],
1451+
walk: &walker::MetaWalkResult,
1452+
rules: walker::WalkRules,
14411453
permit: Option<&Arc<crate::managed::WorkPermit>>,
14421454
mut memory: Option<&mut crate::managed::work::MemoryCharge>,
14431455
) -> Result<BTreeMap<String, ExcludedFile>> {
1456+
if self.options.walk.no_ignore {
1457+
return Ok(BTreeMap::new());
1458+
}
1459+
let listed = &walk.listed_files;
14441460
let check = || match permit {
14451461
Some(permit) => permit.check().map_err(crate::Error::from),
14461462
None => Ok(()),
@@ -1486,6 +1502,8 @@ impl WorktreeView {
14861502
.map_err(crate::Error::from)?;
14871503
}
14881504
let mut records = BTreeMap::new();
1505+
let mut rules = rules;
1506+
let mut proven = rules.matcher.is_some();
14891507
// (directory prefix length, candidate range); candidates are sorted,
14901508
// so every directory's descendants are contiguous and ordered by
14911509
// their remainder after the prefix.
@@ -1541,6 +1559,17 @@ impl WorktreeView {
15411559
continue;
15421560
};
15431561
let path = range[found];
1562+
if !proven {
1563+
rules.matcher = self.walk_matcher(walk, rules.inputs.as_ref(), permit)?;
1564+
proven = true;
1565+
}
1566+
if !rules
1567+
.matcher
1568+
.as_ref()
1569+
.is_some_and(|matcher| matcher.is_ignored(Path::new(path), false))
1570+
{
1571+
continue;
1572+
}
15441573
let Ok(metadata) = entry.metadata() else {
15451574
continue;
15461575
};
@@ -1565,6 +1594,38 @@ impl WorktreeView {
15651594
Ok(records)
15661595
}
15671596

1597+
/// A point-query matcher for the rules `walk` applied, when the walk did
1598+
/// not build one for visibility. A controlled walk's input control serves
1599+
/// the snapshots of the rule files it read and charges the parse. The
1600+
/// walk's case-insensitive matching is left out: it only adds exclusions,
1601+
/// so without it a candidate can only stay unproven.
1602+
fn walk_matcher(
1603+
&self,
1604+
walk: &walker::MetaWalkResult,
1605+
inputs: Option<&Arc<crate::managed::inputs::InputControl>>,
1606+
permit: Option<&Arc<crate::managed::WorkPermit>>,
1607+
) -> Result<Option<crate::gitignore::IgnoreMatcher>> {
1608+
let control = inputs
1609+
.cloned()
1610+
.map(|inputs| inputs as Arc<dyn ignore::gitignore::FileReadControl>);
1611+
let matcher = crate::gitignore::matcher_from_ignore_paths_controlled(
1612+
&self.root,
1613+
&walk.gitignore_files,
1614+
&walk.ignore_files,
1615+
self.options.walk.no_require_git,
1616+
None,
1617+
control.as_ref(),
1618+
);
1619+
if let Some(inputs) = inputs {
1620+
let checked = match permit {
1621+
Some(permit) => permit.check(),
1622+
None => Ok(()),
1623+
};
1624+
inputs.finish(checked).map_err(crate::Error::from)?;
1625+
}
1626+
Ok(matcher)
1627+
}
1628+
15681629
fn prepare(
15691630
&self,
15701631
full: bool,
@@ -1602,28 +1663,29 @@ impl WorktreeView {
16021663
// Metadata alone must not permanently exclude a file whose actual
16031664
// bytes have become small enough. Bound actual reads instead.
16041665
options.max_file_size = None;
1605-
let (walk, memory) = match permit {
1666+
let (walk, memory, rules) = match permit {
16061667
Some(permit) => {
1607-
let (walk, memory) = walker::walk_file_metadata_controlled(
1668+
let (walk, memory, rules) = walker::walk_file_metadata_controlled(
16081669
&self.root,
16091670
&options,
16101671
&repository,
16111672
permit,
16121673
)
16131674
.map_err(crate::Error::from)?;
1614-
(walk, Some(memory))
1675+
(walk, Some(memory), rules)
1676+
}
1677+
None => {
1678+
let (walk, rules) =
1679+
walker::walk_file_metadata_with_rules(&self.root, &options, ignorecase);
1680+
(walk, None, rules)
16151681
}
1616-
None => (
1617-
walker::walk_file_metadata_with_ignorecase(&self.root, &options, ignorecase),
1618-
None,
1619-
),
16201682
};
16211683
if walk.skipped_error != 0 {
16221684
return Err(WorktreeError::IncompleteWalk(walk.skipped_error));
16231685
}
16241686
let mut memory = memory;
16251687
let records = Instant::now();
1626-
let excluded = self.excluded_records(&walk.listed_files, permit, memory.as_mut())?;
1688+
let excluded = self.excluded_records(&walk, rules, permit, memory.as_mut())?;
16271689
let excluded_records_us = crate::managed::work::micros(records.elapsed());
16281690
after_discovery();
16291691
let mut prepared = Prepared {

‎tgrep-core/src/worktrees/tests.rs‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1773,10 +1773,13 @@ fn excluded_records_charge_scratch_before_allocating_and_release_it() {
17731773
commit(root);
17741774
let view = fixture.view(root, &fixture.generation());
17751775
let (_namespace, permit) = fixture.permit();
1776+
let (walk, rules) = walker::walk_file_metadata_with_rules(&view.root, &view.options.walk, None);
1777+
let mut listed = walk.listed_files.clone();
1778+
listed.sort();
1779+
assert_eq!(listed, [".gitignore", "keep.txt"]);
17761780
let mut charge = permit.memory(1000).unwrap();
1777-
let listed = [".gitignore".to_owned(), "keep.txt".to_owned()];
17781781
let records = view
1779-
.excluded_records(&listed, Some(&permit), Some(&mut charge))
1782+
.excluded_records(&walk, rules, Some(&permit), Some(&mut charge))
17801783
.unwrap();
17811784
assert_eq!(records.len(), 200);
17821785
let record_bytes: u64 = records
@@ -1797,6 +1800,48 @@ fn excluded_records_charge_scratch_before_allocating_and_release_it() {
17971800
);
17981801
}
17991802

1803+
#[test]
1804+
fn excluded_records_require_the_walks_rules_to_exclude_a_file() {
1805+
let fixture = Fixture::new();
1806+
let root = &fixture.root;
1807+
write(root, ".gitignore", b"ign/\n");
1808+
write(root, "restored.txt", b"needle\n");
1809+
write(root, "ign/tracked.log", b"needle\n");
1810+
git(root, &["add", "--force", "ign"]);
1811+
commit(root);
1812+
fs::remove_file(root.join("restored.txt")).unwrap();
1813+
let view = fixture.view(root, &fixture.generation());
1814+
let (_namespace, permit) = fixture.permit();
1815+
let mut options = view.options.walk.clone();
1816+
options.max_file_size = None;
1817+
// With the matcher the walk built for visibility, then with one rebuilt
1818+
// from the snapshots its input control kept.
1819+
for rebuild in [false, true] {
1820+
let (walk, mut charge, mut rules) =
1821+
walker::walk_file_metadata_controlled(&view.root, &options, &view.repository, &permit)
1822+
.unwrap();
1823+
assert!(rules.matcher.is_some());
1824+
assert!(!walk.listed_files.iter().any(|path| path == "restored.txt"));
1825+
if rebuild {
1826+
rules.matcher = None;
1827+
}
1828+
// A tracked file restored after the walk is unlisted, but the rules
1829+
// the walk applied don't exclude it, so its notifications must keep
1830+
// invalidating. A rule added since is not one the walk applied.
1831+
write(root, "restored.txt", b"needle\n");
1832+
write(root, ".gitignore", b"ign/\nrestored.txt\n");
1833+
let records = view
1834+
.excluded_records(&walk, rules, Some(&permit), Some(&mut charge))
1835+
.unwrap();
1836+
assert_eq!(records.keys().collect::<Vec<_>>(), ["ign/tracked.log"]);
1837+
// The walk's rules were released on return.
1838+
drop(charge);
1839+
drop(permit.memory(permit.private_limit()).unwrap());
1840+
fs::remove_file(root.join("restored.txt")).unwrap();
1841+
write(root, ".gitignore", b"ign/\n");
1842+
}
1843+
}
1844+
18001845
#[test]
18011846
fn read_and_discovery_errors_close_the_gate_and_force_full_retry() {
18021847
let fixture = Fixture::new();

0 commit comments

Comments
 (0)