Skip to content

Commit 82b88a1

Browse files
shengyfuCopilot
andcommitted
Stop a failed read from hiding a file for good
The reconcile pass skips files that failed to read last time and have not changed since, so one permanently locked file does not make every hourly reconcile rebuild the index. But the stamps published alongside that delta were still built from the whole walk, including the skipped files. A stamp means "indexed at this mtime and size", so a file nothing had read was marked up-to-date: the next reconcile classified it as unchanged and skipped it for a reason the memo could no longer clear, and because the stamp lands in filestamps.json a restart did not help either. Every automatic caller of the stale check passes compare_index_membership = false, which is exactly the check that would have noticed the file was missing from the index. The escape hatch and the memo now agree: a file kept out of the delta is kept out of the stamps too, so it keeps looking new until it changes. Pulling both halves into drop_memoized_failures and stamps_for_indexed lets the test call the real code rather than a copy of it, which is why the old test passed while the bug was live. Also two smaller things from the same review: The case-insensitive ignore pass ignored --no-ignore-vcs, --no-ignore-exclude and --no-ignore-parent. It is only sound as a narrowing of the case-sensitive pass, and those flags widen that pass to let everything through - which left this matcher as the only thing excluding, so the flags excluded more than passing nothing at all. The git index parser reserved capacity straight from the 32-bit header count before checking it against the file length, so a twelve-byte file claiming four billion entries asked the allocator for ~146GB and aborted the process in a way no caller can catch. Clamped to what the file could actually hold. Co-authored-by: Copilot App <[email protected]>
1 parent ebaf986 commit 82b88a1

4 files changed

Lines changed: 285 additions & 56 deletions

File tree

‎tgrep-cli/src/serve.rs‎

Lines changed: 133 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2255,13 +2255,70 @@ fn stream_merge_stale_changes(
22552255
}
22562256
}
22572257

2258+
/// Drop the candidates that failed to read last time and have not changed since.
2259+
///
2260+
/// Returns the paths removed, which the caller must also keep out of the
2261+
/// published stamps — see [`stamps_for_indexed`].
2262+
fn drop_memoized_failures(
2263+
memo: &std::collections::HashMap<String, tgrep_core::meta::FileStamp>,
2264+
current_meta: &[tgrep_core::walker::FileMeta],
2265+
changed: &mut Vec<String>,
2266+
added: &mut Vec<String>,
2267+
) -> std::collections::HashSet<String> {
2268+
if memo.is_empty() {
2269+
return std::collections::HashSet::new();
2270+
}
2271+
// One pass over the walk to pick out the memoized paths, rather than a scan
2272+
// per candidate.
2273+
let still_failing: std::collections::HashSet<String> = current_meta
2274+
.iter()
2275+
.filter(|fm| {
2276+
memo.get(&fm.relative_path)
2277+
.is_some_and(|a| a.mtime == fm.mtime && a.size == fm.size)
2278+
})
2279+
.map(|fm| fm.relative_path.clone())
2280+
.collect();
2281+
changed.retain(|path| !still_failing.contains(path));
2282+
added.retain(|path| !still_failing.contains(path));
2283+
still_failing
2284+
}
2285+
2286+
/// The stamps to publish for a walk, minus the files that were never built.
2287+
///
2288+
/// A published stamp means "indexed at this version". Stamping a file the delta
2289+
/// deliberately skipped would make every later reconcile see it as unchanged,
2290+
/// so it would never be indexed again — and because the stamp lands in
2291+
/// `filestamps.json`, not even a restart would recover it: every automatic
2292+
/// caller of the stale check passes `compare_index_membership = false`, which
2293+
/// is precisely the check that would have noticed the file is missing. Leaving
2294+
/// it unstamped keeps it looking new, which is what makes the retry-on-change
2295+
/// behaviour in [`drop_memoized_failures`] work at all.
2296+
fn stamps_for_indexed(
2297+
current_meta: &[tgrep_core::walker::FileMeta],
2298+
skipped: &std::collections::HashSet<String>,
2299+
) -> std::collections::HashMap<String, tgrep_core::meta::FileStamp> {
2300+
current_meta
2301+
.iter()
2302+
.filter(|fm| !skipped.contains(&fm.relative_path))
2303+
.map(|fm| {
2304+
(
2305+
fm.relative_path.clone(),
2306+
tgrep_core::meta::FileStamp {
2307+
mtime: fm.mtime,
2308+
size: fm.size,
2309+
},
2310+
)
2311+
})
2312+
.collect()
2313+
}
2314+
22582315
fn background_refresh_stale(
22592316
state: &Arc<ServerState>,
22602317
root: &Path,
22612318
index_dir: &Path,
22622319
compare_index_membership: bool,
22632320
) -> bool {
2264-
use tgrep_core::meta::{self, FileStamp};
2321+
use tgrep_core::meta;
22652322
use tgrep_core::walker;
22662323

22672324
let _refresh = state.stale_refresh_lock.lock().unwrap();
@@ -2354,29 +2411,13 @@ fn background_refresh_stale(
23542411
// a file that is gone needs no read to evict.
23552412
let skipped_unreadable = {
23562413
let memo = state.unreadable.read().unwrap();
2357-
if memo.is_empty() {
2358-
0
2359-
} else {
2360-
// One pass over the walk to pick out the memoized paths, rather
2361-
// than a scan per candidate.
2362-
let still_failing: std::collections::HashSet<&str> = current_meta
2363-
.iter()
2364-
.filter(|fm| {
2365-
memo.get(&fm.relative_path)
2366-
.is_some_and(|a| a.mtime == fm.mtime && a.size == fm.size)
2367-
})
2368-
.map(|fm| fm.relative_path.as_str())
2369-
.collect();
2370-
let before = changed.len() + added.len();
2371-
changed.retain(|path| !still_failing.contains(path.as_str()));
2372-
added.retain(|path| !still_failing.contains(path.as_str()));
2373-
before - (changed.len() + added.len())
2374-
}
2414+
drop_memoized_failures(&memo, current_meta, &mut changed, &mut added)
23752415
};
2376-
if skipped_unreadable > 0 {
2416+
if !skipped_unreadable.is_empty() {
23772417
eprintln!(
2378-
"[trace] stale check: {skipped_unreadable} file(s) unchanged since they \
2379-
last failed to read; not retrying them"
2418+
"[trace] stale check: {} file(s) unchanged since they last failed to \
2419+
read; not retrying them",
2420+
skipped_unreadable.len()
23802421
);
23812422
}
23822423

@@ -2406,18 +2447,7 @@ fn background_refresh_stale(
24062447
);
24072448
}
24082449

2409-
let new_stamps: std::collections::HashMap<String, FileStamp> = current_meta
2410-
.iter()
2411-
.map(|fm| {
2412-
(
2413-
fm.relative_path.clone(),
2414-
FileStamp {
2415-
mtime: fm.mtime,
2416-
size: fm.size,
2417-
},
2418-
)
2419-
})
2420-
.collect();
2450+
let new_stamps = stamps_for_indexed(current_meta, &skipped_unreadable);
24212451

24222452
if !stream_merge_stale_changes(
24232453
state,
@@ -3765,26 +3795,80 @@ mod tests {
37653795
&std::collections::HashSet::new(),
37663796
false,
37673797
);
3768-
// Mirrors the filter `background_refresh_stale` applies.
3769-
let still_failing: std::collections::HashSet<&str> = current
3770-
.iter()
3771-
.filter(|fm| {
3772-
memo.get(&fm.relative_path)
3773-
.is_some_and(|a| a.mtime == fm.mtime && a.size == fm.size)
3774-
})
3775-
.map(|fm| fm.relative_path.as_str())
3776-
.collect();
3777-
changed.retain(|path| !still_failing.contains(path.as_str()));
3778-
added.retain(|path| !still_failing.contains(path.as_str()));
3779-
changed.len() + added.len()
3798+
let skipped = drop_memoized_failures(&memo, &current, &mut changed, &mut added);
3799+
(changed.len() + added.len(), skipped)
37803800
};
37813801

37823802
// Unchanged since the failed read: leave it alone.
3783-
assert_eq!(retried(100, 5), 0);
3803+
assert_eq!(retried(100, 5).0, 0);
37843804
// Touched: worth another look.
3785-
assert_eq!(retried(200, 5), 1);
3805+
assert_eq!(retried(200, 5).0, 1);
37863806
// Resized: likewise.
3787-
assert_eq!(retried(100, 6), 1);
3807+
assert_eq!(retried(100, 6).0, 1);
3808+
}
3809+
3810+
/// Skipping a file must not also mark it indexed.
3811+
///
3812+
/// The two halves have to agree. `drop_memoized_failures` keeps a file out
3813+
/// of the delta, so nothing reads it and nothing writes postings for it; if
3814+
/// the stamps published alongside that delta still claimed it was indexed at
3815+
/// its current mtime and size, the next reconcile would classify it as
3816+
/// unchanged and skip it for a completely different reason — one that never
3817+
/// clears, because the memo is not involved and the stamp outlives the
3818+
/// process in `filestamps.json`. A file that failed to read once would be
3819+
/// invisible to search forever, with no error and no way back short of
3820+
/// deleting the index.
3821+
#[test]
3822+
fn a_skipped_file_is_left_unstamped_so_the_next_pass_still_sees_it() {
3823+
use tgrep_core::meta::FileStamp;
3824+
use tgrep_core::walker::FileMeta;
3825+
3826+
let current = vec![
3827+
FileMeta {
3828+
relative_path: "locked.bin".to_string(),
3829+
mtime: 100,
3830+
size: 5,
3831+
},
3832+
FileMeta {
3833+
relative_path: "src/main.rs".to_string(),
3834+
mtime: 100,
3835+
size: 9,
3836+
},
3837+
];
3838+
let memo = std::collections::HashMap::from([(
3839+
"locked.bin".to_string(),
3840+
FileStamp {
3841+
mtime: 100,
3842+
size: 5,
3843+
},
3844+
)]);
3845+
3846+
let (mut changed, mut added, _) = classify_file_changes(
3847+
&current,
3848+
&std::collections::HashMap::new(),
3849+
&std::collections::HashSet::new(),
3850+
false,
3851+
);
3852+
let skipped = drop_memoized_failures(&memo, &current, &mut changed, &mut added);
3853+
assert!(skipped.contains("locked.bin"));
3854+
3855+
let stamps = stamps_for_indexed(&current, &skipped);
3856+
assert!(
3857+
!stamps.contains_key("locked.bin"),
3858+
"published a stamp for a file that was never read: {stamps:?}"
3859+
);
3860+
assert!(stamps.contains_key("src/main.rs"), "{stamps:?}");
3861+
3862+
// And with that stamp withheld, a later pass over an unchanged tree
3863+
// still offers the file up rather than treating it as up-to-date.
3864+
let (changed, added, _) = classify_file_changes(
3865+
&current,
3866+
&stamps,
3867+
&std::collections::HashSet::new(),
3868+
false,
3869+
);
3870+
assert_eq!(changed.len() + added.len(), 1);
3871+
assert!(added.contains(&"locked.bin".to_string()));
37883872
}
37893873

37903874
/// A walk error must not leave the watcher gated forever.

‎tgrep-core/src/git_index.rs‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,12 @@ fn parse_index(bytes: &[u8]) -> Option<TrackedFiles> {
169169
}
170170
let count = u32::from_be_bytes(bytes[8..12].try_into().ok()?) as usize;
171171

172-
let mut paths = HashSet::with_capacity(count);
172+
// The header is attacker-controlled — reserving `count` outright lets a
173+
// 12-byte file claim 4 billion entries and abort the process in the
174+
// allocator, which is not catchable. No entry can be shorter than its
175+
// 62-byte header plus a NUL and padding, so the file length is a hard
176+
// ceiling on how many there really are.
177+
let mut paths = HashSet::with_capacity(count.min((bytes.len() - 12) / (ENTRY_HEADER + 1)));
173178
let mut pos = 12;
174179
let mut previous: Vec<u8> = Vec::new();
175180
for _ in 0..count {
@@ -362,6 +367,19 @@ mod tests {
362367
assert!(parse_index(&index).is_none());
363368
}
364369

370+
#[test]
371+
fn a_header_claiming_four_billion_entries_does_not_reserve_for_them() {
372+
// A bare header claiming u32::MAX entries. Reserving that many would ask
373+
// the allocator for roughly 146GB and abort the process before any of the
374+
// per-entry length checks below ever run, so the clamp has to happen at
375+
// the reservation itself. Reaching the `None` at all proves it did.
376+
let mut index = Vec::from(*b"DIRC");
377+
index.extend_from_slice(&2u32.to_be_bytes());
378+
index.extend_from_slice(&u32::MAX.to_be_bytes());
379+
assert_eq!(index.len(), 12);
380+
assert!(parse_index(&index).is_none());
381+
}
382+
365383
#[test]
366384
fn reads_a_version_4_index_with_prefix_compressed_paths() {
367385
// Written by hand because v4 entries are only decodable in order, so a

‎tgrep-core/src/gitignore.rs‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -422,18 +422,47 @@ pub struct CaseInsensitiveIgnore {
422422
impl CaseInsensitiveIgnore {
423423
/// Returns `None` unless `root` is in a git repository that sets
424424
/// `core.ignorecase` and has repository-wide rules to apply.
425-
pub fn new(root: &Path) -> Option<Self> {
425+
///
426+
/// `use_gitignore`, `use_exclude` and `use_parents` mirror the flags the
427+
/// case-sensitive pass was built with. They are not a convenience: this
428+
/// matcher is only sound as a *narrowing* of that pass, so it must never
429+
/// consult a source the case-sensitive pass was told to ignore. With
430+
/// `--no-ignore-vcs` the case-sensitive pass lets everything through, which
431+
/// would leave this the only thing excluding — the exact opposite of what
432+
/// the flag asks for.
433+
pub fn new(
434+
root: &Path,
435+
use_gitignore: bool,
436+
use_exclude: bool,
437+
use_parents: bool,
438+
) -> Option<Self> {
426439
use ignore::gitignore::GitignoreBuilder;
427440

441+
if !use_gitignore && !use_exclude {
442+
return None;
443+
}
444+
428445
let repo_root = git_repo_root(root)?.to_path_buf();
429446
if !crate::git_index::ignores_case(&repo_root) {
430447
return None;
431448
}
432449

450+
// Everything below is repository-wide, so when the walk starts inside a
451+
// subdirectory these rules reach it only as parent rules. `--no-ignore-parent`
452+
// switches those off in the case-sensitive pass, and we must follow.
453+
// `git_repo_root` returns an ancestor of `root`, so this compares exactly.
454+
if !use_parents && root != repo_root {
455+
return None;
456+
}
457+
433458
let mut builder = GitignoreBuilder::new(&repo_root);
434459
builder.case_insensitive(true).ok()?;
435-
let _ = builder.add(repo_root.join(GITIGNORE_FILENAME));
436-
let _ = builder.add(repo_root.join(".git").join("info").join("exclude"));
460+
if use_gitignore {
461+
let _ = builder.add(repo_root.join(GITIGNORE_FILENAME));
462+
}
463+
if use_exclude {
464+
let _ = builder.add(repo_root.join(".git").join("info").join("exclude"));
465+
}
437466
let matcher = builder.build().ok()?;
438467
if matcher.is_empty() {
439468
return None;

0 commit comments

Comments
 (0)