Prioritize managed work, bound WAL checkpoints, cut scoped query cost; bump to 1.2.5 by shengyfu · Pull Request #217 · microsoft/tgrep

Shengyu Fu (shengyfu) · 2026-10-26T08:18:46Z

Addresses #212, #214 and #216, and bumps the workspace version to 1.2.5.

The issues were found running v1.2.4 managed shared views on a 293k-file Windows monorepo with linked worktrees. The large-checkout numbers below come from a public 184,065-file dotnet checkout.

Commits

6bc57fd: the scheduler, WAL and scoped-query changes.
b00775d: an edit made during a content-verification pass no longer forces a second full pass.
7951103: version bump to 1.2.5 (Cargo.toml, both lockfiles, and the release example in CONTRIBUTING.md).

#212: storage pressure starves reconcile and verification

Root cause

Managed jobs ran first in, first out, so background work could hold the only work slot ahead of catch-up reconciliation.
Unproductive collection and recovery continuations retried without proper backoff.
Long verification jobs shared the global rayon pool with search, so search jobs waited behind them.

Changes

The job queue is bounded and ages work in three classes: foreground, maintenance and background, with maximum offsets of 0 s, 5 s and 10 min. Each offset is also capped at one quarter of the operation's remaining lifetime when enqueued, accounting for time since durable acceptance. A fresh 120-second operation therefore yields to newer foreground jobs for at most 30 seconds. Selection considers shorter-lived jobs within each class; running work and older backlogs can still exhaust a deadline. Queued work is promoted, keeping its original queue time, when its view loses readiness.
Unproductive continuations back off exponentially.
Status reports queued and running jobs, collection activity and per-view work.
Each search window runs on a dedicated tgrep-search-N pool, while the outer query loop stays on the query thread. Submitting whole queries to the pool instead could run one query nested inside another's window.

Results

Stall probe: the worst content-search server time in the tail window fell from 11,364 ms to 76 ms.
A two-query concurrency probe showed no nesting stalls.

#214: the catalog WAL grows to `metadata_bytes/4`

Root cause: the checkpoint threshold was metadata_bytes/4, which is 1 GiB with a 4 GiB budget.

Change

The threshold is now min(metadata_bytes/4, 64 MiB). Synchronization and crash-credit ordering are unchanged.
Collection docs and tuning guidance were added.

Tests: cap computation, a writer checkpointing a large-budget WAL at the cap, contention ordering, and storage admission reclaiming the WAL before refusing.

#216: scoped queries, idle verification and readiness

Root cause

Scoped searches and --files listings evaluated candidates outside their scope.
The client ran Git to discover the daemon.
Each batch open re-verified the root directory twice per file, and concurrent opens of the same directory contend in the kernel.
Every idle verification reread all content.
Readiness was closed even when nothing differed.

Changes

Candidates are restricted to the search scope, --files walks only the requested prefix, and the daemon applies glob and type filters.
The managed v2 client discovers the daemon without running Git.
Files are opened in batches through pinned handles, with an adaptive group size of clamp(ceil(N / min(threads, 6)), 8, 64), where threads is the query-pool width. A failed batch finish re-verifies the root.
Verification has two tiers:
- a metadata pass after at least one poll interval;
- a content pass after at least 15 intervals;
- each tier also waits at least 19× its last cost.
Unchanged metadata carries forward, invalidation can be tree-only, and hints are retained and coarsened. Readiness stays open when verification finds nothing.
Remote-tracking and prefetch ref updates are ignored as Git metadata noise.
b00775d: when a verification on an eventual view finds a difference while a newer hint is queued, the verified state is published without restoring readiness, and only the newer hints remain for the follow-up pass. The receipt reports verification: "superseded" and changed: true. Strict (v1) views keep their existing full retry.

Results

Release builds before and after the batch-open change, on a 240-file fixture. Output was identical to rg, and no Git processes were started.

Query	Server p50	Wall p50
scoped	14.3 → 5.8 ms	51.9 → 43.3 ms
one_dir	16.9 → 8.8 ms	66.8 → 58.8 ms
three_dirs	51.3 → 26.2 ms	153.6 → 128.0 ms
whole	39.0 → 19.9 ms	90.0 → 70.6 ms

On the 184k-file checkout:

Metadata pass: 0 files read in 6–7 s, readiness kept.
Content pass: about 390 s, readiness kept.
Query wall p50: 52 ms idle and 99 ms during a pass, with 0 fallbacks.
600 s idle (measured): 0 content passes and 4 metadata passes that read no files. The daemon used 0.26 cores and 0.93 GiB/h. This window excludes content-verification cost and is not a steady-state measurement.
Long-run idle (estimated): including scheduled content passes, about 0.30 cores and 2.1 GiB/h, versus the measured v1.2.4 figures of 1.43 cores and 47.48 GiB/h. This estimate includes one content pass roughly every two hours, at about 346 CPU-s and 2.6 GiB per pass; it is not a measured long-run result.
Edit 3 s into a forced content pass:

	Before (`6bc57fd`)	After (`b00775d`)
Content passes	2	1, plus a hinted pass that read 1 file
Bytes read	5.64 GB	2.84 GB
CPU	739 s	301 s
Edit searchable after the write	797 s	71 s

This run read about 5× faster than the baseline. At the baseline's read rate, expect the edit to be searchable after about 390 s and the operation to finish in about 400 s, so compare bytes, CPU and pass count.

Validation

cargo fmt --all -- --check, git diff --check, and cargo clippy --locked --workspace --all-targets --features tgrep-cli/managed-test-hooks -- -D warnings.
cargo test --locked --workspace --features tgrep-cli/managed-test-hooks -- --test-threads=2 on 6bc57fd: 0 failures.
On b00775d:
- core lib: 505 passed
- managed_lifecycle: 39 passed
- --bin tgrep: 384 passed, 2 ignored
- shared_daemon: 112 passed, 4 ignored
- MSRV 1.88 check: passed
- release build: passed
The version bump was checked with cargo metadata --locked against both the root and fuzz lockfiles.

Known limits

A running operation can't be preempted, so an edit can wait for the rest of a content pass.
Storage pressure doesn't create headroom.
The WAL threshold triggers a checkpoint; it isn't a hard cap, because readers can pin frames. Collection write amplification is unchanged.
Hinted passes still walk membership.
Each path argument is still a separate search.
Native-watch overflow isn't lossless. Adaptive delays aren't a freshness SLA; use --no-index when current disk bytes are required.
A superseded receipt's reconcile block covers only the hinted follow-up pass. The verification's own reads appear in job counters and admitted_us.
Idle cost not yet attributed: each pass starts about 10 processes and writes 18–20 MB, and about 0.15 cores stays busy between passes.

Addresses #212, #214 and #216. #212: scheduling and starvation - Replace the managed daemon's FIFO job dispatch with a bounded, aged job queue with foreground, maintenance (collection) and background (serving verification, automatic advancement) classes. A free worker runs the job whose queue time plus class delay (0 s, 5 s, 10 min) is earliest, so catch-up reconciliation no longer waits behind background work and no class starves. When a view loses readiness, its queued work moves to the foreground class and keeps its queue time. - Back off exponentially when recovery or collection continuations make no progress, instead of counting verification-only activity as productive. - Report queued and running jobs, collection activity and each view's queued work in status, distinguishing accepted, queued and running work. - Run each parallel query window on a dedicated rayon pool (tgrep-search-N), so long reconciliation jobs on the global pool no longer stall searches. #214: WAL growth - Trigger the pre-mutation catalog checkpoint at min(metadata_bytes / 4, 64 MiB). Only the trigger changes; the checkpoint itself, synchronization and crash-credit ordering are unchanged. The trigger does not cap WAL size while readers pin frames. - Document resumable collection verification and how to tune collection budgets safely. #216: scoped queries, verification cost and readiness churn - Restrict indexed candidates to the queried scope before processing paths, check overlay precedence once per selected file, and walk only the matching prefix of the ordered membership set for shared --files listings. - Filter --files listings by glob and type in the daemon. - Let managed v2 clients derive their attachment from the gitfile, commondir and markers without running Git. The derived identity must match the registration bound to the owner-only credential; any mismatch falls back to Git discovery. Discover once per worktree per invocation and revalidate further paths with lookup. The managed accept loop blocks in accept() instead of polling every 10 ms; a waker connection unblocks it at shutdown. - Open sibling candidates through the group's pinned handles, and size sibling groups from the sibling count and query threads, capped at 64 files, so small directories spread across query threads. Containment is still verified when each batch finishes and against the query's final snapshot. - Carry trusted unchanged metadata through reconciliation without content opens or parent verification batches, including for untracked overlay files. Use tree-only invalidation where possible, keep the original hints when a pass fails, coarsen hints that overflow their bound to ancestor directories, and keep the evidence of an unchanged verification that a newer event superseded. An explicit refresh still rereads every file. - Pace native verification in two tiers: metadata checks after at least one poll interval and content rereads after at least 15, each also waiting 19 times its previous admitted cost. Status reports the tier, the delays and metadata reuse, plus tree and Git event counters. - Ignore modifications of the Git metadata root and remote-tracking or prefetch ref updates. Index, HEAD, local branch and directory changes keep tree checks instead of forcing full rereads. Co-authored-by: Copilot App <[email protected]>

A serving verification that found a difference while a newer invalidation arrived returned ChangedDuringReconcile, and its stale guard then forced a full reread. On an eventual view, an edit made during a content pass therefore cost a second full pass: on a 184,065-file dotnet checkout the operation read 5.6 GB in 802.6 s, and the edit stayed unsearchable for 797 s after the write. An eventual (stale_reads) view now publishes the verified state at a fresh epoch without restoring readiness and returns Verification::PublishedSuperseded. A verification starts only with an empty queue, so everything still queued is newer than the verified state; the same operation's ordinary hinted refresh then processes only those hints. The receipt reports verification "superseded" with changed: true. Strict (v1) views still return ChangedDuringReconcile, and a failed publication still fails closed. Tests: - eventual_verification_publishes_a_difference_and_leaves_only_newer_hints (core): the difference is published and searchable while the view stays unready but serving with one pending path; the follow-up reads one file without a full pass and restores readiness. - superseded_native_content_verification_publishes_edits_without_a_second_full_pass (shared_daemon): the completed receipt reports superseded and changed, full=false, files_read=1, metadata_reused=2; the view serves throughout and the content timer advances. On the same checkout, a forced content pass with an edit 3 s in is now one operation with outcome superseded: one content pass plus a 1-read hinted pass, 2.84 GB of job reads (was 5.64 GB) and 301 CPU-s (was 739). The edit was searchable at the first publish, before the hinted pass; serving held for all 122 polls, and the next content delay is the 1,800 s floor. Co-authored-by: Copilot App <[email protected]>

Co-authored-by: Copilot App <[email protected]>

Copilot

🟡 Changes recommended

Failed content-verification attempts can bypass cost-based retry backoff and repeatedly occupy the managed worker.

1 open issue

Apply retry backoff after failed or cancelled content attempts · New

What changed in this PR

This PR improves managed shared-view responsiveness and reduces background I/O for large checkouts, addressing #212, #214 and #216. It also prepares release 1.2.5.

Changes:

Prioritizes managed work, isolates parallel searches, and expands scheduling diagnostics.
Caps the WAL checkpoint trigger at 64 MiB and introduces metadata/content verification tiers with retained hints.
Reduces scoped-query and discovery costs, adds regression coverage, and updates documentation and release versions.

File	Description
tgrep-core/tests/snapshot_consistency.rs	Tests scoped-query parity across base and overlay data.
tgrep-core/tests/managed_lifecycle.rs	Tests explicit full refresh with pending hints.
tgrep-core/tests/generations.rs	Tests Git-free repository identity calculation.
tgrep-core/src/worktrees/tests.rs	Tests scoped queries, batching, verification, and hint retention.
tgrep-core/src/worktrees.rs	Adds metadata verification, tree invalidation, and superseded publication.
tgrep-core/src/walker.rs	Records metadata versions for listed binary-extension files.
tgrep-core/src/rooted/tests.rs	Tests configurable sibling batch sizes.
tgrep-core/src/rooted.rs	Supports sized sibling batches and retained root access.
tgrep-core/src/reader.rs	Adds prefix-based file ID lookup.
tgrep-core/src/managed/work.rs	Updates test comments describing the checkpoint threshold.
tgrep-core/src/managed/views.rs	Integrates verification tiers and explicit content refresh.
tgrep-core/src/managed/catalog.rs	Updates WAL checkpoint documentation.
tgrep-core/src/managed/catalog_io.rs	Caps the checkpoint trigger and tests large-budget behavior.
tgrep-core/src/hybrid.rs	Restricts query evaluation to scoped candidates.
tgrep-core/src/git_index.rs	Clarifies Git-free discovery authority requirements.
tgrep-core/src/generations/mod.rs	Extracts repository identity calculation.
tgrep-cli/tests/shared_daemon/managed.rs	Tests filtered listings and Git-free managed discovery.
tgrep-cli/tests/shared_daemon/managed_watch.rs	Updates expectations for metadata-only tree checks.
tgrep-cli/tests/shared_daemon/managed_live.rs	Tests verification tiers and superseded content passes.
tgrep-cli/tests/shared_daemon.rs	Tests legacy discovery reuse and filtered listings.
tgrep-cli/src/serve/shared/server.rs	Adds dedicated search workers, adaptive batching, and listing filters.
tgrep-cli/src/serve/shared/protocol.rs	Caches discovery and locates managed attachments without Git.
tgrep-cli/src/serve/shared/protocol_v2.rs	Supports native discovery and filtered-listing capability negotiation.
tgrep-cli/src/serve/shared/managed.rs	Adds prioritized scheduling, verification pacing, and diagnostics.
tgrep-cli/src/serve.rs	Shares path-filter parsing between searches and listings.
tgrep-cli/src/search.rs	Sends listing filters to compatible shared daemons.
SHARED_WORKTREE_INDEXES.md	Documents scheduling, verification, batching, and collection tuning.
README.md	Updates shared-query behavior and freshness guidance.
fuzz/Cargo.lock	Bumps the core package version to 1.2.5.
CONTRIBUTING.md	Updates release examples to 1.2.5.
Cargo.toml	Bumps the workspace version to 1.2.5.
Cargo.lock	Updates first-party package versions to 1.2.5.
AGENTS.md	Updates managed verification and freshness guidance.

_{Review effort: Balanced}

Track terminal content attempts separately from successful rereads so failures and cancellations restart retry backoff without resetting content freshness. Preserve metadata checks and the successful-pass cadence, expose attempt age and retry delay, and cover the behavior with scheduler-boundary and daemon fault-injection regressions. Co-authored-by: Copilot App <[email protected]>

Copilot

🔵 Needs a closer look

Cross-platform validation, concurrent publication, and scheduling changes warrant human review, and the Unix metadata-verification test expectation needs correction.

1 open issue

Fix assertion for Unix ctime changes after metadata repair · New

1 resolved since last review

Apply retry backoff after failed or cancelled content attempts

_{Review effort: Balanced}

Account for Unix ctime changes after size-preserving writes that restore mtime. Assert metadata repair when the complete version changes, content-tier repair otherwise, and the corresponding short-circuit hint-lookup count. Co-authored-by: Copilot App <[email protected]>

Copilot

🔵 Needs a closer look

Concurrent publication, durable checkpoints, and rooted reads create restart-ordering and filesystem identity risks, particularly with Windows handles, that require human review beyond static inspection.

No open issues 🎉

1 resolved since last review

Fix assertion for Unix ctime changes after metadata repair

_{Review effort: Balanced}

Do not mistake an earlier metadata result for a content retry while the new operation is queued or just starting. Wait for an empty work queue and the expected tier as well as the terminal outcome. Co-authored-by: Copilot App <[email protected]>

Copilot

🔵 Needs a closer look

Human assessment is needed because the 600-second background offset can exceed operation deadlines, causing verification to repeatedly expire in the queue without foreground promotion.

1 open issue

The 600-second background offset exceeds valid operation lifetimes, including the documented… · New

_{Review effort: Balanced}

Cap class offsets at one quarter of remaining operation lifetime, sharing the durable acceptance clock with work admission. Consider shorter-lived jobs within each class and preserve queue age during promotion. Co-authored-by: Copilot App <[email protected]>

Copilot

🟡 Changes recommended

Failed or cancelled polling passes omit terminal attempt times and costs, allowing retained work to retry without cost-based backoff and repeatedly occupy the worker.

1 open issue

Apply cost-based backoff to failed polling retries · New

1 resolved since last review

The 600-second background offset exceeds valid operation lifetimes, including the documented…

_{Review effort: Balanced}

Keep the live-reader protection assertions immediate, but bound release checks while concurrent subprocesses may still hold inherited descriptors before exec. Co-authored-by: Copilot App <[email protected]>

Record polling failures and cancellations, gate retained work by its retry delay, and allow newer input epochs to request prompt repair. Keep running receipts pending until retry accounting completes and cover both configured polling and native fallback. Co-authored-by: Copilot App <[email protected]>

Copilot

🔵 Needs a closer look

Human assessment is needed because hint coarsening can discard hints that fit the configured capacity, unnecessarily triggering a full checkout reread.

1 open issue

This can force a full content pass even when the coarsened hints fit the configured capacity. With… · New

1 resolved since last review

Apply cost-based backoff to failed polling retries

_{Review effort: Balanced}

Treat half capacity as a compaction target rather than a hard limit. Cover mixed root and nested hints at even and odd capacities while retaining genuine overflow fallback. Co-authored-by: Copilot App <[email protected]>

Copilot

🔵 Needs a closer look

Human review is needed because the central managed.rs scheduler/watcher diff was not supplied, leaving its queue, retry, and verification-timer interactions with concurrent publication paths unassessed.

No open issues 🎉

1 resolved since last review

This can force a full content pass even when the coarsened hints fit the configured capacity. With…

_{Review effort: Balanced}

A ready publication can precede permit teardown. Drain its reserved slots before taking the test's writer lock so the fixture does not strand a reservation during cleanup. Co-authored-by: Copilot App <[email protected]>

Copilot

🔵 Needs a closer look

The central scheduler and retry diff is missing, and changes to concurrency, cross-platform filesystem safety, and scheduler fairness require maintainer review.

No open issues 🎉

_{Review effort: Balanced}

Shengyu Fu (shengyfu) and others added 3 commits October 26, 2026 06:59

Bump version to 1.2.5

7951103

Co-authored-by: Copilot App <[email protected]>

Copilot AI balanced review requested due to automatic review settings October 26, 2026 08:18

Copilot started reviewing on behalf of Shengyu Fu (shengyfu) October 26, 2026 08:19

View session

Copilot AI reviewed Oct 26, 2026

View reviewed changes

Comment thread tgrep-cli/src/serve/shared/managed.rs Resolved

Copilot AI balanced review requested due to automatic review settings October 26, 2026 08:46

Copilot started reviewing on behalf of Shengyu Fu (shengyfu) October 26, 2026 08:47

View session

Copilot AI reviewed Oct 26, 2026

View reviewed changes

Comment thread tgrep-core/src/worktrees/tests.rs Resolved

Copilot AI balanced review requested due to automatic review settings October 27, 2026 12:54

Copilot started reviewing on behalf of Shengyu Fu (shengyfu) October 27, 2026 12:54

View session