Skip to content

Keep admitted work running when stop-if-idle finds the namespace busy; bump to 1.2.6 - #219

Merged
Shengyu Fu (shengyfu) merged 2 commits into
mainfrom
shengyfu-fix-tgrep-idle-stop-race-v1-2-6
Oct 31, 2026
Merged

Shengyu Fu (shengyfu) merged 2 commits into
mainfrom
shengyfu-fix-tgrep-idle-stop-race-v1-2-6

Conversation

@shengyfu

Copy link
Copy Markdown
Member

Fixes #218.

Problem

In v1.2.5, stop-if-idle closed catalog admission before it counted protected work. It then always tried to lock activity.lock exclusively, even when the count had already found work. Admission stayed closed until the decision committed, and a lock the probe obtained stayed held until the probe returned, even when the decision was busy. Work accepted before the probe could therefore fail its next admission check with retryable busy/namespace-admission-closed, even though the probe then refused the stop. Two checks fail that way:

  • work::ensure_admission rejects while catalog admission is closed. Operation acceptance and owner preparation, registration and claims call it. So do steps inside accepted jobs, including reservations, object creation and pinning, generation opening, metadata mutations, and view attach, reconciliation and publish.
  • ActivityGuard::acquire takes activity.lock shared for reservations, owner preparation, catalog handles, object and root guards, collection, inventory and housekeeping. While the probe holds the lock exclusively, the shared lock fails and is reported with the same code.

In #218 this left an accepted views.attach operation failed.

Fix (tgrep-core/src/managed/maintenance.rs)

  • The probe first samples receipt readers and external work (requests, queries, queued jobs, background batches). A single catalog transaction then counts unreleased leases and owners, reservations and unfinished operations, and closes admission only when all of them are idle. The count uses the same accepted/preparing/cancelling filter as v1.2.5.
  • Operation acceptance, owner preparation and owner registration each call ensure_admission inside the IMMEDIATE catalog transaction that records them, so they serialize with the probe's count-and-close transaction. Every unfinished operation accepted before the closure is counted and makes the probe refuse. A later one gets the documented retryable busy/namespace-admission-closed before acceptance. An accepted operation is never then failed by an idle-probe closure. Owner claims also check admission but write nothing to the catalog. A claim needs an issued, unreleased owner, which the count includes, so an idle probe never closes admission under a claim.
  • Visible work (new busy fast path). The probe neither closes admission nor tests activity.lock, and never reaches the IdleAdmissionClosed fault point. Like the v1.2.5 busy decision, it discards only expired verifier caches (discard_idle_verification(true)). It then records its non-stopping result through the same token and operation-result path. It still writes admission='open' as v1.2.5's busy decision did, which also repairs a closure left behind by a failed reopen.
  • Nothing visible. The v1.2.5 sequence runs unchanged. With admission closed, the probe rechecks those counts and receipt readers and tests independent namespace/root/object guards through activity.lock. It commits a stop only when every category is idle. A busy result reopens admission and a committed stop leaves it closed. Only a committed stop ends in-flight work.
  • Both paths finish through commit_idle_decision, which writes the stopping service record or admission='open' together with the receipt, then hits IdleCommitted.
  • No new deadlines, retries or fault points.

Compatibility

  • No RPC, field or reason-code changes. stop-if-idle, busy, namespace-admission-* and daemon-admission-closed are unchanged, and stopping keeps its meaning. No CLI change.
  • Semantic change to a diagnostic field. IdleOutcome::namespace_readers_or_work is now observed only after admission closes, and is false when the probe refuses before closing it. In v1.2.5 every probe tested activity.lock, so a busy probe could report true. The Copilot runtime reads only stopping, and SHARED_WORKTREE_INDEXES.md did not define this field. The field's doc comment and the stop-if-idle API row now state this.
  • The existing managed::collection::tests::busy_idle_probes_preserve_staging_reclamation_and_storage_admission asserted namespace_readers_or_work for a probe that an issued owner and an accepted attachment had already made busy. It now asserts operations == 1 and !namespace_readers_or_work. The inverted assertion fails on v1.2.5 with IdleOutcome { stopping: false, .., owners: 1, reservations: 0, operations: 1, .., namespace_readers_or_work: true, .. }, so it doubles as regression evidence that a busy probe leaves activity.lock alone.

Why there is no deferral, CLI change or race test

  • There is no residual window to defer. When a probe sees no work, the transaction that counts operations is the one that closes admission, so it counts every unfinished operation accepted before the closure. The ViewManager::execute deferral list is unchanged.
  • The CLI handler holds the process admission mutex for the whole stop-if-idle call. The accept loop and the maintenance tick lock the same mutex, so they wait for the decision rather than being rejected. They see daemon-admission-closed only after a committed stop or an unresolved idle-state-unavailable. Jobs never consult that gate. This corrects step 1 of the issue's code path, which says new connections and receipt polls are rejected mid-probe.
  • A race test would pass on v1.2.5 too: an attach either completes while the daemon stays, or is rejected before acceptance while it stops. Acceptance during a genuine closure was already rejected before acceptance in v1.2.5, and atomic_idle_admission_rejects_new_owners_operations_and_independent_readers covers that.

Tests

The new tests use existing managed-test-hooks fault points (IdleAdmissionClosed, IdleCommitted). They synchronize on hook state or probe completion, without sleeps, retries or longer timeouts. With only tgrep-core/src/managed/maintenance.rs reverted to v1.2.5 (tests unchanged), each one fails:

Test Check it covers Result on v1.2.5
managed_lifecycle: busy_idle_probe_keeps_admission_open_for_an_accepted_attach ensure_admission The probe paused at IdleAdmissionClosed, and the accepted attach ended Failed with busy/namespace-admission-closed. Expected Completed.
managed_lifecycle: busy_idle_decision_leaves_the_activity_lock_to_admitted_work ActivityGuard::acquire With the busy decision paused at IdleCommitted, reserve for the accepted attach returned Error { category: Busy, reason_code: "namespace-admission-closed", retryable: true, committed_state: NotCommitted, .. }.
managed_lifecycle: operation_accepted_before_an_idle_probe_refuses_the_stop_and_completes ensure_admission, with an operation as the only protected work The metadata policy mutation ended Failed with busy/namespace-admission-closed. Expected Completed.
shared_daemon: managed::faults::busy_idle_probe_keeps_admission_open_for_an_accepted_attachment Daemon end to end The sibling views.attach ended "state":"failed" with busy/namespace-admission-closed. Expected "completed".
managed::collection::tests::busy_idle_probes_preserve_staging_reclamation_and_storage_admission (inverted assertion) activity.lock on a busy probe namespace_readers_or_work: true

Adjusted existing daemon test. managed::clients::escaped_cross_process_readers_block_collection_and_idle_stop_then_allow_exact_rebuild expects an escaped cross-process reader to show up as namespace_readers_or_work: true. Its idle_status helper took the first non-busy probe outcome. With this change, a probe that happens to see other daemon work refuses before it closes admission and reports false. Two sources of such work already exist in v1.2.5:

  • the scheduler pass counted in background_batches;
  • the preceding RPC, whose request guard drops only after its response is written.

With the fix and v1.2.5's helper, the test failed this way (left: Bool(false)) in 11 of 20 runs. The helper now repeats the probe until an outcome shows no other work: every lease, owner, reservation, operation, receipt-reader and external count is zero. It polls every 20 ms, like the existing stop() helper, within a 10 s bound. Only a probe that closed admission and tested activity.lock can return an outcome like that. The assertions are unchanged, apart from now printing the outcome. No product code depends on this.

Reproduction and stress runs

At this PR's head, each run a separate cargo test --locked ... --features managed-test-hooks ... --exact invocation. The v1.2.5 rows check out only tgrep-core/src/managed/maintenance.rs from refs/tags/v1.2.5, then restore it.

Product code Tests Runs Result
v1.2.5 collection test (inverted assertion) 3 3 failed
v1.2.5 the three new managed_lifecycle tests 3 all three failed in every run
v1.2.5 new daemon test 10 10 failed
v1.2.5 escaped-reader test, adjusted helper 3 3 passed
This PR collection test 3 3 passed
This PR the three new managed_lifecycle tests 3 all passed
This PR new daemon test 10 10 passed
This PR escaped-reader test, adjusted helper 20 20 passed
This PR escaped-reader test, v1.2.5's managed_clients.rs 20 11 failed at assert_eq!(idle["namespace_readers_or_work"], true) with left: Bool(false); 9 passed

The adjusted helper passes against both v1.2.5 and fixed product code, and still asserts namespace_readers_or_work: true for the escaped reader.

Release

  • Bumps the version to 1.2.6, the same way as 7951103: Cargo.toml, Cargo.lock, fuzz/Cargo.lock and the two CONTRIBUTING.md examples.
  • SHARED_WORKTREE_INDEXES.md updates the typed claim failures, the stop-if-idle API row, and the idle-stop sequence and recovery text. The activity.lock layout entry keeps its v1.2.5 wording, which is still accurate: work and independent readers hold that lock shared, and only a probe that has closed admission tests it exclusively.
  • Not part of this PR: merging, tagging, running release.yml or sign-and-release.yml, or publishing a release.

Validation

Run one at a time at this PR's head, f606863, on Windows 11 Enterprise (16 logical CPUs) with rustc/cargo 1.98.0, rustc 1.88.0 and Python 3.14.7. git status --porcelain was empty before and after. TGREP_V1_BINARY and TGREP_V1_READER, which only the shared_daemon managed tests read, pointed to the debug tgrep and legacy_reader example built as ci.yml builds them: a git archive of the pinned 1120aca, prepared with scripts/qualification/prepare_legacy.py. The Rust 1.88 rows used the windows-msrv job's environment (CARGO_BUILD_JOBS, RAYON_NUM_THREADS and RUST_TEST_THREADS set to 2, plus its hardening RUSTFLAGS) with CARGO_TARGET_DIR=target\msrv.

Command Result
cargo fmt --all -- --check Exit 0
git --no-pager diff --check Exit 0
git --no-pager diff --check refs/tags/v1.2.5..HEAD Exit 0
cargo clippy --locked --workspace --all-targets --features tgrep-cli/managed-test-hooks -- -D warnings Exit 0
cargo test --locked -p tgrep-core --features managed-test-hooks managed:: -- --test-threads=2 Exit 0: 112 passed
cargo test --locked -p tgrep-core --features managed-test-hooks --test managed_lifecycle -- --test-threads=2 Exit 0: 42 passed
cargo test --locked -p tgrep-cli --features managed-test-hooks --test shared_daemon managed:: -- --test-threads=2 Exit 0: 81 passed, 3 ignored
cargo test --locked -p tgrep-cli --features managed-test-hooks --bin tgrep Exit 0: 387 passed, 2 ignored
cargo test --locked --workspace Exit 101 at shared_daemon (66 passed, 1 failed, 1 ignored), after the 11 earlier tgrep-cli targets passed. The failure is the pre-existing one below.
cargo test --locked --workspace --no-fail-fast (to run the remaining targets) Exit 101: 1,472 passed, 1 failed (the same test), 3 ignored, across 25 targets
cargo test --locked --workspace --features tgrep-cli/managed-test-hooks --no-fail-fast -- --test-threads=2 Exit 0: 1,539 passed, 6 ignored, across 25 targets (shared_daemon: 115 passed, 4 ignored), including every new and adjusted test
python -B -m unittest discover -s scripts/qualification -p test_*.py -v OK: 29 tests
python -B -m unittest discover -s scripts -p test_*.py -v OK: 65 tests, 5 skipped
python -B scripts/qualification/qualify.py dependencies Passed; all three lockfiles unchanged
cargo +1.88.0 check --locked --workspace --all-targets --features tgrep-cli/managed-test-hooks Exit 0
cargo +1.88.0 test --locked -p tgrep-core --features managed-test-hooks --test managed_catalog_windows --test managed_lifecycle -- --test-threads=2 Exit 0: 45 passed
cargo +1.88.0 build --release --locked -p tgrep-cli --bin tgrep --target x86_64-pc-windows-msvc Exit 0
The release tgrep.exe --version, then tgrep.exe --no-index -F -q -- "[workspace.package]" Cargo.toml Exit 0 for both; prints tgrep 1.2.6
cargo build --locked --workspace --benches Exit 0
python -B scripts/qualification/qualify.py installed Passed. It installed this checkout's release build into a private root with cargo install --locked, then checked the installed binary: local index and server parity; shared attach, refresh and stale-daemon fallback; restart, detach and worktree deletion; and managed v2 owner proof, generation reuse, indexed parity, migration and replay. All three lockfiles unchanged. Its cleanup retried once while Windows still held the exited tgrep.exe image, which the harness expects.
cargo test --manifest-path vendor/ignore/Cargo.toml --locked --target-dir target\vendor-ignore Exit 0: 148 passed, including 6 doctests. --target-dir keeps the build out of vendor/ignore/target, which .gitignore doesn't cover.
The managed-measurement job's steps: cargo test --locked --release -p tgrep-cli --features managed-test-hooks --test shared_daemon --no-run --message-format=json, then scripts/qualification/capture_managed_artifacts.py, then the captured harness with managed::performance::native_managed_lifecycle_measurement --ignored --exact --test-threads=1 --nocapture Exit 0 for each step. The measurement, which stops its daemons through stop-if-idle, passed in 72 s. The host was shared rather than quiet, so this is a functional pass, not a measurement.
git --no-pager diff --exit-code -- Cargo.lock fuzz\Cargo.lock vendor\ignore\Cargo.lock, after each batch Exit 0

Pre-existing failure, out of scope. Without managed-test-hooks, managed::polling_modes_repair_unreported_restored_mtime_and_membership_changes fails at tgrep-cli/tests/shared_daemon.rs:151 with unknown managed method: testing.reconcile-timer. That RPC exists only with the feature, but the test is not gated on it. Unmodified v1.2.5 fails the same way: from a git archive of v1.2.5, cargo test --locked -p tgrep-cli --test shared_daemon -- managed::polling_modes_repair_unreported_restored_mtime_and_membership_changes --exact exits 101 with the identical panic. CI always enables the feature, and the test passes in the feature runs above. This PR leaves it unchanged.

Not run locally: the Linux and macOS jobs, including the Unix installer test and the scripts/agent integration tests, which CI runs only off Windows.

Shengyu Fu (shengyfu) and others added 2 commits October 29, 2026 12:38
A stop-if-idle probe closed catalog admission and tested activity.lock
before it counted protected work. Jobs of operations accepted before the
probe then failed with busy/namespace-admission-closed even when the probe
decided not to stop.

Count catalog work and close admission in one transaction, and close only
when no work is visible. Acceptance checks admission in its own write
transaction, so the count includes every unfinished operation accepted
before the closure. A probe that sees unreleased leases or owners,
reservations, unfinished operations, receipt readers or external work
now refuses the stop without closing admission or testing activity.lock,
discards only expired verification, and records the decision through the
same receipt path. The closed path is unchanged: the final all-category
count and activity.lock test follow the closure, and only a committed
stop ends in-flight work.

namespace_readers_or_work is now observed only after admission closes and
is false when the probe refuses before closing.

Fixes #218

Co-authored-by: Copilot App <[email protected]>
Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 13:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the changes to busy idle-stop probes.

No open issues 🎉

What changed in this PR

Fixes #218 by letting managed stop-if-idle probes refuse a stop without interrupting already-admitted work when the namespace is visibly busy. Prepares release 1.2.6.

Changes:

  • Counts catalog work and conditionally closes admission in one transaction; busy probes leave activity.lock untouched.
  • Adds regression coverage for accepted operations and adjusts reader tests for the diagnostic field’s updated semantics.
  • Documents the idle-stop behavior and updates release versions and examples.
File Description
tgrep-core/​tests/​managed_lifecycle.rs Adds three regression tests protecting admitted work during busy probes.
tgrep-core/​src/​managed/​maintenance.rs Adds the busy fast path and shared idle-decision commit logic.
tgrep-core/​src/​managed/​collection.rs Updates assertions for busy probes that skip the activity lock.
tgrep-cli/​tests/​shared_daemon/​managed_faults.rs Adds an end-to-end accepted-attachment regression test.
tgrep-cli/​tests/​shared_daemon/​managed_clients.rs Waits for unrelated work to clear before checking escaped readers.
SHARED_WORKTREE_INDEXES.md Documents admission handling and reader diagnostic semantics.
fuzz/​Cargo.lock Updates the locked core version to 1.2.6.
CONTRIBUTING.md Updates release examples to v1.2.6.
Cargo.toml Bumps the workspace version to 1.2.6.
Cargo.lock Updates locked CLI and core versions to 1.2.6.

Review effort: Balanced

@shengyfu
Shengyu Fu (shengyfu) merged commit df30218 into main Oct 31, 2026
16 checks passed
@shengyfu
Shengyu Fu (shengyfu) deleted the shengyfu-fix-tgrep-idle-stop-race-v1-2-6 branch October 31, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v1.2.5: stop-if-idle fails already-accepted view operations with namespace-admission-closed even when it decides busy

2 participants