Skip to content

Fix Windows managed publication and CRLF overlay memory accounting - #205

Merged
Shengyu Fu (shengyfu) merged 5 commits into
mainfrom
shengyfu-windows-publication-metadata
Oct 15, 2026
Merged

Shengyu Fu (shengyfu) merged 5 commits into
mainfrom
shengyfu-windows-publication-metadata

Conversation

@shengyfu

@shengyfu Shengyu Fu (shengyfu) commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Summary

Fix two independent managed-publication/readiness blockers encountered while qualifying v1.2.1:

  • Windows producer re-observation: authenticate a closed member against its original complete BLAKE3 block proofs and manifest before updating its observed metadata. Keep the native write/delete exclusion guard, exact inventory snapshot, identity, ownership, and producer-state checks. Metadata-only ChangeTime drift is no longer rejected before content authentication; unchanged/restored timestamps are not a content-authentication shortcut.
  • Admitted deadlines: authentication uses the existing WorkPermit checks and captured deadline when one is supplied. Only the no-permit path derives a new bounded deadline from current policy.
  • CRLF private overlays: account for backing allocations and their lifetimes instead of retaining extraction growth headroom and accumulating fixed per-posting publication charges. Admit old and replacement table storage during growth, release consumed prepared maps, and preallocate controlled clones. Keep actual CRLF bytes and positional/next-byte masks unchanged. Review follow-up tracks allocation watermarks across deletions and removes replaced entries before sizing growth.

Update the managed-lifecycle documentation. The user-requested version-only follow-up, 2f9d21fb964c5816fb4461ace812c7661a856d09, bumps the workspace crates, root/fuzz lock records, and release instructions to 1.2.2. Historical v1.2.1 deprecation references and immutable candidate evidence are preserved. No budget, timeout, watcher-freshness, or host security-policy changes.

Regression evidence

The ordinary-user-EA regression fails against c7b2ab8 with stale-identity / producer-member-modified. It passes with authentication against the original producer seal; no protected kernel EA is set or modified. The kernel component responsible for the observed metadata drift remains unidentified.

The policy-reduction-after-admission regression fails against 2055b93 and passes with the captured permit deadline. Negative coverage remains for same-length tampering with restored timestamps, ownership/identity changes, incomplete/corrupt producer proofs, conflicting native opens/writable mappings, memory pressure, and cancellation.

The new eight-file CRLF regression fails against 0b6e28a with reserved-private-memory-exhausted under the existing 32-MiB test reservation. It now passes through initial readiness, checkpoint capture, and unchanged full refresh. A separate table-growth test covers overlapping allocation charges, genuine admission failure without mutation, overflow, cancellation, and released headroom.

Original generated case: resolved without budget changes

Release/no-test-hooks executable from 0143cc4b7e59e7aa97759c59e1fd98e9b8b238bf; owned synthetic repositories only, 4,096 files of 8,192 bytes each. The original ReFS-backed CRLF fixture and its content inventory were preserved; a separately labeled LF/NTFS-cache control was not substituted for it. These measurements predate the version-only bump and remain attributed to the unchanged candidate.

Case Cold generation + readiness Fresh warm readiness Peak private-work admission
Original CRLF/ReFS 529.937 s 359.351 s 7,614,786,984 bytes
Separate LF control 256.820 s 6.403 s 95,988,356 bytes

Both retain the original 32-GiB allocation, 8-GiB private and staging limits, one operation worker/slot, 16-MiB sort buffer, and 10,800,000-ms operation timeout. Effective policy and allocation were checked directly.

CRLF cold and warm attachments fully verified and extracted all 4,097 paths, including the linked-worktree gitfile. Warm attachment reused the published generation with zero Git blob reads. Managed/standalone membership matched all 4,097 paths; managed/local-index/filesystem content results matched for the shared anchor (4,096 paths), a 32-byte substring after the CRLF prefix (one path), and an absent pattern (zero paths). LF preserved 4,096 base reuses with only the gitfile extracted.

The original-case daemon's separately observed native peak private/commit memory was 2,200,727,552 bytes. Namespace admission estimates are not an RSS cap. These are single readiness observations, not a speedup claim: full transformed overlays remain expensive, particularly checkpoint/catalog work.

Both daemon/owner pairs exited normally, leases and root handles drained, and all four disposable worktrees were removed. Generated sources, caches, original failure reports, and new qualification evidence remain available locally.

Qualification scope: full watched real-repository qualification has not passed. The separate watcher-budget/polling-fallback issue is being handled independently and is not fixed by this PR. Successful generated publication/CRLF readiness does not establish full release readiness.

Validation

Correction candidate at 0143cc4, Windows / Rust 1.88: 228 selected tests passed (161 core unit, 24 generation integration, 3 native WAL, 37 lifecycle, 3 real CLI). Native Linux: 225 selected tests passed (165 core unit, 24 generation integration, 36 lifecycle); the ignored subprocess entry point was exercised by its supervisor. Formatting passed. Default Clippy completed with the same 10 pre-existing warnings in untouched files; strict -D warnings remains baseline-blocked.

Windows commands (CARGO_BUILD_JOBS=2, test-build Rust flags unset, MSVC environment initialized):

cargo +1.88.0 fmt --all --check
cargo +1.88.0 test --locked -p tgrep-core --features managed-test-hooks --lib -- managed:: worktrees:: live:: trigram:: shared:: --test-threads=2
cargo +1.88.0 test --locked -p tgrep-core --features managed-test-hooks --test generations --test managed_catalog_windows --test managed_lifecycle -- --test-threads=2
cargo +1.88.0 test --locked -p tgrep-cli --features managed-test-hooks --test shared_daemon -- managed::v2_cli_real_backend_migration_replay_and_idle_shutdown managed::live::concurrent_clients_share_one_new_publication_with_distinct_exact_commits_and_private_views managed::faults::process_crashes_at_generation_and_view_boundaries_recover_the_exact_pin --test-threads=2
cargo +1.88.0 clippy --locked --workspace --all-targets --features tgrep-cli/managed-test-hooks

Native Linux ran the same core selectors and the generations / managed_lifecycle integration targets in a native Linux directory, with two build/test threads.

Version-only follow-up at 2f9d21f: both locked graphs resolve tgrep-core/tgrep-cli to 1.2.2 without dependency upgrades. A fresh hardened Windows release build without test-hook features passed; --version, -V, and Windows FileVersion/ProductVersion all report 1.2.2. The 17 existing isolated release-script tests, formatting, and diff checks passed. The earlier generated campaign was not rerun or relabeled for this metadata-only change.

cargo +1.88.0 tree --offline --locked --workspace --depth 0
cargo +1.88.0 tree --offline --locked --manifest-path fuzz\Cargo.toml -p tgrep-core --depth 0
cargo +1.88.0 build --release --locked -p tgrep-cli --bin tgrep --target x86_64-pc-windows-msvc --no-default-features --target-dir target\publication-release
python -B -m unittest discover -s scripts -p test_prepare_release.py -q

Review corrections at 6627cefa439f3b0fed1d07aa8814b9c9b53fda52

Both review findings have deterministic red/green coverage against 2f9d21f:

  • A replacement with only 320 bytes of spare admission previously failed with reserved-private-memory-exhausted. Removing its old entries before sizing growth lets the same-size replacement fit. A truly oversized replacement still fails without changing the published view.
  • Deletion previously reduced a retained charge to 4,726 bytes instead of the required 10,390 while backing storage remained allocated. Persistent allocation watermarks now cover every retained table, nested posting set, and the bookkeeping map. Single/batch deletion tests verify unchanged surviving storage and actual retained namespace accounting; rehash, rebuild, real pressure, clear, and drop paths are covered.

The CRLF regression additionally covers an equal-length content edit, refresh, result parity, and another checkpoint under the original 32-MiB test reservation.

Windows: 261 selected tests passed (164 core unit, 24 generation, 3 native WAL, 37 lifecycle, 30 shared-worktree, 3 real CLI). Native Linux: 257 selected tests passed (168 core unit, 24 generation, 36 lifecycle, 29 shared-worktree); the supervised subprocess entry point is not double-counted. Formatting and a hardened release build pass. Default Clippy still reports only the same 10 pre-existing warnings; strict lint is not claimed to pass.

The review follow-up extends the earlier commands with these core/integration selectors; the three real CLI selectors and Clippy command above were also rerun:

cargo +1.88.0 test --locked -p tgrep-core --features managed-test-hooks --lib -- managed:: worktrees:: live:: trigram:: shared:: hybrid:: --test-threads=2
cargo +1.88.0 test --locked -p tgrep-core --features managed-test-hooks --test generations --test shared_worktrees --test managed_catalog_windows --test managed_lifecycle -- --test-threads=2

Native Linux uses the same core selectors and generations, shared_worktrees, and managed_lifecycle integration targets. Its archived source tree is exactly the committed tree 5fe12555fba5e141c57683ef4fa1f52ac9eb3e89.

A fresh local 1.2.2 release/no-hooks validation build also passed the original CRLF/ReFS cold/warm readiness and exact membership/content parity, without changing policy or allocation: 456.042 s cold, 409.739 s warm, 7,645,982,280 bytes peak private-work admission. Both passes fully read/extracted 4,097 paths; warm reused the generation with zero Git blob reads. Its SHA256 is ee70cf0c948009d49fb26d5bb9610ec05e650d6275635d5b2746822711613c7a.

Teardown caveat: after successful readiness/parity and both lease/root-handle drains, the benchmark harness hit operation-token-conflict while changing stop tokens without advancing the sequence, then forcibly terminated its own daemon. The failure report is preserved. The same executable subsequently recovered the retained cache and completed an atomic idle stop normally. The two new disposable worktrees and native Linux build directories were removed; the source inventory, caches, and evidence remain. This is not an all-clean first-run teardown or watched real-repository qualification. No existing immutable candidate was replaced.

Separately identified candidate

The retained read-only local candidate is not the official v1.2.1 artifact and was not replaced or relabeled by the 1.2.2 source bump or review corrections. Its CLI version remains tgrep 1.2.1; Cargo records show empty tgrep-core/tgrep-cli feature sets and a release, non-test profile. Both generated runs verified that testing.status is unavailable. No executable is committed or attached here.

Source: 0143cc4b7e59e7aa97759c59e1fd98e9b8b238bf
SHA256: fb46b6aa8dc9ccd56f6bf816f9c71c1e4ee79218688d6d9e6cc8264fa537e3ae
Build: cargo +1.88.0 build --release --locked -p tgrep-cli --bin tgrep --target x86_64-pc-windows-msvc --no-default-features --target-dir target\publication-release --message-format=json-render-diagnostics
RUSTFLAGS: -Ccontrol-flow-guard -Ctarget-feature=+crt-static -Clink-arg=/DYNAMICBASE -Clink-arg=/CETCOMPAT

Source diff and build provenance are retained beside the immutable candidate. Earlier candidates and the independent runtime qualification were not changed. This PR does not create a release/tag or merge either change.

Shengyu Fu (shengyfu) and others added 3 commits October 14, 2026 15:44
Reverify closed managed members against their original bounded block proofs under native write/delete exclusion. Keep the original seal, identity and ownership checks, and charge generation verification to its existing work permit. Reject timestamp-restored rewrites and unverified reobservations; add native metadata-only and protection regressions.

Co-authored-by: Copilot App <[email protected]>

Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d
Reuse the existing WorkPermit deadline and checks for admitted Windows verification. Bound only unreserved observations with current policy, and cover a policy timeout reduction after admission.

Co-authored-by: Copilot App <[email protected]>

Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d
Release extraction headroom and consumed input tables instead of retaining cumulative per-posting estimates. Admit hash-table growth with old and replacement capacity both charged, preserve content and mask semantics, and cover CRLF checkpoint/refresh under the existing budget.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d
Copilot AI balanced review requested due to automatic review settings October 15, 2026 01:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Windows publication protections and memory-admission changes need final human review, with a replacement-capacity issue still unresolved.

1 open finding
What changed in this PR

Addresses managed generation publication on Windows and CRLF overlay memory accounting without changing budgets or timeouts.

Changes:

  • Authenticates closed Windows producer files against original proofs using admitted deadlines.
  • Tracks overlay table capacity and allocation lifetimes.
  • Adds regressions and updates lifecycle documentation.
File Description
tgrep-core/​src/​worktrees/​tests.rs Adds CRLF budget regression.
tgrep-core/​src/​worktrees.rs Tracks prepared-map allocation lifetimes.
tgrep-core/​src/​trigram.rs Releases extraction growth headroom.
tgrep-core/​src/​managed/​work.rs Adds charge accessor and admission regressions.
tgrep-core/​src/​managed/​verification.rs Authenticates complete producer proofs.
tgrep-core/​src/​managed/​memory.rs Adds capacity-based table admission.
tgrep-core/​src/​managed/​generations.rs Passes publication work permits.
tgrep-core/​src/​managed/​collection.rs Tests Windows publication protections.
tgrep-core/​src/​managed/​catalog.rs Authenticates closed-member re-observation.
tgrep-core/​src/​live.rs Controls overlay growth and cloning.
SHARED_WORKTREE_INDEXES.md Documents authentication and memory accounting.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tgrep-core/src/live.rs Resolved
Update both first-party lockfiles and release instructions; preserve historical v1.2.1 deprecation and candidate evidence.

Co-authored-by: Copilot App <[email protected]>

Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d
Copilot AI balanced review requested due to automatic review settings October 15, 2026 03:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Retained-memory undercharging remains unresolved, and security-sensitive Windows authentication warrants final human review.

2 open findings

🧠 Review effort: Balanced

Comment thread tgrep-core/src/managed/memory.rs Resolved
Size replacement growth after removing old entries from the unpublished clone. Track backing-capacity watermarks for retained maps, posting sets, and bookkeeping instead of releasing tombstone storage. Add deterministic red/green replacement and deletion regressions, rehash/pressure coverage, and changed-CRLF checkpoint parity.

Co-authored-by: Copilot App <[email protected]>

Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d
Copilot AI balanced review requested due to automatic review settings October 15, 2026 05:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive Windows publication protection and allocation-lifetime accounting warrant final human review.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

@shengyfu
Shengyu Fu (shengyfu) merged commit 29b7e2e into main Oct 15, 2026
16 checks passed
@shengyfu
Shengyu Fu (shengyfu) deleted the shengyfu-windows-publication-metadata branch October 15, 2026 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants