Skip to content

Add opt-in managed shared-index lifecycle - #189

Merged
Shengyu Fu (shengyfu) merged 35 commits into
mainfrom
shengyfu-managed-shared-index-lifecycle
Oct 11, 2026
Merged

Shengyu Fu (shengyfu) merged 35 commits into
mainfrom
shengyfu-managed-shared-index-lifecycle

Conversation

@shengyfu

@shengyfu Shengyu Fu (shengyfu) commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Summary

Add an opt-in, versioned managed lifecycle around the existing shared generation builder, worktree reconciliation, watchers and search engine.

  • Prepare explicit and adaptive exact-HEAD base changes while the previous view serves queries, then atomically publish the reconciled replacement and authoritative current pin. Existing snapshots retain their original generation.
  • Persist namespace ownership, physical publication incarnations, current views, checkpoints and idempotent operation receipts in a durable catalog.
  • Carry OS-backed lifetime protection through owners, generation readers, views and snapshots. Distinguish lease release from the eventual release of readers and worktree root handles.
  • Withdraw checkpoint references before collecting eligible generations. Authenticate catalog-owned objects, recheck current eligibility, and bound collection, proof work, staging and publication overlap. Support cancellation, crash recovery and offline maintenance after repository deletion.
  • Expose versioned policies, allocations, reservations, paginated inspection, operation recovery and atomic stop-if-idle with admission closed before protected-work checks.
  • Authenticate managed RPCs before dispatch using rotating, owner-only credentials, with a separate private capability for genuine legacy clients in compatibility mode.

The catalog uses bundled SQLite 3.53.3 through a pinned rusqlite revision, with native Windows WAL and locking qualification. The workspace version is 1.2.0; no release or tag is created by this PR.

Compatibility

Managed storage is opt-in. Legacy shared operation retains fixed generation pins, retain-all storage and no-expiry leases; ordinary, unborn-repository and non-Git search behavior remains supported. Managed namespaces have a separate format/protocol boundary and do not make legacy stores collectible.

Public namespace/instance IDs do not authorize managed RPCs. Clients authenticate using per-daemon credentials protected by native owner-only file permissions. The CLI handles discovery and authentication automatically; the documented direct-RPC contract includes credential loading and binding to the complete registration. Default v1 daemons started without a policy retain their existing transport.

Storage must be trusted and caller-controlled. Namespace authentication and containment are not an isolation boundary against arbitrary same-user writers or privileged OS administrators.

Qualification and measurements

The results in this section describe the pre-PR baseline; PR checks and review follow-ups are separate.

  • All eight native CI jobs passed, including Windows, Linux and macOS qualification.
  • Additional immutable-source local execution completed with 1,250 passes and five expected ignored tests on Windows, and 1,294 passes and five expected ignored tests on native Linux. Independent hook-free/default Windows qualification completed with 1,212 passes and two expected ignored tests. Normally installed release and external-consumer qualification are separate passing gates.
  • Five native measurement runs use release binaries with managed-test-hooks, without injected pauses, separately from hook-free/default and normally installed release qualification. They exercise real shared-backend/scan parity and bounded collection on bounded synthetic fixtures, not representative production workloads or universal performance thresholds.
  • The broad qualification and measurements above use code, tests and harnesses from 87bdd224633ce4044fbbfa0a40c51e340c982c07. The initial documentation-only descendants end at 7e9acd8, changing only the two documentation files below. Local archive reports explicitly record unavailable Git revision; separately retained source, Cargo and image fingerprints link them to that source.

The measurement document preserves per-run samples, actual build/reuse counters, collection receipts, sampled-resource limitations and unavailable metrics. The successor attachment in this fixture rebuilt every blob, so it does not establish incremental-build savings.

Review follow-ups

8aec641 rejects owner-claim files larger than 1 MiB before deserialization or acquiring the owner guard. Real-CLI regression coverage includes oversized valid prefixes with trailing whitespace or JSON, compact and limit-minus-one inputs, exact-limit acceptance, and preserved owner registration/release. Both targeted owner-holder tests passed on Windows against feature-enabled and default hook-free debug CLI builds; formatting and targeted Clippy checks also passed.

c6cb2ca addresses unauthenticated loopback management. Native private-file checks enforce owner-only access before writing or reading credentials; every managed RPC authenticates before either v2 or compatibility dispatch. Regression coverage exercises missing/wrong/public-ID credentials without mutations, native ACL enforcement, protocol-scoped legacy credentials, discovery tampering, restart rotation and failed-start cleanup.

For this authentication change, local Windows managed-core tests passed (88), and the complete shared-daemon integration suite passed (62, with four expected ignored). Native Linux passed the managed-core tests (88) and complete shared-daemon suite (67, with four expected ignored). The first Linux integration attempt encountered one retryable catalog-lock failure during attachment; the isolated test and unchanged full rerun passed. Both suites used pinned original v1 CLI/core-reader controls. Eight selected authentication, owner-holder, legacy-compatibility and lifecycle tests also passed against a newly built hook-free Windows CLI. Clippy passed on Windows and native Linux. PR native CI for this commit is separate from the historical results above.

a9bd5df reserves only a managed store directory and paths beneath its 64-hex repository namespaces from ordinary index commands, so ordinary indexes beneath an unrelated directory that shares the store name stay indexed; managed generation files remain rejected by format. 3b98aa7 closes a ready shared view only when opening, inspecting or reading the candidate fails; snapshot validation, deadline and cancellation failures propagate unchanged and leave the view ready. ca15e57 makes legacy-baseline preparation reject file/directory conflicts, including the generated probe path, before writing any output.

The new behavioral regression tests fail against the previous implementation. For this source, the complete hooked workspace suite passed locally on Windows (1,262 passed, five expected ignored) and native Linux (1,308 passed, five expected ignored) using pinned original v1 CLI/core-reader controls. Formatting, Clippy and the Python qualification and script suites also passed, as did the POSIX-only agent suite on Linux. All eight native CI jobs for this head passed on Windows, Linux and macOS, and CodeQL and the CLA check also passed.

The version bump and review follow-ups do not relabel the earlier broad qualification or measurement results, which remain attributed to 87bdd22.

Shengyu Fu (shengyfu) and others added 29 commits October 8, 2026 06:58
Add isolated managed storage, guarded reader and owner lifetimes, atomic explicit and adaptive advancement, reference-aware resumable collection, allocations, and versioned live/offline management. Preserve ordinary and retain-all v1 behavior, document the generic client contract, and add native lifecycle qualification.

Co-authored-by: Copilot App <[email protected]>
Restore authoritative full publications after committed failures, retain adaptive decisions across deferral, and checkpoint WAL before mutations. Correct idle-stop authorization and native ownership portability. Exercise live recovery and record checked measurement cleanup and UTC windows.

Co-authored-by: Copilot App <[email protected]>
Use a bounded metadata-root list and choose the shortest matching relative path. Exercise both root orders against a real linked-worktree view, including HEAD, index, configuration, lock and unrelated-event controls.

Co-authored-by: Copilot App <[email protected]>
Use the documented sys/proc_info.h ABI value because the pinned libc crate exposes proc_listpids without PROC_PGRP_ONLY. Keep the bounded, positively identified zombie-only inventory check unchanged.

Co-authored-by: Copilot App <[email protected]>
Serialize headroom admission with SQLite writer acquisition, account for FULL-sync padding and future sidecars, and bound spill-disabled pager payloads. Preserve durable usage and reject unsafe policy reductions before mutation. Cover actual held-reader CLI service and owner-reaper contention.

Co-authored-by: Copilot App <[email protected]>
Use the smallest immutable upstream SQLite correction for canonical Windows paths, cover native read-lock release and real PSOW padding, and serialize metadata execution before deciding whether a receipt is terminal.

Co-authored-by: Copilot App <[email protected]>
Record bounded rendezvous timeouts inside the callback and assert only after restoring the VFS hook. Exercise the sticky timeout path without native callbacks.

Co-authored-by: Copilot App <[email protected]>
Exercise the normally installed release binary through owner lifetime, generation reuse, private views, shared-v2 scan parity, exact migration replay and explicit release. Reuse version-aware backend labels for filename search stats.

Co-authored-by: Copilot App <[email protected]>
Require supported nonzero content evidence for an unmapped owned fixture, invalidate timestamp-restored writes, and reject reusable evidence while a writable mapping survives its creator descriptor. This native proof does not assert a deletion exclusion fence.

Co-authored-by: Copilot App <[email protected]>
Add authentication format discrimination, bounded immutable block proof batches, creation-handle ownership snapshots, exact post-commit proof replay, and conservative producer recovery. Charge authentication scratch memory and catalog storage; compact proof rows in bounded units. This is an intermediate producer slice; native authenticated collection remains separate unfinished work.

Co-authored-by: Copilot App <[email protected]>
Retain native continuity observations, authenticate surviving prefixes and deletion intents, preserve ownership changes, and account for volatile verifier resources.

Co-authored-by: Copilot App <[email protected]>
…rors

Use the Linux F_SETOWN_EX UAPI with its thread-owner struct, and retain cross-platform final-boundary ownership regressions without losing durable intent state.

Co-authored-by: Copilot App <[email protected]>
Record the descriptor-backed absent FILESEC_ACL state separately from bounded ACL encodings and add native absence, ACL-change, and restoration controls.

Co-authored-by: Copilot App <[email protected]>
Centralize bounded unreserved admission with memory-first locking and verify the lock order through a deterministic nonhanging fault boundary.

Co-authored-by: Copilot App <[email protected]>
Retain the cross-platform inventory regression and exercise policy/allocation restarts, expiration, cancellation, and active-versus-idle shutdown resources.

Co-authored-by: Copilot App <[email protected]>
…r guards

Invalidate all version-stale proof without conflicting with retained inventory descriptors. Observe cancellation before destructive I/O and atomically in the final saved result.

Co-authored-by: Copilot App <[email protected]>
Keep missing guards unknown without matching retirement evidence, avoid duplicate control credits, and place the control-removal barrier after confirmed Windows handle release.

Co-authored-by: Copilot App <[email protected]>
Fence matching control opens without blocking unrelated owner admission. Separate root withdrawal from physical cleanup, serialize protected-reader admission with retirement, and retain accepted idle tokens across admission contention. Preserve deterministic same-control, unrelated-owner and same-token regressions.

Co-authored-by: Copilot App <[email protected]>
Reread the authoritative receipt after maintenance admission and hold serialization through error publication. Add a deterministic accepted-receipt interleaving that previously replaced a non-stopping decision with a committed stop.

Co-authored-by: Copilot App <[email protected]>
Revalidate terminal recovery state under maintenance serialization before lifetime checks and execution. Retain the exact pending collection object when another bounded control verifier owns admission, exposing the Busy skip without losing continuation or recrediting removed members. Add deterministic completed/failed/cancelled replay and physical-cleanup continuation controls.

Co-authored-by: Copilot App <[email protected]>
…naries

Scope known-operation pause and error hooks before asynchronous admission becomes visible. Preserve causal reconciliation, crash, recovery, native and legacy controls, and capture the exact Cargo-selected measurement CLI and harness as qualification artifacts.

Co-authored-by: Copilot App <[email protected]>
Preserve the pinned SQLite graph and managed backend labels while merging the upstream pattern resolution, smart-case, boundary and multiline fixes. Exercise the actual managed-v2 backend against scanning for both engines, retaining the empty-pattern-set early exit.

Co-authored-by: Copilot App <[email protected]>
Preserve failed-token replay and exact recovered pins before fresh-token attempts. Exercise post-lease SQLite Busy injection, unchanged lease resumption, and bounded exhaustion without changing shipping retry or serialization policy.

Co-authored-by: Copilot App <[email protected]>
Preserve ready readers and exact checkpoints when queued background work has already been fulfilled. Keep forced repairs, cancellation and version checks, immutable replay, and the periodic verification clock intact.

Co-authored-by: Copilot App <[email protected]>
Keep new attempts limited to authoritative uncommitted catalog contention, preserve absent or pending initial intent and immutable receipts, and continue from the actual consumed sequence. Cover pre-lease transaction failure, persisted-lease resumption, bounded exhaustion, and actual dropped-response replay without altering shipping retry behavior.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 05:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-platform storage and lifetime semantics require human validation, and bounded owner-claim parsing remains incorrect.

1 open finding
What changed in this PR

Adds an opt-in managed v2 lifecycle for shared indexes while preserving legacy v1 retain-all behavior.

Changes:

  • Adds versioned ownership, storage, checkpoints, collection, accounting, and recovery.
  • Adds managed CLI/RPC routing and bounded/cancellable indexing.
  • Expands cross-platform qualification, compatibility tests, and documentation.
File Description
.github/​workflows/​ci.yml Adds managed lifecycle CI and measurements.
AGENTS.md Documents managed-agent lifecycle requirements.
Cargo.lock Locks managed storage dependencies.
Cargo.toml Pins bundled rusqlite.
CONTRIBUTING.md Documents managed qualification.
README.md Documents managed v2 usage.
fuzz/​Cargo.lock Updates fuzz dependency lock.
scripts/​qualification/​capture_managed_artifacts.py Captures exact measurement binaries.
scripts/​qualification/​prepare_legacy.py Prepares pinned v1 controls.
scripts/​qualification/​test_capture_managed_artifacts.py Tests artifact capture.
scripts/​qualification/​test_prepare_legacy.py Tests legacy preparation.
scripts/​qualification/​test_qualify.py Tests managed qualification helpers.
tgrep-cli/​Cargo.toml Adds managed test features/dependencies.
tgrep-cli/​src/​main.rs Adds --shared-policy routing.
tgrep-cli/​src/​search.rs Reports v2 backend usage.
tgrep-cli/​src/​serve/​shared/​mod.rs Registers managed server modules.
tgrep-cli/​src/​serve/​shared/​protocol.rs Adds v2 discovery and lifecycle commands.
tgrep-cli/​src/​serve/​shared/​server.rs Generalizes shared snapshot queries.
tgrep-cli/​tests/​shared_daemon.rs Extends daemon integration harness.
tgrep-cli/​tests/​shared_daemon/​runtime.rs Adds supervised process support.
tgrep-core/​Cargo.toml Adds managed dependencies and hooks.
tgrep-core/​src/​builder.rs Supports managed output layouts.
tgrep-core/​src/​encoding.rs Adds cancellable bounded decoding.
tgrep-core/​src/​error.rs Integrates managed errors.
tgrep-core/​src/​git_index.rs Adds controlled Git metadata handling.
tgrep-core/​src/​lib.rs Exposes managed APIs.
tgrep-core/​src/​live.rs Adds overlay accounting helpers.
tgrep-core/​src/​managed/​authentication.rs Implements member sealing.
tgrep-core/​src/​managed/​checkpoints.rs Implements protected checkpoints.
tgrep-core/​src/​managed/​clock.rs Adds persistent monotonic timing.
tgrep-core/​src/​managed/​error.rs Defines managed error contracts.
tgrep-core/​src/​managed/​faults.rs Adds deterministic test hooks.
tgrep-core/​src/​managed/​lifetime.rs Implements OS-backed guards.
tgrep-core/​src/​managed/​memory.rs Tracks retained memory.
tgrep-core/​src/​managed/​mod.rs Defines managed lifecycle API.
tgrep-core/​src/​managed/​owners.rs Implements owner release/reaping.
tgrep-core/​src/​meta.rs Adds controlled hashing and IDs.
tgrep-core/​src/​ondisk.rs Defines managed index layout.
tgrep-core/​src/​output.rs Adds guarded output abstraction.
tgrep-core/​src/​rooted.rs Exposes cloned directory handles.
tgrep-core/​src/​trigram.rs Adds bounded extraction.
tgrep-core/​src/​visibility.rs Adds controlled visibility processing.
tgrep-core/​src/​walker.rs Adds controlled metadata walks.
tgrep-core/​tests/​managed_storage_macos.rs Qualifies macOS generation evidence.
vendor/​ignore/​PATCHES.md Documents vendored hooks.
vendor/​ignore/​src/​dir.rs Propagates controlled reads.
vendor/​ignore/​src/​gitignore.rs Adds snapshot/read controls.
vendor/​ignore/​src/​walk.rs Adds walker cancellation checks.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Read one byte past the claim limit before deserializing so a valid JSON prefix cannot hide oversized whitespace or trailing data. Cover the boundary and preserved owner lifecycle through real CLI clients.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 10, 2026 03:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cross-platform locking, SQLite durability, reclamation, and crash-recovery semantics require final human validation.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

Co-authored-by: Copilot App <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The unauthenticated loopback management endpoint permits cross-user lifecycle mutations on multi-user hosts.

1 open finding

🧠 Review effort: Balanced

Comment thread tgrep-cli/src/serve/shared/managed.rs Resolved
Copilot AI balanced review requested due to automatic review settings October 10, 2026 04:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Query failures can invalidate healthy managed views, and malformed legacy archives can leave unretryable partial output.

1 open finding
Previously missed (2)

In code that hasn't changed since last review

Medium severity Preflight misses archive file/directory prefix conflicts

scripts/​qualification/​prepare_legacy.py:63

The preflight detects only duplicate names, not file/directory prefix collisions. An archive containing regular entries conflict and conflict/child passes this loop; extraction later raises FileExistsError after creating the destination and writing earlier source files, so the helper leaves a partial output that cannot be retried despite its stated no-reuse guarantee. Reject prefix conflicts during preflight or extract into a temporary sibling and publish only after complete success.

Medium severity Query control failures incorrectly trigger full view invalidation

tgrep-cli/​src/​serve/​shared/​server.rs:1335

A query-control failure is handled as if the candidate file changed. In particular, source.check() inside the read loop can return query-deadline, query-input-changed, or query-snapshot-changed; this branch then calls ManagedQuery::invalidate, which increments the view epoch and performs a full invalidation. A slow or concurrently invalidated query therefore dirties an otherwise healthy view and forces reconciliation. Propagate control/validation failures directly, and reserve invalidate for failures from opening, statting, or reading the candidate itself.

🧠 Review effort: Balanced

Require per-daemon credentials before v2 or compatibility dispatch. Publish and validate native owner-only credential files, bind discovery to the private registration, rotate capabilities at restart, and preserve genuine v1 clients with a separate private instance capability.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 10, 2026 05:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Managed-path detection incorrectly blocks ordinary indexes beneath unrelated directories named tgrep-managed-v2.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

Shengyu Fu (shengyfu) and others added 3 commits October 9, 2026 23:32
Ordinary index commands rejected any path with an ancestor named like the managed store, so an unrelated project or index beneath a directory that merely shares that name fell back to scanning or could not be indexed. Reserve only the store directory itself and paths beneath its 64-hex repository namespaces, compared lexically and case-insensitively after normalization; managed generation files stay rejected by format.

Co-authored-by: Copilot App <[email protected]>
Snapshot validation, deadline and cancellation failures during a shared content query were treated as candidate read failures, which invalidated a ready view and forced full repair. Only open, stat and read failures of the candidate now close the view; control failures propagate unchanged and the view stays ready.

Co-authored-by: Copilot App <[email protected]>
Reject archive entries beneath a file entry, including the generated probe path, before creating the destination so extraction cannot fail partway and leave output that blocks a retry.

Co-authored-by: Copilot App <[email protected]>
Copilot AI balanced review requested due to automatic review settings October 10, 2026 06:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The 89-file lifecycle, authentication, native-locking, and reclamation change requires final human review despite extensive qualification.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

@shengyfu
Shengyu Fu (shengyfu) merged commit 8a05880 into main Oct 11, 2026
15 checks passed
@shengyfu
Shengyu Fu (shengyfu) deleted the shengyfu-managed-shared-index-lifecycle branch October 11, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants