Feature request: worktree overlay indexing for branch-aware agent workflows #573

Activity

1. DeusData

   added

   enhancementNew feature or requestNew feature or request

   on Jun 23, 2026
2. DeusData commented on Jun 23, 2026

   

   DeusData

   on Jun 23, 2026

   Owner

   More actions

   Really well-framed — thank you for pulling the worktree history (#48 / #37 / #351 / #401 / #489 / #493 / #499 / #542) together in one place.

   The merge-base overlay model (one canonical base index + a changed-files-only overlay with tombstones, resolved against the active Git context at query time) is the right shape and avoids the N-near-identical-indexes blowup. Committed-changes-only for a first cut is the correct scope call; dirty-tree overlays can come later.

   Tracking this under the architecture task #398, and the stale-entry-after-removal piece overlaps #286. Labeling enhancement.
3. ctaylor4874 commented on Jun 24, 2026

   

   ctaylor4874

   on Jun 24, 2026

   More actions

   Adding an operational data point from rolling this out org-wide (a large monorepo, auto_index=false), since it reinforces the "committed-changes-only first cut" scope and touches a couple of the open questions.

   The base-reuse gap bites even before overlays exist. As of main / v0.8.1, project identity is still the slug of getcwd() — detect_session() → cbm_project_name_from_path() in src/mcp/mcp.c — and the canonical_root resolved in #401 (src/git/git_context.c) isn't consulted for identity. With auto_index=false, a session started inside a linked worktree therefore:

   • derives a project name that was never indexed, and
   • maybe_auto_index logs autoindex.skip reason=disabled and bootstraps nothing,

   so the worktree gets no reuse of the canonical base graph at all — queries return "not indexed" — even though the base index is sitting in the cache and canonical_root already points at it. With auto_index=true the failure mode flips to the duplication/churn you list (a full from-scratch index per worktree, never GC'd).

   Interim workaround, in case it's useful to others: I run a SessionStart hook that detects a linked worktree (--git-dir != --git-common-dir) and injects the canonical project name (slug of canonical_root, computed to match cbm_project_name_from_path) so the agent passes project=<main> explicitly to the graph tools. It works, but it's a client-side band-aid and only ever exposes the base/committed graph — which is genuinely fine for most agent navigation (definitions, callers, impact); the branch delta is the part that needs the overlay.

   That experience suggests a useful first increment ahead of the full overlay (re Q10): context-aware read resolution to the canonical base when running inside a worktree — when is_worktree and no overlay exists, resolve reads against the canonical_root base index instead of a getcwd-named phantom, and skip auto-indexing worktrees so the shared base is never mutated. That removes both the "not indexed" cliff and the duplicate-index blowup, and it's strictly a subset of the overlay model (overlay = empty set). From this rollout, my bias on the open questions:

   • Q1 / Q6: infer context from the cwd's Git context for ergonomics, but keep an explicit project/context selector as the override — the hook above is really just a manual version of the cwd-inference path.
   • Q2: committed-only for v1, agreed — dirty-tree overlays were pure noise in my usage.
4. DeusData

   added

   priority/normalStandard review queue; useful PR with ordinary maintainer urgency.Standard review queue; useful PR with ordinary maintainer urgency.

   on Jun 30, 2026
5. DeusData

   added this to the 0.9.2-rc milestone on Jul 8, 2026
6. joshweir commented on Jul 27, 2026

   

   joshweir

   on Jul 27, 2026

   More actions

   Answering Q6 from the query side, against main @ d587deab. Confirms @ctaylor4874 and adds one case
   overlays cannot reach.

   Nothing is inferred today. Every graph read tool declares "required":["project"]
   (src/mcp/mcp.c:434 through 640), and the only resolution is get_project_arg() (1385): four
   spellings of the same key, then resolve_project_tail() (1331), a unique-suffix rescue over
   cache-dir filenames. detect_session() (10463) does slug one getcwd(), but no query handler reads
   session_project - it reaches the watcher, auto-index, the daemon session surface and the
   relative-repo_path join at 7521, and stops.

   Nor does anything error. A worktree holds the same files at the same relative paths, so a
   main-checkout answer returns real symbols at plausible lines - just the base ref. A reviewer agent
   asking who calls the function it changed gets the pre-change call graph and reports no regressions.
   resolve_project_tail() widens it: when only the worktree is indexed, passing the repo name finds a
   single tail match and adopts it silently.

   The resolver for this already exists, one layer away.
   ha_resolve_indexed_project_with_root() (src/cli/hook_augment.c:843) maps cwd to project: walk up,
   derive a name per level, probe index_status, first hit wins; on a full miss
   ha_registry_project_for_path() (785) takes the project whose root_path contains the cwd, longest
   prefix winning - which also covers --name <custom>, where no derived slug matches. It is static in
   src/cli/, unreachable from the tool layer. Lifting it is most of @ctaylor4874's increment. Four
   notes:

   • get_architecture already uses path for a directory prefix that scopes analysis (535), so a
     resolution hint needs another key.
   • cbm_mcp_server_set_session_context() (mcp.h:149) is public and already fed a canonical root over
     the wire by the daemon (application.c:2434).
   • resolve_store_internal() (1972) keeps one store open and closes it whenever the project changes,
     so probing index_status per level would reopen the queried DB on every call. Testing for
     <name>.db on disk, as resolve_project_tail() does at 1338, avoids that.
   • A caller-supplied path wants the same allowed_root / CBM_ALLOWED_ROOT check repo_path gets
     (7599).

   The case overlays cannot reach is subagents. A Claude Code subagent has its own working directory but
   shares the parent session's MCP server process, whose getcwd() still returns the parent's
   directory - and detect_session() caches that behind session_detected, never re-reading. No
   session-level inference sees it, and neither does the roots protocol: a subagent's worktree is not a
   session root.
   The path must arrive per call.

   So what I would send: an optional cwd on the read tools, resolved through the lifted helper only
   when project is absent, explicit project always winning, project moving out of required into an
   anyOf over the two - the shape check_index_coverage already ships (612). No storage change, no
   new graph semantics, no behaviour change for existing callers. Once overlays land, the same argument
   can select the overlay context.

   In scope for #573, or its own issue? Resolution is narrower than the overlay model and useful before
   overlays land. Happy to open a focused issue and send that PR if you prefer the split.

   Separately, and settling nothing here: when a query resolves a project whose root_path does not
   contain the caller's path, warn in the response, and likewise when the index predates HEAD
   (cbm_git_context_t already carries head_sha). Own issue if you want it.
7. lishixiang0705 commented on Jul 31, 2026

   

   lishixiang0705

   on Jul 31, 2026

   Contributor

   More actions

   Field report on the scale question, since @DeusData asked for worktree-heavy RC reports.

   Scale: 379 live worktrees off a single 2,432-file repo.

   $ git worktree list | wc -l
   379
   

   Breakdown: 367 persistent (feature branches, PR audits, per-issue workspaces under
   .worktrees/, orca/workspaces/, .claude/worktrees/), 12 ephemeral under agent
   scratchpads, 76 in detached HEAD. This is one repo, one developer, agent-per-worktree.

   Measured cost of the current one-index-per-worktree model (v0.9.1-rc.1 + 2 commits,
   plus an unrelated local patch to detect_changes seed scoping; macOS arm64, measured on
   an isolated clone so as not to disturb the working setup):

   Base repo	2,432 files → 54,129 nodes / 280,047 edges
   Full index, cold index cache (daemon warm)	19.21 s
   Index DB size	336 MB
   git worktree add of identical content, then index	own project name, own 340 MB DB, 54,089 nodes rebuilt from scratch
   Incremental (existing DB, one changed file)	4.66 s

   The worktree shares zero bytes with the base index despite being byte-identical at
   HEAD. Projected across the 379 worktrees: ~126 GB of near-duplicate index for one
   repository.

   Both auto_index settings are lossy here, in different ways — this confirms
   @ctaylor4874's #573 comment from the query side, which I hit independently before
   finding this thread:

   • auto_index=false (the default): a session started in a worktree derives a project
     name that was never indexed, maybe_auto_index logs autoindex.skip reason=disabled,
     and nothing bootstraps. The base graph is sitting in cache and canonical_root
     already resolves to it, but identity is still the getcwd() slug
     (detect_session() → cbm_project_name_from_path()), so there is no reuse.
   • auto_index=true: correct answers per worktree, but a full from-scratch index per
     worktree, and nothing GCs them when the worktree is removed. The watcher's stale-root
     pruning only covers projects that were registered via a session; a worktree indexed
     once and never reopened leaks its DB indefinitely.

   I switched to auto_index=true specifically because @joshweir's point above is the
   worse failure: identical relative paths mean a base-ref answer looks entirely plausible,
   so a review agent asking "who calls what I just changed" gets a pre-change call graph
   and reports no regressions. Paying 340 MB per worktree is preferable to a silently
   stale answer — but at 379 worktrees that tradeoff stops scaling.

   Nothing to add to the design; the merge-base overlay shape in the issue body addresses
   exactly this. Just adding the number, since "many short-lived worktrees" in the original
   framing turns out to be three orders of magnitude of index for one repo.
8. TerrifiedBug commented on Aug 18, 2026

   

   TerrifiedBug

   on Aug 18, 2026

   More actions

   Real-world confirmation of the "query ambiguity" bullet, with the cost measured

   This is the bullet in your list that bites hardest in practice — "query ambiguity when an MCP client is running inside a worktree but the canonical checkout is also indexed" — and I have a measurement for it from an autonomous fleet on v0.9.0 (Linux x64).

   Setup: a dispatcher gives each coding-agent attempt its own git worktree of the repository, and separately maintains a canonical index of the repo's default branch (4,935 nodes, 17,660 edges, HEAD equal to main, refreshed every 20 minutes on a timer). So a correct, fresh index for the code the agent is editing exists the entire time.

   Measured across the twelve most recent agent sessions:

   998  bash
   552  read
   376  edit
   234  grep
     2  glob
     0  graph calls
   

   Zero. And one capped session spent 586 tool calls — 216 edits against 264 read/grep discovery calls — before hitting its ceiling without ever querying the graph. Roughly half of every session's budget went to finding code by grep, which is precisely the work this server exists to make unnecessary.

   The mechanism is the one you describe: the agent's cwd is a worktree whose absolute path was never indexed, so a cwd-derived project resolves to nothing, and the agent falls back to grep rather than discovering that a canonical index exists under a different key. Nothing is broken — the tools behave as documented — but the default path for a worktree-based client is silent emptiness.

   A cheap intermediate step, well short of the full overlay model: when a client's cwd is a linked worktree, resolve the repository's canonical root (git rev-parse --path-format=absolute --git-common-dir, then its parent) and, if an index exists for that root, answer from it — ideally with a flag in the response saying the result reflects the base branch and not the worktree's own edits. That alone converts every one of my zero-graph sessions into a useful one. The overlay model is clearly the better end state, and the tombstone design reads right, but the base-index mapping is what unblocks worktree clients today, and it needs no new indexing work.

   For anyone hitting this before either lands, the workaround is to pass the canonical project key explicitly rather than letting it derive from cwd. Verified here: search_graph with the canonical project returned the target function with exact file and line range in one call, and trace_path for callers of another symbol correctly returned none — both against a worktree cwd. It works well; it just has to be known, which is the part a worktree client cannot discover on its own.
9. sinterqlaas2

   added a commit that references this issue on Sep 1, 2026

   feat(config): opt-in worktree_shared_index — one index per repository…

   ...

   7648f9d
10. DeusData

    added

    editor/integrationEditor compatibility and CLI integrationEditor compatibility and CLI integration

    stability/performanceServer crashes, OOM, hangs, high CPU/memoryServer crashes, OOM, hangs, high CPU/memory

    on Sep 5, 2026
11. DeusData commented on Oct 1, 2026

    

    DeusData

    on Oct 1, 2026

    Owner

    More actions

    Thank you for bringing the worktree history and overlay design together, and to everyone who added concrete measurements, query-selection examples, and smaller intermediate proposals. I appreciate your patience with the long wait, and the earlier maintainer encouragement deserves acknowledgment.

    Shared base indexes, committed-file overlays and tombstones, automatic canonical-base reuse, and per-call context selection remain design work. The operational examples are valuable input to that discussion. This closure does not establish that those capabilities have shipped or that a related PR implements the full proposal.

    We’re keeping Issues focused on actionable defects, so I’m closing this feature request administratively and inviting continued design discussion in GitHub Discussions. The proposal, measurements, and suggestions remain available here; there is no need to resubmit them. Adoption and implementation scope remain undecided.

    Thank you again for the care and detail in this discussion.
12. DeusData

    closed this as not plannedon Oct 1, 2026