Repository navigation
fix(ingest): tie each code graph to its root and prune graphs of deleted roots (#711, slice 1) - #869
Conversation
…ted roots (#711, slice 1) ingest_codebase now writes cortex-root.json (the resolved project root) next to the graph it builds, and prunes every graph under ~/.cache/cortex/code-graphs whose recorded root no longer exists. A directory without the marker is logged as unattributed and kept, never deleted on a guess. This replaces the name-prefix prune, which removed any sibling sharing the project name's first segment (cortex-viz-* when cortex ran) and never removed the graph of a deleted worktree, whose name differs. Refs #711 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYt8uiGG3xcmvaug2eG
…of deleted roots (#711, slice 1) Route the root marker through gated_files.write_json (ADR-1110). Check unit, run 41925545304: test_storage_redaction_gate found a raw write_text in infrastructure/code_graph_cache.py. A marker altered by redaction is removed, so the graph stays unattributed instead of being pruned. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYt8uiGG3xcmvaug2eG
|
Run locally, file by file: Generated by Claude Code |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
Security findingsAdvisory findings (1)
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Claude finished @cdeust's task in 2m 2s —— View job Review of PR #869: review 1 of 3 (ADR-1119)
There are no Blockers. Three Notes are posted under "Notes, no action before merge":
Verdict: approve |
Review 1 of 3 (ADR-1119): PR #869, commit 5ed7909I ran five reviews: code quality, security, performance, test coverage and documentation accuracy. None found a Blocker. Things I checked and found sound:
Notes, no action before merge
Verdict: approve |
|
The three notes from review 1 are left out of this PR, following ADR-1119: a Note does not justify a push.
Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ed7909d7c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # source: ADR-0399 | ||
| _prune_precedent_graphs(output_dir) | ||
| _record_and_prune_graphs(project_path, output_dir) |
There was a problem hiding this comment.
Record ownership at the actual graph location
When ensure_graph reuses a memoised graph, it returns the cached path without using the requested output_dir (ingest_codebase_graph.py, cached-return branch). Recording ownership against output_dir therefore attributes a directory that was not necessarily used: the actual graph can remain unmarked, or an existing graph belonging to another project can have its marker overwritten. In the latter case, deleting the newly recorded project root causes a later pruning pass to delete the other project's graph even though its root still exists. Pass the returned graph_path into the ownership-recording step and identify the directory containing that graph instead of assuming the requested output directory was used.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Traced on 5ed7909. In the default path I did not find a case where another project's graph gets deleted:
- With no
output_dirargument,output_dirisdefault_cache_dir()/project_key(project_path). The key hashes the resolved path, so this directory can only belong to this project. If a marker is written there, it names the right root. - In the cached branch,
find_cached_graphonly returns graphs memoised under this project's tag (code_graph_tag(project_key)). If that graph lives elsewhere, the defaultoutput_dirusually does not exist, theis_dir()check writes nothing, and the real graph stays unattributed, which means kept. That is the conservative case, not a deletion. - The scenario you describe needs a caller who explicitly passes as
output_dirthe cache directory of another project, while this project's cached graph is reused. Without the cache, the same argument already makes automatised-pipeline overwrite the other graph, so this is the explicit argument's own risk, not a new one.
The fix is still right in principle: mark the directory of the graph_path that is actually used. Under ADR-1119 this is not a Blocker (no CI, security or non-negotiable rule is broken in the default path), so it goes into the next slice of #711 rather than a push now. I am leaving the thread open for the owner's call.
Generated by Claude Code
There was a problem hiding this comment.
Addressed in 456b038. The marker is no longer written by the handler after ensure_graph. It is written in _call_analyze, right after upstream has built the graph in the output_dir it was given. A reused graph is never re-marked, so another project's marker can no longer be overwritten. Test: tests_py/handlers/test_ingest_codebase_graph_cache.py::test_analyze_call_records_the_root_of_the_graph_it_builds.
Generated by Claude Code
…the NullProgress test test_null_progress_does_not_alter_handler_result assigned icb.graphmod.ensure_graph and icb.cypher.fetch_files directly, so the real ingest_codebase_graph and ingest_codebase_cypher modules kept the fakes for the rest of the session. Run before test_ingest_codebase.py (the order mutmut's clean run uses), 8 handler tests failed, and the diff-scoped mutation gate stopped with "Failed to run clean test" (check `unit`, run 42051227576). The test now swaps the module references through monkeypatch, as the stage-ordering test above does. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYs2Ko7tuSGHN7Qfbf7
…of deleted roots (#711, slice 1)
|
Vérifié localement : l'ordre fautif passe (39 passed, 2 skipped sur les trois fichiers de test) ; Note : la branche a 14 commits de retard sur Generated by Claude Code |
|
Claude finished @cdeust's task in 1m 51s —— View job Review 2 of 3 (ADR-1119): PR #869
What the two commits do:
I could not run pytest or ruff in this job because those commands were not allowed. The Pipeline run was still going when I posted the review. Verdict: approve |
Review 2 of 3 (ADR-1119): PR #869, commits after 5ed7909Blockers from review 1: none were raised, so nothing needed fixing. Scope: only the commits pushed after 5ed7909, db4953b and 1922fb6. Both change tests only (+7 lines). I did not re-audit unchanged code, including the code behind the open Codex thread on I ran five reviews: code quality, security, performance, test coverage and documentation accuracy. None found a Blocker. What I traced:
Not verified here: I could not run the tests or ruff locally, because the job did not allow those commands. The Pipeline run for 1922fb6 was still running when I posted this. Verdict: approve |
…of deleted roots (#711, slice 1) Check unit, run 42051227576 (ADR-1122 mutation gate). Keep handler() byte-identical to main, so the gate does not charge this slice with the 182 pre-existing survivors of the whole handler. The marker is now written in _call_analyze (the graph Cortex asks upstream to build, which also answers the Codex P2 finding about the reused-graph path). _prune_precedent_graphs keeps its call site and prunes orphans beside output_dir. default_cache_dir is gone (no caller). Tests pin the marker, prune and log behaviour. Local ci_mutation_diff.py: 3 file(s), 0 failed, 0 untested. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYt8uiGG3xcmvaug2eG
|
Merged with
So
Run locally: Generated by Claude Code |
|
Claude finished @cdeust's task in 2m 13s —— View job Review 3 of 3 (ADR-1119): PR #869
What 456b038 changes:
Accepted: the whole PR. No file or commit carries an open Blocker. Not run: ruff and pytest, because this job did not allow those commands. The Pipeline run for 456b038 was still running when I posted the review. Verdict: approve |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 456b038fdd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "language": language, | ||
| }, | ||
| ) | ||
| code_graph_cache.record_root(output_dir, project_path) |
There was a problem hiding this comment.
Record ownership only after analysis succeeds
When an explicit output_dir already contains another project's graph and upstream returns a status: error payload without replacing it, this call still overwrites the existing ownership marker. ensure_graph checks the error only afterward, so the failed ingest leaves the original graph attributed to the wrong root. Removing that root then causes a later pruning pass to delete the original graph even while its actual project still exists. Normalize and validate the analysis response before recording ownership, preserving the existing marker on failure.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The trace is correct, but only on the explicit path. For the marker to name the wrong root, a caller must pass as output_dir a directory that holds another project's graph, and upstream must fail without replacing it. In the default path, output_dir is code-graphs/<project_key(project_path)>, so a marker written after a failure still names the right root.
The fix is to record ownership only after ensure_graph has validated the response. That moves the write into ensure_graph, and under ADR-1122 this PR would then own all of ensure_graph's mutants. This is the third and last review (ADR-1119), so the change goes into the follow-up slice of #711 (a refinement PR with its own budget) together with the guarded os.stat from review 1. Thread left open for the owner.
Generated by Claude Code
Review 3 of 3 (ADR-1119): PR #869, commit 456b038Blockers from reviews 1 and 2: none were raised, so nothing needed fixing. Scope: only 456b038, the one commit pushed after 1922fb6. I did not re-audit unchanged code. I ran five reviews: code quality, security, performance, test coverage and documentation accuracy. None found a Blocker. What I traced in 456b038:
Notes, no action before merge
Accepted: the whole PR, all 5 commits and 7 files. No file or commit carries an open Blocker. Not verified: I could not run ruff or pytest because this job does not allow those commands. The Pipeline run for 456b038 (42058709581) was still running when I posted this, so it is the receipt for the ADR-1122 mutation gate. Verdict: approve |
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: 456b038fdd
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
| elif not Path(root).exists(): | ||
| try: | ||
| shutil.rmtree(entry) |
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Security: Restrict pruning to the managed graph cache
Required conditions: the codebase upstream is enabled and a local MCP caller (including a content-influenced agent) can make two calls with chosen paths. _call_analyze writes cortex-root.json even after a returned status:error, so the first call can mark any existing user-writable directory using a nonexistent project_path; a later successful call with a sibling output_dir makes this loop trust the marker and rmtree that directory. Restrict pruning to the canonical cache, reject non-directory roots, and record only validated successful graph outputs. Cortex runs with the user's filesystem permissions.
SECURITY.md reference: SECURITY.md:L5-L16
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 551041a. The trace is right: a failed analyze wrote a marker in any output_dir with a nonexistent root, and a second call with a sibling output_dir would then rmtree that directory. Two guards:
_prune_precedent_graphsprunes only when the parent of the resolvedoutput_diris the managed cache~/.cache/cortex/code-graphs. An explicitoutput_direlsewhere never deletes its siblings.record_rootwrites no marker whenproject_rootis not an existing directory, so a failed call on a nonexistent path leaves no orphan marker behind.
Tests: test_prune_never_touches_siblings_outside_the_managed_cache and test_record_root_refuses_a_root_that_is_not_a_directory. Local ci_mutation_diff.py (main's scripts): 3 file(s), 0 failed, 0 untested.
Process note: GitHub lost the push event, so the PR stayed on 456b038. I closed and reopened the PR to resync the head (autopilot rule 10).
Generated by Claude Code
…of deleted roots (#711, slice 1) Codex security review on 456b038 (P2, Medium): a failed analyze could mark any user-writable output_dir with a nonexistent root, and a second call with a sibling output_dir would then rmtree it. Pruning is now limited to the managed cache (~/.cache/cortex/code-graphs), and no marker is written for a root that is not an existing directory. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYt8uiGG3xcmvaug2eG
|
CI autopilot : branche en retard de 19 commits sur
Generated by Claude Code |
…ests nothing (#711, slice 2) Slice 1 (#869) pruned graphs whose recorded root is gone, but only after ingest_codebase had built a graph. A session with a fresh graph, no pipeline or a failed analyze left every deleted worktree's graph (0.6 to 1.1 GB each, measured 2026-10-10) on disk. The SessionStart hook now reads the cortex-root.json markers of the managed cache (~/.cache/cortex/code-graphs) and, when at least one names a root that no longer exists, starts the detached code_graph_prune_background worker, so the removal never runs inside the hook's time budget. A directory without a marker is still reported and kept. Refs #711 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYsdTexaL2aS5FFzdeC
Summary
Refs #711 (slice 1 of 4).
ingest_codebasenow ties each graph under~/.cache/cortex/code-graphs/to the project root it was built from, and removes the graphs whose root no longer exists.Root cause. Nothing recorded which project a graph directory belongs to. The only cleanup,
_prune_precedent_graphs, matched siblings by name prefix: a deleted worktree's graph (fix-665-windows-registry-44d852f6) never matched the prefix of a session that was still running, so it stayed on disk forever. The same rule also deleted unrelated projects: forcortex-ab12cd34the prefix iscortex-, which matchescortex-viz-*.Change.
mcp_server/infrastructure/code_graph_cache.py:write_root_markerwritescortex-root.json({"project_root": <resolved path>}, written atomically) andprune_orphan_graphsremoves each graph whose recorded root is gone. A directory with no readable marker is logged as unattributed and kept, never deleted on a guess.ingest_codebasewrites the marker intooutput_dironce a graph exists there, then prunes the default cache. The name-prefix prune is removed.meta.json:meta.jsonis written by automatised-pipeline (ADR-0052, "for cortex-viz"), and its schema is not defined in this repository. Cortex does not read or rewrite a file owned by another tool.Remaining slices of #711 (not in this PR):
2. Bound the total size, with an eviction rule that never touches a graph in use.
3. Prune when a worktree is removed (hook), not only on the next ingest.
4. Let a worktree session reuse the parent project's current graph, or ingest it incrementally, instead of building a full one.
The ~43 existing directories with no marker are reported, not migrated. Tying them to a root would mean guessing from
meta.json, which is why it needs an owner decision.Type of change
Test plan
tests_py/infrastructure/test_code_graph_cache.py, 7 passed. It covers: deleted root → pruned; live root → kept; no marker → reported and kept (the warning names the directory); unreadable marker → unattributed;cortex/cortex-vizprefix collision → untouched; missing cache dir → empty report.tests_py/test_ingest_progress.py14 passed,tests_py/handlers/test_ingest_codebase.py16 passed and 2 skipped,tests_py/hooks/test_ingest_codebase_background.py3 passed,tests_py/scripts/test_composition_root_wiring_coverage.py6 passed,tests_py/infrastructure/test_pipeline_discovery.py30 passed,tests_py/scripts/test_craftsmanship_layer_table.py12 passed.ruff check+ruff format --checkon the changed files (system ruff and theuv.lockversion): clean.python scripts/check_craftsmanship.py: OK.python scripts/check_doc_claims.py: OK.test_ingest_codebase.pyfails 8 tests when it runs in the same process aftertest_ingest_progress.py. It is identical onorigin/main(stash check: 8 failed there too).Not verified: the full
pytestsuite (the local env was installed withuv pip, notuv sync --locked, because the proxy blocked the torch CPU index) and a real end-to-end ingest against an automatised-pipeline binary. CI is the receipt.Audit notes
handlers → infrastructure, andinfrastructureimports stdlib only. The craftsmanship gate passes.Breaking changes
Graph directories that share a name prefix with the current project are no longer deleted. A legacy directory with no marker is now reported in the log, not removed.
Reviewer checklist
docs/module-inventory.mdlists the new module.Generated by Claude Code