docs(audit): what a Cortex backup must contain (#731, slice 1) by cdeust · Pull Request #892 · cdeust/Cortex

cdeust · 2026-11-05T17:30:41Z

Summary

First slice of #731 (verified portable backup and restore). The issue's first acceptance item is "define included/excluded owned resources". This PR is that inventory, with no code change: docs/audits/owned-resources-backup-2026-11-05.md.

Root cause of the gap the issue describes, cited at main 8ef4813:

Cortex already names its personal-data roots (private_roots() / private_files() in mcp_server/shared/private_paths.py:67-74), but nothing turns that list into a backup: no handler under mcp_server/handlers/ exports or restores a store as a whole, and the only backup check in the tree is the pg_dump precondition of scripts/reclassify_team_scope_db.py:12-22.
The inventory classes every file and store (SQLite file with -wal/-shm, managed PostgreSQL cluster and its password, backend marker, wiki pages, artifacts, profiles, capture spool, telemetry, logs, locks, caches, code graphs) as user data, derived data, process state, or not Cortex's, and gives the set a backup carries per backend.
Findings: the code graphs (ingest_codebase.py:108), the consolidation stamp (consolidate_background.py:37) and the pipeline install (pipeline_discovery.py:75-78, pipeline_install_lock.py:15) are built from the home directory and do not follow CORTEX_CLAUDE_DIR; the code graphs also ignore CORTEX_CACHE_DIR/XDG_CACHE_HOME; the wiki lives both as files and as tables, so a snapshot must take both at once.

Refs #731, not Closes.

Type of change

Documentation (audit)

Test plan

python scripts/check_doc_claims.py → doc claims OK; python scripts/check_craftsmanship.py → Craftsmanship gate: OK.
pytest tests_py/scripts/test_check_doc_claims.py and the document-ingest tests → 194 passed.
Every cited line re-read with sed -n before commit.
No test added: the PR changes no behaviour.

Not verified: nothing was run against a user datastore; the statement that a file copy of a running managed cluster is not consistent is inferred from PostgreSQL's file-level backup rules, not reproduced. ruff check on the whole tree reports 8 E501 in files this PR does not touch (scripts/ci_pipeline_source_contract.py, scripts/tests/test_no_main_comparison.py, tests_py/scripts/test_ci_review_required.py), already present on main 8ef4813.

Remaining slices of #731

Versioned manifest of the class A entries with a digest each.
Consistent snapshot per store under concurrent writes.
Restore into a fresh CORTEX_CLAUDE_DIR, then integrity and known-memory/source queries.
Refusal of an incomplete or incompatible archive, target left intact.
Tested recovery and upgrade-rollback documentation per platform.

Generated by Claude Code

Inventory every file and store a Cortex install writes, with the cited line that names its path, classed as user data, derived data, process state or not Cortex's, and the set a backup carries per backend. Finds that the code graphs, the consolidation stamp and the pipeline install are built from the home directory and do not follow CORTEX_CLAUDE_DIR, and that the code graphs ignore the cache override. Refs #731 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYqkZyf6rNbyAG8gFua

…ce-inventory # Conflicts: # CHANGELOG.md

chatgpt-codex-connector · 2026-11-06T13:50:59Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-06T14:21:55.169927Z	`f016253`	New commits
🔒 Security Review	✅ Completed 2026-11-06T14:22:44.897918Z	`f016253`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

claude · 2026-11-06T13:51:45Z

Claude finished @cdeust's task in 2m 37s —— View job

Review 1 of 3 — done

Read the diff (documentation only: the CHANGELOG entry and the audit doc)
Ran five reviews: code quality, documentation accuracy, security, performance, test coverage
Checked the key cited lines myself
Posted the review: docs(audit): what a Cortex backup must contain (#731, slice 1) #892 (comment)

Result: no Blockers, 3 Notes. All the citations I checked hold up. The Notes:

mcp-connections.json can contain credentials.
The inventory is missing a few files, including .last_consolidate, /tmp/cortex_reindex_cooldown.json and some managed-pg/ files.
The finding on backup tools could mention the existing per-project memory export.

I couldn't run check_doc_claims.py or check_craftsmanship.py because running them needed approval this session didn't have.

Verdict: approve

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58a78d36d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-06T13:53:09Z

+| Backend marker (which store this install uses) | `<root>/methodology/backend.json` | A | `mcp_server/infrastructure/backend_marker.py:48` |
+| Wiki pages on disk | `<root>/methodology/wiki/` | A | `mcp_server/infrastructure/config.py:42`; `mcp_server/handlers/ingest_findings_writers.py:173` |
+| Raw-output artifacts | `<root>/methodology/artifacts/` | A | `mcp_server/infrastructure/artifact_store.py:19` |
+| Cognitive profiles | `<root>/methodology/domains/<id>.json`, `index.json`; legacy `profiles.json`, `profiles.json.v1_backup` | D (rebuilt from transcripts by `rebuild_profiles`) | `mcp_server/infrastructure/profile_store.py:20-22`, `:49`; `mcp_server/infrastructure/config.py:38` |


Include accumulated cognitive profiles in the class A backup set

For installs whose older transcripts have expired, these profiles cannot be rebuilt by rebuild_profiles. record_session_end._update_profile() incrementally updates the persisted session count and cognitive-style EMA, and mcp_server/core/profile_rebuild_policy.py explicitly preserves stored profiles when the remaining transcripts contain fewer sessions because that accumulated state is unrecoverable. Classifying the profiles as D excludes them from section 3's backup set and the planned class-A-only manifest, so a restore following this inventory would permanently lose learned profile history. Classify the persisted profiles and their index as A and include them in the per-backend backup sets.

Useful? React with 👍 / 👎.

Confirmed and fixed in 6fd8147. record_session_end._update_profile (record_session_end.py:265-287) accumulates each session into the stored profile, and profile_rebuild_policy.py:1-15 (ADR-1099) keeps that profile because expired transcripts cannot rebuild it. The profiles (domains/, index.json, legacy profiles.json*) are now class A and in the §3 backup set. session-log.json had the same flaw (one entry appended per ended session, record_session_end.py:257/:322, session_lifecycle.py:254/:356), so it moved to class A in the same commit.

Generated by Claude Code

claude · 2026-11-06T13:54:22Z

Review 1 of 3 — PR #892

I ran five reviews: code quality, documentation accuracy, security, performance and test coverage. None found a Blocker. The diff changes only documentation (the audit doc and a CHANGELOG entry), so it can't break CI. I found no false statement about security or behaviour.

The documentation review checked every path:line citation against HEAD 58a78d3. Each one supports the claim next to it, including findings 1–4 (Path.home() at consolidate_background.py:37 and ingest_codebase.py:108, home_dir() in pipeline_discovery.py and pipeline_install_lock.py). The CHANGELOG entry sits under [Unreleased] → Added. I didn't run check_doc_claims.py or check_craftsmanship.py here because this session wasn't allowed to run them. Neither script scans docs/audits/ Markdown anyway.

Notes, no action before merge

mcp-connections.json can hold a literal token in its env block, because only ${VAR} values are resolved (mcp_client_pool.py:44-53). Section 3 says only the password file needs protecting; it is worth saying the whole archive should be owner-only.
The inventory is incomplete, though the CHANGELOG says it "lists every file":
- .last_consolidate has no row in section 2, so the claim in finding 1 that it is class S, X or D can't be checked against the table.
- /tmp/cortex_reindex_cooldown.json (post_commit_reindex.py:56) is missing.
- So are the other managed-pg/ files: installed.json, provisioned.json, the two logs and the locks (managed_pg.py:81-100).
Finding 4 could name handlers/memory_portability.py, which exports one project's memories, and say why that is not a store backup.

Verdict: approve

claude · 2026-11-06T13:54:35Z

Claude finished @cdeust's task in 2m 23s —— View job

I'll analyze this and get back to you.

claude · 2026-11-06T13:57:05Z

Review 1 of 3 — PR #892 (head `6fd8147`)

Five reviews were run: two checked citations, one checked security, one checked quality, and one checked tests and CI. This PR changes only docs (docs/audits/owned-resources-backup-2026-11-05.md and CHANGELOG.md).

Blockers: none.

Citations: each file:line in sections 1–4 was opened in the tree, and every one supports its claim. Findings 1–3 are confirmed in code: consolidate_background.py:37 and ingest_codebase.py:108 use Path.home(), the pipeline paths use home_dir(), and the code-graph path ignores cortex_cache_dir().
Earlier Codex P2 (profiles classed D): already fixed in 6fd8147. Profiles are now class A, cite record_session_end.py:265-287 and profile_rebuild_policy.py:1-15, and appear in the section 3 set.
CI: no gate reads docs/audits/* or CHANGELOG.md. check_doc_claims.py only scans its SCANNED_FILES. The CHANGELOG entry sits under [Unreleased] with the other audit entries.

Notes, no action before merge

mcp-connections.json can hold literal credentials in a server's env block, and ${VAR} substitution is optional (mcp_client_pool.py:44-55). It deserves the same "secret" handling as the managed-pg password.
The brain index is classed D, but nothing in mcp_server/ writes or rebuilds brain-index.json. Its only access is the reader brain_index_store.py:11-19, so it is input Cortex reads, not derived output.
.last_consolidate has no row in section 2, yet Finding 1 assigns it a class. managed-pg/provisioned.json, installed.json and postgres.log (managed_pg.py:81-100) are not classified either; slice 1 of the manifest will need them.

View job run

Verdict: approve

…ce-inventory # Conflicts: # CHANGELOG.md

claude · 2026-11-06T14:21:05Z

Claude finished @cdeust's task in 2m 26s —— View job

Review 3 of 3 — done

Checked the earlier Blockers: reviews 1 and 2 raised none.
Listed the commits after 6fd8147. There is only one, merge f016253 from origin/main. It leaves the audit doc, the PR's CHANGELOG entry and every file the audit cites unchanged.
Ran five reviews: documentation accuracy, code quality, security, performance and test/CI.
Posted the review: docs(audit): what a Cortex backup must contain (#731, slice 1) #892 (comment)

Result: no Blockers. The whole PR is accepted.

Note: [Unreleased] has two ### Added headings. They are already on main, not added by this PR.

Verdict: approve

claude · 2026-11-06T14:23:32Z

Review 3 of 3 — PR #892 (head `f016253`)

Earlier Blockers: reviews 1 and 2 raised none.

New since 6fd8147: one commit, merge f016253 of origin/main. It brings in 3 files from main: the #733 and #730 audit docs and their CHANGELOG entries. It leaves docs/audits/owned-resources-backup-2026-11-05.md unchanged (git diff 6fd8147 HEAD on that file is empty). It also leaves every file the audit cites unchanged, including mcp_server/, scripts/ and PRIVACY.md, so the citations still point at the right lines. Against origin/main the PR is still 2 Markdown files, +123/−0.

Five reviews were run: documentation accuracy, code quality, security, performance and test/CI. None found a Blocker.

No conflict markers in CHANGELOG.md or docs/audits/*.md. The audit table is intact.
The feat: provide a verified portable backup and restore journey for local memory #731 CHANGELOG entry appears once, under ## [Unreleased] → ### Added.
No gate reads CHANGELOG.md or docs/audits/ (check_doc_claims.py:41-71 SCANNED_FILES), and no runtime path changes.

Notes, no action before merge

[Unreleased] has two ### Added headings (lines 9 and 231). They are already on main, not added by this PR.

Accepted: the whole PR: CHANGELOG.md and docs/audits/owned-resources-backup-2026-11-05.md. No file or commit carries an open Blocker.

Verdict: approve

cdeust mentioned this pull request Nov 5, 2026

feat: provide a verified portable backup and restore journey for local memory #731

Open

5 tasks

Merge remote-tracking branch 'origin/main' into fix/731-backup-resour…

58a78d3

…ce-inventory # Conflicts: # CHANGELOG.md

cdeust marked this pull request as ready for review November 6, 2026 13:50

chatgpt-codex-connector Bot reviewed Nov 6, 2026

View reviewed changes

fixup! docs(audit): what a Cortex backup must contain (#731, slice 1)

6fd8147

Merge remote-tracking branch 'origin/main' into fix/731-backup-resour…

f016253

…ce-inventory # Conflicts: # CHANGELOG.md

github-actions Bot merged commit a26fe12 into main Nov 6, 2026
18 checks passed

cdeust mentioned this pull request Nov 10, 2026

feat(backup): a versioned manifest digests every class A file of an install (#731, slice 2) #960

Merged

11 of 12 tasks

Navigation Menu

Repository navigation

docs(audit): what a Cortex backup must contain (#731, slice 1) - #892

docs(audit): what a Cortex backup must contain (#731, slice 1)#892
github-actions[bot] merged 4 commits into
mainfrom
fix/731-backup-resource-inventory

cdeust commented Nov 5, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Nov 6, 2026

Uh oh!

cdeust Nov 6, 2026

Uh oh!

claude Bot commented Nov 6, 2026

Uh oh!

claude Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026

Uh oh!

claude Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

claude Bot commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cdeust commented Nov 5, 2026

Summary

Type of change

Test plan

Remaining slices of #731

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

claude Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review 1 of 3 — done

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

cdeust Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Nov 6, 2026

Review 1 of 3 — PR #892

Uh oh!

claude Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Nov 6, 2026

Review 1 of 3 — PR #892 (head 6fd8147)

Uh oh!

claude Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review 3 of 3 — done

Uh oh!

claude Bot commented Nov 6, 2026

Review 3 of 3 — PR #892 (head f016253)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chatgpt-codex-connector Bot commented Nov 6, 2026 •

edited

Loading

claude Bot commented Nov 6, 2026 •

edited

Loading

claude Bot commented Nov 6, 2026 •

edited

Loading

Review 1 of 3 — PR #892 (head `6fd8147`)

claude Bot commented Nov 6, 2026 •

edited

Loading

Review 3 of 3 — PR #892 (head `f016253`)