Skip to content

Commit 5cc4524

Browse files
committed
feat(backup): a versioned manifest digests every class A file of an install (#731, slice 2)
The manifest names the class A resources of docs/audits/owned-resources-backup-2026-11-05.md by backend and gives a SHA-256 digest and a size per file. A required store that is absent, a symbolic link or a special file is refused, never skipped. Refs #731 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYt9sk9gsvUQZCwMbuF
1 parent fc13eb1 commit 5cc4524

9 files changed

Lines changed: 759 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@ adheres to [Semantic Versioning](https://semver.org/).
1818

1919
### Added
2020

21+
- A versioned backup manifest (`cortex-backup-manifest`, version 1) names the class A
22+
resources of an install by backend (SQLite store, or managed cluster and its
23+
password marked secret; wiki, artifacts, profiles, session log, MCP connections,
24+
capture spool) with a SHA-256 digest and a size for every file. A required store
25+
that is absent, a symbolic link or a special file is refused, never skipped
26+
(#731, slice 2).
2127
- A benchmark run's `MANIFEST.json` now carries `runtime_artifact`: the `mcp_server`
2228
artifact the run imported, as `cortex doctor` names it (installed wheel or source
2329
tree, its revision, uncommitted changes, one digest over its members), with

‎docs/audits/owned-resources-backup-2026-11-05.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,9 @@ not need it). Class **X** = not Cortex's to back up.
108108

109109
## 5. Remaining slices of #731
110110

111-
1. A versioned manifest listing the class A entries above with a digest each.
111+
1. A versioned manifest listing the class A entries above with a digest each
112+
(delivered: `mcp_server/core/backup_manifest.py`, `infrastructure/backup_scan.py`,
113+
`handlers/backup_manifest.py`).
112114
2. A consistent snapshot of each store under concurrent writes (SQLite online
113115
backup; managed PostgreSQL stopped copy or logical dump).
114116
3. Restore into a fresh `CORTEX_CLAUDE_DIR`, then verify integrity and a known

‎docs/module-inventory.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,9 @@ not exist." The registered tools are listed in `docs/mcp-tools.md`.
194194
- `confluence_parser.py` — Pure Confluence storage-format XHTML parser (entities + ac:/ri: namespaces pre-resolved) → `ParsedDocument`
195195
- `document_normalizer.py` — `ParsedDocument` + provenance → wiki page markdown + memory payloads (skipped-image/empty notices, provenance stamping)
196196

197+
*Backup (issue #731):*
198+
- `backup_manifest.py` — Versioned backup manifest: the class A resource table per backend (audit 2026-11-05), build from a scan, canonical JSON, parse refusing another format, version or table
199+
197200
*Undocumented (measured, not yet catalogued individually):* `core/streaming/`
198201
(5 files) and `core/context_assembly/` (14 files — 15 as of the issue #228
199202
split, minus `active_retrieval.py`, deleted by issue #201 as unwired dead
@@ -204,6 +207,7 @@ control/writing, backpressure, and other mechanisms not yet described above.
204207
Run the measurement command in the header to get a current file listing.
205208

206209
## infrastructure/ — All I/O
210+
- `backup_scan.py` — Reads one backup resource under the Cortex root and digests every file (SHA-256, size); refuses symlinks and special files (issue #731)
207211

208212
- `config.py` — Centralized path constants via pathlib. The root (`CLAUDE_DIR`,
209213
default `~/.claude`) is overridable via `CORTEX_CLAUDE_DIR`; every other

‎mcp_server/core/backup_manifest.py‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
"""The versioned manifest of a Cortex backup (issue #731, slice 2).
2+
3+
A manifest names the class A resources of one install (user data that cannot be
4+
rebuilt) and gives a SHA-256 digest and a size for every file it carries. The
5+
resource table is the class A set of
6+
``docs/audits/owned-resources-backup-2026-11-05.md`` §2-3, by backend:
7+
8+
- every backend: backend marker, wiki pages, raw-output artifacts, MCP
9+
connection list, cognitive profiles (``domains/``, ``index.json``, legacy
10+
``profiles.json*``), session log, capture spool;
11+
- ``sqlite``: ``methodology/memory.db`` (required);
12+
- ``managed``: the plugin-managed cluster and its password (both required; the
13+
password is marked ``secret``);
14+
- ``postgresql``: nothing more, the server is the user's (class X).
15+
16+
Paths are POSIX strings relative to the Cortex root (``~/.claude`` or
17+
``CORTEX_CLAUDE_DIR``). A store relocated outside the root
18+
(``CORTEX_MEMORY_DB_PATH``) is therefore absent, and a required absent resource
19+
is refused, never silently omitted. The SQLite ``-wal``/``-shm`` files are not
20+
entries: a backup carries one consistent copy of the database (slice 3), and the
21+
manifest describes the tree it is given.
22+
23+
Pure: the files are read and hashed by ``infrastructure/backup_scan.py``.
24+
25+
source: issue #731; docs/audits/owned-resources-backup-2026-11-05.md
26+
"""
27+
28+
from __future__ import annotations
29+
30+
import json
31+
import re
32+
from dataclasses import dataclass
33+
from typing import Any, Mapping, Sequence
34+
35+
FORMAT = "cortex-backup-manifest"
36+
# source: issue #731 slice 2, the first format; bump on any change to the
37+
# resource table or to the entry and file fields below.
38+
VERSION = 1
39+
40+
SQLITE = "sqlite"
41+
MANAGED = "managed"
42+
POSTGRESQL = "postgresql"
43+
# The marker values of infrastructure/backend_marker.py MARKER_BACKENDS.
44+
BACKENDS = frozenset({SQLITE, MANAGED, POSTGRESQL})
45+
46+
FileRecord = tuple[str, str, int] # (path relative to the root, sha256 hex, size)
47+
48+
_SHA256_HEX = re.compile(r"[0-9a-f]{64}")
49+
50+
51+
class ManifestError(ValueError):
52+
"""A manifest cannot be built or read as asked; the message names why."""
53+
54+
55+
@dataclass(frozen=True)
56+
class ResourceSpec:
57+
"""One class A resource: where it lives under the root and for which backends."""
58+
59+
name: str
60+
path: str
61+
kind: str # "file" or "dir"
62+
backends: frozenset[str]
63+
required: bool
64+
secret: bool = False
65+
66+
67+
RESOURCES: tuple[ResourceSpec, ...] = (
68+
ResourceSpec(
69+
"sqlite_store", "methodology/memory.db", "file", frozenset({SQLITE}), True
70+
),
71+
ResourceSpec(
72+
"managed_cluster",
73+
"methodology/managed-pg/cluster",
74+
"dir",
75+
frozenset({MANAGED}),
76+
True,
77+
),
78+
ResourceSpec(
79+
"managed_password",
80+
"methodology/managed-pg/password",
81+
"file",
82+
frozenset({MANAGED}),
83+
True,
84+
secret=True,
85+
),
86+
ResourceSpec("backend_marker", "methodology/backend.json", "file", BACKENDS, False),
87+
ResourceSpec("wiki", "methodology/wiki", "dir", BACKENDS, False),
88+
ResourceSpec("artifacts", "methodology/artifacts", "dir", BACKENDS, False),
89+
ResourceSpec(
90+
"mcp_connections", "methodology/mcp-connections.json", "file", BACKENDS, False
91+
),
92+
ResourceSpec("profile_domains", "methodology/domains", "dir", BACKENDS, False),
93+
ResourceSpec("profile_index", "methodology/index.json", "file", BACKENDS, False),
94+
ResourceSpec(
95+
"legacy_profiles", "methodology/profiles.json", "file", BACKENDS, False
96+
),
97+
ResourceSpec(
98+
"legacy_profiles_backup",
99+
"methodology/profiles.json.v1_backup",
100+
"file",
101+
BACKENDS,
102+
False,
103+
),
104+
ResourceSpec(
105+
"session_log", "methodology/session-log.json", "file", BACKENDS, False
106+
),
107+
ResourceSpec("capture_spool", ".capture-worker/spool", "dir", BACKENDS, False),
108+
)
109+
110+
111+
def resources_for(backend: str) -> tuple[ResourceSpec, ...]:
112+
"""The class A resources a ``backend`` install carries, in table order."""
113+
if backend not in BACKENDS:
114+
raise ManifestError(
115+
f"unknown backend {backend!r}; expected one of {sorted(BACKENDS)}"
116+
)
117+
return tuple(spec for spec in RESOURCES if backend in spec.backends)
118+
119+
120+
def _checked_files(
121+
spec: ResourceSpec, records: Sequence[FileRecord]
122+
) -> list[dict[str, Any]]:
123+
"""One resource's records as manifest files: sorted, each inside it."""
124+
if spec.kind == "file" and [r[0] for r in records] != [spec.path]:
125+
raise ManifestError(f"{spec.path} is a file resource: one record, its own path")
126+
files = []
127+
for path, sha256, size in sorted(records):
128+
if spec.kind == "dir" and not path.startswith(spec.path + "/"):
129+
raise ManifestError(f"{path} is not inside {spec.path}")
130+
if not _SHA256_HEX.fullmatch(sha256) or size < 0:
131+
raise ManifestError(f"{path}: invalid digest or size")
132+
files.append({"path": path, "sha256": sha256, "size": size})
133+
return files
134+
135+
136+
def _entry(
137+
spec: ResourceSpec, records: Sequence[FileRecord] | None, backend: str
138+
) -> dict:
139+
if records is None and spec.required:
140+
raise ManifestError(
141+
f"{spec.path} is required for a {backend} backup and is absent"
142+
)
143+
return {
144+
"name": spec.name,
145+
"path": spec.path,
146+
"kind": spec.kind,
147+
"secret": spec.secret,
148+
"present": records is not None,
149+
"files": [] if records is None else _checked_files(spec, records),
150+
}
151+
152+
153+
def build_manifest(
154+
backend: str,
155+
scanned: Mapping[str, Sequence[FileRecord] | None],
156+
created_at: str,
157+
) -> dict[str, Any]:
158+
"""The manifest of one install from its scan.
159+
160+
``scanned`` maps every resource name of ``resources_for(backend)`` to its
161+
file records, or to ``None`` when the resource is absent; a missing or an
162+
extra name is refused, so a scan cannot leave a resource out unnoticed.
163+
"""
164+
specs = resources_for(backend)
165+
expected = {spec.name for spec in specs}
166+
if set(scanned) != expected:
167+
raise ManifestError(
168+
f"scan names {sorted(set(scanned))}, "
169+
f"the {backend} table names {sorted(expected)}"
170+
)
171+
return {
172+
"format": FORMAT,
173+
"version": VERSION,
174+
"created_at": created_at,
175+
"backend": backend,
176+
"entries": [_entry(spec, scanned[spec.name], backend) for spec in specs],
177+
}
178+
179+
180+
def to_json(manifest: Mapping[str, Any]) -> str:
181+
"""Canonical text: ASCII, sorted keys, two-space indent, one trailing newline."""
182+
return json.dumps(manifest, sort_keys=True, indent=2) + "\n"
183+
184+
185+
def parse_manifest(text: str) -> dict[str, Any]:
186+
"""Read a manifest; refuse another format, another version or another table."""
187+
try:
188+
manifest = json.loads(text)
189+
except json.JSONDecodeError as exc:
190+
raise ManifestError(f"not JSON: {exc}") from exc
191+
if not isinstance(manifest, dict) or manifest.get("format") != FORMAT:
192+
raise ManifestError(f"not a {FORMAT} document")
193+
if manifest.get("version") != VERSION:
194+
raise ManifestError(
195+
f"manifest version {manifest.get('version')!r}, this Cortex reads {VERSION}"
196+
)
197+
specs = resources_for(manifest.get("backend"))
198+
entries = manifest.get("entries")
199+
if not isinstance(entries, list) or not all(isinstance(e, dict) for e in entries):
200+
raise ManifestError("entries must be a list of objects")
201+
names = [entry.get("name") for entry in entries]
202+
if names != [spec.name for spec in specs]:
203+
raise ManifestError(
204+
f"entries {names} do not match the {manifest['backend']} table"
205+
)
206+
return manifest
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
"""Build the backup manifest of one Cortex root (issue #731, slice 2).
2+
3+
Composition root: the resource table and the format come from
4+
``core/backup_manifest.py``, the files are read by
5+
``infrastructure/backup_scan.py``. No tool is registered yet; the consistent
6+
snapshot and the restore check (later slices of #731) call this.
7+
8+
source: issue #731; docs/audits/owned-resources-backup-2026-11-05.md
9+
"""
10+
11+
from __future__ import annotations
12+
13+
from datetime import datetime, timezone
14+
from pathlib import Path
15+
from typing import Any
16+
17+
from mcp_server.core.backup_manifest import build_manifest, resources_for
18+
from mcp_server.infrastructure.backup_scan import scan_resource
19+
20+
21+
def backup_manifest(
22+
root: Path, backend: str, now: datetime | None = None
23+
) -> dict[str, Any]:
24+
"""The manifest of ``root`` for a ``backend`` install, each file digested."""
25+
specs = resources_for(backend)
26+
scanned = {spec.name: scan_resource(root, spec.path, spec.kind) for spec in specs}
27+
created_at = (now or datetime.now(timezone.utc)).isoformat(timespec="seconds")
28+
return build_manifest(backend, scanned, created_at)
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
"""Read the files of one backup resource and digest each (issue #731, slice 2).
2+
3+
``scan_resource`` returns the ``(path, sha256, size)`` records the pure manifest
4+
(``core/backup_manifest.py``) is built from, paths POSIX and relative to the
5+
root. A resource that does not exist is ``None``. A symbolic link, or anything
6+
that is neither a regular file nor a directory, is refused: following a link
7+
would digest data outside the root, and skipping it would leave a file out of
8+
the backup unnoticed.
9+
10+
source: issue #731; docs/audits/owned-resources-backup-2026-11-05.md
11+
"""
12+
13+
from __future__ import annotations
14+
15+
import hashlib
16+
import os
17+
import stat
18+
from pathlib import Path
19+
20+
21+
class BackupScanError(OSError):
22+
"""A resource holds something a backup cannot carry as a plain file."""
23+
24+
25+
def _record(root: Path, path: Path) -> tuple[str, str, int]:
26+
with path.open("rb") as handle:
27+
size = os.fstat(handle.fileno()).st_size
28+
digest = hashlib.file_digest(handle, hashlib.sha256).hexdigest()
29+
return path.relative_to(root).as_posix(), digest, size
30+
31+
32+
def _checked_mode(path: Path) -> int:
33+
mode = path.lstat().st_mode
34+
if stat.S_ISLNK(mode):
35+
raise BackupScanError(
36+
f"{path} is a symbolic link; a backup does not follow links"
37+
)
38+
if not (stat.S_ISREG(mode) or stat.S_ISDIR(mode)):
39+
raise BackupScanError(f"{path} is neither a regular file nor a directory")
40+
return mode
41+
42+
43+
def _walk(root: Path, directory: Path) -> list[tuple[str, str, int]]:
44+
records = []
45+
for child in sorted(directory.iterdir()):
46+
if stat.S_ISDIR(_checked_mode(child)):
47+
records.extend(_walk(root, child))
48+
else:
49+
records.append(_record(root, child))
50+
return records
51+
52+
53+
def scan_resource(
54+
root: Path, relative: str, kind: str
55+
) -> list[tuple[str, str, int]] | None:
56+
"""The records of ``root/relative``; ``None`` when it does not exist.
57+
58+
``kind`` is ``"file"`` or ``"dir"``; a resource of the other kind is refused.
59+
"""
60+
target = root / relative
61+
if not target.exists() and not target.is_symlink():
62+
return None
63+
is_dir = stat.S_ISDIR(_checked_mode(target))
64+
if is_dir != (kind == "dir"):
65+
raise BackupScanError(f"{target} is expected to be a {kind}")
66+
return _walk(root, target) if is_dir else [_record(root, target)]

0 commit comments

Comments
 (0)