Skip to content

Commit 57cf06f

Browse files
committed
feat(ai-backup): doc-backed manifest scope for AI-state backup
Redefine what enters the mact2->rog backup per documentation research (code.claude.com docs, opencode.ai docs, anomalyco/opencode#13654, sqlite.org/backup.html, libarchive suffix-matching findings): - explicit member list, zero --exclude patterns — no portable exclude syntax exists (bsdtar suffix-matches unanchored patterns; GNU tar reads ^ as a literal), so the enumerated list is the auditable contract on both tar flavors - add $HOME/.claude.json (sibling, may hold OAuth), nested .claude/ tree (real transcripts inside), opencode legacy storage/ (only copy of pre-SQLite sessions per #13654), opencode-multimodal.json, and AI_BACKUP_EXTRA for project .engram/ dirs (restored under ~/extras/<project>/.engram/ to spare the global DB) - drop .config/opencode (HM-generated), Claude HM-deployed assets, caches/ephemeral env-snapshot dirs, opencode bin/log/snapshot dirs - snapshot staging as <real-basename>.db.snapshot with count-based gate (exit 3 unless >=1 non-empty OpenCode snapshot); restore strips the suffix, resolves destination symlinks before moving (never mv onto a live symlink), keeps .pre-restore-<ts> copies, and runs PRAGMA integrity_check per restored DB - COPYFILE_DISABLE=1 suppresses macOS xattr headers (no-op on GNU) - docs/ai-backup.md runbook (security, commands, citations, Keychain re-login, limitations) - openspec change ai-backup-manifest-scope: full SDD artifacts Verified: go test ./... (13 tests), go vet, /bin/sh -n payloads, format-nix && nix flake check --no-build; real-run gate ai-backup-mact2-20260908-221928.tar.zst (237M/433s) — sha256 OK, member presence/absence asserted, extracted snapshot integrity ok.
1 parent fc951c5 commit 57cf06f

8 files changed

Lines changed: 1326 additions & 114 deletions

File tree

‎docs/ai-backup.md‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# AI Backup Runbook (`ai-backup`)
2+
3+
One-shot compressed backup/restore of AI-assistant state (Claude Code,
4+
OpenCode, Engram) as a single `.tar.zst` archive, with consistent SQLite
5+
snapshots taken on the source host.
6+
7+
Binary: `pkgs/nixos-scripts/cmd/ai-backup` (Go; the backup logics are
8+
POSIX-sh payloads embedded in that source, executed via `/bin/sh -s` on
9+
the source host — including macOS).
10+
11+
> ## ⚠️ Security warning — read before sharing
12+
>
13+
> The archive contains **`~/.local/share/opencode/auth.json`** (OpenCode
14+
> API credentials) and **`~/.claude.json`**, which **may contain OAuth
15+
> tokens** for Claude. Anyone who can read the archive can use your AI
16+
> subscriptions. The default destination is a private Samba share that
17+
> is LAN-reachable with guest-ok semantics; treat the destination host
18+
> and LAN as part of the threat model. Keep share permissions private
19+
> (per-user share, no world access) and do not copy archives out of it.
20+
> To rotate after a suspected leak: log out/in and re-auth OpenCode
21+
> (`opencode auth login`).
22+
23+
## Commands
24+
25+
```
26+
ai-backup backup [TARGET] # default target mact2
27+
ai-backup list
28+
ai-backup restore ARCHIVE [--to TARGET] [--dry-run]
29+
```
30+
31+
Examples:
32+
33+
```bash
34+
# Backup the default source (mact2) to the default destination.
35+
ai-backup backup
36+
37+
# Local snapshot of this host, dry-run inspection only.
38+
ai-backup restore /run/media/stuff/samba/backup/ai/mact2/ai-backup-mact2-YYYYMMDD-HHMMSS.tar.zst --dry-run
39+
40+
# Inspect an archive without decompressing to disk:
41+
zstd -dc ARCHIVE | tar -tf - | less
42+
43+
# Verify a published checksum sidecar:
44+
cd /run/media/stuff/samba/backup/ai/mact2 && sha256sum -c ai-backup-mact2-*.sha256
45+
46+
# Restore locally, keeping pre-restore copies of everything replaced:
47+
ai-backup restore ARCHIVE --to local
48+
49+
# Restore onto another host over ssh.
50+
ai-backup restore ARCHIVE --to t14
51+
```
52+
53+
A backup publishes `<dest>/<label>/ai-backup-<label>-<ts>.tar.zst`
54+
atomically (`.part` removed on failure, renamed only when complete) plus
55+
a `.sha256` sidecar. Dry-run and `list` never modify data.
56+
57+
## What is included / skipped
58+
59+
| Family | Archived | Skipped (regenerable or live) | Why |
60+
|---|---|---|---|
61+
| Claude state | `~/.claude/{projects/,history.jsonl,plans/,todos/,keybindings.json}` and the nested `~/.claude/.claude/` tree, sibling `~/.claude.json`[^claude] | settings, `CLAUDE.md`, skills, commands, agents, caches, shell-snapshots, telemetry, plugin caches, Keychain/`.credentials.json` | Sessions and history are irreplaceable state; settings, caches and credentials are regenerable (config ownership belongs to the dotfile system) |
62+
| OpenCode state | `~/.local/share/opencode/{storage/,auth.json,opencode-multimodal.json}` + snapshot of every channel-dependent `opencode-<channel>.db`[^opencode] | `bin/`, `log/`, `snapshot/`, `tool-output/`, `node_modules`, live `*.db`, `-wal`/`-shm` sidecars, `.config/opencode/` | live DBs snapshot via `sqlite3 .backup` (safe under WAL); the rest is runtime/provisioned state[^opencode2] |
63+
| Engram | global `~/.engram/engram.db` as `…/engram.db.snapshot` only; project dirs opt-in via `AI_BACKUP_EXTRA`[^engram] | live `~/.engram/` tree, `engram.db.before-*`/`pre-cleanup.*` backups, `-wal`/`-shm` | snapshot copy is the consistent moment-in-time copy recommended by SQLite's own backup guide[^sqlite] |
64+
| Extra project dirs | `AI_BACKUP_EXTRA`, verbatim (see env reference) | — | opt-in only |
65+
66+
[^claude]: `~/.claude.json` may contain OAuth credentials — plain file,
67+
sibling member, copied to `.claude.json.pre-restore-<ts>` before a
68+
restore overwrites it. Source: <https://code.claude.com/docs/en/claude-directory>.
69+
[^opencode]: DB names are channel-dependent (`opencode-main.db` etc.);
70+
symlinks are resolved before snapshotting and the staged name is
71+
`<real-basename>.db.snapshot`. Source:
72+
<https://github.com/sst/opencode/issues/13654>.
73+
[^opencode2]: Verified against OpenCode's data-layout troubleshooting
74+
guide: <https://opencode.ai/docs/troubleshooting>.
75+
[^engram]: Snapshot semantics follow the store's own backup path in
76+
<https://github.com/Gentleman-Programming/engram> (`store.go`).
77+
[^sqlite]: <https://www.sqlite.org/backup.html>.
78+
79+
## After restore
80+
81+
1. **Claude re-login** — Keychain and `.credentials.json` are excluded,
82+
so Claude Code asks you to authenticate again on the target host; the
83+
`.claude.json` from the archive is preserved as
84+
`.claude.json.pre-restore-<ts>` first (`keep old .claude.json` in
85+
the output). Same for OpenCode if `auth.json` was absent from the
86+
archive (`opencode auth login`).
87+
2. **Snapshot integrity** — the restore payload runs `PRAGMA
88+
integrity_check` on every restored DB and the command exits nonzero
89+
if any output is not exactly `ok`. Verify manually:
90+
```bash
91+
sqlite3 ~/.local/share/opencode/opencode-main-stable.db 'PRAGMA integrity_check;'
92+
```
93+
3. **Legacy session count** — OpenCode migrated its store out of the
94+
legacy `storage/` directory; the count from the archived machine
95+
must match the count after restore:
96+
```bash
97+
find ~/.local/share/opencode/storage -type f | wc -l
98+
```
99+
Zero or a lower count than expected means the storage member did not
100+
survive the round-trip — re-inspect the archive before deleting
101+
anything.
102+
103+
## Restoring snapshots (payload behaviour)
104+
105+
Snapshot members live under `home-snap/…` in the archive. The restore
106+
payload, per db:
107+
108+
- strips the `.db.snapshot` suffix and moves the staged snapshot back to
109+
its live path (realpath basename),
110+
- **resolves an existing destination symlink first** — it never moves
111+
onto a symlink (Home-Manager re-created ones, like rog's
112+
`opencode.db → opencode-stable.db`) and moves onto the resolved file,
113+
- keeps a timestamped `*.pre-restore-<ts>` copy of every file it
114+
replaces,
115+
- integrity-checks every restoration target (including pre-existing
116+
DBs).
117+
118+
## Environment reference
119+
120+
| Env | Default | Meaning |
121+
|---|---|---|
122+
| `AI_BACKUP_DEST` | `/run/media/stuff/samba/backup/ai` | destination root for backup/list |
123+
| `AI_BACKUP_ZSTD_LEVEL` | `6` | `zstd -T0 -<level>` compression level (1–19) |
124+
| `AI_BACKUP_SSH_OPTS` | `-o BatchMode=yes -o ConnectTimeout=8` | extra ssh options for the source host |
125+
| `AI_BACKUP_EXTRA` | *(unset)* | Space-separated list of absolute paths on the source host; only paths that exist are archived, skipped silently when absent. Typically project `.engram/` dirs: their `chunks/`, `manifest.json` and `config.json` members are archived verbatim as `extras/<project>/.engram/...`, and after restore they land in `~/extras/<project>/.engram/` (never over the global `~/.engram`). The live project `engram.db` never enters. Usage example: `AI_BACKUP_EXTRA="/home/glats/dev/gentle-ai/.engram /home/glats/work/notesta/.engram" ai-backup backup mact2` |
126+
127+
## Limitations
128+
129+
- **No transcript path remapping**: Claude stores project transcripts in
130+
cwd-encoded directory names (`projects/-home-glats-nixos/…`). A
131+
restore onto a different host/username means those paths refer to a
132+
machine-local layout and sessions are not re-encoded into the new
133+
host's session list automatically.
134+
- **`.claude/.claude` is included whole** (triage-later): its final
135+
content split is intentionally deferred — it exists in every archive
136+
for the cost of its bytes.
137+
- **Multimodal provenance is not interpreted**: `opencode-multimodal.json`
138+
is archived raw; attachments referenced by storage paths may decay if
139+
a project's unstaged files changed between backup and restore.
140+
- **The live Engram tree never rides**: if project `.engram` state must
141+
ride, name it in `AI_BACKUP_EXTRA` — it is never implicit.
142+
- sqlite3 must exist on the source host; the payload falls back to
143+
`/usr/bin/sqlite3` after a PATH probe.
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Design: AI Backup Manifest Scope
2+
3+
## Overview
4+
5+
Evolve `pkgs/nixos-scripts/cmd/ai-backup/main.go` and its embedded POSIX-sh payloads. `remoteBackupScript` becomes the executable manifest: snapshot real databases, archive approved state from an explicit member list with no exclude patterns and no macOS xattrs, then use the existing Go compression/publication pipeline. Restore protects `.claude.json`, relocates realpath-named snapshots, and preserves symlinks. The proposal's `bin/ai-backup` path predates the Go migration and must not be recreated.
6+
7+
## Decisions
8+
9+
| Decision | Choice | Rationale | Alternatives rejected |
10+
|---|---|---|---|
11+
| Manifest ownership | Embedded payload plus compact include/skip comment table | Keeps one owner and unchanged packaging | Recreate bash; add an unshared `bin/lib` file |
12+
| Member selection | Explicit include list, zero `--exclude` flags | No portable pattern syntax exists: bsdtar suffix-matches unanchored patterns (the confirmed data-loss bug) while GNU tar reads `^` as a literal and would never match — enumerated members are auditable on both | `^`-anchored excludes (bsdtar-only semantics), per-tree invocations |
13+
| Root Claude state | Sibling member via `-C "$HOME_DIR" .claude.json`; pre-restore copy | It is outside `.claude/` and may hold OAuth | Implicit capture; unsafe overwrite |
14+
| Verification artifact | Runbook inspection; no `--manifest` or second decompression | Acceptance already lists the archive | Routine generated file lists |
15+
| Legacy `storage/` check | Runbook session-count spot check | OpenCode owns migration semantics | Reimplement storage parsing |
16+
| Errors | Preserve CLI and `0/1/2/3/4` categories | Avoid caller breakage | New manifest codes |
17+
18+
No Nix option is added or changed. `pkgs/nixos-scripts/default.nix` retains the existing `cmd/ai-backup` registration.
19+
20+
## Component Changes
21+
22+
### `remoteBackupScript`
23+
24+
- Enumerate members explicitly: `.claude/{projects,history.jsonl,plans,todos,keybindings.json,.claude}`, `$HOME/.claude.json`, `.local/share/opencode/{storage,auth.json,opencode-multimodal.json}`, and staged `home-snap`. `.config/opencode` and the live `.engram` tree never enter (engram rides only as its snapshot; project `.engram/` dirs ride via `AI_BACKUP_EXTRA` paths when present).
25+
- No `--exclude` flags remain; the include list mirrors the spec manifest and is the single auditable artifact.
26+
- Run tar under `COPYFILE_DISABLE=1` (suppresses macOS copyfile xattrs; no-op on GNU); keep `/bin/sh` compatibility. Source hosts need no `zstd`.
27+
- `snap()` resolves and opens the `readlink -f` path, emitting `<real-basename>.db.snapshot`. A `snapcount` fails with 3 unless at least one non-empty OpenCode snapshot exists; all staged snapshots are then non-vacuously checked.
28+
29+
### `remoteRestoreScript`
30+
31+
- Strip `.snapshot`, target the realpath-named file, and resolve an existing destination symlink before rollback-copy and move.
32+
- Relocate OpenCode and Engram snapshots; require `PRAGMA integrity_check` output `ok` for each.
33+
- Add a pre-extract POSIX step that copies existing `.claude.json` to `.pre-restore-<ts>` before plain-file overwrite.
34+
35+
### `doBackup` / `doRestore`
36+
37+
Keep flags and targets unchanged. `doBackup` retains rog-side `zstd -T0 -<level>`, `.part` cleanup/rename, then checksum. `doRestore` remains checksum-first/read-only on dry-run, pre-copies `.claude.json`, and maps extraction, relocation, or integrity failure to 4. GNU extraction may add `--warning=no-unknown-keyword`; source suppression is authoritative.
38+
39+
## `docs/ai-backup.md` Outline
40+
41+
1. Security warning for `auth.json`/possible OAuth and private Samba permissions.
42+
2. Backup, list, inspect, checksum, dry-run, and restore commands.
43+
3. Included/skipped table with one citation per family.
44+
4. Claude re-login, SQLite integrity, and legacy-session count checks.
45+
5. Limits: no transcript path remap; nested Claude triage; uninterpreted multimodal provenance; excluded Keychain/credentials.
46+
47+
## Risks
48+
49+
| Risk | Mitigation |
50+
|---|---|
51+
| Tar pattern drift | Fixture tests list required/forbidden members |
52+
| Symlink replacement or partial restore | Realpath target, rollback copies, checksum/integrity gates |
53+
| Sensitive archive exposure | Runbook warning and private Samba guidance |
54+
| Stale baseline | Tasks target Go and correct proposal/spec paths and gate wording |
55+
56+
## Verification Plan (Spec R5)
57+
58+
| Gate | Evidence |
59+
|---|---|
60+
| Payload/unit behavior | `go test ./...`; `/bin/sh -n` both payloads; test exit contracts, zero/empty snapcount, realpath names, excludes/xattrs, symlink-safe restore |
61+
| Repository gates | `format-nix && nix flake check --no-build` |
62+
| Real mact2 archive | Require `.claude.json`/`.db.snapshot`; reject `.config/opencode`, live DBs, `bin/`, `log/`, `snapshot/`, `node_modules` |
63+
| Integrity and publication | Extract a snapshot, require `PRAGMA integrity_check` = `ok`, and run `sha256sum -c` |
64+
65+
R5's `bash -n` is stale after the Go migration; correct it to `/bin/sh -n` over embedded payloads plus `go test ./...` before tasks.
66+
67+
## Threat Matrix
68+
69+
| Boundary | Applicability | Design response | Planned RED tests |
70+
|---|---|---|---|
71+
| Documentation-like paths | N/A — tar members are data, never executable classification | None | None |
72+
| Git repository selection | N/A — no Git invocation | None | None |
73+
| Commit state | N/A — no commit automation | None | None |
74+
| Push state | N/A — no push automation | None | None |
75+
| PR commands | N/A — no PR automation | None | None |
76+
77+
## Migration / Scope Boundary
78+
79+
No data migration or flag is required. Roll back the Go source and runbook; rollback copies remain. No timer, retention, path remap, Samba provisioning, or encryption layer is added.
80+
81+
## Open Questions
82+
83+
None.
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Proposal: AI Backup Manifest Scope
2+
3+
## Intent
4+
5+
Make the one-shot `mact2` → `rog` AI-state backup complete and auditable. The draft omits `~/.claude.json`, risks silent exclusion under macOS `bsdtar`, and does not distinguish irreplaceable state from declarative or regenerable data.
6+
7+
## Scope
8+
9+
### In Scope
10+
- Back up the documented Claude transcripts/state, OpenCode SQLite/auth/legacy storage, Engram snapshots, project `.engram/`, and nested `~/.claude/.claude/`.
11+
- Preserve `.backup`, atomic publication, checksums, listing, dry-run, rollback copies, and integrity checks; restore `~/.claude.json` and realpath-named databases without replacing symlinks.
12+
- Use `^`-anchored excludes, `.db.snapshot` staging with non-vacuous counts, and macOS xattr suppression.
13+
- Document exclusions with citations, Samba security, Claude re-login, legacy-session checks, restore, and known limitations.
14+
- Keep `mact2` as default source while supporting local, `t14`, `thinkcentre`, and `user@host` targets.
15+
16+
### Out of Scope
17+
- Home Manager assets, caches, logs, binaries, environment snapshots, Claude credentials, and macOS Keychain data.
18+
- Path remapping, nested-home deduplication, retention, scheduling, encryption, and Samba provisioning.
19+
20+
## Capabilities
21+
22+
### New Capabilities
23+
- `ai-assistant-state-backup`: Manifest-driven backup, inspection, and safe restore of Claude Code, OpenCode, and Engram state.
24+
25+
### Modified Capabilities
26+
- None; existing canonical specs do not govern AI-state backup.
27+
28+
## Approach
29+
30+
Stage SQLite snapshots as `.db.snapshot`, archive approved state with anchored exclusions and xattrs disabled, then stream through `zstd` to `/run/media/stuff/samba/backup/ai/<label>/`. Extend the current relocation restore. Warn that included OpenCode auth and possible Claude tokens make archives sensitive.
31+
32+
## Affected Areas
33+
34+
| Area | Impact | Description |
35+
|---|---|---|
36+
| `pkgs/nixos-scripts/cmd/ai-backup/` | Modified | Manifest member list, portability defenses, and restore coverage |
37+
| `docs/ai-backup.md` | New | Operator and security runbook |
38+
| `pkgs/nixos-scripts/default.nix` | Retained | Package registration |
39+
40+
## Risks
41+
42+
| Risk | Likelihood | Mitigation |
43+
|---|---|---|
44+
| Silent omission or inconsistent DB | Medium | Anchors, snapshot naming/counts, `.backup`, archive inspection |
45+
| Credential exposure on Samba | Medium | Restricted destination guidance and explicit security warning |
46+
| Restore overwrites symlink/config | Low | Realpath names, pre-restore copies, dry-run |
47+
48+
## Rollback Plan
49+
50+
Restore the prior Go command and runbook; registration remains valid. Failed restores recover from timestamped pre-restore copies.
51+
52+
## Dependencies
53+
54+
- SSH, writable `rog` storage, `sqlite3`, `tar`/`bsdtar`, `zstd`, and SHA-256 tooling.
55+
56+
## Verification Plan
57+
58+
- Run `bash -n`, then `format-nix && nix flake check --no-build`.
59+
- Run a real `mact2` backup; inspect required and excluded members, extract a snapshot, and obtain `ok` from `PRAGMA integrity_check`.
60+
61+
## Success Criteria
62+
63+
- [ ] Required portable state, including `~/.claude.json` and legacy storage, is present without live WAL databases or managed/ephemeral paths.
64+
- [ ] Restore preserves symlinks, creates rollback copies, and verifies every restored database.
65+
- [ ] The runbook enables a documented backup, inspection, re-login, and restore procedure.

0 commit comments

Comments
 (0)