Skip to content

Commit e768750

Browse files
committed
fix: full-codebase audit findings (data-integrity, fail-closed, security, robustness)
Athena full-codebase audit (3 members, dashboard-excluded). Each finding verified against source; real ones fixed, false positives + accepted tradeoffs documented. HIGH — data integrity: - #3 v22 memory rekey UNIQUE collision: the deferred backfill did a blind UPDATE memories SET project_path=identity, which trips UNIQUE(project_path, category, normalized_hash) and aborts the whole batch transaction when two legacy raw paths for one project share a (category, normalized_hash). Now detects the target collision and merges (keep max seen_count, delete source, embedding FK-cascades) instead. +test. - #5 CLI migration dropped v2 compartment tiers: the Pi-session migrator's bespoke INSERT omitted p1-p4/importance/episode_type/legacy, so migrated rows landed legacy=0 + NULL tiers and the decay renderer fell back to full content for every tier (no decay, prompt bloat). Now carries all v2 fields faithfully (legacy v1 rows keep legacy=1). +test. MEDIUM — robustness / correctness: - #1 openDatabase() honest type: JSDoc promised "throws" but the schema-fence path returned `null as unknown as Database` (a type lie). Return type is now Database | null; every call site null-checks (plugin + Pi + CLI). The two sites the audit flagged were already safe, but the contract is now compiler-enforced. recordChildInvocation tolerates a null db (skip telemetry). - #7 CLI migration non-transactional: copyMagicContextState now returns a plan whose commit() runs all INSERTs inside BEGIN IMMEDIATE, and the caller writes the Pi JSONL file BEFORE committing DB rows — so an interruption never orphans shared-DB rows pointing at a session file that was never written. - #8 auto-update npm install pipe deadlock: spawn used stdio:"pipe" with no readers; once npm output exceeds the OS pipe buffer the child blocks forever and the updater spuriously times out. Switched to stdio:"ignore" (output is never read; failure still detected via exit code). - #10 key-files trust: the validator enforced path/size/budget but not candidate-set membership or doc/lockfile exclusion (prompt-only). Both are now enforced in code (the persisted key-files set is injected into every future prompt — a trust boundary). +tests. - #11 RPC auth: the localhost RPC server set wildcard CORS and required no auth on side-effecting endpoints (recomp/upgrade/dismiss). Now generates an unguessable per-process token, publishes it in the (user-private) port file, requires it as Bearer auth on all non-health calls, and drops wildcard CORS. Health stays open for discovery. +tests. Dashboard (user-reported regression from f5cfa31): - resolve_paths_for_table_filter read `SELECT DISTINCT project_path` into a non-Option String; the notes table holds many NULL-project_path rows (session notes), so opening any session detail threw "Invalid column type Null at index: 0, name: project_path" and crashed the whole view. Now reads Option<String> and drops NULLs. +test. Docs: - #9 ARCHITECTURE.md reconciled with the supersede-delta design: in-session ctx_memory mutations (additive AND non-additive) route through m[1], not a project_memory_epoch bump; the epoch is bumped only by dashboard mutations and /ctx-session-upgrade migration. - AUDIT-KNOWN-ISSUES A11: key-files lazy pickup (next natural cache-bust) is the chosen behavior, and not a Pi divergence in v2 (Pi's eager system-prompt signal no longer touches m[1]-resident key-files). Verified FALSE POSITIVE: - #2 (contested HIGH) m[0] rematerialize-on-defer: Opus correct. materialize fires only on genuine content-change markers (epoch/seq/mutation-id/docs-hash/ upgrade-state); Date.now() is deliberately excluded. Defer replay stays byte-identical (test-locked). GPT 5.5 conflated "not gated on isCacheBusting" with "rebuilds spuriously". Deferred (need decisions/sourcing): #6 (Electron native-binding integrity — needs a trusted checksum/signature source). LOW hygiene (#12-#18) batched separately. plugin 1717/0, Pi 403/0, CLI 148/0, dashboard cargo + 18 Rust tests, tsc + biome clean, dist rebuilt.
1 parent f5cfa31 commit e768750

25 files changed

Lines changed: 632 additions & 129 deletions

‎ARCHITECTURE.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,9 +108,10 @@ Injection renders only stored DB content. Disk drift queues a CAS-protected stal
108108
**m[0]/m[1] history layout (v2 — the cache-stability core):** the compacted history is rendered into TWO synthetic message slots instead of one, so the large stable prefix survives steady-state working sessions. Implemented in `inject-compartments.ts` (`renderM0`, `renderM1`, `materializeM0`, `mustMaterialize`) and mirrored in `inject-compartments-pi.ts`.
109109
- **m[0] — cumulative baseline.** Holds `<project-docs>`, baseline `<user-profile>`, and the decay-rendered compartment history as of the last materialization. Treated like a frozen prefix (analogous to `system[0]`): it does NOT change on routine turns, so its prompt-cache bytes persist.
110110
- **m[1] — volatile delta.** Holds everything added *since* the last m[0] materialization: `<key-files>`, new user-profile additions, **new memories** (read via the `maxMemoryId` watermark `id > cachedM0MaxMemoryId`), and the newest compartments (rendered at full tier 1). When there's nothing new it renders a minimal placeholder (m[1] must never be fully empty for Anthropic cache-breakpoint structure).
111-
- **Materialization decision (`mustMaterialize`).** m[0] re-materializes ONLY on: first render, `project_memory_epoch` change (non-additive memory mutations: update/delete/archive/merge — additive write/promote deliberately do NOT bump, so new memories flow through m[1] instead), user-profile-version change, new compartment sequence, pending m[0] mutations, project-docs hash change, or upgrade-state change. **`maxMemoryId` is deliberately NOT a trigger** — it drives m[1] rendering, not m[0] invalidation; triggering on it would bust m[0] on every memory write and defeat the whole design. `session_facts` version is retired as a trigger (facts are memories now).
111+
- **Materialization decision (`mustMaterialize`).** m[0] re-materializes ONLY on: first render, `project_memory_epoch` change, user-profile-version change, new compartment sequence, pending m[0] mutations, project-docs hash change, or upgrade-state change. **`maxMemoryId` is deliberately NOT a trigger** — it drives m[1] rendering, not m[0] invalidation; triggering on it would bust m[0] on every memory write and defeat the whole design. `session_facts` version is retired as a trigger (facts are memories now).
112+
- **Memory mutations route through m[1] (supersede-delta), not the epoch (v2.0).** In-session `ctx_memory` mutations — both additive (`write`/`promote`) and non-additive (`update`/`delete`/`archive`/`merge`) — deliberately do NOT bump `project_memory_epoch`. Additive writes surface in m[1] via the `maxMemoryId` watermark; non-additive mutations record a `memory_mutation_log` row that renders in m[1] as a `<memory-updates>` delta (stale rows linger in the frozen m[0] baseline until reconciled). Both are reconciled into m[0] on the next natural hard bust. The epoch IS bumped — forcing an immediate m[0] re-materialize — only by **dashboard** memory mutations (an external editor cannot otherwise signal a running session) and by `/ctx-session-upgrade` memory migration.
112113
- **Cache-bust taxonomy.** Soft passes replay identical bytes (defer). Hard busts materialize m[0] (fold m[1] into the new baseline). A forced refold also fires at +15% budget drift if no natural hard bust occurred. `applyMarkersToState` updates ALL root `state.cachedM0*` fields post-materialize so the next pass sees the new baseline (the guard against an infinite re-materialize loop).
113-
- **`m0_mutation_log` + `project_state`.** The mutation log carries m[0]-affecting deltas for m[1]; `project_state.project_memory_epoch` is the cross-process invalidation counter bumped by non-additive memory mutations and by `/ctx-session-upgrade` migration.
114+
- **`m0_mutation_log` + `project_state`.** The mutation log carries m[0]-affecting deltas for m[1]; `project_state.project_memory_epoch` is the cross-process invalidation counter bumped by **dashboard** memory mutations and by `/ctx-session-upgrade` migration (in-session `ctx_memory` mutations use the `memory_mutation_log` supersede-delta path instead — see the m[1] memory-mutation note above).
114115

115116
**System-prompt adjunct injection:**
116117
- The `experimental.chat.system.transform` hook in `src/hooks/magic-context/system-prompt-hash.ts` injects only the **Magic Context agent guidance text** (when `system_prompt_injection.enabled=true` and the active agent isn't matched by `skip_signatures`) plus the frozen `Today's date:` line into the system-prompt array.

‎docs/AUDIT-KNOWN-ISSUES.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,28 @@ signal a running session; see the dashboard's own path.) Accepted.
161161
> memory drops on the next materialization); m[1]'s `id > maxMemoryId` filter
162162
> prevents any double-render. Not a separate bug.
163163
164+
### A11. Key-files pickup after a Dreamer commit is lazy (next natural cache-bust), not eager
165+
166+
When the Dreamer commits new/updated key-files mid-session, OpenCode does NOT
167+
force a refresh — the new `<key-files>` content is picked up on the next natural
168+
cache-busting pass (`buildKeyFilesBlock` is re-read fresh whenever m[1] is
169+
recomputed). This is **deliberate**: key-files live in m[1], so any pickup
170+
necessarily rewrites the m[1] cache prefix; forcing that on every Dreamer commit
171+
would bust the Anthropic cache for a low-urgency content change. Lazy pickup is
172+
the chosen behavior (cache-stability over freshness for an infrequently-changing
173+
block).
174+
175+
**Not a Pi divergence in v2** (the audit's "Pi eager vs OpenCode lazy" framing
176+
predates the v2 layout): in v1, key-files lived in the system prompt and Pi's
177+
Dreamer fired `onAdjunctsRefreshNeeded → systemPromptRefreshSessions` to refresh
178+
it. In v2 the system block is guidance-only (`system-prompt.ts` explicitly never
179+
emits `<key-files>`/`<project-docs>`/`<user-profile>`), and that signal targets
180+
the **system-prompt** refresh set, NOT `historyRefreshSessions` (which drives
181+
m[1]). So Pi also picks up key-files lazily via m[1]'s natural busts — same
182+
effective behavior as OpenCode. Pi's lingering system-prompt refresh on Dreamer
183+
commit is a near-no-op in v2 (guidance text doesn't change), not an extra m[1]
184+
bust. Accepted on both harnesses.
185+
164186
## Growing-data factors (bounded or slow; future cleanup)
165187

166188
These are intentionally listed so a future cleanup/maintenance task can address

‎packages/cli/src/commands/doctor-pi.ts‎

Lines changed: 30 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -503,30 +503,42 @@ async function runHealthChecks(options: {
503503
let db: ContextDatabase | null = null;
504504
try {
505505
db = options.deps.openDatabase();
506-
if (options.deps.isDatabasePersisted(db))
507-
add(results, "pass", "openDatabase() opened the shared DB");
508-
else
506+
if (!db) {
507+
// openDatabase() returns null on the schema fence (shared DB newer
508+
// than this binary supports). Report and skip DB-dependent checks;
509+
// the embedding checks below do not need the DB handle.
509510
add(
510511
results,
511512
"fail",
512-
"openDatabase() fell back to an in-memory DB; shared DB is broken or unwritable",
513+
"openDatabase() returned no handle; the shared DB schema is newer than this binary supports (upgrade Magic Context)",
513514
);
515+
} else {
516+
if (options.deps.isDatabasePersisted(db))
517+
add(results, "pass", "openDatabase() opened the shared DB");
518+
else
519+
add(
520+
results,
521+
"fail",
522+
"openDatabase() fell back to an in-memory DB; shared DB is broken or unwritable",
523+
);
514524

515-
const integrity = db.prepare("PRAGMA integrity_check").get() as {
516-
integrity_check?: unknown;
517-
};
518-
if (integrity?.integrity_check === "ok") add(results, "pass", "SQLite integrity_check: ok");
519-
else
520-
add(
521-
results,
522-
"fail",
523-
`SQLite integrity_check: ${String(integrity?.integrity_check ?? "unknown")}`,
524-
);
525+
const integrity = db.prepare("PRAGMA integrity_check").get() as {
526+
integrity_check?: unknown;
527+
};
528+
if (integrity?.integrity_check === "ok")
529+
add(results, "pass", "SQLite integrity_check: ok");
530+
else
531+
add(
532+
results,
533+
"fail",
534+
`SQLite integrity_check: ${String(integrity?.integrity_check ?? "unknown")}`,
535+
);
525536

526-
const counts = ROW_COUNT_TABLES.map(
527-
(table) => `${table}=${countTable(db as ContextDatabase, table) ?? "n/a"}`,
528-
).join(", ");
529-
add(results, "info", `Shared DB row counts: ${counts}`);
537+
const counts = ROW_COUNT_TABLES.map(
538+
(table) => `${table}=${countTable(db as ContextDatabase, table) ?? "n/a"}`,
539+
).join(", ");
540+
add(results, "info", `Shared DB row counts: ${counts}`);
541+
}
530542
} catch (error) {
531543
add(
532544
results,

‎packages/cli/src/commands/migrate.test.ts‎

Lines changed: 55 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,13 @@ function makeCortexkitDb() {
157157
end_message_id TEXT DEFAULT '',
158158
title TEXT NOT NULL,
159159
content TEXT NOT NULL,
160+
p1 TEXT,
161+
p2 TEXT,
162+
p3 TEXT,
163+
p4 TEXT,
164+
importance INTEGER NOT NULL DEFAULT 50,
165+
episode_type TEXT,
166+
legacy INTEGER NOT NULL DEFAULT 0,
160167
created_at INTEGER NOT NULL,
161168
harness TEXT NOT NULL DEFAULT 'opencode',
162169
UNIQUE(session_id, sequence)
@@ -365,13 +372,31 @@ describe("migrateOpenCodeSessionToPi — token & magic-context bridging", () =>
365372
// Two compartments under the source session.
366373
// Compartment 0: covers msg_1 → msg_2 (exact boundary match).
367374
ck.prepare(
368-
"INSERT INTO compartments (session_id, sequence, start_message, end_message, start_message_id, end_message_id, title, content, created_at, harness) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'opencode')",
369-
).run(sessionId, 0, 1, 2, "msg_1", "msg_2", "Comp 0", "summary 0", 5);
375+
"INSERT INTO compartments (session_id, sequence, start_message, end_message, start_message_id, end_message_id, title, content, p1, p2, p3, p4, importance, episode_type, legacy, created_at, harness) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'opencode')",
376+
).run(
377+
sessionId,
378+
0,
379+
1,
380+
2,
381+
"msg_1",
382+
"msg_2",
383+
"Comp 0",
384+
"summary 0",
385+
"p1 verbose",
386+
"p2 mid",
387+
"p3 terse",
388+
"p4 anchor",
389+
72,
390+
"design,bug",
391+
0,
392+
5,
393+
);
370394
// Compartment 1: end boundary "msg_unknown" doesn't directly map →
371-
// must remap to nearest at-or-before.
395+
// must remap to nearest at-or-before. Stored as a legacy (v1) row to
396+
// confirm the legacy flag is preserved verbatim through migration.
372397
ck.prepare(
373-
"INSERT INTO compartments (session_id, sequence, start_message, end_message, start_message_id, end_message_id, title, content, created_at, harness) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'opencode')",
374-
).run(sessionId, 1, 3, 4, "msg_3", "msg_zzzz_unknown", "Comp 1", "summary 1", 6);
398+
"INSERT INTO compartments (session_id, sequence, start_message, end_message, start_message_id, end_message_id, title, content, importance, legacy, created_at, harness) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'opencode')",
399+
).run(sessionId, 1, 3, 4, "msg_3", "msg_zzzz_unknown", "Comp 1", "summary 1", 50, 1, 6);
375400

376401
ck.prepare(
377402
"INSERT INTO session_facts (session_id, category, content, created_at, updated_at, harness) VALUES (?, ?, ?, ?, ?, 'opencode')",
@@ -403,13 +428,36 @@ describe("migrateOpenCodeSessionToPi — token & magic-context bridging", () =>
403428
};
404429
const piCompartments = ck
405430
.prepare(
406-
"SELECT session_id, sequence, start_message_id, end_message_id, title, content, harness FROM compartments WHERE session_id = ? AND harness = 'pi' ORDER BY sequence",
431+
"SELECT session_id, sequence, start_message_id, end_message_id, title, content, p1, p2, p3, p4, importance, episode_type, legacy, harness FROM compartments WHERE session_id = ? AND harness = 'pi' ORDER BY sequence",
407432
)
408-
.all(result.piSessionId) as Row[];
433+
.all(result.piSessionId) as Array<
434+
Row & {
435+
p1: string | null;
436+
p2: string | null;
437+
p3: string | null;
438+
p4: string | null;
439+
importance: number;
440+
episode_type: string | null;
441+
legacy: number;
442+
}
443+
>;
409444
expect(piCompartments).toHaveLength(2);
410445
expect(piCompartments[0].title).toBe("Comp 0");
411446
expect(piCompartments[1].title).toBe("Comp 1");
412447

448+
// v2 tier/metadata must survive migration (regression: bespoke INSERT
449+
// previously dropped p1-p4/importance/episode_type and forced legacy=0).
450+
expect(piCompartments[0].p1).toBe("p1 verbose");
451+
expect(piCompartments[0].p2).toBe("p2 mid");
452+
expect(piCompartments[0].p3).toBe("p3 terse");
453+
expect(piCompartments[0].p4).toBe("p4 anchor");
454+
expect(piCompartments[0].importance).toBe(72);
455+
expect(piCompartments[0].episode_type).toBe("design,bug");
456+
expect(piCompartments[0].legacy).toBe(0);
457+
// Legacy v1 row keeps legacy=1 and NULL tiers.
458+
expect(piCompartments[1].legacy).toBe(1);
459+
expect(piCompartments[1].p1).toBeNull();
460+
413461
// Verify boundary IDs reference real Pi entries in the JSONL.
414462
const entries = readJsonl(result.outputPath);
415463
const entryIds = new Set(

0 commit comments

Comments
 (0)