Skip to content

Commit a1ac807

Browse files
0xb007ab1eclaude
andcommitted
fix(adr): store ADRs in a per-project sidecar DB, untouched by reindex
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review): remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps, polling, or timeout deciding an outcome. store.c: - ADR writes (store/delete/update_sections) use a cached read-write handle opened SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB. - Reads never write: cbm_store_adr_get uses the cached RW handle if open, else opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the file exists, else reads the legacy graph row without creating anything. When the sidecar exists but is NOT marked migrated (half-created), it falls back to the legacy graph row so a partial sidecar never hides an ADR. - One-time migration is atomic: adr_copy_legacy_once wraps the marker check, legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT (re-checking the marker under the lock; ROLLBACK->ERR on failure). New public cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows long-path). Dropped the write-only adr_db_path field. pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and forced-full routes unlink the old generation; preserve-and-abort on failure, and do NOT fail open — a query-open returning NULL preserves unless the final DB genuinely does not exist. Migration targets the FINAL path (staging suffix stripped). The early-capture/re-apply machinery and its test hook are gone. artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always DELETEs project_summaries first, then injects only when an ADR is present, so a deleted ADR never resurrects through a lingering legacy row; import restores the ADR into the destination sidecar via cbm_store_adr_migrate_once (INSERT OR IGNORE — never clobbers a local ADR). mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude *.adr.db from index listing/counting. README + memory-core baselines updated. Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before a forced-full/format reindex with no prior read; migration failure (a directory at <db>.adr.db) preserves-and-aborts with the old generation intact; delete_project via the MCP handler removes the sidecar + journals; the artifact export->import round trip keeps the ADR; a deleted ADR does not resurrect on import; an import does not clobber a local ADR; a half-created sidecar falls back to the legacy row. Closes DeusData#516 Signed-off-by: 0xb007ab1e <[email protected]> Co-Authored-By: Claude Opus 4.8 <[email protected]>
1 parent 64c23fa commit a1ac807

13 files changed

Lines changed: 1142 additions & 228 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -671,7 +671,7 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
671671
|------|-------------|
672672
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh after that. |
673673
| `list_projects` | List all indexed projects with node/edge counts. |
674-
| `delete_project` | Remove a project and all its graph data. |
674+
| `delete_project` | Remove a project and all its graph data, including its ADR sidecar (`<db>.adr.db`). |
675675
| `index_status` | Check indexing status of a project. |
676676
| `check_index_coverage` | Check whether exact paths or a scope are indexed and fresh. A clean result means no recorded gap, not proof of completeness. |
677677

‎scripts/memory-core-baseline.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,9 +77,9 @@ src/pipeline/pass_semantic.c 3
7777
src/pipeline/pass_tests.c 3
7878
src/pipeline/pass_usages.c 4
7979
src/pipeline/path_alias.c 42
80-
src/pipeline/pipeline.c 134
80+
src/pipeline/pipeline.c 131
8181
src/pipeline/pipeline_delta.c 19
82-
src/pipeline/pipeline_incremental.c 144
82+
src/pipeline/pipeline_incremental.c 134
8383
src/pipeline/registry.c 17
8484
src/pipeline/worker_pool.c 2
8585
src/store/store.c 358

‎src/cli/cli.c‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7193,6 +7193,13 @@ static const char *get_cache_dir(const char *home_dir) {
71937193
return cbm_resolve_cache_dir();
71947194
}
71957195

7196+
/* A per-project ADR sidecar ("<db>.adr.db") sits in the cache beside its index
7197+
* but is not itself an index — exclude it from listing/counting. */
7198+
static bool cli_is_adr_sidecar(const char *name, size_t len) {
7199+
const size_t ext = sizeof(".adr.db") - 1;
7200+
return len >= ext && strcmp(name + len - ext, ".adr.db") == 0;
7201+
}
7202+
71967203
int cbm_list_indexes(const char *home_dir) {
71977204
const char *cache_dir = get_cache_dir(home_dir);
71987205
if (!cache_dir) {
@@ -7208,7 +7215,8 @@ int cbm_list_indexes(const char *home_dir) {
72087215
cbm_dirent_t *ent;
72097216
while ((ent = cbm_readdir(d)) != NULL) {
72107217
size_t len = strlen(ent->name);
7211-
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0) {
7218+
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0 &&
7219+
!cli_is_adr_sidecar(ent->name, len)) {
72127220
printf(" %s/%s\n", cache_dir, ent->name);
72137221
count++;
72147222
}
@@ -10437,7 +10445,8 @@ static int count_db_indexes(const char *home) {
1043710445
cbm_dirent_t *ent;
1043810446
while ((ent = cbm_readdir(d)) != NULL) {
1043910447
size_t len = strlen(ent->name);
10440-
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0) {
10448+
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0 &&
10449+
!cli_is_adr_sidecar(ent->name, len)) {
1044110450
count++;
1044210451
}
1044310452
}

‎src/mcp/mcp.c‎

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -671,7 +671,7 @@ static const tool_def_t TOOLS[] = {
671671
"\"offset\":{\"type\":\"integer\",\"default\":0,\"minimum\":0},"
672672
"\"metadata_only\":{\"type\":\"boolean\",\"default\":false,"
673673
"\"description\":\"Compatibility: omit counts, size, and branch.\"}}}"},
674-
{"delete_project", "Delete a project from the index",
674+
{"delete_project", "Delete a project from the index, including its ADR store",
675675
"{\"type\":\"object\",\"properties\":{\"project\":{\"type\":\"string\"}},\"required\":["
676676
"\"project\"]}"},
677677

@@ -2623,6 +2623,11 @@ static bool is_project_db_file(const char *name, size_t len) {
26232623
if (len < MCP_MIN_DB_NAME || strcmp(name + len - MCP_DB_EXT, ".db") != 0) {
26242624
return false;
26252625
}
2626+
/* A per-project ADR sidecar ("<db>.adr.db") ends in ".db" but is not itself
2627+
* an index — skip it in every cache/cross-repo scan (matches the CLI). */
2628+
if (len >= SLEN(".adr.db") && strcmp(name + len - SLEN(".adr.db"), ".adr.db") == 0) {
2629+
return false;
2630+
}
26262631
if (strncmp(name, "_", SLEN("_")) == 0 || strncmp(name, ":memory:", SLEN(":memory:")) == 0) {
26272632
return false;
26282633
}
@@ -6809,7 +6814,9 @@ static char *handle_index_status(cbm_mcp_server_t *srv, const char *args) {
68096814
return result;
68106815
}
68116816

6812-
/* delete_project: just erase the .db file (and WAL/SHM). */
6817+
/* delete_project: erase the .db file (and WAL/SHM) plus the ADR sidecar
6818+
* "<db>.adr.db" (and its WAL/SHM/journal), so deleting a project removes its
6819+
* ADR store with it. */
68136820
static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
68146821
char *name = get_project_arg(args);
68156822
if (!name) {
@@ -6848,18 +6855,37 @@ static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
68486855
const char *error_detail = NULL;
68496856
bool is_error = false;
68506857

6858+
/* ADR sidecar lives beside the graph DB as "<db>.adr.db" — remove it (and
6859+
* its own WAL/SHM/journal) whenever the project is deleted. Sized as the
6860+
* graph path plus the ".adr.db" suffix. */
6861+
char adr_path[CBM_SZ_1K + sizeof(".adr.db")];
6862+
int adr_n = snprintf(adr_path, sizeof(adr_path), "%s.adr.db", path);
6863+
bool have_adr_path = adr_n > 0 && (size_t)adr_n < sizeof(adr_path);
6864+
68516865
if (exists) {
68526866
int rc = cbm_unlink(path);
68536867
(void)cbm_unlink(wal);
68546868
(void)cbm_unlink(shm);
68556869
if (rc == 0) {
68566870
status = "deleted";
6871+
/* Only drop the ADR store once the graph DB is actually gone, so a
6872+
* failed graph delete leaves the project (and its ADR) intact. */
6873+
if (have_adr_path) {
6874+
(void)cbm_unlink(adr_path);
6875+
(void)cbm_remove_db_sidecars(adr_path);
6876+
}
68576877
} else {
68586878
status = "delete_failed";
68596879
error_detail = strerror(errno);
68606880
is_error = true;
68616881
}
68626882
} else {
6883+
/* No graph DB, but a sidecar could still linger (e.g. ADR written then
6884+
* the index deleted out of band) — best-effort clean it up. */
6885+
if (have_adr_path) {
6886+
(void)cbm_unlink(adr_path);
6887+
(void)cbm_remove_db_sidecars(adr_path);
6888+
}
68636889
is_error = true;
68646890
}
68656891

@@ -17173,9 +17199,10 @@ static char *handle_manage_adr(cbm_mcp_server_t *srv, const char *args) {
1717317199
}
1717417200
}
1717517201

17176-
/* ADRs are stored in the SQLite store (project_summaries), the SAME
17177-
* backend the UI /api/adr endpoints use — so writes via the MCP tool and
17178-
* the UI are visible to each other (#256). */
17202+
/* ADRs are stored in the per-project ADR sidecar DB ("<db>.adr.db"), the
17203+
* SAME backend the UI /api/adr endpoints use (via cbm_store_adr_*) — so
17204+
* writes via the MCP tool and the UI are visible to each other (#256), and
17205+
* a reindex of the graph DB never disturbs them. */
1717917206
store_recovery_status_t recovery_status = STORE_RECOVERY_NONE;
1718017207
cbm_store_t *resolved =
1718117208
resolve_store_internal(srv, project, mutation_held, !write_request, &recovery_status, true);

‎src/pipeline/artifact.c‎

Lines changed: 77 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -888,13 +888,54 @@ static void artifact_snapshot_tmp_close(artifact_snapshot_tmp_t *tmp) {
888888
tmp->dir[0] = '\0';
889889
}
890890

891+
/* Make the snapshot's project_summaries carry exactly the current ADR — and
892+
* nothing stale. ADRs now live in the "<db>.adr.db" sidecar, but a legacy
893+
* pre-sidecar row can still linger in the graph DB and gets copied into the
894+
* snapshot by VACUUM INTO; incremental/closure-repair clones keep carrying it.
895+
* So ALWAYS clear project_summaries first, then insert only when an ADR is
896+
* present (`adr` is NULL when the sidecar has none). Otherwise a no-ADR export
897+
* would ship the stale row and a teammate's import would restore a decision
898+
* record that was deleted. Best-effort: on any failure the artifact simply
899+
* omits the ADR, as before the sidecar existed. */
900+
static void snapshot_inject_adr(const char *snapshot_path, const cbm_adr_t *adr) {
901+
sqlite3 *db = NULL;
902+
if (sqlite3_open_v2(snapshot_path, &db, SQLITE_OPEN_READWRITE, NULL) != SQLITE_OK) {
903+
sqlite3_close(db);
904+
return;
905+
}
906+
sqlite3_exec(db,
907+
"CREATE TABLE IF NOT EXISTS project_summaries ("
908+
" project TEXT PRIMARY KEY, summary TEXT NOT NULL, source_hash TEXT NOT NULL,"
909+
" created_at TEXT NOT NULL, updated_at TEXT NOT NULL);",
910+
NULL, NULL, NULL);
911+
/* Drop any stale row (a deleted ADR must not resurrect through the export). */
912+
sqlite3_exec(db, "DELETE FROM project_summaries;", NULL, NULL, NULL);
913+
if (adr && adr->project && adr->content) {
914+
sqlite3_stmt *st = NULL;
915+
if (sqlite3_prepare_v2(db,
916+
"INSERT INTO project_summaries "
917+
"(project, summary, source_hash, created_at, updated_at) "
918+
"VALUES (?1, ?2, '', ?3, ?4);",
919+
-1, &st, NULL) == SQLITE_OK) {
920+
sqlite3_bind_text(st, 1, adr->project, -1, SQLITE_TRANSIENT);
921+
sqlite3_bind_text(st, 2, adr->content, -1, SQLITE_TRANSIENT);
922+
sqlite3_bind_text(st, 3, adr->created_at ? adr->created_at : "", -1, SQLITE_TRANSIENT);
923+
sqlite3_bind_text(st, 4, adr->updated_at ? adr->updated_at : "", -1, SQLITE_TRANSIENT);
924+
sqlite3_step(st);
925+
sqlite3_finalize(st);
926+
}
927+
}
928+
sqlite3_close(db);
929+
}
930+
891931
/* Prepare a stripped DB copy for best-quality export.
892-
* VACUUM INTO → (optionally) drop indexes → VACUUM. Returns malloc'd buffer
893-
* or NULL. VACUUM INTO runs on BOTH quality levels: it is the consistent
894-
* snapshot — the store runs in WAL mode, so raw main-file bytes miss
895-
* committed transactions still in the -wal and can be mid-checkpoint torn
896-
* (#895). Only the index-stripping is BEST-only. */
897-
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes) {
932+
* VACUUM INTO → (optionally) drop indexes → VACUUM → (optionally) inject the
933+
* ADR. Returns malloc'd buffer or NULL. VACUUM INTO runs on BOTH quality
934+
* levels: it is the consistent snapshot — the store runs in WAL mode, so raw
935+
* main-file bytes miss committed transactions still in the -wal and can be
936+
* mid-checkpoint torn (#895). Only the index-stripping is BEST-only. */
937+
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes,
938+
const cbm_adr_t *adr) {
898939
artifact_snapshot_tmp_t tmp;
899940
if (!artifact_snapshot_tmp_open(&tmp)) {
900941
artifact_export_fail("prepare_snapshot_dir", cbm_tmpdir(), "private_tmpdir_failed", errno);
@@ -939,6 +980,11 @@ static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool str
939980
}
940981
}
941982

983+
/* Carry the ADR (from the sidecar) into the snapshot so the exported
984+
* artifact ships the team's decision record. After the strip/VACUUM, so the
985+
* row is not dropped. */
986+
snapshot_inject_adr(tmp_path, adr);
987+
942988
/* Reopened by path rather than held on a descriptor across VACUUM INTO,
943989
* because sqlite owns the create. The private directory is what makes that
944990
* safe: an attacker who cannot enter it cannot swap the file underneath. */
@@ -984,15 +1030,28 @@ int cbm_artifact_export(const char *db_path, const char *repo_path, const char *
9841030
char *db_data = NULL;
9851031
int compression_level = ART_ZSTD_FAST;
9861032

1033+
/* Fetch the ADR from the sidecar (read-only; does not create it) so the
1034+
* snapshot can carry it — the team-shared artifact used to include the ADR
1035+
* via project_summaries, and the sidecar must not silently break that. */
1036+
cbm_adr_t adr = {0};
1037+
bool have_adr = false;
1038+
cbm_store_t *adr_src = cbm_store_open_path_query(db_path);
1039+
if (adr_src) {
1040+
have_adr = cbm_store_adr_get(adr_src, project_name, &adr) == CBM_STORE_OK;
1041+
cbm_store_close(adr_src);
1042+
}
1043+
const cbm_adr_t *adr_arg = have_adr ? &adr : NULL;
1044+
9871045
if (quality == CBM_ARTIFACT_BEST) {
9881046
compression_level = ART_ZSTD_BEST;
989-
db_data = prepare_snapshot_db(db_path, &db_size, true);
1047+
db_data = prepare_snapshot_db(db_path, &db_size, true, adr_arg);
9901048
} else {
9911049
/* FAST keeps zstd-3 and its indexes, but still snapshots via
9921050
* VACUUM INTO: the raw main-file bytes of a live WAL store are a
9931051
* torn copy (#895). */
994-
db_data = prepare_snapshot_db(db_path, &db_size, false);
1052+
db_data = prepare_snapshot_db(db_path, &db_size, false, adr_arg);
9951053
}
1054+
cbm_store_adr_free(&adr);
9961055

9971056
if (!db_data || db_size == 0) {
9981057
free(db_data);
@@ -1202,6 +1261,16 @@ int cbm_artifact_import(const char *repo_path, const char *cache_db_path) {
12021261
cbm_unlink(wal);
12031262
cbm_unlink(shm);
12041263

1264+
/* Restore the imported ADR (carried in the graph DB's project_summaries)
1265+
* into the per-project sidecar, so a teammate's first index starts with the
1266+
* team's decisions. migrate-once uses INSERT OR IGNORE, so it never clobbers
1267+
* an ADR already present locally. Best-effort. */
1268+
cbm_store_t *restore = cbm_store_open_path_existing(cache_db_path);
1269+
if (restore) {
1270+
(void)cbm_store_adr_migrate_once(restore);
1271+
cbm_store_close(restore);
1272+
}
1273+
12051274
cbm_log_info("artifact.import", "db", cache_db_path, "size_mb",
12061275
itoa_buf((int)((size_t)dlen / ART_BYTES_PER_MB)));
12071276

0 commit comments

Comments
 (0)