Skip to content

Commit 2d2c0fd

Browse files
0xb007ab1eclaude
andcommitted
fix(adr): store ADRs in a per-project sidecar DB, untouched by reindex
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review): remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps, polling, or timeout deciding an outcome. store.c: - ADR ops route to the sidecar. Writes (store/delete/update_sections) use a cached read-write handle opened SQLITE_OPEN_READWRITE|CREATE and configured like the graph store — PRAGMA busy_timeout=10000 + journal_mode=WAL — so cross-process access waits rather than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB (never file-swapped, no race). - Reads never write: cbm_store_adr_get uses the cached RW handle if open, else opens the sidecar SQLITE_OPEN_READONLY when the file exists, else reads the legacy graph row without creating anything (a query-only cache is never mutated). - One-time migration guarded by an adr_meta 'migrated' marker: copy the legacy project_summaries row (INSERT OR IGNORE) into the sidecar exactly once, so it can never resurrect a row deleted after migration. New public cbm_store_adr_migrate_once. adr_update_sections' BEGIN IMMEDIATE runs on the ADR connection it protects. - Sidecar path built through cbm_path_for_file_api (Windows long-path prefix). - Dropped the write-only adr_db_path field (keeps the memory-core baseline). pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and forced-full routes unlink the old generation (otherwise an upgrade whose first action is a full reindex loses it); preserve-and-abort on failure. Migration targets the FINAL path (staging suffix stripped), not the discarded stage's sidecar. The early-capture/re-apply machinery and its test hook are gone. artifact.c: the exported artifact carries the ADR — inject the sidecar's ADR into the snapshot's project_summaries before compression; on import, restore it into the destination sidecar via cbm_store_adr_migrate_once. mcp.c: delete_project also removes "<db>.adr.db" (+ its -wal/-shm/-journal), and its tool description says so. cli.c: exclude "*.adr.db" from index listing/ counting. README + memory-core baselines updated. Tests (file-based): an ADR written mid-reindex survives; a legacy row migrates before a forced-full/format reindex with no prior read; delete_project via the MCP handler removes the sidecar and a same-name re-index has no ADR; the artifact export→import round trip keeps the ADR. Closes DeusData#516 Signed-off-by: 0xb007ab1e <[email protected]> Co-Authored-By: Claude Opus 4.8 <[email protected]>
1 parent 64c23fa commit 2d2c0fd

13 files changed

Lines changed: 818 additions & 228 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -671,7 +671,7 @@ JSON arguments can also be piped on stdin, for tools that take arguments. A tool
671671
|------|-------------|
672672
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh after that. |
673673
| `list_projects` | List all indexed projects with node/edge counts. |
674-
| `delete_project` | Remove a project and all its graph data. |
674+
| `delete_project` | Remove a project and all its graph data, including its ADR sidecar (`<db>.adr.db`). |
675675
| `index_status` | Check indexing status of a project. |
676676
| `check_index_coverage` | Check whether exact paths or a scope are indexed and fresh. A clean result means no recorded gap, not proof of completeness. |
677677

‎scripts/memory-core-baseline.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,9 +77,9 @@ src/pipeline/pass_semantic.c 3
7777
src/pipeline/pass_tests.c 3
7878
src/pipeline/pass_usages.c 4
7979
src/pipeline/path_alias.c 42
80-
src/pipeline/pipeline.c 134
80+
src/pipeline/pipeline.c 131
8181
src/pipeline/pipeline_delta.c 19
82-
src/pipeline/pipeline_incremental.c 144
82+
src/pipeline/pipeline_incremental.c 134
8383
src/pipeline/registry.c 17
8484
src/pipeline/worker_pool.c 2
8585
src/store/store.c 358

‎src/cli/cli.c‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7193,6 +7193,13 @@ static const char *get_cache_dir(const char *home_dir) {
71937193
return cbm_resolve_cache_dir();
71947194
}
71957195

7196+
/* A per-project ADR sidecar ("<db>.adr.db") sits in the cache beside its index
7197+
* but is not itself an index — exclude it from listing/counting. */
7198+
static bool cli_is_adr_sidecar(const char *name, size_t len) {
7199+
const size_t ext = sizeof(".adr.db") - 1;
7200+
return len >= ext && strcmp(name + len - ext, ".adr.db") == 0;
7201+
}
7202+
71967203
int cbm_list_indexes(const char *home_dir) {
71977204
const char *cache_dir = get_cache_dir(home_dir);
71987205
if (!cache_dir) {
@@ -7208,7 +7215,8 @@ int cbm_list_indexes(const char *home_dir) {
72087215
cbm_dirent_t *ent;
72097216
while ((ent = cbm_readdir(d)) != NULL) {
72107217
size_t len = strlen(ent->name);
7211-
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0) {
7218+
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0 &&
7219+
!cli_is_adr_sidecar(ent->name, len)) {
72127220
printf(" %s/%s\n", cache_dir, ent->name);
72137221
count++;
72147222
}
@@ -10437,7 +10445,8 @@ static int count_db_indexes(const char *home) {
1043710445
cbm_dirent_t *ent;
1043810446
while ((ent = cbm_readdir(d)) != NULL) {
1043910447
size_t len = strlen(ent->name);
10440-
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0) {
10448+
if (len > DB_EXT_LEN && strcmp(ent->name + len - DB_EXT_LEN, ".db") == 0 &&
10449+
!cli_is_adr_sidecar(ent->name, len)) {
1044110450
count++;
1044210451
}
1044310452
}

‎src/mcp/mcp.c‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -671,7 +671,7 @@ static const tool_def_t TOOLS[] = {
671671
"\"offset\":{\"type\":\"integer\",\"default\":0,\"minimum\":0},"
672672
"\"metadata_only\":{\"type\":\"boolean\",\"default\":false,"
673673
"\"description\":\"Compatibility: omit counts, size, and branch.\"}}}"},
674-
{"delete_project", "Delete a project from the index",
674+
{"delete_project", "Delete a project from the index, including its ADR store",
675675
"{\"type\":\"object\",\"properties\":{\"project\":{\"type\":\"string\"}},\"required\":["
676676
"\"project\"]}"},
677677

@@ -6809,7 +6809,9 @@ static char *handle_index_status(cbm_mcp_server_t *srv, const char *args) {
68096809
return result;
68106810
}
68116811

6812-
/* delete_project: just erase the .db file (and WAL/SHM). */
6812+
/* delete_project: erase the .db file (and WAL/SHM) plus the ADR sidecar
6813+
* "<db>.adr.db" (and its WAL/SHM/journal), so deleting a project removes its
6814+
* ADR store with it. */
68136815
static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
68146816
char *name = get_project_arg(args);
68156817
if (!name) {
@@ -6848,10 +6850,20 @@ static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
68486850
const char *error_detail = NULL;
68496851
bool is_error = false;
68506852

6853+
/* ADR sidecar lives beside the graph DB as "<db>.adr.db" — remove it (and
6854+
* its own WAL/SHM/journal) whenever the project is deleted. */
6855+
char adr_path[CBM_SZ_1K];
6856+
int adr_n = snprintf(adr_path, sizeof(adr_path), "%s.adr.db", path);
6857+
bool have_adr_path = adr_n > 0 && (size_t)adr_n < sizeof(adr_path);
6858+
68516859
if (exists) {
68526860
int rc = cbm_unlink(path);
68536861
(void)cbm_unlink(wal);
68546862
(void)cbm_unlink(shm);
6863+
if (have_adr_path) {
6864+
(void)cbm_unlink(adr_path);
6865+
(void)cbm_remove_db_sidecars(adr_path);
6866+
}
68556867
if (rc == 0) {
68566868
status = "deleted";
68576869
} else {
@@ -6860,6 +6872,12 @@ static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
68606872
is_error = true;
68616873
}
68626874
} else {
6875+
/* No graph DB, but a sidecar could still linger (e.g. ADR written then
6876+
* the index deleted out of band) — best-effort clean it up. */
6877+
if (have_adr_path) {
6878+
(void)cbm_unlink(adr_path);
6879+
(void)cbm_remove_db_sidecars(adr_path);
6880+
}
68636881
is_error = true;
68646882
}
68656883

@@ -17173,9 +17191,10 @@ static char *handle_manage_adr(cbm_mcp_server_t *srv, const char *args) {
1717317191
}
1717417192
}
1717517193

17176-
/* ADRs are stored in the SQLite store (project_summaries), the SAME
17177-
* backend the UI /api/adr endpoints use — so writes via the MCP tool and
17178-
* the UI are visible to each other (#256). */
17194+
/* ADRs are stored in the per-project ADR sidecar DB ("<db>.adr.db"), the
17195+
* SAME backend the UI /api/adr endpoints use (via cbm_store_adr_*) — so
17196+
* writes via the MCP tool and the UI are visible to each other (#256), and
17197+
* a reindex of the graph DB never disturbs them. */
1717917198
store_recovery_status_t recovery_status = STORE_RECOVERY_NONE;
1718017199
cbm_store_t *resolved =
1718117200
resolve_store_internal(srv, project, mutation_held, !write_request, &recovery_status, true);

‎src/pipeline/artifact.c‎

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -888,13 +888,49 @@ static void artifact_snapshot_tmp_close(artifact_snapshot_tmp_t *tmp) {
888888
tmp->dir[0] = '\0';
889889
}
890890

891+
/* Write the ADR into the snapshot's project_summaries so the exported artifact
892+
* carries the team's decision record (ADRs now live in the "<db>.adr.db"
893+
* sidecar, which the VACUUM INTO snapshot does not include). Best-effort: a
894+
* failure just means this artifact omits the ADR, exactly as before the sidecar
895+
* existed. */
896+
static void snapshot_inject_adr(const char *snapshot_path, const cbm_adr_t *adr) {
897+
if (!adr || !adr->project || !adr->content) {
898+
return;
899+
}
900+
sqlite3 *db = NULL;
901+
if (sqlite3_open_v2(snapshot_path, &db, SQLITE_OPEN_READWRITE, NULL) != SQLITE_OK) {
902+
sqlite3_close(db);
903+
return;
904+
}
905+
sqlite3_exec(db,
906+
"CREATE TABLE IF NOT EXISTS project_summaries ("
907+
" project TEXT PRIMARY KEY, summary TEXT NOT NULL, source_hash TEXT NOT NULL,"
908+
" created_at TEXT NOT NULL, updated_at TEXT NOT NULL);",
909+
NULL, NULL, NULL);
910+
sqlite3_stmt *st = NULL;
911+
if (sqlite3_prepare_v2(db,
912+
"INSERT OR REPLACE INTO project_summaries "
913+
"(project, summary, source_hash, created_at, updated_at) "
914+
"VALUES (?1, ?2, '', ?3, ?4);",
915+
-1, &st, NULL) == SQLITE_OK) {
916+
sqlite3_bind_text(st, 1, adr->project, -1, SQLITE_TRANSIENT);
917+
sqlite3_bind_text(st, 2, adr->content, -1, SQLITE_TRANSIENT);
918+
sqlite3_bind_text(st, 3, adr->created_at ? adr->created_at : "", -1, SQLITE_TRANSIENT);
919+
sqlite3_bind_text(st, 4, adr->updated_at ? adr->updated_at : "", -1, SQLITE_TRANSIENT);
920+
sqlite3_step(st);
921+
sqlite3_finalize(st);
922+
}
923+
sqlite3_close(db);
924+
}
925+
891926
/* Prepare a stripped DB copy for best-quality export.
892-
* VACUUM INTO → (optionally) drop indexes → VACUUM. Returns malloc'd buffer
893-
* or NULL. VACUUM INTO runs on BOTH quality levels: it is the consistent
894-
* snapshot — the store runs in WAL mode, so raw main-file bytes miss
895-
* committed transactions still in the -wal and can be mid-checkpoint torn
896-
* (#895). Only the index-stripping is BEST-only. */
897-
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes) {
927+
* VACUUM INTO → (optionally) drop indexes → VACUUM → (optionally) inject the
928+
* ADR. Returns malloc'd buffer or NULL. VACUUM INTO runs on BOTH quality
929+
* levels: it is the consistent snapshot — the store runs in WAL mode, so raw
930+
* main-file bytes miss committed transactions still in the -wal and can be
931+
* mid-checkpoint torn (#895). Only the index-stripping is BEST-only. */
932+
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes,
933+
const cbm_adr_t *adr) {
898934
artifact_snapshot_tmp_t tmp;
899935
if (!artifact_snapshot_tmp_open(&tmp)) {
900936
artifact_export_fail("prepare_snapshot_dir", cbm_tmpdir(), "private_tmpdir_failed", errno);
@@ -939,6 +975,11 @@ static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool str
939975
}
940976
}
941977

978+
/* Carry the ADR (from the sidecar) into the snapshot so the exported
979+
* artifact ships the team's decision record. After the strip/VACUUM, so the
980+
* row is not dropped. */
981+
snapshot_inject_adr(tmp_path, adr);
982+
942983
/* Reopened by path rather than held on a descriptor across VACUUM INTO,
943984
* because sqlite owns the create. The private directory is what makes that
944985
* safe: an attacker who cannot enter it cannot swap the file underneath. */
@@ -984,15 +1025,28 @@ int cbm_artifact_export(const char *db_path, const char *repo_path, const char *
9841025
char *db_data = NULL;
9851026
int compression_level = ART_ZSTD_FAST;
9861027

1028+
/* Fetch the ADR from the sidecar (read-only; does not create it) so the
1029+
* snapshot can carry it — the team-shared artifact used to include the ADR
1030+
* via project_summaries, and the sidecar must not silently break that. */
1031+
cbm_adr_t adr = {0};
1032+
bool have_adr = false;
1033+
cbm_store_t *adr_src = cbm_store_open_path_query(db_path);
1034+
if (adr_src) {
1035+
have_adr = cbm_store_adr_get(adr_src, project_name, &adr) == CBM_STORE_OK;
1036+
cbm_store_close(adr_src);
1037+
}
1038+
const cbm_adr_t *adr_arg = have_adr ? &adr : NULL;
1039+
9871040
if (quality == CBM_ARTIFACT_BEST) {
9881041
compression_level = ART_ZSTD_BEST;
989-
db_data = prepare_snapshot_db(db_path, &db_size, true);
1042+
db_data = prepare_snapshot_db(db_path, &db_size, true, adr_arg);
9901043
} else {
9911044
/* FAST keeps zstd-3 and its indexes, but still snapshots via
9921045
* VACUUM INTO: the raw main-file bytes of a live WAL store are a
9931046
* torn copy (#895). */
994-
db_data = prepare_snapshot_db(db_path, &db_size, false);
1047+
db_data = prepare_snapshot_db(db_path, &db_size, false, adr_arg);
9951048
}
1049+
cbm_store_adr_free(&adr);
9961050

9971051
if (!db_data || db_size == 0) {
9981052
free(db_data);
@@ -1202,6 +1256,16 @@ int cbm_artifact_import(const char *repo_path, const char *cache_db_path) {
12021256
cbm_unlink(wal);
12031257
cbm_unlink(shm);
12041258

1259+
/* Restore the imported ADR (carried in the graph DB's project_summaries)
1260+
* into the per-project sidecar, so a teammate's first index starts with the
1261+
* team's decisions. migrate-once uses INSERT OR IGNORE, so it never clobbers
1262+
* an ADR already present locally. Best-effort. */
1263+
cbm_store_t *restore = cbm_store_open_path_existing(cache_db_path);
1264+
if (restore) {
1265+
(void)cbm_store_adr_migrate_once(restore);
1266+
cbm_store_close(restore);
1267+
}
1268+
12051269
cbm_log_info("artifact.import", "db", cache_db_path, "size_mb",
12061270
itoa_buf((int)((size_t)dlen / ART_BYTES_PER_MB)));
12071271

0 commit comments

Comments
 (0)