Skip to content

Commit 001fbe2

Browse files
0xb007ab1eclaude
andcommitted
fix(adr): store ADRs in a per-project sidecar DB, untouched by reindex
Reworks the ADR-loss-across-reindex fix to Alternative 1 (per the PR review): remove the race rather than narrow it. ADRs live in a per-project sidecar SQLite DB ("<graph_db>.adr.db"), separate from the graph DB a reindex rebuilds or deletes, so there is no capture, re-read, re-apply, or lock — and no sleeps, polling, or timeout deciding an outcome. store.c: - ADR writes (store/delete/update_sections) use a cached read-write handle opened SQLITE_OPEN_READWRITE|CREATE and configured like the graph store (PRAGMA busy_timeout=10000 + journal_mode=WAL) so cross-process access waits rather than failing SQLITE_BUSY. A :memory: store keeps ADRs in the graph DB. - Reads never write: cbm_store_adr_get uses the cached RW handle if open, else opens the sidecar SQLITE_OPEN_READONLY (with the same busy timeout) when the file exists, else reads the legacy graph row without creating anything. When the sidecar exists but is NOT marked migrated (half-created), it falls back to the legacy graph row so a partial sidecar never hides an ADR. - One-time migration is atomic: adr_copy_legacy_once wraps the marker check, legacy copy, and marker insert in a single BEGIN IMMEDIATE...COMMIT (re-checking the marker under the lock; ROLLBACK->ERR on failure). New public cbm_store_adr_migrate_once. Sidecar path via cbm_path_for_file_api (Windows long-path). Dropped the write-only adr_db_path field. pipeline.c: migrate a legacy ADR into the sidecar BEFORE the format-change and forced-full routes unlink the old generation; preserve-and-abort on failure, and do NOT fail open — a query-open returning NULL preserves unless the final DB genuinely does not exist. Migration targets the FINAL path (staging suffix stripped). The early-capture/re-apply machinery and its test hook are gone. artifact.c: the exported artifact carries the ADR — snapshot_inject_adr always DELETEs project_summaries first, then injects only when an ADR is present, so a deleted ADR never resurrects through a lingering legacy row. Import restores the ADR into the destination sidecar via cbm_store_adr_restore_from_graph_if_absent: it copies the artifact's ADR (INSERT OR IGNORE — never clobbers a local one) regardless of the migrated marker, so a teammate who already indexed the project (marked sidecar, no ADR) still receives the shared decision record. store.c factors the row copy into adr_copy_rows, shared by the one-time legacy migration and the ungated restore. mcp.c: delete_project removes "<db>.adr.db" (+ -wal/-shm/-journal) only after the graph unlink succeeds; is_project_db_file skips *.adr.db. cli.c: exclude *.adr.db from index listing/counting. README + memory-core baselines updated. Tests (file-based): ADR written mid-reindex survives on BOTH the full and delta routes (route-pinned FORCED_FULL / CLOSURE_REPAIR); a legacy row migrates before a forced-full/format reindex with no prior read; migration failure (a directory at <db>.adr.db) preserves-and-aborts with the old generation intact; delete_project via the MCP handler removes the sidecar + journals; the artifact export->import round trip keeps the ADR; a deleted ADR does not resurrect on import; an import does not clobber a local ADR; an import into an already-indexed destination (marked sidecar, no ADR) restores the artifact's ADR; a half-created sidecar falls back to the legacy row. The import tests remove the destination graph DB (keeping the sidecar + marker) before importing, matching production (try_artifact_bootstrap imports only when the graph DB is missing) so the rename succeeds on Windows, not only on POSIX. Closes DeusData#516 Signed-off-by: 0xb007ab1e <[email protected]> Co-Authored-By: Claude Opus 4.8 <[email protected]>
1 parent 10ab613 commit 001fbe2

15 files changed

Lines changed: 1300 additions & 231 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -875,7 +875,7 @@ Tools requiring arguments accept JSON piped on stdin. Tools with only optional a
875875
|------|-------------|
876876
| `index_repository` | Index a repository into the graph. Auto-sync keeps it fresh only while it is the project of an open MCP session; repos indexed from the CLI or from another project's session need a re-run (`index_status` shows `watch`). Waits for the whole index by default; pass `async: true` to start it in the daemon and return at once, then poll with `status: true` (see below). |
877877
| `list_projects` | List indexed projects with paging; use `detail: "stats"` for node/edge counts and database sizes. |
878-
| `delete_project` | Remove a project and all its graph data. |
878+
| `delete_project` | Remove a project and all its graph data, including its ADR sidecar (`<db>.adr.db`). |
879879
| `index_status` | Check indexing status of a project. |
880880
| `check_index_coverage` | Check whether exact paths or a scope are indexed and fresh. A clean result means no recorded gap, not proof of completeness. |
881881

‎scripts/memory-core-baseline.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,9 +74,9 @@ src/pipeline/pass_semantic.c 3
7474
src/pipeline/pass_tests.c 3
7575
src/pipeline/pass_usages.c 4
7676
src/pipeline/path_alias.c 40
77-
src/pipeline/pipeline.c 132
77+
src/pipeline/pipeline.c 129
7878
src/pipeline/pipeline_delta.c 19
79-
src/pipeline/pipeline_incremental.c 140
79+
src/pipeline/pipeline_incremental.c 130
8080
src/pipeline/registry.c 17
8181
src/pipeline/worker_pool.c 2
8282
src/store/store.c 354

‎src/cli/cli.c‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7325,6 +7325,13 @@ static const char *get_cache_dir(const char *home_dir) {
73257325
return cbm_resolve_cache_dir();
73267326
}
73277327

7328+
/* A per-project ADR sidecar ("<db>.adr.db") sits in the cache beside its index
7329+
* but is not itself an index — exclude it from listing/counting. */
7330+
static bool cli_is_adr_sidecar(const char *name, size_t len) {
7331+
const size_t ext = sizeof(".adr.db") - 1;
7332+
return len >= ext && strcmp(name + len - ext, ".adr.db") == 0;
7333+
}
7334+
73287335
int cbm_list_indexes(const char *home_dir) {
73297336
const char *cache_dir = get_cache_dir(home_dir);
73307337
if (!cache_dir) {
@@ -7339,6 +7346,7 @@ int cbm_list_indexes(const char *home_dir) {
73397346
int count = 0;
73407347
cbm_dirent_t *ent;
73417348
while ((ent = cbm_readdir(d)) != NULL) {
7349+
/* cbm_is_project_index_db already excludes the ".adr.db" sidecar. */
73427350
if (cbm_is_project_index_db(ent->name)) {
73437351
printf(" %s/%s\n", cache_dir, ent->name);
73447352
count++;
@@ -7362,9 +7370,18 @@ int cbm_remove_indexes(const char *home_dir) {
73627370
int count = 0;
73637371
cbm_dirent_t *ent;
73647372
while ((ent = cbm_readdir(d)) != NULL) {
7365-
if (cbm_is_project_index_db(ent->name)) {
7366-
char path[CLI_BUF_1K];
7367-
snprintf(path, sizeof(path), "%s/%s", cache_dir, ent->name);
7373+
bool is_index = cbm_is_project_index_db(ent->name);
7374+
/* A "<db>.adr.db" ADR sidecar is excluded from cbm_is_project_index_db,
7375+
* but a full index wipe must still remove it (and any orphan whose index
7376+
* is already gone) — otherwise the next re-index of that name inherits a
7377+
* stale ADR. It appears as its own dirent, so remove it directly. */
7378+
bool is_adr = cli_is_adr_sidecar(ent->name, strlen(ent->name));
7379+
if (!is_index && !is_adr) {
7380+
continue;
7381+
}
7382+
char path[CLI_BUF_1K];
7383+
snprintf(path, sizeof(path), "%s/%s", cache_dir, ent->name);
7384+
if (is_index) {
73687385
/* Also remove .db.tmp if present */
73697386
char tmp_path[CLI_FIELD_1040];
73707387
snprintf(tmp_path, sizeof(tmp_path), "%s.tmp", path);
@@ -7378,6 +7395,11 @@ int cbm_remove_indexes(const char *home_dir) {
73787395
* its .db. Sidecars are not indexes, so count is unchanged. */
73797396
cbm_remove_db_sidecars(path);
73807397
cbm_remove_db_sidecars(tmp_path);
7398+
} else {
7399+
/* ADR sidecar: remove it and its own -wal/-shm/-journal; not an
7400+
* index, so count is unchanged. */
7401+
cbm_unlink(path);
7402+
cbm_remove_db_sidecars(path);
73817403
}
73827404
}
73837405
cbm_closedir(d);
@@ -10604,7 +10626,8 @@ static int count_db_indexes(const char *home) {
1060410626
int count = 0;
1060510627
cbm_dirent_t *ent;
1060610628
while ((ent = cbm_readdir(d)) != NULL) {
10607-
if (cbm_is_project_index_db(ent->name)) {
10629+
if (cbm_is_project_index_db(ent->name) &&
10630+
!cli_is_adr_sidecar(ent->name, strlen(ent->name))) {
1060810631
count++;
1060910632
}
1061010633
}

‎src/foundation/str_util.c‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -317,14 +317,22 @@ bool cbm_is_internal_cache_db(const char *filename) {
317317

318318
bool cbm_is_project_index_db(const char *filename) {
319319
static const char db_ext[] = ".db";
320+
static const char adr_ext[] = ".adr.db";
320321
const size_t ext_len = sizeof(db_ext) - 1;
322+
const size_t adr_len = sizeof(adr_ext) - 1;
321323
if (!filename) {
322324
return false;
323325
}
324326
size_t len = strlen(filename);
325327
if (len <= ext_len || strcmp(filename + len - ext_len, db_ext) != 0) {
326328
return false;
327329
}
330+
/* A per-project ADR sidecar ("<db>.adr.db") ends in ".db" but is not itself
331+
* an index — exclude it from every index scan/count (MCP loops + the CLI's
332+
* cbm_list_indexes / count_db_indexes all gate on this helper). */
333+
if (len >= adr_len && strcmp(filename + len - adr_len, adr_ext) == 0) {
334+
return false;
335+
}
328336
return !cbm_is_internal_cache_db(filename);
329337
}
330338

‎src/mcp/mcp.c‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -766,7 +766,7 @@ static const tool_def_t TOOLS[] = {
766766
"\"offset\":{\"type\":\"integer\",\"default\":0,\"minimum\":0},"
767767
"\"metadata_only\":{\"type\":\"boolean\",\"default\":false,"
768768
"\"description\":\"Compatibility: omit counts, size, and branch.\"}}}"},
769-
{"delete_project", "Delete a project from the index",
769+
{"delete_project", "Delete a project from the index, including its ADR store",
770770
"{\"type\":\"object\",\"properties\":{\"project\":{\"type\":\"string\"}},\"required\":["
771771
"\"project\"]}"},
772772

@@ -7508,7 +7508,9 @@ static char *handle_index_status(cbm_mcp_server_t *srv, const char *args) {
75087508
return result;
75097509
}
75107510

7511-
/* delete_project: just erase the .db file (and WAL/SHM). */
7511+
/* delete_project: erase the .db file (and WAL/SHM) plus the ADR sidecar
7512+
* "<db>.adr.db" (and its WAL/SHM/journal), so deleting a project removes its
7513+
* ADR store with it. */
75127514
static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
75137515
char *name = get_project_arg(args);
75147516
if (!name) {
@@ -7547,18 +7549,37 @@ static char *handle_delete_project(cbm_mcp_server_t *srv, const char *args) {
75477549
const char *error_detail = NULL;
75487550
bool is_error = false;
75497551

7552+
/* ADR sidecar lives beside the graph DB as "<db>.adr.db" — remove it (and
7553+
* its own WAL/SHM/journal) whenever the project is deleted. Sized as the
7554+
* graph path plus the ".adr.db" suffix. */
7555+
char adr_path[CBM_SZ_1K + sizeof(".adr.db")];
7556+
int adr_n = snprintf(adr_path, sizeof(adr_path), "%s.adr.db", path);
7557+
bool have_adr_path = adr_n > 0 && (size_t)adr_n < sizeof(adr_path);
7558+
75507559
if (exists) {
75517560
int rc = cbm_unlink(path);
75527561
(void)cbm_unlink(wal);
75537562
(void)cbm_unlink(shm);
75547563
if (rc == 0) {
75557564
status = "deleted";
7565+
/* Only drop the ADR store once the graph DB is actually gone, so a
7566+
* failed graph delete leaves the project (and its ADR) intact. */
7567+
if (have_adr_path) {
7568+
(void)cbm_unlink(adr_path);
7569+
(void)cbm_remove_db_sidecars(adr_path);
7570+
}
75567571
} else {
75577572
status = "delete_failed";
75587573
error_detail = strerror(errno);
75597574
is_error = true;
75607575
}
75617576
} else {
7577+
/* No graph DB, but a sidecar could still linger (e.g. ADR written then
7578+
* the index deleted out of band) — best-effort clean it up. */
7579+
if (have_adr_path) {
7580+
(void)cbm_unlink(adr_path);
7581+
(void)cbm_remove_db_sidecars(adr_path);
7582+
}
75627583
is_error = true;
75637584
}
75647585

@@ -20097,9 +20118,10 @@ static char *handle_manage_adr(cbm_mcp_server_t *srv, const char *args) {
2009720118
}
2009820119
}
2009920120

20100-
/* ADRs are stored in the SQLite store (project_summaries), the SAME
20101-
* backend the UI /api/adr endpoints use — so writes via the MCP tool and
20102-
* the UI are visible to each other (#256). */
20121+
/* ADRs are stored in the per-project ADR sidecar DB ("<db>.adr.db"), the
20122+
* SAME backend the UI /api/adr endpoints use (via cbm_store_adr_*) — so
20123+
* writes via the MCP tool and the UI are visible to each other (#256), and
20124+
* a reindex of the graph DB never disturbs them. */
2010320125
store_recovery_status_t recovery_status = STORE_RECOVERY_NONE;
2010420126
cbm_store_t *resolved =
2010520127
resolve_store_internal(srv, project, mutation_held, !write_request, &recovery_status, true);

‎src/pipeline/artifact.c‎

Lines changed: 78 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -884,13 +884,54 @@ static void artifact_snapshot_tmp_close(artifact_snapshot_tmp_t *tmp) {
884884
tmp->dir[0] = '\0';
885885
}
886886

887+
/* Make the snapshot's project_summaries carry exactly the current ADR — and
888+
* nothing stale. ADRs now live in the "<db>.adr.db" sidecar, but a legacy
889+
* pre-sidecar row can still linger in the graph DB and gets copied into the
890+
* snapshot by VACUUM INTO; incremental/closure-repair clones keep carrying it.
891+
* So ALWAYS clear project_summaries first, then insert only when an ADR is
892+
* present (`adr` is NULL when the sidecar has none). Otherwise a no-ADR export
893+
* would ship the stale row and a teammate's import would restore a decision
894+
* record that was deleted. Best-effort: on any failure the artifact simply
895+
* omits the ADR, as before the sidecar existed. */
896+
static void snapshot_inject_adr(const char *snapshot_path, const cbm_adr_t *adr) {
897+
sqlite3 *db = NULL;
898+
if (sqlite3_open_v2(snapshot_path, &db, SQLITE_OPEN_READWRITE, NULL) != SQLITE_OK) {
899+
sqlite3_close(db);
900+
return;
901+
}
902+
sqlite3_exec(db,
903+
"CREATE TABLE IF NOT EXISTS project_summaries ("
904+
" project TEXT PRIMARY KEY, summary TEXT NOT NULL, source_hash TEXT NOT NULL,"
905+
" created_at TEXT NOT NULL, updated_at TEXT NOT NULL);",
906+
NULL, NULL, NULL);
907+
/* Drop any stale row (a deleted ADR must not resurrect through the export). */
908+
sqlite3_exec(db, "DELETE FROM project_summaries;", NULL, NULL, NULL);
909+
if (adr && adr->project && adr->content) {
910+
sqlite3_stmt *st = NULL;
911+
if (sqlite3_prepare_v2(db,
912+
"INSERT INTO project_summaries "
913+
"(project, summary, source_hash, created_at, updated_at) "
914+
"VALUES (?1, ?2, '', ?3, ?4);",
915+
-1, &st, NULL) == SQLITE_OK) {
916+
sqlite3_bind_text(st, 1, adr->project, -1, SQLITE_TRANSIENT);
917+
sqlite3_bind_text(st, 2, adr->content, -1, SQLITE_TRANSIENT);
918+
sqlite3_bind_text(st, 3, adr->created_at ? adr->created_at : "", -1, SQLITE_TRANSIENT);
919+
sqlite3_bind_text(st, 4, adr->updated_at ? adr->updated_at : "", -1, SQLITE_TRANSIENT);
920+
sqlite3_step(st);
921+
sqlite3_finalize(st);
922+
}
923+
}
924+
sqlite3_close(db);
925+
}
926+
887927
/* Prepare a stripped DB copy for best-quality export.
888-
* VACUUM INTO → (optionally) drop indexes → VACUUM. Returns malloc'd buffer
889-
* or NULL. VACUUM INTO runs on BOTH quality levels: it is the consistent
890-
* snapshot — the store runs in WAL mode, so raw main-file bytes miss
891-
* committed transactions still in the -wal and can be mid-checkpoint torn
892-
* (#895). Only the index-stripping is BEST-only. */
893-
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes) {
928+
* VACUUM INTO → (optionally) drop indexes → VACUUM → (optionally) inject the
929+
* ADR. Returns malloc'd buffer or NULL. VACUUM INTO runs on BOTH quality
930+
* levels: it is the consistent snapshot — the store runs in WAL mode, so raw
931+
* main-file bytes miss committed transactions still in the -wal and can be
932+
* mid-checkpoint torn (#895). Only the index-stripping is BEST-only. */
933+
static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool strip_indexes,
934+
const cbm_adr_t *adr) {
894935
artifact_snapshot_tmp_t tmp;
895936
if (!artifact_snapshot_tmp_open(&tmp)) {
896937
artifact_export_fail("prepare_snapshot_dir", cbm_tmpdir(), "private_tmpdir_failed", errno);
@@ -935,6 +976,11 @@ static char *prepare_snapshot_db(const char *db_path, size_t *out_size, bool str
935976
}
936977
}
937978

979+
/* Carry the ADR (from the sidecar) into the snapshot so the exported
980+
* artifact ships the team's decision record. After the strip/VACUUM, so the
981+
* row is not dropped. */
982+
snapshot_inject_adr(tmp_path, adr);
983+
938984
/* Reopened by path rather than held on a descriptor across VACUUM INTO,
939985
* because sqlite owns the create. The private directory is what makes that
940986
* safe: an attacker who cannot enter it cannot swap the file underneath. */
@@ -980,15 +1026,28 @@ int cbm_artifact_export(const char *db_path, const char *repo_path, const char *
9801026
char *db_data = NULL;
9811027
int compression_level = ART_ZSTD_FAST;
9821028

1029+
/* Fetch the ADR from the sidecar (read-only; does not create it) so the
1030+
* snapshot can carry it — the team-shared artifact used to include the ADR
1031+
* via project_summaries, and the sidecar must not silently break that. */
1032+
cbm_adr_t adr = {0};
1033+
bool have_adr = false;
1034+
cbm_store_t *adr_src = cbm_store_open_path_query(db_path);
1035+
if (adr_src) {
1036+
have_adr = cbm_store_adr_get(adr_src, project_name, &adr) == CBM_STORE_OK;
1037+
cbm_store_close(adr_src);
1038+
}
1039+
const cbm_adr_t *adr_arg = have_adr ? &adr : NULL;
1040+
9831041
if (quality == CBM_ARTIFACT_BEST) {
9841042
compression_level = ART_ZSTD_BEST;
985-
db_data = prepare_snapshot_db(db_path, &db_size, true);
1043+
db_data = prepare_snapshot_db(db_path, &db_size, true, adr_arg);
9861044
} else {
9871045
/* FAST keeps zstd-3 and its indexes, but still snapshots via
9881046
* VACUUM INTO: the raw main-file bytes of a live WAL store are a
9891047
* torn copy (#895). */
990-
db_data = prepare_snapshot_db(db_path, &db_size, false);
1048+
db_data = prepare_snapshot_db(db_path, &db_size, false, adr_arg);
9911049
}
1050+
cbm_store_adr_free(&adr);
9921051

9931052
if (!db_data || db_size == 0) {
9941053
free(db_data);
@@ -1198,6 +1257,17 @@ int cbm_artifact_import(const char *repo_path, const char *cache_db_path) {
11981257
cbm_unlink(wal);
11991258
cbm_unlink(shm);
12001259

1260+
/* Restore the imported ADR (carried in the graph DB's project_summaries) into
1261+
* the per-project sidecar so it arrives with the shared artifact — even for a
1262+
* teammate who already indexed the project (a marked sidecar with no ADR),
1263+
* which the marker-gated migrate-once would skip. INSERT OR IGNORE means a
1264+
* local ADR is never overwritten. Best-effort. */
1265+
cbm_store_t *restore = cbm_store_open_path_existing(cache_db_path);
1266+
if (restore) {
1267+
(void)cbm_store_adr_restore_from_graph_if_absent(restore);
1268+
cbm_store_close(restore);
1269+
}
1270+
12011271
cbm_log_info("artifact.import", "db", cache_db_path, "size_mb",
12021272
itoa_buf((int)((size_t)dlen / ART_BYTES_PER_MB)));
12031273

0 commit comments

Comments
 (0)