Skip to content

Commit 1ed721b

Browse files
committed
feat(store): external per-node annotations, keyed by qualified_name
Nodes carry only what the indexer can derive from a syntax tree. Some facts about a symbol simply are not in the syntax: an elaborated hardware instance hierarchy, a resolved bit width or packed-struct layout, a profiler's sample count. A tool that understands the language semantically can compute them, but there was no way to attach them to the graph — nodes have a fixed column set, and query_graph's Cypher is read-only. Add a side table rather than widening nodes: node_annotations(qualified_name, source, props JSON, updated_at) PRIMARY KEY (qualified_name, source) qualified_name is the key because it is the only node identifier stable across a re-index (ids are AUTOINCREMENT and change on every rebuild). `source` namespaces annotators so several coexist and one can be refreshed alone. Rows whose key matches no node are inert, never an error — which is why upsert reports how many of the written source's rows actually join a node: a payload that matches nothing is the signature of a wrong key convention, and the caller needs to see that instead of a clean success. Read side: cbm_store_has_annotations is a cached one-probe gate so an un-annotated project keeps its previous cost; cbm_store_annotation_prop pulls one key via SQL json_extract on the primary-key index; merged_json returns {"<source>": {props}} per node, keeping annotators that disagree distinguishable. Reads treat a missing table as "no annotations", so DBs written by older builds keep working; the existence probe is cached but its answer is not, so a table created by another process mid-session is picked up. Survive re-index: a full re-index DELETES the .db file and rebuilds it from the graph buffer, which is why the ADR already had explicit carry-over (DeusData#516). Annotations get the same treatment — captured before the delete, restored after the rebuild, with a 200k-row ceiling whose truncation is logged rather than silent. An incremental index never touched them. delete_project still removes everything, annotations included, by design. Verified: annotations survive a real full re-index of a 900-node SystemVerilog project (node ids restart from 1, the rows keep their original import timestamp and still join), plus unit coverage for the rebuild case, source namespacing, replace semantics, the read path and unkeyable rows. Signed-off-by: Oliver Barrera <[email protected]>
1 parent 7d67801 commit 1ed721b

5 files changed

Lines changed: 943 additions & 2 deletions

File tree

‎src/pipeline/pipeline.c‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,18 @@
1212
*/
1313
#include "foundation/constants.h"
1414

15-
enum { CBM_DIR_PERMS = 0755, PL_RING = 4, PL_RING_MASK = 3, PL_SEQ_PASSES = 6, PL_WAL_BUF = 1040 };
15+
enum {
16+
CBM_DIR_PERMS = 0755,
17+
PL_RING = 4,
18+
PL_RING_MASK = 3,
19+
PL_SEQ_PASSES = 6,
20+
PL_WAL_BUF = 1040,
21+
/* Ceiling on annotations carried across a full re-index. Well above any real
22+
* annotated repo (a whole-kernel-scale import is ~10^4 rows); a set larger
23+
* than this is truncated rather than held wholly in memory mid-index, and
24+
* the truncation is logged, never silent. */
25+
PL_ANNOTATIONS_CARRY_MAX = 200000,
26+
};
1627
#define PL_NSEC_PER_SEC 1000000000LL
1728
#include "pipeline/pipeline.h"
1829
#include "pipeline/artifact.h"
@@ -120,6 +131,13 @@ struct cbm_pipeline {
120131
/* ADR (project_summaries) captured before a full-reindex DB delete, so it
121132
* can be restored after the rebuild. NULL when no ADR existed. Issue #516. */
122133
char *saved_adr;
134+
135+
/* External node annotations (node_annotations) captured before the same
136+
* delete. They are keyed by qualified_name — stable across a rebuild — so
137+
* carrying them over is what makes an import survive a re-index instead of
138+
* silently vanishing with the old DB file. NULL/0 when none existed. */
139+
cbm_annotation_t *saved_annotations;
140+
int saved_annotation_count;
123141
};
124142

125143
/* ── Global pkgmap (one active pipeline at a time) ─────────────── */
@@ -242,6 +260,9 @@ void cbm_pipeline_free(cbm_pipeline_t *p) {
242260
free(p->saved_adr); /* freed here too: error paths can exit before the
243261
* restore in dump_and_persist_hashes runs. Issue #516. */
244262
p->saved_adr = NULL;
263+
cbm_store_free_annotations(p->saved_annotations, p->saved_annotation_count);
264+
p->saved_annotations = NULL;
265+
p->saved_annotation_count = 0;
245266
cbm_git_context_free(&p->git_ctx);
246267
/* gbuf, store, registry freed during/after run */
247268
/* Defensively free userconfig in case run() was never called or panicked */
@@ -1135,7 +1156,10 @@ static int try_incremental_or_delete_db(cbm_pipeline_t *p, cbm_file_info_t *file
11351156
}
11361157
cbm_log_info("pipeline.route", "path", "reindex", "action", "deleting old db");
11371158
/* Capture any ADR before deleting the DB so the full-reindex rebuild can
1138-
* restore it (project_summaries is otherwise lost). Issue #516. */
1159+
* restore it (project_summaries is otherwise lost). Issue #516.
1160+
* Same for external node annotations: a full re-index deletes the DB file
1161+
* wholesale, so without this an imported annotation set would disappear on
1162+
* the next index run even though its qualified_name keys stay valid. */
11391163
{
11401164
cbm_store_t *adr_store = cbm_store_open_path(db_path);
11411165
if (adr_store) {
@@ -1147,6 +1171,11 @@ static int try_incremental_or_delete_db(cbm_pipeline_t *p, cbm_file_info_t *file
11471171
}
11481172
cbm_store_adr_free(&existing);
11491173
}
1174+
cbm_store_free_annotations(p->saved_annotations, p->saved_annotation_count);
1175+
p->saved_annotations = NULL;
1176+
p->saved_annotation_count = 0;
1177+
(void)cbm_store_annotations_get(adr_store, NULL, NULL, PL_ANNOTATIONS_CARRY_MAX,
1178+
&p->saved_annotations, &p->saved_annotation_count);
11501179
cbm_store_close(adr_store);
11511180
}
11521181
}
@@ -1242,6 +1271,26 @@ static int dump_and_persist_hashes(cbm_pipeline_t *p, const cbm_file_info_t *fil
12421271
}
12431272
CBM_PROF_END("persist", "3_adr_restore", t_adr);
12441273

1274+
/* Restore the external annotations captured before the DB delete. Their
1275+
* qualified_name keys survive the rebuild, so the rows re-attach to the
1276+
* freshly minted nodes. A failed restore is logged, not swallowed —
1277+
* losing an import silently is the failure mode this exists to avoid. */
1278+
if (p->saved_annotation_count > 0 && p->saved_annotations) {
1279+
if (cbm_store_annotations_upsert(hash_store, p->project_name, p->saved_annotations,
1280+
p->saved_annotation_count, NULL) != CBM_STORE_OK) {
1281+
cbm_log_error("pipeline.err", "phase", "annotations_restore", "project",
1282+
p->project_name, "rows", itoa_buf(p->saved_annotation_count));
1283+
} else {
1284+
cbm_log_info("pipeline.annotations_restored", "project", p->project_name, "rows",
1285+
itoa_buf(p->saved_annotation_count));
1286+
}
1287+
if (p->saved_annotation_count >= PL_ANNOTATIONS_CARRY_MAX) {
1288+
cbm_log_warn("pipeline.annotations_truncated", "project", p->project_name,
1289+
"carried", itoa_buf(PL_ANNOTATIONS_CARRY_MAX), "action",
1290+
"re-run import_annotations to restore the remainder");
1291+
}
1292+
}
1293+
12451294
/* Batch the per-file hash upserts into ONE transaction. The per-file
12461295
* cbm_store_upsert_file_hash path autocommits, i.e. file_count fsyncs
12471296
* (~89k on the kernel); cbm_store_upsert_file_hash_batch wraps the same
@@ -1376,6 +1425,9 @@ static int dump_and_persist_hashes(cbm_pipeline_t *p, const cbm_file_info_t *fil
13761425
}
13771426
free(p->saved_adr);
13781427
p->saved_adr = NULL;
1428+
cbm_store_free_annotations(p->saved_annotations, p->saved_annotation_count);
1429+
p->saved_annotations = NULL;
1430+
p->saved_annotation_count = 0;
13791431

13801432
/* Export persistent artifact if enabled */
13811433
if (p->persistence) {

0 commit comments

Comments
 (0)