Incremental reindex costs ≈ a full index, especially after artifact bootstrap #867

Problem

After artifact bootstrap (and to a lesser degree on steady-state incremental runs), an incremental re-index ends up re-doing almost the work of a full index. Three independent causes, verified against main (74b27f3):

P1 — Artifact bootstrap doesn't actually avoid the full-index cost

try_artifact_bootstrap (src/mcp/mcp.c) byte-copies the shared artifact DB into the local cache; the pipeline then routes to cbm_pipeline_run_incremental because hash rows exist (src/pipeline/pipeline.c, the pipeline.route path=incremental branch).

classify_files (src/pipeline/pipeline_incremental.c) classifies changed/unchanged only by mtime_ns + size. The imported file_hashes rows carry the exporter's mtimes; a fresh clone stamps every file with checkout time → essentially every file mismatches → full extract + resolve + post-passes. Bootstrap saves almost nothing on run 1 — the exact run it exists to make cheap.

Two building blocks already exist but are dead/unused in production:

• cbm_artifact_commit() (src/pipeline/artifact.c) reads the git commit from artifact.json — never called in production.
• cbm_store_upsert_file_hash_batch() (src/store/store.c) — exists, never called in production.

P2 — persist_hashes is row-at-a-time and double-stats every file

persist_hashes (src/pipeline/pipeline_incremental.c) re-stat()s every discovered file (classify already statted them all) and calls cbm_store_upsert_file_hash per row → one implicit SQLite write transaction per file.

Correctness bonus: re-statting after the run records the post-run mtime, so a file edited mid-index is stamped "current" and the next run misses the edit. Using the classify-time stat is both faster and more correct.

P3 — Artifact re-export on every incremental run

dump_and_persist (src/pipeline/pipeline_incremental.c): if an artifact exists, every incremental run — even a 1-file edit — reads the whole DB, zstd-compresses it, and rewrites .codebase-memory/graph.db.zst (+ metadata), churning the user's git working tree and burning I/O/CPU proportional to the whole graph.

Proposed fix (3 focused changes)

I'd like to submit these as three small, stacked PRs referencing this issue, each under ~500 lines and independently green:

1. P1 — Git-based hash re-stamp at artifact import ("bootstrap reconciliation"). Right after a successful artifact import, re-stamp file_hashes rows for files git reports unchanged between the artifact's commit and the local working tree, using local stat() values. Zero changes to the incremental pipeline itself. Graceful degradation: any uncertainty (no git, unknown commit, dirty/stale export) → skip → today's slow-safe behavior. This is the headline win.
2. P2 — Batch + single-stat persist_hashes. classify_files outputs classify-time stamps; persist_hashes builds one cbm_file_hash_t array and calls the existing cbm_store_upsert_file_hash_batch (single transaction), with a row-at-a-time fallback on batch failure. Fixes the mid-run-edit blind spot too.
3. P3 — Debounce artifact auto re-export. Export only when changed_total >= 10 (default) OR the artifact's indexed_at is older than 24 h. CBM_ARTIFACT_EXPORT_MIN_CHANGES=0 restores per-run export. Skip is logged.

Two design points I'd like maintainer sign-off on before/as the PRs land

Per CONTRIBUTING these touch areas that need explicit approval:

• (a) Optional artifact.json trust marker (P1). Reconciliation requires a guarantee that the imported DB matches a clean checked-out tree at a known commit (otherwise a stale/corrupt artifact could mark a genuinely-changed file unchanged → graph corruption). I propose adding an optional "reconcile_basis":"git-clean-head" field to the existing artifact.json — written only when export can prove: commit is a valid hex OID, the working tree is clean outside .codebase-memory/, and every file_hashes row's on-disk mtime+size matches. No new metadata file, no schema_version bump (older binaries ignore the unknown field). Reconciliation skips unless the marker + commit + local checks all pass. I'd appreciate a thumbs-up on reusing artifact.json this way vs. any other convention you prefer.
• (b) Makefile test registration (all PRs). Each PR adds tests following the existing tests/test_*.c + TEST() + tmpdir fixtures conventions. P1/P2/P3 unit tests extend tests/test_artifact.c (no Makefile change). A new offline integration test (tests/test_incremental_bootstrap.c) needs a one-line addition to TEST_INTEGRATION_SRCS in Makefile.cbm — flagged since CONTRIBUTING calls out Makefile changes as approval-gated.

Before/after evidence (full matrix in the PRs)

Synthetic Rust-weighted repos (500 / 2000 / 8000 modules), 3 reps, median. K = 7 changed files (5 modified + 2 added). Full protocol + raw CSV are kept outside the repo.

Candidate S2 wall is 4.8 % of baseline at N=8000 (~21× faster). Graph equivalence: after S2, candidate node/edge counts exactly equal a from-scratch full index of the same tree at every size.

Explicitly deferred (follow-up issues, not in this changeset)

• In-place SQLite delta dump instead of load-all → purge → full dump → full FTS5 rebuild (biggest remaining win, biggest blast radius).
• Gating/incrementalizing pass_similarity + pass_semantic_edges for small change sets.
• Populating real content hashes in the sha256 column + a hash fallback in classify_files (covers non-git repos / mtime noise; P1's git reconciliation covers the bootstrap case).
• Cross-path artifact bootstrap (discovered while benchmarking): cbm_project_name_from_path derives the project name from the full normalized repo path, so two clones at different paths get different project names — the imported artifact DB is keyed by the exporter's project, so the importing teammate finds 0 hash rows and falls through to a full reindex (and reconcile restamps 0). This is pre-existing (independent of these three changes) but worth tracking since it limits P1's real-world team-sharing benefit until addressed. Happy to file it separately.

Happy to split, re-scope, or drop any of the three based on your feedback. Code is written and tested (ASan/UBSan suite green locally; full matrix above) — I'll open the PRs once there's rough agreement on (a)/(b).