semantic_query returns identical (alphabetical) results for any query — vector-less nodes win the ranking

Summary

search_graph with semantic_query returns the same result set for any query whatsoever — different phrases, single terms, different languages all return identical hits, in what appears to be alphabetical order. The vector data in the index is fine (verified by computing cosine similarity manually against the SQLite DB); the bug is in the query executor: it returns exactly the nodes that have NO row in node_vectors, which suggests a LEFT JOIN + ORDER BY similarity DESC where the NULL/NaN similarity of vector-less nodes sorts above every real cosine score.

Environment

v0.8.1, darwin-arm64 (release binary, checksum verified)
Large TypeScript + Java monorepo (~10k files, 154k nodes after mode: "moderate" indexing)
Reproduced identically via cli one-shot mode AND via the stdio MCP server (tools/call)

Reproduction

codebase-memory-mcp cli index_repository '{"repo_path":"<repo>","mode":"moderate"}'

# Any of these return the SAME hits (JSON keys from a registry.json, alphabetical):
codebase-memory-mcp cli search_graph '{"project":"<slug>","semantic_query":["where is client debt calculated"],"limit":6}'
codebase-memory-mcp cli search_graph '{"project":"<slug>","semantic_query":["debt"],"limit":6}'
codebase-memory-mcp cli search_graph '{"project":"<slug>","semantic_query":["cost basis FIFO when selling stocks"],"limit":6}'

Every call returns (excerpt):

$behaviors_note        | apps/.../ui/registry.json
$bulkZipDownload_note  | apps/.../ui/registry.json
$candidateExemplar_note| apps/.../ui/registry.json
...

Note the results are query-independent and alphabetically ordered — the classic signature of sorting on NULL/NaN similarity.

Root-cause evidence (from the index DB)

The data side is healthy:

node_vectors: 46,606 rows × 768d int8 — populated.
token_vectors: 18,049 tokens with vector + idf; all my query terms (debt, overdue, balance, fifo, …) are present.

Manual cosine check (Python over the SQLite DB, query vector = mean of its token vectors):

query "client debt overdue balance" vs CcmaSaldoCalculator  → cos = 0.9492   (the CORRECT hit)
query "client debt overdue balance" vs $behaviors_note      → node has NO row in node_vectors

So the correct ranking exists in the data — CcmaSaldoCalculator (a debt-calculation class) scores 0.949 and should be the top hit. Instead, the executor returns precisely the nodes that are absent from node_vectors (JSON-key nodes from a registry.json, which apparently never get vectors).

Suggested fix

Exclude vector-less nodes from the semantic ranking — INNER JOIN node_vectors (or WHERE nv.vector IS NOT NULL), and/or treat missing similarity as -inf rather than letting NULL/NaN win the sort.

Side note

BM25 (query) and search_code work great on the same index — this report is only about the semantic_query path.

Thanks for the semantic_query diagnosis and the manual vector check. Triage: high-priority query correctness bug under #592/#594.

Vector-less nodes winning semantic-only ranking is exactly the sort of silent false-confidence result we need to prevent. The fix should prove semantic-only queries return vector-ranked primary results and that total/has_more still describe the semantic candidate set.

Verification against current main (post-v0.9.0 semantic rework), with a small mixed fixture indexed in full mode:

semantic_query:["debt","client"] → calculate_client_debt first at score 0.9586
semantic_query:["email","welcome"] → send_welcome_email first at 0.9274

Distinct queries, distinct correct rankings, real scores — the identical-alphabetical symptom does not reproduce. On the diagnosis: both v0.8.1 and current main use an INNER JOIN onto node_vectors (checked the v0.8.1 tag directly), so vector-less nodes can't enter the result set; and the cosine paths are zero-norm-guarded at both the SQL function and the re-rank layer. What best fits your observation (ties resolving to insertion order — alphabetical registry.json keys) is DEGENERATE node vectors in the 0.8.1 moderate-mode index (e.g. body-less JSON-key nodes collapsing to identical vectors → all scores tie). The v0.9.0 semantic pass was substantially reworked (per-node token vectors, storage changes), which most plausibly fixed that generation bug.

Could you re-index your monorepo on v0.9.0 (or 0.9.1, imminent) with the same mode:"moderate" and rerun your three probes? If rankings still collapse, two artifacts would pin it immediately: (1) SELECT count(*), count(DISTINCT vector) FROM node_vectors on the new cache (identical-vector collapse shows up as a tiny distinct count), and (2) the top-6 with scores from any one probe. Thanks for the unusually thorough diagnosis — the manual-cosine verification is what let us rule layers out quickly.

Thanks for the report! To move this forward we need a bit more so we can reproduce it ourselves:

the codebase-memory-mcp --version you're on
the exact steps or command you ran
a public repo (or a small dummy snippet) that shows the problem — please don't paste proprietary code

Once that's here we'll pick it straight back up. Heads-up: issues left awaiting-reporter are automatically closed after a few weeks of silence, but a comment reopens the door anytime.

⚠️ Retracted — see the correction below. The examples in this comment were produced with cli search_graph --semantic-query, which does not parse its array argument, so they measured a broken path. The builtins claim is withdrawn. The underlying vs_fill_sparse_random finding survives and now has a validated fix.

Reproduced on v0.9.0 (this issue was filed against 0.8.1), with a minimal public repro and what I believe is the source-level cause:

https://github.com/Icosa2050/cbm-semantic-query-repro — 84 nodes, synthetic Python across three unrelated domains, ./verify.sh asserts every claim below. Verified identical on macOS 15 arm64 and Linux 7.0 x86_64 from clean clones with a virgin CBM_CACHE_DIR.

Query-time vectors never consult the pretrained table

At index time, cbm_sem_random_index() (src/semantic/semantic.c:437) looks each token up in the vendored nomic-embed-code table and only falls back to sparse random indexing on a miss, so node vectors carry real pretrained semantics.

At query time, vs_build_keyword_vectors() (src/store/store.c:6245) does not use that function. It tries vs_load_enriched_vector() (src/store/store.c:6176), which reads only the per-project token_vectors table:

const char *tv_sql = "SELECT vector, idf FROM token_vectors"
                     " WHERE project = ?1 AND token = ?2 LIMIT 1";

and on a miss falls through to vs_fill_sparse_random() (src/store/store.c:6202), which is a plain XXH3 hash with no nomic lookup.

So any keyword outside the project's harvested vocabulary is turned into a content-free hash and compared against semantically-built node vectors. It is returned silently, with no error and no signal to the caller.

token_vectors is populated only from indexed identifiers (phase3c_export_token_vectors, src/pipeline/pass_semantic_edges.c:1101) — 108 tokens in this 84-node repo.

Consequences

Multi-word keywords can never match, since the lookup is on the whole string and the table holds single tokens:

--semantic-query '["discount tariff currency"]'      -> print, pop, append, upper, lower   (all <python-builtins>)
--semantic-query '["discount","tariff","currency"]'  -> haversine_distance_km, summarize_sensor_alarms, invoice_total

The docs do prescribe single keywords, so the first form is misuse on my part — but it is accepted and answered with noise rather than rejected.

Ordinary words miss too when absent from the identifiers. temperature is not in this project's vocabulary (the field is cabin_temperature_c), so it degrades to a hash and returns negatives, with the actual temperature function ranked third:

--semantic-query '["temperature"]'
  -0.00297  tyre_pressure_low
  -0.01184  split_into_legs
  -0.01191  refrigeration_breached

No score floor. vs_min_cosine_score() (src/store/store.c:6267) takes the min across keywords and nothing rejects a negative. Taking the min also penalises specialists: battery_is_critical is rank 3 on ["battery"] (+0.033) but rank 16 on ["battery","temperature","sensor"], because its temperature component is negative while a generalist scores mediocre-positive on all three.

Builtins are 39% of the semantic corpus. cbm_store_vector_search filters to n.label IN ('Function','Method','Class') (src/store/store.c:6345), which here is 18 rows — 7 of them len, print, upper, lower, append, pop, get.

The project name affects scores. It is tokenised into the vocabulary like any other token (repro appears as a row in token_vectors), so it shifts IDF corpus-wide. Same commit, byte-identical source, only --name differs: rank-1 scores 0.05703263 vs 0.05976143.

On this issue's diagnosis

The LEFT JOIN node_vectors theory does not seem to apply. The vector path already uses an inner join (src/store/store.c:6341):

FROM node_vectors v INNER JOIN nodes n ON n.id = v.node_id

The identical alphabetical results array does reproduce here — three unrelated queries hash the same, total equals the full node count. But per cli search_graph --help, semantic output goes to semantic_results, and results is the structural-filter field; with no query / name_pattern / label supplied there is nothing to filter it by. That may be intended. It is still easy to misread, since total: 84 beside a populated results array looks like 84 matches — which I think is how the LEFT JOIN reading arose.

Possible fixes

Give vs_fill_sparse_random the same nomic fallback cbm_sem_random_index already has, or share one code path. This looks like the root cause; the rest are amplifiers.
Tokenise multi-word keywords, or reject them explicitly.
Exclude <python-builtins> and equivalent stub nodes from node_vectors.
Add a score floor in vs_min_cosine_score, and reconsider min-across-keywords.
Exclude the project name from the tokenised corpus.
Signal out-of-vocabulary keywords in the response instead of returning noise.

BM25 and name_pattern are unaffected throughout — the same phrase via --query returns exactly the three pricing symbols.

Separately, the repro README documents an unrelated bug: .cbmignore exclusions are not applied when re-indexing an existing project (inclusions are; delete_project is currently required). Happy to file that on its own if useful.

Correction to my comment above — please read this one instead.

I traced my own evidence and found it was produced through a broken path. The repro is rewritten; the retraction and the parts that survive are below, plus a validated fix.

What I got wrong

Every example in my previous comment used cli search_graph --semantic-query '["..."]'. That flag does not parse its array. It passes the literal text ["temperature"] as a single keyword, which can never be in any vocabulary, so it always took the fallback path and returned noise. The JSON argument form, piped stdin, and the MCP tool all parse correctly.

So my claim that language builtins outrank project code is withdrawn. With correct parsing:

{"semantic_query":["discount","tariff","currency"]}
  compute_parcel_price     +0.781   src/pricing.py
  compute_bulk_discount    +0.778   src/pricing.py
  apply_currency_surcharge +0.732   src/pricing.py
  invoice_total            +0.705   src/pricing.py

All four pricing functions, no builtins. Semantic search works well when the keyword resolves. Apologies for the noise.

The flag not parsing its array is itself a bug — it fails silently rather than erroring.

What survives, and the actual defect

Index time uses cbm_sem_random_index() (src/semantic/semantic.c:437), which looks the token up in the vendored nomic table and only falls back to sparse random indexing on a miss.

Query time does not. vs_build_keyword_vectors() (src/store/store.c:6245) tries vs_load_enriched_vector() (src/store/store.c:6176), which reads only the per-project token_vectors table, then falls through to vs_fill_sparse_random() (src/store/store.c:6202) — a plain XXH3 hash, no nomic lookup.

token_vectors holds only tokens harvested from indexed identifiers. In this 84-node repo temperature is absent (the field is cabin_temperature_c) even though it is in the nomic vocabulary at index 35753. Because vs_min_cosine_score() (src/store/store.c:6267) takes the minimum across keywords, that one hash keyword collapses the whole query:

query	top 1
`["battery"]`	`battery_is_critical` +0.946 (telemetry)
`["battery","temperature","sensor"]`	`haversine_distance_km` +0.069 (routing)

An order of magnitude lower, and the wrong domain.

Fix

static void vs_fill_sparse_random(const char *token, float *out) {
    cbm_sem_vec_t sv;
    cbm_sem_random_index(token, &sv);   /* nomic first, sparse RI on miss */
    for (int d = 0; d < VS_VEC_DIM && d < CBM_SEM_DIM; d++) {
        out[d] = sv.v[d];
    }
}

plus #include "semantic/semantic.h". VS_VEC_DIM == CBM_SEM_DIM == 768.

query	stock	patched
`["temperature"]`	`haversine_distance_km` +0.069 (routing)	`tyre_pressure_low` +0.396, `refrigeration_breached` +0.337 (telemetry)
`["battery","temperature","sensor"]`	`haversine_distance_km` +0.069	`tyre_pressure_low` +0.396
`["battery"]`	`battery_is_critical` +0.946	unchanged — no fallback taken

make -f Makefile.cbm test → 5934 passed, 1 skipped, byte-identical to the unpatched baseline I ran on the same machine. The single skip is the network-gated incremental test.

Also found

limit changes the winner, not just the row count. fetch_limit = limit * 5 (src/store/store.c:6357) and the SQL pre-filter orders by the first keyword only, then re-ranks by min, so a small pool never sees the best-by-min candidate:

limit	top-1 for `["battery","temperature","sensor"]`
1	`summarize_sensor_alarms` +0.0267
2	`compute_parcel_price` +0.0506
3+	`haversine_distance_km` +0.0690

Scores depend on the project name. It is tokenised into the vocabulary like anything else, shifting IDF corpus-wide. Byte-identical source, only --name differs: battery_is_critical scores 0.94567562 under repro and 0.94138202 under othername.

.cbmignore exclusions are not applied on re-index (inclusions are; delete_project is required first). Unrelated to this issue — happy to file separately.

On this issue's diagnosis

The identical alphabetical results array does reproduce. But the LEFT JOIN node_vectors theory does not fit: the vector path already uses an inner join (src/store/store.c:6341). Per cli search_graph --help, semantic output goes to semantic_results, and results is the structural-filter field — with no query/name_pattern/label supplied there is nothing to filter it by, so this may be intended. total: 84 beside a populated results array is easy to read as "84 matches", which is likely how the LEFT JOIN reading arose.

Repro, rewritten and with verify.sh covering all of the above: https://github.com/Icosa2050/cbm-semantic-query-repro — 11/11 on macOS 15 arm64, 10/11 on Linux 7.0 x86_64 (one check skips without sqlite3), from clean clones.

@Icosa2050

Thanks @Icosa2050 for retracting the invalid evidence and replacing it with a corrected, public, cross-platform reproducer. The awaiting-reporter label has been removed.

The surviving findings are present on current main: vs_load_enriched_vector reads only the per-project token_vectors row, then vs_fill_sparse_random uses XXH3 when the token is absent (src/store/store.c:7570-7607); multi-keyword ranking prefilters limit*5 candidates by the first keyword before min-score reranking (store.c:7734-7757). Those are distinct from the original LEFT JOIN theory, which the INNER JOIN at 7737-7742 correctly disproves.

Keeping this priority/high in 0.9.1-rc with the corrected scope: query-vector fallback parity, typed CLI array parsing, and limit-independent ranking correctness. Please file the .cbmignore reindex behavior separately as offered so it does not get lost here. Thank you for the unusually responsible correction.

Closing as fixed on main: the semantic-only path was isolated from the BM25/alphabetical fallback in de4ec9b (#1319), on top of the compact search_graph output in 4843a34; vector-less nodes no longer win the ranking. The pagination follow-up that grew out of this report (#938) is being resolved separately against main's lossless semantic pagination. Thank you for the clear repro.

fix(mcp): harden semantic search inputs and store-error handling

Navigation Menu

Repository navigation

semantic_query returns identical (alphabetical) results for any query — vector-less nodes win the ranking #915

Summary

Environment

Reproduction

Root-cause evidence (from the index DB)

Suggested fix

Side note

Query-time vectors never consult the pretrained table

Consequences

On this issue's diagnosis

Possible fixes

What I got wrong

What survives, and the actual defect

Fix

Also found

On this issue's diagnosis

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

semantic_query returns identical (alphabetical) results for any query — vector-less nodes win the ranking #915

Description

Summary

Environment

Reproduction

Root-cause evidence (from the index DB)

Suggested fix

Side note

Activity

DeusData commented on Jul 7, 2026

DeusData commented on Jul 11, 2026

github-actions commented on Jul 11, 2026

Icosa2050 commented on Jul 26, 2026

Query-time vectors never consult the pretrained table

Consequences

On this issue's diagnosis

Possible fixes

Icosa2050 commented on Jul 26, 2026

What I got wrong

What survives, and the actual defect

Fix

Also found

On this issue's diagnosis

DeusData commented on Jul 28, 2026

DeusData commented on Sep 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions