Summary

Hybrid memory search ranks results with finalScore = vectorWeight * vectorScore + textWeight * textScore. mergeHybridResults (extensions/memory-core/src/memory/hybrid.ts) applied that formula uniformly to every candidate. Image and audio chunks only carry a synthetic label as text, e.g. Image file: media/IMG_0007.png (buildMemoryMultimodalLabel, packages/memory-host-sdk/src/host/multimodal.ts). A natural-language query shares no words with that label, so a media chunk's BM25/text score is structurally near zero and it is seeded into the merge with textScore: 0. Under the default 0.7 vector / 0.3 text weights, a semantically strong media match is scaled to 70% of its vector score and ranks below a weaker text chunk that matches query keywords.

The fix makes fusion modality-aware. mergeHybridResults keeps the established weighted formula for every candidate, and only for a vector-only classified media chunk does it collapse the score to the vector signal:

-    const score = params.vectorWeight * entry.vectorScore + params.textWeight * entry.textScore;
+    const dropMediaTextSignal =
+      entry.hasVector &&
+      !entry.hasKeyword &&
+      params.vectorWeight > 0 &&
+      params.isNonTextMediaPath?.(entry.path) === true;
+    const contentScore = dropMediaTextSignal
+      ? entry.vectorScore
+      : params.vectorWeight * entry.vectorScore + params.textWeight * keywordScore;

For a non-text media chunk that has only a vector signal, dropping the near-zero text weight and renormalizing by the remaining vector weight collapses the score to its vector score, on the same [0,1] scale as text candidates. Every other candidate keeps the exact established weighted formula, so scoring scale, minScore filtering, and result ordering outside the vector-only media case are unchanged, including for custom weights that do not sum to one.

The manager supplies the predicate from the already-known file path, reusing the classifier the sync path uses (extensions/memory-core/src/memory/manager.ts):

isNonTextMediaPath: (path) =>
  classifyMemoryMultimodalPath(path, this.settings.multimodal) !== null,

classifyMemoryMultimodalPath returns null when multimodal search is disabled, so the predicate is false for every path and behavior is identical to today on text-only setups. Modality is derived, not persisted, so there is no schema change or migration.

Keyword-only media is preserved (review follow-up)

The first revision dropped the text weight for any media path. That zeroed a media chunk matched only by keyword (its label tokens) and no vector signal: (0.7*0 + 0*text)/0.7 = 0, dropping it from results. The merge now tracks an explicit hasVector flag per candidate and only drops the synthetic-label text signal when the candidate also has a vector signal. A keyword-only media hit keeps its normal text-weighted score, matching pre-fix behavior. Covered by a new regression test.

Zero vector-weight configuration preserved (re-review follow-up)

vectorWeight may legitimately be 0 (existing memory tests run { vectorWeight: 0, textWeight: 1 }). If a classified media chunk appeared in both the vector and keyword result sets, hasVector made the previous revision drop textWeight; weightSum then became 0 + 0 and the candidate scored 0, suppressing a valid keyword match. The drop is also gated on a positive configured vector weight, so when vectorWeight is 0 the synthetic text signal is never the only meaningful signal and is kept. Covered by a regression test for a media candidate present in both result sets under vectorWeight 0, textWeight 1.

Both-signal media keeps its real keyword score (re-review follow-up)

Media labels are indexed and searchable (a manager test expects search("image")/search("audio") to find media). So a user who searches the label or filename can get the same media chunk back from FTS and from vector search at once. Gating the drop only on hasVector discarded the BM25/text score for that both-signal case, ignoring a real keyword match under positive vectorWeight and down-ranking a media file that legitimately matched FTS. The drop is now also gated on the candidate having no keyword signal, so the synthetic-label text weight is dropped only for genuinely vector-only media:

     const dropMediaTextSignal =
       entry.hasVector &&
+      !entry.hasKeyword &&
       params.vectorWeight > 0 &&
       params.isNonTextMediaPath?.(entry.path) === true;

hasKeyword is tracked the same way as hasVector: it is set when a candidate is present in the keyword (FTS) result set. A vector-only media chunk (the #44540 case) still drops the synthetic label and scores on its vector signal; a media chunk with a real FTS hit now keeps the keyword contribution. Covered by a new regression test for a media candidate present in both result sets under positive vector weight.

Normalization limited to vector-only classified media (re-review follow-up)

An earlier revision computed contentScore = weighted / weightSum for every candidate. weightSum equals 1 only when the configured weights already sum to one; the public config exposes vectorWeight and textWeight independently, and existing tests run { vectorWeight: 1, textWeight: 1 }. Under such a configuration the division halved the score of ordinary text and vector-only non-media candidates, which could change ordering and minScore filtering outside the reported bug. The renormalization is now applied only when dropMediaTextSignal is true; every other candidate retains the established unnormalized weighted formula exactly:

-    const effectiveTextWeight = dropMediaTextSignal ? 0 : params.textWeight;
-    const weightSum = params.vectorWeight + effectiveTextWeight;
-    const weighted =
-      params.vectorWeight * entry.vectorScore + effectiveTextWeight * keywordScore;
-    const contentScore = weightSum > 0 ? weighted / weightSum : 0;
+    const contentScore = dropMediaTextSignal
+      ? entry.vectorScore
+      : params.vectorWeight * entry.vectorScore + params.textWeight * keywordScore;

Because dropMediaTextSignal is gated on params.vectorWeight > 0, collapsing the media score to entry.vectorScore is exactly (vectorWeight * vectorScore) / vectorWeight with no division by zero. Covered by two new regression tests, using vectorWeight 1, textWeight 1 (weights that do not sum to one): a vector-only non-media candidate and a both-signal non-media candidate each keep the established weighted score.

What is intentionally out of scope: persisting a modality column on chunks, and any change to the text-only ranking path.

Rebase onto current main and the TypeScript LOC ratchet

This branch was rebased onto current main. The changed-file TypeScript LOC ratchet on main forbids an oversized legacy file from growing, and extensions/memory-core/src/memory/manager.ts is already 1671 lines, so plumbing the media predicate into the merge call tripped the gate. The merge wrapper's inline parameter type moved to hybrid.ts, which owns the merge contract, and is referenced as ManagerHybridMergeParams. manager.ts now shrinks to 1660 lines. The modality-aware fusion fix is unchanged.

Real behavior proof

Behavior addressed: hybrid fusion discounted a vector-only image match by the configured text weight, so a semantically strong media result ranked below a weaker keyword-matching text note.
Real environment tested: the real production memory retrieval path driven on pristine main (1708faf85b) and on PR head c02d11dc7f with identical inputs: a real node:sqlite memory index built with the production ensureMemoryIndexSchema, a real FTS5 index scored by the production searchKeyword with real BM25 ranking, the real classifyMemoryMultimodalPath modality classifier wired exactly as MemoryIndexManager wires it, and the real mergeHybridResults fusion under the default vectorWeight 0.7, textWeight 0.3. The corpus is one image file, one keyword-matching note, and 18 filler documents so BM25 has real IDF.
Exact steps or command run after this patch: indexed the corpus, ran the query underwater coral reef survey through the production keyword search and hybrid fusion, then reverted only hybrid.ts to its pristine main contents and re-ran the identical scenario.
Evidence after fix:

# BEFORE (pristine main 1708faf85b)
query: underwater coral reef survey
fusion weights: vectorWeight=0.7 textWeight=0.3
classifyMemoryMultimodalPath("memory/media/IMG_0007.png") -> image
classifyMemoryMultimodalPath("memory/field-notes.md") -> null
ranked memory search results:
  #1  score=0.9376  vector=0.9487  bm25=0.9116  memory/field-notes.md
  #2  score=0.7000  vector=1.0000  bm25=0.0000  memory/media/IMG_0007.png
top result: memory/field-notes.md

# AFTER (this patch, identical inputs, head c02d11dc7f)
query: underwater coral reef survey
fusion weights: vectorWeight=0.7 textWeight=0.3
classifyMemoryMultimodalPath("memory/media/IMG_0007.png") -> image
classifyMemoryMultimodalPath("memory/field-notes.md") -> null
ranked memory search results:
  #1  score=1.0000  vector=1.0000  bm25=0.0000  memory/media/IMG_0007.png
  #2  score=0.9376  vector=0.9487  bm25=0.9116  memory/field-notes.md
top result: memory/media/IMG_0007.png

Observed result after fix: the image chunk carries a strong vector signal (1.0000) but a structurally zero BM25 signal, because its only indexed text is the synthetic Image file: label. Before the patch the 0.3 text weight scaled it down to 0.7000 and it lost to the weaker note at 0.9376. After the patch it scores on its vector signal (1.0000) and surfaces first, while the note keeps its identical 0.9376 score, so the ranking flips to put the strongest semantic match on top.
What was not tested: no live embedding provider request was made, so the query and chunk vectors are deterministic local values rather than Gemini-served embeddings; the approximate-nearest-neighbour vector index lookup was not exercised, and the vector scores were supplied directly to fusion. Everything else on the retrieval path (schema, FTS5, BM25 ranking, modality classification, fusion) is the real production code. A full production build was not run.

Verification

• Real before/after through the production memory retrieval path above (real node:sqlite schema, FTS5/BM25 keyword search, real modality classifier, real fusion), pristine main vs this patch on identical inputs.
• Regression coverage in extensions/memory-core/src/memory/hybrid.test.ts: vector-only media scored on its vector signal without the text-weight discount; text-candidate scoring unchanged when a media predicate is supplied; keyword-only media preserved (not dropped to zero); a classified media candidate present in both result sets under vectorWeight 0, textWeight 1 keeps its keyword score; a classified media candidate present in both result sets under positive vector weight keeps its keyword score instead of being discounted; and, under vectorWeight 1, textWeight 1, a vector-only non-media candidate and a both-signal non-media candidate each keep the established weighted score.
• node scripts/test-projects.mjs extensions/memory-core/src/memory/hybrid.test.ts (22 passed).
• node scripts/test-projects.mjs extensions/memory-core/src/memory/manager-search.test.ts (34 passed).

Closes #44540