fix(memory): detect and recover from Qdrant collection dimension mismatch (#1951)#1954
Merged
fix(memory): detect and recover from Qdrant collection dimension mismatch (#1951)#1954
Conversation
…atch QdrantOps::ensure_collection previously returned Ok(()) if a collection already existed, without verifying that its vector dimensions matched the required size. When the embedding model changed (e.g. 4096-dim → 1536-dim), all collections backed by QdrantOps continued silently until the first insert or search produced a dimension-mismatch error. This was a latent bug for every collection that goes through QdrantOps directly: - zeph_conversations - zeph_session_summaries - zeph_key_facts - zeph_corrections - zeph_graph_entities - zeph-index code collections EmbeddingRegistry (used by skills/MCP) already had its own inline detection, but all other collections bypassed it. Fix: add dimension verification in QdrantOps::ensure_collection and ensure_collection_with_quantization via a shared private helper get_collection_vector_size. On mismatch, the collection is deleted and recreated, with a WARN log listing the old and new dimension. Data loss is expected and documented. Also add four #[ignore] integration tests (require live Qdrant at :6334) that cover idempotency for same-size and recreation on mismatch for both ensure_collection variants. Closes #1951
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QdrantOps::ensure_collectionandensure_collection_with_quantizationpreviously returnedOk(())when a collection existed without checking its vector dimensionszeph_conversations,zeph_session_summaries,zeph_key_facts,zeph_corrections,zeph_graph_entities, and code-index collections in zeph-indexget_collection_vector_sizehelper usingcollection_info; both ensure methods now verify dimensions and recreate the collection (with aWARNlog) when there is a mismatchChanges
crates/zeph-memory/src/qdrant_ops.rs: dimension check + auto-recreation inensure_collectionandensure_collection_with_quantization; newget_collection_vector_sizehelper; four#[ignore]integration testsCHANGELOG.md: entry in[Unreleased]sectionNotes
EmbeddingRegistry(skills, MCP) already had its own inline dimension detection. That logic remains and still works correctly; it now also benefits from the fix in the underlyingQdrantOpscall, but the redundancy is harmless.Test plan
cargo +nightly fmt --checkpassescargo clippy --workspace --features full -- -D warningspassescargo nextest run --config-file .github/nextest.toml --workspace --features full --lib --bins— 6127 tests pass#[ignore]) can be verified manually with a live Qdrant instance at:6334by runningcargo nextest run -p zeph-memory -- --ignoredCloses #1951