Repository navigation
fix(vector): clear every embedding model's vectors on workspace clear - #4263
Open
inamdarmihir wants to merge 1 commit into
Open
inamdarmihir wants to merge 1 commit into
inamdarmihir wants to merge 1 commit into
Conversation
With per-model vector containers, /documents/clear dropped only the active model's container, so switching back to another model could surface chunks the operator had deleted. Add BaseVectorStorage.drop_all_models() (drop() plus drop_sibling_models()) and call it from the clear route. drop() is unchanged so rebuild-vdb and reversible model switching keep working. Siblings are selected by evidence of ownership, not name alone: - Qdrant: model-suffixed collections that index workspace_id; this workspace's points are deleted. - PostgreSQL: model-suffixed tables with workspace and content_vector columns; this workspace's rows are deleted. - Milvus: collections whose schema description carries a new workspace/namespace ownership marker are dropped. Collections created before the marker are left in place and reported. Fixes HKUDS#3974
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/documents/cleardropped only the active embedding model's vector container. Vectors stored under other models stayed in place, so switching back to one of them could return chunks the operator believed were deleted, while KV, graph and doc-status were already cleared.Motivation
Fixes #3974. Per the recommended option in the issue, clearing a workspace now means every model's vectors, including the legacy container where applicable.
What's changed
BaseVectorStorage.drop_all_models()runsdrop()and then the newdrop_sibling_models(). A faileddrop()stops there; a sibling failure is returned as an error./documents/clearcallsdrop_all_models()for the vector stores.drop()is unchanged, solightrag-rebuild-vdband reversible model switching are unaffected.lightrag_vdb_{ns}_*_{dim}dcollections that indexworkspace_id; this workspace's points are deleted.{base}_*_{dim}dtables withworkspaceandcontent_vectorcolumns; this workspace's rows are deleted.[lightrag:workspace=..;namespace=..]marker in the schema description. Only collections with this instance's marker are dropped.docs/design/VectorSpaceProvenance.md.What's not covered
drop_all_models(), so the operator can drop them. Deleting on a name match could remove another workspace's data.Testing
tests/kg/{qdrant,postgres,milvus}_impl/test_*_drop_sibling_models.py,tests/kg/test_drop_all_models.py), using mocked clients. Nothing was run against live Qdrant, PostgreSQL or Milvus.tests/kg/{qdrant,milvus,postgres}_impl, the new base test,tests/apiandtests/test_docstring_budget.py.tests/apihas 98 failures that also occur on an unmodified checkout; everything else passes.ruff checkandruff format --checkare clean.