Skip to content

fix(vector): clear every embedding model's vectors on workspace clear - #4263

Open
inamdarmihir wants to merge 1 commit into
HKUDS:mainfrom
inamdarmihir:fix/clear-all-model-vector-containers
Open

inamdarmihir wants to merge 1 commit into
HKUDS:mainfrom
inamdarmihir:fix/clear-all-model-vector-containers

Conversation

@inamdarmihir

Copy link
Copy Markdown

Summary

/documents/clear dropped only the active embedding model's vector container. Vectors stored under other models stayed in place, so switching back to one of them could return chunks the operator believed were deleted, while KV, graph and doc-status were already cleared.

Motivation

Fixes #3974. Per the recommended option in the issue, clearing a workspace now means every model's vectors, including the legacy container where applicable.

What's changed

  • BaseVectorStorage.drop_all_models() runs drop() and then the new drop_sibling_models(). A failed drop() stops there; a sibling failure is returned as an error.
  • /documents/clear calls drop_all_models() for the vector stores.
  • drop() is unchanged, so lightrag-rebuild-vdb and reversible model switching are unaffected.
  • Siblings are selected by evidence of ownership, not by name pattern alone:
    • Qdrant: lightrag_vdb_{ns}_*_{dim}d collections that index workspace_id; this workspace's points are deleted.
    • PostgreSQL: {base}_*_{dim}d tables with workspace and content_vector columns; this workspace's rows are deleted.
    • Milvus: collection names are a plain prefix and can collide across workspaces, so new collections carry a [lightrag:workspace=..;namespace=..] marker in the schema description. Only collections with this instance's marker are dropped.
  • Design and the accepted residue are documented in docs/design/VectorSpaceProvenance.md.

What's not covered

  • Milvus collections created before this change have no marker. They are left in place and named in the error returned by drop_all_models(), so the operator can drop them. Deleting on a name match could remove another workspace's data.
  • OpenSearch is not touched: main has no per-model indexes yet (feat(opensearch): isolate vector indices by embedding model #3966).

Testing

  • New offline tests for each backend and the base class (tests/kg/{qdrant,postgres,milvus}_impl/test_*_drop_sibling_models.py, tests/kg/test_drop_all_models.py), using mocked clients. Nothing was run against live Qdrant, PostgreSQL or Milvus.
  • Ran tests/kg/{qdrant,milvus,postgres}_impl, the new base test, tests/api and tests/test_docstring_budget.py. tests/api has 98 failures that also occur on an unmodified checkout; everything else passes. ruff check and ruff format --check are clean.

With per-model vector containers, /documents/clear dropped only the active
model's container, so switching back to another model could surface chunks
the operator had deleted.

Add BaseVectorStorage.drop_all_models() (drop() plus drop_sibling_models())
and call it from the clear route. drop() is unchanged so rebuild-vdb and
reversible model switching keep working.

Siblings are selected by evidence of ownership, not name alone:
- Qdrant: model-suffixed collections that index workspace_id; this
  workspace's points are deleted.
- PostgreSQL: model-suffixed tables with workspace and content_vector
  columns; this workspace's rows are deleted.
- Milvus: collections whose schema description carries a new
  workspace/namespace ownership marker are dropped. Collections created
  before the marker are left in place and reported.

Fixes HKUDS#3974

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

clear_documents leaves other embedding models' vector containers intact

1 participant