Repository navigation
Python: key Qdrant records by storage name when serializing and deserializing - #14762
Open
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 1 commit into
Open
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 1 commit into
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 1 commit into
Conversation
…ializing (microsoft#14754) Dict and dataclass records reach the Qdrant serializer keyed by storage name and pydantic models keyed by field name. The serializer popped the key and vector fields by declared name, so dict and dataclass records with a storage_name on the key or vector field failed to upsert with a KeyError. Pydantic payloads were stored under the declared names, so filters on storage names did not match them. The deserializer returned the key and an unnamed vector under the declared names, which the data model step does not read for dict and dataclass records. Closes microsoft#14754 Co-authored-by: VANDRANKI <[email protected]>
MOHAMMED WASIM KHAN (wasim-builds)
requested a review
from a team
as a code owner
November 9, 2026 16:41
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
There was a problem hiding this comment.
Looks correct. Here's what I checked.
- Serialize.
_storage_keyed(qdrant.py:443) copies the record, then renames declared names to storage names only when the storage name isn't already a key. After that the key is popped bykey_field_storage_nameand the vectors by storage name (qdrant.py:461-472). Dict and dataclass records already arrive keyed by storage name, so theKeyErrorfrom #14754 goes away. Pydantic payloads now land under storage names, which is what the lambda filter parser checks against (node.attr not in self.definition.storage_names,qdrant.py:425). The copy also means the caller's dicts are no longer mutated bypop, whichmaindid. - Named vectors.
vectors_config(qdrant.py:519) and the searchusing=(qdrant.py:298,:342) already usestorage_name or name, so the dict built at:470matches the collection's vector names. - Deserialize. The key and an unnamed vector come back under storage names (
qdrant.py:482-493). A named-vector response is already a dict keyed by storage name. In_deserialize_dict_to_data_model(data/vector.py), dict and dataclass records readrecord.get(field.storage_name or field.name), and pydantic records renamestorage_nametonamewhen it's present. So all three record types read their key and vector back. git merge-treeagainstmainis clean.
I didn't run the tests.
Two small notes:
- For pydantic models with a
storage_nameon a data field, the payload key changes from the declared name to the storage name. A collection with older points will then have both spellings, and a filter on the storage name won't match the older points. Reads still work for both. That may be worth a line in the release notes. The same applies to the Chroma change in #14763. _storage_keyedhere is the same function as the one in #14763'schroma.py. If both land, it could move to the base collection class so other connectors can use it.
Reviewed with help from Claude Code.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Closes #14754.
Dict and dataclass records reach the Qdrant serializer keyed by storage name and pydantic models keyed by field name. The serializer popped the key and vector fields by declared name, so dict and dataclass records with a
storage_nameon the key or vector field failed to upsert with aKeyError. Pydantic payloads were stored under the declared names, so filters on storage names did not match them. The deserializer returned the key and an unnamed vector under the declared names, which the data model step does not read for dict and dataclass records.Description
PointStruct,_serialize_dicts_to_store_modelsrenames declared names to storage names in the record when the storage name is not already present. Then it pops the key and vectors by storage name._deserialize_store_models_to_dictsreturns the key and an unnamed vector under their storage names so the data model step can read them for dict, dataclass, and pydantic models.tests/unit/connectors/memory/test_qdrant_local.pycovering dict, dataclass, and pydantic models across named and unnamed vector configurations.Contribution Checklist