Repository navigation
Python: key Chroma records by storage name when serializing and deserializing - #14763
Open
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 1 commit into
Open
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 1 commit into
MOHAMMED WASIM KHAN (wasim-builds) wants to merge 1 commit into
Conversation
…ializing (microsoft#14755) Dict and dataclass records reach the Chroma serializer keyed by storage name and pydantic models keyed by field name. The serializer read the key by declared name and the vector by storage name, so a dict or dataclass record with a storage_name on the key field failed to upsert and a pydantic model with a storage_name on the vector field failed too. The deserializer returned the key and the vector under the declared names, which the data model step does not read for dict and dataclass records, so a vector with a storage_name came back as None. Pydantic metadata was stored under the declared names, so filters on storage names did not match it. Closes microsoft#14755 Co-authored-by: VANDRANKI <[email protected]>
MOHAMMED WASIM KHAN (wasim-builds)
requested a review
from a team
as a code owner
November 9, 2026 16:50
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
There was a problem hiding this comment.
Looks correct. Here's what I checked.
- Which keys reach the serializer.
_serialize_data_model_to_dict(data/vector.py) returnsrecord.model_dump()for pydantic models, which is keyed by declared name. For dicts and dataclasses it buildsstore_model[field.storage_name or field.name]. So the description is right that the two arrive keyed differently, and_storage_keyed(chroma.py:212) moves the pydantic case onto storage names without touching the other. - Which keys the deserializer must produce.
_deserialize_dict_to_data_modelpopsstorage_nametonamefor pydantic records and readsrecord.get(field.storage_name or field.name)for the others. Returning the key and vector under their storage names (chroma.py:249-250) is what both paths expect. Onmain, the vector came back undervector_field.nameand was read asNonefor dict records with a vectorstorage_name. - Collisions.
_storage_keyedonly renames when the storage name isn't already a key, so a field whose storage name equals another field's declared name won't be overwritten. git merge-treeagainstmainis clean.
I didn't run the tests.
One thing worth a line in the release notes: for pydantic models with a storage_name on a data field, metadata used to be written under the declared name and is now written under the storage name. A collection with older records will then have both spellings, and a filter on the storage name won't match the older ones. Reads still work for both, because the pydantic path accepts either key.
Reviewed with help from Claude Code.
4 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Closes #14755.
Dict and dataclass records reach the Chroma serializer keyed by storage name and pydantic models keyed by field name. The serializer read the key by declared name and the vector by storage name, so a dict or dataclass record with a
storage_nameon the key field failed to upsert and a pydantic model with astorage_nameon the vector field failed too. The deserializer returned the key and the vector under the declared names, which the data model step does not read for dict and dataclass records, so a vector with astorage_namecame back as None. Pydantic metadata was stored under the declared names, so filters on storage names did not match it.Description
_serialize_dicts_to_store_modelsrenames declared names to storage names in a copy of the record (only when the storage name is not already present), and then reads the key and vector by storage name._deserialize_store_models_to_dictsreturns the key and vector under their storage names. The data model step maps them back for pydantic records.tests/unit/connectors/memory/test_chroma.py(dict, dataclass, and pydantic, each with and without keystorage_name). All 26 tests intest_chroma.pypass cleanly.Contribution Checklist