Repository navigation
Python: Support list literals in Chroma, MongoDB, Pinecone and Weaviate filters - #14558
PRABHU KIRAN VANDRANKI (VANDRANKI) wants to merge 3 commits into
Conversation
GitHub Actions
Inactive
GitHub Actions
Inactive
semantic-kernel-automation
Bot
added
the
python
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Chroma and Weaviate currently alter apostrophes in list values, causing valid membership filters to miss matching records.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds list-literal support to vector-store filter parsers and fixes Pinecone negation handling.
Changes:
- Parses list literals for Chroma, MongoDB Atlas, Pinecone, and Weaviate.
- Correctly converts Pinecone
not (... in [...])to$nin. - Adds connector-specific unit coverage.
| File | Description |
|---|---|
python/semantic_kernel/connectors/chroma.py |
Adds list parsing and updates return typing. |
python/semantic_kernel/connectors/mongodb.py |
Adds list parsing. |
python/semantic_kernel/connectors/pinecone.py |
Adds list parsing and repairs negation handling. |
python/semantic_kernel/connectors/weaviate.py |
Adds list parsing for contains_any. |
python/tests/unit/connectors/memory/test_chroma.py |
Tests $in and $nin lists. |
python/tests/unit/connectors/memory/mongodb_atlas/test_mongodb_atlas_collection.py |
Tests MongoDB list filters. |
python/tests/unit/connectors/memory/test_pinecone.py |
Tests list and negated filters. |
python/tests/unit/connectors/memory/weaviate/test_weaviate_collection.py |
Tests contains_any translation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…tten List elements were parsed through the string constant branch, which doubles apostrophes, so x.content in ["O'Reilly"] filtered on O''Reilly. List elements now pass string and bytes constants through unchanged.
GitHub Actions
Inactive
|
Approved the reviewed head, but GitHub rejected merge-queue entry because this PR now has merge conflicts. Please rebase onto the current main branch, preserving the filter and apostrophe regressions, and rerun CI. It has not been added to the queue. |
GitHub Actions
Inactive
|
I updated this branch with current main (df1cb33). I merged main in instead of force pushing, so the new head 146d59c is a normal merge commit. Main changed the same spot in two test files. #14560 added the apostrophe tests in test_chroma.py and test_weaviate_collection.py, and that conflicted with my list tests. I kept both sets of tests. The source changes in chroma.py, mongodb.py, pinecone.py and weaviate.py merged with no conflicts and are identical to the head you approved. On the pushed tree I ran test_chroma.py, test_pinecone.py, test_weaviate_collection.py and test_mongodb_atlas_collection.py: 72 passed. With those four connector files taken back to the main version, 11 of the new list tests fail (3 Chroma, 4 Pinecone, 2 Weaviate, 2 MongoDB) and the other 61 pass. I did not run the rest of the memory test folders. The new workflow runs for 146d59c show action_required, so a maintainer needs to approve them before they start. |
Motivation and Context
Filter lambdas that use
inornot inwith a list literal always fail on the Chroma, MongoDB Atlas, Pinecone and Weaviate collections withNotImplementedError: Unsupported AST node: <class 'ast.List'>. Their_lambda_parserimplementations already handleast.Inandast.NotIn, but none of them parse the list on the right-hand side. Qdrant already does.Pinecone also computes
$ninfornot (x in [...])withnext(operand.values()), which raisesTypeError: 'dict_values' object is not an iteratorfor everynotfilter.Fixes #14557
Description
case ast.List(): return [self._lambda_parser(elt) for elt in node.elts]to_lambda_parserinmongodb.pyandpinecone.py, the same wayqdrant.pydoes it.ast.Listcase to_lambda_parserinchroma.pyandweaviate.pytoo. There, string and bytes elements are passed through as written instead of through the scalar string branch, which doubles apostrophes (x.content in ["O'Reilly"]would otherwise filter onO''Reilly). The scalar case is fixed separately in Python: Stop doubling apostrophes in Chroma and Weaviate filter values #14560.pinecone.py, replace thenext(operand.values())calls withnext(iter(operand.items())).not (x in [...])now becomes$nin, and any othernotraises the documentedNotImplementedErrorinstead of aTypeError._lambda_parserreturn annotation to includelist[Any].TextandNumfilters reject a list on the right side, so it needs a separate decision.Tests added for each connector, including an apostrophe case for Chroma and Weaviate. They fail on
mainand pass with this change.tests/unit/dataand the Chroma, Pinecone, MongoDB, Weaviate and Redis unit tests pass, and ruff and mypy report nothing new for the changed lines.Contribution Checklist