Repository navigation
Python: Match float and bool filter values in QdrantCollection - #14777
Harshil Kaneria (Harshil26202) wants to merge 3 commits into
Conversation
The Qdrant filter parser built MatchValue for == and != and MatchAny for in and not in. MatchValue only accepts str, int and bool, MatchAny only a list of str or int, so filters such as 'x.price == 4.5' or 'x.in_stock in [True]' raised a pydantic ValidationError. An int value also never matched a float payload, so 'x.price == 5' missed 5.0. Match numbers with a closed Range, which Qdrant compares numerically, and keep MatchValue for ints above 2**53 because ranges are compared as float64. Keep MatchAny for lists of strings and expand other lists into a should of single-value matches.
GitHub Actions
Active
semantic-kernel-automation
Bot
added
the
python
There was a problem hiding this comment.
🟡 Changes recommended
The numeric filtering helper mishandles the 2**53 integer boundary, causing incorrect matches, and routes large floats to MatchValue, which rejects them.
2 open issues
What changed in this PR
This PR extends Python’s QdrantCollection filter translation to support float comparisons and boolean membership filters, bringing numeric matching closer to InMemoryCollection.
Changes:
- Uses closed numeric ranges for equality comparisons, with an exact-match fallback for large integers.
- Translates non-string membership lists into individual value conditions.
- Adds seven local Qdrant regression cases for float and boolean filters.
| File | Description |
|---|---|
python/tests/unit/connectors/memory/test_qdrant_local.py |
Adds a product fixture and parameterized float and boolean filter tests. |
python/semantic_kernel/connectors/qdrant.py |
Adds shared matching helpers and integrates them into equality and membership parsing. |
Review effort: Balanced
| } | ||
| IN_MEMORY_STRING: Final[str] = ":memory:" | ||
| # The server compares ranges as float64, so larger ints are matched exactly with MatchValue instead. | ||
| MAX_EXACT_FLOAT_INT: Final[int] = 2**53 |
There was a problem hiding this comment.
Good catch, fixed in b570677. The cutoff is now abs(value) < 2**53, so 253 itself stays on MatchValue. Local mode compares ranges with exact Python ints, so a local test can't show the server-side rounding. test_match_value_uses_range_only_where_float64_is_exact pins which condition gets built at 253 - 1, 253 and 260 instead.
| def _match_value(key: str, value: Any) -> FieldCondition: | ||
| """Match a single value, numerically for numbers so 5 also matches 5.0 and floats are accepted.""" | ||
| # MatchValue only accepts str, int and bool, and an int never matches a float payload | ||
| if isinstance(value, (int, float)) and not isinstance(value, bool) and abs(value) <= MAX_EXACT_FLOAT_INT: |
There was a problem hiding this comment.
Agreed, fixed in b570677. The cutoff now applies to ints only, so any float uses Range. I added x.price == 1e20 and != 1e20 to the local filter tests. Both failed with a MatchValue validation error before this commit.
…filters The float64 cutoff also applied to floats, so a value such as 1e20 fell through to MatchValue, which rejects floats. And it included 2**53, which float64 can't tell apart from 2**53 + 1. Apply the cutoff to ints only, and keep ints from 2**53 up on MatchValue.
GitHub Actions
Active
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
Muhammad Abdullah Butt (1AbdullahButt)
left a comment
There was a problem hiding this comment.
Looks correct for the cases it targets. Here's what I checked.
_match_value(qdrant.py:87). Floats, and ints withabs(value) < 2**53that aren't bools, becomeRange(gte=v, lte=v). Bools are excluded by thenot isinstance(value, bool)check, sox.in_stock == Truestill usesMatchValue. So do strings. Bools never reach the range branch, and a bool payload isn't numeric.!=andnot in. Both still wrap the condition inmust_not. So a record that lacks the field is returned, as onmain.not inover a mixed list becomesmust_not=[Filter(should=[...])], which is "none of these". That is the right meaning._match_any(qdrant.py:96). An empty list still takes the all-strbranch (all()of nothing isTrue) and givesMatchAny(any=[]), asmaindid.- Chained comparisons. The chain branch (
qdrant.py:395) calls_lambda_parseron each pair, sox.price == 4.5inside a chain also goes through_match_value. git merge-treeagainstmainis clean.
I didn't run the tests. I also didn't check the server-side float64 behaviour behind the 2**53 cutoff myself. It's plausible, and the description notes that local mode can't show it.
One cost to consider: a list of ints used to be a single MatchAny, and it now becomes one Range condition per value under should. For x.category_id in ids with a few hundred ids, the request and the filter grow linearly. The definition already knows each field's declared type, so _lambda_parser could look up the field by left and keep MatchValue/MatchAny when its type_ is "int". That would use Range only for float fields, or when the type is unknown. It would keep exact int matching (and the 2**53 question) out of int fields entirely, and x.price == 5 on a float field would still match 5.0.
Reviewed with help from Claude Code.
…Qdrant filters A field declared as int holds int payloads, so an int value there can be matched exactly. This keeps a list of ints as a single MatchAny instead of one Range per value, and int fields no longer depend on the 2**53 cutoff. Floats, and ints on fields of any other or unknown type, still use Range.
GitHub Actions
Active
|
Muhammad Abdullah Butt (@1AbdullahButt) thanks for the careful read. Good point on the int lists, done in ff3b3df.
|
Motivation and Context
QdrantCollection._lambda_parserbuildsMatchValuefor==and!=, andMatchAnyforinandnot in. In qdrant-client,MatchValueonly acceptsstr,intandbool, andMatchAnyonly accepts a list ofstror a list ofint. So common filters on float and bool fields fail before the query is sent:There is a quieter case as well. An int value never matches a float payload in Qdrant, so
lambda x: x.price == 5returns nothing for a record stored withprice=5.0.InMemoryCollectionreturns that record, as Python would.Description
==and!=with a number (not a bool) now use a closed range,Range(gte=value, lte=value). Qdrant compares ranges numerically, so this works for float values and matches int and float payloads alike. Ints withabs(value) >= 2**53keepMatchValue, because the server compares ranges as float64, where2**53and2**53 + 1are the same value. Floats of any size use the range.inandnot inkeepMatchAnyfor lists of strings. Other lists become ashouldof the single-value conditions above, so float and bool lists work. Mixed int/float lists also match numerically.==/!=still useMatchValue.Tests:
test_local_filter_with_float_and_bool_valuesintests/unit/connectors/memory/test_qdrant_local.pyruns nine filters against a local:memory:Qdrant collection, including a large float (1e20). All of them fail onmainand pass with this change.test_match_value_uses_range_only_where_float64_is_exactpins the int cutoff, since local mode compares exact Python ints and can't show the server's float64 rounding. I also compared about 600 random nested filters over int, float, bool and string fields betweenQdrantCollection(location=":memory:")andInMemoryCollection, and the results now match. The only remaining errors were chained comparisons with the constant on the left (0 <= x.num < 5), which is a separate issue and not part of this PR.Locally,
tests/unit/dataplus the in-memory, Qdrant, Qdrant local, Chroma and FAISS connector tests pass (263 passed).ruff checkandruff format --checkare clean.mypyreports one error inqdrant.pyat thedeletecall (line 281), which is also there onmain.AI disclosure: I found this with Claude Code while comparing filter results across the Python vector store connectors. It also drafted the fix and the tests.
Contribution Checklist