Repository navigation
Python: InMemoryCollection combines multiple search filters with OR instead of AND #14538
Description
What happens
VectorSearchOptions documents that "When multiple filters are used, they are combined with an AND operator" (python/semantic_kernel/data/vector.py, class VectorSearchOptions). InMemoryCollection combines a list of filters with OR instead.
Where
python/semantic_kernel/connectors/in_memory.py, InMemoryCollection._get_filtered_records:
filtered_records: dict[TKey, AttributeDict] = {}
for key, record in self.inner_storage.items():
for filter in callable_filters:
if self._run_filter(filter, record):
filtered_records[key] = recordA record is kept as soon as any one filter matches. Every other connector (Postgres " AND ", Chroma $and, Mongo $and, Qdrant must, Weaviate all_of, Redis &, Azure AI Search " and ") uses AND.
Repro (checked on current main, 9974625)
import asyncio
from dataclasses import dataclass
from typing import Annotated
from semantic_kernel.connectors.in_memory import InMemoryCollection
from semantic_kernel.data.vector import VectorStoreField, vectorstoremodel
@vectorstoremodel
@dataclass
class R:
id: Annotated[str, VectorStoreField("key")]
color: Annotated[str, VectorStoreField("data", is_indexed=True)]
size: Annotated[int, VectorStoreField("data", is_indexed=True)]
vec: Annotated[list[float] | None, VectorStoreField("vector", dimensions=2)] = None
async def main():
c = InMemoryCollection(record_type=R, collection_name="x")
await c.upsert([R("1", "red", 1, [1, 0]), R("2", "red", 9, [1, 0]),
R("3", "blue", 1, [1, 0]), R("4", "blue", 9, [1, 0])])
r = await c.search(vector=[1, 0], top=10,
filter=["lambda x: x.color == 'red'", "lambda x: x.size > 5"])
print(sorted([x.record.id async for x in r.results]))
asyncio.run(main())Expected: ['2'] (red AND size > 5).
Actual: ['1', '2', '4'] (red OR size > 5).
The same happens when the filters are lambdas instead of strings.
Impact
Any multi-filter search against the in-memory store returns records that violate part of the filter. Code that works on a real store returns extra results here, and tests written against the in-memory store give wrong results.
I have a fix ready (all(...) over the filters) with tests, and will open the PR right away.
Activity