.Net MEVD: Filter-only search API (without vector similarity) by adamsitnik · Pull Request #11112 · microsoft/semantic-kernel

Adam Sitnik (adamsitnik) · 2025-03-21T15:34:37Z

GetFilteredRecordOptions<FilterRecord> options = new();

options.OrderBy
    .Ascending(r => r.Int)
    .Descending(r => r.String);

return await collection.GetAsync(record => record.Int > 2, top: 100, options).ToListAsync();

Implemented with tests for:

fixes #10295

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreDbClient.cs # dotnet/src/Connectors/Connectors.Memory.Postgres/SqlBuilder.cs # dotnet/src/Connectors/Connectors.Postgres.UnitTests/PostgresVectorStoreCollectionSqlBuilderTests.cs # dotnet/src/VectorDataIntegrationTests/AzureAISearchIntegrationTests/Support/AzureAISearchTestStore.cs

…o queryAsync # Conflicts: # dotnet/src/Connectors/VectorData/VectorStorage/LoggingVectorStoreRecordCollection.cs

…o queryAsync # Conflicts: # dotnet/samples/Concepts/Memory/VectorStoreEmbeddingGeneration/TextEmbeddingVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.AzureCosmosDBNoSQL/AzureCosmosDBNoSQLVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.MongoDB/MongoDBVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Qdrant/QdrantVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Weaviate/WeaviateVectorStoreRecordCollection.cs

Shay Rojansky (roji)

Awesome to see this, am really happy we'll have this capability!

Shay Rojansky (roji) · 2025-04-04T13:38:40Z

+        {
+            IsFilterable = dataProperty.IsFilterable,
+            // Sometimes the users ask to also OrderBy given filterable property, so we make it sortable.
+            IsSortable = dataProperty.IsFilterable && !fieldType.IsCollection


Yeah, I think this is one of the reasons we thought about renaming IsFilterable to IsIndexable indeed... In the future, we can have an Azure AI-specific IsSortable flag too.

Shay Rojansky (roji) · 2025-04-04T13:45:17Z

+    public VectorStoreRecordPropertyModel GetDataOrKeyProperty<TRecord>(Expression<Func<TRecord, object?>> expression)
+        => this.GetMatchingProperty<TRecord, VectorStoreRecordPropertyModel>(expression, data: true);
+
+    private TProperty GetMatchingProperty<TRecord, TProperty>(Expression<Func<TRecord, object?>> expression, bool data)


Just noting that this parameter is mainly there because of VectorStoreGenericDataModel, which has different dictionaries for data/vectors; that's going away with the transition to Dictionary<string, object?> tracked by #10802 (PR should be out this weekend).

In other words, I think this is super temporary and will be gone soon (so should be OK to merge as-is for now). We'd also rename GetDataOrKeyProperty to something like BindProperty (I'm already thinking about this also in relation to the filter translators...).

Shay Rojansky (roji) · 2025-04-04T20:49:40Z

+        foreach (var property in model.Properties)
+        {
+            if (options.IncludeVectors || property is not VectorStoreRecordVectorPropertyModel)
+            {
+                query.AppendFormat("\"{0}\",", property.StorageName);
+            }
+        }


Yeah, we should definitely do some passes to consolidate/refactor.

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.AzureCosmosDBMongoDB/AzureCosmosDBMongoDBVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.InMemory/InMemoryVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreDbClient.cs # dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Redis/RedisHashSetVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Redis/RedisJsonVectorStoreRecordCollection.cs # dotnet/src/Connectors/VectorData.Abstractions/CompatibilitySuppressions.xml

Shay Rojansky (roji)

OK, there seem to be a few more unresolved comments, but nothing that looks blocking - LGTM. Thanks Adam Sitnik (@adamsitnik)!

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs

- rename FilterOptions to GetFilteredRecordOptions - rename Sort to OrderBy - remove the limitation to select Key as the Redis ordeby field

QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate

0096e21

Adam Sitnik (adamsitnik) added .NET Issue or Pull requests regarding .NET code msft.ext.vectordata Related to Microsoft.Extensions.VectorData labels Mar 21, 2025

Mark Wallace (markwallace-microsoft) added kernel Issues or pull requests impacting the core kernel memory labels Mar 21, 2025

github-actions Bot changed the title ~~[DRAFT] .Net MEVD: QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate~~ .Net: [DRAFT] .Net MEVD: QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate Mar 21, 2025

Adam Sitnik (adamsitnik) had a problem deploying to integration March 21, 2025 15:35 — with GitHub Actions Failure

don't perform any kind of client-side ordering

c82106b

Adam Sitnik (adamsitnik) had a problem deploying to integration March 21, 2025 16:14 — with GitHub Actions Failure

implement QueryAsync for InMemory, Azure AI Search, MongoDB and CosmosDB

ff98cd8

Adam Sitnik (adamsitnik) had a problem deploying to integration March 24, 2025 17:44 — with GitHub Actions Failure

Adam Sitnik (adamsitnik) changed the title ~~.Net: [DRAFT] .Net MEVD: QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate~~ .Net: [DRAFT] MEVD: QueryAsync Mar 24, 2025

implement QueryAsync for Redis, SQL Server

c93dacd

Adam Sitnik (adamsitnik) had a problem deploying to integration March 25, 2025 17:53 — with GitHub Actions Failure

implement QueryAsync for PostgreSql and Sqlite

ddfc5e5

Adam Sitnik (adamsitnik) had a problem deploying to integration March 26, 2025 14:11 — with GitHub Actions Failure

fix the build

c4bf2b5

Adam Sitnik (adamsitnik) temporarily deployed to integration March 27, 2025 16:14 — with GitHub Actions Inactive

Adam Sitnik (adamsitnik) changed the title ~~.Net: [DRAFT] MEVD: QueryAsync~~ .Net MEVD: QueryAsync Mar 27, 2025

Adam Sitnik (adamsitnik) marked this pull request as ready for review March 27, 2025 16:38

Adam Sitnik (adamsitnik) requested a review from a team as a code owner March 27, 2025 16:38

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

Adam Sitnik (adamsitnik) added 4 commits April 1, 2025 16:47

change the API design to what we have agreed on part 1/2

72319f8

change the API design to what we have agreed on part 2/2

ef18c35

Merge remote-tracking branch 'upstream/feature-vector-data-preb2' int…

0f92d08

…o queryAsync # Conflicts: # dotnet/src/Connectors/VectorData/VectorStorage/LoggingVectorStoreRecordCollection.cs

Adam Sitnik (adamsitnik) had a problem deploying to integration April 2, 2025 11:27 — with GitHub Actions Failure

fix the build

9072402

Adam Sitnik (adamsitnik) temporarily deployed to integration April 2, 2025 11:43 — with GitHub Actions Inactive

Adam Sitnik (adamsitnik) had a problem deploying to integration April 4, 2025 11:51 — with GitHub Actions Failure

Adam Sitnik (adamsitnik) added 2 commits April 4, 2025 16:28

address code review feedback

69d44fc

Adam Sitnik (adamsitnik) had a problem deploying to integration April 4, 2025 14:34 — with GitHub Actions Error

Adam Sitnik (adamsitnik) commented Apr 4, 2025

View reviewed changes

Comment thread dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeVectorStoreRecordCollection.cs Resolved

Apply suggestions from code review

9b226c5

Adam Sitnik (adamsitnik) had a problem deploying to integration April 4, 2025 14:36 — with GitHub Actions Failure

fix the build

281a9e0

Adam Sitnik (adamsitnik) temporarily deployed to integration April 4, 2025 15:01 — with GitHub Actions Inactive

Adam Sitnik (adamsitnik) requested a review from westey (westey-m) April 4, 2025 15:33

Shay Rojansky (roji) requested changes Apr 4, 2025

View reviewed changes

address code review feedback

287b4f6

Adam Sitnik (adamsitnik) temporarily deployed to integration April 7, 2025 10:27 — with GitHub Actions Inactive

Adam Sitnik (adamsitnik) commented Apr 7, 2025

View reviewed changes

Comment thread dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs Resolved

Adam Sitnik (adamsitnik) requested a review from Shay Rojansky (roji) April 7, 2025 10:36

westey (westey-m) approved these changes Apr 7, 2025

View reviewed changes

Adam Sitnik (adamsitnik) had a problem deploying to integration April 7, 2025 15:01 — with GitHub Actions Failure

try to fix a build error that I can't reproduce locally ...

247749f

Adam Sitnik (adamsitnik) had a problem deploying to integration April 7, 2025 15:44 — with GitHub Actions Failure

why on earth is it still failing and only in CI?

304068e

Adam Sitnik (adamsitnik) had a problem deploying to integration April 7, 2025 15:56 — with GitHub Actions Failure

Shay Rojansky (roji) approved these changes Apr 8, 2025

View reviewed changes

Comment thread dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs Resolved

Comment thread dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs Resolved

Adam Sitnik (adamsitnik) added 2 commits April 9, 2025 11:01

Merge remote-tracking branch 'upstream/feature-vector-data-preb2' int…

0cc1bf5

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs

address code review feedback:

de72aec

- rename FilterOptions to GetFilteredRecordOptions - rename Sort to OrderBy - remove the limitation to select Key as the Redis ordeby field

Adam Sitnik (adamsitnik) temporarily deployed to integration April 9, 2025 09:17 — with GitHub Actions Inactive

Adam Sitnik (adamsitnik) merged commit 747531c into microsoft:feature-vector-data-preb2 Apr 9, 2025

This was referenced Apr 9, 2025

[MEVD] Filter-only search API (without vector similarity) #10295

Closed

.Net MEVD Sqlite: Escaping #11534

Merged

.Net MEVD: fix two bugs #11585

Merged

Navigation Menu

Repository navigation

.Net MEVD: Filter-only search API (without vector similarity) - #11112

.Net MEVD: Filter-only search API (without vector similarity)#11112
Adam Sitnik (adamsitnik) merged 24 commits into
microsoft:feature-vector-data-preb2from
adamsitnik:queryAsync

Adam Sitnik (adamsitnik) commented Mar 21, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

This comment was marked as resolved.

Uh oh!

Shay Rojansky (roji) left a comment

Uh oh!

Uh oh!

Shay Rojansky (roji) Apr 4, 2025

Uh oh!

Uh oh!

Shay Rojansky (roji) Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shay Rojansky (roji) Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shay Rojansky (roji) left a comment

Uh oh!

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Adam Sitnik (adamsitnik) commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as resolved.

Uh oh!

Shay Rojansky (roji) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Shay Rojansky (roji) Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Shay Rojansky (roji) Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shay Rojansky (roji) Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shay Rojansky (roji) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Adam Sitnik (adamsitnik) commented Mar 21, 2025 •

edited

Loading