.Net MEVD: Modify ITextSearch to react to the new LINQ-based vector search filtering

Mark Wallace (@markwallace-microsoft)

ITextSearch accepts TextSearchOptions, which currently has an optional TextSearchFilter, which relies on the legacy non-LINQ vector search filtering feature. We need to modify it to use the new search interface. Mark Wallace (@markwallace-microsoft) let me know if/when you want to discuss ideas around this.

Mark Wallace (@markwallace-microsoft)

Mark Wallace (@markwallace-microsoft) just a reminder that this will need to happen at some point, not sure who's best for this...

.Net: Remove obsoleted MEVD APIs (#11891)

Shay Rojansky (@roji)

Hi Shay Rojansky (@roji), Mark Wallace (@markwallace-microsoft), Please let me know if I am on the right track for understanding and tackling this

Issue #10456 Analysis: Modernize ITextSearch to Use LINQ-Based Vector Search Filtering

Context: What Are These Filtering Systems?

Both filtering systems serve the same purpose: allowing developers to narrow down search results based on metadata or record properties before performing the actual search operation.

Common Use Case: A knowledge base with documents containing metadata like category, department, confidentiality, and dateCreated. Developers need to filter searches like:

"Find 'API documentation' but only in technical documents"
"Search 'budget planning' but exclude confidential materials"

The Two Approaches:

Legacy (TextSearchFilter): Clause-based with method chaining: filter.Equality("category", "technical")
Modern (LINQ): Type-safe expressions: doc => doc.Category == "technical" && doc.DateCreated > DateTime.Now.AddMonths(-6)

The LINQ approach provides compile-time safety, IntelliSense support, and complex boolean logic—which is why vector stores moved to it.

Overview

This document analyzes GitHub Issue #10456 which requests modernizing the ITextSearch interface to use LINQ-based filtering instead of the legacy TextSearchFilter approach.

Problem Statement

Current State: Inconsistent Filtering APIs

The Semantic Kernel .NET SDK currently has two different filtering systems:

Legacy Text Search Filtering (used by ITextSearch)
- Uses TextSearchFilter with clause-based filtering
- Location: dotnet/src/SemanticKernel.Abstractions/Data/TextSearch/
Modern Vector Search Filtering (used by vector stores)
- Uses LINQ expressions Expression<Func<TRecord, bool>>
- Location: dotnet/src/VectorData/VectorData.Abstractions/VectorSearch/

The Technical Debt

The current implementation in VectorStoreTextSearch.cs shows the problem:

var vectorSearchOptions = new VectorSearchOptions<TRecord>
{
    #pragma warning disable CS0618 // VectorSearchFilter is obsolete
    OldFilter = searchOptions.Filter?.FilterClauses is not null 
        ? new VectorSearchFilter(searchOptions.Filter.FilterClauses) 
        : null,
    #pragma warning restore CS0618
    // ...
}

This code converts from the legacy TextSearchFilter to the already-obsolete VectorSearchFilter, rather than using the modern LINQ-based approach.

Root Cause Analysis

Related Work: PR #10273

The foundation for LINQ-based filtering was established in PR #10273, which introduced:

VectorSearchOptions<TRecord>.Filter property using Expression<Func<TRecord, bool>>
Deprecation of the old VectorSearchFilter approach
Modern filtering infrastructure across vector database connectors

However, ITextSearch was not updated to use this new infrastructure, creating the current inconsistency.

Current Interface Design

ITextSearch Interface:

public interface ITextSearch
{
    Task<KernelSearchResults<string>> SearchAsync(
        string query,
        TextSearchOptions? searchOptions = null,  // ← Uses legacy filtering
        CancellationToken cancellationToken = default);
}

TextSearchOptions Class:

public sealed class TextSearchOptions
{
    public TextSearchFilter? Filter { get; init; }  // ← Legacy clause-based filter
    // ...
}

[Experimental("SKEXP0001")]
public sealed class TextSearchFilter  // ← Current filtering approach
{
    public IEnumerable<FilterClause> FilterClauses { get; }
    // ...
}

Versus Modern VectorSearchOptions:

public class VectorSearchOptions<TRecord>
{
    [Obsolete("Use Filter instead")]
    public VectorSearchFilter? OldFilter { get; set; }  // ← Being phased out
    
    public Expression<Func<TRecord, bool>>? Filter { get; set; }  // ← Modern LINQ approach
    // ...
}

Impact Assessment

Affected Components

Based on codebase analysis, the following components require updates:

Core Abstractions

Implementation Classes

VectorStoreTextSearch.cs - Primary implementation
BingTextSearch.cs
GoogleTextSearch.cs
TavilyTextSearch.cs
BraveTextSearch.cs

Tests and Samples

40+ test files across unit and integration tests
Multiple sample applications demonstrating text search

Breaking Change Analysis

This is a breaking change that requires careful migration strategy:

API Surface: TextSearchOptions.Filter type changes from TextSearchFilter? to Expression<Func<TRecord, bool>>?
Generic Constraint: Interface likely needs to become generic (ITextSearch<TRecord>)
Implementation Impact: All text search implementations need to adapt to LINQ expressions

Verification: Issue Not Addressed Elsewhere

Checked Related PRs/Issues

✅ PR .Net MEVD: LINQ-based filtering #10273: Established LINQ infrastructure but did not update ITextSearch
✅ Recent commits: No other PRs addressing this specific modernization
✅ Open issues: Issue .Net MEVD: Modify ITextSearch to react to the new LINQ-based vector search filtering #10456 is the primary tracking issue for this work

Current Status

Vector search: ✅ Uses modern LINQ filtering
Text search: ❌ Still uses legacy TextSearchFilter
Integration point: ❌ Uses obsolete conversion between the two systems

Next Steps

The solution requires:

Interface modernization to support LINQ-based filtering
Backward compatibility strategy to minimize breaking changes
Implementation updates across all text search connectors
Comprehensive testing to ensure no regression

References

Primary Issue: microsoft/semantic-kernel#10456
Foundation PR: microsoft/semantic-kernel#10273
Current Implementation: VectorStoreTextSearch.cs
Interface Definition: ITextSearch.cs
Modern Filtering: VectorSearchOptions.cs

Analysis completed on September 19, 2025

.Net: fix: Remove OldFilter dependency from VectorStoreTextSearch (mi…

.Net: fix: Remove OldFilter dependency from VectorStoreTextSearch (#1… (

.Net: Add provider-specific search parameters to Bing/Google connecto…

.Net: Net: Address roji/westey review feedback round 2 on text search…

Remove legacy filtering from MEVD providers

.Net: Remove legacy filtering from MEVD providers (#13636)

Navigation Menu

Repository navigation

.Net MEVD: Modify ITextSearch to react to the new LINQ-based vector search filtering #10456

Issue #10456 Analysis: Modernize ITextSearch to Use LINQ-Based Vector Search Filtering

Context: What Are These Filtering Systems?

Overview

Problem Statement

Current State: Inconsistent Filtering APIs

The Technical Debt

Root Cause Analysis

Related Work: PR #10273

Current Interface Design

Impact Assessment

Affected Components

Core Abstractions

Implementation Classes

Tests and Samples

Breaking Change Analysis

Verification: Issue Not Addressed Elsewhere

Checked Related PRs/Issues

Current Status

Next Steps

References

68 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

.Net MEVD: Modify ITextSearch to react to the new LINQ-based vector search filtering #10456

Description

Activity

roji commented on Mar 5, 2025

alzarei commented on Sep 20, 2025

Issue #10456 Analysis: Modernize ITextSearch to Use LINQ-Based Vector Search Filtering

Context: What Are These Filtering Systems?

Overview

Problem Statement

Current State: Inconsistent Filtering APIs

The Technical Debt

Root Cause Analysis

Related Work: PR #10273

Current Interface Design

Impact Assessment

Affected Components

Core Abstractions

Implementation Classes

Tests and Samples

Breaking Change Analysis

Verification: Issue Not Addressed Elsewhere

Checked Related PRs/Issues

Current Status

Next Steps

References

68 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions