Skip to content

.Net: Introduce a LINQ filter preprocessor in MEVD - #11702

Merged
Shay Rojansky (roji) merged 1 commit into
microsoft:feature-vector-data-preb2from
roji:FilterPreprocessor
Apr 26, 2025
Merged

Shay Rojansky (roji) merged 1 commit into
microsoft:feature-vector-data-preb2from
roji:FilterPreprocessor

Conversation

@roji

Copy link
Copy Markdown
Member

This is a refactor-only change, with no functional changes. It extracts out all logic for identifying and processing captured variables ("parameters") out to a new FilterTranslationProcessor connectos support type in MEVD.

  • Most databases don't support parameterization; for these we simply inline the captured variables. Connector translators see these as constants without ever having to know about captured variables etc.
  • SQL databases do support parameterization. For these, the preprocessor transforms the captured variable (which is a complex tree construct) into a simple QueryParameterExpression, which can easily be pattern-matched in connector translator code.

This is preparatory work for #11673 (we need a centralized place to do this kind of thing).

@roji
Shay Rojansky (roji) requested a review from a team as a code owner April 23, 2025 22:09
@markwallace-microsoft Mark Wallace (markwallace-microsoft) added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory labels Apr 23, 2025
@github-actions github-actions Bot changed the title Introduce a LINQ filter preprocessor in MEVD .Net: Introduce a LINQ filter preprocessor in MEVD Apr 23, 2025
using System.Reflection;
using System.Runtime.CompilerServices;

namespace Microsoft.Extensions.VectorData.ConnectorSupport.Filter;

@dmytrostruk Dmytro Struk (dmytrostruk) Apr 24, 2025 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep everything related to connector support under single Microsoft.Extensions.VectorData.ConnectorSupport namespace to avoid namespace explosion and for better discoverability.

@roji
Shay Rojansky (roji) merged commit 8b8ccd6 into microsoft:feature-vector-data-preb2 Apr 26, 2025

default:
throw new NotSupportedException($"Member access for '{memberExpression.Member.Name}' is unsupported - only member access over the filter parameter are supported");
this._filter.Append(property.StorageName); // TODO: Escape

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shay Rojansky (@roji) Dmytro Struk (@dmytrostruk) I fail to understand why this PR was reviewed and merged before #11676 which aims to solve the problem of escaping.

I did my best to ensure I review other people PRs first so everybody gets unblocked. As a result, I got none of my PRs reviewed, and all with merge conflicts. This is nice... not.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really sorry Adam Sitnik (@adamsitnik), I've been in a tunnel, focusing on my things, and I've neglected your PRs (and forgot that this would cause a conflict... I'm really sorry about that, my bad. I'll go review it now.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can also take care of the conflict resolution too, as it's my fault.

This branch was previously deployed

1 inactive deployment
integration — 9ef845e7 Deployed Apr 26, 2025 by roji
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Issues or pull requests impacting the core kernel memory .NET Issue or Pull requests regarding .NET code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants