Python: Phase 4: Portable Filters & In-Memory Vector Store #4166

Parent: #1188
Depends on: #4165

Goal

Replace the experimental lambda/string filter translation with a portable, data-only filter contract, then provide a zero-dependency in-memory vector store as its first implementation.

Security rationale

Semantic Kernel in-memory filtering was affected by GHSA-xjw9-4gw8-4rqx / CVE-2026-26030, a critical remote-code-execution vulnerability. Phase 4 must not port or preserve source-string parsing, callable inspection, eval, exec, compile, or connector-specific AST semantics.

Part 1: Cross-store filter contract

• Document the data-only filter model, considered alternatives, extension rules, and trust boundary in the vector feature spec
• Add mutable Filter(field_name, operator, value) and FilterGroup(operator, filters) inputs that are snapshotted per operation
• Add Param(name, value_type, ...) references with native type, default, and constraint metadata
• Support common operators: eq, ne, gt, gte, lt, lte, between, in, not_in, is_null, is_not_null, exists, contains, contains_any, contains_all, starts_with, ends_with, and contains_text
• Support explicit and, or, and unary not groups
• Permit namespaced provider operators such as qdrant.geo_radius; reject unknown unqualified operators and unsupported provider operators
• Hard-remove callable/string filters, _LambdaVisitor, source inspection, AST parsing, and connector-specific _lambda_parser
• Pass one validated filter tree to every connector so combination semantics cannot drift
• Validate logical field names, operator/value arity, group arity, depth, node count, dunder/path segments, and finite numbers; fail closed
• Keep provider-native values structured and typed; never accept raw native query strings

Part 2: Search tool parameters

• Discover a Param used as the entire filter value or as a supported paging option
• Build a closed JSON Schema natively, without requiring a Pydantic model
• Validate model arguments against each Param type and inline constraints before substitution
• Allow fixed values and parameterized filter leaves to coexist in one filter tree
• Omit a filter leaf when its optional Param was not supplied; prune empty groups deterministically
• Require repeated references with the same name to have identical declarations
• Never allow a Param to replace a field name, operator, or one element inside a fixed container value
• Preserve fixed developer filters when model-controlled leaves are omitted
• Reject unresolved Param references in direct search() calls
• Preserve bounded model-set paging through Param values

Part 3: In-memory vector store

• Add InMemoryCollection and InMemoryStore in a dedicated core module
• Keep a real shared collection registry with compatible definitions across repeated clients
• Store serialized mappings and use defensive copies; never evaluate filters against model attributes
• Implement batch upsert/get/delete, listing, paging, deterministic order_by, collection lifecycle, and accurate collection names
• Implement flat vector search with pure-Python distance functions and no NumPy/SciPy dependency
• Support only deterministic common filter operators initially; reject full-text, geo, nested paths, and provider operators explicitly
• Apply the shared conservative filter depth and node limits plus in-memory value-type restrictions
• Add functional, isolation, malformed-input, exploit-corpus, and resource-limit tests
• Add direct-filter and search-tool samples and document process-local, nonpersistent, linear-scan behavior

Required security coverage

• Semantic Kernel sandbox-escape payloads remain ordinary data and cannot execute
• Multiple filters use explicit AND/OR semantics; fixed scope cannot be widened by model parameters
• Undeclared, dunder, malformed, oversized, unsupported, cyclic, and non-finite filters fail closed
• Membership direction, null/exists semantics, and range inclusivity are explicit and tested
• Filter evaluation never invokes model properties, descriptors, or arbitrary methods
• Unsupported namespaced operators fail closed in the in-memory implementation

Explicitly out of scope

• FAISS or other optional acceleration
• Keyword/hybrid search
• Raw provider query strings
• Provider-neutral regex, full-text, geo, arbitrary arithmetic, nested paths, or array-count semantics

Mergeable: Yes — the filter contract/tool changes and in-memory implementation can be reviewed separately.