feat: Add native complex types and scalar index tags to the Milvus online store by jiwidi · Pull Request #7120 · feast-dev/feast

jiwidi · 2026-11-09T13:14:15Z

What this PR does / why we need it:

With enable_openai_compatible_store: true the Milvus online store already stores numeric
features as native columns, so that filters compare numbers instead of text. Arrays, Json, Map
and Struct values were still base64 text, so a filter could not reach them and Milvus could not
validate them. This PR completes the native storage behind the same flag:

Feast type	With `enable_openai_compatible_store: true`
`Array` of `String`, `Bool`, `Int32`, `Int64`, `Float32`, `Float64` or `UnixTimestamp`, without `vector_index`	`ARRAY` of the element type, with at most `milvus.max_capacity` elements (default and maximum 4096)
`Json` and `Array(Json)`	`JSON`

Both columns are nullable unless the field has milvus.nullable: "false", so a missing value stays
null. A Json value must be an object or an array: PyMilvus cannot store a top-level string,
number or boolean in a JSON column, and JSON null becomes a null column value. Map and Struct
stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists.

The column types of an existing collection decide how a row is written and read, as #7025 does
for numeric columns, so collections created before this change keep working. Filter keys can
address a JSON path or an array element with Milvus syntax, so retrieve_online_documents_v2 and
online_delete can select rows by them:

ComparisonFilter(type="gt", key='attributes["price"]', value=100)
ComparisonFilter(type="eq", key="tags[0]", value="sale")

A filter on a scalar column scans the collection unless the column has an index. Field tags now
declare scalar indexes, which Feast creates with the collection, in the same way the
milvus.partition_key tag declares the partition key:

Field(name="brand_id", dtype=String, tags={"milvus.index": "INVERTED"}),
Field(name="price", dtype=Float64, tags={"milvus.index": "STL_SORT"}),
Field(
    name="attributes",
    dtype=Json,
    tags={"milvus.json_indexes": '[{"json_path": "attributes[\\"price\\"]", "json_cast_type": "double"}]'},
),

milvus.index names any scalar index type the server supports, milvus.index_params passes its
parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing
collection without them, Feast logs a warning, as it does for a missing partition key. The
Elasticsearch online store indexes every feature value by default, Qdrant creates payload indexes
on the fields its filters use, and Postgres indexes its filter columns; Milvus filters hit the
feature columns, and those have no index unless one is declared.

The docs describe the migration: the flag only applies when Feast creates a collection, a
feast teardown drops the collections of the whole project, and a staged cutover avoids it.

Benchmark. One million rows on a local single-node Milvus 3.0.2 in Docker, through PyMilvus:
a 1000-value brand text column, a price number, a tags array of two to four values and a
doc JSON object, plus an 8-dimensional HNSW vector. Three collections hold the same rows: text
columns as before this PR, native columns, and native columns with an INVERTED index on brand
and price and a JSON path index on doc["a"]. Each cell is the p50 / p95 latency of 30 runs
after 5 warm-ups, for a plain query with the filter and for a top_k=10 vector search with the
filter, measured after every index reported all rows indexed.

	text columns	native columns	native columns with indexes
insert one million rows	14.5 s	14.0 s	14.1 s
`brand == 'b_7'`, query	3.7 / 6.1 ms	3.5 / 4.4 ms	1.4 / 2.1 ms
`brand == 'b_7'`, search	4.0 / 5.2 ms	3.8 / 5.5 ms	2.5 / 2.7 ms
`price > 990`, query	rejected by Milvus	1.4 / 1.7 ms	1.4 / 2.2 ms
`price > 990`, search	rejected by Milvus	2.0 / 2.8 ms	2.9 / 3.8 ms
`ARRAY_CONTAINS(tags, 't_7')`, query	rejected by Milvus	11.8 / 12.8 ms	10.9 / 11.9 ms
`ARRAY_CONTAINS(tags, 't_7')`, search	rejected by Milvus	12.4 / 13.1 ms	14.4 / 15.3 ms
`doc["a"] > 990`, query	rejected by Milvus	62.8 / 64.0 ms	3.6 / 4.2 ms
`doc["a"] > 990`, search	rejected by Milvus	5.1 / 6.4 ms	4.9 / 6.1 ms

Native columns cost nothing at write time and make three kinds of filter possible that text
columns reject. The inverted index cuts the equality filter from 3.5 to 1.4 ms, and the JSON path
index cuts the JSON range query from 63 to 3.6 ms. The numeric index changes nothing at this size,
and a filtered vector search gains less than a plain query because the vector search dominates it.

Which issue(s) this PR fixes:

N/A

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change
Unit (mocked client): Json and arrays get nullable JSON and ARRAY columns with the
element type, milvus.max_capacity and max_length, Map stays VARCHAR; milvus.index and milvus.json_indexes tags are
passed to Milvus as indexes; the tags are validated; a missing index on an existing collection is
logged.
Unit (Milvus Lite): twelve complex values, empty arrays included, round-trip through
online_write_batch and online_read, and a missing value reads as null without any tag; JSON
null reads as a missing value and a top-level JSON string or number is rejected; a JSON column
tagged not nullable rejects a missing value; a JSON path filter selects search hits and an
array element filter selects rows to delete; the filter key syntax accepts paths and rejects
anything else.
Server: the same round-trip, the JSON path filter, the array element delete, and the scalar and
JSON path indexes against a real server.

Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The server test in
sdk/python/tests/integration/online_store/test_milvus_remote.py passes against a local Milvus
3.0.2 server. Milvus Lite does not support JSON path indexes, so that part runs on the server only.
The existing Milvus unit and server tests pass unchanged.

Misc

Stacked on #7119, whose commit is included here until it merges; review the last
commit. Follows #7082, #7083 and #7084 and Simon Hearne's series #6882, #6895, #6917, #6921 and
#6940. When #7025 lands, native_numeric_types will control these columns too, as it is an alias
of the same flag.

With the default configuration the Milvus online store could not return most non-numeric Feast types as written. A Bool failed to read, a UnixTimestamp got no column so every write of it failed, an Array(String) failed to read, a numeric array without vector_index was declared as a vector of embedding_dim and could not be written, Bytes came back as base64 text and Json as the base64 of its protobuf. A missing value was stored as the column's default, which failed to read for a numeric text column and came back as 0 or false for a native one. Search hits decoded only primitive types, and entity columns were written as the text of their Python value. The write path now stores every scalar natively or as text, bytes as base64 and every other value as the base64 of its ValueProto, for features and entity columns alike, and one helper rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column, only a field with vector_index=True becomes a vector, with the dimension from Field.vector_length before embedding_dim, and an empty text in a numeric or timestamp column reads as null. Array(Bool) with vector_index is rejected, because Milvus binary vectors were written unpacked and indexed with a float metric and could not work. The new milvus.nullable field tag makes a column nullable, so a missing value is stored and read as null. Signed-off-by: jiwidi <[email protected]>

…line store With enable_openai_compatible_store the Milvus online store stored numeric features as native columns, but arrays and Json values were still base64 text, so a filter could not reach them and Milvus could not validate them. With the flag, an array of scalars without vector_index is now an ARRAY column of the element type, with milvus.max_capacity elements at most, and a Json value, or an array of them, is a JSON column. Both columns are nullable unless the field says otherwise, so a missing value stays null. A Json value must be an object or an array, because PyMilvus cannot store a top-level string, number or bool in a JSON column, and JSON null is stored as a null column value. Map and Struct values stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, so collections created before this change keep working. Filter keys can address a JSON path or an array element, for example attributes["price"] or tags[0]. Field tags declare scalar indexes, which Feast creates with the collection: milvus.index names the index type of a scalar column, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them Feast logs a warning, as it does for a missing partition key. Signed-off-by: jiwidi <[email protected]>

codecov-commenter · 2026-11-09T13:21:14Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.65979% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.48%. Comparing base (bbaca76) to head (d825b1f).

Files with missing lines	Patch %	Lines
.../infra/online_stores/milvus_online_store/milvus.py	88.65%	18 Missing and 4 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7120      +/-   ##
==========================================
+ Coverage   51.32%   51.48%   +0.15%     
==========================================
  Files         458      458              
  Lines       57441    57534      +93     
  Branches     8421     8442      +21     
==========================================
+ Hits        29482    29620     +138     
+ Misses      25950    25917      -33     
+ Partials     2009     1997      -12

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.96% <88.65%> (+0.16%)`	⬆️

Files with missing lines	Coverage Δ
.../infra/online_stores/milvus_online_store/milvus.py	`88.00% <88.65%> (+10.41%)`	⬆️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbaca76...d825b1f. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

shuchu

lgtm

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

…line store (feast-dev#7120) * fix: Round-trip every Feast type and null in the Milvus online store With the default configuration the Milvus online store could not return most non-numeric Feast types as written. A Bool failed to read, a UnixTimestamp got no column so every write of it failed, an Array(String) failed to read, a numeric array without vector_index was declared as a vector of embedding_dim and could not be written, Bytes came back as base64 text and Json as the base64 of its protobuf. A missing value was stored as the column's default, which failed to read for a numeric text column and came back as 0 or false for a native one. Search hits decoded only primitive types, and entity columns were written as the text of their Python value. The write path now stores every scalar natively or as text, bytes as base64 and every other value as the base64 of its ValueProto, for features and entity columns alike, and one helper rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column, only a field with vector_index=True becomes a vector, with the dimension from Field.vector_length before embedding_dim, and an empty text in a numeric or timestamp column reads as null. Array(Bool) with vector_index is rejected, because Milvus binary vectors were written unpacked and indexed with a float metric and could not work. The new milvus.nullable field tag makes a column nullable, so a missing value is stored and read as null. Signed-off-by: jiwidi <[email protected]> * feat: Add native complex types and scalar index tags to the Milvus online store With enable_openai_compatible_store the Milvus online store stored numeric features as native columns, but arrays and Json values were still base64 text, so a filter could not reach them and Milvus could not validate them. With the flag, an array of scalars without vector_index is now an ARRAY column of the element type, with milvus.max_capacity elements at most, and a Json value, or an array of them, is a JSON column. Both columns are nullable unless the field says otherwise, so a missing value stays null. A Json value must be an object or an array, because PyMilvus cannot store a top-level string, number or bool in a JSON column, and JSON null is stored as a null column value. Map and Struct values stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, so collections created before this change keep working. Filter keys can address a JSON path or an array element, for example attributes["price"] or tags[0]. Field tags declare scalar indexes, which Feast creates with the collection: milvus.index names the index type of a scalar column, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them Feast logs a warning, as it does for a missing partition key. Signed-off-by: jiwidi <[email protected]> --------- Signed-off-by: jiwidi <[email protected]> Signed-off-by: Shuchu Han <[email protected]>

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>

jiwidi added 2 commits November 9, 2026 12:30

jiwidi requested a review from a team as a code owner November 9, 2026 13:14

shuchu approved these changes Nov 10, 2026

View reviewed changes

shuchu merged commit f7068d6 into feast-dev:master Nov 10, 2026
27 of 29 checks passed

patelchaitany mentioned this pull request Nov 10, 2026

chore: Sync upstream feast-dev/feast master opendatahub-io/feast#288

Merged

jiwidi mentioned this pull request Nov 10, 2026

fix: Round-trip every Feast type and null in the Milvus online store #7119

Closed

5 of 7 tasks

simonhearne mentioned this pull request Nov 10, 2026

feat: Add native_numeric_types to the Milvus online store #7025

Closed

4 of 7 tasks

jiwidi mentioned this pull request Nov 10, 2026

feat: Store arrays and Json natively in every new Milvus collection #7123

Merged

5 of 7 tasks

patelchaitany mentioned this pull request Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

rhods-devops-app Bot mentioned this pull request Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

jonasfagerlund mentioned this pull request Nov 12, 2026

feat: Add text analysis tags to the Milvus online store jiwidi/feast#4

Merged

5 of 7 tasks

jiwidi mentioned this pull request Nov 12, 2026

feat: Add text analysis tags to the Milvus online store #7143

Closed

5 of 7 tasks

Navigation Menu

Repository navigation

feat: Add native complex types and scalar index tags to the Milvus online store - #7120

feat: Add native complex types and scalar index tags to the Milvus online store#7120
shuchu merged 2 commits into
feast-dev:masterfrom
jiwidi:feat/milvus-native-types-and-indexes

jiwidi commented Nov 9, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Nov 9, 2026 •

edited

Loading

Uh oh!

shuchu left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jiwidi commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Misc

Uh oh!

codecov-commenter commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jiwidi commented Nov 9, 2026 •

edited

Loading

codecov-commenter commented Nov 9, 2026 •

edited

Loading