Skip to content

Commit 206309f

Browse files
simonhearneclaude
simonhearne
and
claude
committed
docs: Add migration warning and correct VARCHAR filter behaviour
Teardown drops every collection in the project and loses online-only rows, so the docs now warn about it and describe a staged cutover that avoids it. Numeric range filters on VARCHAR collections don't compare lexicographically as the warning and docs said: Milvus rejects numeric filter values on VARCHAR fields, and only string values compare as text. Correct both, and add a regression test that turns on native_numeric_types over an existing VARCHAR collection. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Simon Hearne <[email protected]>
1 parent 675a536 commit 206309f

3 files changed

Lines changed: 73 additions & 26 deletions

File tree

‎docs/reference/online-stores/milvus.md‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -182,19 +182,31 @@ changed; Feast logs a warning for them. To add a partition key to an existing fe
182182
## Numeric types
183183

184184
By default, numeric and bool features are stored as `VARCHAR`, so range filters (`gt`, `gte`, `lt`,
185-
`lte`) compare strings: `"9" > "100"` is true. Set `native_numeric_types: true` to store them as
186-
native Milvus `INT32`, `INT64`, `FLOAT`, `DOUBLE` and `BOOL` fields, which compare numerically.
187-
`enable_openai_compatible_store: true` has the same effect.
185+
`lte`) don't work on them: Milvus rejects a numeric filter value (`comparisons between VarChar and
186+
Int64 are not supported`), and a string value compares as text (`"9" > "50"` is true). Set
187+
`native_numeric_types: true` to store them as native Milvus `INT32`, `INT64`, `FLOAT`, `DOUBLE`
188+
and `BOOL` fields, which compare numerically. `enable_openai_compatible_store: true` has the same
189+
effect.
188190

189191
The setting only applies when Feast creates a collection. Feast detects the storage type of each
190-
existing collection, so reads and writes keep working whichever way the collection was created.
191-
To migrate an existing feature view:
192+
existing collection, so reads and writes keep working whichever way the collection was created,
193+
and Feast logs a warning when numeric range filters are used on a `VARCHAR` collection.
192194

193-
1. Set `native_numeric_types: true`.
194-
2. Run `feast teardown` to drop the collections, then `feast apply` to recreate them.
195-
3. Materialize again to reload the data.
195+
To migrate an existing feature view, set `native_numeric_types: true` and recreate its collection
196+
in one of these ways:
196197

197-
Until a collection is recreated, Feast logs a warning when numeric range filters are used on it.
198+
* **Teardown and reload.** Run `feast teardown` and `feast apply`, then materialize again.
199+
* **Staged cutover.** Add a copy of the feature view under a new name and run `feast apply`, which
200+
creates a new collection with native types. Materialize it, move readers to the new name, then
201+
remove the old feature view and run `feast apply` again to drop its collection.
202+
203+
{% hint style="warning" %}
204+
`feast teardown` drops the collections of every feature view in the project, not just the one you
205+
are migrating. Online reads fail until materialization finishes, and rows that only exist in the
206+
online store, such as those written by push sources or `write_to_online_store`, are lost. For
207+
production, use the staged cutover, or back up the collections first with
208+
[Milvus Backup](https://milvus.io/docs/milvus_backup_overview.md).
209+
{% endhint %}
198210

199211
## Consistency level
200212

‎sdk/python/feast/infra/online_stores/milvus_online_store/milvus.py‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -831,12 +831,12 @@ def retrieve_online_documents_v2(
831831
if not has_native:
832832
logger.warning(
833833
"Numeric comparison filters (gt, gte, lt, lte) are being used "
834-
"but this collection stores numeric fields as VARCHAR. This "
835-
"causes lexicographic comparison instead of numeric comparison "
836-
"(e.g. '9' > '100' is True as a string). To fix this, set "
837-
"'native_numeric_types: true' in your online_store "
838-
"config, then teardown and re-apply your feature store to "
839-
"recreate collections with native numeric types."
834+
"but this collection stores numeric fields as VARCHAR. Milvus "
835+
"rejects numeric filter values on VARCHAR fields, and string "
836+
"values compare lexicographically (e.g. '9' > '50' is True). "
837+
"To fix this, set 'native_numeric_types: true' in your "
838+
"online_store config and recreate the collection; see the "
839+
"Milvus online store docs for migration options."
840840
)
841841

842842
metadata_filter_expr = MilvusFilterTranslator().translate(filters)

‎sdk/python/tests/unit/infra/online_store/test_milvus_online_store.py‎

Lines changed: 46 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
import pytest
99
from pydantic import ValidationError
10-
from pymilvus import DataType, MilvusClient
10+
from pymilvus import DataType, MilvusClient, MilvusException
1111
from pymilvus.client.types import LoadState
1212

1313
from feast import Entity, FeatureView
@@ -769,11 +769,9 @@ def test_native_numeric_types(
769769
assert {f["name"]: f["type"] for f in fields}["price"] == expected_type
770770

771771

772-
def test_native_numeric_range_filter_and_round_trip(tmp_path: Path) -> None:
773-
config = _lite_config(tmp_path, native_numeric_types=True)
774-
fv = _catalog_feature_view()
775-
store = MilvusOnlineStore()
776-
store.update(config, [], [fv], [], [], partial=False)
772+
def _write_priced_products(
773+
store: MilvusOnlineStore, config: RepoConfig, fv: FeatureView
774+
) -> None:
777775
embedding = ValueProto()
778776
embedding.float_list_val.val.extend([1.0, 0.0])
779777
now = datetime.now(timezone.utc)
@@ -797,19 +795,56 @@ def test_native_numeric_range_filter_and_round_trip(tmp_path: Path) -> None:
797795
progress=None,
798796
)
799797

800-
# As strings, "9.0" > "50" would be true; natively only 100.0 matches.
798+
799+
def _titles_priced_above(
800+
store: MilvusOnlineStore, config: RepoConfig, fv: FeatureView, value: Any
801+
) -> List[str]:
801802
results = store.retrieve_online_documents_v2(
802803
config,
803804
fv,
804805
["embedding", "title", "price"],
805806
embedding=[1.0, 0.0],
806807
top_k=5,
807808
distance_metric="COSINE",
808-
filters=ComparisonFilter(type="gt", key="price", value=50),
809+
filters=ComparisonFilter(type="gt", key="price", value=value),
809810
)
810-
assert [values["title"].string_val for _, _, values in results if values] == [
811-
"pricey"
812-
]
811+
return [values["title"].string_val for _, _, values in results if values]
812+
813+
814+
def test_native_numeric_range_filter_and_round_trip(tmp_path: Path) -> None:
815+
config = _lite_config(tmp_path, native_numeric_types=True)
816+
fv = _catalog_feature_view()
817+
store = MilvusOnlineStore()
818+
store.update(config, [], [fv], [], [], partial=False)
819+
_write_priced_products(store, config, fv)
813820

821+
# As strings, "9.0" > "50" would be true; natively only 100.0 matches.
822+
assert _titles_priced_above(store, config, fv, 50) == ["pricey"]
823+
824+
rows = store.online_read(config, fv, [_product_key(1)], ["price", "title"])
825+
assert rows[0][1] is not None and rows[0][1]["price"].float_val == 9.0
826+
827+
828+
def test_native_numeric_types_keeps_existing_varchar_collection(
829+
tmp_path: Path, caplog: pytest.LogCaptureFixture
830+
) -> None:
831+
fv = _catalog_feature_view()
832+
MilvusOnlineStore().update(_lite_config(tmp_path), [], [fv], [], [], partial=False)
833+
834+
# Turn the setting on without a teardown: the collection stays VARCHAR.
835+
config = _lite_config(tmp_path, native_numeric_types=True)
836+
store = MilvusOnlineStore()
837+
store.update(config, [], [fv], [], [], partial=False)
838+
assert store.client is not None
839+
fields = store.client.describe_collection("test_milvus_products")["fields"]
840+
assert {f["name"]: f["type"] for f in fields}["price"] == DataType.VARCHAR
841+
842+
_write_priced_products(store, config, fv)
814843
rows = store.online_read(config, fv, [_product_key(1)], ["price", "title"])
815844
assert rows[0][1] is not None and rows[0][1]["price"].float_val == 9.0
845+
846+
# Milvus rejects numeric values on VARCHAR fields; strings compare as text.
847+
with pytest.raises(MilvusException):
848+
_titles_priced_above(store, config, fv, 50)
849+
assert "stores numeric fields as VARCHAR" in caplog.text
850+
assert _titles_priced_above(store, config, fv, "50") == ["cheap"]

0 commit comments

Comments
 (0)