Skip to content

Commit f7068d6

Browse files
authored
feat: Add native complex types and scalar index tags to the Milvus online store (feast-dev#7120)
* fix: Round-trip every Feast type and null in the Milvus online store With the default configuration the Milvus online store could not return most non-numeric Feast types as written. A Bool failed to read, a UnixTimestamp got no column so every write of it failed, an Array(String) failed to read, a numeric array without vector_index was declared as a vector of embedding_dim and could not be written, Bytes came back as base64 text and Json as the base64 of its protobuf. A missing value was stored as the column's default, which failed to read for a numeric text column and came back as 0 or false for a native one. Search hits decoded only primitive types, and entity columns were written as the text of their Python value. The write path now stores every scalar natively or as text, bytes as base64 and every other value as the base64 of its ValueProto, for features and entity columns alike, and one helper rebuilds the ValueProto for reads and searches from every declared field type. UnixTimestamp gets a column, only a field with vector_index=True becomes a vector, with the dimension from Field.vector_length before embedding_dim, and an empty text in a numeric or timestamp column reads as null. Array(Bool) with vector_index is rejected, because Milvus binary vectors were written unpacked and indexed with a float metric and could not work. The new milvus.nullable field tag makes a column nullable, so a missing value is stored and read as null. Signed-off-by: jiwidi <[email protected]> * feat: Add native complex types and scalar index tags to the Milvus online store With enable_openai_compatible_store the Milvus online store stored numeric features as native columns, but arrays and Json values were still base64 text, so a filter could not reach them and Milvus could not validate them. With the flag, an array of scalars without vector_index is now an ARRAY column of the element type, with milvus.max_capacity elements at most, and a Json value, or an array of them, is a JSON column. Both columns are nullable unless the field says otherwise, so a missing value stays null. A Json value must be an object or an array, because PyMilvus cannot store a top-level string, number or bool in a JSON column, and JSON null is stored as a null column value. Map and Struct values stay serialized, because JSON cannot hold their nested bytes, timestamps or empty typed lists. The column types of an existing collection decide how a row is written and read, so collections created before this change keep working. Filter keys can address a JSON path or an array element, for example attributes["price"] or tags[0]. Field tags declare scalar indexes, which Feast creates with the collection: milvus.index names the index type of a scalar column, milvus.index_params passes its parameters and milvus.json_indexes lists JSON path indexes of a JSON column. For an existing collection without them Feast logs a warning, as it does for a missing partition key. Signed-off-by: jiwidi <[email protected]> --------- Signed-off-by: jiwidi <[email protected]>
1 parent bbaca76 commit f7068d6

4 files changed

Lines changed: 1238 additions & 237 deletions

File tree

‎docs/reference/online-stores/milvus.md‎

Lines changed: 82 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ online_store:
101101
| `partition_key_isolation` | unset | Enables partition key isolation on a new collection that has a partition key. When unset, Milvus uses its default. |
102102
| `vector_enabled` | `true` | Enables vector search. |
103103
| `varchar_max_length` | `65535` | Default `max_length` of VARCHAR fields. Override per field with the `max_length` tag. |
104-
| `enable_openai_compatible_store` | `false` | Store numeric features as native Milvus numeric types. |
104+
| `enable_openai_compatible_store` | `false` | Store numeric, array, `Json`, `Map` and `Struct` features as native Milvus types. See [Field types](#field-types). |
105105

106106
The full set of configuration options is available in [MilvusOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.milvus.MilvusOnlineStoreConfig).
107107

@@ -225,6 +225,87 @@ with an empty group is rejected, so a mistake cannot delete more rows than it na
225225
leaves no tombstone: a later write creates it again, and Feast does not order a delete against
226226
writes from other processes. Deletion works on every collection that Feast created.
227227

228+
## Field types
229+
230+
Each feature is a column of the collection. A value comes back as it was written, whatever its
231+
type.
232+
233+
| Feast type | Default | With `enable_openai_compatible_store: true` |
234+
|:-----------|:--------|:--------------------------------------------|
235+
| `String` | `VARCHAR` | `VARCHAR` |
236+
| `Bool`, `Int32`, `Int64`, `Float32`, `Float64` | `VARCHAR` holding the value as text | `BOOL`, `INT32`, `INT64`, `FLOAT`, `DOUBLE` |
237+
| `UnixTimestamp` | `VARCHAR` holding the seconds as text | `INT64` |
238+
| `Bytes` | `VARCHAR` holding base64 | `VARCHAR` holding base64 |
239+
| `Array` with `vector_index=True` | `FLOAT_VECTOR` with the field's `vector_length`, or `embedding_dim` when it is unset | the same |
240+
| `Array` of `String`, `Bool`, `Int32`, `Int64`, `Float32`, `Float64` or `UnixTimestamp` without `vector_index` | `VARCHAR` holding the serialized value | `ARRAY` of the element type, with at most `milvus.max_capacity` elements. The default and the maximum are 4096 |
241+
| `Json` and `Array(Json)` | `VARCHAR` holding the serialized value | `JSON` |
242+
| Every other type, including `Map` and `Struct` | `VARCHAR` holding the serialized value | the same |
243+
244+
Filters apply to `VARCHAR` and native columns. A column that holds a serialized value can be read
245+
but not filtered on. A filter key can address a JSON path or an array element with Milvus syntax:
246+
247+
```python
248+
ComparisonFilter(type="gt", key='attributes["price"]', value=100)
249+
ComparisonFilter(type="eq", key="tags[0]", value="sale")
250+
```
251+
252+
A `Json` value in a `JSON` column must be an object or an array: a top-level string, number or
253+
boolean is rejected, and `null` is stored as a null column value and reads as a missing value. The
254+
value comes back as the JSON text of the stored document, so its formatting is normalized. `Map` and
255+
`Struct` values stay serialized, because JSON cannot hold their nested bytes, timestamps or empty
256+
typed lists. `JSON` and `ARRAY` columns are nullable unless the field has `milvus.nullable: "false"`.
257+
258+
{% hint style="warning" %}
259+
`enable_openai_compatible_store` only applies when Feast creates a collection. Feast detects the
260+
column types of an existing collection, so reads and writes keep working whichever way it was
261+
created. `feast teardown` drops the collections of every feature view in the project, online reads
262+
fail until materialization finishes, and rows that exist only in the online store are lost. To
263+
migrate without a teardown, add a copy of the feature view under a new name, materialize it, move
264+
readers to it, then remove the old view.
265+
{% endhint %}
266+
267+
`Array(Bool)` cannot be a vector field: Milvus binary vectors are not supported.
268+
269+
A missing or null value is written as the column's default: an empty string, `0` or `false`. Feast
270+
reads an empty string in a numeric or timestamp text column as null, but a missing `String` or
271+
`Bytes` value comes back empty and a `0` in a native numeric column comes back as `0`. Tag a field
272+
with `milvus.nullable: "true"` to store null as `NULL` instead:
273+
274+
```python
275+
Field(name="rating", dtype=Float32, tags={"milvus.nullable": "true"})
276+
```
277+
278+
The tag applies when Feast creates the collection, and it needs Milvus 2.5 or later. Entity,
279+
partition key and vector fields cannot be nullable.
280+
281+
## Scalar indexes
282+
283+
A filter on a scalar column scans the collection unless the column has an index. Declare indexes
284+
with field tags, and Feast creates them with the collection:
285+
286+
```python
287+
Field(name="brand_id", dtype=String, tags={"milvus.index": "INVERTED"}),
288+
Field(name="price", dtype=Float64, tags={"milvus.index": "STL_SORT"}),
289+
Field(
290+
name="attributes",
291+
dtype=Json,
292+
tags={
293+
"milvus.json_indexes": '[{"json_path": "attributes[\\"price\\"]", "json_cast_type": "double"}]'
294+
},
295+
),
296+
```
297+
298+
`milvus.index` names a scalar index type the server supports, such as `INVERTED`, `BITMAP`,
299+
`STL_SORT`, `Trie` or `AUTOINDEX`, and `milvus.index_params` passes a JSON object of parameters.
300+
`milvus.json_indexes` is a JSON list of index definitions for paths of a `JSON` column, each with a
301+
`json_path`, a `json_cast_type` and an optional `index_type` that defaults to `INVERTED`. It needs
302+
native types and a Milvus server with JSON path indexes. Vector indexes come from `index_type` and
303+
`index_params` in the store config.
304+
305+
Indexes only apply when Feast creates a collection. For an existing collection without them, Feast
306+
logs a warning. To add them, run `feast teardown` and `feast apply`, then materialize again, or use
307+
the staged cutover described under [Field types](#field-types).
308+
228309
## Feature views without vectors
229310

230311
Milvus requires every collection to have a vector field. For feature views that have no vector

0 commit comments

Comments
 (0)