Repository navigation
Commit f7068d6
authored
feat: Add native complex types and scalar index tags to the Milvus online store (feast-dev#7120)
* fix: Round-trip every Feast type and null in the Milvus online store
With the default configuration the Milvus online store could not return
most non-numeric Feast types as written. A Bool failed to read, a
UnixTimestamp got no column so every write of it failed, an Array(String)
failed to read, a numeric array without vector_index was declared as a
vector of embedding_dim and could not be written, Bytes came back as
base64 text and Json as the base64 of its protobuf. A missing value was
stored as the column's default, which failed to read for a numeric text
column and came back as 0 or false for a native one. Search hits decoded
only primitive types, and entity columns were written as the text of
their Python value.
The write path now stores every scalar natively or as text, bytes as
base64 and every other value as the base64 of its ValueProto, for
features and entity columns alike, and one helper rebuilds the ValueProto
for reads and searches from every declared field type. UnixTimestamp
gets a column, only a field with vector_index=True becomes a vector, with
the dimension from Field.vector_length before embedding_dim, and an
empty text in a numeric or timestamp column reads as null. Array(Bool)
with vector_index is rejected, because Milvus binary vectors were written
unpacked and indexed with a float metric and could not work. The new
milvus.nullable field tag makes a column nullable, so a missing value is
stored and read as null.
Signed-off-by: jiwidi <[email protected]>
* feat: Add native complex types and scalar index tags to the Milvus online store
With enable_openai_compatible_store the Milvus online store stored
numeric features as native columns, but arrays and Json values were
still base64 text, so a filter could not reach them and Milvus could not
validate them.
With the flag, an array of scalars without vector_index is now an ARRAY
column of the element type, with milvus.max_capacity elements at most,
and a Json value, or an array of them, is a JSON column. Both columns
are nullable unless the field says otherwise, so a missing value stays
null. A Json value must be an object or an array, because PyMilvus
cannot store a top-level string, number or bool in a JSON column, and
JSON null is stored as a null column value. Map and Struct values stay
serialized, because JSON cannot hold their nested bytes, timestamps or
empty typed lists. The column types of an existing collection decide
how a row is written and read, so collections created before this
change keep working. Filter keys can address a JSON path or an array
element, for example attributes["price"] or tags[0].
Field tags declare scalar indexes, which Feast creates with the
collection: milvus.index names the index type of a scalar column,
milvus.index_params passes its parameters and milvus.json_indexes lists
JSON path indexes of a JSON column. For an existing collection without
them Feast logs a warning, as it does for a missing partition key.
Signed-off-by: jiwidi <[email protected]>
---------
Signed-off-by: jiwidi <[email protected]>1 parent bbaca76 commit f7068d6
4 files changed
Lines changed: 1238 additions & 237 deletions
File tree
- docs/reference/online-stores
- sdk/python
- feast/infra/online_stores/milvus_online_store
- tests
- integration/online_store
- unit/infra/online_store
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
104 | | - | |
| 104 | + | |
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
| |||
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
228 | 309 | | |
229 | 310 | | |
230 | 311 | | |
| |||
0 commit comments