Skip to content

Commit 1669661

Browse files
zsetafranciscojavierarceo
zseta
and
franciscojavierarceo
authored
feat: Add ScyllaDB online store with vector search (#6508)
* feat: add ScyllaDB online store with vector search Signed-off-by: Attila Toth <[email protected]> * refactor: update integration tests to use local Docker stack Signed-off-by: Attila Toth <[email protected]> * add testcontainers that support vectors Signed-off-by: Attila Toth <[email protected]> * fix vector support and refactor Signed-off-by: Attila Toth <[email protected]> * docs update Signed-off-by: Attila Toth <[email protected]> * add support for native TTL, set local_quorum as default Signed-off-by: Attila Toth <[email protected]> * docs fixes Signed-off-by: Attila Toth <[email protected]> * add tests Signed-off-by: Attila Toth <[email protected]> * lint fix Signed-off-by: Attila Toth <[email protected]> * refactor: use one table for both vector and non-vector data Signed-off-by: Attila Toth <[email protected]> * fix lint Signed-off-by: Attila Toth <[email protected]> * fix lint Signed-off-by: Attila Toth <[email protected]> --------- Signed-off-by: Attila Toth <[email protected]> Co-authored-by: Francisco Javier Arceo <[email protected]>
1 parent 7358fb8 commit 1669661

10 files changed

Lines changed: 1489 additions & 43 deletions

File tree

‎docs/getting-started/genai.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Feast integrates with popular vector databases to store and retrieve embedding v
1515
* **Elasticsearch**: Scalable vector search capabilities
1616
* **Postgres with PGVector**: SQL-based vector operations
1717
* **Qdrant**: Purpose-built vector database integration
18+
* **ScyllaDB**: Native `vector<float, N>` type with HNSW ANN index, full `retrieve_online_documents_v2` support
1819

1920
These integrations allow you to:
2021
- Store embeddings as features

‎docs/reference/alpha-vector-database.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Below are supported vector databases and implemented features:
1515
| Faiss | [ ] | [ ] | [] | [] |
1616
| SQLite | [x] | [ ] | [x] | [x] |
1717
| Qdrant | [x] | [x] | [] | [] |
18+
| ScyllaDB | [x] | [x] | [x] | [x] |
1819

1920
*Note: V2 Support means the SDK supports retrieval of features along with vector embeddings from vector similarity search.
2021

@@ -30,7 +31,7 @@ Beyond that, we will then have `retrieve_online_documents` and `retrieve_online_
3031
backwards compatibility and the adopt industry standard naming conventions.
3132
{% endhint %}
3233

33-
**Note**: Milvus and SQLite implement the v2 `retrieve_online_documents_v2` method in the SDK. This will be the longer-term solution so that Data Scientists can easily enable vector similarity search by just flipping a flag.
34+
**Note**: Milvus, SQLite, and ScyllaDB implement the v2 `retrieve_online_documents_v2` method in the SDK. This will be the longer-term solution so that Data Scientists can easily enable vector similarity search by just flipping a flag.
3435

3536
## Examples
3637

‎docs/reference/online-stores/overview.md‎

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -29,26 +29,26 @@ See this [issue](https://github.com/feast-dev/feast/issues/2254) for a discussio
2929
## Functionality Matrix
3030

3131
There are currently five core online store implementations: `SqliteOnlineStore`, `RedisOnlineStore`, `DynamoDBOnlineStore`, `SnowflakeOnlineStore`, and `DatastoreOnlineStore`.
32-
There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore` and `CassandraOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
32+
There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, `CassandraOnlineStore` and `ScyllaDBOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
3333
Details for each specific online store, such as how to configure it in a `feature_store.yaml`, can be found [here](README.md).
3434

3535
Below is a matrix indicating which online stores support what functionality.
3636

37-
| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | [[Cassandra](https://cassandra.apache.org/_/index.html) / [Astra DB](https://www.datastax.com/products/datastax-astra?utm_source=feast)] | Milvus |
38-
| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |:----|
39-
| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
40-
| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
41-
| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
42-
| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
43-
| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes | no |
44-
| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes | yes |
45-
| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes | yes |
46-
| readable by Java | no | yes | no | no | no | no | no | no | no |
47-
| readable by Go | yes | yes | no | no | no | no | no | no | no |
48-
| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes | yes |
49-
| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no | yes |
50-
| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no | no |
51-
| support for deleting expired data | no | yes | no | no | no | no | no | no | no |
52-
| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes | no |
53-
| collocated by feature service | no | no | no | no | no | no | no | no | no |
54-
| collocated by entity key | no | yes | no | no | no | no | no | no | yes |
37+
| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | [[Cassandra](https://cassandra.apache.org/_/index.html) / [Astra DB](https://www.datastax.com/products/datastax-astra?utm_source=feast)] | Milvus | ScyllaDB |
38+
| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |:----| :-- |
39+
| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
40+
| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
41+
| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
42+
| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
43+
| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes | no | no |
44+
| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
45+
| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
46+
| readable by Java | no | yes | no | no | no | no | no | no | no | no |
47+
| readable by Go | yes | yes | no | no | no | no | no | no | no | no |
48+
| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
49+
| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no | yes | no |
50+
| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no | no | yes |
51+
| support for deleting expired data | no | yes | no | no | no | no | no | no | no | yes |
52+
| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes | no | yes |
53+
| collocated by feature service | no | no | no | no | no | no | no | no | no | no |
54+
| collocated by entity key | no | yes | no | no | no | no | no | no | yes | no |

‎docs/reference/online-stores/scylladb.md‎

Lines changed: 72 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -2,20 +2,15 @@
22

33
## Description
44

5-
ScyllaDB is a low-latency and high-performance Cassandra-compatible (uses CQL) database. You can use the existing Cassandra connector to use ScyllaDB as an online store in Feast.
6-
7-
The [ScyllaDB](https://www.scylladb.com/) online store provides support for materializing feature values into a ScyllaDB or [ScyllaDB Cloud](https://www.scylladb.com/product/scylla-cloud/) cluster for serving online features real-time.
5+
[ScyllaDB](https://www.scylladb.com/) is a distributed real-time NoSQL database with vector search support.
6+
This integration uses the native **`scylla-driver`** Python driver for optimised performance and supports materializing feature values into a [ScyllaDB Cloud](https://www.scylladb.com/product/scylla-cloud/) cluster for real-time online feature serving.
87

98
## Getting started
109

11-
Install Feast with Cassandra support:
12-
```bash
13-
pip install "feast[cassandra]"
14-
```
10+
Install Feast with the `scylladb` extra, which pulls in `scylla-driver` automatically:
1511

16-
Create a new Feast project:
1712
```bash
18-
feast init REPO_NAME -t cassandra
13+
pip install feast[scylladb]
1914
```
2015

2116
### Example (ScyllaDB)
@@ -26,7 +21,7 @@ project: scylla_feature_repo
2621
registry: data/registry.db
2722
provider: local
2823
online_store:
29-
type: cassandra
24+
type: scylladb
3025
hosts:
3126
- 172.17.0.2
3227
keyspace: feast
@@ -43,44 +38,99 @@ project: scylla_feature_repo
4338
registry: data/registry.db
4439
provider: local
4540
online_store:
46-
type: cassandra
41+
type: scylladb
4742
hosts:
4843
- node-0.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
4944
- node-1.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
5045
- node-2.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
5146
keyspace: feast
5247
username: scylla
53-
password: password
48+
password: xxxxxx
49+
local_dc: AWS_US_EAST_1
5450
```
5551
{% endcode %}
5652
57-
58-
The full set of configuration options is available in [CassandraOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.cassandra_online_store.cassandra_online_store.CassandraOnlineStoreConfig).
59-
For a full explanation of configuration options please look at file
60-
`sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md`.
53+
## Configuration options
54+
55+
| Parameter | Type | Default | Description |
56+
| :--- | :--- | :--- | :--- |
57+
| `hosts` | list[str] | *(required)* | Contact-point host addresses. |
58+
| `port` | int | `9042` | CQL port. |
59+
| `keyspace` | str | `feast_keyspace` | Target ScyllaDB keyspace. |
60+
| `username` | str | `None` | Auth username. |
61+
| `password` | str | `None` | Auth password. |
62+
| `local_dc` | str | `None` | Local datacenter name for DC-aware load balancing. |
63+
| `request_timeout` | float | `None` | Driver request timeout in seconds. |
64+
| `read_concurrency` | int | `100` | `concurrency` argument passed to the driver's `execute_concurrent_with_args` for reads. Controls how many CQL statements are in-flight at once. |
65+
| `write_concurrency` | int | `100` | `concurrency` argument passed to the driver's `execute_concurrent_with_args` for writes. Controls how many CQL statements are in-flight at once. |
66+
| `vector_similarity_function` | str | `COSINE` | Default similarity function for vector indexes. Supported: `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN`. Can be overridden per-feature via the `similarity_function` Field tag. |
6167

6268
Storage specifications can be found at `docs/specs/online_store_format.md`.
6369

70+
## Vector Search
71+
72+
ScyllaDB Cloud supports approximate nearest-neighbour (ANN) vector search.
73+
To enable it for a feature view, tag the embedding `Field` with `vector_index=true` and specify the number of dimensions:
74+
75+
{% code title="feature_definitions.py" %}
76+
```python
77+
from feast import FeatureView, Field
78+
from feast.types import Array, Float32, String
79+
80+
documents_fv = FeatureView(
81+
name="documents",
82+
entities=[item],
83+
schema=[
84+
Field(name="text", dtype=String),
85+
Field(
86+
name="embedding",
87+
dtype=Array(Float32),
88+
tags={
89+
"vector_index": "true",
90+
"dimensions": "768",
91+
"similarity_function": "COSINE", # COSINE | DOT_PRODUCT | EUCLIDEAN
92+
},
93+
),
94+
],
95+
online=True,
96+
source=push_source,
97+
)
98+
```
99+
{% endcode %}
100+
101+
When `feast apply` runs, the store automatically creates the necessary tables and HNSW ANN index for any feature view with vector-tagged fields.
102+
103+
To query the top-k most similar documents:
104+
105+
```python
106+
result = store.retrieve_online_documents_v2(
107+
features=["documents:text", "documents:embedding"],
108+
query=[0.1, 0.2, ...], # your query embedding
109+
top_k=10,
110+
distance_metric="COSINE",
111+
)
112+
```
113+
64114
## Functionality Matrix
65115

66116
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
67-
Below is a matrix indicating which functionality is supported by the Cassandra plugin.
117+
Below is a matrix indicating which functionality is supported by the ScyllaDB online store.
68118

69-
| | Cassandra |
119+
| | ScyllaDB |
70120
| :-------------------------------------------------------- | :-------- |
71121
| write feature values to the online store | yes |
72122
| read feature values from the online store | yes |
73123
| update infrastructure (e.g. tables) in the online store | yes |
74124
| teardown infrastructure (e.g. tables) in the online store | yes |
75-
| generate a plan of infrastructure changes | yes |
125+
| generate a plan of infrastructure changes | no |
76126
| support for on-demand transforms | yes |
77127
| readable by Python SDK | yes |
78128
| readable by Java | no |
79129
| readable by Go | no |
80130
| support for entityless feature views | yes |
81131
| support for concurrent writing to the same key | no |
82-
| support for ttl (time to live) at retrieval | no |
83-
| support for deleting expired data | no |
132+
| support for ttl (time to live) at retrieval | yes |
133+
| support for deleting expired data | yes |
84134
| collocated by feature view | yes |
85135
| collocated by feature service | no |
86136
| collocated by entity key | no |
@@ -89,6 +139,6 @@ To compare this set of functionality against other online stores, please see the
89139

90140
## Resources
91141

92-
* [Sample application with ScyllaDB](https://feature-store.scylladb.com/stable/)
142+
* [ScyllaDB Vector Search documentation](https://cloud.docs.scylladb.com/stable/vector-search/)
93143
* [ScyllaDB website](https://www.scylladb.com/)
94144
* [ScyllaDB Cloud documentation](https://cloud.docs.scylladb.com/stable/)

‎pyproject.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ redis = [
133133
"redis>=4.2.2,<8",
134134
"hiredis>=2.0.0,<4",
135135
]
136+
scylladb = ["scylla-driver>=3.28.0,<4"]
136137
singlestore = ["singlestoredb<1.8.0"]
137138
snowflake = [
138139
"snowflake-connector-python[pandas]>=3.7,<5",
@@ -161,7 +162,7 @@ test = [
161162
]
162163

163164
ci = [
164-
"feast[test, aws, azure, cassandra, clickhouse, couchbase, delta, docling, duckdb, elasticsearch, faiss, gcp, ge, go, grpcio, hazelcast, hbase, ibis, image, k8s, mcp, milvus, mlflow, mongodb, mssql, mysql, openlineage, opentelemetry, oracle, spark, trino, postgres, pytorch, qdrant, rag, ray, redis, singlestore, snowflake, sqlite_vec]",
165+
"feast[test, aws, azure, cassandra, clickhouse, couchbase, delta, docling, duckdb, elasticsearch, faiss, gcp, ge, go, grpcio, hazelcast, hbase, ibis, image, k8s, mcp, milvus, mlflow, mongodb, mssql, mysql, openlineage, opentelemetry, oracle, scylladb, spark, trino, postgres, pytorch, qdrant, rag, ray, redis, singlestore, snowflake, sqlite_vec]",
165166
"build",
166167
"virtualenv==20.23.0",
167168
"dbt-artifacts-parser",

‎sdk/python/feast/infra/online_stores/scylladb_online_store/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)