Skip to content

Commit b05d50b

Browse files
mehmettokgozmehmettokgoz
authored
feat: Add Hazelcast as an online store (#3523)
* feat: Add Hazelcast as an online store Signed-off-by: mehmettokgoz <[email protected]> * Fix template bugs and address review comments. Signed-off-by: mehmettokgoz <[email protected]> --------- Signed-off-by: mehmettokgoz <[email protected]>
1 parent 8dd7217 commit b05d50b

23 files changed

Lines changed: 1073 additions & 0 deletions

File tree

‎Makefile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,27 @@ test-python-universal-cassandra:
259259
python -m pytest -x --integration \
260260
sdk/python/tests
261261

262+
test-python-universal-hazelcast:
263+
PYTHONPATH='.' \
264+
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.hazelcast_repo_configuration \
265+
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.hazelcast \
266+
FEAST_USAGE=False \
267+
IS_TEST=True \
268+
python -m pytest -n 8 --integration \
269+
-k "not test_universal_cli and \
270+
not test_go_feature_server and \
271+
not test_feature_logging and \
272+
not test_reorder_columns and \
273+
not test_logged_features_validation and \
274+
not test_lambda_materialization_consistency and \
275+
not test_offline_write and \
276+
not test_push_features_to_offline_store and \
277+
not gcs_registry and \
278+
not s3_registry and \
279+
not test_universal_types and \
280+
not test_snowflake" \
281+
sdk/python/tests
282+
262283
test-python-universal-cassandra-no-cloud-providers:
263284
PYTHONPATH='.' \
264285
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@
9797
* [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md)
9898
* [MySQL (contrib)](reference/online-stores/mysql.md)
9999
* [Rockset (contrib)](reference/online-stores/rockset.md)
100+
* [Hazelcast (contrib)](reference/online-stores/hazelcast.md)
100101
* [Providers](reference/providers/README.md)
101102
* [Local](reference/providers/local.md)
102103
* [Google Cloud Platform](reference/providers/google-cloud-platform.md)

‎docs/reference/online-stores/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,8 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
4646
[rockset.md](rockset.md)
4747
{% endcontent-ref %}
4848

49+
{% content-ref url="hazelcast.md" %}
50+
[hazelcast.md](hazelcast.md)
51+
{% endcontent-ref %}
52+
53+

‎docs/reference/online-stores/hazelcast.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Hazelcast online store
2+
3+
## Description
4+
5+
Hazelcast online store is in alpha development.
6+
7+
The [Hazelcast](htpps://hazelcast.com) online store provides support for materializing feature values into a Hazelcast cluster for serving online features in real-time.
8+
In order to use Hazelcast as online store, you need to have a running Hazelcast cluster. You can create a cluster using Hazelcast Viridian Serverless. See this [getting started](https://hazelcast.com/get-started/) page for more details.
9+
10+
* Each feature view is mapped one-to-one to a specific Hazelcast IMap
11+
* This implementation inherits all strengths of Hazelcast such as high availability, fault-tolerance, and data distribution.
12+
* Secure TSL/SSL connection is supported by Hazelcast online store.
13+
* You can set TTL (Time-To-Live) setting for your features in Hazelcast cluster.
14+
15+
Each feature view corresponds to an IMap in Hazelcast cluster and the entries in that IMap corresponds to features of entities.
16+
Each feature value stored separately and can be retrieved individually.
17+
18+
## Getting started
19+
20+
In order to use Hazelcast online store, you'll need to run `pip install 'feast[hazelcast]'`. You can then get started with the command `feast init REPO_NAME -t hazelcast`.
21+
22+
23+
## Examples
24+
25+
{% code title="feature_store.yaml" %}
26+
```yaml
27+
project: my_feature_repo
28+
registry: data/registry.db
29+
provider: local
30+
online_store:
31+
type: hazelcast
32+
cluster_name: dev
33+
cluster_members: ["localhost:5701"]
34+
key_ttl_seconds: 36000
35+
```
36+
37+
## Functionality Matrix
38+
39+
| | Hazelcast |
40+
| :-------------------------------------------------------- |:----------|
41+
| write feature values to the online store | yes |
42+
| read feature values from the online store | yes |
43+
| update infrastructure (e.g. tables) in the online store | yes |
44+
| teardown infrastructure (e.g. tables) in the online store | yes |
45+
| generate a plan of infrastructure changes | no |
46+
| support for on-demand transforms | yes |
47+
| readable by Python SDK | yes |
48+
| readable by Java | no |
49+
| readable by Go | no |
50+
| support for entityless feature views | yes |
51+
| support for concurrent writing to the same key | yes |
52+
| support for ttl (time to live) at retrieval | yes |
53+
| support for deleting expired data | yes |
54+
| collocated by feature view | no |
55+
| collocated by feature service | no |
56+
| collocated by entity key | yes |
57+
58+
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
59+

‎sdk/python/docs/source/feast.infra.online_stores.contrib.hazelcast_online_store.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
feast.infra.online\_stores.contrib.hazelcast\_online\_store package
2+
===================================================================
3+
4+
Submodules
5+
----------
6+
7+
feast.infra.online\_stores.contrib.hazelcast\_online\_store.hazelcast\_online\_store module
8+
-------------------------------------------------------------------------------------------
9+
10+
.. automodule:: feast.infra.online_stores.contrib.hazelcast_online_store.hazelcast_online_store
11+
:members:
12+
:undoc-members:
13+
:show-inheritance:
14+
15+
Module contents
16+
---------------
17+
18+
.. automodule:: feast.infra.online_stores.contrib.hazelcast_online_store
19+
:members:
20+
:undoc-members:
21+
:show-inheritance:

‎sdk/python/docs/source/feast.infra.online_stores.contrib.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Subpackages
88
:maxdepth: 4
99

1010
feast.infra.online_stores.contrib.cassandra_online_store
11+
feast.infra.online_stores.contrib.hazelcast_online_store
1112
feast.infra.online_stores.contrib.hbase_online_store
1213
feast.infra.online_stores.contrib.mysql_online_store
1314
feast.infra.online_stores.contrib.rockset_online_store
@@ -23,6 +24,14 @@ feast.infra.online\_stores.contrib.cassandra\_repo\_configuration module
2324
:undoc-members:
2425
:show-inheritance:
2526

27+
feast.infra.online\_stores.contrib.hazelcast\_repo\_configuration module
28+
------------------------------------------------------------------------
29+
30+
.. automodule:: feast.infra.online_stores.contrib.hazelcast_repo_configuration
31+
:members:
32+
:undoc-members:
33+
:show-inheritance:
34+
2635
feast.infra.online\_stores.contrib.hbase\_repo\_configuration module
2736
--------------------------------------------------------------------
2837

‎sdk/python/feast/cli.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -641,6 +641,7 @@ def materialize_incremental_command(ctx: click.Context, end_ts: str, views: List
641641
"hbase",
642642
"cassandra",
643643
"rockset",
644+
"hazelcast",
644645
],
645646
case_sensitive=False,
646647
),

‎sdk/python/feast/infra/online_stores/contrib/hazelcast_online_store/README.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Hazelcast Online Store
2+
3+
This contribution makes it possible to use [Hazelcast](https://hazelcast.com/) as online store for Feast.
4+
5+
Once the Hazelcast client configuration is given inside `feature_store.yaml` file, everything else
6+
is handled as with any other online store: schema creation, read/write from/to Hazelcast and remove operations.
7+
8+
## Quick usage
9+
10+
The following refers to the [Feast quickstart](https://docs.feast.dev/getting-started/quickstart) page.
11+
Only the Step 2 is different from this tutorial since it requires you to configure your Hazelcast online store.
12+
13+
### Creating the feature repository
14+
15+
The easiest way to get started is to use the Feast CLI to initialize a new
16+
feature store. Once Feast is installed, the command
17+
18+
```
19+
feast init FEATURE_STORE_NAME -t hazelcast
20+
```
21+
22+
will interactively help you create the `feature_store.yaml` with the
23+
required configuration details to access your Hazelcast cluster.
24+
25+
Alternatively, you can run `feast init -t FEATURE_STORE_NAME`, as described
26+
in the quickstart, and then manually edit the `online_store` section in
27+
the `feature_store.yaml` file as detailed below.
28+
29+
The following steps (setup of feature definitions, deployment of the store,
30+
generation of training data, materialization, fetching of online/offline
31+
features) proceed exactly as in the general Feast quickstart instructions.
32+
33+
#### Hazelcast setup
34+
35+
In order to use [Hazelcast](https://hazelcast.com) as online store, you need to have a running Hazelcast cluster.
36+
You can create a cluster using Hazelcast Viridian Serverless easily or deploy one on your local/remote machine.
37+
See this [getting started](https://hazelcast.com/get-started/) page for more details.
38+
39+
Hazelcast online store provides capability to connect local/remote or Hazelcast Viridian Serverless cluster.
40+
Following is an example to connect local cluster named "dev" running on port 5701 with TLS/SSL enabled.
41+
42+
```yaml
43+
[...]
44+
online_store:
45+
type: hazelcast
46+
cluster_name: dev
47+
cluster_members: ["localhost:5701"]
48+
ssl_cafile_path: /path/to/ca/file
49+
ssl_certfile_path: /path/to/cert/file
50+
ssl_keyfile_path: /path/to/key/file
51+
ssl_password: ${SSL_PASSWORD} # The password will be read form the `SSL_PASSWORD` environment variable.
52+
key_ttl_seconds: 86400 # The default is 0 and means infinite.
53+
```
54+
55+
If you want to connect your Hazelcast Viridian cluster instead of local/remote one, specify your configuration as follows:
56+
57+
```yaml
58+
[...]
59+
online_store:
60+
type: hazelcast
61+
cluster_name: YOUR_CLUSTER_ID
62+
discovery_token: YOUR_DISCOVERY_TOKEN
63+
ssl_cafile_path: /path/to/ca/file
64+
ssl_certfile_path: /path/to/cert/file
65+
ssl_keyfile_path: /path/to/key/file
66+
ssl_password: ${SSL_PASSWORD} # The password will be read form the `SSL_PASSWORD` environment variable.
67+
key_ttl_seconds: 86400 # The default is 0 and means infinite.
68+
```
69+
70+
#### TTL configuration
71+
72+
TTL is the maximum time in seconds for each feature to stay idle in the map.
73+
It limits the lifetime of the features relative to the time of the last read or write access performed on them.
74+
The features whose idle period exceeds this limit are expired and evicted automatically.
75+
A feature is idle if no get or put is called on it.
76+
Valid values are integers between 0 and Integer.MAX_VALUE.
77+
Its default value is 0, which means infinite.
78+
79+
```yaml
80+
[...]
81+
online_store:
82+
[...]
83+
key_ttl_seconds: 86400
84+
```
85+
86+
### More info
87+
88+
You can learn about Hazelcast more from the [Hazelcast Documentation](https://docs.hazelcast.com/home/).
89+

‎sdk/python/feast/infra/online_stores/contrib/hazelcast_online_store/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)