Skip to content

Commit 01fa2f6

Browse files
authored
Merge pull request feast-dev#2 from dmartinol/remote_offline
remote offline store example
2 parents 0d0cffd + fe4186c commit 01fa2f6

11 files changed

Lines changed: 350 additions & 12 deletions

File tree

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Feast Offline store with Arrow Flight remote server
2+
3+
This POC proposes a solution to implement the issue [Remote offline feature server deployment](https://github.com/feast-dev/feast/issues/4032)
4+
using [Arrow Flight](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) server as the remote Feast offline store.
5+
6+
## Architecture
7+
The [offline_server](./offline_server) folder contains a sample Feast repository using the `feast init` command.
8+
We can serve remote offline requests using the `feast server_offline` command that has been introduced as part of this new feature:
9+
* Spins up an `Arrow Flight` server at default port 8815.
10+
* The server accepts `do_get` requests for the `get_historical_features` command and delegates the implementation to the
11+
current `FeatureStore`.
12+
13+
The [offline_client](./offline_client) folder includes a test python function that uses an offline store of type `remote`, leveraging the remote server as the
14+
actual data provider. The offline store is implementated by the new `RemoteOfflineStore` class:
15+
* For now it implements only the `get_historical_features` method.
16+
* Since the implementation is lazy, the returned `RetrievalJob` does not run the `do_get` request until any method to synchronously execute
17+
the underlying query is invoked (e.g., `to_df` or `to_arrow`).
18+
19+
## Parameter transfer protocol
20+
The `Apache Arrow Flight` protocol defines generic APIs for efficient data streaming to and from the server, leveraging the gRPC communication framework.
21+
22+
The server exposes services to retrieve or push data streams associated with a particular `Flight` descriptor, using the `do_get` and `do_put` endpoints,
23+
but the APIs are not sufficiently detailed for the use that we have in mind, e.g. **adopt the Arrow Flight server as a generic gRPC server
24+
to serve the Feast offline APIs**.
25+
26+
Each API in the `OfflineStore` interface has multiple parameters that have to be transferred to the server to perform the actual implementation.
27+
The way we implement the parameter transfer protocol is the following:
28+
* The client, e.g. the `RemoteOfflineStore` instance, receives a call to the `get_historical_features` API with the required parameters
29+
(e.g., `entity_df` and `feature_refs`).
30+
* The client creates a unique identifier for a new command, using the UUID format, and generates a `Flight Descriptor` to represent it.
31+
* Then the client sends the received API parameters to the server using multiple calls to the `do_put` service
32+
* Each call includes the data stream of the parameter value (e.g. the `entity_df` DataFrame).
33+
* Each call also adds additional metadata values to the data schema to let the server identify the exact API to invoke:
34+
* A `command` metadata with the unique command identifier calculated before.
35+
* An `api` metadata with the name of the API to be invoked remotely, e.g. `get_historical_features`.
36+
* A `param` metadata with the name of each parameter of the API.
37+
* When the server receives the `do_put` calls, it stores the data in memory, using an ad-hoc `flights` dictionary indexed by the unique
38+
`command` identifier and storing a document with the streamed metadata values:
39+
```json
40+
{
41+
"(b'8d6a366a-c8d3-4f96-b085-f9f686111815')": {
42+
"command": "8d6a366a-c8d3-4f96-b085-f9f686111815",
43+
"api": "get_historical_features",
44+
"entity_df": ".....",
45+
"features": "...."
46+
}
47+
}
48+
```
49+
* Indexing a flight descriptor by a unique `command` identifier enables the server to efficiently handle concurrent requests from multiple clients
50+
for the same service without any overlaps.
51+
* Since the client implementation is lazy, the returned instance of `RemoteRetrievalJob` invokes the `do_get` service on the server only when
52+
the data is requested, e.g. in the `_to_arrow_internal` method.
53+
* When the server receives the `do_get` request, it unpacks the API parameters from the `flights` dictionary and, if the requested API is
54+
set to `get_historical_features`, forwards the request to the internal instance of `FeatureStore`.
55+
* Once the `do_get` request is consumed, the associated flight is removed from the `flights` dictionary, as we do not expect the same
56+
API request to be executed twice from any client.
57+
58+
Other APIs of the `OfflineStore` interface can be implemented the same way, assuming that both the client and the server implementation
59+
agree on the parameter transfer protocol to be used to let the server execute the service remotely.
60+
61+
As an alternative, APIs that do not have any returned data may be implemented as a `do_action` service in the server.
62+
63+
## Validating the POC
64+
### Launch the offline server
65+
A default Feast store has been defined in [offline_server/feature_repo](./offline_server/feature_repo/) using the `feast init` command
66+
and must be first initialized with:
67+
```console
68+
cd offline_server
69+
feast -c feature_repo apply
70+
```
71+
72+
Then the offline server can be started at the default port 8815 with:
73+
```console
74+
feast -c feature_repo serve_offline
75+
```
76+
77+
Sample output:
78+
```console
79+
Serving on grpc+tcp://127.0.0.1:8815
80+
get_historical_features for: entity_df from 0 to 2, features from driver_hourly_stats:conv_rate to transformed_conv_rate:conv_rate_plus_val2
81+
```
82+
83+
## Launch a remote offline client
84+
The test class is located under [offline_client](./offline_client/) and uses a remote configuration of the offline store to delegate the actual
85+
implementation to the offline store server:
86+
```yaml
87+
offline_store:
88+
type: remote
89+
host: localhost
90+
port: 8815
91+
```
92+
93+
The test code in [test.py](./offline_client/test.py) initializes the store from the local configuration and then fetches the historical features
94+
from the store like any other Feast client, but the actual implementation is delegated to the offline server
95+
```py
96+
store = FeatureStore(repo_path=".")
97+
training_df = store.get_historical_features(entity_df, features).to_df()
98+
```
99+
100+
Sample output of `cd offline_client; python test.py`:
101+
```console
102+
config.offline_store is <class 'feast.infra.offline_stores.remote.RemoteOfflineStoreConfig'>
103+
----- Feature schema -----
104+
105+
<class 'pandas.core.frame.DataFrame'>
106+
RangeIndex: 3 entries, 0 to 2
107+
Data columns (total 10 columns):
108+
# Column Non-Null Count Dtype
109+
--- ------ -------------- -----
110+
0 driver_id 3 non-null int64
111+
1 event_timestamp 3 non-null datetime64[ns, UTC]
112+
2 label_driver_reported_satisfaction 3 non-null int64
113+
3 val_to_add 3 non-null int64
114+
4 val_to_add_2 3 non-null int64
115+
5 conv_rate 3 non-null float32
116+
6 acc_rate 3 non-null float32
117+
7 avg_daily_trips 3 non-null int32
118+
8 conv_rate_plus_val1 3 non-null float64
119+
9 conv_rate_plus_val2 3 non-null float64
120+
dtypes: datetime64[ns, UTC](1), float32(2), float64(2), int32(1), int64(4)
121+
memory usage: 332.0 bytes
122+
None
123+
124+
----- Features -----
125+
126+
driver_id event_timestamp label_driver_reported_satisfaction ... avg_daily_trips conv_rate_plus_val1 conv_rate_plus_val2
127+
0 1001 2021-04-12 10:59:42+00:00 1 ... 590 1.022378 10.022378
128+
1 1002 2021-04-12 08:12:10+00:00 5 ... 974 2.762213 20.762213
129+
2 1003 2021-04-12 16:40:26+00:00 3 ... 127 3.419828 30.419828
130+
131+
[3 rows x 10 columns]
132+
------training_df----
133+
driver_id event_timestamp label_driver_reported_satisfaction ... avg_daily_trips conv_rate_plus_val1 conv_rate_plus_val2
134+
0 1001 2021-04-12 10:59:42+00:00 1 ... 590 1.022378 10.022378
135+
1 1002 2021-04-12 08:12:10+00:00 5 ... 974 2.762213 20.762213
136+
2 1003 2021-04-12 16:40:26+00:00 3 ... 127 3.419828 30.419828
137+
138+
[3 rows x 10 columns]
139+
```
140+

‎examples/remote-offline-store/offline_client/__init__.py‎

Whitespace-only changes.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
project: offline_server
2+
# By default, the registry is a file (but can be turned into a more scalable SQL-backed registry)
3+
registry: ../offline_server/feature_repo/data/registry.db
4+
# The provider primarily specifies default offline / online stores & storing the registry in a given cloud
5+
provider: local
6+
offline_store:
7+
type: remote
8+
host: localhost
9+
port: 8815
10+
entity_key_serialization_version: 2
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
from datetime import datetime
2+
from feast import FeatureStore
3+
import pandas as pd
4+
5+
entity_df = pd.DataFrame.from_dict(
6+
{
7+
"driver_id": [1001, 1002, 1003],
8+
"event_timestamp": [
9+
datetime(2021, 4, 12, 10, 59, 42),
10+
datetime(2021, 4, 12, 8, 12, 10),
11+
datetime(2021, 4, 12, 16, 40, 26),
12+
],
13+
"label_driver_reported_satisfaction": [1, 5, 3],
14+
"val_to_add": [1, 2, 3],
15+
"val_to_add_2": [10, 20, 30],
16+
}
17+
)
18+
19+
features = [
20+
"driver_hourly_stats:conv_rate",
21+
"driver_hourly_stats:acc_rate",
22+
"driver_hourly_stats:avg_daily_trips",
23+
"transformed_conv_rate:conv_rate_plus_val1",
24+
"transformed_conv_rate:conv_rate_plus_val2",
25+
]
26+
27+
store = FeatureStore(repo_path=".")
28+
29+
training_df = store.get_historical_features(entity_df, features).to_df()
30+
31+
print("----- Feature schema -----\n")
32+
print(training_df.info())
33+
34+
print()
35+
print("----- Features -----\n")
36+
print(training_df.head())
37+
38+
print("------training_df----")
39+
40+
print(training_df)

‎examples/remote-offline-store/offline_server/__init__.py‎

Whitespace-only changes.

‎examples/remote-offline-store/offline_server/feature_repo/__init__.py‎

Whitespace-only changes.
Binary file not shown.
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# This is an example feature definition file
2+
3+
from datetime import timedelta
4+
5+
import pandas as pd
6+
import os
7+
8+
from feast import (
9+
Entity,
10+
FeatureService,
11+
FeatureView,
12+
Field,
13+
FileSource,
14+
PushSource,
15+
RequestSource,
16+
)
17+
from feast.on_demand_feature_view import on_demand_feature_view
18+
from feast.types import Float32, Float64, Int64
19+
20+
# Define an entity for the driver. You can think of an entity as a primary key used to
21+
# fetch features.
22+
driver = Entity(name="driver", join_keys=["driver_id"])
23+
24+
# Read data from parquet files. Parquet is convenient for local development mode. For
25+
# production, you can use your favorite DWH, such as BigQuery. See Feast documentation
26+
# for more info.
27+
driver_stats_source = FileSource(
28+
name="driver_hourly_stats_source",
29+
path=f"{os.path.dirname(os.path.abspath(__file__))}/data/driver_stats.parquet",
30+
timestamp_field="event_timestamp",
31+
created_timestamp_column="created",
32+
)
33+
34+
# Our parquet files contain sample data that includes a driver_id column, timestamps and
35+
# three feature column. Here we define a Feature View that will allow us to serve this
36+
# data to our model online.
37+
driver_stats_fv = FeatureView(
38+
# The unique name of this feature view. Two feature views in a single
39+
# project cannot have the same name
40+
name="driver_hourly_stats",
41+
entities=[driver],
42+
ttl=timedelta(days=1),
43+
# The list of features defined below act as a schema to both define features
44+
# for both materialization of features into a store, and are used as references
45+
# during retrieval for building a training dataset or serving features
46+
schema=[
47+
Field(name="conv_rate", dtype=Float32),
48+
Field(name="acc_rate", dtype=Float32),
49+
Field(name="avg_daily_trips", dtype=Int64, description="Average daily trips"),
50+
],
51+
online=True,
52+
source=driver_stats_source,
53+
# Tags are user defined key/value pairs that are attached to each
54+
# feature view
55+
tags={"team": "driver_performance"},
56+
)
57+
58+
# Define a request data source which encodes features / information only
59+
# available at request time (e.g. part of the user initiated HTTP request)
60+
input_request = RequestSource(
61+
name="vals_to_add",
62+
schema=[
63+
Field(name="val_to_add", dtype=Int64),
64+
Field(name="val_to_add_2", dtype=Int64),
65+
],
66+
)
67+
68+
69+
# Define an on demand feature view which can generate new features based on
70+
# existing feature views and RequestSource features
71+
@on_demand_feature_view(
72+
sources=[driver_stats_fv, input_request],
73+
schema=[
74+
Field(name="conv_rate_plus_val1", dtype=Float64),
75+
Field(name="conv_rate_plus_val2", dtype=Float64),
76+
],
77+
)
78+
def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
79+
df = pd.DataFrame()
80+
df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"]
81+
df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"]
82+
return df
83+
84+
85+
# This groups features into a model version
86+
driver_activity_v1 = FeatureService(
87+
name="driver_activity_v1",
88+
features=[
89+
driver_stats_fv[["conv_rate"]], # Sub-selects a feature from a feature view
90+
transformed_conv_rate, # Selects all features from the feature view
91+
],
92+
)
93+
driver_activity_v2 = FeatureService(
94+
name="driver_activity_v2", features=[driver_stats_fv, transformed_conv_rate]
95+
)
96+
97+
# Defines a way to push data (to be available offline, online or both) into Feast.
98+
driver_stats_push_source = PushSource(
99+
name="driver_stats_push_source",
100+
batch_source=driver_stats_source,
101+
)
102+
103+
# Defines a slightly modified version of the feature view from above, where the source
104+
# has been changed to the push source. This allows fresh features to be directly pushed
105+
# to the online store for this feature view.
106+
driver_stats_fresh_fv = FeatureView(
107+
name="driver_hourly_stats_fresh",
108+
entities=[driver],
109+
ttl=timedelta(days=1),
110+
schema=[
111+
Field(name="conv_rate", dtype=Float32),
112+
Field(name="acc_rate", dtype=Float32),
113+
Field(name="avg_daily_trips", dtype=Int64),
114+
],
115+
online=True,
116+
source=driver_stats_push_source, # Changed from above
117+
tags={"team": "driver_performance"},
118+
)
119+
120+
121+
# Define an on demand feature view which can generate new features based on
122+
# existing feature views and RequestSource features
123+
@on_demand_feature_view(
124+
sources=[driver_stats_fresh_fv, input_request], # relies on fresh version of FV
125+
schema=[
126+
Field(name="conv_rate_plus_val1", dtype=Float64),
127+
Field(name="conv_rate_plus_val2", dtype=Float64),
128+
],
129+
)
130+
def transformed_conv_rate_fresh(inputs: pd.DataFrame) -> pd.DataFrame:
131+
df = pd.DataFrame()
132+
df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"]
133+
df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"]
134+
return df
135+
136+
137+
driver_activity_v3 = FeatureService(
138+
name="driver_activity_v3",
139+
features=[driver_stats_fresh_fv, transformed_conv_rate_fresh],
140+
)
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
project: offline_server
2+
# By default, the registry is a file (but can be turned into a more scalable SQL-backed registry)
3+
registry: data/registry.db
4+
# The provider primarily specifies default offline / online stores & storing the registry in a given cloud
5+
provider: local
6+
online_store:
7+
type: sqlite
8+
path: data/online_store.db
9+
entity_key_serialization_version: 2

‎sdk/python/feast/infra/offline_stores/remote.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55

66
import pandas as pd
77
import pyarrow as pa
8+
import pyarrow.flight as fl
89
import pyarrow.parquet
910
from pydantic import StrictInt, StrictStr
1011

@@ -41,16 +42,14 @@ def __init__(
4142
# Generate unique command identifier
4243
self.command = str(uuid.uuid4())
4344
# Initialize the client connection
44-
self.client = pa.flight.connect(
45+
self.client = fl.connect(
4546
f"grpc://{config.offline_store.host}:{config.offline_store.port}"
4647
)
4748
# Put API parameters
4849
self._put_parameters(feature_refs, entity_df)
4950

5051
def _put_parameters(self, feature_refs, entity_df):
51-
historical_flight_descriptor = pa.flight.FlightDescriptor.for_command(
52-
self.command
53-
)
52+
historical_flight_descriptor = fl.FlightDescriptor.for_command(self.command)
5453

5554
entity_df_table = pa.Table.from_pandas(entity_df)
5655
writer, _ = self.client.do_put(
@@ -89,7 +88,7 @@ def _to_df_internal(self, timeout: Optional[int] = None) -> pd.DataFrame:
8988
# Invoked to synchronously execute the underlying query and return the result as an arrow table
9089
# This is where do_get service is invoked
9190
def _to_arrow_internal(self, timeout: Optional[int] = None) -> pa.Table:
92-
upload_descriptor = pa.flight.FlightDescriptor.for_command(self.command)
91+
upload_descriptor = fl.FlightDescriptor.for_command(self.command)
9392
flight = self.client.get_flight_info(upload_descriptor)
9493
ticket = flight.endpoints[0].ticket
9594

0 commit comments

Comments
 (0)