|
| 1 | +# Feast Offline store with Arrow Flight remote server |
| 2 | + |
| 3 | +This POC proposes a solution to implement the issue [Remote offline feature server deployment](https://github.com/feast-dev/feast/issues/4032) |
| 4 | +using [Arrow Flight](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) server as the remote Feast offline store. |
| 5 | + |
| 6 | +## Architecture |
| 7 | +The [offline_server](./offline_server) folder contains a sample Feast repository using the `feast init` command. |
| 8 | +We can serve remote offline requests using the `feast server_offline` command that has been introduced as part of this new feature: |
| 9 | +* Spins up an `Arrow Flight` server at default port 8815. |
| 10 | +* The server accepts `do_get` requests for the `get_historical_features` command and delegates the implementation to the |
| 11 | +current `FeatureStore`. |
| 12 | + |
| 13 | +The [offline_client](./offline_client) folder includes a test python function that uses an offline store of type `remote`, leveraging the remote server as the |
| 14 | +actual data provider. The offline store is implementated by the new `RemoteOfflineStore` class: |
| 15 | +* For now it implements only the `get_historical_features` method. |
| 16 | +* Since the implementation is lazy, the returned `RetrievalJob` does not run the `do_get` request until any method to synchronously execute |
| 17 | +the underlying query is invoked (e.g., `to_df` or `to_arrow`). |
| 18 | + |
| 19 | +## Parameter transfer protocol |
| 20 | +The `Apache Arrow Flight` protocol defines generic APIs for efficient data streaming to and from the server, leveraging the gRPC communication framework. |
| 21 | + |
| 22 | +The server exposes services to retrieve or push data streams associated with a particular `Flight` descriptor, using the `do_get` and `do_put` endpoints, |
| 23 | +but the APIs are not sufficiently detailed for the use that we have in mind, e.g. **adopt the Arrow Flight server as a generic gRPC server |
| 24 | +to serve the Feast offline APIs**. |
| 25 | + |
| 26 | +Each API in the `OfflineStore` interface has multiple parameters that have to be transferred to the server to perform the actual implementation. |
| 27 | +The way we implement the parameter transfer protocol is the following: |
| 28 | +* The client, e.g. the `RemoteOfflineStore` instance, receives a call to the `get_historical_features` API with the required parameters |
| 29 | +(e.g., `entity_df` and `feature_refs`). |
| 30 | +* The client creates a unique identifier for a new command, using the UUID format, and generates a `Flight Descriptor` to represent it. |
| 31 | +* Then the client sends the received API parameters to the server using multiple calls to the `do_put` service |
| 32 | + * Each call includes the data stream of the parameter value (e.g. the `entity_df` DataFrame). |
| 33 | + * Each call also adds additional metadata values to the data schema to let the server identify the exact API to invoke: |
| 34 | + * A `command` metadata with the unique command identifier calculated before. |
| 35 | + * An `api` metadata with the name of the API to be invoked remotely, e.g. `get_historical_features`. |
| 36 | + * A `param` metadata with the name of each parameter of the API. |
| 37 | +* When the server receives the `do_put` calls, it stores the data in memory, using an ad-hoc `flights` dictionary indexed by the unique |
| 38 | +`command` identifier and storing a document with the streamed metadata values: |
| 39 | +```json |
| 40 | +{ |
| 41 | + "(b'8d6a366a-c8d3-4f96-b085-f9f686111815')": { |
| 42 | + "command": "8d6a366a-c8d3-4f96-b085-f9f686111815", |
| 43 | + "api": "get_historical_features", |
| 44 | + "entity_df": ".....", |
| 45 | + "features": "...." |
| 46 | + } |
| 47 | +} |
| 48 | +``` |
| 49 | +* Indexing a flight descriptor by a unique `command` identifier enables the server to efficiently handle concurrent requests from multiple clients |
| 50 | +for the same service without any overlaps. |
| 51 | +* Since the client implementation is lazy, the returned instance of `RemoteRetrievalJob` invokes the `do_get` service on the server only when |
| 52 | +the data is requested, e.g. in the `_to_arrow_internal` method. |
| 53 | +* When the server receives the `do_get` request, it unpacks the API parameters from the `flights` dictionary and, if the requested API is |
| 54 | +set to `get_historical_features`, forwards the request to the internal instance of `FeatureStore`. |
| 55 | +* Once the `do_get` request is consumed, the associated flight is removed from the `flights` dictionary, as we do not expect the same |
| 56 | +API request to be executed twice from any client. |
| 57 | + |
| 58 | +Other APIs of the `OfflineStore` interface can be implemented the same way, assuming that both the client and the server implementation |
| 59 | +agree on the parameter transfer protocol to be used to let the server execute the service remotely. |
| 60 | + |
| 61 | +As an alternative, APIs that do not have any returned data may be implemented as a `do_action` service in the server. |
| 62 | + |
| 63 | +## Validating the POC |
| 64 | +### Launch the offline server |
| 65 | +A default Feast store has been defined in [offline_server/feature_repo](./offline_server/feature_repo/) using the `feast init` command |
| 66 | +and must be first initialized with: |
| 67 | +```console |
| 68 | +cd offline_server |
| 69 | +feast -c feature_repo apply |
| 70 | +``` |
| 71 | + |
| 72 | +Then the offline server can be started at the default port 8815 with: |
| 73 | +```console |
| 74 | +feast -c feature_repo serve_offline |
| 75 | +``` |
| 76 | + |
| 77 | +Sample output: |
| 78 | +```console |
| 79 | +Serving on grpc+tcp://127.0.0.1:8815 |
| 80 | +get_historical_features for: entity_df from 0 to 2, features from driver_hourly_stats:conv_rate to transformed_conv_rate:conv_rate_plus_val2 |
| 81 | +``` |
| 82 | + |
| 83 | +## Launch a remote offline client |
| 84 | +The test class is located under [offline_client](./offline_client/) and uses a remote configuration of the offline store to delegate the actual |
| 85 | +implementation to the offline store server: |
| 86 | +```yaml |
| 87 | +offline_store: |
| 88 | + type: remote |
| 89 | + host: localhost |
| 90 | + port: 8815 |
| 91 | +``` |
| 92 | +
|
| 93 | +The test code in [test.py](./offline_client/test.py) initializes the store from the local configuration and then fetches the historical features |
| 94 | +from the store like any other Feast client, but the actual implementation is delegated to the offline server |
| 95 | +```py |
| 96 | +store = FeatureStore(repo_path=".") |
| 97 | +training_df = store.get_historical_features(entity_df, features).to_df() |
| 98 | +``` |
| 99 | + |
| 100 | +Sample output of `cd offline_client; python test.py`: |
| 101 | +```console |
| 102 | +config.offline_store is <class 'feast.infra.offline_stores.remote.RemoteOfflineStoreConfig'> |
| 103 | +----- Feature schema ----- |
| 104 | + |
| 105 | +<class 'pandas.core.frame.DataFrame'> |
| 106 | +RangeIndex: 3 entries, 0 to 2 |
| 107 | +Data columns (total 10 columns): |
| 108 | + # Column Non-Null Count Dtype |
| 109 | +--- ------ -------------- ----- |
| 110 | + 0 driver_id 3 non-null int64 |
| 111 | + 1 event_timestamp 3 non-null datetime64[ns, UTC] |
| 112 | + 2 label_driver_reported_satisfaction 3 non-null int64 |
| 113 | + 3 val_to_add 3 non-null int64 |
| 114 | + 4 val_to_add_2 3 non-null int64 |
| 115 | + 5 conv_rate 3 non-null float32 |
| 116 | + 6 acc_rate 3 non-null float32 |
| 117 | + 7 avg_daily_trips 3 non-null int32 |
| 118 | + 8 conv_rate_plus_val1 3 non-null float64 |
| 119 | + 9 conv_rate_plus_val2 3 non-null float64 |
| 120 | +dtypes: datetime64[ns, UTC](1), float32(2), float64(2), int32(1), int64(4) |
| 121 | +memory usage: 332.0 bytes |
| 122 | +None |
| 123 | + |
| 124 | +----- Features ----- |
| 125 | + |
| 126 | + driver_id event_timestamp label_driver_reported_satisfaction ... avg_daily_trips conv_rate_plus_val1 conv_rate_plus_val2 |
| 127 | +0 1001 2021-04-12 10:59:42+00:00 1 ... 590 1.022378 10.022378 |
| 128 | +1 1002 2021-04-12 08:12:10+00:00 5 ... 974 2.762213 20.762213 |
| 129 | +2 1003 2021-04-12 16:40:26+00:00 3 ... 127 3.419828 30.419828 |
| 130 | + |
| 131 | +[3 rows x 10 columns] |
| 132 | +------training_df---- |
| 133 | + driver_id event_timestamp label_driver_reported_satisfaction ... avg_daily_trips conv_rate_plus_val1 conv_rate_plus_val2 |
| 134 | +0 1001 2021-04-12 10:59:42+00:00 1 ... 590 1.022378 10.022378 |
| 135 | +1 1002 2021-04-12 08:12:10+00:00 5 ... 974 2.762213 20.762213 |
| 136 | +2 1003 2021-04-12 16:40:26+00:00 3 ... 127 3.419828 30.419828 |
| 137 | + |
| 138 | +[3 rows x 10 columns] |
| 139 | +``` |
| 140 | + |
0 commit comments