|
| 1 | +# Apache Flink |
| 2 | + |
| 3 | +## Description |
| 4 | + |
| 5 | +The Apache Flink compute engine provides a distributed execution engine for |
| 6 | +feature pipelines through the PyFlink Table API. It implements Feast's unified |
| 7 | +`ComputeEngine` interface and can be used for batch materialization operations |
| 8 | +(`materialize` and `materialize-incremental`) and historical retrieval |
| 9 | +(`get_historical_features`). |
| 10 | + |
| 11 | +The engine reads data through the configured Feast offline store and executes |
| 12 | +the Feast DAG as PyFlink tables. Offline stores that expose a native |
| 13 | +`to_flink_table(table_env)` retrieval job hand Flink tables directly to the |
| 14 | +engine. The engine then uses Flink Table/SQL operations for join, filter, |
| 15 | +aggregate, dedupe, and projection steps, and writes materialization results to |
| 16 | +the configured online and/or offline store. |
| 17 | + |
| 18 | +## Configuration |
| 19 | + |
| 20 | +Install the Flink extra from a Feast source checkout with `uv` before using the |
| 21 | +engine: |
| 22 | + |
| 23 | +```bash |
| 24 | +uv sync --extra flink --no-dev |
| 25 | +``` |
| 26 | + |
| 27 | +The `flink` extra installs PyFlink directly. PyFlink currently requires |
| 28 | +`pyarrow<21`, while the default Feast install keeps `pyarrow>=21`; Feast's uv |
| 29 | +lock resolves the Flink extra in a separate dependency fork so normal Feast |
| 30 | +installs do not downgrade Arrow. |
| 31 | + |
| 32 | +Configure the engine in `feature_store.yaml`: |
| 33 | + |
| 34 | +```yaml |
| 35 | +project: my_project |
| 36 | +registry: data/registry.db |
| 37 | +provider: local |
| 38 | +offline_store: |
| 39 | + type: file |
| 40 | +online_store: |
| 41 | + type: sqlite |
| 42 | + path: data/online_store.db |
| 43 | +batch_engine: |
| 44 | + type: flink.engine |
| 45 | + execution_mode: batch |
| 46 | + parallelism: 4 |
| 47 | + table_config: |
| 48 | + pipeline.name: "Feast Flink Compute Engine" |
| 49 | + pandas_split_num: 4 |
| 50 | +``` |
| 51 | +
|
| 52 | +## Configuration Options |
| 53 | +
|
| 54 | +| Option | Type | Default | Description | |
| 55 | +| --- | --- | --- | --- | |
| 56 | +| `type` | string | `flink.engine` | Must be `flink.engine`. | |
| 57 | +| `execution_mode` | string | `batch` | PyFlink execution mode: `batch` or `streaming`. | |
| 58 | +| `parallelism` | integer | `null` | Default Flink parallelism for jobs created by the engine. | |
| 59 | +| `table_config` | map | `null` | Additional PyFlink table configuration entries. | |
| 60 | +| `pandas_split_num` | integer | `1` | Number of PyFlink Arrow source splits when converting pandas entity DataFrames into Flink tables. | |
| 61 | + |
| 62 | +## Flink Transformations |
| 63 | + |
| 64 | +Use `mode="flink"` when a `BatchFeatureView` transformation should receive and |
| 65 | +return PyFlink table objects: |
| 66 | + |
| 67 | +```python |
| 68 | +from feast import BatchFeatureView, Field |
| 69 | +from feast.types import Float32 |
| 70 | +
|
| 71 | +
|
| 72 | +def double_rates(table): |
| 73 | + # In production this can use PyFlink Table API operations and return a table. |
| 74 | + return table |
| 75 | +
|
| 76 | +
|
| 77 | +driver_stats = BatchFeatureView( |
| 78 | + name="driver_stats", |
| 79 | + entities=[driver], |
| 80 | + mode="flink", |
| 81 | + udf=double_rates, |
| 82 | + schema=[Field(name="conv_rate", dtype=Float32)], |
| 83 | + source=driver_stats_source, |
| 84 | + online=True, |
| 85 | +) |
| 86 | +``` |
| 87 | + |
| 88 | +Flink transformations must return PyFlink table objects. pandas-returning UDFs |
| 89 | +are not accepted by the Flink compute engine. |
| 90 | + |
| 91 | +## DAG Support |
| 92 | + |
| 93 | +The Flink engine implements Feast's compute DAG with Flink-specific nodes: |
| 94 | + |
| 95 | +- Source reads from Feast offline stores, preferring native Flink tables when a |
| 96 | + retrieval job supports `to_flink_table(table_env)`. |
| 97 | +- Transform nodes pass PyFlink tables to `mode="flink"` UDFs and preserve native |
| 98 | + Flink table outputs. |
| 99 | +- Join nodes use Flink SQL temporary views for feature joins and entity joins. |
| 100 | +- Filter nodes apply point-in-time, TTL, and custom filter expressions in Flink |
| 101 | + SQL. |
| 102 | +- Aggregate nodes support non-windowed Feast aggregations using Flink SQL |
| 103 | + aggregate functions. |
| 104 | +- Dedupe nodes use `ROW_NUMBER()` over entity keys or internal entity-row ids so |
| 105 | + historical retrieval keeps one latest feature row per entity row. |
| 106 | +- Validation nodes check required output columns. JSON value validation must be |
| 107 | + handled upstream in Flink SQL. |
| 108 | +- Output nodes write only for materialization tasks; historical retrieval is |
| 109 | + read-only. |
| 110 | +- Historical retrieval accepts pandas entity DataFrames and SQL-string entity |
| 111 | + DataFrames. SQL strings are interpreted as Flink SQL queries against the |
| 112 | + configured TableEnvironment/catalog and must select an `event_timestamp` |
| 113 | + column. |
| 114 | + |
| 115 | +## Current Limitations |
| 116 | + |
| 117 | +- Windowed aggregations are not yet implemented in the Flink compute engine. Use |
| 118 | + non-windowed Feast aggregations or pre-window upstream in Flink. |
| 119 | +- Offline store retrieval jobs must implement `to_flink_table(table_env)`. |
| 120 | + Arrow/pandas-only retrieval jobs are rejected instead of converted. |
| 121 | +- JSON value validation is not implemented inside the Flink compute engine |
| 122 | + because the engine does not collect intermediate data out of Flink for |
| 123 | + validation. |
0 commit comments