Skip to content

Commit 9636d6a

Browse files
authored
feat: Add apache flink compute engine (feast-dev#6476)
* feat: Add apache flink compute engine Signed-off-by: Le Xuan An <[email protected]> * fix: Address flink review comments Signed-off-by: Le Xuan An <[email protected]> * fix: Address flink copilot review comments Signed-off-by: Le Xuan An <[email protected]> * fix: Refresh pixi lock for flink extra Signed-off-by: Le Xuan An <[email protected]> * fix: Satisfy flink feature builder typing Signed-off-by: Le Xuan An <[email protected]> * refactor: Share compute engine timestamp helpers Signed-off-by: Le Xuan An <[email protected]> * test: Cover flink transformation mode Signed-off-by: Le Xuan An <[email protected]> * test: Stabilize feature server performance check Signed-off-by: Le Xuan An <[email protected]> * test: Mock image extractor model loading Signed-off-by: Le Xuan An <[email protected]> * test: Use localhost for remote offline clients Signed-off-by: Le Xuan An <[email protected]> * test: Stabilize local Milvus integration index Signed-off-by: Le Xuan An <[email protected]> * test: Stabilize feature server benchmark timing Signed-off-by: Le Xuan An <[email protected]> * test: Clean up remote offline server processes Signed-off-by: Le Xuan An <[email protected]> * test: Revert unrelated universal test changes Signed-off-by: Le Xuan An <[email protected]> * fix: Address Flink review blockers Signed-off-by: Le Xuan An <[email protected]> --------- Signed-off-by: Le Xuan An <[email protected]>
1 parent 7d1ef73 commit 9636d6a

31 files changed

Lines changed: 3109 additions & 41 deletions

File tree

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,7 @@
177177
* [Snowflake](reference/compute-engine/snowflake.md)
178178
* [AWS Lambda (alpha)](reference/compute-engine/lambda.md)
179179
* [Spark (contrib)](reference/compute-engine/spark.md)
180+
* [Apache Flink](reference/compute-engine/flink.md)
180181
* [Ray (contrib)](reference/compute-engine/ray.md)
181182
* [Feature repository](reference/feature-repository/README.md)
182183
* [feature\_store.yaml](reference/feature-repository/feature-store-yaml.md)

‎docs/getting-started/components/compute-engine.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ engines.
2424
| SparkComputeEngine | Runs on Apache Spark, designed for large-scale distributed feature generation. | ✅ | |
2525
| SnowflakeComputeEngine | Runs on Snowflake, designed for scalable feature generation using Snowflake SQL. | ✅ | |
2626
| LambdaComputeEngine | Runs on AWS Lambda, designed for serverless feature generation. | ✅ | |
27-
| FlinkComputeEngine | Runs on Apache Flink, designed for stream processing and real-time feature generation. | ❌ | |
27+
| FlinkComputeEngine | Runs on Apache Flink, designed for distributed feature generation through PyFlink Table API. | ✅ | |
2828
| RayComputeEngine | Runs on Ray, designed for distributed feature generation and machine learning workloads. | ✅ | |
2929
```
3030

@@ -156,4 +156,4 @@ DAG nodes are defined as follows:
156156
+----------------+ +----------------+
157157
| OnlineStoreWrite| OfflineStoreWrite|
158158
+----------------+ +----------------+
159-
```
159+
```

‎docs/reference/compute-engine/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,14 @@ An example of built output from FeatureBuilder:
5757
- Supports point-in-time joins and large-scale materialization
5858
- Integrates with `SparkOfflineStore` and `SparkMaterializationJob`
5959

60+
### 🌊 FlinkComputeEngine
61+
62+
{% page-ref page="flink.md" %}
63+
64+
- Distributed DAG execution through Apache Flink's PyFlink Table API
65+
- Supports materialization and historical retrieval with Feast offline stores
66+
- Integrates with `FlinkMaterializationJob` and `FlinkDAGRetrievalJob`
67+
6068
### ⚡ RayComputeEngine (contrib)
6169

6270
- Distributed DAG execution via Ray
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Apache Flink
2+
3+
## Description
4+
5+
The Apache Flink compute engine provides a distributed execution engine for
6+
feature pipelines through the PyFlink Table API. It implements Feast's unified
7+
`ComputeEngine` interface and can be used for batch materialization operations
8+
(`materialize` and `materialize-incremental`) and historical retrieval
9+
(`get_historical_features`).
10+
11+
The engine reads data through the configured Feast offline store and executes
12+
the Feast DAG as PyFlink tables. Offline stores that expose a native
13+
`to_flink_table(table_env)` retrieval job hand Flink tables directly to the
14+
engine. The engine then uses Flink Table/SQL operations for join, filter,
15+
aggregate, dedupe, and projection steps, and writes materialization results to
16+
the configured online and/or offline store.
17+
18+
## Configuration
19+
20+
Install the Flink extra from a Feast source checkout with `uv` before using the
21+
engine:
22+
23+
```bash
24+
uv sync --extra flink --no-dev
25+
```
26+
27+
The `flink` extra installs PyFlink directly. PyFlink currently requires
28+
`pyarrow<21`, while the default Feast install keeps `pyarrow>=21`; Feast's uv
29+
lock resolves the Flink extra in a separate dependency fork so normal Feast
30+
installs do not downgrade Arrow.
31+
32+
Configure the engine in `feature_store.yaml`:
33+
34+
```yaml
35+
project: my_project
36+
registry: data/registry.db
37+
provider: local
38+
offline_store:
39+
type: file
40+
online_store:
41+
type: sqlite
42+
path: data/online_store.db
43+
batch_engine:
44+
type: flink.engine
45+
execution_mode: batch
46+
parallelism: 4
47+
table_config:
48+
pipeline.name: "Feast Flink Compute Engine"
49+
pandas_split_num: 4
50+
```
51+
52+
## Configuration Options
53+
54+
| Option | Type | Default | Description |
55+
| --- | --- | --- | --- |
56+
| `type` | string | `flink.engine` | Must be `flink.engine`. |
57+
| `execution_mode` | string | `batch` | PyFlink execution mode: `batch` or `streaming`. |
58+
| `parallelism` | integer | `null` | Default Flink parallelism for jobs created by the engine. |
59+
| `table_config` | map | `null` | Additional PyFlink table configuration entries. |
60+
| `pandas_split_num` | integer | `1` | Number of PyFlink Arrow source splits when converting pandas entity DataFrames into Flink tables. |
61+
62+
## Flink Transformations
63+
64+
Use `mode="flink"` when a `BatchFeatureView` transformation should receive and
65+
return PyFlink table objects:
66+
67+
```python
68+
from feast import BatchFeatureView, Field
69+
from feast.types import Float32
70+
71+
72+
def double_rates(table):
73+
# In production this can use PyFlink Table API operations and return a table.
74+
return table
75+
76+
77+
driver_stats = BatchFeatureView(
78+
name="driver_stats",
79+
entities=[driver],
80+
mode="flink",
81+
udf=double_rates,
82+
schema=[Field(name="conv_rate", dtype=Float32)],
83+
source=driver_stats_source,
84+
online=True,
85+
)
86+
```
87+
88+
Flink transformations must return PyFlink table objects. pandas-returning UDFs
89+
are not accepted by the Flink compute engine.
90+
91+
## DAG Support
92+
93+
The Flink engine implements Feast's compute DAG with Flink-specific nodes:
94+
95+
- Source reads from Feast offline stores, preferring native Flink tables when a
96+
retrieval job supports `to_flink_table(table_env)`.
97+
- Transform nodes pass PyFlink tables to `mode="flink"` UDFs and preserve native
98+
Flink table outputs.
99+
- Join nodes use Flink SQL temporary views for feature joins and entity joins.
100+
- Filter nodes apply point-in-time, TTL, and custom filter expressions in Flink
101+
SQL.
102+
- Aggregate nodes support non-windowed Feast aggregations using Flink SQL
103+
aggregate functions.
104+
- Dedupe nodes use `ROW_NUMBER()` over entity keys or internal entity-row ids so
105+
historical retrieval keeps one latest feature row per entity row.
106+
- Validation nodes check required output columns. JSON value validation must be
107+
handled upstream in Flink SQL.
108+
- Output nodes write only for materialization tasks; historical retrieval is
109+
read-only.
110+
- Historical retrieval accepts pandas entity DataFrames and SQL-string entity
111+
DataFrames. SQL strings are interpreted as Flink SQL queries against the
112+
configured TableEnvironment/catalog and must select an `event_timestamp`
113+
column.
114+
115+
## Current Limitations
116+
117+
- Windowed aggregations are not yet implemented in the Flink compute engine. Use
118+
non-windowed Feast aggregations or pre-window upstream in Flink.
119+
- Offline store retrieval jobs must implement `to_flink_table(table_env)`.
120+
Arrow/pandas-only retrieval jobs are rejected instead of converted.
121+
- JSON value validation is not implemented inside the Flink compute engine
122+
because the engine does not collect intermediate data out of Flink for
123+
validation.

‎pixi.lock‎

Lines changed: 5 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,8 @@ dependencies = [
2323
"mmh3",
2424
"numpy>=2.0.0,<3",
2525
"pandas>=1.4.3,<3",
26-
"pyarrow>=21.0.0",
26+
"pyarrow>=21.0.0; extra != 'flink'",
27+
"pyarrow>=16.1.0,<21.0.0; extra == 'flink'",
2728
"pydantic>=2.10.6",
2829
"pygments>=2.12.0,<3",
2930
"PyYAML>=5.4.0,<7",
@@ -63,6 +64,7 @@ docling = ["docling==2.27.0"]
6364
duckdb = ["ibis-framework[duckdb]>=10.0.0"]
6465
elasticsearch = ["elasticsearch>=8.13.0"]
6566
faiss = ["faiss-cpu>=1.7.0,<=1.10.0"]
67+
flink = ["apache-flink>=2.2.1,<3"]
6668
gcp = [
6769
"google-api-core>=1.23.0,<3",
6870
"googleapis-common-protos>=1.52.0,<2",
@@ -278,6 +280,26 @@ dev = [
278280
"pytest-xdist>=3.8.0",
279281
]
280282

283+
[tool.uv]
284+
conflicts = [
285+
[
286+
{ extra = "flink" },
287+
{ extra = "ge" },
288+
],
289+
[
290+
{ extra = "flink" },
291+
{ extra = "ci" },
292+
],
293+
[
294+
{ extra = "flink" },
295+
{ extra = "dev" },
296+
],
297+
[
298+
{ extra = "flink" },
299+
{ extra = "docs" },
300+
],
301+
]
302+
281303
# Pixi configuration
282304
[tool.pixi.workspace]
283305
channels = ["conda-forge"]

‎sdk/python/feast/batch_feature_view.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -169,13 +169,14 @@ def get_feature_transformation(self) -> Optional[Transformation]:
169169
TransformationMode.PYTHON,
170170
TransformationMode.SQL,
171171
TransformationMode.RAY,
172-
) or self.mode in ("pandas", "python", "sql", "ray"):
172+
TransformationMode.FLINK,
173+
) or self.mode in ("pandas", "python", "sql", "ray", "flink"):
173174
return Transformation(
174175
mode=self.mode, udf=self.udf, udf_string=self.udf_string or ""
175176
)
176177
else:
177178
raise ValueError(
178-
f"Unsupported transformation mode: {self.mode} for StreamFeatureView"
179+
f"Unsupported transformation mode: {self.mode} for BatchFeatureView"
179180
)
180181

181182

‎sdk/python/feast/infra/compute_engines/dag/model.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ class DAGFormat(str, Enum):
66
PANDAS = "pandas"
77
ARROW = "arrow"
88
RAY = "ray"
9+
FLINK = "flink"

‎sdk/python/feast/infra/compute_engines/feature_builder.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,10 +158,13 @@ def get_column_info(
158158
# we need to read ALL source columns, not just the output feature columns.
159159
# This is specifically for transformations that create new columns or need raw data.
160160
mode = getattr(getattr(view, "feature_transformation", None), "mode", None)
161-
if mode in ("ray", "pandas", "python") or getattr(mode, "value", None) in (
161+
if mode in ("ray", "pandas", "python", "flink") or getattr(
162+
mode, "value", None
163+
) in (
162164
"ray",
163165
"pandas",
164166
"python",
167+
"flink",
165168
):
166169
# Signal to read all columns by passing empty list for feature_cols.
167170
# "python" (BatchFeatureView) transformations need all raw source columns — the
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
from __future__ import annotations
2+
3+
from feast.infra.compute_engines.flink.compute import (
4+
FlinkComputeEngine,
5+
FlinkComputeEngineConfig,
6+
)
7+
8+
__all__ = [
9+
"FlinkComputeEngine",
10+
"FlinkComputeEngineConfig",
11+
]

0 commit comments

Comments
 (0)