Repository navigation
Support for DeltaLake, Iceberg, Hudi as an offline source covered by Spark, Ray and Dask engines #2406
Description
Is your feature request related to a problem? Please describe.
Feast currently can run on spark, dask and ray. Using this engines the support for deltalake, apache iceberg and apache hudi (which was requested by the community) can be added. This feature will be very helpful for the teams which base on Spark but also for integrations with Flink, Spark Streaming eg. when historical features are saved in the data lake formats.
Describe the solution you'd like
The solution assumes adding new data lake sources:
DeltaDataSourceIcebergDataSourceHudiDataSource
and also support for CSV files.
The data lake sources support will be covered by the Spark engine (which is already in contrib) for users which use Feast on Spark (or Databricks) but also for Dask and Ray.
Additional assumptions:
- Data sources can be mixed eg. you can use
DeltaDataSource,CSVandIcebergDataSourceto fetch historical features. - The engine change won't require changes in the code (only
feature_store.yamlconfiguration) thus user can test on the laptop using Dask (without any cluster setup) and then deploy to Spark cluster (or Dask and Ray clusters) - The implementation should enable to simply add new DataSources like Apache Arrow Flight (if the python api will be added) and simply mix them with other data sources in the future.
Delta Lake
The support for Delta Lake for Feast on Spark is already proposed and tested on Databricks and local spark (https://github.com/qooba/feast-pyspark). The solution is based on the plain pyspark rather than on Spark SQL and Jinja thus it is to decide which implementation will be more desirable and maintainable.
The support for Delta Lake for Feast on Dask (Ray) can be implemented using delta python interface:
- https://github.com/delta-io/delta-rs
- https://github.com/rajagurunath/dask_deltatable
Apache Iceberg
The Apache Iceberg is covered by the Spark engine but also by the python api which can be used to add Dask/Ray implementation.
Apache Hudi
The Apache Hudi is covered by the Spark. Currently there is no python api (as far as I know).
CSV
The support for csv files will be dedicated for the data scientists which would like to conduct ad-hoc experiments.
Describe alternatives you've considered
N/A
Additional context
N/A
Activity
tianlinzx commentedon Apr 20, 2022
Will this feature be picked up ？
qooba commentedon Apr 27, 2022
Yes, but this requires a lot of changes in the Feast architecture thus I have decided to move it to separate repository and create Feast extension: https://github.com/qooba/yummy
LeonardAukea commentedon May 19, 2022
I had a quick look and it seems like the
IcebergDataSourceis still missing. Are you planning to add it or has anything changed? Do you need help?qooba commentedon May 19, 2022
I'll add
Icebergsupport soon :)qooba commentedon Jul 25, 2022
@LeonardAukea - I have finished feast integration with Iceberg here you have video introduction: https://www.youtube.com/watch?v=kv0iWuSf4jw
andywalner commentedon Mar 19, 2025
Just came across this. Hudi now offers a Python API via hudi-rs. More at https://hudi.apache.org/docs/python-rust-quick-start-guide/
franciscojavierarceo commentedon May 15, 2025
H @qooba would you be open to adding this into Feast by any chance?
CC @HaoXuAI
haoxu0 commentedon Jun 13, 2025
I will look into this. One thing is to make the open table format time travel consistent with Feast time travel utility.