Repository navigation
Distribute Feast with Dask #1953
Description
Is your feature request related to a problem? Please describe.
I would like to add possibility to distribute Feast feature store with Dask library
Describe the solution you'd like
I will extend Feast with the Dask library. Thanks to this the users will be able to distribute historical feature handling on the clusters (eg. VMs, k8s, yarn, local machine). This will help to handle huge datasets.
Describe alternatives you've considered
Dask is the best solution because we stay in the python stack others like Spark will need additional setup and maintenance.
Additional context
No
Reacted by vtslab and HUANG SIZHE
Activity
judahrand commentedon Nov 5, 2021
Is there a good reason to reject Modin?
qooba commentedon Nov 8, 2021
@judahrand
Modinis great if you want to boost the pandas code with minimal effort. If you want to have full control (partitioning, persistence in memory etc.) and better performance you need to go deeper and use directly eg. Dask or Ray (current Modin backends).