Repository navigation
Lack of Spark Support for On-Demand Feature Views #5475
Description
Currently, stream feature views support both spark and spark_sql execution modes, which enables them to scale efficiently with large datasets. However, on-demand feature views are limited to pandas or python modes only.
This limitation becomes a bottleneck when trying to perform transformations on features that haven't been precomputed, especially in cases where those transformations need to scale to large data volumes. Without Spark support, users are forced to rely on local computation, which significantly hinders scalability and performance.
Reacted by Hao Xu
Activity
haoxu0 commentedon Aug 10, 2025
Team is working on it 👍
franciscojavierarceo commentedon Aug 10, 2025
A temporary hack is to use the ODFV as a UDF. That's ultimately what the real implementation will look like.
kwin-wang commentedon Mar 3, 2026
@franciscojavierarceo hi, can you give me an example that how to implement this way? thanks