You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
OnDemandFeatureView: Allow python transformations for offline processing. #4490
Is your feature request related to a problem? Please describe.
When running a PythonTransformation for offline feature retrieval, an exception is raised here. I understand that python transformations on very large datasets do not scale well. However, in some cases, a user might still want to apply a python transformation to a smaller dataset in the offline store. Due to this error (and the missing implementation of PythonTransformation().transform_arrow()), this is currently not possible.
Describe the solution you'd like
Instead of raising an exception, we could still run the transformation, and also log a warning indicating that it may not scale well. On top of that we can implement the functionality in transform_arrow to make the transformation happen.
Oh we should definitely discuss this because I agree. Also, for scaling Spark should really call the Transformation as a UDF and then it can scale very well.
Activity
franciscojavierarceo commentedon Oct 6, 2024
Oh we should definitely discuss this because I agree. Also, for scaling Spark should really call the Transformation as a UDF and then it can scale very well.
franciscojavierarceo commentedon Oct 20, 2024
This has been implemented. 👍
job-almekinders commentedon Oct 21, 2024
Thanks a lot for this! @franciscojavierarceo
FYI @TomSteenbergen :)