Skip to content

OnDemandFeatureView: Allow python transformations for offline processing.  #4490

Closed

Description

@job-almekinders

Is your feature request related to a problem? Please describe.
When running a PythonTransformation for offline feature retrieval, an exception is raised here. I understand that python transformations on very large datasets do not scale well. However, in some cases, a user might still want to apply a python transformation to a smaller dataset in the offline store. Due to this error (and the missing implementation of PythonTransformation().transform_arrow()), this is currently not possible.

Describe the solution you'd like
Instead of raising an exception, we could still run the transformation, and also log a warning indicating that it may not scale well. On top of that we can implement the functionality in transform_arrow to make the transformation happen.

Describe alternatives you've considered

Additional context

Activity

  1. franciscojavierarceo commented on Oct 6, 2024

    @franciscojavierarceo
    Member

    Oh we should definitely discuss this because I agree. Also, for scaling Spark should really call the Transformation as a UDF and then it can scale very well.

  2. franciscojavierarceo commented on Oct 20, 2024

    @franciscojavierarceo
    Member

    This has been implemented. 👍

  3. job-almekinders commented on Oct 21, 2024

    @job-almekinders
    ContributorAuthor

    Thanks a lot for this! @franciscojavierarceo

    FYI @TomSteenbergen :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions