Skip to content

Native UDFs: config to disable or restrict library loading #6175

Closed

Description

@andygrove

CometNativeUDF.register loads any shared library at any path the caller names, on the driver and
then on every executor, with the full privileges of the JVM process. There is no configuration an
operator can set to turn the feature off or to limit which paths may be loaded.

Options:

  • spark.comet.nativeUdf.enabled, checked in register on the driver and in the planner on the
    executor, so a plan built elsewhere cannot bypass it. Whether it defaults to on (the feature is
    already opt-in by API call) or off (loading native code should need an operator's consent) is
    the main question for this issue.
  • spark.comet.nativeUdf.allowedPaths, a list of directory prefixes. Paths would be canonicalized
    before the check so .. and symlinks cannot escape it.

The executor-side check matters as much as the driver-side one: the library path travels in the
NativeScalarUdf proto, so the executor must not trust that the driver checked it.

Follow-up to #4459.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions