Skip to content

Native UDFs: lift the 4-argument cap and add registerAll #6177

Open

Description

@andygrove

Arity. CometNativeUDF.register installs a Spark catalog stub so that SQL and DataFrame name
resolution succeed, and that stub is built from functions.udf with a fixed-arity Scala lambda, one
case per arity from 0 to 4. A UDF with five or more arguments is rejected at registration. Neither
the ABI nor the native adapter has an arity limit; only the stub does. Spark's UDF5 through
UDF22 interfaces (or building the ScalaUDF directly) would remove the cap.

registerAll. A library can export any number of kernels, but each has to be registered by
name with its own signature. A registerAll(spark, libraryPath) that registered every exported
kernel would need the library to report each kernel's signature, which ties into #5597 (derive the
return type from the library rather than asking the caller to declare it).

Follow-up to #4459.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions