You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9419790
Browse filesBrowse the repository at this point in the historyBrowse files
refactor: register native UDFs as session-scoped Catalyst expressions
Match the registration in #6697. A native UDF's calls now resolve to a
`NativeUdfCall` expression rather than a `ScalaUDF` holding a marker
function, and `CometNativeUdfCall`, a serde of its own, emits the
`NativeScalarUdf`. `CometScalaUDF` and `DataTypeSupport` are back to main's
versions.
- Spark's analyzer checks the argument types (`ExpectsInputTypes`),
disregarding nullability and field metadata, and inserts no casts. A
mismatch fails analysis instead of Comet's planning, so Comet's own type
check, `equalsIgnoreNullability` and `CometNativeUdfArgumentTypeException`
are gone.
- A call with the wrong number of arguments fails analysis with Spark's own
error.
- Nothing caps the arity any more. Four was the most `ScalaUDF`'s `FunctionN`
casts allowed for, so `sum_c` and a test cover a five-argument call.
- The codegen dispatcher declines an ordinary UDF whose arguments hold a
native call, so Spark gets that operator and the failure names the real
cause.
- If Spark evaluates the call, it throws `CometUdfNotEvaluatedException`.
`ShimSessionFunctionRegistry`, `CometUdfErrors` and `CometUdfExceptions` are
byte-identical copies of #6697's, so whichever pull request lands second
merges them without a conflict.
0 commit comments