Skip to content

Commit 9419790

Browse files
committed
refactor: register native UDFs as session-scoped Catalyst expressions
Match the registration in #6697. A native UDF's calls now resolve to a `NativeUdfCall` expression rather than a `ScalaUDF` holding a marker function, and `CometNativeUdfCall`, a serde of its own, emits the `NativeScalarUdf`. `CometScalaUDF` and `DataTypeSupport` are back to main's versions. - Spark's analyzer checks the argument types (`ExpectsInputTypes`), disregarding nullability and field metadata, and inserts no casts. A mismatch fails analysis instead of Comet's planning, so Comet's own type check, `equalsIgnoreNullability` and `CometNativeUdfArgumentTypeException` are gone. - A call with the wrong number of arguments fails analysis with Spark's own error. - Nothing caps the arity any more. Four was the most `ScalaUDF`'s `FunctionN` casts allowed for, so `sum_c` and a test cover a five-argument call. - The codegen dispatcher declines an ordinary UDF whose arguments hold a native call, so Spark gets that operator and the failure names the real cause. - If Spark evaluates the call, it throws `CometUdfNotEvaluatedException`. `ShimSessionFunctionRegistry`, `CometUdfErrors` and `CometUdfExceptions` are byte-identical copies of #6697's, so whichever pull request lands second merges them without a conflict.
1 parent 7d5b363 commit 9419790

17 files changed

Lines changed: 400 additions & 261 deletions

File tree

‎docs/source/user-guide/latest/rust_udfs.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -285,13 +285,12 @@ This feature is at an early stage. The current limitations are:
285285
`register`. Comet does not distribute it for you: stage it with your image, a mounted volume, or
286286
your cluster's own file distribution, and pass a path that is valid cluster-wide. A path that
287287
exists only on the driver will fail at execution time.
288-
- **Up to 4 arguments** per function.
289288
- **No type coercion.** Every call must pass exactly the argument types given to `register`,
290-
apart from nullability and struct field metadata such as a column comment. Comet does not cast
291-
arguments to those types; a call that passes, say, an `int` column to a UDF registered with
292-
`LongType` is refused at planning time, and the fix is an explicit `cast` in the query.
293-
`return_field` should still reject anything it does not handle. Downcast defensively in `invoke`
294-
and return a clear `Err` rather than assuming a particular array layout.
289+
apart from nullability. Spark's analyzer checks each call and inserts no casts, so a call that
290+
passes, say, an `int` column to a UDF registered with `LongType` fails analysis, and the fix is
291+
an explicit `cast` in the query. `return_field` should still reject anything it does not handle.
292+
Downcast defensively in `invoke` and return a clear `Err` rather than assuming a particular array
293+
layout.
295294
- **Loading a library is loading native code.** It runs with the full privileges of the executor
296295
process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
297296
Only register libraries you trust and control.

‎native/comet-test-udfs/src/lib.rs‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
//! - `add_one_c`: `(Int64) -> Int64`, the basic compute path
2323
//! - `sub_c`: `(Int64, Int64) -> Int64`, `a - b`, which is order-sensitive
2424
//! so a call that delivered its arguments swapped gives a wrong answer
25+
//! - `sum_c`: `(Int64, ...) -> Int64`, the sum of any number of arguments
2526
//! - `echo_c`: identity over any type, used to check that each supported
2627
//! Spark type survives the round trip through the ABI with its nulls
2728
//! - `stringify_c`: `(any) -> Utf8`, which forces the UDF to actually
@@ -124,6 +125,47 @@ impl CometCScalarUdf for SubC {
124125
}
125126
}
126127

128+
/// The sum of one or more `Int64` arguments, NULL where any of them is.
129+
///
130+
/// Takes any number of arguments, so the suite can call it with more than
131+
/// the four a registration was once limited to.
132+
#[derive(Default)]
133+
pub struct SumC;
134+
135+
impl CometCScalarUdf for SumC {
136+
fn name(&self) -> &str {
137+
"sum_c"
138+
}
139+
140+
fn return_field(&self, args: &[Field]) -> Result<Field, String> {
141+
if args.is_empty() || args.iter().any(|a| a.data_type() != &DataType::Int64) {
142+
return Err("sum_c expects one or more Int64 arguments".to_string());
143+
}
144+
Ok(Field::new("sum_c", DataType::Int64, true))
145+
}
146+
147+
fn invoke(&self, args: &[ArrayRef], n_rows: usize) -> Result<ArrayRef, String> {
148+
let arrays = args
149+
.iter()
150+
.enumerate()
151+
.map(|(i, a)| {
152+
a.as_any()
153+
.downcast_ref::<Int64Array>()
154+
.ok_or_else(|| format!("arg #{i}: expected Int64Array"))
155+
})
156+
.collect::<Result<Vec<_>, _>>()?;
157+
let out: Int64Array = (0..n_rows)
158+
.map(|row| {
159+
arrays
160+
.iter()
161+
.map(|a| a.is_valid(row).then(|| a.value(row)))
162+
.sum::<Option<i64>>()
163+
})
164+
.collect();
165+
Ok(Arc::new(out))
166+
}
167+
}
168+
127169
/// Identity over any input type: declares the argument's own type as the
128170
/// return type and hands the array back.
129171
///
@@ -396,6 +438,7 @@ impl CometCScalarUdf for PanicsOnReturnField {
396438
comet_c_udf_export!(
397439
AddOneC,
398440
SubC,
441+
SumC,
399442
EchoC,
400443
StringifyC,
401444
MakeTsUtcC,

‎native/core/src/execution/c_udf/loader.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,7 @@ mod tests {
273273
for expected in [
274274
"add_one_c",
275275
"sub_c",
276+
"sum_c",
276277
"echo_c",
277278
"stringify_c",
278279
"make_ts_utc_c",

‎spark/src/main/scala/org/apache/comet/DataTypeSupport.scala‎

Lines changed: 0 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -157,23 +157,6 @@ object DataTypeSupport {
157157
case other => other
158158
}
159159

160-
/**
161-
* Whether `left` and `right` are the same type once nullability is disregarded at every nesting
162-
* level. Struct field names have to match, but their metadata, such as a column comment, is not
163-
* compared, since it is not part of the SQL type. Re-derives Spark's
164-
* `DataType.equalsIgnoreNullability`, which is `private[sql]` on Spark 3.4.
165-
*/
166-
def equalsIgnoreNullability(left: DataType, right: DataType): Boolean = (left, right) match {
167-
case (ArrayType(l, _), ArrayType(r, _)) => equalsIgnoreNullability(l, r)
168-
case (MapType(lk, lv, _), MapType(rk, rv, _)) =>
169-
equalsIgnoreNullability(lk, rk) && equalsIgnoreNullability(lv, rv)
170-
case (StructType(l), StructType(r)) =>
171-
l.length == r.length && l.zip(r).forall { case (lf, rf) =>
172-
lf.name == rf.name && equalsIgnoreNullability(lf.dataType, rf.dataType)
173-
}
174-
case (l, r) => l == r
175-
}
176-
177160
def hasTemporalType(t: DataType): Boolean = t match {
178161
case DataTypes.DateType | DataTypes.TimestampType | DataTypes.TimestampNTZType =>
179162
true

‎spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,17 +182,21 @@ object CometBatchKernelCodegen extends Logging with CometExprTraitShim with Come
182182
// `Unevaluable`: rejected by default. `isCodegenInertUnevaluable` exempts version-specific
183183
// leaves that are `Unevaluable` but never invoked by codegen (e.g. Spark 4.0's
184184
// `ResolvedCollation` in `Collate.collation`, where `Collate.genCode` delegates to its child).
185+
//
186+
// A native UDF call is a `CodegenFallback` whose `eval` only throws, since its implementation
187+
// runs in the native library.
185188
boundExpr.find {
186189
case _: org.apache.spark.sql.catalyst.expressions.aggregate.AggregateFunction => true
187190
case _: org.apache.spark.sql.catalyst.expressions.Generator => true
188191
case u: Unevaluable if isCodegenInertUnevaluable(u) => false
189192
case _: Unevaluable => true
193+
case _: org.apache.comet.udf.NativeUdfCall => true
190194
case _ => false
191195
} match {
192196
case Some(bad) =>
193197
return Some(
194198
s"codegen dispatch: expression ${bad.getClass.getSimpleName} not supported " +
195-
"(aggregate, generator, or unevaluable)")
199+
"(aggregate, generator, unevaluable, or native UDF)")
196200
case None =>
197201
}
198202
val badRef = boundExpr.collectFirst {
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.comet.serde
21+
22+
import org.apache.spark.sql.catalyst.expressions.Attribute
23+
24+
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
25+
import org.apache.comet.serde.ExprOuterClass.Expr
26+
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, serializeDataType}
27+
import org.apache.comet.udf.NativeUdfCall
28+
29+
/**
30+
* Emits a `NativeScalarUdf` naming the registered function and the library it lives in, with each
31+
* argument serialized as its own native expression. Executors load the library from that path on
32+
* first use, so they need no registration of their own.
33+
*/
34+
object CometNativeUdfCall extends CometExpressionSerde[NativeUdfCall] {
35+
36+
override def convert(
37+
expr: NativeUdfCall,
38+
inputs: Seq[Attribute],
39+
binding: Boolean): Option[Expr] = {
40+
val args = expr.children.map(exprToProtoInternal(_, inputs, binding))
41+
// An argument that could not be converted has already recorded why.
42+
if (args.contains(None)) {
43+
return None
44+
}
45+
val returnType = serializeDataType(expr.dataType).getOrElse {
46+
withFallbackReason(
47+
expr,
48+
s"native UDF '${expr.name}': unsupported return type ${expr.dataType}")
49+
return None
50+
}
51+
val udf = ExprOuterClass.NativeScalarUdf
52+
.newBuilder()
53+
.setName(expr.name)
54+
.setLibraryPath(expr.libraryPath)
55+
.setReturnType(returnType)
56+
.setDeterministic(expr.deterministic)
57+
args.flatten.foreach(udf.addArgs)
58+
Some(Expr.newBuilder().setNativeScalarUdf(udf.build()).build())
59+
}
60+
}

‎spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala‎

Lines changed: 2 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,9 @@ import org.apache.spark.sql.types.BinaryType
2828
import org.apache.comet.CometConf
2929
import org.apache.comet.CometExplainInfo
3030
import org.apache.comet.CometSparkSessionExtensions.{withCodegenDispatchExpr, withFallbackReason}
31-
import org.apache.comet.DataTypeSupport.equalsIgnoreNullability
3231
import org.apache.comet.codegen.CometBatchKernelCodegen
3332
import org.apache.comet.serde.ExprOuterClass.Expr
3433
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, serializeDataType}
35-
import org.apache.comet.udf.{CometNativeUdfArgumentTypeException, CometNativeUdfFunction, NativeUdfMetadata}
3634
import org.apache.comet.udf.codegen.CometScalaUDFCodegen
3735

3836
/**
@@ -57,71 +55,8 @@ import org.apache.comet.udf.codegen.CometScalaUDFCodegen
5755
*/
5856
object CometScalaUDF extends CometExpressionSerde[ScalaUDF] {
5957

60-
override def convert(expr: ScalaUDF, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = {
61-
// A native UDF is emitted as NativeScalarUdf and dispatched to its shared library rather than
62-
// to the JVM codegen dispatcher. It is recognized by the function its registration put in the
63-
// `ScalaUDF`, not by name, so an ordinary UDF registered under the same name, in this session
64-
// or another, holds its own function and stays on the JVM path.
65-
expr.function match {
66-
case native: CometNativeUdfFunction =>
67-
emitNativeScalarUdf(expr, native, inputs, binding)
68-
case _ =>
69-
emitJvmCodegenDispatch(expr, inputs, binding)
70-
}
71-
}
72-
73-
private def emitNativeScalarUdf(
74-
expr: ScalaUDF,
75-
native: CometNativeUdfFunction,
76-
inputs: Seq[Attribute],
77-
binding: Boolean): Option[Expr] = {
78-
val name = native.name
79-
val meta = native.meta
80-
checkArgumentTypes(name, expr, meta)
81-
val argProtos = expr.children.map(c => exprToProtoInternal(c, inputs, binding))
82-
if (argProtos.exists(_.isEmpty)) {
83-
withFallbackReason(expr, "one or more native UDF arguments are not supported")
84-
return None
85-
}
86-
val returnTypeProto = serializeDataType(meta.returnType).getOrElse {
87-
withFallbackReason(expr, s"return type ${meta.returnType} not serializable")
88-
return None
89-
}
90-
val callBuilder = ExprOuterClass.NativeScalarUdf
91-
.newBuilder()
92-
.setName(name)
93-
.setLibraryPath(meta.libraryPath)
94-
.setReturnType(returnTypeProto)
95-
.setDeterministic(expr.deterministic)
96-
argProtos.foreach(a => callBuilder.addArgs(a.get))
97-
Some(ExprOuterClass.Expr.newBuilder().setNativeScalarUdf(callBuilder.build()).build())
98-
}
99-
100-
/**
101-
* Refuse a call whose argument types differ from the ones the UDF was registered with.
102-
*
103-
* The `ScalaUDF` a native UDF's calls resolve to declares no input types, so Spark inserts no
104-
* casts for it and a call reaches this point with whatever types its arguments happen to have.
105-
* Converting them here would be a semantic choice Spark never made, so the call is refused
106-
* instead, naming both signatures. Nullability and struct field metadata are disregarded,
107-
* because neither changes the values a UDF receives.
108-
*
109-
* This throws rather than falling back: a native UDF cannot be evaluated on the JVM, so a
110-
* fallback would only fail later with a less useful message.
111-
*/
112-
private def checkArgumentTypes(name: String, expr: ScalaUDF, meta: NativeUdfMetadata): Unit = {
113-
val actual = expr.children.map(_.dataType)
114-
val matches = actual.length == meta.inputTypes.length &&
115-
actual.zip(meta.inputTypes).forall { case (a, d) => equalsIgnoreNullability(a, d) }
116-
if (!matches) {
117-
def render(types: Seq[org.apache.spark.sql.types.DataType]) =
118-
types.map(_.catalogString).mkString("(", ", ", ")")
119-
throw new CometNativeUdfArgumentTypeException(
120-
s"native UDF '$name' was registered with argument types ${render(meta.inputTypes)} " +
121-
s"but is called with ${render(actual)}. Cast the arguments to the registered types, " +
122-
"or register the UDF with the types this call produces.")
123-
}
124-
}
58+
override def convert(expr: ScalaUDF, inputs: Seq[Attribute], binding: Boolean): Option[Expr] =
59+
emitJvmCodegenDispatch(expr, inputs, binding)
12560

12661
/**
12762
* Bind `expr`, closure-serialize it, and emit a `JvmScalarUdf` proto routed through

‎spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ import org.apache.comet.serde.Types.{DataType => ProtoDataType}
4747
import org.apache.comet.serde.Types.DataType._
4848
import org.apache.comet.serde.literals.CometLiteral
4949
import org.apache.comet.shims.{CometExprShim, CometTypeShim}
50+
import org.apache.comet.udf.NativeUdfCall
5051

5152
/**
5253
* An utility object for query plan and expression serialization.
@@ -370,6 +371,7 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
370371
classOf[Literal] -> CometLiteral,
371372
classOf[MakeDecimal] -> CometMakeDecimal,
372373
classOf[MonotonicallyIncreasingID] -> CometMonotonicallyIncreasingId,
374+
classOf[NativeUdfCall] -> CometNativeUdfCall,
373375
classOf[ScalarSubquery] -> CometScalarSubquery,
374376
classOf[ScalaUDF] -> CometScalaUDF,
375377
classOf[SparkPartitionID] -> CometSparkPartitionId,

‎spark/src/main/scala/org/apache/comet/udf/CometNativeUDF.scala‎

Lines changed: 26 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@
2020
package org.apache.comet.udf
2121

2222
import org.apache.spark.sql.SparkSession
23-
import org.apache.spark.sql.catalyst.expressions.{Expression, ScalaUDF}
23+
import org.apache.spark.sql.comet.CometUdfErrors
2424
import org.apache.spark.sql.types.DataType
2525

26+
import org.apache.comet.shims.ShimSessionFunctionRegistry
27+
2628
/**
2729
* Entry point for registering scalar UDFs that run as native code inside Comet.
2830
*
@@ -45,18 +47,19 @@ object CometNativeUDF {
4547
* Register a single native UDF with an explicit signature.
4648
*
4749
* Validates the library on the driver (loads it, confirms a UDF named `name` exists), then
48-
* registers `name` as a temporary function in `spark`'s function registry, as
49-
* `spark.udf.register` does. Like any temporary function it is visible only in that session,
50-
* and registering another function under the same name there replaces it.
50+
* installs `name` as a temporary function of the session, as `spark.udf.register` would: other
51+
* sessions do not see it, and registering another function under the same name replaces it.
5152
*
5253
* Executors need no registration: the library path travels with the plan in the
5354
* `NativeScalarUdf` proto, and each executor loads the library itself on first use. The path
5455
* must therefore be valid on every executor, not just the driver.
5556
*
56-
* `inputTypes` is the signature every call must match. Comet does not convert arguments to
57-
* these types: a call whose argument types differ, other than in nullability or struct field
58-
* metadata, is refused at planning time with both signatures named, so cast the arguments in
59-
* the query instead.
57+
* `inputTypes` is the signature every call must match. Spark's analyzer rejects a call whose
58+
* argument types differ, other than in nullability, and inserts no casts, so cast the arguments
59+
* in the query instead.
60+
*
61+
* Spark cannot evaluate the call itself: if Comet does not take the operator holding it, the
62+
* query fails.
6063
*
6164
* `deterministic` must be `true`. Comet plans every imported kernel as immutable, so a
6265
* nondeterministic UDF cannot yet be expressed; passing `false` fails here rather than silently
@@ -82,28 +85,21 @@ object CometNativeUDF {
8285
"See https://github.com/apache/datafusion-comet/issues/5249")
8386
}
8487
validateLibrary(libraryPath, name)
85-
val function = CometNativeUdfFunction(
86-
name,
87-
NativeUdfMetadata(libraryPath, inputTypes, returnType, deterministic))
88-
// Every call resolved to `name` gets the same function, so the serde recognizes the call by
89-
// it, and two identical calls are still equal expressions.
90-
def builder(children: Seq[Expression]): Expression = {
91-
if (children.length != inputTypes.length) {
92-
throw new CometNativeUdfArgumentTypeException(
93-
s"native UDF '$name' was registered with ${inputTypes.length} argument(s) but is " +
94-
s"called with ${children.length}")
95-
}
96-
// No input encoders, so Spark inserts no casts and does no null handling for the
97-
// arguments. The serde checks their types instead.
98-
ScalaUDF(
99-
function,
100-
returnType,
101-
children,
102-
inputEncoders = Seq.fill(children.length)(None),
103-
udfName = Some(name),
104-
udfDeterministic = deterministic)
105-
}
106-
spark.sessionState.functionRegistry.createOrReplaceTempFunction(name, builder, "scala_udf")
88+
ShimSessionFunctionRegistry
89+
.functionRegistry(spark)
90+
.createOrReplaceTempFunction(
91+
name,
92+
children => {
93+
// Checked here, as built-in functions check it, because the analyzer's type coercion
94+
// pairs arguments with types positionally before any check on the call would run.
95+
if (children.length != inputTypes.length) {
96+
throw CometUdfErrors.wrongNumArgs(name, inputTypes.length, children.length)
97+
}
98+
NativeUdfCall(name, libraryPath, inputTypes, returnType, deterministic, children)
99+
},
100+
// `ExpressionInfo` accepts only Spark's own source names. This is the one
101+
// `spark.udf.register` gives a Scala UDF.
102+
"scala_udf")
107103
}
108104

109105
/**

0 commit comments

Comments
 (0)