Skip to content

Commit 74d7560

Browse files
committed
fix: ignore struct field metadata when checking native UDF argument types
`checkArgumentTypes` compared `deepNullable` forms, which keep each `StructField`'s metadata. A struct argument whose field carries a comment was refused against a registration with the same field name and type, with an error naming `struct<a:int>` on both sides. `DataTypeSupport.equalsIgnoreNullability` re-derives Spark's `DataType.equalsIgnoreNullability`, which compares struct field names but neither nullability nor metadata. Spark's own is `private[sql]` on 3.4. The new test passes a struct whose field has a comment, built with `struct(col.as(name, metadata))`.
1 parent dc962d5 commit 74d7560

5 files changed

Lines changed: 43 additions & 10 deletions

File tree

‎docs/source/user-guide/latest/rust_udfs.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -276,10 +276,11 @@ This feature is at an early stage. The current limitations are:
276276
exists only on the driver will fail at execution time.
277277
- **Up to 4 arguments** per function.
278278
- **No type coercion.** Every call must pass exactly the argument types given to `register`,
279-
apart from nullability. Comet does not cast arguments to those types; a call that passes, say,
280-
an `int` column to a UDF registered with `LongType` is refused at planning time, and the fix is
281-
an explicit `cast` in the query. `return_field` should still reject anything it does not handle. Downcast defensively in `invoke` and return a clear `Err`
282-
rather than assuming a particular array layout.
279+
apart from nullability and struct field metadata such as a column comment. Comet does not cast
280+
arguments to those types; a call that passes, say, an `int` column to a UDF registered with
281+
`LongType` is refused at planning time, and the fix is an explicit `cast` in the query.
282+
`return_field` should still reject anything it does not handle. Downcast defensively in `invoke`
283+
and return a clear `Err` rather than assuming a particular array layout.
283284
- **Loading a library is loading native code.** It runs with the full privileges of the executor
284285
process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
285286
Only register libraries you trust and control.

‎spark/src/main/scala/org/apache/comet/DataTypeSupport.scala‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,23 @@ object DataTypeSupport {
157157
case other => other
158158
}
159159

160+
/**
161+
* Whether `left` and `right` are the same type once nullability is disregarded at every nesting
162+
* level. Struct field names have to match, but their metadata, such as a column comment, is not
163+
* compared, since it is not part of the SQL type. Re-derives Spark's
164+
* `DataType.equalsIgnoreNullability`, which is `private[sql]` on Spark 3.4.
165+
*/
166+
def equalsIgnoreNullability(left: DataType, right: DataType): Boolean = (left, right) match {
167+
case (ArrayType(l, _), ArrayType(r, _)) => equalsIgnoreNullability(l, r)
168+
case (MapType(lk, lv, _), MapType(rk, rv, _)) =>
169+
equalsIgnoreNullability(lk, rk) && equalsIgnoreNullability(lv, rv)
170+
case (StructType(l), StructType(r)) =>
171+
l.length == r.length && l.zip(r).forall { case (lf, rf) =>
172+
lf.name == rf.name && equalsIgnoreNullability(lf.dataType, rf.dataType)
173+
}
174+
case (l, r) => l == r
175+
}
176+
160177
def hasTemporalType(t: DataType): Boolean = t match {
161178
case DataTypes.DateType | DataTypes.TimestampType | DataTypes.TimestampNTZType =>
162179
true

‎spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ import org.apache.spark.sql.types.BinaryType
2828
import org.apache.comet.CometConf
2929
import org.apache.comet.CometExplainInfo
3030
import org.apache.comet.CometSparkSessionExtensions.{withCodegenDispatchExpr, withFallbackReason}
31-
import org.apache.comet.DataTypeSupport.deepNullable
31+
import org.apache.comet.DataTypeSupport.equalsIgnoreNullability
3232
import org.apache.comet.codegen.CometBatchKernelCodegen
3333
import org.apache.comet.serde.ExprOuterClass.Expr
3434
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, serializeDataType}
@@ -107,15 +107,16 @@ object CometScalaUDF extends CometExpressionSerde[ScalaUDF] {
107107
* The catalog stub Comet installs is untyped, so Spark inserts no casts for it and a call
108108
* reaches this point with whatever types its arguments happen to have. Converting them here
109109
* would be a semantic choice Spark never made, so the call is refused instead, naming both
110-
* signatures. Nullability is disregarded because it does not change the values a UDF receives.
110+
* signatures. Nullability and struct field metadata are disregarded, because neither changes
111+
* the values a UDF receives.
111112
*
112113
* This throws rather than falling back: the stub cannot evaluate the UDF on the JVM, so a
113114
* fallback would only fail later with a less useful message.
114115
*/
115116
private def checkArgumentTypes(name: String, expr: ScalaUDF, meta: NativeUdfMetadata): Unit = {
116117
val actual = expr.children.map(_.dataType)
117118
val matches = actual.length == meta.inputTypes.length &&
118-
actual.zip(meta.inputTypes).forall { case (a, d) => deepNullable(a) == deepNullable(d) }
119+
actual.zip(meta.inputTypes).forall { case (a, d) => equalsIgnoreNullability(a, d) }
119120
if (!matches) {
120121
def render(types: Seq[org.apache.spark.sql.types.DataType]) =
121122
types.map(_.catalogString).mkString("(", ", ", ")")

‎spark/src/main/scala/org/apache/comet/udf/CometNativeUDF.scala‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,8 +55,9 @@ object CometNativeUDF {
5555
* must therefore be valid on every executor, not just the driver.
5656
*
5757
* `inputTypes` is the signature every call must match. Comet does not convert arguments to
58-
* these types: a call whose argument types differ, other than in nullability, is refused at
59-
* planning time with both signatures named, so cast the arguments in the query instead.
58+
* these types: a call whose argument types differ, other than in nullability or struct field
59+
* metadata, is refused at planning time with both signatures named, so cast the arguments in
60+
* the query instead.
6061
*
6162
* `deterministic` must be `true`. Comet plans every imported kernel as immutable, so a
6263
* nondeterministic UDF cannot yet be expressed; passing `false` fails here rather than silently

‎spark/src/test/scala/org/apache/comet/CometNativeUdfSuite.scala‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ import java.io.File
2323
import java.util.Locale
2424

2525
import org.apache.spark.sql.CometTestBase
26-
import org.apache.spark.sql.functions.expr
26+
import org.apache.spark.sql.functions.{col, expr, struct}
2727
import org.apache.spark.sql.types._
2828

2929
import org.apache.comet.udf.{CometNativeUDF, CometNativeUdfAbiException, CometNativeUdfLoadException}
@@ -371,6 +371,19 @@ class CometNativeUdfSuite extends CometTestBase {
371371
assert(fixed == Seq(1L, 2L, 3L))
372372
}
373373

374+
test("argument types are compared without struct field metadata") {
375+
// Spark keeps a field's metadata, such as a column comment, inside the struct type, but it is
376+
// not part of the SQL type and does not change what the UDF receives.
377+
val structType = StructType(Seq(StructField("a", IntegerType)))
378+
CometNativeUDF.register(spark, "echo_c", libPath, Seq(structType), structType)
379+
val comment = new MetadataBuilder().putString("comment", "documented").build()
380+
val df = spark.range(0, 3).select(struct(col("id").cast("int").as("a", comment)).as("c"))
381+
val argType = df.schema.head.dataType.asInstanceOf[StructType]
382+
assert(argType.head.metadata.contains("comment"), s"no comment on the field: $argType")
383+
val out = df.selectExpr("echo_c(c) AS y").collect().map(_.getStruct(0).getInt(0)).toSeq
384+
assert(out == Seq(0, 1, 2))
385+
}
386+
374387
test("echo_c rejects a call whose argument count it does not accept") {
375388
CometNativeUDF.register(spark, "echo_c", libPath, Seq(LongType), LongType)
376389
// The catalog stub is arity-1, so a 2-arg call is rejected during analysis.

0 commit comments

Comments
 (0)