You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 74d7560
Browse filesBrowse the repository at this point in the historyBrowse files
fix: ignore struct field metadata when checking native UDF argument types
`checkArgumentTypes` compared `deepNullable` forms, which keep each
`StructField`'s metadata. A struct argument whose field carries a comment was
refused against a registration with the same field name and type, with an
error naming `struct<a:int>` on both sides.
`DataTypeSupport.equalsIgnoreNullability` re-derives Spark's
`DataType.equalsIgnoreNullability`, which compares struct field names but
neither nullability nor metadata. Spark's own is `private[sql]` on 3.4. The
new test passes a struct whose field has a comment, built with
`struct(col.as(name, metadata))`.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/rust_udfs.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -276,10 +276,11 @@ This feature is at an early stage. The current limitations are:
276
276
exists only on the driver will fail at execution time.
277
277
-**Up to 4 arguments** per function.
278
278
-**No type coercion.** Every call must pass exactly the argument types given to `register`,
279
-
apart from nullability. Comet does not cast arguments to those types; a call that passes, say,
280
-
an `int` column to a UDF registered with `LongType` is refused at planning time, and the fix is
281
-
an explicit `cast` in the query. `return_field` should still reject anything it does not handle. Downcast defensively in `invoke` and return a clear `Err`
282
-
rather than assuming a particular array layout.
279
+
apart from nullability and struct field metadata such as a column comment. Comet does not cast
280
+
arguments to those types; a call that passes, say, an `int` column to a UDF registered with
281
+
`LongType` is refused at planning time, and the fix is an explicit `cast` in the query.
282
+
`return_field` should still reject anything it does not handle. Downcast defensively in `invoke`
283
+
and return a clear `Err` rather than assuming a particular array layout.
283
284
-**Loading a library is loading native code.** It runs with the full privileges of the executor
284
285
process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
0 commit comments