You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4c335a8
Browse filesBrowse the repository at this point in the historyBrowse files
Skip the placeholder row when terminate emits nothing and mark the carried
columns nullable, so an empty partition with a non-nullable column no longer
fails. Snapshot generator rows into UnsafeRow so a reused buffer cannot alias
earlier rows, and drop the unused partition index. Lower ReplicateRows to
inline over array_repeat so EXCEPT ALL and INTERSECT ALL skip the JVM. Document
that the JVM generator works a batch at a time. Add regression tests, including
a state check across batch boundaries, and SQL file tests for json_tuple and
EXCEPT ALL. Adapt the stack case to the updated inline and stack PR.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/scala_java_udfs.md
+8-5Lines changed: 8 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -44,18 +44,21 @@ This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabl
44
44
- Aggregate UDFs (`ScalaAggregator`, `TypedImperativeAggregate`, the legacy `UserDefinedAggregateFunction`).
45
45
- Python UDTFs (`@udtf`).
46
46
- Python `@udf` and Pandas `@pandas_udf`.
47
+
- Hive `GenericUDF` and `SimpleUDF`.
48
+
-`NullType` and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
49
+
- Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
50
+
51
+
When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.
47
52
48
53
## Generators
49
54
50
55
A deterministic generator that Comet has no native implementation of, such as a Hive `GenericUDTF` registered with `CREATE FUNCTION` or Spark's `json_tuple`, runs inside Comet's native generate operator. The operator calls the generator on the JVM for each batch and keeps one generator instance per partition, so state the generator holds carries across batches. Once a partition's input ends, the rows of the generator's `terminate()` (a Hive UDTF's `close()`) follow, carrying the partition's last input row, as in Spark.
51
56
52
-
Set `spark.comet.exec.jvmGenerator.enabled` to `false` to run such a generator in Spark instead.
57
+
The operator works a batch at a time rather than a row at a time. It evaluates the generator for every row of a batch, and holds all the rows one input row generates, before it passes any of them on. A `LIMIT` above the generator does not stop evaluation partway through a batch, so a generator that raises an error on a later row can fail a query Spark would have satisfied from earlier rows, and a generator that emits a very large number of rows for one input row needs memory for all of them.
53
58
54
-
- Hive `GenericUDF` and `SimpleUDF`.
55
-
-`NullType` and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
56
-
- Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
59
+
Spark's `EXCEPT ALL` and `INTERSECT ALL` use an internal generator, `ReplicateRows`. Comet runs it natively as `inline` over `array_repeat`, without the JVM.
57
60
58
-
When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.
61
+
Set `spark.comet.exec.jvmGenerator.enabled` to `false` to run such a generator in Spark instead.
0 commit comments