Skip to content

Commit 4c335a8

Browse files
committed
fix: address review of JVM generators
Skip the placeholder row when terminate emits nothing and mark the carried columns nullable, so an empty partition with a non-nullable column no longer fails. Snapshot generator rows into UnsafeRow so a reused buffer cannot alias earlier rows, and drop the unused partition index. Lower ReplicateRows to inline over array_repeat so EXCEPT ALL and INTERSECT ALL skip the JVM. Document that the JVM generator works a batch at a time. Add regression tests, including a state check across batch boundaries, and SQL file tests for json_tuple and EXCEPT ALL. Adapt the stack case to the updated inline and stack PR.
1 parent d7f3e42 commit 4c335a8

9 files changed

Lines changed: 439 additions & 52 deletions

File tree

‎docs/source/user-guide/latest/scala_java_udfs.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,18 +44,21 @@ This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabl
4444
- Aggregate UDFs (`ScalaAggregator`, `TypedImperativeAggregate`, the legacy `UserDefinedAggregateFunction`).
4545
- Python UDTFs (`@udtf`).
4646
- Python `@udf` and Pandas `@pandas_udf`.
47+
- Hive `GenericUDF` and `SimpleUDF`.
48+
- `NullType` and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
49+
- Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
50+
51+
When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.
4752

4853
## Generators
4954

5055
A deterministic generator that Comet has no native implementation of, such as a Hive `GenericUDTF` registered with `CREATE FUNCTION` or Spark's `json_tuple`, runs inside Comet's native generate operator. The operator calls the generator on the JVM for each batch and keeps one generator instance per partition, so state the generator holds carries across batches. Once a partition's input ends, the rows of the generator's `terminate()` (a Hive UDTF's `close()`) follow, carrying the partition's last input row, as in Spark.
5156

52-
Set `spark.comet.exec.jvmGenerator.enabled` to `false` to run such a generator in Spark instead.
57+
The operator works a batch at a time rather than a row at a time. It evaluates the generator for every row of a batch, and holds all the rows one input row generates, before it passes any of them on. A `LIMIT` above the generator does not stop evaluation partway through a batch, so a generator that raises an error on a later row can fail a query Spark would have satisfied from earlier rows, and a generator that emits a very large number of rows for one input row needs memory for all of them.
5358

54-
- Hive `GenericUDF` and `SimpleUDF`.
55-
- `NullType` and `UserDefinedType` arguments and return types. UDT-typed columns fall back to Spark; to keep execution in the Comet pipeline, store and read the underlying representation directly (e.g. write MLlib `Vector` outputs as `Struct<type: Byte, size: Int, indices: Array<Int>, values: Array<Double>>` rather than `VectorUDT`).
56-
- Trees whose total nested-field count (output plus all input columns the UDF tree references) exceeds `spark.sql.codegen.maxFields` (default 100). Comet refuses these at plan time and the operator falls back to Spark.
59+
Spark's `EXCEPT ALL` and `INTERSECT ALL` use an internal generator, `ReplicateRows`. Comet runs it natively as `inline` over `array_repeat`, without the JVM.
5760

58-
When a UDF is rejected, the reason surfaces through Comet's standard fallback diagnostics; the query still runs on Spark.
61+
Set `spark.comet.exec.jvmGenerator.enabled` to `false` to run such a generator in Spark instead.
5962

6063
## Behavior
6164

‎native/core/src/execution/planner/explode.rs‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,8 +139,16 @@ impl OperatorBuilder for ExplodeBuilder {
139139
let mut output_fields: Vec<Field> = Vec::new();
140140

141141
// Add all projection columns (non-array columns)
142+
// The rows `terminate` yields carry the last input row's columns, which are all null when
143+
// the input is empty, so those columns are nullable whatever the input says.
144+
let carried_nullable = explode.terminate.is_some();
142145
for i in 0..projections.len() {
143-
output_fields.push(project_schema.field(i).clone());
146+
let field = project_schema.field(i).clone();
147+
output_fields.push(if carried_nullable {
148+
field.with_nullable(true)
149+
} else {
150+
field
151+
});
144152
}
145153

146154
let array_input_index = if explode.position {

‎native/operators/src/explode.rs‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -524,7 +524,23 @@ impl ExplodeStream {
524524
RecordBatch::try_new_with_options(Arc::new(Schema::empty()), vec![], &options)?;
525525
let rows = terminate.evaluate(&no_columns)?.into_array(1)?;
526526

527-
let input_schema = self.input.schema();
527+
// Nothing to emit, so there is no need to build the placeholder row below. That row's
528+
// columns are null when no input row arrived, which a non-nullable column rejects.
529+
let list = rows.as_list::<i32>();
530+
if list.is_null(0) || list.value_length(0) == 0 {
531+
return Ok(None);
532+
}
533+
534+
// The output schema marks every column terminate may null as nullable (see the planner),
535+
// so the placeholder batch has to as well.
536+
let input_schema = Arc::new(Schema::new(
537+
self.input
538+
.schema()
539+
.fields()
540+
.iter()
541+
.map(|f| f.as_ref().clone().with_nullable(true))
542+
.collect::<Vec<_>>(),
543+
));
528544
let mut columns = match self.last_row.take() {
529545
Some(row) => row.columns().to_vec(),
530546
None => input_schema
@@ -2065,6 +2081,60 @@ mod tests {
20652081
);
20662082
}
20672083

2084+
/// `terminate` over an empty input whose carried column is non-nullable, with the nullable
2085+
/// output column the planner declares for it.
2086+
async fn terminate_over_empty_non_nullable_input(terminate: Vec<i32>) -> usize {
2087+
let input_schema = Arc::new(Schema::new(vec![
2088+
Field::new("id", DataType::Int32, false),
2089+
Field::new("l", DataType::new_list(DataType::Int32, true), true),
2090+
]));
2091+
let source = MemorySourceConfig::try_new_exec(
2092+
&[vec![RecordBatch::new_empty(Arc::clone(&input_schema))]],
2093+
input_schema,
2094+
None,
2095+
)
2096+
.unwrap();
2097+
let output_schema = Arc::new(Schema::new(vec![
2098+
Field::new("id", DataType::Int32, true),
2099+
Field::new("l", DataType::Int32, true),
2100+
]));
2101+
let rows = ListArray::from_iter_primitive::<Int32Type, _, _>(vec![Some(
2102+
terminate.into_iter().map(Some).collect::<Vec<_>>(),
2103+
)]);
2104+
let terminate = Arc::new(datafusion::physical_expr::expressions::Literal::new(
2105+
datafusion::common::ScalarValue::List(Arc::new(rows)),
2106+
));
2107+
let explode = ExplodeExec::new(
2108+
source,
2109+
vec![ListUnnest {
2110+
index_in_input_schema: 1,
2111+
depth: 1,
2112+
}],
2113+
vec![],
2114+
output_schema,
2115+
UnnestOptions::new().with_null_handling(NullHandling::Drop),
2116+
)
2117+
.unwrap()
2118+
.with_terminate(terminate)
2119+
.unwrap();
2120+
datafusion::physical_plan::common::collect(explode.execute(0, Arc::default()).unwrap())
2121+
.await
2122+
.unwrap()
2123+
.iter()
2124+
.map(|b| b.num_rows())
2125+
.sum()
2126+
}
2127+
2128+
#[tokio::test]
2129+
async fn empty_terminate_over_empty_non_nullable_input_adds_nothing() {
2130+
assert_eq!(terminate_over_empty_non_nullable_input(vec![]).await, 0);
2131+
}
2132+
2133+
#[tokio::test]
2134+
async fn terminate_over_empty_non_nullable_input_emits_null_carried_column() {
2135+
assert_eq!(terminate_over_empty_non_nullable_input(vec![7, 8]).await, 2);
2136+
}
2137+
20682138
#[test]
20692139
fn flatten_struct_cols_nulls_fields_under_a_null_struct() {
20702140
// Row 1's struct is null but its fields hold values, which Arrow allows. Spark's `inline`

‎spark/src/main/scala/org/apache/spark/sql/comet/CometGeneratorUDF.scala‎

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ import org.apache.arrow.vector.{BitVector, ValueVector, VarBinaryVector}
3131
import org.apache.arrow.vector.types.pojo.Field
3232
import org.apache.spark.SparkEnv
3333
import org.apache.spark.sql.catalyst.InternalRow
34-
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference, AttributeSeq, BindReferences, Expression, Generator, GenericInternalRow, Literal, Nondeterministic, Unevaluable}
34+
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference, AttributeSeq, BindReferences, Expression, Generator, GenericInternalRow, Literal, Unevaluable, UnsafeProjection}
3535
import org.apache.spark.sql.catalyst.util.GenericArrayData
3636
import org.apache.spark.sql.comet.execution.arrow.ArrowWriter
3737
import org.apache.spark.sql.execution.ExecSubqueryExpression
@@ -84,15 +84,17 @@ class CometGeneratorUDF extends CometUDF {
8484
val digest = ByteBuffer.wrap(binaryScalar(inputs(0)))
8585
val bound = generators.getOrElseUpdate(
8686
(digest, planId),
87-
BoundGenerator(deserialize(binaryScalar(inputs(1))), partitionIndex))
87+
new BoundGenerator(deserialize(binaryScalar(inputs(1)))))
8888

8989
val output = CometBatchKernelCodegen.allocateOutput(bound.outputField, numRows, 0)
9090
try {
9191
val writer = ArrowWriter.createFieldWriter(output)
9292
val outputRow = new GenericInternalRow(1)
9393
def writeRows(rows: Any): Unit = {
94+
// A generator may reuse a row's buffers from one `next` to the next, which
95+
// `GenericInternalRow.copy` shares rather than copies.
9496
val copies = new ArrayBuffer[Any]
95-
iteratorOf(rows).foreach(row => copies += row.copy())
97+
iteratorOf(rows).foreach(row => copies += bound.snapshot(row))
9698
outputRow.update(0, new GenericArrayData(copies.toArray))
9799
writer.write(outputRow, 0)
98100
}
@@ -217,16 +219,13 @@ object CometGeneratorUDF {
217219
/**
218220
* A generator ready to run in one native plan, and the Arrow field its output is exported as.
219221
*/
220-
private case class BoundGenerator(generator: Generator, outputField: Field)
222+
private class BoundGenerator(val generator: Generator) {
223+
val outputField: Field = CometGeneratorUDF.outputField(generator)
221224

222-
private object BoundGenerator {
223-
def apply(generator: Generator, partitionIndex: Int): BoundGenerator = {
224-
generator.foreach {
225-
case n: Nondeterministic => n.initialize(partitionIndex)
226-
case _ =>
227-
}
228-
BoundGenerator(generator, outputField(generator))
229-
}
225+
private val projection = UnsafeProjection.create(generator.elementSchema)
226+
227+
/** A copy of `row` that owns its bytes, so it survives the generator reusing its buffers. */
228+
def snapshot(row: InternalRow): InternalRow = projection(row).copy()
230229
}
231230

232231
/** The type of each element of the output: the rows one input row generates. */

‎spark/src/main/scala/org/apache/spark/sql/comet/operators.scala‎

Lines changed: 43 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ import org.apache.spark.broadcast.Broadcast
2929
import org.apache.spark.internal.Logging
3030
import org.apache.spark.rdd.RDD
3131
import org.apache.spark.sql.catalyst.InternalRow
32-
import org.apache.spark.sql.catalyst.expressions.{Ascending, Attribute, AttributeSeq, AttributeSet, CodegenObjectFactoryMode, CreateArray, CreateNamedStruct, ExplodeBase, Expression, ExpressionSet, Generator, Inline, LeafExpression, Literal, NamedExpression, SortOrder, Stack, XXH64}
32+
import org.apache.spark.sql.catalyst.expressions.{ArrayRepeat, Ascending, Attribute, AttributeSeq, AttributeSet, Cast, CodegenObjectFactoryMode, CreateArray, CreateNamedStruct, ExplodeBase, Expression, ExpressionSet, Generator, GreaterThan, If, Inline, LeafExpression, Literal, NamedExpression, ReplicateRows, SortOrder, Stack, XXH64}
3333
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, AggregateMode, Average, CollectList, CollectSet, Final, ImperativeAggregate, Mode, Partial, PartialMerge, Percentile, Sum}
3434
import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
3535
import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight, BuildSide}
@@ -60,7 +60,7 @@ import org.apache.comet.{CometConf, CometExecIterator, CometRuntimeException, Co
6060
import org.apache.comet.CometSparkSessionExtensions.{isCometShuffleEnabled, isSpark35Plus, withFallbackReason}
6161
import org.apache.comet.parquet.{CometParquetUtils, HadoopFileOutput}
6262
import org.apache.comet.rules.CometExecRule
63-
import org.apache.comet.serde.{CometCollectBuffer, CometOperatorSerde, Compatible, OperatorOuterClass, QueryContextInterner, SupportLevel, Unsupported}
63+
import org.apache.comet.serde.{CometCollectBuffer, CometOperatorSerde, Compatible, ExprOuterClass, OperatorOuterClass, QueryContextInterner, SupportLevel, Unsupported}
6464
import org.apache.comet.serde.OperatorOuterClass.{AggregateMode => CometAggregateMode, Operator}
6565
import org.apache.comet.serde.QueryPlanSerde
6666
import org.apache.comet.serde.QueryPlanSerde.{aggExprToProto, exprToProto, hasNonDefaultStringCollation, isStringCollationType, supportedSortType}
@@ -1985,27 +1985,18 @@ object CometExplodeExec extends CometOperatorSerde[GenerateExec] {
19851985
exprToProto(inline.child, op.child.output)
19861986
case stack: Stack =>
19871987
explodeBuilder.setInline(true)
1988-
val array = stackAsStructArray(stack)
1989-
exprToProto(array, op.child.output).orElse {
1990-
// The array of structs is built here rather than taken from the plan, so carry any
1991-
// reason recorded on the built nodes to the generator, where the operator collects it.
1992-
liftFallbackReasons(array, op.generator)
1993-
None
1994-
}
1988+
exprToProto(stackAsStructArray(stack), op.child.output)
19951989
case explode: ExplodeBase =>
19961990
explodeBuilder.setPosition(explode.position)
19971991
exprToProto(explode.child, op.child.output)
1998-
case generator =>
1999-
// The JVM generator yields, for each input row, an array of the structs it emits.
1992+
case rows: ReplicateRows =>
20001993
explodeBuilder.setInline(true)
2001-
CometGeneratorUDF.calls(generator, op.child.output) match {
2002-
case Right((process, terminate)) =>
2003-
explodeBuilder.setTerminate(terminate)
2004-
Some(process)
2005-
case Left(reason) =>
2006-
withFallbackReason(op, reason)
2007-
None
2008-
}
1994+
// The rewrite is native end to end, so it is preferred to the JVM generator.
1995+
exprToProto(replicateRowsAsStructArray(rows), op.child.output)
1996+
.orElse(jvmGeneratorCall(op, explodeBuilder))
1997+
case _ =>
1998+
explodeBuilder.setInline(true)
1999+
jvmGeneratorCall(op, explodeBuilder)
20092000
}
20102001

20112002
if (childExprProto.isEmpty) {
@@ -2029,6 +2020,39 @@ object CometExplodeExec extends CometOperatorSerde[GenerateExec] {
20292020
Some(builder.setExplode(explodeBuilder).build())
20302021
}
20312022

2023+
/**
2024+
* The expression that runs `op.generator` on the JVM, which yields for each input row an array
2025+
* of the structs it emits, and sets the one that runs its `terminate` on `explodeBuilder`.
2026+
*/
2027+
private def jvmGeneratorCall(
2028+
op: GenerateExec,
2029+
explodeBuilder: OperatorOuterClass.Explode.Builder): Option[ExprOuterClass.Expr] =
2030+
CometGeneratorUDF.calls(op.generator, op.child.output) match {
2031+
case Right((process, terminate)) =>
2032+
explodeBuilder.setTerminate(terminate)
2033+
Some(process)
2034+
case Left(reason) =>
2035+
withFallbackReason(op, reason)
2036+
None
2037+
}
2038+
2039+
/**
2040+
* `ReplicateRows(n, v1, ..., vk)` emits `n` copies of the row `(v1, ..., vk)`, which is
2041+
* `inline` over `array_repeat` of that row as a struct. A NULL or negative `n` gives no rows in
2042+
* both. The count is a `Long` and `array_repeat` takes an `Int`, so a count beyond
2043+
* `Int.MaxValue` is clamped rather than wrapped; no array that long fits in memory.
2044+
*/
2045+
private def replicateRowsAsStructArray(rows: ReplicateRows): Expression = {
2046+
val count = rows.children.head
2047+
val row = CreateNamedStruct(rows.elementSchema.fields.toSeq.zip(rows.children.tail).flatMap {
2048+
case (field, value) => Seq(Literal(field.name), value)
2049+
})
2050+
val maxInt = Literal(Int.MaxValue.toLong)
2051+
ArrayRepeat(
2052+
row,
2053+
If(GreaterThan(count, maxInt), Literal(Int.MaxValue), Cast(count, IntegerType)))
2054+
}
2055+
20322056
/**
20332057
* `stack(n, v1, ..., vk)` emits `n` rows of `ceil(k / n)` columns, filled row by row and padded
20342058
* with nulls. That is `inline` over an array holding one struct per output row.
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- json_tuple has no native implementation. It runs on the JVM inside the native generate operator,
19+
-- one row per input row, with NULL for a field the document lacks and a row of NULLs for a NULL or
20+
-- invalid document.
21+
22+
statement
23+
CREATE TABLE test_json_tuple(id int, js string) USING parquet
24+
25+
statement
26+
INSERT INTO test_json_tuple VALUES
27+
(1, '{"a": 1, "b": "x", "c": true}'),
28+
(2, '{"a": null, "b": "y"}'),
29+
(3, '{"b": "multi word", "d": {"e": 5}}'),
30+
(4, '{"a": [1, 2, 3], "b": 1.5}'),
31+
(5, 'not json'),
32+
(6, ''),
33+
(7, NULL),
34+
(8, '{"a": "é日本", "b": ""}'),
35+
(NULL, '{"a": 9}')
36+
37+
query
38+
SELECT id, json_tuple(js, 'a', 'b') FROM test_json_tuple
39+
40+
query
41+
SELECT json_tuple(js, 'a', 'b', 'c', 'd') FROM test_json_tuple
42+
43+
-- a field the documents lack, and a repeated field
44+
query
45+
SELECT id, json_tuple(js, 'missing', 'a', 'a') FROM test_json_tuple
46+
47+
query
48+
SELECT id, a, b FROM test_json_tuple LATERAL VIEW json_tuple(js, 'a', 'b') j AS a, b
49+
50+
-- the outer form keeps the rows that yield no values
51+
query
52+
SELECT id, a FROM test_json_tuple LATERAL VIEW OUTER json_tuple(js, 'a') j AS a
53+
54+
-- a filter and an aggregate above the generator
55+
query
56+
SELECT id, a FROM test_json_tuple LATERAL VIEW json_tuple(js, 'a') j AS a WHERE a IS NOT NULL
57+
58+
query
59+
SELECT b, count(*) FROM test_json_tuple LATERAL VIEW json_tuple(js, 'b') j AS b GROUP BY b

0 commit comments

Comments
 (0)