Skip to content

Commit fd8bb59

Browse files
authored
fix: array expression audit follow-ups (#4503) (#4713)
* fix: array expression audit follow-ups (#4503) Address the actionable array expression audit follow-ups from #4503: - ArrayReverse: report Incompatible for struct/map element types so they route through the JVM codegen dispatcher and stay native, instead of reporting Compatible while convert silently declined and fell back. - SortArray: accept any foldable boolean ascendingOrder (Spark 4.0+ widens it beyond a boolean Literal) and evaluate it at convert time. - ArrayJoin: fall back for non-default string collations, mirroring ArrayIntersect, so the limitation surfaces in EXPLAIN. - ArrayExcept: surface the native element-type restriction in getSupportLevel rather than only in convert. - Remove the dead ArrayCompact serde registration (RuntimeReplaceable in all supported versions, dispatched via ArrayFilter) and document ArrayAppend as reachable only on Spark 3.x. Item 2 (NaN/signed-zero gating for array_contains/distinct/union/max/min) is intentionally omitted: existing SQL tests show native DataFusion already canonicalizes NaN and signed zero to match Spark, so reporting Incompatible would force a needless fallback to the codegen dispatcher with no correctness benefit. Item 5 (array_union ordering) is tracked in #4681. * refactor: simplify CometArrayExcept support-level and convert guards Collapse the dual-Incompatible match in getSupportLevel into a single Incompatible construction, and tighten the convert-time type guard to the same find idiom (dropping a needless toSet). Behavior preserving. * fix: keep collated array_join and array_intersect native via codegen dispatch Address review feedback on #4713: report Incompatible instead of Unsupported for non-default collations so both route through the JVM codegen dispatcher (Spark's own doGenCode) and stay native while matching Spark, reconciling the three #2190 collation cases with CometReverse. array_join concatenation is collation-independent, so only the output string's collation metadata is dropped. array_intersect membership is collation-aware, so the dispatcher (not the byte-based native path) produces the correct result. Add collation SQL tests for both.
1 parent 464afe1 commit fd8bb59

6 files changed

Lines changed: 257 additions & 33 deletions

File tree

‎spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,13 @@ import org.apache.comet.shims.{CometExprShim, CometTypeShim}
5151
object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
5252

5353
private[comet] val arrayExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(
54+
// ArrayAppend is a concrete expression only on Spark 3.x. Spark 4.0+ marks it
55+
// RuntimeReplaceable and rewrites it to ArrayInsert before serde, so this entry is
56+
// unreachable there and is kept solely for Spark 3.4/3.5.
5457
classOf[ArrayAppend] -> CometArrayAppend,
55-
classOf[ArrayCompact] -> CometArrayCompact,
58+
// ArrayCompact is RuntimeReplaceable in all supported Spark versions (rewritten to
59+
// ArrayFilter(arr, IsNotNull(...))), so it never reaches serde directly; dispatch flows
60+
// through CometArrayFilter -> CometArrayCompact instead. No direct registration here.
5661
classOf[ArrayContains] -> CometArrayContains,
5762
classOf[ArrayDistinct] -> CometScalarFunction("array_distinct"),
5863
classOf[ArrayExcept] -> CometArrayExcept,

‎spark/src/main/scala/org/apache/comet/serde/arrays.scala‎

Lines changed: 72 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -152,9 +152,11 @@ object CometSortArray extends CometExpressionSerde[SortArray] with CodegenDispat
152152
.strictFloatingPointReason(elementType, "Sorting on floating-point")
153153
.map(reason => Incompatible(Some(reason)))
154154
.getOrElse(expr.ascendingOrder match {
155-
case Literal(_: Boolean, BooleanType) => Compatible()
155+
// Spark 3.x requires a boolean Literal; Spark 4.0+ widens ascendingOrder to any
156+
// foldable boolean. Accept both; convert evaluates the foldable expression.
157+
case ao if ao.foldable && ao.dataType == BooleanType => Compatible()
156158
case other =>
157-
Unsupported(Some(s"ascendingOrder must be a boolean literal: $other"))
159+
Unsupported(Some(s"ascendingOrder must be a foldable boolean: $other"))
158160
})
159161
}
160162
}
@@ -164,17 +166,13 @@ object CometSortArray extends CometExpressionSerde[SortArray] with CodegenDispat
164166
inputs: Seq[Attribute],
165167
binding: Boolean): Option[ExprOuterClass.Expr] = {
166168
val arrayExprProto = exprToProtoInternal(expr.base, inputs, binding)
167-
val (sortDirectionExprProto, nullOrderingExprProto) = expr.ascendingOrder match {
168-
case Literal(value: Boolean, BooleanType) =>
169-
val direction = if (value) "ASC" else "DESC"
170-
val nullOrdering = if (value) "NULLS FIRST" else "NULLS LAST"
171-
(
172-
exprToProtoInternal(Literal(direction), inputs, binding),
173-
exprToProtoInternal(Literal(nullOrdering), inputs, binding))
174-
case _ =>
175-
// Unreachable: getSupportLevel gates a non-boolean-literal ascendingOrder.
176-
(None, None)
177-
}
169+
// ascendingOrder is a foldable boolean (gated in getSupportLevel). Evaluate it; a null result
170+
// unboxes to false, matching Spark's `right.eval().asInstanceOf[Boolean]`.
171+
val ascending = expr.ascendingOrder.eval(EmptyRow).asInstanceOf[Boolean]
172+
val direction = if (ascending) "ASC" else "DESC"
173+
val nullOrdering = if (ascending) "NULLS FIRST" else "NULLS LAST"
174+
val sortDirectionExprProto = exprToProtoInternal(Literal(direction), inputs, binding)
175+
val nullOrderingExprProto = exprToProtoInternal(Literal(nullOrdering), inputs, binding)
178176

179177
val sortArrayScalarExpr =
180178
scalarFunctionExprToProto(
@@ -195,16 +193,20 @@ object CometArrayIntersect
195193
"Result array element order may differ from Spark when the right array is longer " +
196194
"than the left (DataFusion probes the longer side)."
197195

198-
private val unsupportedCollationReason: String =
199-
"array_intersect on collated strings is not supported."
200-
201-
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason)
196+
private val collationReason: String =
197+
"array_intersect does not propagate non-UTF8_BINARY collations to the output array elements " +
198+
"(https://github.com/apache/datafusion-comet/issues/2190)"
202199

203-
override def getUnsupportedReasons(): Seq[String] = Seq(unsupportedCollationReason)
200+
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason, collationReason)
204201

205202
override def getSupportLevel(expr: ArrayIntersect): SupportLevel = {
203+
// The native array_intersect dedups by raw bytes, which is wrong under non-default collations,
204+
// so report Incompatible rather than Unsupported: the JVM codegen dispatcher (Spark's own
205+
// doGenCode) performs collation-aware set membership and keeps execution native, matching
206+
// Spark. Only the output elements' collation metadata is dropped, consistent with CometReverse
207+
// and CometArrayJoin.
206208
if (hasNonDefaultStringCollation(expr.dataType)) {
207-
Unsupported(Some(unsupportedCollationReason))
209+
Incompatible(Some(collationReason))
208210
} else {
209211
Incompatible(Some(incompatReason))
210212
}
@@ -339,8 +341,17 @@ object CometArrayExcept
339341

340342
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason)
341343

342-
override def getSupportLevel(expr: ArrayExcept): SupportLevel = Incompatible(
343-
Some(incompatReason))
344+
override def getSupportLevel(expr: ArrayExcept): SupportLevel = {
345+
// Surface the native element-type restriction in EXPLAIN. We report Incompatible (not
346+
// Unsupported) for these types so the JVM codegen dispatcher still evaluates them natively
347+
// under the default config; the convert-time guard below is only reached under
348+
// allowIncompatible=true, where the native array_except cannot handle them.
349+
val reason = expr.children.map(_.dataType).find(dt => !isTypeSupported(dt)) match {
350+
case Some(dt) => s"native array_except does not support element type $dt: $incompatReason"
351+
case None => incompatReason
352+
}
353+
Incompatible(Some(reason))
354+
}
344355

345356
@tailrec
346357
def isTypeSupported(dt: DataType): Boolean = {
@@ -361,12 +372,14 @@ object CometArrayExcept
361372
expr: ArrayExcept,
362373
inputs: Seq[Attribute],
363374
binding: Boolean): Option[ExprOuterClass.Expr] = {
364-
val inputTypes = expr.children.map(_.dataType).toSet
365-
for (dt <- inputTypes) {
366-
if (!isTypeSupported(dt)) {
375+
// Defensive: only reached under allowIncompatible=true (the default-config Incompatible path
376+
// routes through the codegen dispatcher before convert). Native array_except cannot handle
377+
// these element types, so decline and let Spark evaluate.
378+
expr.children.map(_.dataType).find(dt => !isTypeSupported(dt)) match {
379+
case Some(dt) =>
367380
withFallbackReason(expr, s"data type not supported: $dt")
368381
return None
369-
}
382+
case None =>
370383
}
371384
val leftArrayExprProto = exprToProto(expr.left, inputs, binding)
372385
val rightArrayExprProto = exprToProto(expr.right, inputs, binding)
@@ -377,13 +390,31 @@ object CometArrayExcept
377390
}
378391
}
379392

380-
object CometArrayJoin extends CometExpressionSerde[ArrayJoin] with CodegenDispatchFallback {
393+
object CometArrayJoin
394+
extends CometExpressionSerde[ArrayJoin]
395+
with CometTypeShim
396+
with CodegenDispatchFallback {
381397

382398
private val incompatReason = "Null handling may differ from Spark"
383399

384-
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason)
400+
private val collationReason =
401+
"array_join does not propagate non-UTF8_BINARY collations to the output string " +
402+
"(https://github.com/apache/datafusion-comet/issues/2190)"
385403

386-
override def getSupportLevel(expr: ArrayJoin): SupportLevel = Incompatible(Some(incompatReason))
404+
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason, collationReason)
405+
406+
override def getSupportLevel(expr: ArrayJoin): SupportLevel = {
407+
// Spark 4.0 widens ArrayJoin's input to StringTypeWithCollation. Concatenation itself is
408+
// collation-independent, so the joined value is always correct; only the output string's
409+
// collation metadata is dropped (Comet columns are UTF8_BINARY). Report Incompatible rather
410+
// than Unsupported so the JVM codegen dispatcher (Spark's own doGenCode) keeps collated
411+
// array_join native and matching Spark, consistent with CometReverse's #2190 handling.
412+
if (hasNonDefaultStringCollation(expr.array.dataType)) {
413+
Incompatible(Some(collationReason))
414+
} else {
415+
Incompatible(Some(incompatReason))
416+
}
417+
}
387418

388419
override def convert(
389420
expr: ArrayJoin,
@@ -592,22 +623,31 @@ object CometGetArrayItem extends CometExpressionSerde[GetArrayItem] {
592623
}
593624

594625
object CometArrayReverse extends CometExpressionSerde[Reverse] with ArraysBase {
595-
val unsupportedReason = "reverse on array containing binary is not supported"
626+
val unsupportedReason =
627+
"native reverse does not support arrays whose element type contains binary, struct, or map"
596628

597629
override def getIncompatibleReasons(): Seq[String] = Seq(unsupportedReason)
598630

599631
override def getSupportLevel(expr: Reverse): SupportLevel = {
600-
if (SupportLevel.containsType(expr.child.dataType, classOf[BinaryType])) {
601-
Incompatible(Some(unsupportedReason))
602-
} else {
632+
// Mirror the native impl's element-type support. Report Incompatible (not Unsupported) for
633+
// element types the native array_reverse cannot handle so the expression routes through the
634+
// JVM codegen dispatcher (via CometReverse, which mixes in CodegenDispatchFallback) instead
635+
// of silently falling back to Spark. Previously StructType reported Compatible here while
636+
// convert rejected it, so such arrays silently fell back.
637+
if (isTypeSupported(expr.child.dataType)) {
603638
Compatible(None)
639+
} else {
640+
Incompatible(Some(unsupportedReason))
604641
}
605642
}
606643

607644
override def convert(
608645
expr: Reverse,
609646
inputs: Seq[Attribute],
610647
binding: Boolean): Option[ExprOuterClass.Expr] = {
648+
// Defensive: only reached under allowIncompatible=true (the default-config Incompatible path
649+
// routes through the codegen dispatcher before convert). Native array_reverse cannot handle
650+
// these element types, so decline and let Spark evaluate.
611651
if (!isTypeSupported(expr.child.dataType)) {
612652
withFallbackReason(expr, s"child data type not supported: ${expr.child.dataType}")
613653
return None
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- MinSparkVersion: 4.0
19+
20+
-- Spark 4.0+ widens ArrayIntersect's input to collated strings. Set membership is collation-aware,
21+
-- so Comet's native byte-based dedup would be wrong. Comet instead routes collated array_intersect
22+
-- through the JVM codegen dispatcher (Spark's own doGenCode), which performs collation-aware
23+
-- membership and matches Spark. Only the output elements' collation metadata is dropped (#2190).
24+
25+
-- UTF8_LCASE: 'A' and 'a' are equal, so the intersection is non-empty and case is taken from the
26+
-- left array.
27+
query
28+
SELECT array_intersect(array('A' COLLATE UTF8_LCASE, 'b' COLLATE UTF8_LCASE), array('a' COLLATE UTF8_LCASE))
29+
30+
-- UTF8_LCASE with duplicates and a non-matching element
31+
query
32+
SELECT array_intersect(array('Hello' COLLATE UTF8_LCASE, 'WORLD' COLLATE UTF8_LCASE), array('hello' COLLATE UTF8_LCASE, 'x' COLLATE UTF8_LCASE))
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- MinSparkVersion: 4.0
19+
20+
-- Spark 4.0+ widens ArrayJoin's input to collated strings. Concatenation is collation-independent,
21+
-- so the joined value is always correct; Comet routes collated array_join through the JVM codegen
22+
-- dispatcher (Spark's own doGenCode), keeping it native and matching Spark. Only the output
23+
-- string's collation metadata is dropped (tracked by #2190).
24+
25+
-- collated array elements
26+
query
27+
SELECT array_join(array('a' COLLATE UTF8_LCASE, 'b' COLLATE UTF8_LCASE), ',')
28+
29+
-- collated elements with null replacement
30+
query
31+
SELECT array_join(array('a' COLLATE UTF8_LCASE, NULL, 'c' COLLATE UTF8_LCASE), ',', 'NULL')
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- ===== INT arrays (native array_reverse) =====
19+
20+
statement
21+
CREATE TABLE test_array_reverse_int(arr array<int>) USING parquet
22+
23+
statement
24+
INSERT INTO test_array_reverse_int VALUES
25+
(array(1, 2, 3)),
26+
(array(1, NULL, 3)),
27+
(array()),
28+
(NULL)
29+
30+
query
31+
SELECT reverse(arr) FROM test_array_reverse_int
32+
33+
-- ===== STRING arrays =====
34+
35+
statement
36+
CREATE TABLE test_array_reverse_string(arr array<string>) USING parquet
37+
38+
statement
39+
INSERT INTO test_array_reverse_string VALUES
40+
(array('a', 'b', 'c')),
41+
(array('a', NULL, 'c')),
42+
(array()),
43+
(NULL)
44+
45+
query
46+
SELECT reverse(arr) FROM test_array_reverse_string
47+
48+
-- ===== STRUCT arrays =====
49+
-- Native array_reverse cannot handle struct element types, so getSupportLevel reports
50+
-- Incompatible and Comet routes these through the JVM codegen dispatcher, staying native instead
51+
-- of silently falling back to Spark.
52+
53+
statement
54+
CREATE TABLE test_array_reverse_struct(c1 struct<a:int, b:string>, c2 struct<a:int, b:string>)
55+
USING parquet
56+
57+
statement
58+
INSERT INTO test_array_reverse_struct VALUES
59+
(named_struct('a', 1, 'b', 'x'), named_struct('a', 2, 'b', 'y')),
60+
(named_struct('a', 3, 'b', NULL), named_struct('a', 4, 'b', 'z')),
61+
(NULL, named_struct('a', 5, 'b', 'w'))
62+
63+
query
64+
SELECT reverse(array(c1, c2)) FROM test_array_reverse_struct
65+
66+
query
67+
SELECT reverse(array(array(c1), array(c2))) FROM test_array_reverse_struct
68+
69+
-- literal arguments (constant folding is disabled by the suite, so these run natively)
70+
query
71+
SELECT
72+
reverse(array(1, 2, 3)),
73+
reverse(array('a', 'b', 'c')),
74+
reverse(array(named_struct('a', 1), named_struct('a', 2))),
75+
reverse(cast(NULL as array<int>))
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- MinSparkVersion: 4.0
19+
20+
-- Spark 4.0+ widens SortArray.ascendingOrder from a boolean Literal to any foldable boolean
21+
-- expression. Spark 3.x rejects a non-Literal ascendingOrder at analysis, so this file is gated
22+
-- to 4.0+. CometSqlFileTestSuite disables ConstantFolding, so the foldable casts below reach the
23+
-- serde unfolded (as Cast nodes, not Literals) and exercise the convert-time evaluation path.
24+
25+
statement
26+
CREATE TABLE test_sort_array_foldable(arr array<int>) USING parquet
27+
28+
statement
29+
INSERT INTO test_sort_array_foldable VALUES
30+
(array(3, 1, 4, 1, 5)),
31+
(array(3, NULL, 1, NULL, 2)),
32+
(array()),
33+
(NULL)
34+
35+
-- foldable ascending (cast(1 as boolean) => true)
36+
query
37+
SELECT sort_array(arr, cast(1 as boolean)) FROM test_sort_array_foldable
38+
39+
-- foldable descending (cast(0 as boolean) => false)
40+
query
41+
SELECT sort_array(arr, cast(0 as boolean)) FROM test_sort_array_foldable

0 commit comments

Comments
 (0)