Skip to content

Commit ee57cac

Browse files
committed
fix: dispatch collated array extrema through Spark codegen
1 parent 5522ab7 commit ee57cac

8 files changed

Lines changed: 109 additions & 26 deletions

File tree

‎docs/source/contributor-guide/expression-audits/array_funcs.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,15 +95,15 @@
9595
- Spark 3.5.8 (audited 2026-08-22): `ArrayMax` skips NULL elements and returns NULL for an empty or all-NULL array. `SQLOrderingUtil` treats all NaNs as equal and greater than non-NaN values, and signed zeros as equal. The first equal maximum is retained. Nested arrays and structs compare lexicographically, with NULL fields or elements ordered first.
9696
- Spark 4.0.1 (audited 2026-08-22): `NullIntolerant` becomes a `nullIntolerant` field. Extrema semantics are unchanged; string ordering can use non-default collations.
9797
- Spark 4.1.1 (audited 2026-08-22): identical to 4.0.1.
98-
- Current status: `CometArrayMax` uses the native `SparkArrayExtrema` UDF. Typed float/double scans and recursive array/struct comparisons follow Spark's ordering and preserve the original first equal element, including its zero sign and NaN representation. This path is used in both strict and non-strict floating-point modes without the JVM codegen dispatcher. Other scalar element types retain the existing DataFusion implementation. Non-UTF8_BINARY string collations, including nested fields, are flagged `Incompatible` ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
98+
- Current status: `CometArrayMax` uses the native `SparkArrayExtrema` UDF. Typed float/double scans and recursive array/struct comparisons follow Spark's ordering and preserve the original first equal element, including its zero sign and NaN representation. This path is used in both strict and non-strict floating-point modes without the JVM codegen dispatcher. Other scalar element types retain the existing DataFusion implementation. Non-UTF8_BINARY string collations, including nested fields, use Spark's JVM codegen dispatcher inside the Comet pipeline by default. If the dispatcher is disabled, these cases fall back to Spark unless incompatible native execution is explicitly enabled ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
9999

100100
## array_min
101101

102102
- Spark 3.4.3 (audited 2026-08-22): identical to 3.5.8.
103103
- Spark 3.5.8 (audited 2026-08-22): mirrors `ArrayMax`, retaining the first equal minimum. The NULL, NaN, signed-zero, and nested comparison rules are the same.
104104
- Spark 4.0.1 (audited 2026-08-22): same trait refactor and collation support as `array_max`, with no change in floating-point extrema semantics.
105105
- Spark 4.1.1 (audited 2026-08-22): identical to 4.0.1.
106-
- Current status: `CometArrayMin` shares the native `SparkArrayExtrema` implementation and support boundary with `array_max`. Both floating-point modes use Spark-compatible native ordering, preserving the original first equal minimum. Non-default string collations remain `Incompatible` ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
106+
- Current status: `CometArrayMin` shares the native `SparkArrayExtrema` implementation and support boundary with `array_max`. Both floating-point modes use Spark-compatible native ordering, preserving the original first equal minimum. Non-default string collations use the same JVM codegen dispatch and dispatcher-disabled fallback as `array_max` ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
107107

108108
## array_position
109109

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -156,8 +156,8 @@ The tables below list every Spark built-in expression with its current status.
156156
| `array_insert` | ✅ | Native | |
157157
| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
158158
| `array_join` | ✅ | Hybrid | Native for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) |
159-
| `array_max` | ✅ | Native | Spark-compatible floating-point and nested ordering; non-default string collations fall back ([details](compatibility/expressions/array.md)) |
160-
| `array_min` | ✅ | Native | Spark-compatible floating-point and nested ordering; non-default string collations fall back ([details](compatibility/expressions/array.md)) |
159+
| `array_max` | ✅ | Hybrid | Native Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) |
160+
| `array_min` | ✅ | Hybrid | Native Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) |
161161
| `array_position` | ✅ | Native | Binary/struct/map/null elements fall back |
162162
| `array_prepend` | ✅ | — | |
163163
| `array_remove` | ✅ | Native | |

‎native/spark-expr/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -425,4 +425,4 @@ harness = false
425425

426426
[[bench]]
427427
name = "json_array_length"
428-
harness = false
428+
harness = false

‎spark/src/main/scala/org/apache/comet/serde/arrays.scala‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -247,14 +247,18 @@ private object ArrayExtremaSupport extends CometTypeShim {
247247

248248
def getSupportLevel(elementType: DataType): SupportLevel = {
249249
if (hasNonDefaultStringCollation(elementType)) {
250+
// The dispatcher runs Spark's own comparison with the original collation IDs, including
251+
// strings nested in arrays or structs. Keep the native bytewise comparison opt-in only.
250252
Incompatible(Some(incompatReason))
251253
} else {
252254
Compatible()
253255
}
254256
}
255257
}
256258

257-
object CometArrayMax extends CometExpressionSerde[ArrayMax] {
259+
object CometArrayMax extends CometExpressionSerde[ArrayMax] with CodegenDispatchFallback {
260+
override def hasConditionalNativeDefault: Boolean = true
261+
258262
override def getIncompatibleReasons(): Seq[String] = Seq(ArrayExtremaSupport.incompatReason)
259263

260264
override def getSupportLevel(expr: ArrayMax): SupportLevel =
@@ -272,7 +276,9 @@ object CometArrayMax extends CometExpressionSerde[ArrayMax] {
272276
}
273277
}
274278

275-
object CometArrayMin extends CometExpressionSerde[ArrayMin] {
279+
object CometArrayMin extends CometExpressionSerde[ArrayMin] with CodegenDispatchFallback {
280+
override def hasConditionalNativeDefault: Boolean = true
281+
276282
override def getIncompatibleReasons(): Seq[String] = Seq(ArrayExtremaSupport.incompatReason)
277283

278284
override def getSupportLevel(expr: ArrayMin): SupportLevel =

‎spark/src/test/resources/sql-tests/expressions/array/array_max_collation.sql‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,10 @@
1616
-- under the License.
1717

1818
-- MinSparkVersion: 4.0
19-
-- Non-binary collations still require Spark's ordering, including when nested.
20-
-- Neither strict floating-point mode nor enabling the dispatcher may bypass this guard.
19+
-- Non-binary collations use Spark's ordering through the dispatcher, including when nested.
2120
-- Config: spark.comet.expression.ArrayMax.allowIncompatible=false
2221
-- ConfigMatrix: spark.comet.exec.strictFloatingPoint=false,true
23-
-- ConfigMatrix: spark.comet.exec.scalaUDF.codegen.enabled=false,true
22+
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
2423

2524
statement
2625
CREATE TABLE test_array_max_collation(
@@ -36,36 +35,40 @@ INSERT INTO test_array_max_collation VALUES
3635
(6, NULL, NULL, NULL, NULL)
3736

3837
-- Binary string ordering remains native, including inside arrays and structs.
39-
query
38+
query expect_native(array_max)
4039
SELECT id, array_max(array(a, b)) FROM test_array_max_collation
4140

42-
query
41+
query expect_native(array_max)
4342
SELECT id, array_max(array(array(a), array(b))),
4443
array_max(array(named_struct('s', a, 'f', x), named_struct('s', b, 'f', y)))
4544
FROM test_array_max_collation
4645

46+
-- Enabling the dispatcher must not move floating-point extrema off the native path.
47+
query expect_native(array_max)
48+
SELECT id, array_max(array(x, y)) FROM test_array_max_collation
49+
4750
-- Lowercase ordering differs from binary ordering for the column values 'a' and 'B'.
48-
query expect_fallback(Array extrema use binary string ordering)
51+
query expect_dispatch(array_max)
4952
SELECT id, array_max(array(
5053
CAST(a AS STRING COLLATE UTF8_LCASE),
5154
CAST(b AS STRING COLLATE UTF8_LCASE)))
5255
FROM test_array_max_collation
5356

54-
query expect_fallback(Array extrema use binary string ordering)
57+
query expect_dispatch(array_max)
5558
SELECT id, array_max(array(
5659
array(CAST(a AS STRING COLLATE UTF8_LCASE)),
5760
array(CAST(b AS STRING COLLATE UTF8_LCASE))))
5861
FROM test_array_max_collation
5962

6063
-- A floating-point field does not make a collated struct eligible for native ordering.
61-
query expect_fallback(Array extrema use binary string ordering)
64+
query expect_dispatch(array_max)
6265
SELECT id, array_max(array(
6366
named_struct('s', CAST(a AS STRING COLLATE UTF8_LCASE), 'f', x),
6467
named_struct('s', CAST(b AS STRING COLLATE UTF8_LCASE), 'f', y)))
6568
FROM test_array_max_collation
6669

6770
-- The collation check must recurse through both struct fields and nested array elements.
68-
query expect_fallback(Array extrema use binary string ordering)
71+
query expect_dispatch(array_max)
6972
SELECT id, array_max(array(
7073
named_struct('s', array(CAST(a AS STRING COLLATE UTF8_LCASE)), 'f', x),
7174
named_struct('s', array(CAST(b AS STRING COLLATE UTF8_LCASE)), 'f', y)))

‎spark/src/test/resources/sql-tests/expressions/array/array_min_collation.sql‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,10 @@
1616
-- under the License.
1717

1818
-- MinSparkVersion: 4.0
19-
-- Non-binary collations still require Spark's ordering, including when nested.
20-
-- Neither strict floating-point mode nor enabling the dispatcher may bypass this guard.
19+
-- Non-binary collations use Spark's ordering through the dispatcher, including when nested.
2120
-- Config: spark.comet.expression.ArrayMin.allowIncompatible=false
2221
-- ConfigMatrix: spark.comet.exec.strictFloatingPoint=false,true
23-
-- ConfigMatrix: spark.comet.exec.scalaUDF.codegen.enabled=false,true
22+
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
2423

2524
statement
2625
CREATE TABLE test_array_min_collation(
@@ -36,36 +35,40 @@ INSERT INTO test_array_min_collation VALUES
3635
(6, NULL, NULL, NULL, NULL)
3736

3837
-- Binary string ordering remains native, including inside arrays and structs.
39-
query
38+
query expect_native(array_min)
4039
SELECT id, array_min(array(a, b)) FROM test_array_min_collation
4140

42-
query
41+
query expect_native(array_min)
4342
SELECT id, array_min(array(array(a), array(b))),
4443
array_min(array(named_struct('s', a, 'f', x), named_struct('s', b, 'f', y)))
4544
FROM test_array_min_collation
4645

46+
-- Enabling the dispatcher must not move floating-point extrema off the native path.
47+
query expect_native(array_min)
48+
SELECT id, array_min(array(x, y)) FROM test_array_min_collation
49+
4750
-- Lowercase ordering differs from binary ordering for the column values 'a' and 'B'.
48-
query expect_fallback(Array extrema use binary string ordering)
51+
query expect_dispatch(array_min)
4952
SELECT id, array_min(array(
5053
CAST(a AS STRING COLLATE UTF8_LCASE),
5154
CAST(b AS STRING COLLATE UTF8_LCASE)))
5255
FROM test_array_min_collation
5356

54-
query expect_fallback(Array extrema use binary string ordering)
57+
query expect_dispatch(array_min)
5558
SELECT id, array_min(array(
5659
array(CAST(a AS STRING COLLATE UTF8_LCASE)),
5760
array(CAST(b AS STRING COLLATE UTF8_LCASE))))
5861
FROM test_array_min_collation
5962

6063
-- A floating-point field does not make a collated struct eligible for native ordering.
61-
query expect_fallback(Array extrema use binary string ordering)
64+
query expect_dispatch(array_min)
6265
SELECT id, array_min(array(
6366
named_struct('s', CAST(a AS STRING COLLATE UTF8_LCASE), 'f', x),
6467
named_struct('s', CAST(b AS STRING COLLATE UTF8_LCASE), 'f', y)))
6568
FROM test_array_min_collation
6669

6770
-- The collation check must recurse through both struct fields and nested array elements.
68-
query expect_fallback(Array extrema use binary string ordering)
71+
query expect_dispatch(array_min)
6972
SELECT id, array_min(array(
7073
named_struct('s', array(CAST(a AS STRING COLLATE UTF8_LCASE)), 'f', x),
7174
named_struct('s', array(CAST(b AS STRING COLLATE UTF8_LCASE)), 'f', y)))

‎spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -631,6 +631,32 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
631631
}
632632
}
633633

634+
test("array extrema - collations fall back when the dispatcher is disabled") {
635+
assume(isSpark40Plus)
636+
withParquetTable(Seq(("a", "B"), ("B", "a"), ("A", "a")), "collated_extrema") {
637+
val a = "CAST(_1 AS STRING COLLATE UTF8_LCASE)"
638+
val b = "CAST(_2 AS STRING COLLATE UTF8_LCASE)"
639+
val inputs = Seq(
640+
s"array($a, $b)",
641+
s"array(array($a), array($b))",
642+
s"array(named_struct('s', $a), named_struct('s', $b))",
643+
s"array(named_struct('s', array($a)), named_struct('s', array($b)))")
644+
for (strict <- Seq(false, true)) {
645+
withSQLConf(
646+
CometConf.COMET_EXEC_STRICT_FLOATING_POINT.key -> strict.toString,
647+
CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false",
648+
CometConf.getExprAllowIncompatConfigKey(classOf[ArrayMin]) -> "false",
649+
CometConf.getExprAllowIncompatConfigKey(classOf[ArrayMax]) -> "false") {
650+
for (function <- Seq("array_min", "array_max"); input <- inputs) {
651+
checkSparkAnswerAndFallbackReason(
652+
s"SELECT $function($input) FROM collated_extrema",
653+
"Array extrema use binary string ordering")
654+
}
655+
}
656+
}
657+
}
658+
}
659+
634660
test("array extrema - runtime NaN representations") {
635661
withParquetTable(Seq((Float.NaN, Double.NaN)), "floating_point_extrema") {
636662
for (strict <- Seq(false, true)) {

‎spark/src/test/scala/org/apache/comet/CometCodegenSourceSuite.scala‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,12 @@ import org.scalatest.funsuite.AnyFunSuite
2424
import org.apache.spark.SparkConf
2525
import org.apache.spark.serializer.JavaSerializer
2626
import org.apache.spark.sql.catalyst.InternalRow
27-
import org.apache.spark.sql.catalyst.expressions.{Add, AddMonths, BoundReference, Cast, Coalesce, Concat, CreateArray, CreateMap, DateFormatClass, ElementAt, Expression, GetStructField, IntegralDivide, LeafExpression, Length, Literal, MakeTimestamp, MicrosToTimestamp, MillisToTimestamp, MonthsBetween, Nondeterministic, Rand, Size, Substring, ToUnixTimestamp, Unevaluable, UnixMicros, UnixMillis, UnixSeconds, Upper}
27+
import org.apache.spark.sql.catalyst.expressions.{Add, AddMonths, ArrayMax, ArrayMin, BoundReference, Cast, Coalesce, Concat, CreateArray, CreateMap, DateFormatClass, ElementAt, Expression, GetStructField, IntegralDivide, LeafExpression, Length, Literal, MakeTimestamp, MicrosToTimestamp, MillisToTimestamp, MonthsBetween, Nondeterministic, Rand, Size, Substring, ToUnixTimestamp, Unevaluable, UnixMicros, UnixMillis, UnixSeconds, Upper}
2828
import org.apache.spark.sql.catalyst.expressions.codegen.{CodeFormatter, CodegenContext, CodegenFallback, ExprCode}
2929
import org.apache.spark.sql.types._
3030
import org.apache.spark.unsafe.types.UTF8String
3131

32+
import org.apache.comet.CometSparkSessionExtensions.isSpark40Plus
3233
import org.apache.comet.codegen.CometBatchKernelCodegen
3334
import org.apache.comet.codegen.CometBatchKernelCodegen.{ArrayColumnSpec, ArrowColumnSpec, MapColumnSpec, ScalarColumnSpec, StructColumnSpec, StructFieldSpec}
3435
import org.apache.comet.udf.codegen.CometScalaUDFCodegen
@@ -1435,6 +1436,50 @@ class CometCodegenSourceSuite extends AnyFunSuite {
14351436
}
14361437
}
14371438

1439+
test("array extrema preserve collations through closure serialization and codegen") {
1440+
assume(isSpark40Plus, "COLLATE requires Spark 4.0")
1441+
val collated = DataType.fromDDL("STRING COLLATE UTF8_LCASE")
1442+
assert(collated != StringType)
1443+
val serializer = new JavaSerializer(new SparkConf()).newInstance()
1444+
1445+
def inputSpec(dt: DataType): ArrowColumnSpec = dt match {
1446+
case _: StringType => ScalarColumnSpec(varCharVectorClass, nullable = true)
1447+
case ArrayType(elementType, _) =>
1448+
ArrayColumnSpec(nullable = true, elementType, inputSpec(elementType))
1449+
case st: StructType =>
1450+
StructColumnSpec(
1451+
nullable = true,
1452+
st.fields
1453+
.map(f => StructFieldSpec(f.name, f.dataType, f.nullable, inputSpec(f.dataType)))
1454+
.toSeq)
1455+
case other => fail(s"unexpected input type $other")
1456+
}
1457+
1458+
val elementTypes = Seq(
1459+
collated,
1460+
ArrayType(collated),
1461+
StructType(Seq(StructField("s", collated))),
1462+
StructType(Seq(StructField("s", ArrayType(collated)))))
1463+
for (elementType <- elementTypes) {
1464+
val inputType = ArrayType(elementType)
1465+
val input = BoundReference(0, inputType, nullable = true)
1466+
for (expr <- Seq(ArrayMin(input), ArrayMax(input))) {
1467+
withClue(s"${expr.prettyName} over $inputType: ") {
1468+
// The dispatcher transports the bound Catalyst tree separately from the Arrow schema,
1469+
// which represents collated strings as UTF-8 bytes without a collation ID.
1470+
val restored = serializer.deserialize[Expression](serializer.serialize(expr))
1471+
assert(restored.dataType == elementType)
1472+
assert(restored.children.head.dataType == inputType)
1473+
assert(CometBatchKernelCodegen.canHandle(restored).isEmpty)
1474+
val src = gen(restored, inputSpec(inputType))
1475+
assert(
1476+
src.contains(".semanticCompare("),
1477+
s"expected Spark's collation-aware string comparison; got:\n$src")
1478+
}
1479+
}
1480+
}
1481+
}
1482+
14381483
test("closure-serialized bytes diverge for failOnError / roundOff / timeZoneId variants") {
14391484
// The dispatcher caches kernels by the closure-serialized bytes of the bound expression
14401485
// (`CacheKey.bytesKey`). For expression classes that carry a runtime-dependent boolean

0 commit comments

Comments
 (0)