Skip to content

Commit 2ecb05f

Browse files
committed
fix: keep collated struct fields out of native array_distinct and array_union
The spark_ variants normalize floats and then let DataFusion compare the elements, which compares strings by their bytes. Spark compares a collated string under its collation, so under UTF8_LCASE two structs that differ only in 'a' and 'A' are one element. Widening the gate to 4.0.5+, 4.1.4+ and 4.2.x would have sent such elements to the native kernels, and 4.2.0 already did. Elements that hold both a float and a non-UTF8_BINARY collated string now fall back on every version. Collated strings without floats are tracked separately in #6470.
1 parent eb6f4dd commit 2ecb05f

5 files changed

Lines changed: 113 additions & 6 deletions

File tree

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,11 @@ the elements the same way. Spark 3.4, 3.5, 4.0.0 to 4.0.4, and 4.1.0 to 4.1.3 ke
9999
negative zero distinct in flat arrays, so on those versions these functions fall back to Spark.
100100
Other element types remain native on every version.
101101

102+
An element type that holds both a floating-point value and a string with a non-`UTF8_BINARY`
103+
collation, such as a struct with a `DOUBLE` and a `STRING COLLATE UTF8_LCASE` field, falls back
104+
on every version. Spark compares such strings under their collation, so `'a'` and `'A'` are one
105+
value under `UTF8_LCASE`, while the native kernels compare their bytes.
106+
102107
Where these functions fall back, the check is based on the element type, not the values. It also
103108
applies to NULL or empty floating-point arrays and columns that never contain negative zero. The
104109
entire projection falls back to Spark, introducing a `CometColumnarToRow` transition and moving

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ The tables below list every Spark built-in expression with its current status.
156156
| `array_append` | ✅ | Native | |
157157
| `array_compact` | ✅ | — | |
158158
| `array_contains` | ✅ | Native | Float/double element arrays route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible |
159-
| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)) |
159+
| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2, and on every version when they also hold a collated string; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)) |
160160
| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
161161
| `array_insert` | ✅ | Native | |
162162
| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
@@ -167,7 +167,7 @@ The tables below list every Spark built-in expression with its current status.
167167
| `array_prepend` | ✅ | — | |
168168
| `array_remove` | ✅ | Native | |
169169
| `array_repeat` | ✅ | Native | |
170-
| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)) |
170+
| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2, and on every version when they also hold a collated string; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)) |
171171
| `arrays_overlap` | ✅ | Native | |
172172
| `arrays_zip` | ✅ | Native | |
173173
| `element_at` | ✅ | Native | |

‎spark/src/main/scala/org/apache/comet/serde/arrays.scala‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -544,6 +544,11 @@ private[comet] object ArraySetSupport {
544544
"4.0.5+, 4.1.4+ and 4.2+, which treat -0.0 and 0.0 as one value in these functions " +
545545
"(SPARK-54918, SPARK-59602)"
546546

547+
val collationReason: String =
548+
"Elements that hold both a floating-point value and a non-UTF8_BINARY collated string fall " +
549+
"back to Spark, which compares the strings under their collation, while Comet's native " +
550+
"kernels compare their raw bytes"
551+
547552
// Spark 4.2.0 normalizes the arguments of these functions in the plan (SPARK-54918), and 4.0.5,
548553
// 4.1.4 and 4.2.1 normalize while evaluating them (SPARK-59602). Either way, Spark treats -0.0
549554
// and 0.0, and every NaN, as one value at any depth, which the spark_ variants match. Earlier
@@ -558,7 +563,12 @@ private[comet] object ArraySetSupport {
558563
}
559564

560565
def supportLevel(dataType: DataType): SupportLevel = {
561-
if (hasFloats(dataType) && !normalizesFloats(SPARK_VERSION)) {
566+
if (hasFloats(dataType) && hasNonDefaultStringCollation(dataType)) {
567+
// The spark_ variants normalize the floats, then DataFusion compares the elements, strings
568+
// included, by their bytes. Collated strings without floats take the plain DataFusion
569+
// functions: https://github.com/apache/datafusion-comet/issues/6470.
570+
Incompatible(Some(collationReason))
571+
} else if (hasFloats(dataType) && !normalizesFloats(SPARK_VERSION)) {
562572
Incompatible(Some(floatingPointReason))
563573
} else {
564574
Compatible()
@@ -577,7 +587,8 @@ private[comet] object ArraySetSupport {
577587
// Use projection fallback to avoid codegen dispatch overhead for array-valued results.
578588
// The native implementation remains available through opt-in.
579589
object CometArrayDistinct extends CometExpressionSerde[ArrayDistinct] {
580-
override def getIncompatibleReasons(): Seq[String] = Seq(ArraySetSupport.floatingPointReason)
590+
override def getIncompatibleReasons(): Seq[String] =
591+
Seq(ArraySetSupport.floatingPointReason, ArraySetSupport.collationReason)
581592

582593
override def getSupportLevel(expr: ArrayDistinct): SupportLevel =
583594
ArraySetSupport.supportLevel(expr.dataType)
@@ -594,7 +605,8 @@ object CometArrayDistinct extends CometExpressionSerde[ArrayDistinct] {
594605
}
595606

596607
object CometArrayUnion extends CometExpressionSerde[ArrayUnion] {
597-
override def getIncompatibleReasons(): Seq[String] = Seq(ArraySetSupport.floatingPointReason)
608+
override def getIncompatibleReasons(): Seq[String] =
609+
Seq(ArraySetSupport.floatingPointReason, ArraySetSupport.collationReason)
598610

599611
override def getSupportLevel(expr: ArrayUnion): SupportLevel =
600612
ArraySetSupport.supportLevel(expr.dataType)
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- MinSparkVersion: 4.0
19+
20+
-- Spark compares a collated string inside an array_distinct or array_union element under its
21+
-- collation, so 'a' and 'A' are one value under UTF8_LCASE. The native kernels for elements with
22+
-- a float normalize the floats but compare strings by their bytes, so elements that hold both
23+
-- fall back to Spark on every version, including the releases whose floats run natively.
24+
-- A cast keeps the collated strings in the Comet plan, unlike collate(), which falls back itself.
25+
26+
statement
27+
CREATE TABLE array_set_collated(s1 string, s2 string, d double) USING parquet
28+
29+
statement
30+
INSERT INTO array_set_collated VALUES
31+
('a', 'A', 1.0),
32+
('a', 'b', 1.0),
33+
('x', 'X', 0.0),
34+
('n', 'N', NULL),
35+
(NULL, 'A', 2.0)
36+
37+
query expect_fallback(non-UTF8_BINARY collated string)
38+
SELECT
39+
size(array_distinct(array(
40+
named_struct('s', CAST(s1 AS STRING COLLATE UTF8_LCASE), 'd', d),
41+
named_struct('s', CAST(s2 AS STRING COLLATE UTF8_LCASE), 'd', d)))),
42+
array_distinct(array(
43+
named_struct('s', CAST(s1 AS STRING COLLATE UTF8_LCASE), 'd', d),
44+
named_struct('s', CAST(s2 AS STRING COLLATE UTF8_LCASE), 'd', d)))
45+
FROM array_set_collated
46+
47+
query expect_fallback(non-UTF8_BINARY collated string)
48+
SELECT
49+
size(array_union(
50+
array(named_struct('s', CAST(s1 AS STRING COLLATE UTF8_LCASE), 'd', d)),
51+
array(named_struct('s', CAST(s2 AS STRING COLLATE UTF8_LCASE), 'd', d)))),
52+
array_union(
53+
array(named_struct('s', CAST(s1 AS STRING COLLATE UTF8_LCASE), 'd', d)),
54+
array(named_struct('s', CAST(s2 AS STRING COLLATE UTF8_LCASE), 'd', d)))
55+
FROM array_set_collated
56+
57+
-- The float differs only in the sign of zero, so a match needs both the collation and the
58+
-- float normalization.
59+
query expect_fallback(non-UTF8_BINARY collated string)
60+
SELECT
61+
size(array_distinct(array(
62+
named_struct('s', CAST(s1 AS STRING COLLATE UTF8_LCASE), 'd', d),
63+
named_struct('s', CAST(s2 AS STRING COLLATE UTF8_LCASE), 'd', -d)))),
64+
size(array_union(
65+
array(named_struct('s', CAST(s1 AS STRING COLLATE UTF8_LCASE), 'd', d)),
66+
array(named_struct('s', CAST(s2 AS STRING COLLATE UTF8_LCASE), 'd', -d))))
67+
FROM array_set_collated
68+
69+
-- Nested one level deeper: the collated string sits in an array inside the struct.
70+
query expect_fallback(non-UTF8_BINARY collated string)
71+
SELECT size(array_distinct(array(
72+
named_struct('s', array(CAST(s1 AS STRING COLLATE UTF8_LCASE)), 'd', d),
73+
named_struct('s', array(CAST(s2 AS STRING COLLATE UTF8_LCASE)), 'd', d))))
74+
FROM array_set_collated

‎spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Cast, Crea
2929
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
3030
import org.apache.spark.sql.functions._
3131
import org.apache.spark.sql.internal.SQLConf
32-
import org.apache.spark.sql.types.{ArrayType, DoubleType, FloatType, IntegerType, StringType, StructType}
32+
import org.apache.spark.sql.types.{ArrayType, DataType, DoubleType, FloatType, IntegerType, StringType, StructType}
3333

3434
import org.apache.comet.CometSparkSessionExtensions.{isSpark35Plus, isSpark40Plus}
3535
import org.apache.comet.DataTypeSupport.isComplexType
@@ -81,6 +81,22 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
8181
}
8282
}
8383

84+
test("array set elements with floats and collated strings fall back on every version") {
85+
assume(isSpark40Plus)
86+
// The spark_ variants normalize the floats but compare strings by their bytes, so an element
87+
// that holds both declines on every version, including those whose floats run natively.
88+
// expressions/array/array_set_collated_floats.sql checks the answers on the pinned versions.
89+
Seq(
90+
"STRUCT<s: STRING COLLATE UTF8_LCASE, d: DOUBLE>",
91+
"STRUCT<s: ARRAY<STRING COLLATE UTF8_LCASE>, f: FLOAT>",
92+
"ARRAY<STRUCT<s: STRING COLLATE UTF8_LCASE, d: DOUBLE>>").foreach { ddl =>
93+
val child = AttributeReference("a", ArrayType(DataType.fromDDL(ddl)))()
94+
val expected = Incompatible(Some(ArraySetSupport.collationReason))
95+
assert(CometArrayDistinct.getSupportLevel(ArrayDistinct(child)) == expected, ddl)
96+
assert(CometArrayUnion.getSupportLevel(ArrayUnion(child, child)) == expected, ddl)
97+
}
98+
}
99+
84100
test("array set signed-zero fallback and native opt-in") {
85101
withSQLConf(
86102
SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->

0 commit comments

Comments
 (0)