Skip to content

Commit 7d8d9b1

Browse files
authored
feat: normalize floats in native array_distinct and array_union (#6563)
* feat: normalize floats in native array_distinct and array_union Spark 4.2.0 normalizes the arguments of array_distinct and array_union in the plan (SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize inside the expressions instead (SPARK-59602), so those releases pass Comet the raw input. DataFusion folds -0.0 into 0.0 only in a flat float array and compares NaNs by their bits, which is why #5750 limited native execution of float elements to 4.2.0. Arrays with a float at any depth now go to spark_array_distinct and spark_array_union, which run normalize_nested_floats on their inputs and then delegate to DataFusion, as SparkArrayExtrema does. That matches every release that merges signed zeros, so float elements run natively by default on 4.0.5+, 4.1.4+ and 4.2+, and only 3.4, 3.5, 4.0.0 to 4.0.4 and 4.1.0 to 4.1.3 fall back. Older Spark merges NaNs and nested zeros too, so the opt-in now differs from it only on signed zeros. Within 4.0 and 4.1 the default path now depends on the patch release, so the 4.0/4.1 signed-zero fixture checks answers only and CometArrayExpressionSuite checks the routing. The 4.2+ fixture checks the default path instead of the opt-ins. * fix: keep collated struct fields out of native array_distinct and array_union The spark_ variants normalize floats and then let DataFusion compare the elements, which compares strings by their bytes. Spark compares a collated string under its collation, so under UTF8_LCASE two structs that differ only in 'a' and 'A' are one element. Widening the gate to 4.0.5+, 4.1.4+ and 4.2.x would have sent such elements to the native kernels, and 4.2.0 already did. Elements that hold both a float and a non-UTF8_BINARY collated string now fall back on every version. Collated strings without floats are tracked separately in #6470. * fix: skip array_union's right operand on rows whose left array is NULL Spark's ArrayUnion returns NULL for a NULL left array without evaluating the right operand. The native function evaluates both over the whole batch, so a right operand that throws, such as slice(b, 0, 1) or an ANSI cast, failed on rows that Spark never evaluates it for. That already happened on main for every element type that runs natively, and this PR would have extended it to float elements on 4.0.5+ and 4.1.4+. CometArrayUnion now guards the call with CASE WHEN <left> IS NOT NULL, as CometElementAt does, unless the right operand is a column or a literal, which cannot throw. A nullable nondeterministic left operand, which the guard would evaluate twice, stays on Spark. * revert: drop the array_union NULL guard in favor of #6716 The CASE WHEN <left> IS NOT NULL guard serialized the left operand twice, in the IS NOT NULL test and in the call, so nested unions grew exponentially: twelve nested array_union calls became 4,095 union nodes, and each level evaluated its left operand twice. #6716 adds a native NullShortCircuit that evaluates the left operand once and the right operand only on the rows where the left is not NULL. It covers array_union for every element type, so it fixes the eager evaluation this guard was for without the duplication. This reverts commit 88ec629. * perf: normalize only the child range a sliced list references normalize_nested_floats rebuilt a list's whole child array, but a sliced list keeps its child whole, and explode hands out slices of one child array for a whole input batch. Rebuild only the range the offsets reference, at every depth, and rebase the offsets onto it. The float_arrays bench gains array_distinct and array_union cases over whole and sliced lists. * test: cover array_union over ARRAY<DOUBLE> with a NULL left array #6716's array_null_short_circuit*.sql fixtures only use array<int>, which runs through DataFusion's array_union. A float element type goes through spark_array_union, so nothing checked that withNullShortCircuit reaches it. Without the guard, native evaluation raises on rows Spark skips: INVALID_PARAMETER_VALUE.START for slice(b, 0, 1) over a NULL array, and CAST_INVALID_INPUT for an ANSI cast of a malformed string. The fixture opts ArrayUnion in so every Spark version runs the float path natively, not only 4.0.5+, 4.1.4+ and 4.2+. The data has no signed zero, which earlier releases keep apart.
1 parent 33f4f88 commit 7d8d9b1

15 files changed

Lines changed: 695 additions & 120 deletions

File tree

‎docs/source/contributor-guide/expression-audits/array_funcs.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@
5656
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet`, which merges NaNs but keeps `+0.0` and `-0.0` distinct. Wired as `CometScalarFunction("array_distinct")`.
5757
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
5858
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
59-
- Float/double element types run natively only on Spark 4.2.0, which normalizes the arguments in the plan (SPARK-54918); every other version falls back by default. See [Array distinct and union](../../user-guide/latest/compatibility/floating-point.md#array-distinct-and-union).
59+
- Float/double element types run natively on Spark 4.0.5+, 4.1.4+, and 4.2+ through `spark_array_distinct`, which normalizes the elements as those releases do (SPARK-54918 in the plan on 4.2.0, SPARK-59602 during evaluation from 4.0.5, 4.1.4, and 4.2.1); earlier versions fall back by default. See [Array distinct and union](../../user-guide/latest/compatibility/floating-point.md#array-distinct-and-union).
6060

6161
## array_except
6262

@@ -141,7 +141,7 @@
141141
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
142142
- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; overflow path uses `arrayFunctionWithElementsExceedLimitError`.
143143
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
144-
- Float/double element types follow the same Spark 4.2.0-only gate as `array_distinct`. Result element ordering matches Spark (left-side distinct elements followed by new right-side elements).
144+
- Float/double element types follow the same gate as `array_distinct`, through `spark_array_union`. Result element ordering matches Spark (left-side distinct elements followed by new right-side elements).
145145

146146
## arrays_overlap
147147

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 26 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -113,18 +113,24 @@ the two can differ.
113113

114114
## Array distinct and union
115115

116-
`array_distinct` and `array_union` fall back to Spark when their element type contains
117-
`FLOAT` or `DOUBLE`, on every Spark version except 4.2.0. Spark 4.2.0 normalizes signed zeros
118-
and NaNs in the arguments of these functions before they run (SPARK-54918), so native execution
119-
returns the same results. Spark 3.4, 3.5, 4.0.0 to 4.0.4, and 4.1.0 to 4.1.3 keep positive and
120-
negative zero distinct in flat arrays. Spark 4.0.5+, 4.1.4+, and 4.2.1+ normalize while these
121-
functions evaluate instead (SPARK-59602), which native execution does not match for NaNs or for
122-
zeros nested in arrays or structs. Other element types remain native.
123-
124-
The check is based on the element type, not the values. It also applies to NULL or empty
125-
floating-point arrays and columns that never contain negative zero. The entire projection
126-
falls back to Spark, introducing a `CometColumnarToRow` transition and moving unrelated
127-
expressions in the same projection out of Comet. For example,
116+
When their element type contains `FLOAT` or `DOUBLE`, `array_distinct` and `array_union` run
117+
natively on Spark 4.0.5+, 4.1.4+, and 4.2+. These releases treat `-0.0` and `0.0`, and every NaN
118+
representation, as one value in these functions, including inside nested arrays and structs, and
119+
return the normalized value. Spark 4.2.0 normalizes the arguments in the plan (SPARK-54918), and
120+
4.0.5, 4.1.4, and 4.2.1 normalize while these functions evaluate (SPARK-59602). Comet normalizes
121+
the elements the same way. Spark 3.4, 3.5, 4.0.0 to 4.0.4, and 4.1.0 to 4.1.3 keep positive and
122+
negative zero distinct in flat arrays, so on those versions these functions fall back to Spark.
123+
Other element types remain native on every version.
124+
125+
Elements that hold a string with a non-`UTF8_BINARY` collation fall back on every version,
126+
including a struct with a `DOUBLE` and a `STRING COLLATE UTF8_LCASE` field. Spark compares such
127+
strings under their collation, so `'a'` and `'A'` are one value under `UTF8_LCASE`, while the
128+
native kernels compare their bytes.
129+
130+
Where these functions fall back, the check is based on the element type, not the values. It also
131+
applies to NULL or empty floating-point arrays and columns that never contain negative zero. The
132+
entire projection falls back to Spark, introducing a `CometColumnarToRow` transition and moving
133+
unrelated expressions in the same projection out of Comet. For example,
128134
`SELECT id + 1, array_distinct(a), i[0] + 5` evaluates all three expressions in a Spark `Project`.
129135

130136
This can have a substantial cost. A local Spark 4.1.3 benchmark of
@@ -133,18 +139,14 @@ projection fallback about 15 times slower than native opt-in (best of five runs)
133139
The slowdown depends on the workload.
134140

135141
Setting `spark.comet.expression.ArrayDistinct.allowIncompatible=true` or
136-
`spark.comet.expression.ArrayUnion.allowIncompatible=true` restores native execution on other
137-
versions, but signed-zero and NaN results may differ from Spark. Native execution can keep
138-
NaNs with different signs or payloads distinct. Signed-zero differences also depend on the
139-
element type: native execution merges positive and negative zero in flat floating-point arrays,
140-
but can keep them distinct inside nested arrays or structs. Only opt in if these differences
141-
are acceptable for your data.
142-
143-
The check uses the Spark version number, not the changes a build contains. A build that reports
144-
any version other than 4.2.0 falls back even if it includes SPARK-54918. A vendor build that
145-
reports 4.2.0 but includes SPARK-59602 still runs natively; set
146-
`spark.comet.expression.ArrayDistinct.enabled=false` and
147-
`spark.comet.expression.ArrayUnion.enabled=false` on such a build.
142+
`spark.comet.expression.ArrayUnion.allowIncompatible=true` restores native execution on the older
143+
versions, but signed-zero results may differ from Spark. Native execution treats `-0.0` and `0.0`
144+
as one value and returns `0.0`. The older versions keep both zeros in a flat floating-point array,
145+
and inside nested arrays or structs they return whichever zero came first. Only opt in if these
146+
differences are acceptable for your data.
147+
148+
The check uses the Spark version number, not the changes a build contains. A vendor build that
149+
reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.
148150

149151
## `array_contains`, `array_remove` and `sort_array`
150152

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ The tables below list every Spark built-in expression with its current status.
156156
| `array_append` | ✅ | Native | |
157157
| `array_compact` | ✅ | — | |
158158
| `array_contains` | ✅ | Hybrid | Flat float/double element arrays use a native kernel with Spark's equality; nested float elements, non-default string collations, and a value that is not a literal or column over a nullable array, route through the JVM codegen dispatcher |
159-
| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back |
159+
| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back |
160160
| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
161161
| `array_insert` | ✅ | Native | |
162162
| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
@@ -167,7 +167,7 @@ The tables below list every Spark built-in expression with its current status.
167167
| `array_prepend` | ✅ | — | |
168168
| `array_remove` | ✅ | Native | |
169169
| `array_repeat` | ✅ | Native | |
170-
| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back |
170+
| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back |
171171
| `arrays_overlap` | ✅ | Hybrid | Non-default string collations use the JVM codegen dispatcher |
172172
| `arrays_zip` | ✅ | Native | |
173173
| `element_at` | ✅ | Native | |

‎native/spark-expr/benches/float_arrays.rs‎

Lines changed: 61 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,10 @@
1818
//! Compare Comet's float `sort_array` and `array_remove` with DataFusion's `array_sort` and
1919
//! `array_remove_all` on doubles without zeros or NaNs, where both return the same arrays, which
2020
//! is checked before timing; special-value semantics belong in tests.
21+
//!
22+
//! Comet's `array_distinct` and `array_union` normalize their arguments and then call DataFusion's
23+
//! kernels. Their `_sliced` cases pass a slice of a longer list, as `explode` returns, to show
24+
//! that the normalization covers only the rows of the slice.
2125
2226
use arrow::array::{Array, ArrayRef, Float64Array, ListArray};
2327
use arrow::buffer::{NullBuffer, OffsetBuffer};
@@ -26,9 +30,12 @@ use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
2630
use datafusion::common::config::ConfigOptions;
2731
use datafusion::common::ScalarValue;
2832
use datafusion::functions_nested::remove::array_remove_all_udf;
33+
use datafusion::functions_nested::set_ops::{array_distinct_udf, array_union_udf};
2934
use datafusion::functions_nested::sort::array_sort_udf;
3035
use datafusion::logical_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDF};
31-
use datafusion_comet_spark_expr::{SparkArrayRemove, SparkFloatArrayContains, SparkSortArray};
36+
use datafusion_comet_spark_expr::{
37+
SparkArrayRemove, SparkArraySetOp, SparkFloatArrayContains, SparkSortArray,
38+
};
3239
use datafusion_spark::function::array::array_contains::SparkArrayContains;
3340
use std::hint::black_box;
3441
use std::sync::Arc;
@@ -41,22 +48,28 @@ fn value(i: usize) -> f64 {
4148
((i * 7919) % 101) as f64 * 0.5 + 1.0
4249
}
4350

44-
/// `ROWS` lists of `len` doubles, with every tenth list and every seventh element null if
51+
/// `rows` lists of `len` doubles, with every tenth list and every seventh element null if
4552
/// `nulls`.
46-
fn lists(len: usize, nulls: bool) -> ArrayRef {
53+
fn lists(rows: usize, len: usize, nulls: bool) -> ArrayRef {
4754
let values = if nulls {
48-
Float64Array::from_iter((0..ROWS * len).map(|i| (i % 7 != 0).then(|| value(i))))
55+
Float64Array::from_iter((0..rows * len).map(|i| (i % 7 != 0).then(|| value(i))))
4956
} else {
50-
Float64Array::from_iter_values((0..ROWS * len).map(value))
57+
Float64Array::from_iter_values((0..rows * len).map(value))
5158
};
5259
Arc::new(ListArray::new(
5360
Arc::new(Field::new_list_field(DataType::Float64, true)),
54-
OffsetBuffer::from_lengths(std::iter::repeat_n(len, ROWS)),
61+
OffsetBuffer::from_lengths(std::iter::repeat_n(len, rows)),
5562
Arc::new(values),
56-
nulls.then(|| NullBuffer::from_iter((0..ROWS).map(|row| row % 10 != 0))),
63+
nulls.then(|| NullBuffer::from_iter((0..rows).map(|row| row % 10 != 0))),
5764
))
5865
}
5966

67+
/// `ROWS` lists sliced from the middle of 16 times as many. `explode` of a list of lists returns
68+
/// slices like this, one per output batch, which all keep the same child array.
69+
fn sliced_lists(len: usize, nulls: bool) -> ArrayRef {
70+
lists(16 * ROWS, len, nulls).slice(8 * ROWS, ROWS)
71+
}
72+
6073
fn args(args: Vec<ColumnarValue>, list: &ArrayRef) -> ScalarFunctionArgs {
6174
ScalarFunctionArgs {
6275
arg_fields: args
@@ -81,9 +94,12 @@ fn criterion_benchmark(c: &mut Criterion) {
8194
// The native path other element types take: array_has plus Spark's null semantics.
8295
let datafusion_contains = Arc::new(ScalarUDF::new_from_impl(SparkArrayContains::default()));
8396
let (datafusion_sort, datafusion_remove) = (array_sort_udf(), array_remove_all_udf());
97+
let comet_distinct = ScalarUDF::new_from_impl(SparkArraySetOp::distinct());
98+
let comet_union = ScalarUDF::new_from_impl(SparkArraySetOp::union());
99+
let (datafusion_distinct, datafusion_union) = (array_distinct_udf(), array_union_udf());
84100
for len in [8, 50] {
85101
for nulls in [false, true] {
86-
let list = lists(len, nulls);
102+
let list = lists(ROWS, len, nulls);
87103
let data = format!("len={len}_null={nulls}");
88104
let column = ColumnarValue::Array(Arc::clone(&list));
89105
let scalar = |value: ScalarValue| ColumnarValue::Scalar(value);
@@ -121,9 +137,17 @@ fn criterion_benchmark(c: &mut Criterion) {
121137
let row_values =
122138
Float64Array::from_iter_values((0..ROWS).map(|row| value(row * len + 1)));
123139
let contains_column_args = args(
124-
vec![column, ColumnarValue::Array(Arc::new(row_values))],
140+
vec![column.clone(), ColumnarValue::Array(Arc::new(row_values))],
125141
&list,
126142
);
143+
let distinct_args = args(vec![column.clone()], &list);
144+
// A union of a list with itself: the right side adds no elements, but it is still
145+
// normalized and looked up.
146+
let union_args = args(vec![column.clone(), column], &list);
147+
let sliced = sliced_lists(len, nulls);
148+
let sliced_column = ColumnarValue::Array(Arc::clone(&sliced));
149+
let sliced_distinct_args = args(vec![sliced_column.clone()], &sliced);
150+
let sliced_union_args = args(vec![sliced_column.clone(), sliced_column], &sliced);
127151
let cases = [
128152
(
129153
"sort_array",
@@ -160,6 +184,34 @@ fn criterion_benchmark(c: &mut Criterion) {
160184
&contains_column_args,
161185
&contains_column_args,
162186
),
187+
(
188+
"array_distinct",
189+
&comet_distinct,
190+
&datafusion_distinct,
191+
&distinct_args,
192+
&distinct_args,
193+
),
194+
(
195+
"array_distinct_sliced",
196+
&comet_distinct,
197+
&datafusion_distinct,
198+
&sliced_distinct_args,
199+
&sliced_distinct_args,
200+
),
201+
(
202+
"array_union",
203+
&comet_union,
204+
&datafusion_union,
205+
&union_args,
206+
&union_args,
207+
),
208+
(
209+
"array_union_sliced",
210+
&comet_union,
211+
&datafusion_union,
212+
&sliced_union_args,
213+
&sliced_union_args,
214+
),
163215
];
164216
for (op, comet, datafusion, comet_args, datafusion_args) in cases {
165217
let evaluate = |udf: &ScalarUDF, args: &ScalarFunctionArgs| {

0 commit comments

Comments
 (0)