Skip to content

Commit ac9ae94

Browse files
authored
fix: match Spark's float ordering in min, max, greatest and least (apache#6457)
* fix: match Spark's float ordering in min, max, greatest and least Spark orders floats with SQLOrderingUtil.compareDoubles, in which NaN is larger than every other value and -0.0 equals 0.0, and its max, min, greatest and least keep the first of equal values. DataFusion's versions order floats by IEEE 754 total order, so a NaN with the sign bit set is the smallest value; its grouped max and min start each group at f64::MIN and f64::MAX, so a group of only -Infinity (or Infinity) returns that finite value; and its greatest and least let a later argument win a tie. Add SparkMinMax for float min and max in aggregates and windows, with ungrouped, grouped and sliding-frame accumulators, and SparkGreatestLeast for float greatest and least, including arrays and structs with a float leaf. Other types keep DataFusion's implementations. Strict floating-point mode no longer makes min and max fall back. Part of apache#6385. * test: add min/max and greatest/least benchmarks and FLOAT sliding-window coverage Add criterion benchmarks comparing SparkMinMax and SparkGreatestLeast with DataFusion's versions, which check that both return the same values before timing. Cover SlidingMinMaxAccumulator over FLOAT in the unit test and the SQL fixture, share the edge values and the first-wins oracle between the two test modules through float_semantics, and route window min/max through min_max_udaf so the float routing lives in one place. * test: avoid scan-order assumptions in float min/max checks * test: retire fixed float min/max and greatest/least gaps * fix: harden float extrema planner routing * test: share float_semantics' edge values with the array function tests array_remove and sort_array, from apache#6518, kept their own copy of the edge values that min/max and greatest/least share through float_semantics. The two arrays were identical. * test: check signed-zero ties in aggregate min and max again eb5c7af stopped depending on scan order by dropping the aggregate tie queries from the SQL fixture, and by testing each zero on its own in CometAggregateSuite, where any tie rule returns the same value. Write both tables as a single file instead, so that Spark and Comet read the rows in the same order in one task, and check ties through the ungrouped and grouped accumulators again. The ungrouped checks run once per group, because over groups 2 and 3 together the first and the last zero have the same sign. * refactor: drop the unreachable missing-result branch from greatest and least invoke_with_args checks for at least two arguments before it starts, so the result can no longer be missing after the loop. Start from the first argument instead of an Option, as return_type already does.
1 parent 9866221 commit ac9ae94

21 files changed

Lines changed: 1667 additions & 57 deletions

File tree

‎docs/source/contributor-guide/expression-audits/math_funcs.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@ Internal fused expression that replaces the `CheckOverflow(Cast(expr, Decimal128
149149
## greatest
150150

151151
- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): NULL-skipping variadic. Wired as `CometScalarFunction("greatest")` to DataFusion's `GreatestFunc`. Comet does not gate input types, so interval inputs and other Spark-only orderings rely on the native UDF accepting them; no explicit fallback path.
152+
- Current status: `FLOAT` and `DOUBLE` arguments, and arrays and structs with a floating-point leaf, use the native `SparkGreatestLeast` UDF in both floating-point modes. It follows `SQLOrderingUtil` at any depth (all NaNs equal and greater than non-NaN values, signed zeros equal) and replaces its result only with a strictly greater argument, so the first of equal arguments is returned. Other types use DataFusion's `GreatestFunc`.
152153

153154
## hex
154155

@@ -157,6 +158,7 @@ Internal fused expression that replaces the `CheckOverflow(Cast(expr, Decimal128
157158
## least
158159

159160
- Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 (audited 2026-05-27): mirror of `greatest`; same caveats. Spark 4.1.1 adds `contextIndependentFoldable` (no Comet impact).
161+
- Current status: mirror of `greatest`. Floating-point arguments, including nested ones, use `SparkGreatestLeast` and return the first of equal arguments; other types use DataFusion's `LeastFunc`.
160162

161163
## ln
162164

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,19 @@ greater than non-NaN values. The original first equal element is retained: for e
7575
The same ordering applies recursively to floating-point fields in arrays and structs. These
7676
expressions do not require Spark's codegen dispatcher for floating-point compatibility.
7777

78+
## `min`, `max`, `greatest`, and `least`
79+
80+
The `min` and `max` aggregates, including over window frames, and `greatest` and `least` compare
81+
`FLOAT` and `DOUBLE` values in the same order: NaN is larger than every other value and `-0.0`
82+
equals `0.0`. Of equal values the first one wins, as in Spark: `greatest(-0.0D, 0.0D)` returns
83+
`-0.0`, and `max` returns whichever zero it reads first. `greatest` and `least` follow the same
84+
order inside arrays and structs. These run natively in strict floating-point mode too.
85+
86+
Spark treats `greatest` and `least` as commutative when it matches expressions, so a projection
87+
that holds both `greatest(a, b)` and `greatest(b, a)` evaluates only one of them and returns the
88+
same value for both. Comet evaluates each one, so when `a` and `b` are zeros of different signs
89+
the two can differ.
90+
7891
## Array distinct and union
7992

8093
`array_distinct` and `array_union` fall back to Spark when their element type contains

‎native/core/src/execution/planner.rs‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -143,8 +143,8 @@ use datafusion_comet_spark_expr::{
143143
ArrayInsert, Avg, AvgDecimal, Cast, CheckOverflow, Correlation, Covariance, CreateNamedStruct,
144144
DecimalRescaleCheckOverflow, GetArrayStructFields, GetStructField, HllPlusPlus, HllSketchAgg,
145145
HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero, NormalizeNestedFloats,
146-
Regr, RegrType, SparkCastOptions, Stddev, SumDecimal, ToJson, UnboundColumn, Variance,
147-
WideDecimalBinaryExpr, WideDecimalOp,
146+
Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal, ToJson, UnboundColumn,
147+
Variance, WideDecimalBinaryExpr, WideDecimalOp,
148148
};
149149
use itertools::Itertools;
150150
use jni::objects::{Global, JObject};
@@ -2757,9 +2757,10 @@ impl PhysicalPlanner {
27572757
AggExprStruct::Min(expr) => {
27582758
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
27592759
let datatype = to_arrow_datatype(expr.datatype.as_ref().unwrap());
2760+
let func = min_max_udaf(&datatype, false);
27602761
let child = Arc::new(CastExpr::new(child, datatype.clone(), None));
27612762

2762-
AggregateExprBuilder::new(min_udaf(), vec![child])
2763+
AggregateExprBuilder::new(func, vec![child])
27632764
.schema(schema)
27642765
.alias("min")
27652766
.with_ignore_nulls(false)
@@ -2770,9 +2771,10 @@ impl PhysicalPlanner {
27702771
AggExprStruct::Max(expr) => {
27712772
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
27722773
let datatype = to_arrow_datatype(expr.datatype.as_ref().unwrap());
2774+
let func = min_max_udaf(&datatype, true);
27732775
let child = Arc::new(CastExpr::new(child, datatype.clone(), None));
27742776

2775-
AggregateExprBuilder::new(max_udaf(), vec![child])
2777+
AggregateExprBuilder::new(func, vec![child])
27762778
.schema(schema)
27772779
.alias("max")
27782780
.with_ignore_nulls(false)
@@ -3400,11 +3402,13 @@ impl PhysicalPlanner {
34003402
}
34013403
Some(AggExprStruct::Min(expr)) => {
34023404
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
3403-
Ok((by_name("min")?, vec![child]))
3405+
let func = min_max_udaf(&child.data_type(&schema)?, false);
3406+
Ok((WindowFunctionDefinition::AggregateUDF(func), vec![child]))
34043407
}
34053408
Some(AggExprStruct::Max(expr)) => {
34063409
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
3407-
Ok((by_name("max")?, vec![child]))
3410+
let func = min_max_udaf(&child.data_type(&schema)?, true);
3411+
Ok((WindowFunctionDefinition::AggregateUDF(func), vec![child]))
34083412
}
34093413
Some(AggExprStruct::Sum(expr)) => {
34103414
// For ever-expanding frames, use Comet's Spark-compatible Sum UDAFs
@@ -3739,6 +3743,20 @@ impl PhysicalPlanner {
37393743
}
37403744
}
37413745

3746+
/// `min` or `max` over `data_type`: Spark's version for floats, which DataFusion orders
3747+
/// differently, and DataFusion's for every other type.
3748+
fn min_max_udaf(data_type: &DataType, is_max: bool) -> Arc<AggregateUDF> {
3749+
// Spark has only 32- and 64-bit floating-point types. Keep Float16 on DataFusion's path so
3750+
// an internal plan using it does not reach SparkMinMax, whose accumulators reject that type.
3751+
if matches!(data_type, DataType::Float32 | DataType::Float64) {
3752+
Arc::new(AggregateUDF::new_from_impl(SparkMinMax::new(is_max)))
3753+
} else if is_max {
3754+
max_udaf()
3755+
} else {
3756+
min_udaf()
3757+
}
3758+
}
3759+
37423760
fn percentile_value(expr: &spark_expression::Expr) -> Result<f64, ExecutionError> {
37433761
match &expr.expr_struct {
37443762
Some(ExprStruct::Literal(literal)) if !literal.is_null => match &literal.value {

‎native/spark-expr/Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -473,3 +473,11 @@ harness = false
473473
[[bench]]
474474
name = "float_arrays"
475475
harness = false
476+
477+
[[bench]]
478+
name = "min_max"
479+
harness = false
480+
481+
[[bench]]
482+
name = "greatest_least"
483+
harness = false
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Compare Spark's float `greatest` and `least` with DataFusion's on ordinary data, where both
19+
//! return the same values, which is checked before timing; special-value semantics belong in
20+
//! tests.
21+
22+
use arrow::array::{Array, ArrayRef, Float64Array};
23+
use arrow::datatypes::{DataType, Field};
24+
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
25+
use datafusion::common::config::ConfigOptions;
26+
use datafusion::functions::core::{greatest, least};
27+
use datafusion::logical_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDF};
28+
use datafusion_comet_spark_expr::SparkGreatestLeast;
29+
use std::hint::black_box;
30+
use std::sync::Arc;
31+
use std::time::Duration;
32+
33+
const ROWS: usize = 8192;
34+
const ARGS: usize = 3;
35+
36+
/// Doubles in no particular order, different for each `seed`, with every tenth row null if
37+
/// `nulls`.
38+
fn column(seed: usize, nulls: bool) -> ArrayRef {
39+
Arc::new(Float64Array::from_iter((0..ROWS).map(|i| {
40+
(!nulls || !(i + seed).is_multiple_of(10))
41+
.then_some(((i * 7919 + seed * 104729) % 10007) as f64 * 0.5 - 2000.0)
42+
})))
43+
}
44+
45+
fn criterion_benchmark(c: &mut Criterion) {
46+
let mut group = c.benchmark_group("greatest_least");
47+
group.sample_size(20);
48+
group.warm_up_time(Duration::from_millis(250));
49+
group.measurement_time(Duration::from_secs(1));
50+
for is_greatest in [true, false] {
51+
let op = if is_greatest { "greatest" } else { "least" };
52+
let comet = ScalarUDF::new_from_impl(SparkGreatestLeast::new(is_greatest));
53+
let datafusion = if is_greatest { greatest() } else { least() };
54+
for nulls in [false, true] {
55+
let args = ScalarFunctionArgs {
56+
args: (0..ARGS)
57+
.map(|seed| ColumnarValue::Array(column(seed, nulls)))
58+
.collect(),
59+
arg_fields: (0..ARGS)
60+
.map(|i| Arc::new(Field::new(format!("c{i}"), DataType::Float64, true)))
61+
.collect(),
62+
number_rows: ROWS,
63+
return_field: Arc::new(Field::new("result", DataType::Float64, true)),
64+
config_options: Arc::new(ConfigOptions::default()),
65+
};
66+
let evaluate = |udf: &ScalarUDF| {
67+
udf.invoke_with_args(args.clone())
68+
.unwrap()
69+
.into_array(ROWS)
70+
.unwrap()
71+
};
72+
assert_eq!(evaluate(&comet).to_data(), evaluate(&datafusion).to_data());
73+
let data = if nulls { "null_10pct" } else { "no_nulls" };
74+
for (engine, udf) in [("comet", &comet), ("datafusion", datafusion.as_ref())] {
75+
group.bench_function(
76+
BenchmarkId::new(format!("{op}_{ARGS}_args_{engine}"), data),
77+
|b| {
78+
b.iter(|| black_box(udf.invoke_with_args(black_box(args.clone())).unwrap()))
79+
},
80+
);
81+
}
82+
}
83+
}
84+
group.finish();
85+
}
86+
87+
criterion_group!(benches, criterion_benchmark);
88+
criterion_main!(benches);
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Compare Spark's float `min` and `max` with DataFusion's; special-value semantics belong in
19+
//! tests. Both return the same values on the data without NaNs, which is checked before timing.
20+
//!
21+
//! The grouped case updates 1024 groups from eight batches. Its loop scatters into the groups, so
22+
//! it cannot vectorize, and the comparison form and bounds checks in `MinMaxGroupsAccumulator`
23+
//! decide its speed.
24+
25+
use arrow::array::{Array, ArrayRef, Float64Array};
26+
use arrow::datatypes::{DataType, Field, Schema};
27+
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
28+
use datafusion::common::ScalarValue;
29+
use datafusion::functions_aggregate::min_max::{max_udaf, min_udaf};
30+
use datafusion::logical_expr::function::AccumulatorArgs;
31+
use datafusion::logical_expr::{AggregateUDF, EmitTo};
32+
use datafusion::physical_expr::expressions::Column;
33+
use datafusion::physical_expr::PhysicalExpr;
34+
use datafusion_comet_spark_expr::SparkMinMax;
35+
use std::hint::black_box;
36+
use std::sync::Arc;
37+
use std::time::Duration;
38+
39+
const ROWS: usize = 8192;
40+
const GROUPS: usize = 1024;
41+
const BATCHES: usize = 8;
42+
43+
/// Doubles in no particular order, with a NaN in every `nan_every`-th row if it is not zero.
44+
fn values(nan_every: usize) -> ArrayRef {
45+
Arc::new(Float64Array::from_iter_values((0..ROWS).map(|i| {
46+
if nan_every > 0 && i % nan_every == 3 {
47+
f64::NAN
48+
} else {
49+
((i * 7919) % 10007) as f64 * 0.5 - 2000.0
50+
}
51+
})))
52+
}
53+
54+
fn criterion_benchmark(c: &mut Criterion) {
55+
let schema = Schema::new(vec![Field::new("d", DataType::Float64, true)]);
56+
let return_field = Arc::new(Field::new("d", DataType::Float64, true));
57+
let exprs: [Arc<dyn PhysicalExpr>; 1] = [Arc::new(Column::new("d", 0))];
58+
let args = |name| AccumulatorArgs {
59+
return_field: Arc::clone(&return_field),
60+
schema: &schema,
61+
expr_fields: &[],
62+
ignore_nulls: false,
63+
order_bys: &[],
64+
is_reversed: false,
65+
name,
66+
is_distinct: false,
67+
exprs: &exprs,
68+
};
69+
let group_indices: Vec<usize> = (0..ROWS).map(|i| (i * 31) % GROUPS).collect();
70+
71+
let mut group = c.benchmark_group("min_max");
72+
group.sample_size(20);
73+
group.warm_up_time(Duration::from_millis(250));
74+
group.measurement_time(Duration::from_secs(1));
75+
for is_max in [true, false] {
76+
let op = if is_max { "max" } else { "min" };
77+
let comet = AggregateUDF::new_from_impl(SparkMinMax::new(is_max));
78+
let datafusion = if is_max { max_udaf() } else { min_udaf() };
79+
for (data, nan_every) in [("no_nan", 0), ("nan_10pct", 10)] {
80+
let batch = values(nan_every);
81+
let ungrouped = |udaf: &AggregateUDF| -> ScalarValue {
82+
let mut acc = udaf.accumulator(args(op)).unwrap();
83+
acc.update_batch(&[Arc::clone(&batch)]).unwrap();
84+
acc.evaluate().unwrap()
85+
};
86+
let grouped = |udaf: &AggregateUDF| -> ArrayRef {
87+
let mut acc = udaf.create_groups_accumulator(args(op)).unwrap();
88+
for _ in 0..BATCHES {
89+
acc.update_batch(&[Arc::clone(&batch)], &group_indices, None, GROUPS)
90+
.unwrap();
91+
}
92+
acc.evaluate(EmitTo::All).unwrap()
93+
};
94+
if nan_every == 0 {
95+
assert_eq!(ungrouped(&comet), ungrouped(&datafusion));
96+
assert_eq!(grouped(&comet).to_data(), grouped(&datafusion).to_data());
97+
}
98+
for (engine, udaf) in [("comet", &comet), ("datafusion", datafusion.as_ref())] {
99+
group.bench_function(
100+
BenchmarkId::new(format!("{op}_ungrouped_{engine}"), data),
101+
|b| b.iter(|| black_box(ungrouped(udaf))),
102+
);
103+
group.bench_function(
104+
BenchmarkId::new(format!("{op}_grouped_{engine}"), data),
105+
|b| b.iter(|| black_box(grouped(udaf))),
106+
);
107+
}
108+
}
109+
}
110+
group.finish();
111+
}
112+
113+
criterion_group!(benches, criterion_benchmark);
114+
criterion_main!(benches);

0 commit comments

Comments
 (0)