Skip to content

Commit 9e54913

Browse files
committed
Merge branch 'main' into perf/2545-hash-join-build-size-guard
2 parents d657266 + b563496 commit 9e54913

22 files changed

Lines changed: 1691 additions & 141 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,6 +408,7 @@ For full documentation on the test file format, including directives like `Confi
408408

409409
- **Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively.
410410
- **Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.
411+
- **For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.** [Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it.
411412
- **Keep one file per expression** to make failures easy to locate.
412413
- **Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md).
413414

‎docs/source/contributor-guide/floating_point.md‎

Lines changed: 207 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/source/contributor-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ Development Guide <development>
4747
Comet Plugin Overview <plugin_overview>
4848
Arrow FFI <ffi>
4949
Timezone Handling <timezones>
50+
Floating-Point Semantics <floating_point>
5051
JVM Shuffle <jvm_shuffle>
5152
Native Shuffle <native_shuffle>
5253
Memory Management <memory_management>

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,20 +23,43 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
2323
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
2424
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
2525
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
26-
For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
27-
execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
28-
also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
29-
literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
30-
forms when a list contains zero.
26+
For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
27+
normalizes both operands before native execution, including noncanonical NaN literals. This
28+
applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
29+
clauses, join conditions, sort keys, and generator arguments.
30+
31+
A native Parquet scan prunes row groups and pages with its data filters. So that this pruning
32+
still applies, a `FLOAT` or `DOUBLE` column compared with a constant other than NaN in a data
33+
filter is compared without normalizing the column, and the filter above the scan evaluates the
34+
comparison again with Spark's semantics. Every other comparison in a data filter is normalized.
35+
Bloom filters store `-0.0` and `0.0` separately, so `=` against either zero checks them for both.
36+
With `spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also drops the rows its data
37+
filters reject, so every comparison in a data filter is normalized, and a `FLOAT` or `DOUBLE`
38+
comparison in a data filter does not prune row groups or pages
39+
([#6702](https://github.com/apache/datafusion-comet/issues/6702)).
40+
41+
Spark's own Parquet reader keeps `-0.0` and `0.0` apart when it prunes row groups with dictionaries
42+
and bloom filters, so it can skip a row group holding rows that its filter matches. For example,
43+
`WHERE d = -0.0D` can skip a row group that holds `0.0` but not `-0.0`, and so can `<=>` or `IN`
44+
against `-0.0D`. `WHERE d >= 0.0D` can skip a dictionary-encoded row group whose largest value is
45+
`-0.0`, and `WHERE d <= -0.0D` one whose smallest value is `0.0`. Comet reads those row groups, so
46+
it can return rows that Spark does not. Comet's result is the one that Spark's comparison semantics
47+
call for, and Spark returns the same rows with `spark.sql.parquet.filterPushdown=false`.
48+
49+
Top-level `IN`, `InSet`, and `NOT IN` membership also normalize dynamic candidates and lists
50+
containing NaN. When every candidate is a non-NaN literal, Comet keeps DataFusion's static filter
51+
and pruning path, enumerating both signed-zero forms when a list contains zero.
3152

3253
This scalar membership handling does not yet recurse into floating-point leaves nested in arrays
3354
or structs; see [#6019](https://github.com/apache/datafusion-comet/issues/6019).
3455

35-
## Nested equality and membership
56+
## Nested comparisons and membership
3657

3758
For arrays and structs containing `FLOAT` or `DOUBLE`, native `=`, `<>`, `IN`, and `NOT IN`
3859
compare signed zeros as equal and all NaN representations as equal, matching Spark. This also
39-
covers single-candidate membership that Spark rewrites into equality.
60+
covers single-candidate membership that Spark rewrites into equality. `<=>`, `<`, `<=`, `>`, and
61+
`>=` normalize the floating-point values inside both operands first, so they follow Spark's order
62+
too, in which NaN sorts above every other value.
4063

4164
Equality and dynamic membership compare nested elements directly and stop at the first mismatch.
4265
Constant membership sets use normalized comparison values for static lookup. These operations

‎docs/source/user-guide/latest/tuning/scans.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ decode step and lazily materializes projected columns for surviving rows. This c
2828
CPU and memory when the filter is highly selective on a small subset of columns. It is disabled by default
2929
because it can hurt when the filter is not selective or when most columns must be read anyway. Row-group,
3030
page-index, and bloom-filter pruning happen regardless of this flag whenever Spark's
31-
`spark.sql.parquet.filterPushdown` is on.
31+
`spark.sql.parquet.filterPushdown` is on, except for filters that compare `FLOAT` or `DOUBLE` values,
32+
which do not prune while this flag is on (see [Floating-point Number Comparison](../compatibility/floating-point.md)).
3233

3334
### Parquet Native Scans
3435

‎native/core/src/execution/planner.rs‎

Lines changed: 176 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -144,10 +144,10 @@ use datafusion_comet_proto::{
144144
use datafusion_comet_spark_expr::{
145145
create_case_when, create_if_expr, jvm_udf::JvmScalarUdfExpr, spark_in_list, ApproxPercentile,
146146
ArrayInsert, Avg, AvgDecimal, Cast, CheckOverflow, Correlation, Covariance, CreateNamedStruct,
147-
DecimalRescaleCheckOverflow, GetArrayStructFields, GetStructField, HllPlusPlus, HllSketchAgg,
148-
HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero, NormalizeNestedFloats,
149-
Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal, ToJson, UnboundColumn,
150-
Variance, WideDecimalBinaryExpr, WideDecimalOp,
147+
DecimalRescaleCheckOverflow, FloatOperands, GetArrayStructFields, GetStructField, HllPlusPlus,
148+
HllSketchAgg, HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero,
149+
NormalizeNestedFloats, Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal,
150+
ToJson, UnboundColumn, Variance, WideDecimalBinaryExpr, WideDecimalOp,
151151
};
152152
use itertools::Itertools;
153153
use jni::objects::{Global, JObject};
@@ -325,6 +325,7 @@ pub struct BinaryExprOptions {
325325
pub const TEST_EXEC_CONTEXT_ID: i64 = -1;
326326

327327
/// The query planner for converting Spark query plans to DataFusion query plans.
328+
#[derive(Clone)]
328329
pub struct PhysicalPlanner {
329330
// The execution context id of this planner.
330331
exec_context_id: i64,
@@ -346,6 +347,9 @@ pub struct PhysicalPlanner {
346347
/// Task-owned destination for remote shuffle blocks, registered on the driving Spark task
347348
/// thread before native planning. Only explicit RSS destinations may use it.
348349
shuffle_partition_pusher: Option<Arc<dyn ShufflePartitionPusher>>,
350+
/// How comparisons treat floating-point operands. `Raw` only while planning a scan's data
351+
/// filters; see [`Self::create_data_filter`].
352+
float_operands: FloatOperands,
349353
}
350354

351355
impl Default for PhysicalPlanner {
@@ -365,6 +369,7 @@ impl PhysicalPlanner {
365369
task_context: None,
366370
class_loader: None,
367371
shuffle_partition_pusher: None,
372+
float_operands: FloatOperands::Normalize,
368373
}
369374
}
370375

@@ -477,6 +482,11 @@ impl PhysicalPlanner {
477482
self
478483
}
479484

485+
/// How comparisons treat floating-point operands.
486+
pub fn float_operands(&self) -> FloatOperands {
487+
self.float_operands
488+
}
489+
480490
/// Return session context of this planner.
481491
pub fn session_ctx(&self) -> &Arc<SessionContext> {
482492
&self.session_ctx
@@ -1129,6 +1139,45 @@ impl PhysicalPlanner {
11291139
}
11301140
}
11311141

1142+
/// Create a data filter that a scan pushes into the Parquet reader, with float operands
1143+
/// treated as [`Self::data_filter_float_operands`] decides.
1144+
fn create_data_filter(
1145+
&self,
1146+
spark_expr: &Expr,
1147+
input_schema: SchemaRef,
1148+
float_operands: FloatOperands,
1149+
) -> Result<Arc<dyn PhysicalExpr>, ExecutionError> {
1150+
let planner = Self {
1151+
float_operands,
1152+
..self.clone()
1153+
};
1154+
planner.create_expr(spark_expr, input_schema)
1155+
}
1156+
1157+
/// How a scan's data filters treat float operands. The Parquet reader prunes row groups and
1158+
/// pages with them, and with row-level pushdown (`pushdown_filters`) it also drops the rows
1159+
/// they reject, which Spark's Filter above the scan then never sees.
1160+
///
1161+
/// Without row-level pushdown the filters only prune, and pruning only recognizes a column
1162+
/// compared with a literal, so that shape keeps the raw column ([`FloatOperands::Raw`]). With
1163+
/// it, every operand is normalized and float comparisons give up pruning: a raw column would
1164+
/// drop a stored NaN that Spark matches, such as one with the sign bit set, which Arrow orders
1165+
/// below every other value.
1166+
fn data_filter_float_operands(&self) -> FloatOperands {
1167+
if self
1168+
.session_ctx
1169+
.copied_config()
1170+
.options()
1171+
.execution
1172+
.parquet
1173+
.pushdown_filters
1174+
{
1175+
FloatOperands::Normalize
1176+
} else {
1177+
FloatOperands::Raw
1178+
}
1179+
}
1180+
11321181
/// Normalize floating-point comparison keys without changing output values: a `FLOAT` or
11331182
/// `DOUBLE` key, and an array or struct key with a float at any depth, whose order Arrow
11341183
/// otherwise takes from the raw bits. Sort, Window, and WindowGroupLimit must use identical
@@ -1910,10 +1959,17 @@ impl PhysicalPlanner {
19101959
.cloned()
19111960
.collect::<Vec<FieldRef>>(),
19121961
));
1962+
let float_operands = self.data_filter_float_operands();
19131963
common
19141964
.data_filters
19151965
.iter()
1916-
.map(|expr| self.create_expr(expr, Arc::clone(&filter_schema)))
1966+
.map(|expr| {
1967+
self.create_data_filter(
1968+
expr,
1969+
Arc::clone(&filter_schema),
1970+
float_operands,
1971+
)
1972+
})
19171973
.collect()
19181974
};
19191975

@@ -5826,6 +5882,121 @@ mod tests {
58265882
assert_eq!(0, filter_exec.additional_native_plans.len());
58275883
}
58285884

5885+
/// Comparisons normalize float operands, except in the data filters that a scan pushes into
5886+
/// the Parquet reader, where pruning has to see the column itself.
5887+
#[test]
5888+
fn scan_data_filters_compare_float_columns_directly() {
5889+
use datafusion::physical_expr::expressions::BinaryExpr;
5890+
use datafusion_comet_spark_expr::NormalizeNaNAndZero;
5891+
let double = spark_expression::DataType {
5892+
type_id: 6,
5893+
type_info: None,
5894+
};
5895+
let operand = |expr_struct| Expr {
5896+
expr_struct: Some(expr_struct),
5897+
query_context: None,
5898+
expr_id: None,
5899+
};
5900+
let expr = operand(Gt(Box::new(spark_expression::BinaryExpr {
5901+
left: Some(Box::new(operand(Bound(spark_expression::BoundReference {
5902+
index: 0,
5903+
datatype: Some(double.clone()),
5904+
})))),
5905+
right: Some(Box::new(operand(Literal(spark_expression::Literal {
5906+
value: Some(literal::Value::DoubleVal(500.0)),
5907+
datatype: Some(double),
5908+
is_null: false,
5909+
})))),
5910+
})));
5911+
let schema = Arc::new(Schema::new(vec![Field::new("d", DataType::Float64, true)]));
5912+
let left_operand = |expr: Arc<dyn PhysicalExpr>| {
5913+
let comparison = expr.downcast_ref::<BinaryExpr>().expect("a comparison");
5914+
Arc::clone(comparison.left())
5915+
};
5916+
let planner = PhysicalPlanner::default();
5917+
let comparison = planner.create_expr(&expr, Arc::clone(&schema)).unwrap();
5918+
assert!(left_operand(comparison)
5919+
.downcast_ref::<NormalizeNaNAndZero>()
5920+
.is_some());
5921+
let data_filter = planner
5922+
.create_data_filter(&expr, schema, planner.data_filter_float_operands())
5923+
.unwrap();
5924+
assert!(left_operand(data_filter).downcast_ref::<Column>().is_some());
5925+
}
5926+
5927+
/// With row-level pushdown the Parquet reader drops the rows a data filter rejects, so the
5928+
/// filter normalizes the column as well as the literal. A raw column would drop a stored NaN
5929+
/// whose bits differ from the normalized literal, and a stored NaN with the sign bit set
5930+
/// under any ordering comparison, both of which Spark matches.
5931+
#[test]
5932+
fn scan_data_filters_normalize_float_columns_with_row_level_pushdown() {
5933+
use arrow::array::{AsArray, BooleanArray};
5934+
use datafusion::physical_expr::expressions::BinaryExpr;
5935+
use datafusion_comet_spark_expr::{FloatOperands, NormalizeNaNAndZero};
5936+
let double = spark_expression::DataType {
5937+
type_id: 6,
5938+
type_info: None,
5939+
};
5940+
let operand = |expr_struct| Expr {
5941+
expr_struct: Some(expr_struct),
5942+
query_context: None,
5943+
expr_id: None,
5944+
};
5945+
let column_and = |value: f64| {
5946+
Box::new(spark_expression::BinaryExpr {
5947+
left: Some(Box::new(operand(Bound(spark_expression::BoundReference {
5948+
index: 0,
5949+
datatype: Some(double.clone()),
5950+
})))),
5951+
right: Some(Box::new(operand(Literal(spark_expression::Literal {
5952+
value: Some(literal::Value::DoubleVal(value)),
5953+
datatype: Some(double.clone()),
5954+
is_null: false,
5955+
})))),
5956+
})
5957+
};
5958+
// Spark folds `-double('NaN')` into a literal with the sign bit set.
5959+
let negative_nan = f64::from_bits(0xfff8_0000_0000_0000);
5960+
let schema = Arc::new(Schema::new(vec![Field::new("d", DataType::Float64, true)]));
5961+
let batch = RecordBatch::try_new(
5962+
Arc::clone(&schema),
5963+
vec![Arc::new(Float64Array::from(vec![
5964+
negative_nan,
5965+
f64::NAN,
5966+
1.0,
5967+
]))],
5968+
)
5969+
.unwrap();
5970+
let config =
5971+
SessionConfig::new().set_bool("datafusion.execution.parquet.pushdown_filters", true);
5972+
let planner = PhysicalPlanner::new(Arc::new(SessionContext::new_with_config(config)), 0);
5973+
assert_eq!(
5974+
planner.data_filter_float_operands(),
5975+
FloatOperands::Normalize
5976+
);
5977+
for (expr, expected) in [
5978+
(operand(Eq(column_and(negative_nan))), [true, true, false]),
5979+
(operand(Gt(column_and(0.0))), [true, true, true]),
5980+
] {
5981+
let data_filter = planner
5982+
.create_data_filter(
5983+
&expr,
5984+
Arc::clone(&schema),
5985+
planner.data_filter_float_operands(),
5986+
)
5987+
.unwrap();
5988+
let comparison = data_filter
5989+
.downcast_ref::<BinaryExpr>()
5990+
.expect("a comparison");
5991+
assert!(comparison
5992+
.left()
5993+
.downcast_ref::<NormalizeNaNAndZero>()
5994+
.is_some());
5995+
let matched = data_filter.evaluate(&batch).unwrap().into_array(3).unwrap();
5996+
assert_eq!(matched.as_boolean(), &BooleanArray::from(expected.to_vec()));
5997+
}
5998+
}
5999+
58296000
#[test]
58306001
fn spark_plan_metrics_hash_join() {
58316002
let op_scan = create_scan();

‎native/core/src/execution/planner/macros.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ macro_rules! binary_expr_builder {
9898
$operator,
9999
right,
100100
input_schema.as_ref(),
101+
planner.float_operands(),
101102
)
102103
.map_err(Into::into)
103104
}

0 commit comments

Comments
 (0)