Skip to content

Commit 7a24bf9

Browse files
authored
Merge branch 'main' into fix/6746-multi-store-scan-partitions
2 parents e0227d2 + b563496 commit 7a24bf9

22 files changed

Lines changed: 1691 additions & 141 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,6 +408,7 @@ For full documentation on the test file format, including directives like `Confi
408408

409409
- **Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively.
410410
- **Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.
411+
- **For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.** [Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it.
411412
- **Keep one file per expression** to make failures easy to locate.
412413
- **Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md).
413414

‎docs/source/contributor-guide/floating_point.md‎

Lines changed: 207 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/source/contributor-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ Development Guide <development>
4747
Comet Plugin Overview <plugin_overview>
4848
Arrow FFI <ffi>
4949
Timezone Handling <timezones>
50+
Floating-Point Semantics <floating_point>
5051
JVM Shuffle <jvm_shuffle>
5152
Native Shuffle <native_shuffle>
5253
Memory Management <memory_management>

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,20 +23,43 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
2323
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
2424
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
2525
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
26-
For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
27-
execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
28-
also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
29-
literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
30-
forms when a list contains zero.
26+
For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
27+
normalizes both operands before native execution, including noncanonical NaN literals. This
28+
applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
29+
clauses, join conditions, sort keys, and generator arguments.
30+
31+
A native Parquet scan prunes row groups and pages with its data filters. So that this pruning
32+
still applies, a `FLOAT` or `DOUBLE` column compared with a constant other than NaN in a data
33+
filter is compared without normalizing the column, and the filter above the scan evaluates the
34+
comparison again with Spark's semantics. Every other comparison in a data filter is normalized.
35+
Bloom filters store `-0.0` and `0.0` separately, so `=` against either zero checks them for both.
36+
With `spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also drops the rows its data
37+
filters reject, so every comparison in a data filter is normalized, and a `FLOAT` or `DOUBLE`
38+
comparison in a data filter does not prune row groups or pages
39+
([#6702](https://github.com/apache/datafusion-comet/issues/6702)).
40+
41+
Spark's own Parquet reader keeps `-0.0` and `0.0` apart when it prunes row groups with dictionaries
42+
and bloom filters, so it can skip a row group holding rows that its filter matches. For example,
43+
`WHERE d = -0.0D` can skip a row group that holds `0.0` but not `-0.0`, and so can `<=>` or `IN`
44+
against `-0.0D`. `WHERE d >= 0.0D` can skip a dictionary-encoded row group whose largest value is
45+
`-0.0`, and `WHERE d <= -0.0D` one whose smallest value is `0.0`. Comet reads those row groups, so
46+
it can return rows that Spark does not. Comet's result is the one that Spark's comparison semantics
47+
call for, and Spark returns the same rows with `spark.sql.parquet.filterPushdown=false`.
48+
49+
Top-level `IN`, `InSet`, and `NOT IN` membership also normalize dynamic candidates and lists
50+
containing NaN. When every candidate is a non-NaN literal, Comet keeps DataFusion's static filter
51+
and pruning path, enumerating both signed-zero forms when a list contains zero.
3152

3253
This scalar membership handling does not yet recurse into floating-point leaves nested in arrays
3354
or structs; see [#6019](https://github.com/apache/datafusion-comet/issues/6019).
3455

35-
## Nested equality and membership
56+
## Nested comparisons and membership
3657

3758
For arrays and structs containing `FLOAT` or `DOUBLE`, native `=`, `<>`, `IN`, and `NOT IN`
3859
compare signed zeros as equal and all NaN representations as equal, matching Spark. This also
39-
covers single-candidate membership that Spark rewrites into equality.
60+
covers single-candidate membership that Spark rewrites into equality. `<=>`, `<`, `<=`, `>`, and
61+
`>=` normalize the floating-point values inside both operands first, so they follow Spark's order
62+
too, in which NaN sorts above every other value.
4063

4164
Equality and dynamic membership compare nested elements directly and stop at the first mismatch.
4265
Constant membership sets use normalized comparison values for static lookup. These operations

‎docs/source/user-guide/latest/tuning/scans.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ decode step and lazily materializes projected columns for surviving rows. This c
2828
CPU and memory when the filter is highly selective on a small subset of columns. It is disabled by default
2929
because it can hurt when the filter is not selective or when most columns must be read anyway. Row-group,
3030
page-index, and bloom-filter pruning happen regardless of this flag whenever Spark's
31-
`spark.sql.parquet.filterPushdown` is on.
31+
`spark.sql.parquet.filterPushdown` is on, except for filters that compare `FLOAT` or `DOUBLE` values,
32+
which do not prune while this flag is on (see [Floating-point Number Comparison](../compatibility/floating-point.md)).
3233

3334
### Parquet Native Scans
3435

‎native/core/src/execution/planner.rs‎

Lines changed: 176 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -145,10 +145,10 @@ use datafusion_comet_proto::{
145145
use datafusion_comet_spark_expr::{
146146
create_case_when, create_if_expr, jvm_udf::JvmScalarUdfExpr, spark_in_list, ApproxPercentile,
147147
ArrayInsert, Avg, AvgDecimal, Cast, CheckOverflow, Correlation, Covariance, CreateNamedStruct,
148-
DecimalRescaleCheckOverflow, GetArrayStructFields, GetStructField, HllPlusPlus, HllSketchAgg,
149-
HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero, NormalizeNestedFloats,
150-
Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal, ToJson, UnboundColumn,
151-
Variance, WideDecimalBinaryExpr, WideDecimalOp,
148+
DecimalRescaleCheckOverflow, FloatOperands, GetArrayStructFields, GetStructField, HllPlusPlus,
149+
HllSketchAgg, HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero,
150+
NormalizeNestedFloats, Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal,
151+
ToJson, UnboundColumn, Variance, WideDecimalBinaryExpr, WideDecimalOp,
152152
};
153153
use itertools::Itertools;
154154
use jni::objects::{Global, JObject};
@@ -326,6 +326,7 @@ pub struct BinaryExprOptions {
326326
pub const TEST_EXEC_CONTEXT_ID: i64 = -1;
327327

328328
/// The query planner for converting Spark query plans to DataFusion query plans.
329+
#[derive(Clone)]
329330
pub struct PhysicalPlanner {
330331
// The execution context id of this planner.
331332
exec_context_id: i64,
@@ -347,6 +348,9 @@ pub struct PhysicalPlanner {
347348
/// Task-owned destination for remote shuffle blocks, registered on the driving Spark task
348349
/// thread before native planning. Only explicit RSS destinations may use it.
349350
shuffle_partition_pusher: Option<Arc<dyn ShufflePartitionPusher>>,
351+
/// How comparisons treat floating-point operands. `Raw` only while planning a scan's data
352+
/// filters; see [`Self::create_data_filter`].
353+
float_operands: FloatOperands,
350354
}
351355

352356
impl Default for PhysicalPlanner {
@@ -366,6 +370,7 @@ impl PhysicalPlanner {
366370
task_context: None,
367371
class_loader: None,
368372
shuffle_partition_pusher: None,
373+
float_operands: FloatOperands::Normalize,
369374
}
370375
}
371376

@@ -478,6 +483,11 @@ impl PhysicalPlanner {
478483
self
479484
}
480485

486+
/// How comparisons treat floating-point operands.
487+
pub fn float_operands(&self) -> FloatOperands {
488+
self.float_operands
489+
}
490+
481491
/// Return session context of this planner.
482492
pub fn session_ctx(&self) -> &Arc<SessionContext> {
483493
&self.session_ctx
@@ -1153,6 +1163,45 @@ impl PhysicalPlanner {
11531163
}
11541164
}
11551165

1166+
/// Create a data filter that a scan pushes into the Parquet reader, with float operands
1167+
/// treated as [`Self::data_filter_float_operands`] decides.
1168+
fn create_data_filter(
1169+
&self,
1170+
spark_expr: &Expr,
1171+
input_schema: SchemaRef,
1172+
float_operands: FloatOperands,
1173+
) -> Result<Arc<dyn PhysicalExpr>, ExecutionError> {
1174+
let planner = Self {
1175+
float_operands,
1176+
..self.clone()
1177+
};
1178+
planner.create_expr(spark_expr, input_schema)
1179+
}
1180+
1181+
/// How a scan's data filters treat float operands. The Parquet reader prunes row groups and
1182+
/// pages with them, and with row-level pushdown (`pushdown_filters`) it also drops the rows
1183+
/// they reject, which Spark's Filter above the scan then never sees.
1184+
///
1185+
/// Without row-level pushdown the filters only prune, and pruning only recognizes a column
1186+
/// compared with a literal, so that shape keeps the raw column ([`FloatOperands::Raw`]). With
1187+
/// it, every operand is normalized and float comparisons give up pruning: a raw column would
1188+
/// drop a stored NaN that Spark matches, such as one with the sign bit set, which Arrow orders
1189+
/// below every other value.
1190+
fn data_filter_float_operands(&self) -> FloatOperands {
1191+
if self
1192+
.session_ctx
1193+
.copied_config()
1194+
.options()
1195+
.execution
1196+
.parquet
1197+
.pushdown_filters
1198+
{
1199+
FloatOperands::Normalize
1200+
} else {
1201+
FloatOperands::Raw
1202+
}
1203+
}
1204+
11561205
/// Normalize floating-point comparison keys without changing output values: a `FLOAT` or
11571206
/// `DOUBLE` key, and an array or struct key with a float at any depth, whose order Arrow
11581207
/// otherwise takes from the raw bits. Sort, Window, and WindowGroupLimit must use identical
@@ -1934,10 +1983,17 @@ impl PhysicalPlanner {
19341983
.cloned()
19351984
.collect::<Vec<FieldRef>>(),
19361985
));
1986+
let float_operands = self.data_filter_float_operands();
19371987
common
19381988
.data_filters
19391989
.iter()
1940-
.map(|expr| self.create_expr(expr, Arc::clone(&filter_schema)))
1990+
.map(|expr| {
1991+
self.create_data_filter(
1992+
expr,
1993+
Arc::clone(&filter_schema),
1994+
float_operands,
1995+
)
1996+
})
19411997
.collect()
19421998
};
19431999

@@ -5860,6 +5916,121 @@ mod tests {
58605916
assert_eq!(0, filter_exec.additional_native_plans.len());
58615917
}
58625918

5919+
/// Comparisons normalize float operands, except in the data filters that a scan pushes into
5920+
/// the Parquet reader, where pruning has to see the column itself.
5921+
#[test]
5922+
fn scan_data_filters_compare_float_columns_directly() {
5923+
use datafusion::physical_expr::expressions::BinaryExpr;
5924+
use datafusion_comet_spark_expr::NormalizeNaNAndZero;
5925+
let double = spark_expression::DataType {
5926+
type_id: 6,
5927+
type_info: None,
5928+
};
5929+
let operand = |expr_struct| Expr {
5930+
expr_struct: Some(expr_struct),
5931+
query_context: None,
5932+
expr_id: None,
5933+
};
5934+
let expr = operand(Gt(Box::new(spark_expression::BinaryExpr {
5935+
left: Some(Box::new(operand(Bound(spark_expression::BoundReference {
5936+
index: 0,
5937+
datatype: Some(double.clone()),
5938+
})))),
5939+
right: Some(Box::new(operand(Literal(spark_expression::Literal {
5940+
value: Some(literal::Value::DoubleVal(500.0)),
5941+
datatype: Some(double),
5942+
is_null: false,
5943+
})))),
5944+
})));
5945+
let schema = Arc::new(Schema::new(vec![Field::new("d", DataType::Float64, true)]));
5946+
let left_operand = |expr: Arc<dyn PhysicalExpr>| {
5947+
let comparison = expr.downcast_ref::<BinaryExpr>().expect("a comparison");
5948+
Arc::clone(comparison.left())
5949+
};
5950+
let planner = PhysicalPlanner::default();
5951+
let comparison = planner.create_expr(&expr, Arc::clone(&schema)).unwrap();
5952+
assert!(left_operand(comparison)
5953+
.downcast_ref::<NormalizeNaNAndZero>()
5954+
.is_some());
5955+
let data_filter = planner
5956+
.create_data_filter(&expr, schema, planner.data_filter_float_operands())
5957+
.unwrap();
5958+
assert!(left_operand(data_filter).downcast_ref::<Column>().is_some());
5959+
}
5960+
5961+
/// With row-level pushdown the Parquet reader drops the rows a data filter rejects, so the
5962+
/// filter normalizes the column as well as the literal. A raw column would drop a stored NaN
5963+
/// whose bits differ from the normalized literal, and a stored NaN with the sign bit set
5964+
/// under any ordering comparison, both of which Spark matches.
5965+
#[test]
5966+
fn scan_data_filters_normalize_float_columns_with_row_level_pushdown() {
5967+
use arrow::array::{AsArray, BooleanArray};
5968+
use datafusion::physical_expr::expressions::BinaryExpr;
5969+
use datafusion_comet_spark_expr::{FloatOperands, NormalizeNaNAndZero};
5970+
let double = spark_expression::DataType {
5971+
type_id: 6,
5972+
type_info: None,
5973+
};
5974+
let operand = |expr_struct| Expr {
5975+
expr_struct: Some(expr_struct),
5976+
query_context: None,
5977+
expr_id: None,
5978+
};
5979+
let column_and = |value: f64| {
5980+
Box::new(spark_expression::BinaryExpr {
5981+
left: Some(Box::new(operand(Bound(spark_expression::BoundReference {
5982+
index: 0,
5983+
datatype: Some(double.clone()),
5984+
})))),
5985+
right: Some(Box::new(operand(Literal(spark_expression::Literal {
5986+
value: Some(literal::Value::DoubleVal(value)),
5987+
datatype: Some(double.clone()),
5988+
is_null: false,
5989+
})))),
5990+
})
5991+
};
5992+
// Spark folds `-double('NaN')` into a literal with the sign bit set.
5993+
let negative_nan = f64::from_bits(0xfff8_0000_0000_0000);
5994+
let schema = Arc::new(Schema::new(vec![Field::new("d", DataType::Float64, true)]));
5995+
let batch = RecordBatch::try_new(
5996+
Arc::clone(&schema),
5997+
vec![Arc::new(Float64Array::from(vec![
5998+
negative_nan,
5999+
f64::NAN,
6000+
1.0,
6001+
]))],
6002+
)
6003+
.unwrap();
6004+
let config =
6005+
SessionConfig::new().set_bool("datafusion.execution.parquet.pushdown_filters", true);
6006+
let planner = PhysicalPlanner::new(Arc::new(SessionContext::new_with_config(config)), 0);
6007+
assert_eq!(
6008+
planner.data_filter_float_operands(),
6009+
FloatOperands::Normalize
6010+
);
6011+
for (expr, expected) in [
6012+
(operand(Eq(column_and(negative_nan))), [true, true, false]),
6013+
(operand(Gt(column_and(0.0))), [true, true, true]),
6014+
] {
6015+
let data_filter = planner
6016+
.create_data_filter(
6017+
&expr,
6018+
Arc::clone(&schema),
6019+
planner.data_filter_float_operands(),
6020+
)
6021+
.unwrap();
6022+
let comparison = data_filter
6023+
.downcast_ref::<BinaryExpr>()
6024+
.expect("a comparison");
6025+
assert!(comparison
6026+
.left()
6027+
.downcast_ref::<NormalizeNaNAndZero>()
6028+
.is_some());
6029+
let matched = data_filter.evaluate(&batch).unwrap().into_array(3).unwrap();
6030+
assert_eq!(matched.as_boolean(), &BooleanArray::from(expected.to_vec()));
6031+
}
6032+
}
6033+
58636034
#[test]
58646035
fn spark_plan_metrics_hash_join() {
58656036
let op_scan = create_scan();

‎native/core/src/execution/planner/macros.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ macro_rules! binary_expr_builder {
9898
$operator,
9999
right,
100100
input_schema.as_ref(),
101+
planner.float_operands(),
101102
)
102103
.map_err(Into::into)
103104
}

0 commit comments

Comments
 (0)