Skip to content

Commit 12e95e6

Browse files
committed
fix: normalize computed operands in scan data filters
With spark.comet.parquet.rowFilterPushdown.enabled the Parquet reader evaluates a scan's data filters on each row, and a row it drops never reaches the Filter above the scan. Data filters left every float operand raw, so a computed operand such as -d, whose NaN has the sign bit set, dropped the NaN row that Spark keeps. Leave only a float column compared with a literal raw, the shape Parquet pruning reads, and normalize every other comparison. Run float_comparisons.sql with row-level pushdown off and on, add filters on stored edge values, add a benchmark of Spark's float comparisons against DataFusion's, and share one helper for nested types with a float leaf.
1 parent 5fd00d1 commit 12e95e6

8 files changed

Lines changed: 218 additions & 35 deletions

File tree

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,14 @@ normalizes both operands before native execution, including noncanonical NaN lit
2828
applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
2929
clauses, join conditions, sort keys, and generator arguments.
3030

31-
The data filters that a native Parquet scan uses to skip row groups compare values without
32-
normalizing them, so that statistics pruning still applies, and the filter above the scan
33-
evaluates them again with Spark's semantics. With
31+
A native Parquet scan skips row groups with its data filters. So that statistics pruning still
32+
applies, a `FLOAT` or `DOUBLE` column compared with a constant in a data filter is compared
33+
without normalizing the column, and the filter above the scan evaluates the comparison again with
34+
Spark's semantics. Every other comparison in a data filter is normalized. With
3435
`spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also filters rows with these
35-
comparisons, so a noncanonical NaN, such as one with the sign bit set, can be filtered differently
36-
from Spark. Spark's Parquet writer only writes canonical NaNs.
36+
comparisons, so a noncanonical NaN stored in the file, such as one with the sign bit set, can be
37+
filtered differently from Spark when compared with a constant. Spark's Parquet writer only writes
38+
canonical NaNs.
3739

3840
Top-level `IN`, `InSet`, and `NOT IN` membership also normalize dynamic candidates and lists
3941
containing NaN. When every candidate is a non-NaN literal, Comet keeps DataFusion's static filter

‎native/core/src/execution/planner.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1019,9 +1019,9 @@ impl PhysicalPlanner {
10191019

10201020
/// Create a data filter that a scan pushes into the Parquet reader. The filter prunes row
10211021
/// groups and pages, and rows too when row-level pushdown is enabled. Pruning only recognizes
1022-
/// a column compared with a literal, so the filter's comparisons leave float operands as they
1023-
/// are rather than normalizing them. Spark's Filter above the scan evaluates the filter again
1024-
/// with Spark's semantics.
1022+
/// a column compared with a literal, so that shape leaves a float column as it is rather than
1023+
/// normalizing it, and every other comparison is normalized; see [`FloatOperands::Raw`].
1024+
/// Spark's Filter above the scan evaluates the filter again with Spark's semantics.
10251025
fn create_data_filter(
10261026
&self,
10271027
spark_expr: &Expr,

‎native/spark-expr/Cargo.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -443,3 +443,7 @@ harness = false
443443
[[bench]]
444444
name = "nested_comparison"
445445
harness = false
446+
447+
[[bench]]
448+
name = "float_comparison"
449+
harness = false
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Compare Spark's float comparisons, which normalize each float operand first, with DataFusion's
19+
//! on the same data, and time `normalize_floats` on its own. Both comparisons return the same
20+
//! answer except on a NaN with the sign bit set, which is checked before timing; special-value
21+
//! semantics belong in tests.
22+
23+
use arrow::array::{Array, ArrayRef, Float64Array};
24+
use arrow::datatypes::{DataType, Field, Schema};
25+
use arrow::record_batch::RecordBatch;
26+
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
27+
use datafusion::common::ScalarValue;
28+
use datafusion::logical_expr::Operator;
29+
use datafusion::physical_expr::expressions::{BinaryExpr, Column, Literal};
30+
use datafusion::physical_expr::PhysicalExpr;
31+
use datafusion_comet_spark_expr::{normalize_floats, spark_comparison, FloatOperands};
32+
use std::hint::black_box;
33+
use std::sync::Arc;
34+
use std::time::Duration;
35+
36+
const ROWS: usize = 8192;
37+
38+
/// A NaN with the sign bit set, which arithmetic produces on x86-64.
39+
const NEGATIVE_NAN: f64 = f64::from_bits(0xfff8_0000_0000_0000);
40+
41+
/// Two columns of doubles, with `special` in every tenth row of each if given.
42+
fn batch(special: Option<f64>) -> RecordBatch {
43+
let column = |seed: usize| -> ArrayRef {
44+
Arc::new(Float64Array::from_iter_values((0..ROWS).map(
45+
|i| match special {
46+
Some(value) if i % 10 == seed => value,
47+
_ => ((i * 7 + seed) % 1000) as f64 * 1.5,
48+
},
49+
)))
50+
};
51+
let schema = Arc::new(Schema::new(vec![
52+
Field::new("a", DataType::Float64, true),
53+
Field::new("b", DataType::Float64, true),
54+
]));
55+
RecordBatch::try_new(schema, vec![column(1), column(3)]).unwrap()
56+
}
57+
58+
fn criterion_benchmark(c: &mut Criterion) {
59+
let mut group = c.benchmark_group("float_comparison");
60+
group.sample_size(20);
61+
group.warm_up_time(Duration::from_millis(250));
62+
group.measurement_time(Duration::from_secs(1));
63+
for (data, special) in [
64+
("no_special", None),
65+
("neg_zero_10pct", Some(-0.0)),
66+
("nan_10pct", Some(f64::NAN)),
67+
("sign_bit_nan_10pct", Some(NEGATIVE_NAN)),
68+
] {
69+
let batch = batch(special);
70+
let schema = batch.schema();
71+
let a: Arc<dyn PhysicalExpr> = Arc::new(Column::new("a", 0));
72+
let b: Arc<dyn PhysicalExpr> = Arc::new(Column::new("b", 1));
73+
let literal: Arc<dyn PhysicalExpr> =
74+
Arc::new(Literal::new(ScalarValue::Float64(Some(500.0))));
75+
for (shape, right) in [("column_column", b), ("column_literal", literal)] {
76+
let datafusion: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
77+
Arc::clone(&a),
78+
Operator::Lt,
79+
Arc::clone(&right),
80+
));
81+
let spark = spark_comparison(
82+
Arc::clone(&a),
83+
Operator::Lt,
84+
right,
85+
&schema,
86+
FloatOperands::Normalize,
87+
)
88+
.unwrap();
89+
let evaluate = |expr: &Arc<dyn PhysicalExpr>| {
90+
expr.evaluate(&batch)
91+
.unwrap()
92+
.into_array(ROWS)
93+
.unwrap()
94+
.to_data()
95+
};
96+
if special.map(f64::to_bits) != Some(NEGATIVE_NAN.to_bits()) {
97+
assert_eq!(evaluate(&spark), evaluate(&datafusion));
98+
}
99+
for (engine, expr) in [("spark", &spark), ("datafusion", &datafusion)] {
100+
group.bench_function(BenchmarkId::new(format!("{shape}_{engine}"), data), |b| {
101+
b.iter(|| black_box(expr.evaluate(black_box(&batch)).unwrap()))
102+
});
103+
}
104+
}
105+
let column = Arc::clone(batch.column(0));
106+
group.bench_function(BenchmarkId::new("normalize_floats", data), |b| {
107+
b.iter(|| black_box(normalize_floats(black_box(&column))))
108+
});
109+
}
110+
group.finish();
111+
}
112+
113+
criterion_group!(benches, criterion_benchmark);
114+
criterion_main!(benches);

‎native/spark-expr/src/array_funcs/nested_comparison.rs‎

Lines changed: 68 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@
1919
//! values without materializing normalized columns.
2020
2121
use crate::float_semantics::{
22-
has_float_leaf, normalize_comparison_operand, normalize_nested_floats, spark_equality,
23-
NormalizeNestedFloats,
22+
is_nested_with_float_leaf, normalize_comparison_operand, normalize_nested_floats,
23+
spark_equality, NormalizeNestedFloats,
2424
};
2525
use arrow::array::{Array, ArrayRef, AsArray, BooleanArray};
2626
use arrow::buffer::{BooleanBuffer, NullBuffer};
@@ -29,17 +29,13 @@ use arrow::datatypes::{DataType, Schema};
2929
use arrow::record_batch::RecordBatch;
3030
use datafusion::common::{internal_err, DFSchema, Result, ScalarValue};
3131
use datafusion::logical_expr::{ColumnarValue, Operator};
32-
use datafusion::physical_expr::expressions::{in_list, BinaryExpr, InListExpr};
32+
use datafusion::physical_expr::expressions::{in_list, BinaryExpr, Column, InListExpr, Literal};
3333
use datafusion::physical_expr::PhysicalExpr;
3434
use datafusion::physical_expr_common::physical_expr::is_volatile;
3535
use std::fmt::{Display, Formatter};
3636
use std::hash::{Hash, Hasher};
3737
use std::sync::Arc;
3838

39-
fn needs_spark_equality(dt: &DataType) -> bool {
40-
dt.is_nested() && has_float_leaf(dt)
41-
}
42-
4339
struct Operand {
4440
array: ArrayRef,
4541
scalar: bool,
@@ -226,9 +222,13 @@ impl PhysicalExpr for NestedPredicate {
226222
pub enum FloatOperands {
227223
/// Normalize them, so that the comparison follows Spark's SQL ordering.
228224
Normalize,
229-
/// Leave them as they are. Only a scan's pushed-down data filters use this: Parquet pruning
230-
/// recognizes a column compared with a literal but not a normalized column, and Spark's
231-
/// Filter above the scan evaluates each of those filters again.
225+
/// Leave a Float32 or Float64 column compared with a literal as it is, and normalize every
226+
/// other operand. Only a scan's pushed-down data filters use this: Parquet pruning recognizes
227+
/// a column compared with a literal but not a normalized column. With row-level pushdown the
228+
/// reader also evaluates the filters on each row, and a row it drops never reaches Spark's
229+
/// Filter above the scan, so any other shape, which pruning cannot use anyway, is normalized.
230+
/// A computed operand such as `-d` can hold a NaN with the sign bit set, which a raw
231+
/// comparison sorts below every other value.
232232
Raw,
233233
}
234234

@@ -260,7 +260,7 @@ pub fn spark_comparison(
260260
let (Ok(left_type), Ok(_)) = (left.data_type(schema), right.data_type(schema)) else {
261261
return Ok(Arc::new(BinaryExpr::new(left, op, right)));
262262
};
263-
if matches!(op, Eq | NotEq) && needs_spark_equality(&left_type) {
263+
if matches!(op, Eq | NotEq) && is_nested_with_float_leaf(&left_type) {
264264
validate_types(&left, std::slice::from_ref(&right), schema)?;
265265
return Ok(Arc::new(NestedPredicate {
266266
value: left,
@@ -269,16 +269,33 @@ pub fn spark_comparison(
269269
membership: false,
270270
}));
271271
}
272-
let (left, right) = match float_operands {
273-
FloatOperands::Normalize => (
272+
let raw = float_operands == FloatOperands::Raw;
273+
let (left, right) = if raw && is_float_column(&left, schema) && is_literal(&right) {
274+
(left, normalize_comparison_operand(right, schema)?)
275+
} else if raw && is_literal(&left) && is_float_column(&right, schema) {
276+
(normalize_comparison_operand(left, schema)?, right)
277+
} else {
278+
(
274279
normalize_comparison_operand(left, schema)?,
275280
normalize_comparison_operand(right, schema)?,
276-
),
277-
FloatOperands::Raw => (left, right),
281+
)
278282
};
279283
Ok(Arc::new(BinaryExpr::new(left, op, right)))
280284
}
281285

286+
/// Whether `expr` is a Float32 or Float64 column, the operand that Parquet pruning reads.
287+
fn is_float_column(expr: &Arc<dyn PhysicalExpr>, schema: &Schema) -> bool {
288+
expr.downcast_ref::<Column>().is_some()
289+
&& matches!(
290+
expr.data_type(schema),
291+
Ok(DataType::Float32 | DataType::Float64)
292+
)
293+
}
294+
295+
fn is_literal(expr: &Arc<dyn PhysicalExpr>) -> bool {
296+
expr.downcast_ref::<Literal>().is_some()
297+
}
298+
282299
fn validate_types(
283300
value: &Arc<dyn PhysicalExpr>,
284301
candidates: &[Arc<dyn PhysicalExpr>],
@@ -301,7 +318,7 @@ pub fn spark_in_list(
301318
negated: bool,
302319
schema: &Schema,
303320
) -> Result<Arc<dyn PhysicalExpr>> {
304-
if !needs_spark_equality(&value.data_type(schema)?) || candidates.is_empty() {
321+
if !is_nested_with_float_leaf(&value.data_type(schema)?) || candidates.is_empty() {
305322
return in_list(value, candidates, &negated, schema);
306323
}
307324
validate_types(&value, &candidates, schema)?;
@@ -948,7 +965,8 @@ mod tests {
948965
}
949966

950967
/// A literal operand is normalized while planning, so the comparison stays `column op
951-
/// literal`. With `FloatOperands::Raw`, as for a scan's data filters, nothing is wrapped.
968+
/// literal`. With `FloatOperands::Raw`, as for a scan's data filters, only the column of that
969+
/// shape is left unwrapped.
952970
#[test]
953971
fn float_operand_shapes() -> Result<()> {
954972
use crate::float_semantics::NormalizeNaNAndZero;
@@ -974,6 +992,8 @@ mod tests {
974992
let folded = binary.right().downcast_ref::<Literal>().unwrap();
975993
assert!(matches!(folded.value(), ScalarValue::Float64(Some(v)) if v.to_bits() == 0));
976994

995+
// With `FloatOperands::Raw`, a float column compared with a literal keeps the column, on
996+
// either side, and the literal is still normalized.
977997
let expr = spark_comparison(
978998
Arc::new(Column::new("a", 0)),
979999
Operator::Lt,
@@ -983,10 +1003,39 @@ mod tests {
9831003
)?;
9841004
let binary = expr.downcast_ref::<BinaryExpr>().unwrap();
9851005
assert!(binary.left().downcast_ref::<Column>().is_some());
986-
assert!(Arc::ptr_eq(binary.right(), &negative_zero));
1006+
let folded = binary.right().downcast_ref::<Literal>().unwrap();
1007+
assert!(matches!(folded.value(), ScalarValue::Float64(Some(v)) if v.to_bits() == 0));
1008+
let expr = spark_comparison(
1009+
Arc::clone(&negative_zero),
1010+
Operator::Lt,
1011+
Arc::new(Column::new("a", 0)),
1012+
&schema,
1013+
FloatOperands::Raw,
1014+
)?;
1015+
let binary = expr.downcast_ref::<BinaryExpr>().unwrap();
1016+
assert!(binary.right().downcast_ref::<Column>().is_some());
9871017

988-
// Other operators are not comparisons and keep their operands.
1018+
// Any other shape is normalized even with `FloatOperands::Raw`, because a reader with
1019+
// row-level pushdown drops the rows it rejects.
9891020
let (a, b) = columns();
1021+
let expr = spark_comparison(
1022+
Arc::clone(&a),
1023+
Operator::Lt,
1024+
Arc::clone(&b),
1025+
&schema,
1026+
FloatOperands::Raw,
1027+
)?;
1028+
let binary = expr.downcast_ref::<BinaryExpr>().unwrap();
1029+
assert!(binary
1030+
.left()
1031+
.downcast_ref::<NormalizeNaNAndZero>()
1032+
.is_some());
1033+
assert!(binary
1034+
.right()
1035+
.downcast_ref::<NormalizeNaNAndZero>()
1036+
.is_some());
1037+
1038+
// Other operators are not comparisons and keep their operands.
9901039
let expr = spark_comparison(
9911040
Arc::clone(&a),
9921041
Operator::Plus,

‎native/spark-expr/src/float_semantics/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ mod compare;
4545
mod normalize;
4646

4747
pub use compare::{spark_comparator, spark_equality};
48+
pub(crate) use normalize::is_nested_with_float_leaf;
4849
pub use normalize::{
4950
has_float_leaf, normalize_comparison_operand, normalize_floats, normalize_nested_floats,
5051
NormalizeNaNAndZero, NormalizeNestedFloats,

‎native/spark-expr/src/float_semantics/normalize.rs‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -273,14 +273,9 @@ pub fn has_float_leaf(dt: &DataType) -> bool {
273273
}
274274
}
275275

276-
fn is_nested_with_float_leaf(dt: &DataType) -> bool {
277-
matches!(
278-
dt,
279-
DataType::List(_)
280-
| DataType::LargeList(_)
281-
| DataType::FixedSizeList(_, _)
282-
| DataType::Struct(_)
283-
) && has_float_leaf(dt)
276+
/// Whether `dt` is a list or struct with a Float32 or Float64 field at any depth.
277+
pub(crate) fn is_nested_with_float_leaf(dt: &DataType) -> bool {
278+
dt.is_nested() && has_float_leaf(dt)
284279
}
285280

286281
/// Recursively rebuilds nested arrays with `-0.0` normalized to `0.0` and NaN canonicalized

‎spark/src/test/resources/sql-tests/expressions/conditional/float_comparisons.sql‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,10 @@
2222
-- `-d` flips the sign bit, so `-d` of the NaN row is a NaN with the sign bit set on every
2323
-- platform. That is the NaN that arithmetic produces on x86-64, and Arrow's total order sorts
2424
-- it below -Infinity.
25+
--
26+
-- A scan pushes its filters into the Parquet reader, which with row-level pushdown drops rows
27+
-- itself, before Spark's Filter above the scan sees them.
28+
-- ConfigMatrix: spark.comet.parquet.rowFilterPushdown.enabled=false,true
2529

2630
statement
2731
CREATE TABLE float_cmp(id INT, d DOUBLE, f FLOAT) USING parquet
@@ -57,6 +61,20 @@ SELECT id FROM float_cmp WHERE -d > 0.0D
5761
query
5862
SELECT id FROM float_cmp WHERE -f >= f
5963

64+
-- A stored column compared with a constant, the filter shape that Parquet pruning reads. The
65+
-- scan still returns the -0.0 and NaN rows that Spark's ordering keeps.
66+
query
67+
SELECT id FROM float_cmp WHERE d >= 0.0D
68+
69+
query
70+
SELECT id FROM float_cmp WHERE f <= -0.0F
71+
72+
query
73+
SELECT id FROM float_cmp WHERE d = double('NaN')
74+
75+
query
76+
SELECT id FROM float_cmp WHERE d > double('Infinity')
77+
6078
-- Aggregate argument
6179
query
6280
SELECT sum(if(-d > 0.0D, 1, 0)), sum(if(-d = d, 1, 0)), sum(if(-f < f, 1, 0)) FROM float_cmp

0 commit comments

Comments
 (0)