Skip to content

Commit b563496

Browse files
authored
fix: normalize float operands in the native comparison builder (apache#6447)
* fix: normalize float operands in the native comparison builder Part of apache#6385. CometExecRule.normalize wrapped comparison operands in NormalizeNaNAndZero only inside ProjectExec and FilterExec, so a comparison in an aggregate argument, a FILTER clause, a join condition, a sort key or a generator compared raw Arrow values, where a NaN with the sign bit set sorts below every other value. Arrays and structs had the same gap for <=>, <, <=, > and >= in every operator. spark_comparison, which builds every native comparison, now normalizes float operands for all eight comparison operators: scalars through NormalizeNaNAndZero, arrays and structs through NormalizeNestedFloats, and literals while planning. Nested = and <> keep spark_equality. A scan's data filters are built with FloatOperands::Raw so that Parquet pruning still sees a column compared with a literal; the Filter above the scan evaluates them again. CometExecRule.normalize is removed, including its Divide and Remainder divisor normalization, which native division and remainder do not need. NormalizeNaNAndZero keeps a scalar child a scalar. TPC-DS golden files count one fewer native expression where the rewrite had added one. * fix: match Spark's float ordering in min, max, greatest and least Spark orders floats with SQLOrderingUtil.compareDoubles, in which NaN is larger than every other value and -0.0 equals 0.0, and its max, min, greatest and least keep the first of equal values. DataFusion's versions order floats by IEEE 754 total order, so a NaN with the sign bit set is the smallest value; its grouped max and min start each group at f64::MIN and f64::MAX, so a group of only -Infinity (or Infinity) returns that finite value; and its greatest and least let a later argument win a tie. Add SparkMinMax for float min and max in aggregates and windows, with ungrouped, grouped and sliding-frame accumulators, and SparkGreatestLeast for float greatest and least, including arrays and structs with a float leaf. Other types keep DataFusion's implementations. Strict floating-point mode no longer makes min and max fall back. Part of apache#6385. * test: add min/max and greatest/least benchmarks and FLOAT sliding-window coverage Add criterion benchmarks comparing SparkMinMax and SparkGreatestLeast with DataFusion's versions, which check that both return the same values before timing. Cover SlidingMinMaxAccumulator over FLOAT in the unit test and the SQL fixture, share the edge values and the first-wins oracle between the two test modules through float_semantics, and route window min/max through min_max_udaf so the float routing lives in one place. * fix: normalize computed operands in scan data filters With spark.comet.parquet.rowFilterPushdown.enabled the Parquet reader evaluates a scan's data filters on each row, and a row it drops never reaches the Filter above the scan. Data filters left every float operand raw, so a computed operand such as -d, whose NaN has the sign bit set, dropped the NaN row that Spark keeps. Leave only a float column compared with a literal raw, the shape Parquet pruning reads, and normalize every other comparison. Run float_comparisons.sql with row-level pushdown off and on, add filters on stored edge values, add a benchmark of Spark's float comparisons against DataFusion's, and share one helper for nested types with a float leaf. * test: avoid scan-order assumptions in float min/max checks * test: drop the sweep suite's known gaps that normalized comparisons close With comparisons normalized in the native comparison builder, the cases for comparisons outside Project and Filter, Parquet scan filters against a sign-bit NaN literal, and nested array and struct comparisons (apache#6157) match Spark, so the suite now requires them to. * test: retire fixed float min/max and greatest/least gaps * fix: harden float extrema planner routing * test: share float_semantics' edge values with the array function tests array_remove and sort_array, from apache#6518, kept their own copy of the edge values that min/max and greatest/least share through float_semantics. The two arrays were identical. * test: check signed-zero ties in aggregate min and max again eb5c7af stopped depending on scan order by dropping the aggregate tie queries from the SQL fixture, and by testing each zero on its own in CometAggregateSuite, where any tie rule returns the same value. Write both tables as a single file instead, so that Spark and Comet read the rows in the same order in one task, and check ties through the ungrouped and grouped accumulators again. The ungrouped checks run once per group, because over groups 2 and 3 together the first and the last zero have the same sign. * refactor: drop the unreachable missing-result branch from greatest and least invoke_with_args checks for at least two arguments before it starts, so the result can no longer be missing after the loop. Start from the first argument instead of an Option, as return_type already does. * test: cover IS DISTINCT FROM, two-level nesting and a NaN row group float_comparisons.sql now runs IS DISTINCT FROM, which Comet plans as one native operator, on scalars, constants, arrays and structs, and compares ARRAY<STRUCT<v: DOUBLE>> and STRUCT<a: ARRAY<DOUBLE>> columns under every operator. The float pruning test puts a NaN in a row group whose other values are all below 500, and checks that `d > 500.0D` still returns it. * test: hash quotients of a NaN divisor with the sign bit set The removed Project and Filter rewrite normalized the divisor of Divide and Remainder, which also canonicalized a NaN divisor. Without it the quotient keeps the sign bit, and hash and xxhash64 must still hash it as Spark does, in both ANSI modes. * test: order quotients of a NaN divisor in greatest, least, max and min Without the removed divisor rewrite, least and greatest of 1.0D / (-d) see a NaN with the sign bit set, and so do max and min of a quotient computed in a projection. The merged apache#6457 orders floats the Spark way in all four, so cover them in nan_divisor.sql with ANSI off and on. * fix: keep normalizing float divisors, and normalize data filters that drop rows With spark.comet.parquet.rowFilterPushdown.enabled, the Parquet reader drops the rows a scan's data filter rejects. FloatOperands::Raw left a float column raw against a normalized literal, so a stored NaN with other bits, such as one with the sign bit set, failed `d = -double('NaN')` and every ordering comparison that Spark matches. Data filters now leave the column raw only when row-level pushdown is off, where the reader uses them for pruning alone and the Filter above the scan applies Spark's semantics to each row. With it on, they normalize both sides and float comparisons give up statistics pruning. Removing normalizePlan also removed the normalization of Divide and Remainder divisors. Comparisons no longer need it, but it canonicalized a NaN quotient, and percentile_approx still orders a NaN with the sign bit set below every other value (apache#6519). normalizeDivisors keeps that part, in Projects and Filters as before, until apache#6519 is fixed. The TPC-DS plans count NormalizeNaNAndZero again, so their expression counts go back to main's. * fix: match either signed zero when a float data filter probes a bloom filter A data filter that only prunes compared the raw float column with a normalized literal, and a bloom filter probe hashes the literal's bits, so `d = -0.0D` probed for `0.0` and pruned a row group holding `-0.0` but no `0.0`. `=` against either zero now probes for both. Other comparisons keep the literal as it is, since DataFusion's comparisons, and with them statistics pruning, already treat the two zeros as equal. A NaN literal normalizes the column too, as the IN path does, because no literal's bits match every NaN a file can hold. Also document that float comparisons in data filters do not prune with spark.comet.parquet.rowFilterPushdown.enabled=true (apache#6702), in the floating-point guide, the config doc and the scan tuning page. * docs: add a contributor guide page on Spark's floating-point semantics Adds a Floating-Point Semantics page next to Timezone Handling. It covers which Spark rule a function follows for -0.0 and NaN, the releases in which a rule changed, how Arrow and DataFusion differ, where Comet applies each rule (the float_semantics helpers, key normalization in the planner, the comparison builder and the Scala version policies), and how to test for it. The expression guide's testing tips link to it. Part of apache#6385. * docs: correct the collect_set and OpenHashSet rules, and note where Spark's reader skips the other zero Before Spark 4.2, collect_set keys its buffer with Scala's `==`, which merges the two zeros and keeps every NaN apart, so the page now lists that rule separately. OpenHashSet and OpenHashMap follow Double.equals only from Spark 3.5.2 and 4.0.0 (SPARK-45599), which also qualifies mode. Spark's own Parquet reader keeps the two zeros apart in its dictionary and bloom filters, so it can skip a row group that Comet reads. The compatibility guide says so, and the contributor guide's data filter paragraph now describes the bloom filter expansion and the NaN literal.
1 parent e038cf0 commit b563496

22 files changed

Lines changed: 1691 additions & 141 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,6 +408,7 @@ For full documentation on the test file format, including directives like `Confi
408408

409409
- **Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively.
410410
- **Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.
411+
- **For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.** [Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it.
411412
- **Keep one file per expression** to make failures easy to locate.
412413
- **Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md).
413414

‎docs/source/contributor-guide/floating_point.md‎

Lines changed: 207 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/source/contributor-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ Development Guide <development>
4747
Comet Plugin Overview <plugin_overview>
4848
Arrow FFI <ffi>
4949
Timezone Handling <timezones>
50+
Floating-Point Semantics <floating_point>
5051
JVM Shuffle <jvm_shuffle>
5152
Native Shuffle <native_shuffle>
5253
Memory Management <memory_management>

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,20 +23,43 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
2323
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
2424
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
2525
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
26-
For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
27-
execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
28-
also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
29-
literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
30-
forms when a list contains zero.
26+
For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
27+
normalizes both operands before native execution, including noncanonical NaN literals. This
28+
applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
29+
clauses, join conditions, sort keys, and generator arguments.
30+
31+
A native Parquet scan prunes row groups and pages with its data filters. So that this pruning
32+
still applies, a `FLOAT` or `DOUBLE` column compared with a constant other than NaN in a data
33+
filter is compared without normalizing the column, and the filter above the scan evaluates the
34+
comparison again with Spark's semantics. Every other comparison in a data filter is normalized.
35+
Bloom filters store `-0.0` and `0.0` separately, so `=` against either zero checks them for both.
36+
With `spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also drops the rows its data
37+
filters reject, so every comparison in a data filter is normalized, and a `FLOAT` or `DOUBLE`
38+
comparison in a data filter does not prune row groups or pages
39+
([#6702](https://github.com/apache/datafusion-comet/issues/6702)).
40+
41+
Spark's own Parquet reader keeps `-0.0` and `0.0` apart when it prunes row groups with dictionaries
42+
and bloom filters, so it can skip a row group holding rows that its filter matches. For example,
43+
`WHERE d = -0.0D` can skip a row group that holds `0.0` but not `-0.0`, and so can `<=>` or `IN`
44+
against `-0.0D`. `WHERE d >= 0.0D` can skip a dictionary-encoded row group whose largest value is
45+
`-0.0`, and `WHERE d <= -0.0D` one whose smallest value is `0.0`. Comet reads those row groups, so
46+
it can return rows that Spark does not. Comet's result is the one that Spark's comparison semantics
47+
call for, and Spark returns the same rows with `spark.sql.parquet.filterPushdown=false`.
48+
49+
Top-level `IN`, `InSet`, and `NOT IN` membership also normalize dynamic candidates and lists
50+
containing NaN. When every candidate is a non-NaN literal, Comet keeps DataFusion's static filter
51+
and pruning path, enumerating both signed-zero forms when a list contains zero.
3152

3253
This scalar membership handling does not yet recurse into floating-point leaves nested in arrays
3354
or structs; see [#6019](https://github.com/apache/datafusion-comet/issues/6019).
3455

35-
## Nested equality and membership
56+
## Nested comparisons and membership
3657

3758
For arrays and structs containing `FLOAT` or `DOUBLE`, native `=`, `<>`, `IN`, and `NOT IN`
3859
compare signed zeros as equal and all NaN representations as equal, matching Spark. This also
39-
covers single-candidate membership that Spark rewrites into equality.
60+
covers single-candidate membership that Spark rewrites into equality. `<=>`, `<`, `<=`, `>`, and
61+
`>=` normalize the floating-point values inside both operands first, so they follow Spark's order
62+
too, in which NaN sorts above every other value.
4063

4164
Equality and dynamic membership compare nested elements directly and stop at the first mismatch.
4265
Constant membership sets use normalized comparison values for static lookup. These operations

‎docs/source/user-guide/latest/tuning/scans.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ decode step and lazily materializes projected columns for surviving rows. This c
2828
CPU and memory when the filter is highly selective on a small subset of columns. It is disabled by default
2929
because it can hurt when the filter is not selective or when most columns must be read anyway. Row-group,
3030
page-index, and bloom-filter pruning happen regardless of this flag whenever Spark's
31-
`spark.sql.parquet.filterPushdown` is on.
31+
`spark.sql.parquet.filterPushdown` is on, except for filters that compare `FLOAT` or `DOUBLE` values,
32+
which do not prune while this flag is on (see [Floating-point Number Comparison](../compatibility/floating-point.md)).
3233

3334
### Parquet Native Scans
3435

‎native/core/src/execution/planner.rs‎

Lines changed: 176 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -144,10 +144,10 @@ use datafusion_comet_proto::{
144144
use datafusion_comet_spark_expr::{
145145
create_case_when, create_if_expr, jvm_udf::JvmScalarUdfExpr, spark_in_list, ApproxPercentile,
146146
ArrayInsert, Avg, AvgDecimal, Cast, CheckOverflow, Correlation, Covariance, CreateNamedStruct,
147-
DecimalRescaleCheckOverflow, GetArrayStructFields, GetStructField, HllPlusPlus, HllSketchAgg,
148-
HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero, NormalizeNestedFloats,
149-
Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal, ToJson, UnboundColumn,
150-
Variance, WideDecimalBinaryExpr, WideDecimalOp,
147+
DecimalRescaleCheckOverflow, FloatOperands, GetArrayStructFields, GetStructField, HllPlusPlus,
148+
HllSketchAgg, HllUnionAgg, IfExpr, ListExtract, MaxMinBy, Mode, NormalizeNaNAndZero,
149+
NormalizeNestedFloats, Regr, RegrType, SparkCastOptions, SparkMinMax, Stddev, SumDecimal,
150+
ToJson, UnboundColumn, Variance, WideDecimalBinaryExpr, WideDecimalOp,
151151
};
152152
use itertools::Itertools;
153153
use jni::objects::{Global, JObject};
@@ -325,6 +325,7 @@ pub struct BinaryExprOptions {
325325
pub const TEST_EXEC_CONTEXT_ID: i64 = -1;
326326

327327
/// The query planner for converting Spark query plans to DataFusion query plans.
328+
#[derive(Clone)]
328329
pub struct PhysicalPlanner {
329330
// The execution context id of this planner.
330331
exec_context_id: i64,
@@ -346,6 +347,9 @@ pub struct PhysicalPlanner {
346347
/// Task-owned destination for remote shuffle blocks, registered on the driving Spark task
347348
/// thread before native planning. Only explicit RSS destinations may use it.
348349
shuffle_partition_pusher: Option<Arc<dyn ShufflePartitionPusher>>,
350+
/// How comparisons treat floating-point operands. `Raw` only while planning a scan's data
351+
/// filters; see [`Self::create_data_filter`].
352+
float_operands: FloatOperands,
349353
}
350354

351355
impl Default for PhysicalPlanner {
@@ -365,6 +369,7 @@ impl PhysicalPlanner {
365369
task_context: None,
366370
class_loader: None,
367371
shuffle_partition_pusher: None,
372+
float_operands: FloatOperands::Normalize,
368373
}
369374
}
370375

@@ -477,6 +482,11 @@ impl PhysicalPlanner {
477482
self
478483
}
479484

485+
/// How comparisons treat floating-point operands.
486+
pub fn float_operands(&self) -> FloatOperands {
487+
self.float_operands
488+
}
489+
480490
/// Return session context of this planner.
481491
pub fn session_ctx(&self) -> &Arc<SessionContext> {
482492
&self.session_ctx
@@ -1129,6 +1139,45 @@ impl PhysicalPlanner {
11291139
}
11301140
}
11311141

1142+
/// Create a data filter that a scan pushes into the Parquet reader, with float operands
1143+
/// treated as [`Self::data_filter_float_operands`] decides.
1144+
fn create_data_filter(
1145+
&self,
1146+
spark_expr: &Expr,
1147+
input_schema: SchemaRef,
1148+
float_operands: FloatOperands,
1149+
) -> Result<Arc<dyn PhysicalExpr>, ExecutionError> {
1150+
let planner = Self {
1151+
float_operands,
1152+
..self.clone()
1153+
};
1154+
planner.create_expr(spark_expr, input_schema)
1155+
}
1156+
1157+
/// How a scan's data filters treat float operands. The Parquet reader prunes row groups and
1158+
/// pages with them, and with row-level pushdown (`pushdown_filters`) it also drops the rows
1159+
/// they reject, which Spark's Filter above the scan then never sees.
1160+
///
1161+
/// Without row-level pushdown the filters only prune, and pruning only recognizes a column
1162+
/// compared with a literal, so that shape keeps the raw column ([`FloatOperands::Raw`]). With
1163+
/// it, every operand is normalized and float comparisons give up pruning: a raw column would
1164+
/// drop a stored NaN that Spark matches, such as one with the sign bit set, which Arrow orders
1165+
/// below every other value.
1166+
fn data_filter_float_operands(&self) -> FloatOperands {
1167+
if self
1168+
.session_ctx
1169+
.copied_config()
1170+
.options()
1171+
.execution
1172+
.parquet
1173+
.pushdown_filters
1174+
{
1175+
FloatOperands::Normalize
1176+
} else {
1177+
FloatOperands::Raw
1178+
}
1179+
}
1180+
11321181
/// Normalize floating-point comparison keys without changing output values: a `FLOAT` or
11331182
/// `DOUBLE` key, and an array or struct key with a float at any depth, whose order Arrow
11341183
/// otherwise takes from the raw bits. Sort, Window, and WindowGroupLimit must use identical
@@ -1910,10 +1959,17 @@ impl PhysicalPlanner {
19101959
.cloned()
19111960
.collect::<Vec<FieldRef>>(),
19121961
));
1962+
let float_operands = self.data_filter_float_operands();
19131963
common
19141964
.data_filters
19151965
.iter()
1916-
.map(|expr| self.create_expr(expr, Arc::clone(&filter_schema)))
1966+
.map(|expr| {
1967+
self.create_data_filter(
1968+
expr,
1969+
Arc::clone(&filter_schema),
1970+
float_operands,
1971+
)
1972+
})
19171973
.collect()
19181974
};
19191975

@@ -5826,6 +5882,121 @@ mod tests {
58265882
assert_eq!(0, filter_exec.additional_native_plans.len());
58275883
}
58285884

5885+
/// Comparisons normalize float operands, except in the data filters that a scan pushes into
5886+
/// the Parquet reader, where pruning has to see the column itself.
5887+
#[test]
5888+
fn scan_data_filters_compare_float_columns_directly() {
5889+
use datafusion::physical_expr::expressions::BinaryExpr;
5890+
use datafusion_comet_spark_expr::NormalizeNaNAndZero;
5891+
let double = spark_expression::DataType {
5892+
type_id: 6,
5893+
type_info: None,
5894+
};
5895+
let operand = |expr_struct| Expr {
5896+
expr_struct: Some(expr_struct),
5897+
query_context: None,
5898+
expr_id: None,
5899+
};
5900+
let expr = operand(Gt(Box::new(spark_expression::BinaryExpr {
5901+
left: Some(Box::new(operand(Bound(spark_expression::BoundReference {
5902+
index: 0,
5903+
datatype: Some(double.clone()),
5904+
})))),
5905+
right: Some(Box::new(operand(Literal(spark_expression::Literal {
5906+
value: Some(literal::Value::DoubleVal(500.0)),
5907+
datatype: Some(double),
5908+
is_null: false,
5909+
})))),
5910+
})));
5911+
let schema = Arc::new(Schema::new(vec![Field::new("d", DataType::Float64, true)]));
5912+
let left_operand = |expr: Arc<dyn PhysicalExpr>| {
5913+
let comparison = expr.downcast_ref::<BinaryExpr>().expect("a comparison");
5914+
Arc::clone(comparison.left())
5915+
};
5916+
let planner = PhysicalPlanner::default();
5917+
let comparison = planner.create_expr(&expr, Arc::clone(&schema)).unwrap();
5918+
assert!(left_operand(comparison)
5919+
.downcast_ref::<NormalizeNaNAndZero>()
5920+
.is_some());
5921+
let data_filter = planner
5922+
.create_data_filter(&expr, schema, planner.data_filter_float_operands())
5923+
.unwrap();
5924+
assert!(left_operand(data_filter).downcast_ref::<Column>().is_some());
5925+
}
5926+
5927+
/// With row-level pushdown the Parquet reader drops the rows a data filter rejects, so the
5928+
/// filter normalizes the column as well as the literal. A raw column would drop a stored NaN
5929+
/// whose bits differ from the normalized literal, and a stored NaN with the sign bit set
5930+
/// under any ordering comparison, both of which Spark matches.
5931+
#[test]
5932+
fn scan_data_filters_normalize_float_columns_with_row_level_pushdown() {
5933+
use arrow::array::{AsArray, BooleanArray};
5934+
use datafusion::physical_expr::expressions::BinaryExpr;
5935+
use datafusion_comet_spark_expr::{FloatOperands, NormalizeNaNAndZero};
5936+
let double = spark_expression::DataType {
5937+
type_id: 6,
5938+
type_info: None,
5939+
};
5940+
let operand = |expr_struct| Expr {
5941+
expr_struct: Some(expr_struct),
5942+
query_context: None,
5943+
expr_id: None,
5944+
};
5945+
let column_and = |value: f64| {
5946+
Box::new(spark_expression::BinaryExpr {
5947+
left: Some(Box::new(operand(Bound(spark_expression::BoundReference {
5948+
index: 0,
5949+
datatype: Some(double.clone()),
5950+
})))),
5951+
right: Some(Box::new(operand(Literal(spark_expression::Literal {
5952+
value: Some(literal::Value::DoubleVal(value)),
5953+
datatype: Some(double.clone()),
5954+
is_null: false,
5955+
})))),
5956+
})
5957+
};
5958+
// Spark folds `-double('NaN')` into a literal with the sign bit set.
5959+
let negative_nan = f64::from_bits(0xfff8_0000_0000_0000);
5960+
let schema = Arc::new(Schema::new(vec![Field::new("d", DataType::Float64, true)]));
5961+
let batch = RecordBatch::try_new(
5962+
Arc::clone(&schema),
5963+
vec![Arc::new(Float64Array::from(vec![
5964+
negative_nan,
5965+
f64::NAN,
5966+
1.0,
5967+
]))],
5968+
)
5969+
.unwrap();
5970+
let config =
5971+
SessionConfig::new().set_bool("datafusion.execution.parquet.pushdown_filters", true);
5972+
let planner = PhysicalPlanner::new(Arc::new(SessionContext::new_with_config(config)), 0);
5973+
assert_eq!(
5974+
planner.data_filter_float_operands(),
5975+
FloatOperands::Normalize
5976+
);
5977+
for (expr, expected) in [
5978+
(operand(Eq(column_and(negative_nan))), [true, true, false]),
5979+
(operand(Gt(column_and(0.0))), [true, true, true]),
5980+
] {
5981+
let data_filter = planner
5982+
.create_data_filter(
5983+
&expr,
5984+
Arc::clone(&schema),
5985+
planner.data_filter_float_operands(),
5986+
)
5987+
.unwrap();
5988+
let comparison = data_filter
5989+
.downcast_ref::<BinaryExpr>()
5990+
.expect("a comparison");
5991+
assert!(comparison
5992+
.left()
5993+
.downcast_ref::<NormalizeNaNAndZero>()
5994+
.is_some());
5995+
let matched = data_filter.evaluate(&batch).unwrap().into_array(3).unwrap();
5996+
assert_eq!(matched.as_boolean(), &BooleanArray::from(expected.to_vec()));
5997+
}
5998+
}
5999+
58296000
#[test]
58306001
fn spark_plan_metrics_hash_join() {
58316002
let op_scan = create_scan();

‎native/core/src/execution/planner/macros.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ macro_rules! binary_expr_builder {
9898
$operator,
9999
right,
100100
input_schema.as_ref(),
101+
planner.float_operands(),
101102
)
102103
.map_err(Into::into)
103104
}

0 commit comments

Comments
 (0)