You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b563496
Browse filesBrowse the repository at this point in the historyBrowse files
fix: normalize float operands in the native comparison builder (apache#6447)
* fix: normalize float operands in the native comparison builder
Part of apache#6385. CometExecRule.normalize wrapped comparison operands in
NormalizeNaNAndZero only inside ProjectExec and FilterExec, so a
comparison in an aggregate argument, a FILTER clause, a join condition,
a sort key or a generator compared raw Arrow values, where a NaN with the
sign bit set sorts below every other value. Arrays and structs had the
same gap for <=>, <, <=, > and >= in every operator.
spark_comparison, which builds every native comparison, now normalizes
float operands for all eight comparison operators: scalars through
NormalizeNaNAndZero, arrays and structs through NormalizeNestedFloats,
and literals while planning. Nested = and <> keep spark_equality. A
scan's data filters are built with FloatOperands::Raw so that Parquet
pruning still sees a column compared with a literal; the Filter above
the scan evaluates them again.
CometExecRule.normalize is removed, including its Divide and Remainder
divisor normalization, which native division and remainder do not need.
NormalizeNaNAndZero keeps a scalar child a scalar. TPC-DS golden files
count one fewer native expression where the rewrite had added one.
* fix: match Spark's float ordering in min, max, greatest and least
Spark orders floats with SQLOrderingUtil.compareDoubles, in which NaN is
larger than every other value and -0.0 equals 0.0, and its max, min,
greatest and least keep the first of equal values. DataFusion's versions
order floats by IEEE 754 total order, so a NaN with the sign bit set is the
smallest value; its grouped max and min start each group at f64::MIN and
f64::MAX, so a group of only -Infinity (or Infinity) returns that finite
value; and its greatest and least let a later argument win a tie.
Add SparkMinMax for float min and max in aggregates and windows, with
ungrouped, grouped and sliding-frame accumulators, and SparkGreatestLeast
for float greatest and least, including arrays and structs with a float
leaf. Other types keep DataFusion's implementations. Strict floating-point
mode no longer makes min and max fall back.
Part of apache#6385.
* test: add min/max and greatest/least benchmarks and FLOAT sliding-window coverage
Add criterion benchmarks comparing SparkMinMax and SparkGreatestLeast with
DataFusion's versions, which check that both return the same values before
timing. Cover SlidingMinMaxAccumulator over FLOAT in the unit test and the
SQL fixture, share the edge values and the first-wins oracle between the
two test modules through float_semantics, and route window min/max through
min_max_udaf so the float routing lives in one place.
* fix: normalize computed operands in scan data filters
With spark.comet.parquet.rowFilterPushdown.enabled the Parquet reader
evaluates a scan's data filters on each row, and a row it drops never
reaches the Filter above the scan. Data filters left every float operand
raw, so a computed operand such as -d, whose NaN has the sign bit set,
dropped the NaN row that Spark keeps. Leave only a float column compared
with a literal raw, the shape Parquet pruning reads, and normalize every
other comparison.
Run float_comparisons.sql with row-level pushdown off and on, add filters
on stored edge values, add a benchmark of Spark's float comparisons
against DataFusion's, and share one helper for nested types with a float
leaf.
* test: avoid scan-order assumptions in float min/max checks
* test: drop the sweep suite's known gaps that normalized comparisons close
With comparisons normalized in the native comparison builder, the cases
for comparisons outside Project and Filter, Parquet scan filters against
a sign-bit NaN literal, and nested array and struct comparisons (apache#6157)
match Spark, so the suite now requires them to.
* test: retire fixed float min/max and greatest/least gaps
* fix: harden float extrema planner routing
* test: share float_semantics' edge values with the array function tests
array_remove and sort_array, from apache#6518, kept their own copy of the edge
values that min/max and greatest/least share through float_semantics.
The two arrays were identical.
* test: check signed-zero ties in aggregate min and max again
eb5c7af stopped depending on scan order by dropping the aggregate tie
queries from the SQL fixture, and by testing each zero on its own in
CometAggregateSuite, where any tie rule returns the same value. Write
both tables as a single file instead, so that Spark and Comet read the
rows in the same order in one task, and check ties through the ungrouped
and grouped accumulators again. The ungrouped checks run once per group,
because over groups 2 and 3 together the first and the last zero have
the same sign.
* refactor: drop the unreachable missing-result branch from greatest and least
invoke_with_args checks for at least two arguments before it starts, so
the result can no longer be missing after the loop. Start from the first
argument instead of an Option, as return_type already does.
* test: cover IS DISTINCT FROM, two-level nesting and a NaN row group
float_comparisons.sql now runs IS DISTINCT FROM, which Comet plans as one
native operator, on scalars, constants, arrays and structs, and compares
ARRAY<STRUCT<v: DOUBLE>> and STRUCT<a: ARRAY<DOUBLE>> columns under every
operator. The float pruning test puts a NaN in a row group whose other
values are all below 500, and checks that `d > 500.0D` still returns it.
* test: hash quotients of a NaN divisor with the sign bit set
The removed Project and Filter rewrite normalized the divisor of Divide
and Remainder, which also canonicalized a NaN divisor. Without it the
quotient keeps the sign bit, and hash and xxhash64 must still hash it as
Spark does, in both ANSI modes.
* test: order quotients of a NaN divisor in greatest, least, max and min
Without the removed divisor rewrite, least and greatest of 1.0D / (-d)
see a NaN with the sign bit set, and so do max and min of a quotient
computed in a projection. The merged apache#6457 orders floats the Spark way in
all four, so cover them in nan_divisor.sql with ANSI off and on.
* fix: keep normalizing float divisors, and normalize data filters that drop rows
With spark.comet.parquet.rowFilterPushdown.enabled, the Parquet reader drops
the rows a scan's data filter rejects. FloatOperands::Raw left a float column
raw against a normalized literal, so a stored NaN with other bits, such as one
with the sign bit set, failed `d = -double('NaN')` and every ordering
comparison that Spark matches. Data filters now leave the column raw only
when row-level pushdown is off, where the reader uses them for pruning alone
and the Filter above the scan applies Spark's semantics to each row. With it
on, they normalize both sides and float comparisons give up statistics
pruning.
Removing normalizePlan also removed the normalization of Divide and
Remainder divisors. Comparisons no longer need it, but it canonicalized a NaN
quotient, and percentile_approx still orders a NaN with the sign bit set
below every other value (apache#6519). normalizeDivisors keeps that part, in
Projects and Filters as before, until apache#6519 is fixed. The TPC-DS plans count
NormalizeNaNAndZero again, so their expression counts go back to main's.
* fix: match either signed zero when a float data filter probes a bloom filter
A data filter that only prunes compared the raw float column with a
normalized literal, and a bloom filter probe hashes the literal's bits, so
`d = -0.0D` probed for `0.0` and pruned a row group holding `-0.0` but no
`0.0`. `=` against either zero now probes for both. Other comparisons keep
the literal as it is, since DataFusion's comparisons, and with them
statistics pruning, already treat the two zeros as equal. A NaN literal
normalizes the column too, as the IN path does, because no literal's bits
match every NaN a file can hold.
Also document that float comparisons in data filters do not prune with
spark.comet.parquet.rowFilterPushdown.enabled=true (apache#6702), in the
floating-point guide, the config doc and the scan tuning page.
* docs: add a contributor guide page on Spark's floating-point semantics
Adds a Floating-Point Semantics page next to Timezone Handling. It covers
which Spark rule a function follows for -0.0 and NaN, the releases in which
a rule changed, how Arrow and DataFusion differ, where Comet applies each
rule (the float_semantics helpers, key normalization in the planner, the
comparison builder and the Scala version policies), and how to test for
it. The expression guide's testing tips link to it.
Part of apache#6385.
* docs: correct the collect_set and OpenHashSet rules, and note where Spark's reader skips the other zero
Before Spark 4.2, collect_set keys its buffer with Scala's `==`, which merges the two zeros and
keeps every NaN apart, so the page now lists that rule separately. OpenHashSet and OpenHashMap
follow Double.equals only from Spark 3.5.2 and 4.0.0 (SPARK-45599), which also qualifies mode.
Spark's own Parquet reader keeps the two zeros apart in its dictionary and bloom filters, so it
can skip a row group that Comet reads. The compatibility guide says so, and the contributor
guide's data filter paragraph now describes the bloom filter expansion and the NaN literal.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/adding_a_new_expression.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -408,6 +408,7 @@ For full documentation on the test file format, including directives like `Confi
408
408
409
409
-**Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively.
410
410
-**Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.
411
+
-**For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.**[Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it.
411
412
-**Keep one file per expression** to make failures easy to locate.
412
413
-**Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md).
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/floating-point.md
+30-7Lines changed: 30 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,20 +23,43 @@ Spark normalizes NaN and zero for floating point numbers for several cases. See
23
23
However, one exception is comparison. Spark does not normalize NaN and zero when comparing values
24
24
because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison
25
25
functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)).
26
-
For top-level `FLOAT` and `DOUBLE` comparisons, Comet normalizes both operands before native
27
-
execution, including noncanonical NaN literals. Top-level `IN`, `InSet`, and `NOT IN` membership
28
-
also normalize dynamic candidates and lists containing NaN. When every candidate is a non-NaN
29
-
literal, Comet keeps DataFusion's static filter and pruning path, enumerating both signed-zero
30
-
forms when a list contains zero.
26
+
For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet
27
+
normalizes both operands before native execution, including noncanonical NaN literals. This
28
+
applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER`
29
+
clauses, join conditions, sort keys, and generator arguments.
30
+
31
+
A native Parquet scan prunes row groups and pages with its data filters. So that this pruning
32
+
still applies, a `FLOAT` or `DOUBLE` column compared with a constant other than NaN in a data
33
+
filter is compared without normalizing the column, and the filter above the scan evaluates the
34
+
comparison again with Spark's semantics. Every other comparison in a data filter is normalized.
35
+
Bloom filters store `-0.0` and `0.0` separately, so `=` against either zero checks them for both.
36
+
With `spark.comet.parquet.rowFilterPushdown.enabled=true` the scan also drops the rows its data
37
+
filters reject, so every comparison in a data filter is normalized, and a `FLOAT` or `DOUBLE`
38
+
comparison in a data filter does not prune row groups or pages
0 commit comments