Skip to content

Commit ef23062

Browse files
committed
docs: add a contributor guide page on Spark's floating-point semantics
Adds a Floating-Point Semantics page next to Timezone Handling. It covers which Spark rule a function follows for -0.0 and NaN, the releases in which a rule changed, how Arrow and DataFusion differ, where Comet applies each rule (the float_semantics helpers, key normalization in the planner, the comparison builder and the Scala version policies), and how to test for it. The expression guide's testing tips link to it. Part of apache#6385.
1 parent 2f003c2 commit ef23062

3 files changed

Lines changed: 194 additions & 0 deletions

File tree

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,6 +408,7 @@ For full documentation on the test file format, including directives like `Confi
408408

409409
- **Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively.
410410
- **Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters.
411+
- **For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.** [Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it.
411412
- **Keep one file per expression** to make failures easy to locate.
412413
- **Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md).
413414

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Floating-Point Semantics
21+
22+
This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE`
23+
values. It is aimed at contributors working on an expression or operator that compares, orders,
24+
hashes, deduplicates or groups floating-point values, including values nested in arrays, structs
25+
and maps. For user-facing differences from Spark, see
26+
[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work
27+
to apply these rules systematically is tracked in
28+
[#6385](https://github.com/apache/datafusion-comet/issues/6385).
29+
30+
The short version: Spark has no single rule for floating-point equality. Each function inherits one
31+
from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order,
32+
which matches none of them. A native implementation must follow the rule of the Spark function it
33+
replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must
34+
include a NaN with the sign bit set.
35+
36+
## Spark's rules
37+
38+
| Rule | `-0.0` and `0.0` | NaN | Used by |
39+
| ----------------------------------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
40+
| SQL ordering (`SQLOrderingUtil.compareDoubles`, `genEqual`) | Equal | All NaNs are equal, and NaN sorts above all values | Comparisons, `IN`, sorting and ranking, `min`/`max`, `greatest`/`least`, `sort_array`, `array_min`/`array_max`, `array_contains`, `array_position`, `array_remove`, and comparisons of arrays and structs |
41+
| `NormalizeNaNAndZero`, inserted by Spark's optimizer | `-0.0` becomes `0.0` | Every NaN becomes the canonical NaN | Grouping keys, join keys and window partition keys |
42+
| `Murmur3Hash` and `XxHash64` | Same hash | Hashed through `doubleToLongBits`, which canonicalizes NaN | `hash`, `xxhash64` and hash partitioning |
43+
| `java.lang.Double.equals`, for boxed values | Distinct | All NaNs are equal | Hash sets and maps of boxed values |
44+
| `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code |
45+
46+
Some functions changed rules in a Spark release, so a native path has to follow the Spark version
47+
it runs against:
48+
49+
- `collect_set` keys its buffer by `Double.equals` before Spark 4.2. Spark 4.2 normalizes NaN and
50+
`-0.0` first (SPARK-57298).
51+
- `mode` keys its frequency map by `Double.equals` before Spark 4.2. Spark 4.2 folds `-0.0` into
52+
`0.0` first (SPARK-57329).
53+
- `array_distinct` and `array_union` keep signed zeros apart in a flat array before Spark 4.2.0.
54+
Spark 4.2.0 normalizes their arguments in the plan (SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1
55+
they normalize while they evaluate instead (SPARK-59602).
56+
- Map construction (`ArrayBasedMapBuilder`) finds duplicate keys by `Double.equals` in Spark 3.4
57+
and 3.5. From Spark 4.0 it normalizes each key first, unless
58+
`spark.sql.legacy.disableMapKeyNormalization` is set
59+
([#6549](https://github.com/apache/datafusion-comet/issues/6549)).
60+
61+
## How Arrow and DataFusion differ
62+
63+
Arrow compares floats by IEEE 754 total order. `-0.0` sorts below `0.0`, NaNs compare by their bit
64+
patterns, and a NaN with the sign bit set sorts below `-Infinity`. Arrow's sort, row format and
65+
hash kernels all work from the same bits. DataFusion 55 folds `-0.0` into `0.0` in some kernels but
66+
does not canonicalize NaN, and that has changed between DataFusion releases. Don't rely on it:
67+
normalize the values, or compare them with the helpers below.
68+
69+
NaNs with the sign bit set are the normal case on x86-64, not a corner case. Every NaN that
70+
arithmetic produces at run time, such as `sqrt(-1)` or `Infinity - Infinity`, is
71+
`0xfff8000000000000` in both Rust and the JVM on x86-64, while aarch64 produces
72+
`0x7ff8000000000000`. Spark hides the difference through `doubleToLongBits`. A native path that
73+
compares raw values puts those NaNs below every other value, so a query that passes on an Apple
74+
Silicon laptop can fail on a Linux x86 cluster.
75+
76+
## Where Comet applies the rules
77+
78+
### The `float_semantics` module
79+
80+
`native/spark-expr/src/float_semantics/` holds the per-value rules and the kernels built on them.
81+
Its module documentation is the reference. In short:
82+
83+
| Helper | Rule | Use it to |
84+
| ---------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------- |
85+
| `compare_floats`, `float_lt`, `float_gt` | SQL ordering | Compare values in a kernel that returns the original bits, as `array_min` and `greatest` do |
86+
| `spark_comparator`, `spark_equality` | SQL ordering, at any depth | Compare arrays and structs in place |
87+
| `normalize_float`, `normalize_floats`, `normalize_nested_floats` | `NormalizeNaNAndZero` | Normalize values before an Arrow kernel sorts, row-encodes, hashes or compares them |
88+
| `NormalizeNaNAndZero`, `NormalizeNestedFloats` | `NormalizeNaNAndZero` | Wrap a key or operand expression. `wrap_if_needed` skips other types and keys that Spark already normalized |
89+
| `canonicalize_nan` | `java.lang.Double.equals` | Key a hash set or map the way boxed Java values do |
90+
| `compare_floats_java` | `java.lang.Double.compare` | Sort the way `java.util.Arrays.sort` sorts a primitive array |
91+
| `hash_input` | `Murmur3Hash` and `XxHash64` | Get the value to hash in place of a float |
92+
93+
Once `-0.0` is folded and every NaN is canonical, Arrow's total order agrees with Spark's SQL
94+
ordering. That is why normalizing the inputs of an Arrow kernel works, as long as only the keys are
95+
normalized and the output keeps the original values.
96+
97+
### Keys, sorting and comparisons
98+
99+
- Spark's optimizer wraps grouping, join and window partition keys in `NormalizeNaNAndZero`
100+
(`NormalizeFloatingNumbers`). Comet serializes it like any other expression.
101+
- `create_normalized_key_expr` in `native/core/src/execution/planner.rs` normalizes sort keys,
102+
window order and partition keys, and the partition keys of `WindowGroupLimit`, so that Arrow's
103+
sort orders them the Spark way.
104+
- `spark_comparison` in `native/spark-expr/src/array_funcs/nested_comparison.rs` builds every native
105+
`=`, `<>`, `<=>`, `<`, `<=`, `>`, `>=` and `IS DISTINCT FROM`, whichever operator evaluates it. It
106+
normalizes float operands, and folds a literal while the plan is built so that it stays a
107+
literal. Nested `=` and `<>` compare in place with `spark_equality`.
108+
- A scan's pushed-down data filters leave a float column compared with a literal unwrapped
109+
(`FloatOperands::Raw`) so that Parquet pruning still recognizes it, but only while the reader
110+
prunes with them without filtering rows. With
111+
`spark.comet.parquet.rowFilterPushdown.enabled=true` they normalize both sides, because the
112+
reader drops the rows a filter rejects, and a raw column would reject a stored NaN that Spark
113+
matches.
114+
- `IN` normalizes its operands in the serde (`normalizeInOperand` in `predicates.scala`).
115+
- `hash`, `xxhash64`, the native shuffle's hash partitioner and `approx_count_distinct` hash floats
116+
through `hash_input`.
117+
118+
### Spark version differences
119+
120+
Rules that depend on the Spark version are decided in Scala. The serde either chooses the native
121+
path or passes a flag to native code, as `Mode`'s `normalize_neg_zero` does. Where a patch release
122+
changed the rule, check the runtime version down to the patch with
123+
`Utils.majorMinorPatchVersion(SPARK_VERSION)`, as `ArraySetSupport` in `arrays.scala` does.
124+
125+
Until an expression has a native implementation that follows its rule, report it `Incompatible`
126+
for floating-point input so that Spark evaluates it.
127+
128+
`spark.comet.exec.strictFloatingPoint=true` makes Comet fall back for floating-point operations
129+
that can still differ from Spark. To gate such an operation, use
130+
`SupportLevel.strictFloatingPointReason(dataType, what)`. It returns a reason only in strict mode,
131+
and only for a type that contains a `FLOAT` or `DOUBLE`.
132+
133+
## Choosing the rule for an expression
134+
135+
1. Read Spark's implementation, both `eval` and `doGenCode`, in every Spark version that Comet
136+
supports, at the latest patch release of each. What it calls decides the rule. `ordering.compare`,
137+
`ordering.equiv`, `ctx.genComp` and `ctx.genEqual` mean SQL ordering. A `java.util.HashMap`, an
138+
`OpenHashSet` or another collection of boxed values means `Double.equals`. `java.util.Arrays.sort`
139+
on a primitive array means `Double.compare`.
140+
2. Check whether Spark's optimizer normalizes the input in the plan. If it does, Comet receives
141+
normalized values on that version only.
142+
3. Apply the rule at every depth. A float inside an array, struct or map follows the same rule, and
143+
some DataFusion kernels fold `-0.0` only in a flat array.
144+
4. Record what you found in the expression's [audit](expression-audits/index.md) and, for a
145+
difference users can see, in the compatibility guide.
146+
147+
## Guidelines
148+
149+
- Don't compare floating-point values with Arrow's comparison, sort or hash kernels, or with
150+
`total_cmp` or `partial_cmp`, where Spark uses SQL ordering, unless you normalize them first.
151+
- Normalize keys and operands, not results. Spark returns the original bits: `greatest(-0.0D, 0.0D)`
152+
returns `-0.0`, `max` returns whichever zero it reads first, and `array_min` returns the first of
153+
equal elements.
154+
- Don't write a local copy of a rule. If a helper is missing, add it to `float_semantics`.
155+
- In a hot loop, use the `float_lt` and `float_gt` predicates. Comparing `compare_floats(a, b)` with
156+
an ordering known only at run time was several times slower in a scan for a minimum.
157+
- When equality on nested values can stop early, keep that exit. `spark_equality` rejects lists of
158+
different lengths without comparing their elements, which an equality built on an ordering
159+
comparator loses.
160+
161+
## Testing
162+
163+
- Use the edge values `0.0`, `-0.0`, a canonical NaN, a NaN with the sign bit set, `1.0`, `-1.0`,
164+
`Infinity`, `-Infinity` and `NULL`, for both `FLOAT` and `DOUBLE`.
165+
- Make the sign-bit NaN at query time. Spark's Parquet writer canonicalizes NaN, so a stored NaN
166+
reads back canonical. Negate it in the query (`-d`), which flips the sign bit on every platform.
167+
Arithmetic such as `sqrt(-1)` yields a sign-bit NaN only on x86-64, so a test that relies on it
168+
passes on Apple Silicon without exercising anything. The writer keeps the sign of zero, so `-0.0`
169+
can be stored directly.
170+
- Read the values from a Parquet table, and also compare them with literals on both sides. The
171+
[Comet SQL Tests](sql-file-tests.md) turn constant folding off, so `-0.0D` stays a literal with the
172+
sign bit set, while `double('NaN')` is a cast.
173+
- Cover every place the rule applies. For a comparison, that is `Project`, `Filter`, aggregate
174+
arguments, `FILTER` clauses, join conditions, sort keys, generators and windows.
175+
- Nest the values in arrays and structs, two levels deep, and make the nested NaN a sign-bit one.
176+
- When arithmetic produces the NaN, run with ANSI mode both on and off
177+
(`-- ConfigMatrix: spark.sql.ansi.enabled=false,true`), because division and remainder take
178+
different native paths.
179+
- Where the rule changed in a Spark release, gate the fixtures with `MinSparkVersion` or
180+
`MaxSparkVersion`, and add the `run-all-spark-profiles` label to the pull request. CI pins one
181+
patch release per Spark line, so when a patch release changed the rule, also test the routing in
182+
Scala.
183+
- Add the expression to `CometFloatSemanticsSuite`, which crosses the edge values with operator
184+
contexts and with every expression and aggregate that accepts floating-point input. New
185+
expressions go in its `expressions` or `aggregates` list. If one does not match Spark yet, add a
186+
`KnownGap` that links the issue. Known gaps are strict: a case that starts matching Spark fails,
187+
so the fix also removes the entry.
188+
189+
Fixtures to start from include `expressions/conditional/float_comparisons.sql`,
190+
`expressions/math/nan_divisor.sql`, `expressions/math/greatest_least_floating_point.sql`,
191+
`expressions/aggregate/min_max_floating_point.sql`, `expressions/array/sort_array_floating_point.sql`
192+
and `windows/nested_float_order_keys.sql`, all under `spark/src/test/resources/sql-tests/`.

‎docs/source/contributor-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ Development Guide <development>
4747
Comet Plugin Overview <plugin_overview>
4848
Arrow FFI <ffi>
4949
Timezone Handling <timezones>
50+
Floating-Point Semantics <floating_point>
5051
JVM Shuffle <jvm_shuffle>
5152
Native Shuffle <native_shuffle>
5253
Memory Management <memory_management>

0 commit comments

Comments
 (0)