|
| 1 | +<!-- |
| 2 | +Licensed to the Apache Software Foundation (ASF) under one |
| 3 | +or more contributor license agreements. See the NOTICE file |
| 4 | +distributed with this work for additional information |
| 5 | +regarding copyright ownership. The ASF licenses this file |
| 6 | +to you under the Apache License, Version 2.0 (the |
| 7 | +"License"); you may not use this file except in compliance |
| 8 | +with the License. You may obtain a copy of the License at |
| 9 | +
|
| 10 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 11 | +
|
| 12 | +Unless required by applicable law or agreed to in writing, |
| 13 | +software distributed under the License is distributed on an |
| 14 | +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 15 | +KIND, either express or implied. See the License for the |
| 16 | +specific language governing permissions and limitations |
| 17 | +under the License. |
| 18 | +--> |
| 19 | + |
| 20 | +# Floating-Point Semantics |
| 21 | + |
| 22 | +This page describes how Comet matches Spark's handling of `-0.0` and NaN in `FLOAT` and `DOUBLE` |
| 23 | +values. It is aimed at contributors working on an expression or operator that compares, orders, |
| 24 | +hashes, deduplicates or groups floating-point values, including values nested in arrays, structs |
| 25 | +and maps. For user-facing differences from Spark, see |
| 26 | +[Floating-point Number Comparison](../user-guide/latest/compatibility/floating-point.md). The work |
| 27 | +to apply these rules systematically is tracked in |
| 28 | +[#6385](https://github.com/apache/datafusion-comet/issues/6385). |
| 29 | + |
| 30 | +The short version: Spark has no single rule for floating-point equality. Each function inherits one |
| 31 | +from the Java API that its implementation calls. Arrow and DataFusion follow IEEE 754 total order, |
| 32 | +which matches none of them. A native implementation must follow the rule of the Spark function it |
| 33 | +replaces, using the shared helpers in `native/spark-expr/src/float_semantics/`, and its tests must |
| 34 | +include a NaN with the sign bit set. |
| 35 | + |
| 36 | +## Spark's rules |
| 37 | + |
| 38 | +| Rule | `-0.0` and `0.0` | NaN | Used by | |
| 39 | +| ----------------------------------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 40 | +| SQL ordering (`SQLOrderingUtil.compareDoubles`, `genEqual`) | Equal | All NaNs are equal, and NaN sorts above all values | Comparisons, `IN`, sorting and ranking, `min`/`max`, `greatest`/`least`, `sort_array`, `array_min`/`array_max`, `array_contains`, `array_position`, `array_remove`, and comparisons of arrays and structs | |
| 41 | +| `NormalizeNaNAndZero`, inserted by Spark's optimizer | `-0.0` becomes `0.0` | Every NaN becomes the canonical NaN | Grouping keys, join keys and window partition keys | |
| 42 | +| `Murmur3Hash` and `XxHash64` | Same hash | Hashed through `doubleToLongBits`, which canonicalizes NaN | `hash`, `xxhash64` and hash partitioning | |
| 43 | +| `java.lang.Double.equals`, for boxed values | Distinct | All NaNs are equal | Hash sets and maps of boxed values | |
| 44 | +| `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code | |
| 45 | + |
| 46 | +Some functions changed rules in a Spark release, so a native path has to follow the Spark version |
| 47 | +it runs against: |
| 48 | + |
| 49 | +- `collect_set` keys its buffer by `Double.equals` before Spark 4.2. Spark 4.2 normalizes NaN and |
| 50 | + `-0.0` first (SPARK-57298). |
| 51 | +- `mode` keys its frequency map by `Double.equals` before Spark 4.2. Spark 4.2 folds `-0.0` into |
| 52 | + `0.0` first (SPARK-57329). |
| 53 | +- `array_distinct` and `array_union` keep signed zeros apart in a flat array before Spark 4.2.0. |
| 54 | + Spark 4.2.0 normalizes their arguments in the plan (SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 |
| 55 | + they normalize while they evaluate instead (SPARK-59602). |
| 56 | +- Map construction (`ArrayBasedMapBuilder`) finds duplicate keys by `Double.equals` in Spark 3.4 |
| 57 | + and 3.5. From Spark 4.0 it normalizes each key first, unless |
| 58 | + `spark.sql.legacy.disableMapKeyNormalization` is set |
| 59 | + ([#6549](https://github.com/apache/datafusion-comet/issues/6549)). |
| 60 | + |
| 61 | +## How Arrow and DataFusion differ |
| 62 | + |
| 63 | +Arrow compares floats by IEEE 754 total order. `-0.0` sorts below `0.0`, NaNs compare by their bit |
| 64 | +patterns, and a NaN with the sign bit set sorts below `-Infinity`. Arrow's sort, row format and |
| 65 | +hash kernels all work from the same bits. DataFusion 55 folds `-0.0` into `0.0` in some kernels but |
| 66 | +does not canonicalize NaN, and that has changed between DataFusion releases. Don't rely on it: |
| 67 | +normalize the values, or compare them with the helpers below. |
| 68 | + |
| 69 | +NaNs with the sign bit set are the normal case on x86-64, not a corner case. Every NaN that |
| 70 | +arithmetic produces at run time, such as `sqrt(-1)` or `Infinity - Infinity`, is |
| 71 | +`0xfff8000000000000` in both Rust and the JVM on x86-64, while aarch64 produces |
| 72 | +`0x7ff8000000000000`. Spark hides the difference through `doubleToLongBits`. A native path that |
| 73 | +compares raw values puts those NaNs below every other value, so a query that passes on an Apple |
| 74 | +Silicon laptop can fail on a Linux x86 cluster. |
| 75 | + |
| 76 | +## Where Comet applies the rules |
| 77 | + |
| 78 | +### The `float_semantics` module |
| 79 | + |
| 80 | +`native/spark-expr/src/float_semantics/` holds the per-value rules and the kernels built on them. |
| 81 | +Its module documentation is the reference. In short: |
| 82 | + |
| 83 | +| Helper | Rule | Use it to | |
| 84 | +| ---------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------- | |
| 85 | +| `compare_floats`, `float_lt`, `float_gt` | SQL ordering | Compare values in a kernel that returns the original bits, as `array_min` and `greatest` do | |
| 86 | +| `spark_comparator`, `spark_equality` | SQL ordering, at any depth | Compare arrays and structs in place | |
| 87 | +| `normalize_float`, `normalize_floats`, `normalize_nested_floats` | `NormalizeNaNAndZero` | Normalize values before an Arrow kernel sorts, row-encodes, hashes or compares them | |
| 88 | +| `NormalizeNaNAndZero`, `NormalizeNestedFloats` | `NormalizeNaNAndZero` | Wrap a key or operand expression. `wrap_if_needed` skips other types and keys that Spark already normalized | |
| 89 | +| `canonicalize_nan` | `java.lang.Double.equals` | Key a hash set or map the way boxed Java values do | |
| 90 | +| `compare_floats_java` | `java.lang.Double.compare` | Sort the way `java.util.Arrays.sort` sorts a primitive array | |
| 91 | +| `hash_input` | `Murmur3Hash` and `XxHash64` | Get the value to hash in place of a float | |
| 92 | + |
| 93 | +Once `-0.0` is folded and every NaN is canonical, Arrow's total order agrees with Spark's SQL |
| 94 | +ordering. That is why normalizing the inputs of an Arrow kernel works, as long as only the keys are |
| 95 | +normalized and the output keeps the original values. |
| 96 | + |
| 97 | +### Keys, sorting and comparisons |
| 98 | + |
| 99 | +- Spark's optimizer wraps grouping, join and window partition keys in `NormalizeNaNAndZero` |
| 100 | + (`NormalizeFloatingNumbers`). Comet serializes it like any other expression. |
| 101 | +- `create_normalized_key_expr` in `native/core/src/execution/planner.rs` normalizes sort keys, |
| 102 | + window order and partition keys, and the partition keys of `WindowGroupLimit`, so that Arrow's |
| 103 | + sort orders them the Spark way. |
| 104 | +- `spark_comparison` in `native/spark-expr/src/array_funcs/nested_comparison.rs` builds every native |
| 105 | + `=`, `<>`, `<=>`, `<`, `<=`, `>`, `>=` and `IS DISTINCT FROM`, whichever operator evaluates it. It |
| 106 | + normalizes float operands, and folds a literal while the plan is built so that it stays a |
| 107 | + literal. Nested `=` and `<>` compare in place with `spark_equality`. |
| 108 | +- A scan's pushed-down data filters leave a float column compared with a literal unwrapped |
| 109 | + (`FloatOperands::Raw`) so that Parquet pruning still recognizes it, but only while the reader |
| 110 | + prunes with them without filtering rows. With |
| 111 | + `spark.comet.parquet.rowFilterPushdown.enabled=true` they normalize both sides, because the |
| 112 | + reader drops the rows a filter rejects, and a raw column would reject a stored NaN that Spark |
| 113 | + matches. |
| 114 | +- `IN` normalizes its operands in the serde (`normalizeInOperand` in `predicates.scala`). |
| 115 | +- `hash`, `xxhash64`, the native shuffle's hash partitioner and `approx_count_distinct` hash floats |
| 116 | + through `hash_input`. |
| 117 | + |
| 118 | +### Spark version differences |
| 119 | + |
| 120 | +Rules that depend on the Spark version are decided in Scala. The serde either chooses the native |
| 121 | +path or passes a flag to native code, as `Mode`'s `normalize_neg_zero` does. Where a patch release |
| 122 | +changed the rule, check the runtime version down to the patch with |
| 123 | +`Utils.majorMinorPatchVersion(SPARK_VERSION)`, as `ArraySetSupport` in `arrays.scala` does. |
| 124 | + |
| 125 | +Until an expression has a native implementation that follows its rule, report it `Incompatible` |
| 126 | +for floating-point input so that Spark evaluates it. |
| 127 | + |
| 128 | +`spark.comet.exec.strictFloatingPoint=true` makes Comet fall back for floating-point operations |
| 129 | +that can still differ from Spark. To gate such an operation, use |
| 130 | +`SupportLevel.strictFloatingPointReason(dataType, what)`. It returns a reason only in strict mode, |
| 131 | +and only for a type that contains a `FLOAT` or `DOUBLE`. |
| 132 | + |
| 133 | +## Choosing the rule for an expression |
| 134 | + |
| 135 | +1. Read Spark's implementation, both `eval` and `doGenCode`, in every Spark version that Comet |
| 136 | + supports, at the latest patch release of each. What it calls decides the rule. `ordering.compare`, |
| 137 | + `ordering.equiv`, `ctx.genComp` and `ctx.genEqual` mean SQL ordering. A `java.util.HashMap`, an |
| 138 | + `OpenHashSet` or another collection of boxed values means `Double.equals`. `java.util.Arrays.sort` |
| 139 | + on a primitive array means `Double.compare`. |
| 140 | +2. Check whether Spark's optimizer normalizes the input in the plan. If it does, Comet receives |
| 141 | + normalized values on that version only. |
| 142 | +3. Apply the rule at every depth. A float inside an array, struct or map follows the same rule, and |
| 143 | + some DataFusion kernels fold `-0.0` only in a flat array. |
| 144 | +4. Record what you found in the expression's [audit](expression-audits/index.md) and, for a |
| 145 | + difference users can see, in the compatibility guide. |
| 146 | + |
| 147 | +## Guidelines |
| 148 | + |
| 149 | +- Don't compare floating-point values with Arrow's comparison, sort or hash kernels, or with |
| 150 | + `total_cmp` or `partial_cmp`, where Spark uses SQL ordering, unless you normalize them first. |
| 151 | +- Normalize keys and operands, not results. Spark returns the original bits: `greatest(-0.0D, 0.0D)` |
| 152 | + returns `-0.0`, `max` returns whichever zero it reads first, and `array_min` returns the first of |
| 153 | + equal elements. |
| 154 | +- Don't write a local copy of a rule. If a helper is missing, add it to `float_semantics`. |
| 155 | +- In a hot loop, use the `float_lt` and `float_gt` predicates. Comparing `compare_floats(a, b)` with |
| 156 | + an ordering known only at run time was several times slower in a scan for a minimum. |
| 157 | +- When equality on nested values can stop early, keep that exit. `spark_equality` rejects lists of |
| 158 | + different lengths without comparing their elements, which an equality built on an ordering |
| 159 | + comparator loses. |
| 160 | + |
| 161 | +## Testing |
| 162 | + |
| 163 | +- Use the edge values `0.0`, `-0.0`, a canonical NaN, a NaN with the sign bit set, `1.0`, `-1.0`, |
| 164 | + `Infinity`, `-Infinity` and `NULL`, for both `FLOAT` and `DOUBLE`. |
| 165 | +- Make the sign-bit NaN at query time. Spark's Parquet writer canonicalizes NaN, so a stored NaN |
| 166 | + reads back canonical. Negate it in the query (`-d`), which flips the sign bit on every platform. |
| 167 | + Arithmetic such as `sqrt(-1)` yields a sign-bit NaN only on x86-64, so a test that relies on it |
| 168 | + passes on Apple Silicon without exercising anything. The writer keeps the sign of zero, so `-0.0` |
| 169 | + can be stored directly. |
| 170 | +- Read the values from a Parquet table, and also compare them with literals on both sides. The |
| 171 | + [Comet SQL Tests](sql-file-tests.md) turn constant folding off, so `-0.0D` stays a literal with the |
| 172 | + sign bit set, while `double('NaN')` is a cast. |
| 173 | +- Cover every place the rule applies. For a comparison, that is `Project`, `Filter`, aggregate |
| 174 | + arguments, `FILTER` clauses, join conditions, sort keys, generators and windows. |
| 175 | +- Nest the values in arrays and structs, two levels deep, and make the nested NaN a sign-bit one. |
| 176 | +- When arithmetic produces the NaN, run with ANSI mode both on and off |
| 177 | + (`-- ConfigMatrix: spark.sql.ansi.enabled=false,true`), because division and remainder take |
| 178 | + different native paths. |
| 179 | +- Where the rule changed in a Spark release, gate the fixtures with `MinSparkVersion` or |
| 180 | + `MaxSparkVersion`, and add the `run-all-spark-profiles` label to the pull request. CI pins one |
| 181 | + patch release per Spark line, so when a patch release changed the rule, also test the routing in |
| 182 | + Scala. |
| 183 | +- Add the expression to `CometFloatSemanticsSuite`, which crosses the edge values with operator |
| 184 | + contexts and with every expression and aggregate that accepts floating-point input. New |
| 185 | + expressions go in its `expressions` or `aggregates` list. If one does not match Spark yet, add a |
| 186 | + `KnownGap` that links the issue. Known gaps are strict: a case that starts matching Spark fails, |
| 187 | + so the fix also removes the entry. |
| 188 | + |
| 189 | +Fixtures to start from include `expressions/conditional/float_comparisons.sql`, |
| 190 | +`expressions/math/nan_divisor.sql`, `expressions/math/greatest_least_floating_point.sql`, |
| 191 | +`expressions/aggregate/min_max_floating_point.sql`, `expressions/array/sort_array_floating_point.sql` |
| 192 | +and `windows/nested_float_order_keys.sql`, all under `spark/src/test/resources/sql-tests/`. |
0 commit comments