You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 432523f
Browse filesBrowse the repository at this point in the historyBrowse files
fix: hash NaN the way Spark does in hash, xxhash64 and native shuffle (apache#6413)
* refactor: move native -0.0 and NaN handling into one float_semantics module
Part of apache#6385. Each native expression that follows one of Spark's float
rules carried its own copy of that rule. This moves them onto a new
`float_semantics` module in `spark-expr`, with no change in behavior.
- `normalize_float`, `canonicalize_nan`, `compare_floats` and `hash_input`
are the per-value rules: `NormalizeNaNAndZero`, `Double.equals`,
`compareDoubles`, and the `Murmur3Hash`/`XxHash64` input.
- `normalize_floats`, `normalize_nested_floats` and `has_float_leaf`
normalize arrays. `NormalizeNaNAndZero` and `NormalizeNestedFloats`
move into the module, and `NormalizeNaNAndZero::wrap_if_needed` takes
over the planner's check in `create_normalized_key_expr`.
- `spark_comparator` replaces both `nested_equality` (nested `=` and
`IN`) and the comparator local to `array_min`/`array_max`.
hll_plus_plus, max_min_by, mode, percentile, array_extrema,
array_position, arrays_overlap, nested_comparison, the float hash macros,
and the planner's sort keys and range partition bounds now call the
module.
* Restore hot-loop speed and narrow the float_semantics API
- compare_floats uses the if-chain form again, so `.is_eq()` in
array_position and percentile compiles to a single comparison.
- float_extrema calls new float_lt/float_gt predicates, which compile to
the same instructions as the inline tests they replace. The three-way
comparison was up to 9x slower in that loop.
- spark_equality gives nested `=` and `IN` back the length check that
skips comparing the elements of lists of different lengths.
- float_semantics is a private module again, re-exporting
normalize_floats, canonicalize_nan and the two expressions.
- The Parquet variant cast canonicalizes NaN with canonicalize_nan, and
gains a test.
- Tests share the NaN constants and check the per-value rules in one
table, and the max_by doc defers to the module docs.
* Return an error, not a panic, for mismatched physical types in the comparator
`check_types` compares logical types, so a `Dictionary(_, Float64)` passes
as `Float64` and then panicked when `float_comparator` downcast it. The
comparator now matches on both sides' types and returns the internal error.
`mismatched_types_are_rejected` covers a dictionary on either side, at the
top level and inside a list.
* fix: hash NaN the way Spark does in hash, xxhash64 and native shuffle
Part of apache#6385. Spark hashes a float through doubleToLongBits or
floatToIntBits, which canonicalize NaN, so every NaN hashes alike. Comet
hashed the raw bits, so `hash(-d)` and `xxhash64(-d)` differed from Spark
for a NaN, and on x86-64 so did the hash of every NaN that arithmetic
produces. `hash_input` now returns `normalize_float`, which covers `hash`,
`xxhash64`, list, struct and dictionary elements, the native shuffle's hash
partitioner and `approx_count_distinct`.
At the default seed, `xxhash64` handed float arguments to datafusion-spark's
`SparkXxhash64`, which also hashes the raw bits, so floats now stay on
Comet's kernel. `approx_count_distinct` no longer normalizes floats before
hashing, since `xxhash64` now does.
* test: cover infinities in hash NaN tests and link the upstream xxhash64 issue
Add Infinity and -Infinity rows to the hash fixture, so hash(-d) and
hash(-f) show that both infinities keep their own hashes through NaN
canonicalization. Link apache/datafusion#25913, the upstream report of
SparkXxhash64 hashing a NaN's raw bits, at the gate that keeps floats on
Comet's kernel, and list floats among the retained cases in the xxhash64
audit entry.
* test: cover maps and nested floats in the hash NaN tests
* test: drop the sweep suite's known gap for hash and xxhash64
hash, xxhash64 and the native shuffle's hash partitioner now hash NaN
the way Spark does, so the suite requires those cases to match.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/hash_funcs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,6 +70,6 @@
70
70
- Spark 3.5.8 (audited 2026-05-27): baseline. `XxHash64(children, seed) extends HashExpression[Long]`; produces an xxHash64 hash with a configurable Long seed and `LongType` result. Comet routes via `CometXxHash64` to the native `xxhash64` UDF.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
73
-
- Upstream (2026-09-15): Comet's native `xxhash64` UDF delegates compatible arguments at Spark's default seed (`42`) to `datafusion-spark::SparkXxhash64`. Differential tests in `native/spark-expr/src/hash_funcs/xxhash64_diff.rs` compare Comet's kernel against `SparkXxhash64` for primitives, both Decimal128 widths, dictionaries, lists, maps, and nested combinations. The Comet kernel is retained for a non-default seed (`SparkXxhash64` hardcodes 42), `Struct` (upstream does not push a parent null mask into children; see #5753), a `Dictionary` nested in a list/map (upstream restarts those hashes from 42), and `Time64`. `murmur3` is unchanged; `create_xxhash64_hashes` remains for `approx_count_distinct` and the fallback path.
73
+
- Upstream (2026-09-15): Comet's native `xxhash64` UDF delegates compatible arguments at Spark's default seed (`42`) to `datafusion-spark::SparkXxhash64`. Differential tests in `native/spark-expr/src/hash_funcs/xxhash64_diff.rs` compare Comet's kernel against `SparkXxhash64` for primitives, both Decimal128 widths, dictionaries, lists, maps, and nested combinations, and record where the two differ on a NaN whose bits are not canonical. The Comet kernel is retained for a non-default seed (`SparkXxhash64` hardcodes 42), `Struct` (upstream does not push a parent null mask into children; see #5753), a `Dictionary` nested in a list/map (upstream restarts those hashes from 42), `Time64`, and `Float32`/`Float64`, including values nested in lists and maps (upstream hashes the raw bits of a NaN, where Spark hashes every NaN as the canonical NaN; see [apache/datafusion#25913](https://github.com/apache/datafusion/issues/25913)). `murmur3` is unchanged; `create_xxhash64_hashes` remains for `approx_count_distinct` and the fallback path.
0 commit comments