You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7dd1c6a
Browse filesBrowse the repository at this point in the historyBrowse files
feat: Support IEEE 754 negative zero semantics (apache#22835)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes#123` indicates that this PR will close issue #123.
-->
- Closesapache#22826 .
- Closesapache#22490
- Closesapache#11108
## Rationale for this change
SQL (per PG and IEEE 754) treats `+0.0` and `-0.0` as equal in `=`, `IS
DISTINCT FROM`, DISTINCT, GROUP BY, UNION/INTERSECT/EXCEPT, equi-joins,
and `array_*` set ops. DataFusion treated them as
distinct because:
- Arrow's `cmp::eq`/`gt`/`lt` use IEEE 754 totalOrder for floats —
`arrow-ord-58.3.0/src/cmp.rs:71-75` explicitly says *"please normalize
zeros before calling this kernel"*.
- Arrow's `RowConverter` row-encodes floats with totalOrder; ±0 produce
different bytes.
- DataFusion's primitive float hashing used raw `to_bits()` /
`to_ne_bytes()`, so ±0 hashed to different buckets.
## What changes are included in this PR?
**Helper** in `datafusion/common/src/utils/mod.rs`:
- `normalize_float_zero(&ArrayRef) -> ArrayRef` — rewrites `-0.0 → +0.0`
for Float16/32/64 via `PrimitiveArray::unary`; `Arc::clone` for
non-float. NaN payloads preserved (`bits << 1 == 0` matches only
±0).
- `normalize_float_zero_scalar(ScalarValue) -> ScalarValue` — symmetric
for scalars.
**Applied at six boundary sites** where DataFusion hands float data to
Arrow:
| Site | Fixes |
|---|---|
| `physical-expr-common/src/datum.rs::apply_cmp` | BinaryExpr `=`, `<`,
`>`, `IS DISTINCT FROM` |
| `physical-plan/src/joins/utils.rs::eq_dyn_null` | HashJoin row
equality → INNER JOIN, INTERSECT, EXCEPT |
| `physical-plan/src/aggregates/group_values/row.rs::intern` |
Multi-column row-encoded GROUP BY |
| `functions-nested/src/set_ops.rs::general_array_distinct` |
`array_distinct` |
| `functions-nested/src/set_ops.rs::generic_set_lists` | `array_union`,
`array_intersect` |
| `functions-nested/src/except.rs::general_except` | `array_except` |
**Float hash macros** normalize for consistency:
- `datafusion/common/src/hash_utils.rs::hash_float_value!` —
`create_hashes`, hash joins, shuffle.
-
`datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs::hash_float!`
— single-column primitive GROUP BY fast path.
**Single-column primitive GROUP BY / DISTINCT**
(`GroupValuesPrimitive::intern`,
`PrimitiveGroupValueBuilder::{append_val,vectorized_append,equal_to,vectorized_equal_to_*}`)
canonicalize the input via
a new default-identity `canonicalize` method on the local `HashValue`
trait (float override only). Trait visibility lifted to `pub` so the
multi-column file can use it.
0 commit comments