You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 94486b1
Browse filesBrowse the repository at this point in the historyBrowse files
## Which issue does this PR close?
- Part of #26138 (shared floating-point semantics).
- Covers the IN-list regression reported in #27121.
- Closes#27121
- Closes#27122
## Rationale for this change
DataFusion equality treats `+0.0` and `-0.0` as equal, but the optimized
`IN` filters compare or hash floating-point bit patterns. Because the
two zeros have different encodings, an `IN` predicate could therefore
return a different result depending on whether the optimizer retained it
or rewrote it into equality comparisons.
This signed-zero limitation predates the IN-list optimization epic
#19241: [#7378](#7378) already
used bit-pattern equality for floating-point membership in August 2023.
[#22835](#22835) corrected
ordinary comparisons in June 2026 but did not fix `IN` membership. This
PR addresses that remaining correctness gap.
This PR makes `Float16`, `Float32`, and `Float64` `IN` predicates,
including floats inside nested and encoded values, follow the same
signed-zero equality as ordinary comparisons. The shared static-filter
factory adds the missing zero encoding before choosing a primitive
filter. Strategy selection uses the expanded physical non-null list
length, so both encodings count toward the existing cutoffs. Generic
static filters and non-static lists use the shared `normalize_floats`
and recursive `ScalarValue::normalize` APIs before comparison. Distinct
NaN payloads remain distinct, preserving the existing bit-equality
behavior for NaNs.
## What changes are included in this PR?
- Add the missing signed-zero encoding once in the shared static-filter
factory, after flattening dictionary lists.
- Select branchless, bitmap, or hash-set lookup using the expanded
physical non-null key count.
- Extend `ScalarValue::normalize` to nested and encoded values, and use
`float_semantics::normalize_floats` for arrays in dynamic comparisons
and generic static filters.
- Prevent structural literal equality from driving IN-list set-operation
rewrites for types containing floating-point values.
- Preserve null behavior, dictionary inputs, `IN`, and `NOT IN`.
## What is the testing strategy for this PR?
SQL regressions in `negative_zero.slt` cover both signed-zero
directions, static filters, dynamic scalar and array operands, nested
values, IN-list intersection/difference, `IN`, `NOT IN`, and nulls.
Focused Rust tests cover dictionary inputs, distinct NaN payloads,
encoded scalar metadata, and sliced values. Factory tests exercise both
zero signs immediately below and at each floating-point branchless
cutoff, including dispatch after adding the missing physical key. SQL
tests also check constant nested scalar self-membership; an existing
Rust test covers nested dictionary scalar inputs.
Validation after rebasing onto `upstream/main` (`4bf32cb46d`):
formatting, CI Clippy with its standard feature set, all 111 focused
Rust tests, and all 546 sqllogictest files (including Parquet
encryption) passed.
## Are there any user-facing changes?
Yes. Floating-point `IN` and `NOT IN`, including nested and encoded
values, now treat `+0.0` and `-0.0` as equal, consistently with ordinary
DataFusion equality. `ScalarValue::normalize` now applies these rules
recursively; public API signatures are unchanged.
Prepared with AI assistance.
---------
Co-authored-by: Andrew Lamb <[email protected]>
0 commit comments