Fix IN expr for NaN - #7378

…lists (apache#25187)

## Which issue does this PR close?

- Part of apache#19241.

## Rationale for this change

DataFusion currently rewrites short `IN` lists like `x IN (1, 2)` as `x
= 1 OR x = 2`. Evaluating `IN` directly can be faster.

This PR keeps the `IN` form for the types below, so short lists can use
DataFusion's existing optimized `IN` filters.

## What changes are included in this PR?

- Keep two- and three-value `IN` and `NOT IN` lists for `Int32`,
`Int64`, `UInt64`, `Float32`, `Float64`, and `FixedSizeBinary` columns
with widths of 1, 2, 4, 8, or 16 bytes. The list must contain constant
values of the column's type.
- Add benchmarks that use normal optimization to compare performance
before and after this change.
- Add optimizer and SQL logic tests covering list sizes, types, NULLs,
and non-constant list values.

## What is the testing strategy for this PR?

All 935 optimizer unit tests and all 533 SQL logic files passed,
including TPC-H. Formatting, Clippy, and lint checks passed. All 336
benchmark cases passed on both builds.

The benchmark bot's ARM
[two-value](apache#25187 (comment))
and
[three-value](apache#25187 (comment))
results show:

- Integer lists take about 32–55% less time when half the rows match or
none do. In the extreme case where every row matches the first value,
they can take up to about twice as long.
- Floating-point and fixed-size binary lists are faster in every tested
case, including NULLs and small batches.

<details>
<summary>Benchmark setup</summary>

Both builds use the same 336 cases. The
[baseline](apache@31a333d)
contains `main` at `06aa131cda0` plus the same benchmark code as the
[PR](apache@8aea24b).

Queries use normal optimization. Timings cover running the chosen
expressions; query planning and table setup happen before timing.

</details>

## Are there any user-facing changes?

`EXPLAIN` shows `IN` or `NOT IN` for these short lists in place of the
expanded `OR` or `AND` comparisons.

This performance PR aligns constant two- and three-value
`Float32`/`Float64` lists with existing behavior for 4+ values,
including the signed-zero bug (`+0.0` versus `-0.0`). That bug [predates
this optimization
series](apache#7378) and already
affects longer lists with constant or column values; fixing it is left
to apache#25186.

---------

Co-authored-by: Andrew Lamb <[email protected]>

## Which issue does this PR close?

- Part of apache#26138 (shared floating-point semantics).
- Covers the IN-list regression reported in apache#27121.
- Closes apache#27121
- Closes apache#27122

## Rationale for this change

DataFusion equality treats `+0.0` and `-0.0` as equal, but the optimized
`IN` filters compare or hash floating-point bit patterns. Because the
two zeros have different encodings, an `IN` predicate could therefore
return a different result depending on whether the optimizer retained it
or rewrote it into equality comparisons.

This signed-zero limitation predates the IN-list optimization epic
apache#19241: [apache#7378](apache#7378) already
used bit-pattern equality for floating-point membership in August 2023.
[apache#22835](apache#22835) corrected
ordinary comparisons in June 2026 but did not fix `IN` membership. This
PR addresses that remaining correctness gap.

This PR makes `Float16`, `Float32`, and `Float64` `IN` predicates,
including floats inside nested and encoded values, follow the same
signed-zero equality as ordinary comparisons. The shared static-filter
factory adds the missing zero encoding before choosing a primitive
filter. Strategy selection uses the expanded physical non-null list
length, so both encodings count toward the existing cutoffs. Generic
static filters and non-static lists use the shared `normalize_floats`
and recursive `ScalarValue::normalize` APIs before comparison. Distinct
NaN payloads remain distinct, preserving the existing bit-equality
behavior for NaNs.

## What changes are included in this PR?

- Add the missing signed-zero encoding once in the shared static-filter
factory, after flattening dictionary lists.
- Select branchless, bitmap, or hash-set lookup using the expanded
physical non-null key count.
- Extend `ScalarValue::normalize` to nested and encoded values, and use
`float_semantics::normalize_floats` for arrays in dynamic comparisons
and generic static filters.
- Prevent structural literal equality from driving IN-list set-operation
rewrites for types containing floating-point values.
- Preserve null behavior, dictionary inputs, `IN`, and `NOT IN`.

## What is the testing strategy for this PR?

SQL regressions in `negative_zero.slt` cover both signed-zero
directions, static filters, dynamic scalar and array operands, nested
values, IN-list intersection/difference, `IN`, `NOT IN`, and nulls.
Focused Rust tests cover dictionary inputs, distinct NaN payloads,
encoded scalar metadata, and sliced values. Factory tests exercise both
zero signs immediately below and at each floating-point branchless
cutoff, including dispatch after adding the missing physical key. SQL
tests also check constant nested scalar self-membership; an existing
Rust test covers nested dictionary scalar inputs.

Validation after rebasing onto `upstream/main` (`4bf32cb46d`):
formatting, CI Clippy with its standard feature set, all 111 focused
Rust tests, and all 546 sqllogictest files (including Parquet
encryption) passed.

## Are there any user-facing changes?

Yes. Floating-point `IN` and `NOT IN`, including nested and encoded
values, now treat `+0.0` and `-0.0` as equal, consistently with ordinary
DataFusion equality. `ScalarValue::normalize` now applies these rules
recursively; public API signatures are unchanged.

Prepared with AI assistance.

---------

Co-authored-by: Andrew Lamb <[email protected]>