Skip to content

fix(spark): array_contains compares floats the way Spark does - #83

Merged
andygrove merged 1 commit into
apache:mainfrom
Yupsecous:array-contains-float-equality-34
Nov 13, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
Yupsecous:array-contains-float-equality-34

Conversation

@Yupsecous

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

array_contains delegates the element comparison to array_has, which compares floating point values by their bits. Spark compares them with -0.0 == 0.0 and NaN == NaN: CodegenContext.genEqual in Spark 4.2.0 generates (java.lang.Double.isNaN(c1) && java.lang.Double.isNaN(c2)) || c1 == c2 (and the same for Float).

So array_contains(array(-0.0D), 0.0D) returned false here and true in Spark, and two NaNs with different sign bits or payloads did not match, at any depth of list and struct elements.

What changes are included in this PR?

  • array_contains.rs rewrites the float values of both arguments before calling array_has: -0.0 becomes 0.0 and every NaN becomes the canonical NaN. It applies to scalar and array arguments, and to Float32/Float64 at any depth of List, LargeList, FixedSizeList and Struct. Arrays that hold no floats are returned as they are. The existing null handling is unchanged.
  • FixedSizeList is rebuilt with try_new_with_length, so a zero-width list keeps its rows.
  • array/array_contains.slt: new queries for both zeros in both directions (double and float), NaN with a flipped sign bit, LargeList and FixedSizeList haystacks, nested lists and structs, null semantics, and a table with column arguments.
  • Unit tests in array_contains.rs for the normalization (scalar and nested shapes, zero-width lists, reuse of children without floats).

An earlier attempt at this issue, apache/datafusion#26011, was closed before the move. Its review asked for a length-preserving rebuild of zero-width FixedSizeList and tests for it, which this PR includes.

What is the testing strategy for this PR?

  • cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, cargo test -p datafusion-spark (331 passed) and cargo test -p datafusion-spark-sqllogictest (245 of 245 files) pass.
  • With array_contains.rs from main, the new float queries fail. Run one by one, 9 of 14 probe queries fail on main (top-level -0.0, NaN with a flipped sign bit at every depth, LargeList/FixedSizeList); nested -0.0 already compares equal on main and is kept as a guard.
  • Each branch of the new code is covered by at least one test that fails when the branch is removed (checked by hand: no normalization, NaN branch, zero branch for f32 and f64, struct, list, scalar path, zero-width length, early return).

Unknowns and assumptions

Are there any user-facing changes?

array_contains on float or double elements now matches Spark for -0.0 and NaN. No API change.

array_contains delegated the element comparison to array_has, which
compares floating point values by their bits. Spark treats -0.0 as equal
to 0.0 and every NaN as equal to every other NaN, at any depth of list
and struct elements.

Rewrite -0.0 to 0.0 and every NaN to the canonical NaN in both arguments
before calling array_has. Arrays that hold no floats are returned as they
are, and FixedSizeList keeps its length so zero-width lists keep their rows.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Float equality now matches Spark (NaN == NaN, -0.0 == 0.0), including within nested arrays/structs, and the new tests fail on main. Thanks @Yupsecous!

Optional follow-ups: skip the values-buffer copy when no -0.0/NaN is present, and add a Float32 nested case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants