Repository navigation
fix(spark): array_contains compares floats the way Spark does - #83
Merged
andygrove merged 1 commit intoNov 13, 2026
Merged
Conversation
array_contains delegated the element comparison to array_has, which compares floating point values by their bits. Spark treats -0.0 as equal to 0.0 and every NaN as equal to every other NaN, at any depth of list and struct elements. Rewrite -0.0 to 0.0 and every NaN to the canonical NaN in both arguments before calling array_has. Arrays that hold no floats are returned as they are, and FixedSizeList keeps its length so zero-width lists keep their rows.
andygrove
approved these changes
Nov 13, 2026
andygrove
left a comment
Member
There was a problem hiding this comment.
LGTM. Float equality now matches Spark (NaN == NaN, -0.0 == 0.0), including within nested arrays/structs, and the new tests fail on main. Thanks @Yupsecous!
Optional follow-ups: skip the values-buffer copy when no -0.0/NaN is present, and add a Float32 nested case.
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
NULLargument difference mentioned in the comments there is not changed by this PR, so I left the issue open.Rationale for this change
array_containsdelegates the element comparison toarray_has, which compares floating point values by their bits. Spark compares them with-0.0 == 0.0andNaN == NaN:CodegenContext.genEqualin Spark 4.2.0 generates(java.lang.Double.isNaN(c1) && java.lang.Double.isNaN(c2)) || c1 == c2(and the same forFloat).So
array_contains(array(-0.0D), 0.0D)returnedfalsehere andtruein Spark, and two NaNs with different sign bits or payloads did not match, at any depth of list and struct elements.What changes are included in this PR?
array_contains.rsrewrites the float values of both arguments before callingarray_has:-0.0becomes0.0and every NaN becomes the canonical NaN. It applies to scalar and array arguments, and toFloat32/Float64at any depth ofList,LargeList,FixedSizeListandStruct. Arrays that hold no floats are returned as they are. The existing null handling is unchanged.FixedSizeListis rebuilt withtry_new_with_length, so a zero-width list keeps its rows.array/array_contains.slt: new queries for both zeros in both directions (doubleandfloat), NaN with a flipped sign bit,LargeListandFixedSizeListhaystacks, nested lists and structs, null semantics, and a table with column arguments.array_contains.rsfor the normalization (scalar and nested shapes, zero-width lists, reuse of children without floats).An earlier attempt at this issue, apache/datafusion#26011, was closed before the move. Its review asked for a length-preserving rebuild of zero-width
FixedSizeListand tests for it, which this PR includes.What is the testing strategy for this PR?
cargo fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warnings,cargo test -p datafusion-spark(331 passed) andcargo test -p datafusion-spark-sqllogictest(245 of 245 files) pass.array_contains.rsfrommain, the new float queries fail. Run one by one, 9 of 14 probe queries fail onmain(top-level-0.0, NaN with a flipped sign bit at every depth,LargeList/FixedSizeList); nested-0.0already compares equal onmainand is kept as a guard.f32andf64, struct, list, scalar path, zero-width length, early return).Unknowns and assumptions
genEqual, 4.2.0) and from the report in spark: array_contains compares floats by bits, where Spark treats -0.0 as 0.0 and all NaNs as equal #34, not from a live PySpark session.arrays_overlap(mentioned in spark: array_contains compares floats by bits, where Spark treats -0.0 as 0.0 and all NaNs as equal #34) is not touched.Are there any user-facing changes?
array_containson float or double elements now matches Spark for-0.0and NaN. No API change.