Skip to content

array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701

New issue
New issue
Closed
#6563
#5750
Closed
array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918#5701
#6563
#5750
Assignees
peterxcli
Labels
area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:mediumFunctional bugs, performance regressions, broken featuresregressionA bug that did not affect the most recent Comet release

Description

@andygrove
andygrove
opened on Sep 5, 2026

Describe the bug

array_distinct and array_union run natively by default (ArrayDistinct via a plain CometScalarFunction, ArrayUnion via CometArrayUnion; neither overrides getSupportLevel). DataFusion collapses -0.0 and 0.0, so on a Spark version that does not normalize signed zeros in array functions, Comet silently returns a different answer:

SELECT array_distinct(array(0.0, double('-0.0'), 1.0))
Spark:  [0.0, -0.0, 1.0]
Comet:  [0.0, 1.0]

The plan is CometProject with no fallback. NormalizeFloatingNumbers only rewrites grouping keys, join keys, window partition specs and Distinct, so a plain SELECT diverges for literals as well as column-sourced values.

SPARK-54918 makes Spark normalize signed zeros in array_distinct, array_union, array_intersect, array_except and arrays_overlap, which makes [0.0, 1.0] the correct answer. Its fix versions are 4.2.0, 4.1.4 and 4.0.5. Comet also supports 3.4 and 3.5, which will never get it, and any 4.0.x before 4.0.5 or 4.1.x before 4.1.4.

array_intersect and array_except already report Incompatible, so they only reach the native path under allowIncompatible. array_distinct and array_union do not.

Steps to reproduce

Run the query above on Spark 3.5, or on 4.1.3 (the version the build currently pins).

Expected behavior

On a Spark version without SPARK-54918, either match Spark or fall back. On a version with it, keep the native path.

Additional context

The split is at the patch level, so a compile-time shim keyed on shims.minorVerSrc is not sufficient: the build pins 4.0.4 and 4.1.3, but Comet runs against any patch release of those lines. It needs a runtime check.

Note that CometSparkSessionExtensions.isSparkNNPlus compares org.apache.spark.SPARK_VERSION as a string, so SPARK_VERSION >= "4.0.5" would be wrong once 4.0.10 ships. A numeric comparison helper is needed.

Suggested shape: a getSupportLevel override on the ArrayDistinct and ArrayUnion serdes returning Incompatible when the element type contains FloatType/DoubleType and the running Spark version predates the fix, in the shape CometArrayExcept already uses for its unsupported element types. That keeps the fast path for every other element type and every fixed Spark version.

The signed-zero cases in array_distinct.sql, array_union.sql, array_except.sql and array_intersect.sql are currently query ignore(...); they should be restored once behavior is version-correct.

Found while reviewing #5262.

Activity

  1. andygrove
    added
    bugSomething isn't working
    correctness
    priority:highCrashes, panics, segfaults, major functional breakage
    area:expressionsExpression evaluation
    on Sep 5, 2026
  2. github-actions
    added
    requires-triage
    on Sep 5, 2026

  3. andygrove commented on Sep 5, 2026

    @andygrove
    andygrove
    on Sep 5, 2026
    MemberAuthor

    Tracked under #5709 ([EPIC] DataFusion 55 upgrade follow-ups).

  4. peterxcli commented on Sep 6, 2026

    @peterxcli
    peterxcli
    on Sep 6, 2026
    Member

    take

  5. github-actions
    assigned
    peterxcli
    on Sep 6, 2026
  6. andygrove
    added
    priority:criticalData corruption, silent wrong results, security issues
    and removed
    priority:highCrashes, panics, segfaults, major functional breakage
    requires-triage
    on Sep 14, 2026

  7. andygrove commented on Sep 14, 2026

    @andygrove
    andygrove
    on Sep 14, 2026
    MemberAuthor

    Checked against apache/main at commit 481aefea9c60592612650de73bd2e1c7aa173979 (2026-09-13), Spark 3.5.3, JDK 17, Linux x86_64, stock config (spark.comet.enabled=true, spark.comet.exec.enabled=true, spark.comet.exec.shuffle.enabled=true, spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager, off-heap 8g).

    array_distinct and array_union still reproduce as described here. The same -0.0 handling also shows up in three more functions that don't seem to be tracked yet; all three execute in Comet (CometProject over CometNativeScan). array_except and array_intersect are marked Incompatible and only run natively with spark.comet.expression.ArrayExcept.allowIncompatible=true / spark.comet.expression.ArrayIntersect.allowIncompatible=true; they are included because the divergence is the same signed-zero handling. Values come from parquet tables so the queries are not constant-folded away:

    CREATE TABLE t_except(a array<double>, b array<double>) USING parquet;
INSERT INTO t_except VALUES (array(0.0D, -0.0D, 1.0D), array(0.0D)), (array(0.0D, 1.0D), array(-0.0D));
SELECT a, b, array_except(a, b) FROM t_except;
-- Spark: [0.0, -0.0, 1.0] | [0.0]  -> [-0.0, 1.0]     Comet: [1.0]
-- Spark: [0.0, 1.0]       | [-0.0] -> [0.0, 1.0]      Comet: [1.0]

CREATE TABLE t_intersect(a array<double>, b array<double>) USING parquet;
INSERT INTO t_intersect VALUES (array(0.0D), array(-0.0D)), (array(-0.0D), array(0.0D)), (array(-0.0D), array(-0.0D));
SELECT a, b, array_intersect(a, b) FROM t_intersect;
-- Spark: [0.0]  | [-0.0] -> []      Comet: [0.0]
-- Spark: [-0.0] | [0.0]  -> []      Comet: [0.0]
-- Spark: [-0.0] | [-0.0] -> [-0.0]  Comet: [0.0]

CREATE TABLE t_sort(arr array<double>) USING parquet;
INSERT INTO t_sort VALUES (array(CAST('Infinity' AS DOUBLE), CAST('-Infinity' AS DOUBLE), CAST('NaN' AS DOUBLE), 3.0D, 1.0D, -0.0D, 0.0D, NULL));
SELECT sort_array(arr, false) FROM t_sort;
-- Spark: [NaN, Infinity, 3.0, 1.0, -0.0, 0.0, -Infinity, NULL]
-- Comet: [NaN, Infinity, 3.0, 1.0, 0.0, -0.0, -Infinity, NULL]

    array_min on the same kind of input is covered by #5401.

  8. kazuyukitanimura commented on Sep 14, 2026

    @kazuyukitanimura
    kazuyukitanimura
    on Sep 14, 2026
    Contributor

    This is a Spark bug, Spark side is fixed. Lowering the priority.

  9. kazuyukitanimura
    added
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed
    priority:criticalData corruption, silent wrong results, security issues
    on Sep 14, 2026
  10. andygrove
    added
    regressionA bug that did not affect the most recent Comet release
    on Oct 1, 2026

  11. andygrove commented on Oct 1, 2026

    @andygrove
    andygrove
    on Oct 1, 2026
    MemberAuthor

    This is a regression in 1.1.0. The 1.1.0 regression audit (#6399) ran the same Parquet-backed query on both releases with the Spark 4.1 profile. On 1.0.0, which used DataFusion 54.1, native array_distinct and array_union keep -0.0 and 0.0 apart and match Spark 4.1.3. On 1.1.0-rc1, which uses DataFusion 55.1, they merge the two and turn -0.0 into 0.0, so size(array_distinct(array(0.0, -0.0, 1.0))) is 2 instead of 3. Both releases run it natively. It's tracked in #6402.

  12. andygrove
    added a commit that references this issue on Oct 1, 2026

    docs: list the DataFusion 55 regressions (apache#5701, apache#6254) i…

    bc7c5a9
  13. peterxcli
    added a commit that references this issue on Oct 2, 2026

    test: drop the apache#5701 float semantics known gap

    e7c4584
  14. andygrove
    closed this as completedin #5750on Oct 3, 2026
  15. andygrove
    added a commit that references this issue on Oct 3, 2026

    docs: drop the Iceberg nested-field and array set regressions, which …

    0b9c490
  16. dwsmith1983
    added a commit that references this issue on Oct 3, 2026

    fix: gate array distinct and union signed-zero semantics by Spark ver…

    1d313f5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:mediumFunctional bugs, performance regressions, broken featuresregressionA bug that did not affect the most recent Comet release

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions