Array operations do not normalize -0.0 to 0.0

XMLWordPrintableJSON

      IEEE 754 defines -0.0 == 0.0, but they have different binary representations and different hash codes. This causes array operations to behave incorrectly when arrays contain both -0.0 and 0.0.

      Spark normalizes -0.0 to 0.0 in join keys, window partition keys, and aggregate grouping keys via NormalizeFloatingNumbers. However, hash-based array operations do not apply this normalization, causing inconsistent behavior.

      Affected operations:

      • array_distinct
      • array_union
      • array_intersect
      • array_except
      • arrays_overlap

      Examples:

      Seq(Array(0.0, -0.0, 1.0)).toDF("values").selectExpr("array_distinct(values)").show()
      // Returns [0.0, -0.0, 1.0] instead of [0.0, 1.0]
      
      Seq((Array(0.0), Array(-0.0))).toDF("a", "b").selectExpr("array_union(a, b)").show()
      // Returns [0.0, -0.0] instead of [0.0]
      
      Seq((Array(0.0, 1.0), Array(-0.0, 2.0))).toDF("a", "b").selectExpr("array_intersect(a, b)").show()
      // Returns [] instead of [0.0]
      
      Seq((Array(0.0, 1.0), Array(-0.0))).toDF("a", "b").selectExpr("array_except(a, b)").show()
      // Returns [0.0, 1.0] instead of [1.0]
      

      Subsumes SPARK-51475

            Assignee:
            Albert Sugranyes
            Reporter:
            Albert Sugranyes
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 10m
                1h 10m