-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major
-
Affects Version/s: 4.2.0
-
Component/s: SQL
IEEE 754 defines -0.0 == 0.0, but they have different binary representations and different hash codes. This causes array operations to behave incorrectly when arrays contain both -0.0 and 0.0.
Spark normalizes -0.0 to 0.0 in join keys, window partition keys, and aggregate grouping keys via NormalizeFloatingNumbers. However, hash-based array operations do not apply this normalization, causing inconsistent behavior.
Affected operations:
- array_distinct
- array_union
- array_intersect
- array_except
- arrays_overlap
Examples:
Seq(Array(0.0, -0.0, 1.0)).toDF("values").selectExpr("array_distinct(values)").show() // Returns [0.0, -0.0, 1.0] instead of [0.0, 1.0] Seq((Array(0.0), Array(-0.0))).toDF("a", "b").selectExpr("array_union(a, b)").show() // Returns [0.0, -0.0] instead of [0.0] Seq((Array(0.0, 1.0), Array(-0.0, 2.0))).toDF("a", "b").selectExpr("array_intersect(a, b)").show() // Returns [] instead of [0.0] Seq((Array(0.0, 1.0), Array(-0.0))).toDF("a", "b").selectExpr("array_except(a, b)").show() // Returns [0.0, 1.0] instead of [1.0]
Subsumes SPARK-51475
- is a parent of
-
SPARK-51475 ArrayDistinct Producing Inconsistent Behavior For -0.0 and +0.0
-
- Resolved
-
- links to
(3 links to)