-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major
-
Affects Version/s: 5.0.0
-
Component/s: SQL
`collect_set` is documented to return a set of distinct elements, and under Spark SQL semantics all NaN values are equal (and `-0.0 == 0.0`). However, `collect_set` over float/double columns returns duplicate `NaN` elements:
-- Top-level NaN: HashSet compares boxed values with primitive equality (NaN != NaN) SELECT collect_set(v) FROM VALUES (double('NaN')), (double('NaN')) AS t(v); -- before: [NaN, NaN] after: [NaN] -- Nested -0.0 / 0.0: GenericArrayData/GenericInternalRow hash -0.0 and 0.0 to -- different buckets, so they are not deduplicated SELECT collect_set(a) FROM VALUES (array(-0.0D)), (array(0.0D)) AS t(a); -- before: [[-0.0], [0.0]] after: [[0.0]] SELECT collect_set(named_struct('a', v)) FROM VALUES (-0.0D), (0.0D) AS t(v); -- before: [{a:-0.0}, {a:0.0}] after: [{a:0.0}]
This is inconsistent with array_distinct , count(DISTINCT), grouping/join/window keys, and the array set operations - all of which treat NaNs as equal.