collect_set returns duplicate NaN/-0.0 values for float/double columns

XMLWordPrintableJSON

      `collect_set` is documented to return a set of distinct elements, and under Spark SQL semantics all NaN values are equal (and `-0.0 == 0.0`). However, `collect_set` over float/double columns returns duplicate `NaN` elements:

      -- Top-level NaN: HashSet compares boxed values with primitive equality (NaN != NaN)
      SELECT collect_set(v) FROM VALUES (double('NaN')), (double('NaN')) AS t(v);
      -- before: [NaN, NaN]   after: [NaN]
      
      -- Nested -0.0 / 0.0: GenericArrayData/GenericInternalRow hash -0.0 and 0.0 to
      -- different buckets, so they are not deduplicated
      SELECT collect_set(a) FROM VALUES (array(-0.0D)), (array(0.0D)) AS t(a);
      -- before: [[-0.0], [0.0]]   after: [[0.0]]
      SELECT collect_set(named_struct('a', v)) FROM VALUES (-0.0D), (0.0D) AS t(v);
      -- before: [{a:-0.0}, {a:0.0}]   after: [{a:0.0}]
      

      This is inconsistent with array_distinct , count(DISTINCT), grouping/join/window keys, and the array set operations - all of which treat NaNs as equal.
       
       

            Assignee:
            Eric Yang
            Reporter:
            Eric Yang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 40m
                1h 40m