Mismatched min value for -0.0 and 0.0

Describe the bug

test case:

  test("SUM, MIN, MAX, AVG for NaN, -0.0 and 0.0") {
    // NaN should be grouped together, and -0.0 and 0.0 should also be grouped together
    Seq(true, false).foreach { dictionaryEnabled =>
      val data: Seq[(Float, Int)] = Seq(
        (Float.NaN, 1),
        (-0.0.asInstanceOf[Float], 2),
        (0.0.asInstanceOf[Float], 3),
        (Float.NaN, 4))
      withSQLConf(CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true") {
        withParquetTable(data, "tbl", dictionaryEnabled) {
          checkSparkAnswer("SELECT SUM(_2), MIN(_2), MAX(_2), _1 FROM tbl GROUP BY _1")
          checkSparkAnswer("SELECT MIN(_1), MAX(_1), MIN(_2), MAX(_2) FROM tbl")
          checkSparkAnswer("SELECT AVG(_2), _1 FROM tbl GROUP BY _1")
          checkSparkAnswer("SELECT AVG(_1), AVG(_2) FROM tbl")
        }
      }
    }
  }

error:

== Results ==
!== Correct Answer - 1 ==                                      == Spark Answer - 1 ==
 struct<min(_1):float,max(_1):float,min(_2):int,max(_2):int>   struct<min(_1):float,max(_1):float,min(_2):int,max(_2):int>
![0.0,NaN,1,4]                                                 [-0.0,NaN,1,4]

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

@wForget

Thanks @wForget
might be related to apache/datafusion#11051

Possibly relevant code:

datafusion-comet/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala

Lines 600 to 612 in 03c0626

    
           private def normalizeNaNAndZero(expr: Expression): Expression = { 
        
             expr match { 
        
               case _: KnownFloatingPointNormalized => expr 
        
               case FloatLiteral(f) if !f.equals(-0.0f) => expr 
        
               case DoubleLiteral(d) if !d.equals(-0.0d) => expr 
        
               case _ => 
        
                 expr.dataType match { 
        
                   case _: FloatType | _: DoubleType => 
        
                     KnownFloatingPointNormalized(NormalizeNaNAndZero(expr)) 
        
                   case _ => expr 
        
                 } 
        
             } 
        
           }

made a quick investigation, it fails on MIN if shuffle enabled. Let me check if quick fix available

I dont see normalized code is called, checking

but Spark and DF has diff vision on min

scala> spark.sql("select min(a) from (select cast(-0.0 as float) a union all select cast(0.0 as float))").show(false)
+------+
|min(a)|
+------+
|0.0   |
+------+

> select min(a) from (select 0.0::float a union all select -0.0::float);
+--------+
| min(a) |
+--------+
| -0.0   |
+--------+

I dont see normalized code is called, checking

datafusion-comet/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala

Lines 575 to 578 in 03c0626

    
           // However, one exception is for comparison operators. Spark does not normalize NaN and zero 
        
           // because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the 
        
           // comparison functions in arrow-rs do not normalize NaN and zero. So we need to normalize NaN 
        
           // and zero for comparison operators in Comet.

As mentioned in the comment, SQLOrderingUtil.compareFloats is used in vanilla Spark to compare float types .

We might be able to simply fix this by wrapping NormalizeNaNAndZero in the min/max agg for float types in comet.

We might be able to simply fix this by wrapping NormalizeNaNAndZero in the min/max agg for float types in comet.

This may not be a solution. Spark only does normalization when sorting, so returning -0.0 or 0.0 is unstable.

  test("test min on -0.0 and 0.0") {
      withSQLConf(
        CometConf.COMET_SPARK_TO_ARROW_ENABLED.key -> "true",
        CometConf.COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST.key -> "LocalTableScan",
        CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true"
      ) {
        // succeed with -0.0
        checkSparkAnswer("SELECT min(col1) FROM VALUES (-0.0f), (0.0f)")
        // failed, spark returns 0.0, comet returns -0.0
        checkSparkAnswer("SELECT min(col1) FROM VALUES (0.0f), (-0.0f)")
      }
  }

This may not be a solution. Spark only does normalization when sorting, so returning -0.0 or 0.0 is unstable.

          checkSparkAnswer("select min(_1) from (SELECT _1 FROM tbl order by _1 desc)")

passed

whereas

          checkSparkAnswer("select min(_1) from (SELECT _1 FROM tbl order by _1 asc)")

failed

I'll check little bit more today

Navigation Menu

Repository navigation

Mismatched `min` value for -0.0 and 0.0 #2448

Describe the bug

Steps to reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	private def normalizeNaNAndZero(expr: Expression): Expression = {
	expr match {
	case _: KnownFloatingPointNormalized => expr
	case FloatLiteral(f) if !f.equals(-0.0f) => expr
	case DoubleLiteral(d) if !d.equals(-0.0d) => expr
	case _ =>
	expr.dataType match {
	case _: FloatType \| _: DoubleType =>
	KnownFloatingPointNormalized(NormalizeNaNAndZero(expr))
	case _ => expr
	}
	}
	}

	// However, one exception is for comparison operators. Spark does not normalize NaN and zero
	// because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the
	// comparison functions in arrow-rs do not normalize NaN and zero. So we need to normalize NaN
	// and zero for comparison operators in Comet.

Mismatched min value for -0.0 and 0.0 #2448

Description

Describe the bug

Steps to reproduce

Expected behavior

Additional context

Activity

comphead commented on Sep 24, 2025

wForget commented on Sep 26, 2025

comphead commented on Sep 26, 2025

comphead commented on Sep 27, 2025

wForget commented on Sep 28, 2025

wForget commented on Sep 29, 2025

comphead commented on Sep 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Mismatched `min` value for -0.0 and 0.0 #2448