Skip to content

Commit 031070a

Browse files
authored
refactor: stop normalizing float divisors now that percentile_approx orders NaN as Spark does (#6866)
* refactor: stop normalizing float divisors now that percentile_approx orders NaN as Spark does CometExecRule wrapped the divisor of a FLOAT or DOUBLE Divide or Remainder in a Project or Filter in NormalizeNaNAndZero. Comparisons don't need it, since the native comparison builder follows Spark's ordering, including the check for a zero divisor. It stayed only because it also canonicalized a NaN quotient: 1.0D / (-d) of a NaN d is a NaN with the sign bit set, and percentile_approx ordered that NaN below every other value (#6519). #6829 fixed percentile_approx, so the rule can go. nan_divisor.sql keeps its percentile_approx query, which now checks the sign-bit NaN quotient directly, along with the hash and extrema queries for the earlier consumers. * test: regenerate TPC-DS plans without the divisor normalization Without NormalizeNaNAndZero around float divisors, the 11 approved plans that hold a float division in a Project or Filter count one native expression fewer, and q78 no longer lists knownfloatingpointnormalized and normalizenanandzero among its dispatched expressions. Regenerated on Spark 3.4, 3.5, 4.0, 4.1 and 4.2; 4.1 and 4.2 need no override.
1 parent 16dcf84 commit 031070a

13 files changed

Lines changed: 18 additions & 57 deletions

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala‎

Lines changed: 3 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,8 @@ import scala.collection.mutable.ListBuffer
2525
import scala.jdk.CollectionConverters._
2626

2727
import org.apache.spark.sql.SparkSession
28-
import org.apache.spark.sql.catalyst.expressions.{Divide, DoubleLiteral, Expression, FloatLiteral, KnownFloatingPointNormalized, NamedExpression, Remainder, SortOrder}
28+
import org.apache.spark.sql.catalyst.expressions.{Expression, SortOrder}
2929
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateMode, Final, Partial, PartialMerge}
30-
import org.apache.spark.sql.catalyst.optimizer.NormalizeNaNAndZero
3130
import org.apache.spark.sql.catalyst.rules.Rule
3231
import org.apache.spark.sql.catalyst.trees.TreeNodeTag
3332
import org.apache.spark.sql.catalyst.util.sideBySide
@@ -937,41 +936,6 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
937936
newPlan
938937
}
939938

940-
/**
941-
* Wraps the divisor of a floating-point `Divide` or `Remainder` in a Project or Filter in
942-
* `NormalizeNaNAndZero`. Native comparisons already follow Spark's ordering, including the
943-
* check for a zero divisor, so the quotient does not need this. It also canonicalizes a NaN
944-
* divisor, though, and with it the NaN quotient: `1.0D / (-d)` of a NaN `d` is otherwise a NaN
945-
* with the sign bit set, which `percentile_approx` still orders below every other value
946-
* (https://github.com/apache/datafusion-comet/issues/6519). Remove this once that is fixed.
947-
*/
948-
private def normalizeDivisors(plan: SparkPlan): SparkPlan = {
949-
def normalize(expr: Expression): Expression = expr.transformUp {
950-
case Divide(left, right, evalMode) => Divide(left, normalizeNaNAndZero(right), evalMode)
951-
case Remainder(left, right, evalMode) =>
952-
Remainder(left, normalizeNaNAndZero(right), evalMode)
953-
}
954-
plan.transformUp {
955-
case p: ProjectExec =>
956-
ProjectExec(p.projectList.map(normalize(_).asInstanceOf[NamedExpression]), p.child)
957-
case f: FilterExec => FilterExec(normalize(f.condition), f.child)
958-
}
959-
}
960-
961-
private def normalizeNaNAndZero(expr: Expression): Expression = {
962-
expr match {
963-
case _: KnownFloatingPointNormalized => expr
964-
case FloatLiteral(f) if !f.isNaN && !f.equals(-0.0f) => expr
965-
case DoubleLiteral(d) if !d.isNaN && !d.equals(-0.0d) => expr
966-
case _ =>
967-
expr.dataType match {
968-
case _: FloatType | _: DoubleType =>
969-
KnownFloatingPointNormalized(NormalizeNaNAndZero(expr))
970-
case _ => expr
971-
}
972-
}
973-
}
974-
975939
private def _apply(plan: SparkPlan): SparkPlan = {
976940
// We shouldn't transform Spark query plan if Comet is not loaded.
977941
if (!isCometLoaded(conf)) {
@@ -992,17 +956,15 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
992956
plan
993957
}
994958
} else {
995-
val normalizedPlan = normalizeDivisors(plan)
996-
997959
// The rewrite removes the sorts of each join it converts, so put back any sort a kept
998960
// operator above still needs. This runs before transform(), which converts the sorts it
999961
// adds like any other SortExec.
1000962
val planWithJoinRewritten = if (CometConf.COMET_FORCE_SHJ.get()) {
1001-
RewriteJoin.restoreRequiredOrdering(normalizedPlan.transformUp { case p =>
963+
RewriteJoin.restoreRequiredOrdering(plan.transformUp { case p =>
1002964
RewriteJoin.rewrite(p, conf)
1003965
})
1004966
} else {
1005-
normalizedPlan
967+
plan
1006968
}
1007969

1008970
// Tag Partial aggregates that must not be converted to Comet because a

‎spark/src/test/resources/sql-tests/expressions/math/nan_divisor.sql‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,8 @@ query
4343
SELECT max(q), min(q)
4444
FROM (SELECT 1.0D / (-d) AS q FROM nan_divisor UNION ALL SELECT 0.0D AS q FROM nan_divisor)
4545

46-
-- percentile_approx orders a NaN with the sign bit set below every other value
47-
-- (https://github.com/apache/datafusion-comet/issues/6519). Until that is fixed, Comet keeps
48-
-- normalizing the divisor, which makes the quotient a canonical NaN.
46+
-- The quotient keeps the sign bit of its NaN, and percentile_approx orders that NaN above every
47+
-- other value, as Spark does.
4948
query
5049
SELECT percentile_approx(q, 1.0D), percentile_approx(q, 0.0D)
5150
FROM (SELECT /*+ COALESCE(1) */ q

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark3_5/q83/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,4 +101,4 @@ CometColumnarToRow
101101
+- CometFilter
102102
+- CometNativeScan parquet spark_catalog.default.date_dim
103103

104-
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch.
104+
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q83.ansi/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,4 +101,4 @@ CometColumnarToRow
101101
+- CometFilter
102102
+- CometNativeScan parquet spark_catalog.default.date_dim
103103

104-
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch.
104+
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q21/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,4 @@ CometColumnarToRow
2727
+- CometFilter
2828
+- CometNativeScan parquet spark_catalog.default.date_dim
2929

30-
Comet accelerated 24 out of 24 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 13 native, 0 codegen dispatch.
30+
Comet accelerated 24 out of 24 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q34/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,4 +37,4 @@ CometColumnarToRow
3737
+- CometFilter
3838
+- CometNativeScan parquet spark_catalog.default.customer
3939

40-
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch.
40+
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 14 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q39a/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,4 +60,4 @@ CometColumnarToRow
6060
+- CometFilter
6161
+- CometNativeScan parquet spark_catalog.default.date_dim
6262

63-
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.
63+
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 10 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q39b/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,4 +60,4 @@ CometColumnarToRow
6060
+- CometFilter
6161
+- CometNativeScan parquet spark_catalog.default.date_dim
6262

63-
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.
63+
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 10 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q73/extended.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,4 +37,4 @@ CometColumnarToRow
3737
+- CometFilter
3838
+- CometNativeScan parquet spark_catalog.default.customer
3939

40-
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch.
40+
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 14 native, 0 codegen dispatch.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q78/extended.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
CometColumnarToRow
22
+- CometTakeOrderedAndProject [COMET-INFO: JVM codegen dispatcher: add, cast, coalesce, divide, round]
3-
+- CometProject [COMET-INFO: JVM codegen dispatcher: add, cast, coalesce, divide, knownfloatingpointnormalized, normalizenanandzero, round]
3+
+- CometProject [COMET-INFO: JVM codegen dispatcher: add, cast, coalesce, divide, round]
44
+- CometSortMergeJoin
55
:- CometProject
66
: +- CometSortMergeJoin
@@ -76,4 +76,4 @@ CometColumnarToRow
7676
+- CometFilter
7777
+- CometNativeScan parquet spark_catalog.default.date_dim
7878

79-
Comet accelerated 65 out of 65 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 7 codegen dispatch.
79+
Comet accelerated 65 out of 65 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 5 codegen dispatch.

0 commit comments

Comments
 (0)