You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 031070a
Browse filesBrowse the repository at this point in the historyBrowse files
refactor: stop normalizing float divisors now that percentile_approx orders NaN as Spark does (#6866)
* refactor: stop normalizing float divisors now that percentile_approx orders NaN as Spark does
CometExecRule wrapped the divisor of a FLOAT or DOUBLE Divide or Remainder in a Project or Filter in NormalizeNaNAndZero. Comparisons don't need it, since the native comparison builder follows Spark's ordering, including the check for a zero divisor. It stayed only because it also canonicalized a NaN quotient: 1.0D / (-d) of a NaN d is a NaN with the sign bit set, and percentile_approx ordered that NaN below every other value (#6519). #6829 fixed percentile_approx, so the rule can go.
nan_divisor.sql keeps its percentile_approx query, which now checks the sign-bit NaN quotient directly, along with the hash and extrema queries for the earlier consumers.
* test: regenerate TPC-DS plans without the divisor normalization
Without NormalizeNaNAndZero around float divisors, the 11 approved plans that hold a float division in a Project or Filter count one native expression fewer, and q78 no longer lists knownfloatingpointnormalized and normalizenanandzero among its dispatched expressions. Regenerated on Spark 3.4, 3.5, 4.0, 4.1 and 4.2; 4.1 and 4.2 need no override.
@@ -937,41 +936,6 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
937
936
newPlan
938
937
}
939
938
940
-
/**
941
-
* Wraps the divisor of a floating-point `Divide` or `Remainder` in a Project or Filter in
942
-
* `NormalizeNaNAndZero`. Native comparisons already follow Spark's ordering, including the
943
-
* check for a zero divisor, so the quotient does not need this. It also canonicalizes a NaN
944
-
* divisor, though, and with it the NaN quotient: `1.0D / (-d)` of a NaN `d` is otherwise a NaN
945
-
* with the sign bit set, which `percentile_approx` still orders below every other value
946
-
* (https://github.com/apache/datafusion-comet/issues/6519). Remove this once that is fixed.
Copy file name to clipboardExpand all lines: spark/src/test/resources/sql-tests/expressions/math/nan_divisor.sql
+2-3Lines changed: 2 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -43,9 +43,8 @@ query
43
43
SELECTmax(q), min(q)
44
44
FROM (SELECT1.0D / (-d) AS q FROM nan_divisor UNION ALLSELECT0.0D AS q FROM nan_divisor)
45
45
46
-
-- percentile_approx orders a NaN with the sign bit set below every other value
47
-
-- (https://github.com/apache/datafusion-comet/issues/6519). Until that is fixed, Comet keeps
48
-
-- normalizing the divisor, which makes the quotient a canonical NaN.
46
+
-- The quotient keeps the sign bit of its NaN, and percentile_approx orders that NaN above every
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch.
104
+
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch.
104
+
Comet accelerated 52 out of 52 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.
Comet accelerated 24 out of 24 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 13 native, 0 codegen dispatch.
30
+
Comet accelerated 24 out of 24 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 0 codegen dispatch.
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch.
40
+
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 14 native, 0 codegen dispatch.
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.
63
+
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 10 native, 0 codegen dispatch.
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 11 native, 0 codegen dispatch.
63
+
Comet accelerated 46 out of 46 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 10 native, 0 codegen dispatch.
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 15 native, 0 codegen dispatch.
40
+
Comet accelerated 33 out of 33 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 14 native, 0 codegen dispatch.
Comet accelerated 65 out of 65 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 7 codegen dispatch.
79
+
Comet accelerated 65 out of 65 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 12 native, 5 codegen dispatch.
0 commit comments