Skip to content

Commit 1f0ac9d

Browse files
committed
fix: fall back to Spark for regr_* aggregates until their merge matches Spark
The native merge of partial aggregates (variance_merge and covariance_merge in welford.rs) orders its floating-point operations differently from Spark's CentralMomentAgg and Covariance. For a variable that is constant at a value binary floating point cannot represent exactly, merged from two or more partial aggregates, m2 ends up around 1e-34 instead of 0, so Spark's exact m2 == 0 degenerate-case checks never fire and regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy and regr_sxy return wrong values. Mark the five serdes behind these six functions Incompatible, so they fall back to Spark by default as they did in 1.0.0 and run natively only with the per-expression allowIncompatible config. The regr SQL-file tests opt in to keep covering the native path, and a new test from the issue's reproducer checks that the default plan falls back and matches Spark. Closes #6423.
1 parent 8e4bded commit 1f0ac9d

4 files changed

Lines changed: 86 additions & 18 deletions

File tree

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -126,12 +126,12 @@ The tables below list every Spark built-in expression with its current status.
126126
| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
127127
| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
128128
| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
129-
| `regr_intercept` | ✅ | Native | |
130-
| `regr_r2` | ✅ | Native | |
131-
| `regr_slope` | ✅ | Native | |
132-
| `regr_sxx` | ✅ | Native | |
133-
| `regr_sxy` | ✅ | Native | |
134-
| `regr_syy` | ✅ | Native | |
129+
| `regr_intercept` | ✅ | Native | Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrIntercept.allowIncompatible=true` |
130+
| `regr_r2` | ✅ | Native | Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrR2.allowIncompatible=true` |
131+
| `regr_slope` | ✅ | Native | Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrSlope.allowIncompatible=true` |
132+
| `regr_sxx` | ✅ | Native | Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrReplacement.allowIncompatible=true` (Spark plans `regr_sxx` as `RegrReplacement`) |
133+
| `regr_sxy` | ✅ | Native | Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrSXY.allowIncompatible=true` |
134+
| `regr_syy` | ✅ | Native | Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrReplacement.allowIncompatible=true` (Spark plans `regr_syy` as `RegrReplacement`) |
135135
| `skewness` | 🔜 | — | Not yet implemented natively |
136136
| `some` | ✅ | — | |
137137
| `std` | ✅ | Native | |

‎spark/src/main/scala/org/apache/comet/serde/aggregates.scala‎

Lines changed: 37 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ package org.apache.comet.serde
2222
import scala.jdk.CollectionConverters._
2323

2424
import org.apache.spark.sql.catalyst.expressions.{Attribute, Cast, Expression, Literal}
25-
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, ApproximatePercentile, Average, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAggregate, CentralMomentAgg, CollectList, CollectSet, Complete, Corr, Count, Covariance, CovPopulation, CovSample, First, HyperLogLogPlusPlus, Last, Max, MaxBy, MaxMinBy, Min, MinBy, Mode, Partial, Percentile, RegrIntercept, RegrR2, RegrReplacement, RegrSlope, RegrSXY, StddevPop, StddevSamp, Sum, VariancePop, VarianceSamp}
25+
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, AggregateFunction, ApproximatePercentile, Average, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAggregate, CentralMomentAgg, CollectList, CollectSet, Complete, Corr, Count, Covariance, CovPopulation, CovSample, First, HyperLogLogPlusPlus, Last, Max, MaxBy, MaxMinBy, Min, MinBy, Mode, Partial, Percentile, RegrIntercept, RegrR2, RegrReplacement, RegrSlope, RegrSXY, StddevPop, StddevSamp, Sum, VariancePop, VarianceSamp}
2626
import org.apache.spark.sql.catalyst.util.ArrayData
2727
import org.apache.spark.sql.comet.CometExecUtils
2828
import org.apache.spark.sql.internal.SQLConf
@@ -929,7 +929,27 @@ private[comet] object RegrSparkVersions {
929929
* variable (y) and `child2` is the independent variable (x), matching the native accumulator's
930930
* `regr_*(y, x)` convention.
931931
*/
932-
trait CometRegrBase {
932+
trait CometRegrBase[T <: AggregateFunction] extends CometAggregateExpressionSerde[T] {
933+
934+
/** The SQL function or functions this serde implements, named in the incompatibility note. */
935+
protected def sqlFunctions: String
936+
937+
// The native merge (`variance_merge` and `covariance_merge` in welford.rs) orders its
938+
// floating-point operations differently from Spark's CentralMomentAgg and Covariance. Merging
939+
// the first partial buffer into the zero-initialized final buffer can leave a one-ULP error on
940+
// the mean, so a constant variable ends up with a tiny non-zero m2 and Spark's exact `m2 == 0`
941+
// degenerate-case checks never fire. Porting Spark's merge order would make these Compatible.
942+
private def mergeOrderReason: String =
943+
s"Comet merges the partial aggregates of $sqlFunctions in a different floating-point " +
944+
"operation order from Spark. When a group's rows come from more than one partial " +
945+
"aggregate and a variable is constant at a value that binary floating point cannot " +
946+
"represent exactly, such as 0.1, Comet returns a wrong value where Spark returns NULL, " +
947+
"0.0 or 1.0 (https://github.com/apache/datafusion-comet/issues/6423)"
948+
949+
override def getIncompatibleReasons(): Seq[String] = Seq(mergeOrderReason)
950+
951+
override def getSupportLevel(expr: T): SupportLevel = Incompatible(Some(mergeOrderReason))
952+
933953
def convertRegr(
934954
aggExpr: AggregateExpression,
935955
regrType: ExprOuterClass.Regr.RegrType,
@@ -966,7 +986,9 @@ trait CometRegrBase {
966986
}
967987
}
968988

969-
object CometRegrSlope extends CometAggregateExpressionSerde[RegrSlope] with CometRegrBase {
989+
object CometRegrSlope extends CometRegrBase[RegrSlope] {
990+
override protected def sqlFunctions: String = "`regr_slope`"
991+
970992
override def convert(
971993
aggExpr: AggregateExpression,
972994
expr: RegrSlope,
@@ -982,9 +1004,9 @@ object CometRegrSlope extends CometAggregateExpressionSerde[RegrSlope] with Come
9821004
binding)
9831005
}
9841006

985-
object CometRegrIntercept
986-
extends CometAggregateExpressionSerde[RegrIntercept]
987-
with CometRegrBase {
1007+
object CometRegrIntercept extends CometRegrBase[RegrIntercept] {
1008+
override protected def sqlFunctions: String = "`regr_intercept`"
1009+
9881010
override def convert(
9891011
aggExpr: AggregateExpression,
9901012
expr: RegrIntercept,
@@ -1000,7 +1022,9 @@ object CometRegrIntercept
10001022
binding)
10011023
}
10021024

1003-
object CometRegrR2 extends CometAggregateExpressionSerde[RegrR2] with CometRegrBase {
1025+
object CometRegrR2 extends CometRegrBase[RegrR2] {
1026+
override protected def sqlFunctions: String = "`regr_r2`"
1027+
10041028
override def convert(
10051029
aggExpr: AggregateExpression,
10061030
expr: RegrR2,
@@ -1010,7 +1034,9 @@ object CometRegrR2 extends CometAggregateExpressionSerde[RegrR2] with CometRegrB
10101034
convertRegr(aggExpr, ExprOuterClass.Regr.RegrType.R2, expr.y, expr.x, inputs, binding)
10111035
}
10121036

1013-
object CometRegrSXY extends CometAggregateExpressionSerde[RegrSXY] with CometRegrBase {
1037+
object CometRegrSXY extends CometRegrBase[RegrSXY] {
1038+
override protected def sqlFunctions: String = "`regr_sxy`"
1039+
10141040
override def convert(
10151041
aggExpr: AggregateExpression,
10161042
expr: RegrSXY,
@@ -1027,9 +1053,9 @@ object CometRegrSXY extends CometAggregateExpressionSerde[RegrSXY] with CometReg
10271053
* deviations) of its single child. We serialize it as the `SXX` regression statistic with the
10281054
* child duplicated, since `regr_sxx(c, c) = m2(c)`.
10291055
*/
1030-
object CometRegrReplacement
1031-
extends CometAggregateExpressionSerde[RegrReplacement]
1032-
with CometRegrBase {
1056+
object CometRegrReplacement extends CometRegrBase[RegrReplacement] {
1057+
override protected def sqlFunctions: String = "`regr_sxx` and `regr_syy`"
1058+
10331059
override def convert(
10341060
aggExpr: AggregateExpression,
10351061
expr: RegrReplacement,

‎spark/src/test/resources/sql-tests/expressions/aggregate/regr.sql‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,16 @@
1919
-- regr_avgy, regr_sxx, regr_syy, regr_sxy, regr_slope, regr_intercept, regr_r2.
2020
-- All functions take (y, x) and operate only on rows where BOTH y and x are non-null.
2121

22+
-- regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy and regr_sxy fall back to Spark by
23+
-- default because their native merge of partial aggregates differs from Spark's
24+
-- (https://github.com/apache/datafusion-comet/issues/6423). Opt in so the queries below cover
25+
-- the native path. Spark plans regr_sxx and regr_syy as RegrReplacement.
26+
-- Config: spark.comet.expression.RegrSlope.allowIncompatible=true
27+
-- Config: spark.comet.expression.RegrIntercept.allowIncompatible=true
28+
-- Config: spark.comet.expression.RegrR2.allowIncompatible=true
29+
-- Config: spark.comet.expression.RegrSXY.allowIncompatible=true
30+
-- Config: spark.comet.expression.RegrReplacement.allowIncompatible=true
31+
2232
statement
2333
CREATE TABLE test_regr(y double, x double, grp string) USING parquet
2434

‎spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ import org.apache.spark.{CometListenerBusUtils, SparkConf}
2828
import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd}
2929
import org.apache.spark.sql.{CometTestBase, DataFrame, Row}
3030
import org.apache.spark.sql.catalyst.expressions.Cast
31-
import org.apache.spark.sql.catalyst.expressions.aggregate.{Final, Partial, PartialMerge}
31+
import org.apache.spark.sql.catalyst.expressions.aggregate.{Final, Partial, PartialMerge, RegrIntercept, RegrR2, RegrReplacement, RegrSlope, RegrSXY}
3232
import org.apache.spark.sql.catalyst.optimizer.EliminateSorts
3333
import org.apache.spark.sql.catalyst.plans.physical.{HashPartitioning, RangePartitioning}
3434
import org.apache.spark.sql.comet.{CometFilterExec, CometHashAggregateExec, CometNativeExec, CometProjectExec}
@@ -1815,6 +1815,38 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
18151815
}
18161816
}
18171817

1818+
test("regression aggregates fall back to Spark by default") {
1819+
// https://github.com/apache/datafusion-comet/issues/6423: the native merge of partial
1820+
// aggregates orders its floating-point operations differently from Spark. For a variable that
1821+
// is constant at 0.1 and merged from two partial aggregates, m2 ends up around 1e-34 instead
1822+
// of 0, Spark's degenerate-case checks never fire, and the native results are wrong. Until the
1823+
// merge matches Spark, every regr aggregate has to fall back unless the user opts in.
1824+
withTempPath { dir =>
1825+
val path = dir.getCanonicalPath
1826+
spark
1827+
.range(0, 6, 1, 2)
1828+
.selectExpr("CAST(id AS DOUBLE) AS y", "0.1D AS x")
1829+
.write
1830+
.parquet(path)
1831+
withParquetTable(path, "t") {
1832+
// The wrong results need rows merged from more than one partial aggregate, one per file.
1833+
assert(sql("SELECT * FROM t").rdd.getNumPartitions == 2)
1834+
val optInKeys = Seq[Class[_]](
1835+
classOf[RegrSlope],
1836+
classOf[RegrIntercept],
1837+
classOf[RegrR2],
1838+
classOf[RegrSXY],
1839+
// regr_sxx and regr_syy
1840+
classOf[RegrReplacement]).map(cls => CometConf.getExprAllowIncompatConfigKey(cls))
1841+
checkSparkAnswerAndFallbackReasons(
1842+
"SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x), regr_sxx(y, x), " +
1843+
"regr_sxy(y, x), regr_r2(x, y), regr_syy(x, y) FROM t",
1844+
optInKeys.map(key => s"set $key=true").toSet +
1845+
"https://github.com/apache/datafusion-comet/issues/6423")
1846+
}
1847+
}
1848+
}
1849+
18181850
test("avg/sum overflow on decimal(38, _)") {
18191851
val table = "overflow_decimal_38"
18201852
withTable(table) {

0 commit comments

Comments
 (0)