You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1f0ac9d
Browse filesBrowse the repository at this point in the historyBrowse files
fix: fall back to Spark for regr_* aggregates until their merge matches Spark
The native merge of partial aggregates (variance_merge and covariance_merge
in welford.rs) orders its floating-point operations differently from Spark's
CentralMomentAgg and Covariance. For a variable that is constant at a value
binary floating point cannot represent exactly, merged from two or more
partial aggregates, m2 ends up around 1e-34 instead of 0, so Spark's exact
m2 == 0 degenerate-case checks never fire and regr_slope, regr_intercept,
regr_r2, regr_sxx, regr_syy and regr_sxy return wrong values.
Mark the five serdes behind these six functions Incompatible, so they fall
back to Spark by default as they did in 1.0.0 and run natively only with the
per-expression allowIncompatible config. The regr SQL-file tests opt in to
keep covering the native path, and a new test from the issue's reproducer
checks that the default plan falls back and matches Spark.
Closes#6423.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/expressions.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -126,12 +126,12 @@ The tables below list every Spark built-in expression with its current status.
126
126
|`regr_avgx`| ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
127
127
|`regr_avgy`| ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
128
128
|`regr_count`| ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
129
-
|`regr_intercept`| ✅ | Native ||
130
-
|`regr_r2`| ✅ | Native ||
131
-
|`regr_slope`| ✅ | Native ||
132
-
|`regr_sxx`| ✅ | Native ||
133
-
|`regr_sxy`| ✅ | Native ||
134
-
|`regr_syy`| ✅ | Native ||
129
+
|`regr_intercept`| ✅ | Native |Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrIntercept.allowIncompatible=true`|
130
+
|`regr_r2`| ✅ | Native |Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrR2.allowIncompatible=true`|
131
+
|`regr_slope`| ✅ | Native |Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrSlope.allowIncompatible=true`|
132
+
|`regr_sxx`| ✅ | Native |Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrReplacement.allowIncompatible=true` (Spark plans `regr_sxx` as `RegrReplacement`)|
133
+
|`regr_sxy`| ✅ | Native |Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrSXY.allowIncompatible=true`|
134
+
|`regr_syy`| ✅ | Native |Falls back by default because the native merge of partial aggregates differs from Spark ([#6423](https://github.com/apache/datafusion-comet/issues/6423)); the native path is opt-in via `spark.comet.expression.RegrReplacement.allowIncompatible=true` (Spark plans `regr_syy` as `RegrReplacement`)|
135
135
|`skewness`| 🔜 | — | Not yet implemented natively |
-- All functions take (y, x) and operate only on rows where BOTH y and x are non-null.
21
21
22
+
-- regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy and regr_sxy fall back to Spark by
23
+
-- default because their native merge of partial aggregates differs from Spark's
24
+
-- (https://github.com/apache/datafusion-comet/issues/6423). Opt in so the queries below cover
25
+
-- the native path. Spark plans regr_sxx and regr_syy as RegrReplacement.
@@ -1815,6 +1815,38 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
1815
1815
}
1816
1816
}
1817
1817
1818
+
test("regression aggregates fall back to Spark by default") {
1819
+
// https://github.com/apache/datafusion-comet/issues/6423: the native merge of partial
1820
+
// aggregates orders its floating-point operations differently from Spark. For a variable that
1821
+
// is constant at 0.1 and merged from two partial aggregates, m2 ends up around 1e-34 instead
1822
+
// of 0, Spark's degenerate-case checks never fire, and the native results are wrong. Until the
1823
+
// merge matches Spark, every regr aggregate has to fall back unless the user opts in.
1824
+
withTempPath { dir =>
1825
+
valpath= dir.getCanonicalPath
1826
+
spark
1827
+
.range(0, 6, 1, 2)
1828
+
.selectExpr("CAST(id AS DOUBLE) AS y", "0.1D AS x")
1829
+
.write
1830
+
.parquet(path)
1831
+
withParquetTable(path, "t") {
1832
+
// The wrong results need rows merged from more than one partial aggregate, one per file.
1833
+
assert(sql("SELECT * FROM t").rdd.getNumPartitions ==2)
0 commit comments