Skip to content

Commit 81d637b

Browse files
authored
feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates (#4775)
* feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates Add native support for the six simple linear regression aggregates that previously fell back to Spark. regr_avgx, regr_avgy and regr_count were already accelerated because Spark rewrites them to Average/Count. The native accumulators are composed from Comet's existing Spark-compatible covariance and variance accumulators so the partial aggregation state matches the buffer layout Spark's planner expects between partial and final stages: RegrReplacement (regr_sxx/regr_syy) -> 3 fields, Covariance (regr_sxy) -> 4, PearsonCorrelation (regr_r2) -> 6, and the slope/intercept composite -> 7. regr_r2 matches Spark's behavior of returning 1.0 when the dependent variable is constant but the independent variable varies (a perfect horizontal fit), which differs from DataFusion's regr_r2. * fix: match Spark's per-version regr_slope/intercept/r2 semantics The regr aggregates diverged from Spark in three ways that surfaced as CI failures across Spark versions: - regr_r2 degenerate cases were inverted. Spark 3.4/3.5/4.0 return null when the dependent variable is constant and 1.0 when the independent variable is constant; Comet had these swapped. - Spark 4.1 swapped that degenerate handling again (constant dependent -> 1.0, constant independent -> null). Route the behaviour through a new r2_constant_dependent_is_perfect_fit proto flag set from isSpark41Plus. - regr_slope/regr_intercept compute VariancePop(x) over both-non-null pairs on Spark 3.5+, but over every x-non-null row on Spark 3.4. Route this through a new filter_var_by_pair_nulls proto flag set from isSpark35Plus. Also evaluate regr_r2 as corr = ck / sqrt(m2_y * m2_x); corr * corr to mirror Spark's exact float rounding, so the golden-file postgres aggregates tests match bit-for-bit. * fix: regr_r2 degenerate-case swap (SPARK-55969) applies from Spark 3.5, not 4.1 SPARK-55969 swapped which regr_r2 degenerate case returns NULL and which returns 1.0. It shipped in Spark 3.5.9, 4.0.3 and 4.1.0, so every Spark version Comet builds against except 3.4 has the new behavior. Comet was gating it on isSpark41Plus, which made regr_r2 disagree with Spark on the 3.5 and 4.0 profiles.
1 parent 9cac716 commit 81d637b

8 files changed

Lines changed: 963 additions & 37 deletions

File tree

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -121,12 +121,12 @@ The tables below list every Spark built-in expression with its current status.
121121
| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
122122
| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
123123
| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
124-
| `regr_intercept` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop` accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
125-
| `regr_r2` | 🔜 | — | Falls back; can reuse the `corr` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
126-
| `regr_slope` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop` accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
127-
| `regr_sxx` | 🔜 | — | Falls back; can reuse `var_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
128-
| `regr_sxy` | 🔜 | — | Falls back; can reuse `covar_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
129-
| `regr_syy` | 🔜 | — | Falls back; can reuse `var_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
124+
| `regr_intercept` | ✅ | Native | |
125+
| `regr_r2` | ✅ | Native | |
126+
| `regr_slope` | ✅ | Native | |
127+
| `regr_sxx` | ✅ | Native | |
128+
| `regr_sxy` | ✅ | Native | |
129+
| `regr_syy` | ✅ | Native | |
130130
| `skewness` | 🔜 | — | Not yet implemented natively |
131131
| `some` | ✅ | — | |
132132
| `std` | ✅ | Native | |

‎native/core/src/execution/planner.rs‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,9 @@ use datafusion_comet_proto::{
147147
use datafusion_comet_spark_expr::{
148148
jvm_udf::JvmScalarUdfExpr, ApproxPercentile, ArrayInsert, Avg, AvgDecimal, Cast, CheckOverflow,
149149
Correlation, Covariance, CreateNamedStruct, DecimalRescaleCheckOverflow, GetArrayStructFields,
150-
GetStructField, HllPlusPlus, IfExpr, ListExtract, NormalizeNaNAndZero, SparkCastOptions,
151-
Stddev, SumDecimal, ToJson, UnboundColumn, Variance, WideDecimalBinaryExpr, WideDecimalOp,
150+
GetStructField, HllPlusPlus, IfExpr, ListExtract, NormalizeNaNAndZero, Regr, RegrType,
151+
SparkCastOptions, Stddev, SumDecimal, ToJson, UnboundColumn, Variance, WideDecimalBinaryExpr,
152+
WideDecimalOp,
152153
};
153154
use itertools::Itertools;
154155
use jni::objects::{Global, JObject};
@@ -3029,6 +3030,29 @@ impl PhysicalPlanner {
30293030
));
30303031
Self::create_aggr_func_expr("correlation", schema, vec![child1, child2], func)
30313032
}
3033+
AggExprStruct::Regr(expr) => {
3034+
let child1 =
3035+
self.create_expr(expr.child1.as_ref().unwrap(), Arc::clone(&schema))?;
3036+
let child2 =
3037+
self.create_expr(expr.child2.as_ref().unwrap(), Arc::clone(&schema))?;
3038+
let (regr_type, name) = match expr.regr_type() {
3039+
spark_expression::regr::RegrType::Slope => (RegrType::Slope, "regr_slope"),
3040+
spark_expression::regr::RegrType::Intercept => {
3041+
(RegrType::Intercept, "regr_intercept")
3042+
}
3043+
spark_expression::regr::RegrType::R2 => (RegrType::R2, "regr_r2"),
3044+
spark_expression::regr::RegrType::Sxx => (RegrType::SXX, "regr_sxx"),
3045+
spark_expression::regr::RegrType::Syy => (RegrType::SYY, "regr_syy"),
3046+
spark_expression::regr::RegrType::Sxy => (RegrType::SXY, "regr_sxy"),
3047+
};
3048+
let func = AggregateUDF::new_from_impl(Regr::new(
3049+
regr_type,
3050+
name,
3051+
expr.filter_var_by_pair_nulls,
3052+
expr.r2_constant_dependent_is_perfect_fit,
3053+
));
3054+
Self::create_aggr_func_expr(name, schema, vec![child1, child2], func)
3055+
}
30323056
AggExprStruct::Percentile(expr) => {
30333057
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
30343058
let percentile =

‎native/proto/src/proto/expr.proto‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,7 @@ message AggExpr {
162162
ApproxPercentile approxPercentile = 19;
163163
HllPlusPlus hllpp = 20;
164164
CollectList collectList = 21;
165+
Regr regr = 22;
165166
}
166167

167168
// Optional filter expression for SQL FILTER (WHERE ...) clause.
@@ -263,6 +264,36 @@ message Correlation {
263264
DataType datatype = 4;
264265
}
265266

267+
// Simple linear regression aggregates (regr_slope, regr_intercept, regr_r2,
268+
// regr_sxx, regr_syy, regr_sxy). child1 is the dependent variable (y) and
269+
// child2 is the independent variable (x).
270+
message Regr {
271+
enum RegrType {
272+
SLOPE = 0;
273+
INTERCEPT = 1;
274+
R2 = 2;
275+
SXX = 3;
276+
SYY = 4;
277+
SXY = 5;
278+
}
279+
Expr child1 = 1;
280+
Expr child2 = 2;
281+
RegrType regr_type = 3;
282+
DataType datatype = 4;
283+
// Only consulted for SLOPE and INTERCEPT. When true (Spark 3.5+), VariancePop(x)
284+
// is computed only over rows where both y and x are non-null. When false
285+
// (Spark 3.4), VariancePop(x) includes every row where x is non-null even if y
286+
// is null, matching the pre-fix Spark 3.4 semantics.
287+
bool filter_var_by_pair_nulls = 5;
288+
// Only consulted for R2. Spark swapped the degenerate-case handling of
289+
// regr_r2. When true, a constant dependent variable (m2(y) = 0) returns 1.0
290+
// and a constant independent variable (m2(x) = 0) returns null. When false
291+
// those two cases are reversed. The swap is present in the Spark versions
292+
// Comet builds against for 3.5 and later (3.5.9, 4.0.3+, 4.1, 4.2) but not
293+
// in 3.4.
294+
bool r2_constant_dependent_is_perfect_fit = 6;
295+
}
296+
266297
message Percentile {
267298
Expr child = 1;
268299
// Single percentile in [0.0, 1.0] as a literal double expression.

‎native/spark-expr/src/agg_funcs/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ mod hll_plus_plus;
2424
mod hll_plus_plus_const;
2525
mod percentile;
2626
mod quantile_summaries;
27+
mod regr;
2728
mod stddev;
2829
mod sum_decimal;
2930
mod sum_int;
@@ -37,6 +38,7 @@ pub use correlation::Correlation;
3738
pub use covariance::Covariance;
3839
pub use hll_plus_plus::{hllpp_precision, HllPlusPlus};
3940
pub use percentile::SparkPercentile;
41+
pub use regr::{Regr, RegrType};
4042
pub use stddev::Stddev;
4143
pub use sum_decimal::SumDecimal;
4244
pub use sum_int::SumInteger;

0 commit comments

Comments
 (0)