Repository navigation
Commit 81d637b
authored
feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates (#4775)
* feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates
Add native support for the six simple linear regression aggregates that
previously fell back to Spark. regr_avgx, regr_avgy and regr_count were
already accelerated because Spark rewrites them to Average/Count.
The native accumulators are composed from Comet's existing Spark-compatible
covariance and variance accumulators so the partial aggregation state matches
the buffer layout Spark's planner expects between partial and final stages:
RegrReplacement (regr_sxx/regr_syy) -> 3 fields, Covariance (regr_sxy) -> 4,
PearsonCorrelation (regr_r2) -> 6, and the slope/intercept composite -> 7.
regr_r2 matches Spark's behavior of returning 1.0 when the dependent variable
is constant but the independent variable varies (a perfect horizontal fit),
which differs from DataFusion's regr_r2.
* fix: match Spark's per-version regr_slope/intercept/r2 semantics
The regr aggregates diverged from Spark in three ways that surfaced as CI
failures across Spark versions:
- regr_r2 degenerate cases were inverted. Spark 3.4/3.5/4.0 return null when
the dependent variable is constant and 1.0 when the independent variable is
constant; Comet had these swapped.
- Spark 4.1 swapped that degenerate handling again (constant dependent -> 1.0,
constant independent -> null). Route the behaviour through a new
r2_constant_dependent_is_perfect_fit proto flag set from isSpark41Plus.
- regr_slope/regr_intercept compute VariancePop(x) over both-non-null pairs on
Spark 3.5+, but over every x-non-null row on Spark 3.4. Route this through a
new filter_var_by_pair_nulls proto flag set from isSpark35Plus.
Also evaluate regr_r2 as corr = ck / sqrt(m2_y * m2_x); corr * corr to mirror
Spark's exact float rounding, so the golden-file postgres aggregates tests match
bit-for-bit.
* fix: regr_r2 degenerate-case swap (SPARK-55969) applies from Spark 3.5, not 4.1
SPARK-55969 swapped which regr_r2 degenerate case returns NULL and which
returns 1.0. It shipped in Spark 3.5.9, 4.0.3 and 4.1.0, so every Spark
version Comet builds against except 3.4 has the new behavior. Comet was
gating it on isSpark41Plus, which made regr_r2 disagree with Spark on the
3.5 and 4.0 profiles.1 parent 9cac716 commit 81d637b
8 files changed
Lines changed: 963 additions & 37 deletions
File tree
- docs/source/user-guide/latest
- native
- core/src/execution
- proto/src/proto
- spark-expr/src/agg_funcs
- spark/src
- main/scala/org/apache/comet/serde
- test/resources/sql-tests/expressions/aggregate
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
150 | | - | |
151 | | - | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
152 | 153 | | |
153 | 154 | | |
154 | 155 | | |
| |||
3029 | 3030 | | |
3030 | 3031 | | |
3031 | 3032 | | |
| 3033 | + | |
| 3034 | + | |
| 3035 | + | |
| 3036 | + | |
| 3037 | + | |
| 3038 | + | |
| 3039 | + | |
| 3040 | + | |
| 3041 | + | |
| 3042 | + | |
| 3043 | + | |
| 3044 | + | |
| 3045 | + | |
| 3046 | + | |
| 3047 | + | |
| 3048 | + | |
| 3049 | + | |
| 3050 | + | |
| 3051 | + | |
| 3052 | + | |
| 3053 | + | |
| 3054 | + | |
| 3055 | + | |
3032 | 3056 | | |
3033 | 3057 | | |
3034 | 3058 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| 165 | + | |
165 | 166 | | |
166 | 167 | | |
167 | 168 | | |
| |||
263 | 264 | | |
264 | 265 | | |
265 | 266 | | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
266 | 297 | | |
267 | 298 | | |
268 | 299 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| |||
0 commit comments