Skip to content

Commit e5bd4cc

Browse files
committed
fix: fall back for an ungrouped decimal SUM at maximum precision without codegen
Spark turns codegen off for a hash aggregate with an imperative function or when whole-stage codegen is disabled, and then buffers an ungrouped sum in an UnsafeRow that latches once it leaves the precision. The native accumulator keeps the unbounded intermediate, so that case now falls back, and the compatibility note names the cases planning cannot see.
1 parent c7fdb5d commit e5bd4cc

3 files changed

Lines changed: 104 additions & 17 deletions

File tree

‎docs/source/user-guide/latest/compatibility/index.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -147,10 +147,12 @@ so users hunting an unexpected value have a single place to check:
147147
([#5022](https://github.com/apache/datafusion-comet/issues/5022)).
148148
- Ungrouped decimal `SUM` keeps an unbounded intermediate and checks the result precision only
149149
when a partial is written out or the sum is evaluated, which matches Spark's whole-stage codegen
150-
path. With `spark.sql.codegen.wholeStage=false` Spark's ungrouped aggregate buffers in an
151-
`UnsafeRow` and latches as soon as a running sum leaves the precision, so an intermediate
152-
overflow that later cancels out returns `NULL` (or raises under ANSI) in Spark but the recovered
153-
value in Comet. Spark's own decimal overflow tests accept either outcome.
150+
path. Without codegen Spark buffers the aggregate in an `UnsafeRow` and latches as soon as a
151+
running sum leaves the precision. Comet falls back at precision 38 when codegen is disabled by
152+
`spark.sql.codegen.wholeStage` or by an imperative sibling aggregate, but not when Spark
153+
disables it for a stage with more than `spark.sql.codegen.maxFields` fields or after a codegen
154+
compile failure, where an intermediate overflow that later cancels out returns `NULL` (or raises
155+
under ANSI) in Spark but the recovered value in Comet.
154156

155157
## Object store cache
156158

‎spark/src/main/scala/org/apache/spark/sql/comet/operators.scala‎

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ import org.apache.spark.internal.Logging
3131
import org.apache.spark.rdd.RDD
3232
import org.apache.spark.sql.catalyst.InternalRow
3333
import org.apache.spark.sql.catalyst.expressions.{Ascending, Attribute, AttributeSeq, AttributeSet, Expression, ExpressionSet, Generator, NamedExpression, SortOrder, XXH64}
34-
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, AggregateMode, CollectList, CollectSet, Final, Mode, Partial, PartialMerge, Percentile, Sum}
34+
import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, AggregateMode, CollectList, CollectSet, Final, ImperativeAggregate, Mode, Partial, PartialMerge, Percentile, Sum}
3535
import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight, BuildSide}
3636
import org.apache.spark.sql.catalyst.plans._
3737
import org.apache.spark.sql.catalyst.plans.physical._
@@ -1847,6 +1847,20 @@ case class CometUnionExec(
18471847

18481848
trait CometBaseAggregate {
18491849

1850+
/**
1851+
* Whether a decimal SUM's result precision is DecimalType.MAX_PRECISION, the only case with no
1852+
* headroom above the input where an intermediate overflow can change the answer.
1853+
*/
1854+
protected def hasMaxPrecisionDecimalSum(op: BaseAggregateExec): Boolean =
1855+
op.aggregateExpressions.exists(_.aggregateFunction match {
1856+
case sum: Sum =>
1857+
sum.dataType match {
1858+
case decimal: DecimalType => decimal.precision == DecimalType.MAX_PRECISION
1859+
case _ => false
1860+
}
1861+
case _ => false
1862+
})
1863+
18501864
def doConvert(
18511865
aggregate: BaseAggregateExec,
18521866
builder: Operator.Builder,
@@ -2199,6 +2213,17 @@ object CometHashAggregateExec
21992213
op.aggregateExpressions.exists(_.mode == Final)) {
22002214
return Unsupported(Some("Final aggregates disabled via test config"))
22012215
}
2216+
// Without codegen Spark buffers an ungrouped aggregate in an UnsafeRow, which latches a
2217+
// decimal sum that leaves the precision, while the native ungrouped accumulator keeps the
2218+
// unbounded intermediate. Spark turns codegen off for an imperative aggregate or by config.
2219+
val codegenOff = !op.conf.wholeStageEnabled ||
2220+
op.aggregateExpressions.exists(_.aggregateFunction.isInstanceOf[ImperativeAggregate])
2221+
if (op.groupingExpressions.isEmpty && codegenOff && hasMaxPrecisionDecimalSum(op)) {
2222+
return Unsupported(
2223+
Some(
2224+
"Ungrouped decimal SUM at maximum precision without codegen cannot match Spark's " +
2225+
"latching UnsafeRow buffer"))
2226+
}
22022227
Compatible()
22032228
}
22042229

@@ -2259,18 +2284,9 @@ object CometObjectHashAggregateExec
22592284
"aggregate across Comet and Spark"))
22602285
}
22612286
// Spark's object aggregation buffers the intermediate decimal sum unbounded, while Comet's
2262-
// grouped accumulator latches to null once a running sum leaves the precision. Only a result
2263-
// precision of DecimalType.MAX_PRECISION has no headroom above the input, so only that case
2264-
// can diverge and declines here. Decimal AVG has the same gap and is tracked separately.
2265-
val hasMaxPrecisionDecimalSum = op.aggregateExpressions.exists(_.aggregateFunction match {
2266-
case sum: Sum =>
2267-
sum.dataType match {
2268-
case decimal: DecimalType => decimal.precision == DecimalType.MAX_PRECISION
2269-
case _ => false
2270-
}
2271-
case _ => false
2272-
})
2273-
if (op.groupingExpressions.nonEmpty && hasMaxPrecisionDecimalSum) {
2287+
// grouped accumulator latches to null once a running sum leaves the precision, so the
2288+
// grouped case declines. Decimal AVG has the same gap and is tracked separately.
2289+
if (op.groupingExpressions.nonEmpty && hasMaxPrecisionDecimalSum(op)) {
22742290
return Unsupported(
22752291
Some(
22762292
"Grouped decimal SUM at maximum precision cannot match Spark's unbounded object " +

‎spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1545,6 +1545,75 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
15451545
}
15461546
}
15471547

1548+
test("decimal sum without codegen falls back at maximum precision") {
1549+
// Without codegen Spark buffers the ungrouped sum in an UnsafeRow and latches once it
1550+
// leaves the precision; an imperative sibling such as approx_count_distinct or the
1551+
// whole-stage switch turns codegen off, and falling back keeps Spark's answer.
1552+
val reason = "Ungrouped decimal SUM at maximum precision without codegen cannot match " +
1553+
"Spark's latching UnsafeRow buffer"
1554+
Seq(true, false).foreach { ansiEnabled =>
1555+
withSQLConf(
1556+
SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString,
1557+
CometConf.COMET_SHUFFLE_ENABLED.key -> "true",
1558+
CometConf.COMET_SHUFFLE_MODE.key -> "native") {
1559+
withMaxPrecisionDecimalTable(intermediateOverflowRows, "dec_no_codegen") {
1560+
assertDecimalSumFallsBackLikeSpark(
1561+
sql("SELECT SUM(v), approx_count_distinct(k) FROM dec_no_codegen"),
1562+
reason,
1563+
ansiEnabled,
1564+
Row(null, 1L))
1565+
withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false") {
1566+
assertDecimalSumFallsBackLikeSpark(
1567+
sql("SELECT SUM(v) FROM dec_no_codegen"),
1568+
reason,
1569+
ansiEnabled,
1570+
Row(null))
1571+
}
1572+
}
1573+
}
1574+
}
1575+
// With headroom above the input precision the imperative sibling keeps the aggregate native.
1576+
withTempPath { dir =>
1577+
intermediateOverflowRows
1578+
.toDF("k", "raw_v")
1579+
.selectExpr("k", "CAST(raw_v AS DECIMAL(10,2)) AS v")
1580+
.coalesce(1)
1581+
.write
1582+
.parquet(dir.toString)
1583+
withParquetTable(dir.toString, "dec_no_codegen_small") {
1584+
checkSparkAnswerAndOperator(
1585+
"SELECT SUM(v), approx_count_distinct(k) FROM dec_no_codegen_small")
1586+
}
1587+
}
1588+
}
1589+
1590+
/**
1591+
* Asserts that `df` runs the aggregate in Spark with `reason` recorded and gives Spark's
1592+
* latched result: `expected` in legacy mode, and the same failure as Spark under ANSI.
1593+
*/
1594+
private def assertDecimalSumFallsBackLikeSpark(
1595+
df: DataFrame,
1596+
reason: String,
1597+
ansiEnabled: Boolean,
1598+
expected: Row): Unit = {
1599+
if (ansiEnabled) {
1600+
val (sparkError, cometError) = checkSparkAnswerMaybeThrows(df)
1601+
assert(
1602+
sparkError.isDefined && cometError.isDefined,
1603+
s"expected both engines to fail, got Spark $sparkError and Comet $cometError")
1604+
} else {
1605+
checkSparkAnswerAndFallbackReason(df, reason)
1606+
val answer = df.collect().toSeq
1607+
assert(answer == Seq(expected), s"without codegen returned $answer, expected $expected")
1608+
}
1609+
val nativeAggregates = stripAQEPlan(df.queryExecution.executedPlan).collect {
1610+
case agg: CometHashAggregateExec => agg
1611+
}
1612+
assert(
1613+
nativeAggregates.isEmpty,
1614+
s"expected the aggregate to run in Spark, got $nativeAggregates")
1615+
}
1616+
15481617
test("decimal sum with object hash aggregate falls back at maximum precision") {
15491618
withSQLConf(
15501619
SQLConf.USE_OBJECT_HASH_AGG.key -> "true",

0 commit comments

Comments
 (0)