Skip to content

Commit d16f7bb

Browse files
authored
feat: support SortAggregateExec (#4565)
* feat: support SortAggregateExec [skip ci] Wire SortAggregateExec through the existing CometBaseAggregate.doConvert path so that queries planned with useObjectHashAggregate=false (or with TypedImperativeAggregate functions whose buffer formats prevent hash aggregation) can run their Partial->Final pair natively instead of falling back to Spark. CollectSet was the motivating function; the same wiring covers any other natively-implemented TypedImperativeAggregate. No proto changes: DataFusion's AggregateExec::try_new auto-detects InputOrderMode::Sorted from the child's output ordering, so a sorted input naturally produces sorted output. Wrapper reuse: createExec produces a CometHashAggregateExec (matching CometObjectHashAggregateExec). CometExec.outputOrdering already defaults to originalPlan.outputOrdering, so SortAggregateExec.outputOrdering flows through unchanged for downstream operators that elided a sort against it. Cleanups landed in passing: - Lifted baseAggregateSupportLevel and adjustOutputForNativeState onto the CometBaseAggregate trait (was duplicated 3x and 2x). - Collapsed isAggregate / findCometPartialAgg's Spark-side branches / canAggregateBeConverted's shuffle guard to BaseAggregateExec. * test: add comprehensive SortAggregate coverage via SQL file tests Add a SQL file test (sort_aggregate.sql) that forces collect_set through SortAggregateExec by disabling ObjectHashAggregate, covering global and grouped aggregation, multiple grouping keys, expression keys, NULL/empty/ single-row groups, mixed aggregates, multiple collect_set, DISTINCT, HAVING, a representative data-type spread, and a clean fallback when the aggregate input is incompatible. Add a global (no grouping keys) plan-shape assertion to CometAggregateSuite alongside the existing grouped one, since the SQL framework cannot assert that a SortAggregateExec was actually planned. * fix: distinguish CometSortAggregateExec and update tests for SortAggregate support Introduce a dedicated CometSortAggregateExec wrapper instead of reusing CometHashAggregateExec for converted SortAggregateExec nodes, so the executed plan reflects whether Spark planned a hash or a sort aggregate. A shared CometBaseAggregateExec base carries the common rendering and serialization, and findCometPartialAgg matches the base type so a partial sort aggregate (e.g. collect_set) is still paired with its final aggregate instead of falling back. Update the Spark test diffs to recognize the new wrapper: ReplaceHashWithSortAgg counts CometSortAggregateExec as a sort aggregate, and the generic aggregate collectors in AdaptiveQueryExecSuite and StreamingAggregationDistributionSuite match CometBaseAggregateExec so they see through both wrappers. Refresh the affected SQL file tests now that SortAggregate runs natively: - min_max.sql: string min/max still falls back, but the reason is now the StringType limitation rather than SortAggregate being unsupported. - first_last.sql: the multi-type first/last IGNORE NULLS queries now run natively; shape test_types so each group has a single non-null value, since first/last are non-deterministic across engines with multiple non-null rows. * fix: widen post-merge aggregate guards to CometBaseAggregateExec Self-review after merging main. Three safety predicates landed on main after this branch was cut and matched only CometHashAggregateExec, so a CometSortAggregateExec Partial slipped past all of them: - CometExecRule.preserveSparkAggregateBuffers: a Celeborn exchange that falls back to Spark must restore the Comet Partial below it. Missing the sort variant let a native collect_list buffer reach a Spark Final. Added a regression test in CometCelebornShufflePlanningSuite that fails without this change. - RevertNativeForTransitionHeavyStages.hasUnsafeMixedAggregateAtStageBoundary: same class of guard at a stage boundary. - CometMetricNode.withoutAggregateMetrics: range-partition sampling would double-count sort-aggregate spill and memory metrics. Also drop the duplicate adjustOutputForNativeState that the merge left behind (main grew its own copy handling CollectList and PartialMerge; the branch's older copy handled neither), move main's aggregate metrics override onto the shared CometBaseAggregateExec base, rename COMET_EXEC_SHUFFLE_ENABLED to COMET_SHUFFLE_ENABLED, and mark SortAggregateExec supported in operators.md. * refactor: share the aggregate serde and plan node equality CometBaseAggregate is now parameterized on the Spark aggregate type and supplies enabledConfig, getSupportLevel and convert for all three aggregate serdes. Each serde object keeps only its operator-specific support check, a requiresCometShuffle flag and createExec. The shared getSupportLevel notes that the partial and final test knobs gate sort aggregates too. equals and hashCode move into CometBaseAggregateExec and require the same class, so the two wrappers keep only withNewChildInternal. * fix: walk through sorts when pairing sort aggregate buffers Spark puts a SortExec between each SortAggregateExec and the exchange below it. The passes that pair a buffer-consuming aggregate with the Partial that produces its buffers stopped at that sort, so a native collect_list Partial could stay native under a Spark Final or merge stage. Spark then read Comet's array buffer where it expected binary, and a distinct collect_list chain slipped past the issue #4724 guard and failed natively. findPartialAggInPlan, revertUnsafePartialAggregates and hasUnrepairedNativeBuffer now walk through SortExec and CometSortExec. * fix: keep sort aggregate output in grouping-key order natively CometSortAggregateExec reports SortAggregateExec's grouping-key output ordering, and Spark may remove sorts above it on that basis. DataFusion emits groups in input order only when it sees the input sorted on them. When the ordering comes from outside the native plan, as with a cached sorted relation read through an unordered scan, its vectorized grouping can emit groups out of order: NULL and an empty array hash alike, so an ORDER BY over array keys returned [1] before []. A new HashAggregate field, ordered_by_grouping_keys, is set for sort aggregates. The native planner then sorts the aggregate output on the grouping columns unless the aggregate's output ordering already satisfies them. It also normalizes float grouping keys the way native sorts do, so that DataFusion sees the sort below a float-keyed final aggregate and streams it instead of falling back to the hash path. * test: run the collect_set fixtures through SortAggregateExec collect_set.sql and collect_set_floating_fallback.sql gain a ConfigMatrix over useObjectHashAggregateExec, so their whole fixtures run through both ObjectHashAggregateExec and SortAggregateExec. sort_aggregate.sql keeps only the shapes those files do not cover (several grouping keys, an expression key, hashable aggregates riding along) and adds the decimal SUM fallback. Its floating-point fallback query, which expected a fallback that Spark 4.2 no longer has, moves to the matrix. A new fixture checks that SortAggregateExec stays in Spark, with the shuffle reason, when Comet shuffle is disabled. min_by.sql and max_by.sql drop the claim that Comet never converts SortAggregate and check the type fallback without the TypedImperativeAggregate too. The two native collect_set Scala tests become one test over both queries. * docs: note how first and last can differ under sort aggregation Spark plans a sort aggregate for first and last over a column whose buffer cannot use hash aggregation, such as a string, and Comet now runs it natively. Within a group the row order is not defined, and Spark and Comet can order rows with equal grouping keys differently, so a group with more than one candidate value can return a different, equally valid, value. * test: run the mixed-engine collect_list tests with sort aggregates Each mixed-engine collect_list test now also runs with useObjectHashAggregateExec=false, so both halves of the split are covered for SortAggregateExec, where a sort sits between each half and its exchange. The tests count every Comet aggregate wrapper rather than only CometHashAggregateExec, and check which aggregate operator Spark planned. * fix: keep decimal AVG with a maximum-precision sum out of sort aggregates Spark's Average keeps its decimal sum at precision p + 10, capped at 38. When a sort aggregate's buffer has a field that is not mutable, such as a string FIRST, Spark holds that sum in a generic row where it is unbounded, so 0.6 + 0.6 - 0.4 over DECIMAL(38,38) recovers to 0.8. The native AVG records the overflow at 1.2 and returns NULL, or raises under ANSI. These queries stayed in Spark before sort aggregates converted. CometSortAggregateExec now declines a decimal AVG or TRY_AVG whose sum is at maximum precision, next to the existing decimal SUM check. The compatibility guide notes both fallbacks. * test: expect the distinct collect_list sort aggregate chain to run natively #4727 added native support for collect_list and collect_set alongside a distinct aggregate, so the PartialMerge stages of the distinct rewrite no longer force the chain back to Spark (#4724). Under sort aggregates the whole chain now runs natively and matches Spark. Check that, and keep the fallback case by checking that the chain stays in Spark, through the sort below each stage, when its final cannot convert.
1 parent fa25924 commit d16f7bb

25 files changed

Lines changed: 962 additions & 210 deletions

File tree

‎dev/diffs/3.4.3.diff‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1647,26 +1647,31 @@ index 30ce940b032..0d3f6c6c934 100644
16471647

16481648
private def checkSorts(query: String, enabledCount: Int, disabledCount: Int): Unit = {
16491649
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
1650-
index 47679ed7865..9ffbaecb98e 100644
1650+
index 47679ed7865..cbcabe00992 100644
16511651
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
16521652
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
16531653
@@ -18,6 +18,7 @@
16541654
package org.apache.spark.sql.execution
16551655

16561656
import org.apache.spark.sql.{DataFrame, QueryTest}
1657-
+import org.apache.spark.sql.comet.CometHashAggregateExec
1657+
+import org.apache.spark.sql.comet.{CometHashAggregateExec, CometSortAggregateExec}
16581658
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecutionSuite, EnableAdaptiveExecutionSuite}
16591659
import org.apache.spark.sql.execution.aggregate.{HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec}
16601660
import org.apache.spark.sql.internal.SQLConf
1661-
@@ -31,7 +32,7 @@ abstract class ReplaceHashWithSortAggSuiteBase
1661+
@@ -31,9 +32,11 @@ abstract class ReplaceHashWithSortAggSuiteBase
16621662
private def checkNumAggs(df: DataFrame, hashAggCount: Int, sortAggCount: Int): Unit = {
16631663
val plan = df.queryExecution.executedPlan
16641664
assert(collectWithSubqueries(plan) {
16651665
- case s @ (_: HashAggregateExec | _: ObjectHashAggregateExec) => s
16661666
+ case s @ (_: HashAggregateExec | _: ObjectHashAggregateExec | _: CometHashAggregateExec ) => s
16671667
}.length == hashAggCount)
1668-
assert(collectWithSubqueries(plan) { case s: SortAggregateExec => s }.length == sortAggCount)
1668+
- assert(collectWithSubqueries(plan) { case s: SortAggregateExec => s }.length == sortAggCount)
1669+
+ assert(collectWithSubqueries(plan) {
1670+
+ case s @ (_: SortAggregateExec | _: CometSortAggregateExec) => s
1671+
+ }.length == sortAggCount)
16691672
}
1673+
1674+
private def checkAggs(
16701675
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
16711676
index eec396b2e39..bf3f1c769d6 100644
16721677
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
@@ -1737,7 +1742,7 @@ index ac710c32296..2854b433dd3 100644
17371742

17381743
import testImplicits._
17391744
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
1740-
index 593bd7bb4ba..b327d84d5cc 100644
1745+
index 593bd7bb4ba..47285d87416 100644
17411746
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
17421747
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
17431748
@@ -26,9 +26,11 @@ import org.scalatest.time.SpanSugar._
@@ -1797,7 +1802,7 @@ index 593bd7bb4ba..b327d84d5cc 100644
17971802
private def findTopLevelAggregate(plan: SparkPlan): Seq[BaseAggregateExec] = {
17981803
collect(plan) {
17991804
case agg: BaseAggregateExec => agg
1800-
+ case agg: CometHashAggregateExec => agg.originalPlan.asInstanceOf[BaseAggregateExec]
1805+
+ case agg: CometBaseAggregateExec => agg.originalPlan.asInstanceOf[BaseAggregateExec]
18011806
}
18021807
}
18031808

@@ -3104,30 +3109,30 @@ index ef5b8a769fe..84fe1bfabc9 100644
31043109
}
31053110
}
31063111
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
3107-
index b4c4ec7acbf..20579284856 100644
3112+
index b4c4ec7acbf..e57aff1f170 100644
31083113
--- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
31093114
+++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
31103115
@@ -23,6 +23,7 @@ import org.apache.commons.io.FileUtils
31113116
import org.scalatest.Assertions
31123117

31133118
import org.apache.spark.sql.catalyst.plans.physical.UnspecifiedDistribution
3114-
+import org.apache.spark.sql.comet.CometHashAggregateExec
3119+
+import org.apache.spark.sql.comet.CometBaseAggregateExec
31153120
import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
31163121
import org.apache.spark.sql.execution.streaming.{MemoryStream, StateStoreRestoreExec, StateStoreSaveExec}
31173122
import org.apache.spark.sql.functions.count
31183123
@@ -67,6 +68,7 @@ class StreamingAggregationDistributionSuite extends StreamTest
31193124
// verify aggregations in between, except partial aggregation
31203125
val allAggregateExecs = query.lastExecution.executedPlan.collect {
31213126
case a: BaseAggregateExec => a
3122-
+ case c: CometHashAggregateExec => c.originalPlan
3127+
+ case c: CometBaseAggregateExec => c.originalPlan
31233128
}
31243129

31253130
val aggregateExecsWithoutPartialAgg = allAggregateExecs.filter {
31263131
@@ -201,6 +203,7 @@ class StreamingAggregationDistributionSuite extends StreamTest
31273132
// verify aggregations in between, except partial aggregation
31283133
val allAggregateExecs = executedPlan.collect {
31293134
case a: BaseAggregateExec => a
3130-
+ case c: CometHashAggregateExec => c.originalPlan
3135+
+ case c: CometBaseAggregateExec => c.originalPlan
31313136
}
31323137

31333138
val aggregateExecsWithoutPartialAgg = allAggregateExecs.filter {

‎dev/diffs/3.5.9.diff‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1645,26 +1645,31 @@ index 46ed8fdfd21..4585dbab5b8 100644
16451645

16461646
private def checkWindowGroupLimits(query: String, count: Int): Unit = {
16471647
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
1648-
index 47679ed7865..9ffbaecb98e 100644
1648+
index 47679ed7865..cbcabe00992 100644
16491649
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
16501650
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
16511651
@@ -18,6 +18,7 @@
16521652
package org.apache.spark.sql.execution
16531653

16541654
import org.apache.spark.sql.{DataFrame, QueryTest}
1655-
+import org.apache.spark.sql.comet.CometHashAggregateExec
1655+
+import org.apache.spark.sql.comet.{CometHashAggregateExec, CometSortAggregateExec}
16561656
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecutionSuite, EnableAdaptiveExecutionSuite}
16571657
import org.apache.spark.sql.execution.aggregate.{HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec}
16581658
import org.apache.spark.sql.internal.SQLConf
1659-
@@ -31,7 +32,7 @@ abstract class ReplaceHashWithSortAggSuiteBase
1659+
@@ -31,9 +32,11 @@ abstract class ReplaceHashWithSortAggSuiteBase
16601660
private def checkNumAggs(df: DataFrame, hashAggCount: Int, sortAggCount: Int): Unit = {
16611661
val plan = df.queryExecution.executedPlan
16621662
assert(collectWithSubqueries(plan) {
16631663
- case s @ (_: HashAggregateExec | _: ObjectHashAggregateExec) => s
16641664
+ case s @ (_: HashAggregateExec | _: ObjectHashAggregateExec | _: CometHashAggregateExec ) => s
16651665
}.length == hashAggCount)
1666-
assert(collectWithSubqueries(plan) { case s: SortAggregateExec => s }.length == sortAggCount)
1666+
- assert(collectWithSubqueries(plan) { case s: SortAggregateExec => s }.length == sortAggCount)
1667+
+ assert(collectWithSubqueries(plan) {
1668+
+ case s @ (_: SortAggregateExec | _: CometSortAggregateExec) => s
1669+
+ }.length == sortAggCount)
16671670
}
1671+
1672+
private def checkAggs(
16681673
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
16691674
index eec396b2e39..bf3f1c769d6 100644
16701675
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
@@ -1735,7 +1740,7 @@ index 5a413c77754..207b66e1d7b 100644
17351740

17361741
import testImplicits._
17371742
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
1738-
index 2f8e401e743..7849c685b19 100644
1743+
index 2f8e401e743..f7923d2b9a7 100644
17391744
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
17401745
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
17411746
@@ -27,9 +27,11 @@ import org.scalatest.time.SpanSugar._
@@ -1795,7 +1800,7 @@ index 2f8e401e743..7849c685b19 100644
17951800
private def findTopLevelAggregate(plan: SparkPlan): Seq[BaseAggregateExec] = {
17961801
collect(plan) {
17971802
case agg: BaseAggregateExec => agg
1798-
+ case agg: CometHashAggregateExec => agg.originalPlan.asInstanceOf[BaseAggregateExec]
1803+
+ case agg: CometBaseAggregateExec => agg.originalPlan.asInstanceOf[BaseAggregateExec]
17991804
}
18001805
}
18011806

@@ -3116,30 +3121,30 @@ index c97979a57a5..45a998db0e0 100644
31163121
}
31173122
}
31183123
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
3119-
index b4c4ec7acbf..20579284856 100644
3124+
index b4c4ec7acbf..e57aff1f170 100644
31203125
--- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
31213126
+++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
31223127
@@ -23,6 +23,7 @@ import org.apache.commons.io.FileUtils
31233128
import org.scalatest.Assertions
31243129

31253130
import org.apache.spark.sql.catalyst.plans.physical.UnspecifiedDistribution
3126-
+import org.apache.spark.sql.comet.CometHashAggregateExec
3131+
+import org.apache.spark.sql.comet.CometBaseAggregateExec
31273132
import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
31283133
import org.apache.spark.sql.execution.streaming.{MemoryStream, StateStoreRestoreExec, StateStoreSaveExec}
31293134
import org.apache.spark.sql.functions.count
31303135
@@ -67,6 +68,7 @@ class StreamingAggregationDistributionSuite extends StreamTest
31313136
// verify aggregations in between, except partial aggregation
31323137
val allAggregateExecs = query.lastExecution.executedPlan.collect {
31333138
case a: BaseAggregateExec => a
3134-
+ case c: CometHashAggregateExec => c.originalPlan
3139+
+ case c: CometBaseAggregateExec => c.originalPlan
31353140
}
31363141

31373142
val aggregateExecsWithoutPartialAgg = allAggregateExecs.filter {
31383143
@@ -201,6 +203,7 @@ class StreamingAggregationDistributionSuite extends StreamTest
31393144
// verify aggregations in between, except partial aggregation
31403145
val allAggregateExecs = executedPlan.collect {
31413146
case a: BaseAggregateExec => a
3142-
+ case c: CometHashAggregateExec => c.originalPlan
3147+
+ case c: CometBaseAggregateExec => c.originalPlan
31433148
}
31443149

31453150
val aggregateExecsWithoutPartialAgg = allAggregateExecs.filter {

‎dev/diffs/4.0.4.diff‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2070,26 +2070,31 @@ index 46ed8fdfd21..4585dbab5b8 100644
20702070

20712071
private def checkWindowGroupLimits(query: String, count: Int): Unit = {
20722072
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
2073-
index 47679ed7865..9ffbaecb98e 100644
2073+
index 47679ed7865..cbcabe00992 100644
20742074
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
20752075
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
20762076
@@ -18,6 +18,7 @@
20772077
package org.apache.spark.sql.execution
20782078

20792079
import org.apache.spark.sql.{DataFrame, QueryTest}
2080-
+import org.apache.spark.sql.comet.CometHashAggregateExec
2080+
+import org.apache.spark.sql.comet.{CometHashAggregateExec, CometSortAggregateExec}
20812081
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecutionSuite, EnableAdaptiveExecutionSuite}
20822082
import org.apache.spark.sql.execution.aggregate.{HashAggregateExec, ObjectHashAggregateExec, SortAggregateExec}
20832083
import org.apache.spark.sql.internal.SQLConf
2084-
@@ -31,7 +32,7 @@ abstract class ReplaceHashWithSortAggSuiteBase
2084+
@@ -31,9 +32,11 @@ abstract class ReplaceHashWithSortAggSuiteBase
20852085
private def checkNumAggs(df: DataFrame, hashAggCount: Int, sortAggCount: Int): Unit = {
20862086
val plan = df.queryExecution.executedPlan
20872087
assert(collectWithSubqueries(plan) {
20882088
- case s @ (_: HashAggregateExec | _: ObjectHashAggregateExec) => s
20892089
+ case s @ (_: HashAggregateExec | _: ObjectHashAggregateExec | _: CometHashAggregateExec ) => s
20902090
}.length == hashAggCount)
2091-
assert(collectWithSubqueries(plan) { case s: SortAggregateExec => s }.length == sortAggCount)
2091+
- assert(collectWithSubqueries(plan) { case s: SortAggregateExec => s }.length == sortAggCount)
2092+
+ assert(collectWithSubqueries(plan) {
2093+
+ case s @ (_: SortAggregateExec | _: CometSortAggregateExec) => s
2094+
+ }.length == sortAggCount)
20922095
}
2096+
2097+
private def checkAggs(
20932098
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
20942099
index aa619c5cde8..4b5a42e683f 100644
20952100
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala
@@ -2382,7 +2387,7 @@ index a3cfdc5a240..3793b6191bf 100644
23822387
})
23832388
checkAnswer(distinctWithId, Seq(Row(1, 0), Row(1, 0)))
23842389
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
2385-
index fb8fab6a80f..403eb411920 100644
2390+
index fb8fab6a80f..c5768935805 100644
23862391
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
23872392
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
23882393
@@ -28,12 +28,14 @@ import org.apache.spark.SparkException
@@ -2454,7 +2459,7 @@ index fb8fab6a80f..403eb411920 100644
24542459
private def findTopLevelAggregate(plan: SparkPlan): Seq[BaseAggregateExec] = {
24552460
collect(plan) {
24562461
case agg: BaseAggregateExec => agg
2457-
+ case agg: CometHashAggregateExec => agg.originalPlan.asInstanceOf[BaseAggregateExec]
2462+
+ case agg: CometBaseAggregateExec => agg.originalPlan.asInstanceOf[BaseAggregateExec]
24582463
}
24592464
}
24602465

@@ -3872,30 +3877,30 @@ index b0967d5ffdf..3d567f913de 100644
38723877
}
38733878
}
38743879
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
3875-
index b4c4ec7acbf..20579284856 100644
3880+
index b4c4ec7acbf..e57aff1f170 100644
38763881
--- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
38773882
+++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationDistributionSuite.scala
38783883
@@ -23,6 +23,7 @@ import org.apache.commons.io.FileUtils
38793884
import org.scalatest.Assertions
38803885

38813886
import org.apache.spark.sql.catalyst.plans.physical.UnspecifiedDistribution
3882-
+import org.apache.spark.sql.comet.CometHashAggregateExec
3887+
+import org.apache.spark.sql.comet.CometBaseAggregateExec
38833888
import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
38843889
import org.apache.spark.sql.execution.streaming.{MemoryStream, StateStoreRestoreExec, StateStoreSaveExec}
38853890
import org.apache.spark.sql.functions.count
38863891
@@ -67,6 +68,7 @@ class StreamingAggregationDistributionSuite extends StreamTest
38873892
// verify aggregations in between, except partial aggregation
38883893
val allAggregateExecs = query.lastExecution.executedPlan.collect {
38893894
case a: BaseAggregateExec => a
3890-
+ case c: CometHashAggregateExec => c.originalPlan
3895+
+ case c: CometBaseAggregateExec => c.originalPlan
38913896
}
38923897

38933898
val aggregateExecsWithoutPartialAgg = allAggregateExecs.filter {
38943899
@@ -201,6 +203,7 @@ class StreamingAggregationDistributionSuite extends StreamTest
38953900
// verify aggregations in between, except partial aggregation
38963901
val allAggregateExecs = executedPlan.collect {
38973902
case a: BaseAggregateExec => a
3898-
+ case c: CometHashAggregateExec => c.originalPlan
3903+
+ case c: CometBaseAggregateExec => c.originalPlan
38993904
}
39003905

39013906
val aggregateExecsWithoutPartialAgg = allAggregateExecs.filter {

0 commit comments

Comments
 (0)