Repository navigation
Commit d16f7bb
authored
feat: support SortAggregateExec (#4565)
* feat: support SortAggregateExec [skip ci]
Wire SortAggregateExec through the existing CometBaseAggregate.doConvert
path so that queries planned with useObjectHashAggregate=false (or with
TypedImperativeAggregate functions whose buffer formats prevent hash
aggregation) can run their Partial->Final pair natively instead of falling
back to Spark. CollectSet was the motivating function; the same wiring
covers any other natively-implemented TypedImperativeAggregate.
No proto changes: DataFusion's AggregateExec::try_new auto-detects
InputOrderMode::Sorted from the child's output ordering, so a sorted
input naturally produces sorted output.
Wrapper reuse: createExec produces a CometHashAggregateExec (matching
CometObjectHashAggregateExec). CometExec.outputOrdering already defaults
to originalPlan.outputOrdering, so SortAggregateExec.outputOrdering
flows through unchanged for downstream operators that elided a sort
against it.
Cleanups landed in passing:
- Lifted baseAggregateSupportLevel and adjustOutputForNativeState onto
the CometBaseAggregate trait (was duplicated 3x and 2x).
- Collapsed isAggregate / findCometPartialAgg's Spark-side branches /
canAggregateBeConverted's shuffle guard to BaseAggregateExec.
* test: add comprehensive SortAggregate coverage via SQL file tests
Add a SQL file test (sort_aggregate.sql) that forces collect_set through
SortAggregateExec by disabling ObjectHashAggregate, covering global and
grouped aggregation, multiple grouping keys, expression keys, NULL/empty/
single-row groups, mixed aggregates, multiple collect_set, DISTINCT,
HAVING, a representative data-type spread, and a clean fallback when the
aggregate input is incompatible.
Add a global (no grouping keys) plan-shape assertion to CometAggregateSuite
alongside the existing grouped one, since the SQL framework cannot assert
that a SortAggregateExec was actually planned.
* fix: distinguish CometSortAggregateExec and update tests for SortAggregate support
Introduce a dedicated CometSortAggregateExec wrapper instead of reusing
CometHashAggregateExec for converted SortAggregateExec nodes, so the executed
plan reflects whether Spark planned a hash or a sort aggregate. A shared
CometBaseAggregateExec base carries the common rendering and serialization, and
findCometPartialAgg matches the base type so a partial sort aggregate (e.g.
collect_set) is still paired with its final aggregate instead of falling back.
Update the Spark test diffs to recognize the new wrapper: ReplaceHashWithSortAgg
counts CometSortAggregateExec as a sort aggregate, and the generic aggregate
collectors in AdaptiveQueryExecSuite and StreamingAggregationDistributionSuite
match CometBaseAggregateExec so they see through both wrappers.
Refresh the affected SQL file tests now that SortAggregate runs natively:
- min_max.sql: string min/max still falls back, but the reason is now the
StringType limitation rather than SortAggregate being unsupported.
- first_last.sql: the multi-type first/last IGNORE NULLS queries now run
natively; shape test_types so each group has a single non-null value, since
first/last are non-deterministic across engines with multiple non-null rows.
* fix: widen post-merge aggregate guards to CometBaseAggregateExec
Self-review after merging main. Three safety predicates landed on main after
this branch was cut and matched only CometHashAggregateExec, so a
CometSortAggregateExec Partial slipped past all of them:
- CometExecRule.preserveSparkAggregateBuffers: a Celeborn exchange that falls
back to Spark must restore the Comet Partial below it. Missing the sort
variant let a native collect_list buffer reach a Spark Final. Added a
regression test in CometCelebornShufflePlanningSuite that fails without this
change.
- RevertNativeForTransitionHeavyStages.hasUnsafeMixedAggregateAtStageBoundary:
same class of guard at a stage boundary.
- CometMetricNode.withoutAggregateMetrics: range-partition sampling would
double-count sort-aggregate spill and memory metrics.
Also drop the duplicate adjustOutputForNativeState that the merge left behind
(main grew its own copy handling CollectList and PartialMerge; the branch's
older copy handled neither), move main's aggregate metrics override onto the
shared CometBaseAggregateExec base, rename COMET_EXEC_SHUFFLE_ENABLED to
COMET_SHUFFLE_ENABLED, and mark SortAggregateExec supported in operators.md.
* refactor: share the aggregate serde and plan node equality
CometBaseAggregate is now parameterized on the Spark aggregate type and
supplies enabledConfig, getSupportLevel and convert for all three aggregate
serdes. Each serde object keeps only its operator-specific support check, a
requiresCometShuffle flag and createExec. The shared getSupportLevel notes
that the partial and final test knobs gate sort aggregates too.
equals and hashCode move into CometBaseAggregateExec and require the same
class, so the two wrappers keep only withNewChildInternal.
* fix: walk through sorts when pairing sort aggregate buffers
Spark puts a SortExec between each SortAggregateExec and the exchange
below it. The passes that pair a buffer-consuming aggregate with the
Partial that produces its buffers stopped at that sort, so a native
collect_list Partial could stay native under a Spark Final or merge
stage. Spark then read Comet's array buffer where it expected binary, and
a distinct collect_list chain slipped past the issue #4724 guard and
failed natively.
findPartialAggInPlan, revertUnsafePartialAggregates and
hasUnrepairedNativeBuffer now walk through SortExec and CometSortExec.
* fix: keep sort aggregate output in grouping-key order natively
CometSortAggregateExec reports SortAggregateExec's grouping-key output
ordering, and Spark may remove sorts above it on that basis. DataFusion
emits groups in input order only when it sees the input sorted on them.
When the ordering comes from outside the native plan, as with a cached
sorted relation read through an unordered scan, its vectorized grouping
can emit groups out of order: NULL and an empty array hash alike, so an
ORDER BY over array keys returned [1] before [].
A new HashAggregate field, ordered_by_grouping_keys, is set for sort
aggregates. The native planner then sorts the aggregate output on the
grouping columns unless the aggregate's output ordering already satisfies
them. It also normalizes float grouping keys the way native sorts do, so
that DataFusion sees the sort below a float-keyed final aggregate and
streams it instead of falling back to the hash path.
* test: run the collect_set fixtures through SortAggregateExec
collect_set.sql and collect_set_floating_fallback.sql gain a ConfigMatrix
over useObjectHashAggregateExec, so their whole fixtures run through both
ObjectHashAggregateExec and SortAggregateExec. sort_aggregate.sql keeps only
the shapes those files do not cover (several grouping keys, an expression
key, hashable aggregates riding along) and adds the decimal SUM fallback.
Its floating-point fallback query, which expected a fallback that Spark 4.2
no longer has, moves to the matrix.
A new fixture checks that SortAggregateExec stays in Spark, with the
shuffle reason, when Comet shuffle is disabled. min_by.sql and max_by.sql
drop the claim that Comet never converts SortAggregate and check the type
fallback without the TypedImperativeAggregate too. The two native
collect_set Scala tests become one test over both queries.
* docs: note how first and last can differ under sort aggregation
Spark plans a sort aggregate for first and last over a column whose buffer
cannot use hash aggregation, such as a string, and Comet now runs it
natively. Within a group the row order is not defined, and Spark and Comet
can order rows with equal grouping keys differently, so a group with more
than one candidate value can return a different, equally valid, value.
* test: run the mixed-engine collect_list tests with sort aggregates
Each mixed-engine collect_list test now also runs with
useObjectHashAggregateExec=false, so both halves of the split are covered
for SortAggregateExec, where a sort sits between each half and its
exchange. The tests count every Comet aggregate wrapper rather than only
CometHashAggregateExec, and check which aggregate operator Spark planned.
* fix: keep decimal AVG with a maximum-precision sum out of sort aggregates
Spark's Average keeps its decimal sum at precision p + 10, capped at 38.
When a sort aggregate's buffer has a field that is not mutable, such as a
string FIRST, Spark holds that sum in a generic row where it is unbounded,
so 0.6 + 0.6 - 0.4 over DECIMAL(38,38) recovers to 0.8. The native AVG
records the overflow at 1.2 and returns NULL, or raises under ANSI. These
queries stayed in Spark before sort aggregates converted.
CometSortAggregateExec now declines a decimal AVG or TRY_AVG whose sum is
at maximum precision, next to the existing decimal SUM check. The
compatibility guide notes both fallbacks.
* test: expect the distinct collect_list sort aggregate chain to run natively
#4727 added native support for collect_list and collect_set alongside a
distinct aggregate, so the PartialMerge stages of the distinct rewrite no
longer force the chain back to Spark (#4724). Under sort aggregates the
whole chain now runs natively and matches Spark. Check that, and keep the
fallback case by checking that the chain stays in Spark, through the sort
below each stage, when its final cannot convert.1 parent fa25924 commit d16f7bb
25 files changed
Lines changed: 962 additions & 210 deletions
File tree
- dev/diffs
- docs/source/user-guide/latest
- compatibility
- native
- core/src/execution
- proto/src/proto
- spark/src
- main/scala/org/apache
- comet/rules
- spark/sql/comet
- test
- resources/sql-tests/expressions/aggregate
- scala/org/apache
- comet
- exec
- rules
- spark/sql/comet/execution/shuffle
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1647 | 1647 | | |
1648 | 1648 | | |
1649 | 1649 | | |
1650 | | - | |
| 1650 | + | |
1651 | 1651 | | |
1652 | 1652 | | |
1653 | 1653 | | |
1654 | 1654 | | |
1655 | 1655 | | |
1656 | 1656 | | |
1657 | | - | |
| 1657 | + | |
1658 | 1658 | | |
1659 | 1659 | | |
1660 | 1660 | | |
1661 | | - | |
| 1661 | + | |
1662 | 1662 | | |
1663 | 1663 | | |
1664 | 1664 | | |
1665 | 1665 | | |
1666 | 1666 | | |
1667 | 1667 | | |
1668 | | - | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
1669 | 1672 | | |
| 1673 | + | |
| 1674 | + | |
1670 | 1675 | | |
1671 | 1676 | | |
1672 | 1677 | | |
| |||
1737 | 1742 | | |
1738 | 1743 | | |
1739 | 1744 | | |
1740 | | - | |
| 1745 | + | |
1741 | 1746 | | |
1742 | 1747 | | |
1743 | 1748 | | |
| |||
1797 | 1802 | | |
1798 | 1803 | | |
1799 | 1804 | | |
1800 | | - | |
| 1805 | + | |
1801 | 1806 | | |
1802 | 1807 | | |
1803 | 1808 | | |
| |||
3104 | 3109 | | |
3105 | 3110 | | |
3106 | 3111 | | |
3107 | | - | |
| 3112 | + | |
3108 | 3113 | | |
3109 | 3114 | | |
3110 | 3115 | | |
3111 | 3116 | | |
3112 | 3117 | | |
3113 | 3118 | | |
3114 | | - | |
| 3119 | + | |
3115 | 3120 | | |
3116 | 3121 | | |
3117 | 3122 | | |
3118 | 3123 | | |
3119 | 3124 | | |
3120 | 3125 | | |
3121 | 3126 | | |
3122 | | - | |
| 3127 | + | |
3123 | 3128 | | |
3124 | 3129 | | |
3125 | 3130 | | |
3126 | 3131 | | |
3127 | 3132 | | |
3128 | 3133 | | |
3129 | 3134 | | |
3130 | | - | |
| 3135 | + | |
3131 | 3136 | | |
3132 | 3137 | | |
3133 | 3138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1645 | 1645 | | |
1646 | 1646 | | |
1647 | 1647 | | |
1648 | | - | |
| 1648 | + | |
1649 | 1649 | | |
1650 | 1650 | | |
1651 | 1651 | | |
1652 | 1652 | | |
1653 | 1653 | | |
1654 | 1654 | | |
1655 | | - | |
| 1655 | + | |
1656 | 1656 | | |
1657 | 1657 | | |
1658 | 1658 | | |
1659 | | - | |
| 1659 | + | |
1660 | 1660 | | |
1661 | 1661 | | |
1662 | 1662 | | |
1663 | 1663 | | |
1664 | 1664 | | |
1665 | 1665 | | |
1666 | | - | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
1667 | 1670 | | |
| 1671 | + | |
| 1672 | + | |
1668 | 1673 | | |
1669 | 1674 | | |
1670 | 1675 | | |
| |||
1735 | 1740 | | |
1736 | 1741 | | |
1737 | 1742 | | |
1738 | | - | |
| 1743 | + | |
1739 | 1744 | | |
1740 | 1745 | | |
1741 | 1746 | | |
| |||
1795 | 1800 | | |
1796 | 1801 | | |
1797 | 1802 | | |
1798 | | - | |
| 1803 | + | |
1799 | 1804 | | |
1800 | 1805 | | |
1801 | 1806 | | |
| |||
3116 | 3121 | | |
3117 | 3122 | | |
3118 | 3123 | | |
3119 | | - | |
| 3124 | + | |
3120 | 3125 | | |
3121 | 3126 | | |
3122 | 3127 | | |
3123 | 3128 | | |
3124 | 3129 | | |
3125 | 3130 | | |
3126 | | - | |
| 3131 | + | |
3127 | 3132 | | |
3128 | 3133 | | |
3129 | 3134 | | |
3130 | 3135 | | |
3131 | 3136 | | |
3132 | 3137 | | |
3133 | 3138 | | |
3134 | | - | |
| 3139 | + | |
3135 | 3140 | | |
3136 | 3141 | | |
3137 | 3142 | | |
3138 | 3143 | | |
3139 | 3144 | | |
3140 | 3145 | | |
3141 | 3146 | | |
3142 | | - | |
| 3147 | + | |
3143 | 3148 | | |
3144 | 3149 | | |
3145 | 3150 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2070 | 2070 | | |
2071 | 2071 | | |
2072 | 2072 | | |
2073 | | - | |
| 2073 | + | |
2074 | 2074 | | |
2075 | 2075 | | |
2076 | 2076 | | |
2077 | 2077 | | |
2078 | 2078 | | |
2079 | 2079 | | |
2080 | | - | |
| 2080 | + | |
2081 | 2081 | | |
2082 | 2082 | | |
2083 | 2083 | | |
2084 | | - | |
| 2084 | + | |
2085 | 2085 | | |
2086 | 2086 | | |
2087 | 2087 | | |
2088 | 2088 | | |
2089 | 2089 | | |
2090 | 2090 | | |
2091 | | - | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
2092 | 2095 | | |
| 2096 | + | |
| 2097 | + | |
2093 | 2098 | | |
2094 | 2099 | | |
2095 | 2100 | | |
| |||
2382 | 2387 | | |
2383 | 2388 | | |
2384 | 2389 | | |
2385 | | - | |
| 2390 | + | |
2386 | 2391 | | |
2387 | 2392 | | |
2388 | 2393 | | |
| |||
2454 | 2459 | | |
2455 | 2460 | | |
2456 | 2461 | | |
2457 | | - | |
| 2462 | + | |
2458 | 2463 | | |
2459 | 2464 | | |
2460 | 2465 | | |
| |||
3872 | 3877 | | |
3873 | 3878 | | |
3874 | 3879 | | |
3875 | | - | |
| 3880 | + | |
3876 | 3881 | | |
3877 | 3882 | | |
3878 | 3883 | | |
3879 | 3884 | | |
3880 | 3885 | | |
3881 | 3886 | | |
3882 | | - | |
| 3887 | + | |
3883 | 3888 | | |
3884 | 3889 | | |
3885 | 3890 | | |
3886 | 3891 | | |
3887 | 3892 | | |
3888 | 3893 | | |
3889 | 3894 | | |
3890 | | - | |
| 3895 | + | |
3891 | 3896 | | |
3892 | 3897 | | |
3893 | 3898 | | |
3894 | 3899 | | |
3895 | 3900 | | |
3896 | 3901 | | |
3897 | 3902 | | |
3898 | | - | |
| 3903 | + | |
3899 | 3904 | | |
3900 | 3905 | | |
3901 | 3906 | | |
| |||
0 commit comments