Repository navigation
Commit d8ead26
authored
fix: spill a native PartialMerge whose input is sorted on some of its keys (#7222)
* fix: spill a native PartialMerge whose input is sorted on some of its keys
Comet runs a Spark PartialMerge as DataFusion Single so that it spills
rather than emit a key twice (#6857). When the input is sorted on a
prefix of the grouping keys, DataFusion 55.2 runs Single with the legacy
GroupedHashAggregateStream, whose spill asks for more than its share of
the pool to sort the batch it writes, and the task fails.
Plan that PartialMerge as DataFusion Final instead. Final runs an
OrderedFinalAggregateStream there, which spills and reads its spill
files back, and MergeAsPartial evaluates to its state under Final as it
does under Single. Final reads each state from the columns after the
keys and spills its groups in the input's types, so the planner switches
only when the input is the grouping keys followed by native-typed
states. A Spark partial's binary collect buffers keep the Single plan.
Closes #6880.
* refactor: decide the PartialMerge aggregate mode before building it
Read the input order from the child's equivalence properties, as
AggregateExec does, instead of building the aggregate as Single and
again as Final. Simplify the state type guard, note when the switch can
be rechecked, and cover input sorted on every key in the planner test.
* address review: lazy sort check, document fair-limit exemption, cover changing g and a sort-merge join
* address review: walk state types once, share planner test helpers
* test: cover the PartialMerge inputs Final rejects and run the join case under AQE
Add planner cases for swapped keys, states after one key and a column
after the states, which each keep the PartialMerge as Single. Check that
the join test's PartialMerge reaches the sort-merge join without an
exchange, and keep AQE on by disabling the suite's runtime broadcast
threshold. Name the DataFusion 56 stream as it is on main.
* address review: keep Single for a PartialMerge over fully sorted input
* address review: shorten the PartialMerge planning comment1 parent 61fbbe2 commit d8ead26
7 files changed
Lines changed: 419 additions & 86 deletions
File tree
- docs/source/contributor-guide
- native/core/src/execution
- memory_pools
- planner
- spark/src/test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
396 | 396 | | |
397 | 397 | | |
398 | 398 | | |
399 | | - | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
400 | 405 | | |
401 | 406 | | |
402 | 407 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
166 | 172 | | |
167 | 173 | | |
168 | 174 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
182 | 182 | | |
183 | 183 | | |
184 | 184 | | |
| 185 | + | |
185 | 186 | | |
186 | 187 | | |
187 | 188 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
| |||
51 | 54 | | |
52 | 55 | | |
53 | 56 | | |
54 | | - | |
| 57 | + | |
55 | 58 | | |
56 | 59 | | |
57 | 60 | | |
58 | 61 | | |
59 | | - | |
| 62 | + | |
60 | 63 | | |
61 | 64 | | |
62 | 65 | | |
| |||
165 | 168 | | |
166 | 169 | | |
167 | 170 | | |
168 | | - | |
| 171 | + | |
169 | 172 | | |
170 | 173 | | |
171 | 174 | | |
| |||
278 | 281 | | |
279 | 282 | | |
280 | 283 | | |
281 | | - | |
282 | | - | |
283 | | - | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
284 | 287 | | |
285 | 288 | | |
286 | 289 | | |
| |||
0 commit comments