Repository navigation
Commit 8af7cb1
perf: Improve output materializing speed for
# perf: Avoid copying when materializing output in
OrderedFinalAggregateStream
## Which issue does this PR close?
- Closes [apache#25157](apache#25157).
- Follow-up to
[apache#25312](apache#25312).
## Rationale for this change
See [apache#25312](apache#25312) for the
rationale and implementation details. This PR mechanically applies the
same pattern to `OrderedFinalAggregateStream`.
I repeated the original 10-million-row experiment on an M4 Pro MacBook
Pro, with 14 partitions and `batch_size = 8192`:
```text
main (already includes apache#25312): 450.82 ms
this PR: 329.25 ms
legacy aggregation (migration disabled): 336.10 ms
```
## What changes are included in this PR?
1. A clean-up refactor for `OrderedFinalAggregateStream` implementation,
see apache#23974 for idea.
2. Materialize all completed final results once and emit slices,
retaining the output buffers' reservation until the last slice is handed
off. If those buffers cannot be reserved, hand off the whole batch.
3. Preserve spill replay and count its final output once in the operator
metrics.
To read this PR, start at `OrderedFinalAggregateStream::into_stream()`.
Its stage diagram describes the execution flow, and the
incremental-output notes link to the explanation in the partial stream.
The diff is not easy to read directly due to the clean-up refactor
## What is the testing strategy for this PR?
For correctness, existing tests have covered it.
To prevent similar perf regression, we can do
- apache#25310
## Are there any user-facing changes?
Faster ordered final aggregation. Query results and public APIs are
unchanged.OrderedFinalAggregateStream (apache#25639)1 parent a11f858 commit 8af7cb1
4 files changed
Lines changed: 269 additions & 561 deletions
File tree
- datafusion/physical-plan/src/aggregates
- aggregate_hash_table
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
83 | | - | |
84 | | - | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
85 | 88 | | |
86 | 89 | | |
87 | | - | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
88 | 98 | | |
89 | 99 | | |
90 | 100 | | |
| 101 | + | |
91 | 102 | | |
92 | 103 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
719 | 719 | | |
720 | 720 | | |
721 | 721 | | |
722 | | - | |
| 722 | + | |
723 | 723 | | |
724 | 724 | | |
725 | 725 | | |
| |||
0 commit comments