Repository navigation
Commit e7c51c7
feat: stream aggregates over grouped input (apache#24497)
## Which issue does this PR close?
- Closes apache#24438.
- Builds on apache#24737, apache#24697, and apache#24698 (all merged).
## Rationale for this change
An aggregate can emit a completed group once it knows that the same
complete grouping tuple cannot appear again in its input partition.
`EquivalenceProperties::grouping_satisfy` now provides that guarantee
for both lexicographically ordered input and explicitly grouped input.
For example, these complete `(key, time_bin)` groups are contiguous even
though their values are not globally sorted:
```text
(A, 20), (A, 20) | (B, 20), (B, 20) | (A, 0), (A, 0)
```
The aggregate can therefore release each group as its run ends. Its
clustering mode describes that guarantee, while equivalence properties
describe any actual output ordering.
## What changes are included in this PR?
- Query the input equivalence properties for the complete `GROUP BY`
tuple during `AggregateExec` construction.
- Select `GroupClusteringMode::Full` when complete grouping tuple is
satisfied, at least one grouping expression is non-constant, and
grouping uses one non-nulling mask.
- Select clustered execution for grouped input, allowing completed
groups to be emitted incrementally.
- Use input group metadata for clustered execution in `Single`,
`SinglePartitioned`, and `Partial` aggregations. In a two-stage plan,
the partial aggregate clears explicit grouping metadata from its output,
so this property alone does not enabled clustered final aggregation. The
final stage can still use ordering provided by its own input.
```text
input grouping: [[key, time_bin]]
GROUP BY: [key, time_bin]
output ordering: none
grouping_satisfy(GROUP BY)
-> GroupClusteringMode::Full
-> EmissionType::Incremental
-> ClusteredSingleAggregateStream
```
## Stack
1. [apache#24737 — test: cover unsorted contiguous groups in one
partition](apache#24737) (merged)
2. [apache#24697 — refactor: use group clustering terminology in aggregate
execution](apache#24697) (merged)
3. [apache#24698 — feat: add grouping equivalence
properties](apache#24698) (merged)
4. [apache#24497 — feat: stream aggregates over grouped
input](apache#24497) ← **this PR**
## Are these changes tested?
The characterization test from apache#24737 declares its complete grouping
tuple and verifies `GroupClusteringMode::Full`,
`EmissionType::Incremental`, and `ClusteredSingleAggregateStream`. It
uses `BarrierExec` to hold back the second input batch until the first
completed group is emitted, then checks the full result, including a
group spanning both batches.
Regression tests cover `PartialReduce` with ordered and explicitly
grouped input, and constant-only grouping with both a literal and a
column made constant by a filter. A projection test verifies that a
semantically equivalent `date_bin` grouping tuple reaches
`AggregateExec` and enables incremental emission.
A repeated-mask regression compares grouped and unannotated inputs and
verifies that both produce the same merged groups with
`GroupClusteringMode::None`.
## Are there any user-facing changes?
Yes. Aggregates whose complete grouping tuple is satisfied by the input
equivalence properties can execute with full group clustering and
incremental emission on unsorted input.
## Review this layer
[View only this PR
layer](xavlee/datafusion@a616831...3b42751)
---------
Co-authored-by: Jay Zhan <[email protected]>1 parent fbc3895 commit e7c51c7
3 files changed
Lines changed: 512 additions & 72 deletions
0 commit comments