Skip to content

Commit e7c51c7

Browse files
xavleejayzhan211
andauthored
feat: stream aggregates over grouped input (apache#24497)
## Which issue does this PR close? - Closes apache#24438. - Builds on apache#24737, apache#24697, and apache#24698 (all merged). ## Rationale for this change An aggregate can emit a completed group once it knows that the same complete grouping tuple cannot appear again in its input partition. `EquivalenceProperties::grouping_satisfy` now provides that guarantee for both lexicographically ordered input and explicitly grouped input. For example, these complete `(key, time_bin)` groups are contiguous even though their values are not globally sorted: ```text (A, 20), (A, 20) | (B, 20), (B, 20) | (A, 0), (A, 0) ``` The aggregate can therefore release each group as its run ends. Its clustering mode describes that guarantee, while equivalence properties describe any actual output ordering. ## What changes are included in this PR? - Query the input equivalence properties for the complete `GROUP BY` tuple during `AggregateExec` construction. - Select `GroupClusteringMode::Full` when complete grouping tuple is satisfied, at least one grouping expression is non-constant, and grouping uses one non-nulling mask. - Select clustered execution for grouped input, allowing completed groups to be emitted incrementally. - Use input group metadata for clustered execution in `Single`, `SinglePartitioned`, and `Partial` aggregations. In a two-stage plan, the partial aggregate clears explicit grouping metadata from its output, so this property alone does not enabled clustered final aggregation. The final stage can still use ordering provided by its own input. ```text input grouping: [[key, time_bin]] GROUP BY: [key, time_bin] output ordering: none grouping_satisfy(GROUP BY) -> GroupClusteringMode::Full -> EmissionType::Incremental -> ClusteredSingleAggregateStream ``` ## Stack 1. [apache#24737 — test: cover unsorted contiguous groups in one partition](apache#24737) (merged) 2. [apache#24697 — refactor: use group clustering terminology in aggregate execution](apache#24697) (merged) 3. [apache#24698 — feat: add grouping equivalence properties](apache#24698) (merged) 4. [apache#24497 — feat: stream aggregates over grouped input](apache#24497) ← **this PR** ## Are these changes tested? The characterization test from apache#24737 declares its complete grouping tuple and verifies `GroupClusteringMode::Full`, `EmissionType::Incremental`, and `ClusteredSingleAggregateStream`. It uses `BarrierExec` to hold back the second input batch until the first completed group is emitted, then checks the full result, including a group spanning both batches. Regression tests cover `PartialReduce` with ordered and explicitly grouped input, and constant-only grouping with both a literal and a column made constant by a filter. A projection test verifies that a semantically equivalent `date_bin` grouping tuple reaches `AggregateExec` and enables incremental emission. A repeated-mask regression compares grouped and unannotated inputs and verifies that both produce the same merged groups with `GroupClusteringMode::None`. ## Are there any user-facing changes? Yes. Aggregates whose complete grouping tuple is satisfied by the input equivalence properties can execute with full group clustering and incremental emission on unsorted input. ## Review this layer [View only this PR layer](xavlee/datafusion@a616831...3b42751) --------- Co-authored-by: Jay Zhan <[email protected]>
1 parent fbc3895 commit e7c51c7

3 files changed

Lines changed: 512 additions & 72 deletions

File tree

0 commit comments

Comments
 (0)