Skip to content

Commit add66e4

Browse files
authored
refactor: clean up aggregation planning (apache#25104)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#24318 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> See issue ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No.
1 parent a7305f1 commit add66e4

1 file changed

Lines changed: 61 additions & 124 deletions

File tree

  • datafusion/physical-plan/src/aggregates

‎datafusion/physical-plan/src/aggregates/mod.rs‎

Lines changed: 61 additions & 124 deletions
Original file line numberDiff line numberDiff line change
@@ -889,6 +889,10 @@ pub struct AggregateExec {
889889
metrics: ExecutionPlanMetricsSet,
890890
required_input_ordering: Option<OrderingRequirements>,
891891
/// Describes how the input is ordered relative to the group by columns
892+
///
893+
/// This field is also overloaded to mean "the output MUST preserve this
894+
/// input order". When that is not possible, the constructor overwrites it
895+
/// with the unordered variant [`InputOrderMode::Linear`].
892896
input_order_mode: InputOrderMode,
893897
cache: Arc<PlanProperties>,
894898
/// During initialization, if the plan supports dynamic filtering (see [`AggrDynFilter`]),
@@ -1224,143 +1228,76 @@ impl AggregateExec {
12241228
));
12251229
}
12261230

1227-
// Select the stream type based on the query shape and configuration.
1228-
// For an overview, see the `Aggregate planning` section in this file's
1229-
// documentation.
1230-
//
1231-
// # Implementation Note
1232-
//
1233-
// `GroupedHashAggregateStream` is the legacy implementation of all the
1234-
// branches below. It has been split into dedicated streams, and the
1235-
// `enable_migration_aggregate` config option selects between the new
1236-
// streams and the legacy implementation.
1237-
//
1238-
// See the `grouped_hash_stream` module documentation for the
1239-
// deprecation schedule.
1240-
if context
1231+
// `GroupedHashAggregateStream` is the legacy implementation of every
1232+
// grouped path below. It is only planned as an opt-in fallback, see the
1233+
// `grouped_hash_stream` module documentation for the deprecation
1234+
// schedule.
1235+
if !context
12411236
.session_config()
12421237
.options()
12431238
.execution
12441239
.enable_migration_aggregate
12451240
{
1246-
if self.should_use_ordered_partial_aggregate_stream(context) {
1247-
return Ok(StreamType::OrderedPartialAggregate(
1248-
OrderedPartialAggregateStream::new(self, context, partition)?,
1249-
));
1250-
}
1241+
return Ok(StreamType::GroupedHash(GroupedHashAggregateStream::new(
1242+
self, context, partition,
1243+
)?));
1244+
}
12511245

1252-
if self.should_use_partial_hash_stream(context) {
1253-
return Ok(StreamType::PartialHash(PartialHashAggregateStream::new(
1254-
self, context, partition,
1255-
)?));
1256-
}
1246+
// Grouping sets are expanded by the raw-input stages, so the stages
1247+
// consuming partial state must receive the expanded keys as a plain
1248+
// group by (see `PhysicalGroupBy::as_final`).
1249+
if self.mode.input_mode() == AggregateInputMode::Partial
1250+
&& !self.group_by.is_single()
1251+
{
1252+
return internal_err!(
1253+
"Grouping sets must be expanded before {:?} aggregation, which consumes partial state",
1254+
self.mode
1255+
);
1256+
}
12571257

1258-
if self.should_use_partial_reduce_hash_stream(context) {
1259-
return Ok(StreamType::PartialReduceHash(
1260-
PartialReduceHashAggregateStream::new(self, context, partition)?,
1261-
));
1258+
// Choose the execution path based on (aggregation mode, ordering).
1259+
//
1260+
// Note that `self.input_order_mode` represents both input ordering and output
1261+
// order promise. See its comment for details.
1262+
use AggregateMode::*;
1263+
use InputOrderMode::*;
1264+
let stream = match (self.mode, &self.input_order_mode) {
1265+
(Partial, Linear) => StreamType::PartialHash(
1266+
PartialHashAggregateStream::new(self, context, partition)?,
1267+
),
1268+
(Partial, Sorted | PartiallySorted(_)) => {
1269+
StreamType::OrderedPartialAggregate(OrderedPartialAggregateStream::new(
1270+
self, context, partition,
1271+
)?)
12621272
}
1263-
1264-
if self.should_use_ordered_final_aggregate_stream(context) {
1265-
return Ok(StreamType::OrderedFinalAggregate(
1266-
OrderedFinalAggregateStream::new(self, context, partition)?,
1267-
));
1273+
(PartialReduce, Linear) => StreamType::PartialReduceHash(
1274+
PartialReduceHashAggregateStream::new(self, context, partition)?,
1275+
),
1276+
(PartialReduce, Sorted | PartiallySorted(_)) => {
1277+
// See the comment above: the builder enforces `Linear` order for
1278+
// `PartialReduce` mode.
1279+
return internal_err!(
1280+
"PartialReduce aggregation must use InputOrderMode::Linear"
1281+
);
12681282
}
1269-
1270-
if self.should_use_final_hash_stream(context) {
1271-
return Ok(StreamType::FinalHash(FinalHashAggregateStream::new(
1283+
(Final | FinalPartitioned, Linear) => StreamType::FinalHash(
1284+
FinalHashAggregateStream::new(self, context, partition)?,
1285+
),
1286+
(Final | FinalPartitioned, Sorted | PartiallySorted(_)) => {
1287+
StreamType::OrderedFinalAggregate(OrderedFinalAggregateStream::new(
12721288
self, context, partition,
1273-
)?));
1289+
)?)
12741290
}
1275-
1276-
if self.should_use_ordered_single_aggregate_stream(context) {
1277-
return Ok(StreamType::OrderedSingleAggregate(
1278-
OrderedSingleAggregateStream::new(self, context, partition)?,
1279-
));
1280-
}
1281-
1282-
if self.should_use_single_hash_stream(context) {
1283-
return Ok(StreamType::SingleHash(SingleHashAggregateStream::new(
1291+
(Single | SinglePartitioned, Linear) => StreamType::SingleHash(
1292+
SingleHashAggregateStream::new(self, context, partition)?,
1293+
),
1294+
(Single | SinglePartitioned, Sorted | PartiallySorted(_)) => {
1295+
StreamType::OrderedSingleAggregate(OrderedSingleAggregateStream::new(
12841296
self, context, partition,
1285-
)?));
1297+
)?)
12861298
}
1287-
1288-
return internal_err!(
1289-
"All aggregate cases should be able to handle by the new path"
1290-
);
1291-
}
1292-
1293-
// `enable_migration_aggregate` is disabled: fall back to the legacy
1294-
// implementation, which handles every grouped execution path itself.
1295-
Ok(StreamType::GroupedHash(GroupedHashAggregateStream::new(
1296-
self, context, partition,
1297-
)?))
1298-
}
1299-
1300-
// # Grouping sets
1301-
//
1302-
// `GROUPING SETS`, `CUBE` and `ROLLUP` are expanded by the raw-input stages
1303-
// (partial and single aggregation), which evaluate every grouping set of
1304-
// each input batch into the same hash table. They are always planned with
1305-
// `InputOrderMode::Linear` (see `try_new_with_schema`), so they only reach
1306-
// the unordered hash streams. State-input stages (final and partial-reduce
1307-
// aggregation) must receive the expanded keys as a plain group by (see
1308-
// `PhysicalGroupBy::as_final`), which their predicates check explicitly.
1309-
1310-
fn should_use_partial_hash_stream(&self, _context: &TaskContext) -> bool {
1311-
self.mode == AggregateMode::Partial
1312-
&& self.input_order_mode == InputOrderMode::Linear
1313-
&& !self.group_by.is_true_no_grouping()
1314-
}
1315-
1316-
fn should_use_ordered_partial_aggregate_stream(
1317-
&self,
1318-
_context: &TaskContext,
1319-
) -> bool {
1320-
self.mode == AggregateMode::Partial
1321-
&& self.input_order_mode != InputOrderMode::Linear
1322-
&& !self.group_by.is_true_no_grouping()
1323-
}
1324-
1325-
fn should_use_final_hash_stream(&self, _context: &TaskContext) -> bool {
1326-
matches!(
1327-
self.mode,
1328-
AggregateMode::Final | AggregateMode::FinalPartitioned
1329-
) && self.input_order_mode == InputOrderMode::Linear
1330-
&& !self.group_by.is_true_no_grouping()
1331-
&& self.group_by.is_single()
1332-
}
1333-
1334-
fn should_use_partial_reduce_hash_stream(&self, _context: &TaskContext) -> bool {
1335-
self.mode == AggregateMode::PartialReduce
1336-
&& self.input_order_mode == InputOrderMode::Linear
1337-
&& !self.group_by.is_true_no_grouping()
1338-
&& self.group_by.is_single()
1339-
}
1340-
1341-
fn should_use_single_hash_stream(&self, _context: &TaskContext) -> bool {
1342-
matches!(
1343-
self.mode,
1344-
AggregateMode::Single | AggregateMode::SinglePartitioned
1345-
) && self.input_order_mode == InputOrderMode::Linear
1346-
&& !self.group_by.is_true_no_grouping()
1347-
}
1348-
1349-
fn should_use_ordered_single_aggregate_stream(&self, _context: &TaskContext) -> bool {
1350-
matches!(
1351-
self.mode,
1352-
AggregateMode::Single | AggregateMode::SinglePartitioned
1353-
) && self.input_order_mode != InputOrderMode::Linear
1354-
&& !self.group_by.is_true_no_grouping()
1355-
}
1356-
1357-
fn should_use_ordered_final_aggregate_stream(&self, _context: &TaskContext) -> bool {
1358-
matches!(
1359-
self.mode,
1360-
AggregateMode::Final | AggregateMode::FinalPartitioned
1361-
) && self.input_order_mode != InputOrderMode::Linear
1362-
&& !self.group_by.is_true_no_grouping()
1363-
&& self.group_by.is_single()
1299+
};
1300+
Ok(stream)
13641301
}
13651302

13661303
/// Finds the DataType and SortDirection for this Aggregate, if there is one

0 commit comments

Comments
 (0)