Skip to content

Commit aff5850

Browse files
authored
Add per-aggregate argument evaluation metrics for grouped hash aggregation (apache#24024)
#### Which issue does this PR close? * Part of apache#23570 --- ## Rationale for this change Grouped hash aggregation currently exposes a single `aggregate_arguments_time` metric for all aggregate argument evaluation. This makes it difficult to determine which aggregate expression is responsible for argument evaluation cost when multiple aggregates are present (for example, `SUM(a)` versus `SUM(b)`). This change adds per-aggregate argument evaluation metrics while preserving the existing operator-level metric for compatibility. --- ## What changes are included in this PR? * Add an `AggregateArgumentMetrics` helper that registers one timer per aggregate expression during operator construction. * Label per-aggregate metrics using the aggregate expression display text (or alias), allowing otherwise identical aggregate functions on different inputs (for example, `SUM(a)` and `SUM(b)`) to be distinguished. * Wrap aggregate argument evaluation with the corresponding per-aggregate timer in: * grouped hash aggregation stream * migrated hash table aggregation paths * ordered aggregation table paths * Introduce `OrderedAggregateTableMetrics` to carry both group-by and aggregate argument metrics through ordered aggregation replay/spill paths. * Factor out a reusable `aggregate_metric_label` helper for consistent metric labels. * Preserve the existing `aggregate_arguments_time` metric so whole-phase timing continues to be reported. --- ## Are these changes tested? Yes. This PR adds and updates tests including: * `test_groupby_aggregate_argument_metrics_distinguish_inputs`, which verifies that separate metrics are created for `SUM(a)` and `SUM(b)` with distinct metric names and aggregate labels. * Existing group-by metrics tests updated to use shared aggregate construction helpers. * Grouped hash aggregation stream tests updated to verify both: * the existing `aggregate_arguments_time` metric remains present and non-zero, and * the new per-aggregate metric (`agg_expr_0_arguments_time`) is emitted and records time. --- ## Are there any user-facing changes? Yes. `EXPLAIN ANALYZE` and execution plan metrics for the covered grouped hash aggregation paths now include per-aggregate argument evaluation timers (for example, `agg_expr_0_arguments_time`) with aggregate labels identifying the corresponding aggregate expression, while retaining the existing `aggregate_arguments_time` metric for compatibility. --- ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 9fdf144 commit aff5850

15 files changed

Lines changed: 343 additions & 86 deletions

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,11 +27,14 @@ use datafusion_expr::{EmitTo, GroupsAccumulator};
2727
use datafusion_physical_expr::aggregate::AggregateFunctionExpr;
2828

2929
use crate::PhysicalExpr;
30-
use crate::aggregates::group_values::{GroupByMetrics, GroupValues, new_group_values};
30+
use crate::aggregates::group_values::{
31+
AggregateArgumentMetrics, GroupByMetrics, GroupValues, new_group_values,
32+
};
3133
use crate::aggregates::grouped_hash_stream::create_group_accumulator;
3234
use crate::aggregates::order::GroupOrdering;
3335
use crate::aggregates::{
34-
AggregateExec, PhysicalGroupBy, aggregate_expressions, evaluate_group_by,
36+
AggregateExec, PhysicalGroupBy, aggregate_expressions, aggregate_metric_label,
37+
evaluate_group_by,
3538
};
3639

3740
/// Marker for raw rows -> partial state aggregation.
@@ -75,6 +78,9 @@ pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
7578
/// Grouping and accumulator-specific timing metrics.
7679
pub(super) group_by_metrics: GroupByMetrics,
7780

81+
/// Per-aggregate timing metrics for evaluating aggregate arguments.
82+
pub(super) aggregate_argument_metrics: AggregateArgumentMetrics,
83+
7884
/// Raw input schema, used to evaluate expressions and synthesize empty
7985
/// grouping-set rows.
8086
pub(super) input_schema: SchemaRef,
@@ -134,8 +140,17 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
134140
let group_schema = agg.group_by.group_schema(&input_schema)?;
135141
let group_values = new_group_values(group_schema, &GroupOrdering::None)?;
136142

143+
let aggregate_argument_metrics = AggregateArgumentMetrics::new(
144+
&agg.metrics,
145+
partition,
146+
agg.aggr_expr
147+
.iter()
148+
.map(|agg_expr| aggregate_metric_label(agg_expr)),
149+
);
150+
137151
Ok(Self {
138152
group_by_metrics: GroupByMetrics::new(&agg.metrics, partition),
153+
aggregate_argument_metrics,
139154
input_schema,
140155
output_schema,
141156
state_schema,
@@ -169,7 +184,11 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
169184
.building()
170185
.accumulators
171186
.iter()
172-
.map(|acc| acc.evaluate_acc_args(batch))
187+
.enumerate()
188+
.map(|(idx, acc)| {
189+
self.aggregate_argument_metrics
190+
.time(idx, || acc.evaluate_acc_args(batch))
191+
})
173192
.collect::<Result<Vec<_>>>()?;
174193
drop(timer);
175194

@@ -288,10 +307,6 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
288307
}
289308
}
290309

291-
pub(in crate::aggregates) fn group_by_metrics(&self) -> &GroupByMetrics {
292-
&self.group_by_metrics
293-
}
294-
295310
/// Returns the number of distinct groups accumulated so far.
296311
pub(in crate::aggregates) fn building_group_count(&self) -> usize {
297312
self.state.building().group_values.len()

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/common_ordered.rs‎

Lines changed: 52 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -30,19 +30,52 @@ use datafusion_expr::EmitTo;
3030

3131
use crate::InputOrderMode;
3232
use crate::PhysicalExpr;
33-
use crate::aggregates::group_values::{GroupByMetrics, GroupValues, new_group_values};
33+
use crate::aggregates::group_values::{
34+
AggregateArgumentMetrics, GroupByMetrics, GroupValues, new_group_values,
35+
};
3436
use crate::aggregates::grouped_hash_stream::create_group_accumulator;
3537
use crate::aggregates::order::GroupOrdering;
3638
use crate::aggregates::{
3739
AggregateExec, AggregateMode, PhysicalGroupBy, aggregate_expressions,
38-
evaluate_group_by,
40+
aggregate_metric_label, evaluate_group_by,
3941
};
4042

4143
use super::common::{
42-
AggregateAccumulator, AggregateBatchFn, EvaluatedAggregateBatch,
44+
AggregateAccumulator, AggregateBatchFn, AggregateHashTable, EvaluatedAggregateBatch,
4345
MaterializeAccumulatorFn,
4446
};
4547

48+
#[derive(Clone)]
49+
pub(in crate::aggregates) struct OrderedAggregateTableMetrics {
50+
pub(super) group_by: GroupByMetrics,
51+
pub(super) aggregate_arguments: AggregateArgumentMetrics,
52+
}
53+
54+
impl OrderedAggregateTableMetrics {
55+
pub(in crate::aggregates) fn new(agg: &AggregateExec, partition: usize) -> Self {
56+
let aggregate_arguments = AggregateArgumentMetrics::new(
57+
&agg.metrics,
58+
partition,
59+
agg.aggr_expr
60+
.iter()
61+
.map(|agg_expr| aggregate_metric_label(agg_expr)),
62+
);
63+
Self {
64+
group_by: GroupByMetrics::new(&agg.metrics, partition),
65+
aggregate_arguments,
66+
}
67+
}
68+
69+
pub(in crate::aggregates) fn from_hash_table<AggrMode>(
70+
table: &AggregateHashTable<AggrMode>,
71+
) -> Self {
72+
Self {
73+
group_by: table.group_by_metrics.clone(),
74+
aggregate_arguments: table.aggregate_argument_metrics.clone(),
75+
}
76+
}
77+
}
78+
4679
/// Aggregate table shared by the ordered single, partial and final paths.
4780
///
4881
/// # Ordering optimization
@@ -100,6 +133,9 @@ pub(in crate::aggregates) struct OrderedAggregateTable<OrderedAggrMode> {
100133
/// Grouping and accumulator-specific timing metrics.
101134
pub(super) group_by_metrics: GroupByMetrics,
102135

136+
/// Per-aggregate timing metrics for evaluating aggregate arguments.
137+
pub(super) aggregate_argument_metrics: AggregateArgumentMetrics,
138+
103139
/// Group keys, ordering state, and accumulator states.
104140
pub(super) buffer: OrderedAggregateTableBuffer,
105141

@@ -149,7 +185,7 @@ impl<AggrMode> OrderedAggregateTable<AggrMode> {
149185
input_order_mode: &InputOrderMode,
150186
aggregate_mode: &AggregateMode,
151187
filters: Vec<Option<Arc<dyn PhysicalExpr>>>,
152-
group_by_metrics: GroupByMetrics,
188+
metrics: OrderedAggregateTableMetrics,
153189
) -> Result<Self> {
154190
assert_or_internal_err!(
155191
batch_size > 0,
@@ -184,7 +220,8 @@ impl<AggrMode> OrderedAggregateTable<AggrMode> {
184220
output_schema,
185221
state_schema,
186222
batch_size,
187-
group_by_metrics,
223+
group_by_metrics: metrics.group_by,
224+
aggregate_argument_metrics: metrics.aggregate_arguments,
188225
buffer: OrderedAggregateTableBuffer {
189226
group_by: Arc::clone(&agg.group_by),
190227
group_ordering,
@@ -213,7 +250,11 @@ impl<AggrMode> OrderedAggregateTable<AggrMode> {
213250
.buffer
214251
.accumulators
215252
.iter()
216-
.map(|acc| acc.evaluate_acc_args(batch))
253+
.enumerate()
254+
.map(|(idx, acc)| {
255+
self.aggregate_argument_metrics
256+
.time(idx, || acc.evaluate_acc_args(batch))
257+
})
217258
.collect::<Result<Vec<_>>>()?;
218259
drop(timer);
219260

@@ -259,8 +300,11 @@ impl<AggrMode> OrderedAggregateTable<AggrMode> {
259300
+ self.buffer.group_indices.allocated_size()
260301
}
261302

262-
pub(in crate::aggregates) fn group_by_metrics(&self) -> GroupByMetrics {
263-
self.group_by_metrics.clone()
303+
pub(in crate::aggregates) fn metrics(&self) -> OrderedAggregateTableMetrics {
304+
OrderedAggregateTableMetrics {
305+
group_by: self.group_by_metrics.clone(),
306+
aggregate_arguments: self.aggregate_argument_metrics.clone(),
307+
}
264308
}
265309

266310
/// Takes every intermediate aggregate state and resets the table so it can

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,4 @@ pub(super) use common::{
2929
AggregateHashTable, FinalMarker, PartialMarker, PartialReduceMarker,
3030
PartialSkipMarker, SingleMarker,
3131
};
32-
pub(super) use common_ordered::OrderedAggregateTable;
32+
pub(super) use common_ordered::{OrderedAggregateTable, OrderedAggregateTableMetrics};

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/ordered_final_table.rs‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,11 +27,10 @@ use datafusion_common::Result;
2727

2828
use crate::InputOrderMode;
2929
use crate::aggregates::aggregate_hash_table::FinalMarker;
30-
use crate::aggregates::group_values::GroupByMetrics;
3130
use crate::aggregates::{AggregateExec, AggregateMode};
3231

3332
use super::common::HashAggregateAccumulator;
34-
use super::common_ordered::OrderedAggregateTable;
33+
use super::common_ordered::{OrderedAggregateTable, OrderedAggregateTableMetrics};
3534

3635
/// Implementation specific to final aggregation, where the table stores partial
3736
/// aggregate states and the input rows are also partial states.
@@ -49,7 +48,7 @@ impl OrderedAggregateTable<FinalMarker> {
4948
output_schema: SchemaRef,
5049
batch_size: usize,
5150
input_order_mode: &InputOrderMode,
52-
group_by_metrics: GroupByMetrics,
51+
metrics: OrderedAggregateTableMetrics,
5352
) -> Result<Self> {
5453
Self::new_for_mode(
5554
agg,
@@ -60,7 +59,7 @@ impl OrderedAggregateTable<FinalMarker> {
6059
input_order_mode,
6160
&AggregateMode::Final,
6261
vec![None; agg.aggr_expr.len()],
63-
group_by_metrics,
62+
metrics,
6463
)
6564
}
6665

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/ordered_partial_table.rs‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,10 @@ use datafusion_common::Result;
3737

3838
use crate::aggregates::{
3939
AggregateExec, AggregateMode, aggregate_hash_table::PartialMarker,
40-
group_values::GroupByMetrics,
4140
};
4241

4342
use super::common::HashAggregateAccumulator;
44-
use super::common_ordered::OrderedAggregateTable;
43+
use super::common_ordered::{OrderedAggregateTable, OrderedAggregateTableMetrics};
4544

4645
/// Implementation specific to partial aggregation, where the table stores
4746
/// partial aggregate states and the input rows are raw rows.
@@ -61,7 +60,7 @@ impl OrderedAggregateTable<PartialMarker> {
6160
) -> Result<Self> {
6261
let input_schema = agg.input().schema();
6362
let state_schema = Arc::clone(&output_schema);
64-
let group_by_metrics = GroupByMetrics::new(&agg.metrics, partition);
63+
let metrics = OrderedAggregateTableMetrics::new(agg, partition);
6564
Self::new_for_mode(
6665
agg,
6766
&input_schema,
@@ -71,7 +70,7 @@ impl OrderedAggregateTable<PartialMarker> {
7170
&agg.input_order_mode,
7271
&AggregateMode::Partial,
7372
agg.filter_expr.iter().cloned().collect(),
74-
group_by_metrics,
73+
metrics,
7574
)
7675
}
7776

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/ordered_single_table.rs‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,10 @@ use arrow::record_batch::RecordBatch;
2424
use datafusion_common::Result;
2525

2626
use crate::aggregates::aggregate_hash_table::SingleMarker;
27-
use crate::aggregates::group_values::GroupByMetrics;
2827
use crate::aggregates::{AggregateExec, AggregateMode};
2928

3029
use super::common::HashAggregateAccumulator;
31-
use super::common_ordered::OrderedAggregateTable;
30+
use super::common_ordered::{OrderedAggregateTable, OrderedAggregateTableMetrics};
3231

3332
/// Implementation specific to single aggregation, where the table stores final
3433
/// aggregate values and the input rows are raw rows.
@@ -53,7 +52,7 @@ impl OrderedAggregateTable<SingleMarker> {
5352
));
5453

5554
let input_schema = agg.input().schema();
56-
let group_by_metrics = GroupByMetrics::new(&agg.metrics, partition);
55+
let metrics = OrderedAggregateTableMetrics::new(agg, partition);
5756
Self::new_for_mode(
5857
agg,
5958
&input_schema,
@@ -63,7 +62,7 @@ impl OrderedAggregateTable<SingleMarker> {
6362
&agg.input_order_mode,
6463
&agg.mode,
6564
agg.filter_expr.iter().cloned().collect(),
66-
group_by_metrics,
65+
metrics,
6766
)
6867
}
6968

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/partial_table.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ impl AggregateHashTable<PartialMarker> {
8686

8787
Ok(AggregateHashTable {
8888
group_by_metrics: self.group_by_metrics.clone(),
89+
aggregate_argument_metrics: self.aggregate_argument_metrics.clone(),
8990
input_schema: Arc::clone(&self.input_schema),
9091
output_schema: Arc::clone(&self.output_schema),
9192
state_schema: Arc::clone(&self.state_schema),

0 commit comments

Comments
 (0)