Skip to content

Commit 8a92281

Browse files
authored
Refine and document AggregateExec metrics (apache#24757)
## Which issue does this PR close? * Part of apache#23570 ## Rationale for this change Aggregate-specific metrics are a user-visible observability surface in `EXPLAIN ANALYZE`, but their phase boundaries and relationship to individual aggregate expressions need to be clear and consistent. In particular, users should be able to understand what each aggregate timer measures, map indexed metrics such as `agg_expr_0_arguments_time` back to the corresponding expression in the operator's `aggr=[...]` list, and understand which metrics are available on different grouped aggregation paths without reading the implementation. This PR also tightens the timing boundaries of the grouped aggregate metrics so that the documented metrics correspond more directly to the work they describe. ## What changes are included in this PR? * Documents `AggregateExec` operator-level and per-aggregate metrics, including their phase boundaries, indexed naming convention, aggregate expression labels, visibility, and path-specific coverage. * Groups aggregate timing into explicit phases for: * group-key preparation, * aggregate argument and filter evaluation, * accumulator `update` / `merge` operations, and * output emission. * Ensures accumulator `aggregation_time` measures a single interval containing the relevant accumulator operations rather than accumulating overlapping/prefix timing. * Includes group interning and ordering setup in `time_calculating_group_ids`. * Includes collectively evaluated aggregate filters in `aggregate_arguments_time`. * Adds `topk_maintenance_time` for Grouped TopK priority-map maintenance and avoids registering `aggregation_time` for that accumulator-free path. * Times grouped output materialization consistently with `emitting_time`. * Adds `EXPLAIN ANALYZE` coverage demonstrating how indexed per-aggregate metrics map to multiple aggregate expressions and how expression labels appear in verbose output. ## Are these changes tested? Yes. The patch adds and updates tests that verify: * `EXPLAIN ANALYZE` reports `agg_expr_0_arguments_time`, `agg_expr_1_arguments_time`, and `agg_expr_2_arguments_time` for a query containing `SUM(c5)`, `SUM(c6)`, and `COUNT(c7)`. * `EXPLAIN ANALYZE VERBOSE` associates those indexed metrics with the corresponding rendered aggregate expressions. * Grouped TopK reports `topk_maintenance_time` and `time_calculating_group_ids` while not registering `aggregation_time`. * Regular aggregate paths do not report `topk_maintenance_time`. * Aggregate execution without grouped emission does not report `emitting_time`. * `GroupByMetrics::new_topk` does not register `aggregation_time`. ## Are there any user-facing changes? Yes. Aggregate metrics shown through `EXPLAIN ANALYZE` are clarified and their timing boundaries are made more consistent. The metrics documentation now describes the `AggregateExec` operator-level metrics and per-expression timer phases, explains how indexed metrics map to aggregate expressions, and documents path-specific differences such as Grouped TopK and non-grouped aggregation. These are observability and documentation changes; this PR does not introduce a public API change. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 84ccbb1 commit 8a92281

9 files changed

Lines changed: 706 additions & 410 deletions

File tree

‎datafusion/core/tests/sql/explain_analyze.rs‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1064,6 +1064,47 @@ async fn csv_explain_analyze_verbose() {
10641064
assert_contains!(formatted, verbose_needle);
10651065
}
10661066

1067+
#[tokio::test]
1068+
#[cfg_attr(coverage, ignore)]
1069+
async fn explain_analyze_aggregate_metrics_map_indices_to_expressions() {
1070+
let ctx =
1071+
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(1));
1072+
register_aggregate_csv_by_sql(&ctx).await;
1073+
1074+
let query =
1075+
"SELECT c1, SUM(c5), SUM(c6), COUNT(c7) FROM aggregate_test_100 GROUP BY c1";
1076+
let normal = execute_to_batches(&ctx, &format!("EXPLAIN ANALYZE {query}")).await;
1077+
let normal = arrow::util::pretty::pretty_format_batches(&normal)
1078+
.unwrap()
1079+
.to_string();
1080+
assert_contains!(
1081+
normal.as_str(),
1082+
"aggr=[sum(aggregate_test_100.c5), sum(aggregate_test_100.c6), count(aggregate_test_100.c7)]"
1083+
);
1084+
assert_contains!(normal.as_str(), "agg_expr_0_arguments_time");
1085+
assert_contains!(normal.as_str(), "agg_expr_1_arguments_time");
1086+
assert_contains!(normal.as_str(), "agg_expr_2_arguments_time");
1087+
assert!(!normal.contains("aggregate="));
1088+
1089+
let verbose =
1090+
execute_to_batches(&ctx, &format!("EXPLAIN ANALYZE VERBOSE {query}")).await;
1091+
let verbose = arrow::util::pretty::pretty_format_batches(&verbose)
1092+
.unwrap()
1093+
.to_string();
1094+
assert_contains!(
1095+
verbose.as_str(),
1096+
"agg_expr_0_arguments_time{partition=0, aggregate=sum(aggregate_test_100.c5)}"
1097+
);
1098+
assert_contains!(
1099+
verbose.as_str(),
1100+
"agg_expr_1_arguments_time{partition=0, aggregate=sum(aggregate_test_100.c6)}"
1101+
);
1102+
assert_contains!(
1103+
verbose.as_str(),
1104+
"agg_expr_2_arguments_time{partition=0, aggregate=count(aggregate_test_100.c7)}"
1105+
);
1106+
}
1107+
10671108
#[tokio::test]
10681109
async fn explain_logical_plan_only() {
10691110
let mut config = ConfigOptions::new();

‎datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs‎

Lines changed: 113 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -173,24 +173,23 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
173173
batch: &RecordBatch,
174174
) -> Result<EvaluatedAggregateBatch> {
175175
let state = self.state.building();
176-
let timer = self.group_by_metrics.time_calculating_group_ids.timer();
177-
// outer vec: one per each grouping set
178-
// inner vec: all group by exprs for the current grouping set
179-
let grouping_set_args = evaluate_group_by(&state.group_by, batch)?;
180-
drop(timer);
181-
182-
let timer = self.group_by_metrics.aggregate_arguments_time.timer();
183-
// The evaluated args for each accumulator
184-
let accumulator_args = state
185-
.accumulators
186-
.iter()
187-
.enumerate()
188-
.map(|(idx, acc)| {
189-
self.aggregate_argument_metrics
190-
.time(idx, || acc.evaluate_acc_args(batch))
191-
})
192-
.collect::<Result<Vec<_>>>()?;
193-
drop(timer);
176+
// Outer vec: one per grouping set; inner vec: group-by expressions.
177+
let grouping_set_args = self
178+
.group_by_metrics
179+
.time_group_key_preparation(|| evaluate_group_by(&state.group_by, batch))?;
180+
181+
// The evaluated args for each accumulator.
182+
let accumulator_args = self.group_by_metrics.time_aggregate_arguments(|| {
183+
state
184+
.accumulators
185+
.iter()
186+
.enumerate()
187+
.map(|(idx, acc)| {
188+
self.aggregate_argument_metrics
189+
.time(idx, || acc.evaluate_acc_args(batch))
190+
})
191+
.collect::<Result<Vec<_>>>()
192+
})?;
194193

195194
Ok(EvaluatedAggregateBatch {
196195
grouping_set_args,
@@ -212,26 +211,31 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
212211
) -> Result<()> {
213212
let evaluated_batch = self.evaluate_batch(batch)?;
214213
let accumulator_metrics = Arc::clone(&self.aggregate_accumulator_metrics);
214+
let group_by_metrics = self.group_by_metrics.clone();
215215
let state = self.state.building_mut();
216216

217-
let _timer = self.group_by_metrics.aggregation_time.timer();
218217
for group_values in &evaluated_batch.grouping_set_args {
219-
state
220-
.group_values
221-
.intern(group_values, &mut state.batch_group_indices)?;
218+
group_by_metrics.time_group_key_preparation(|| {
219+
state
220+
.group_values
221+
.intern(group_values, &mut state.batch_group_indices)
222+
})?;
223+
222224
let group_indices = &state.batch_group_indices;
223225
let total_num_groups = state.group_values.len();
224-
225-
for (idx, (acc, values)) in state
226-
.accumulators
227-
.iter_mut()
228-
.zip(evaluated_batch.accumulator_args.iter())
229-
.enumerate()
230-
{
231-
accumulator_metrics.time(idx, accumulator_phase, || {
232-
aggregate_fn(acc, values, group_indices, total_num_groups)
233-
})?;
234-
}
226+
group_by_metrics.time_aggregation(|| {
227+
for (idx, (acc, values)) in state
228+
.accumulators
229+
.iter_mut()
230+
.zip(evaluated_batch.accumulator_args.iter())
231+
.enumerate()
232+
{
233+
accumulator_metrics.time(idx, accumulator_phase, || {
234+
aggregate_fn(acc, values, group_indices, total_num_groups)
235+
})?;
236+
}
237+
Ok::<(), datafusion_common::DataFusionError>(())
238+
})?;
235239
}
236240

237241
Ok(())
@@ -265,16 +269,17 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
265269
// Accumulator output consumes internal state. Materialize all
266270
// groups once, then slice the materialized batch on later polls.
267271
let emit_to = EmitTo::All;
268-
let timer = self.group_by_metrics.emitting_time.timer();
269-
let mut columns = state.group_values.emit(emit_to)?;
270-
for (idx, acc) in state.accumulators.iter_mut().enumerate() {
271-
columns.extend(accumulator_metrics.time(
272-
idx,
273-
accumulator_phase,
274-
|| materialize_accumulator_fn(acc, emit_to),
275-
)?);
276-
}
277-
drop(timer);
272+
let columns = self.group_by_metrics.time_emitting(|| {
273+
let mut columns = state.group_values.emit(emit_to)?;
274+
for (idx, acc) in state.accumulators.iter_mut().enumerate() {
275+
columns.extend(accumulator_metrics.time(
276+
idx,
277+
accumulator_phase,
278+
|| materialize_accumulator_fn(acc, emit_to),
279+
)?);
280+
}
281+
Ok::<_, datafusion_common::DataFusionError>(columns)
282+
})?;
278283

279284
let batch = RecordBatch::try_new(output_schema, columns)?;
280285
debug_assert!(batch.num_rows() > 0);
@@ -334,19 +339,23 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
334339
) -> Result<Option<RecordBatch>> {
335340
let state_schema = Arc::clone(&self.state_schema);
336341
let accumulator_metrics = Arc::clone(&self.aggregate_accumulator_metrics);
342+
let group_by_metrics = self.group_by_metrics.clone();
337343
let state = self.state.building_mut();
338344
if state.group_values.is_empty() {
339345
return Ok(None);
340346
}
341347

342-
let mut output = state.group_values.emit(EmitTo::All)?;
343-
for (idx, acc) in state.accumulators.iter_mut().enumerate() {
344-
output.extend(accumulator_metrics.time(
345-
idx,
346-
AccumulatorPhase::State,
347-
|| acc.state(EmitTo::All),
348-
)?);
349-
}
348+
let output = group_by_metrics.time_emitting(|| {
349+
let mut output = state.group_values.emit(EmitTo::All)?;
350+
for (idx, acc) in state.accumulators.iter_mut().enumerate() {
351+
output.extend(accumulator_metrics.time(
352+
idx,
353+
AccumulatorPhase::State,
354+
|| acc.state(EmitTo::All),
355+
)?);
356+
}
357+
Ok::<_, datafusion_common::DataFusionError>(output)
358+
})?;
350359

351360
let batch = RecordBatch::try_new(state_schema, output)?;
352361
debug_assert!(batch.num_rows() > 0);
@@ -401,57 +410,71 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
401410
/// already expanded keys as plain group columns (see
402411
/// [`PhysicalGroupBy::as_final`]) and never own a grouping set.
403412
pub(super) fn init_empty_grouping_sets(&mut self) -> Result<()> {
413+
let group_by_metrics = self.group_by_metrics.clone();
404414
let state = self.state.building_mut();
405415
if !state.group_by.has_grouping_set() || !state.group_values.is_empty() {
406416
return Ok(());
407417
}
408418

409419
let accumulator_metrics = Arc::clone(&self.aggregate_accumulator_metrics);
410-
let max_ordinal = max_duplicate_ordinal(state.group_by.groups());
411-
let mut ordinals: HashMap<&[bool], usize> = HashMap::new();
412-
let group_schema = state.group_by.group_schema(&self.input_schema)?;
413-
let n_expr = state.group_by.expr().len();
414-
let mut any_interned = false;
415-
416-
for group in state.group_by.groups() {
417-
let ordinal = {
418-
let entry = ordinals.entry(group.as_slice()).or_insert(0);
419-
let ordinal = *entry;
420-
*entry += 1;
421-
ordinal
422-
};
423-
424-
if !group.iter().all(|&is_null| is_null) {
425-
continue;
426-
}
420+
let any_interned = group_by_metrics.time_group_key_preparation(|| {
421+
let max_ordinal = max_duplicate_ordinal(state.group_by.groups());
422+
let mut ordinals: HashMap<&[bool], usize> = HashMap::new();
423+
let group_schema = state.group_by.group_schema(&self.input_schema)?;
424+
let n_expr = state.group_by.expr().len();
425+
let mut any_interned = false;
426+
427+
for group in state.group_by.groups() {
428+
let ordinal = {
429+
let entry = ordinals.entry(group.as_slice()).or_insert(0);
430+
let ordinal = *entry;
431+
*entry += 1;
432+
ordinal
433+
};
427434

428-
let mut cols: Vec<ArrayRef> = group_schema
429-
.fields()
430-
.iter()
431-
.take(n_expr)
432-
.map(|field| new_null_array(field.data_type(), 1))
433-
.collect();
434-
cols.push(group_id_array(group, ordinal, max_ordinal, 1)?);
435+
if !group.iter().all(|&is_null| is_null) {
436+
continue;
437+
}
435438

436-
state
437-
.group_values
438-
.intern(&cols, &mut state.batch_group_indices)?;
439-
any_interned = true;
440-
}
439+
let mut cols: Vec<ArrayRef> = group_schema
440+
.fields()
441+
.iter()
442+
.take(n_expr)
443+
.map(|field| new_null_array(field.data_type(), 1))
444+
.collect();
445+
cols.push(group_id_array(group, ordinal, max_ordinal, 1)?);
446+
447+
state
448+
.group_values
449+
.intern(&cols, &mut state.batch_group_indices)?;
450+
any_interned = true;
451+
}
452+
Ok::<_, datafusion_common::DataFusionError>(any_interned)
453+
})?;
441454

442455
if any_interned {
443456
let total_groups = state.group_values.len();
444457
let false_filter = BooleanArray::from(vec![false]);
445-
for (idx, acc) in state.accumulators.iter_mut().enumerate() {
446-
let null_args = acc.null_arguments(&self.input_schema)?;
447-
let values = EvaluatedAccumulatorArgs {
448-
arguments: null_args,
449-
filter: Some(Arc::new(false_filter.clone())),
450-
};
451-
accumulator_metrics.time(idx, AccumulatorPhase::Update, || {
452-
acc.update_batch(&values, &[0], total_groups)
453-
})?;
454-
}
458+
let values = state
459+
.accumulators
460+
.iter()
461+
.map(|acc| {
462+
Ok(EvaluatedAccumulatorArgs {
463+
arguments: acc.null_arguments(&self.input_schema)?,
464+
filter: Some(Arc::new(false_filter.clone())),
465+
})
466+
})
467+
.collect::<Result<Vec<_>>>()?;
468+
group_by_metrics.time_aggregation(|| {
469+
for (idx, (acc, values)) in
470+
state.accumulators.iter_mut().zip(values.iter()).enumerate()
471+
{
472+
accumulator_metrics.time(idx, AccumulatorPhase::Update, || {
473+
acc.update_batch(values, &[0], total_groups)
474+
})?;
475+
}
476+
Ok::<(), datafusion_common::DataFusionError>(())
477+
})?;
455478
}
456479

457480
Ok(())

0 commit comments

Comments
 (0)