You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 31ddcdf
Browse filesBrowse the repository at this point in the historyBrowse files
## Which issue does this PR close?
* Closesapache#23570
This is the last of a series of PRs to close the issue
## Rationale for this change
Existing per-aggregate operation metrics attribute complete `update`,
`merge`, `state`, and `evaluate` calls to each aggregate expression, but
they cannot show how much time an aggregate spends in meaningful
internal subphases.
This PR adds a separate contract for optional aggregate-owned submetrics
so implementations can expose internal timing while retaining a clear
owning aggregate expression and stable metric identity across repeated
aggregates and partitions. The existing aggregate operation timers
remain separate from these internal submetrics.
`array_agg(DISTINCT ...)` is used as the representative implementation,
exposing time spent in its distinct-value handling.
## What changes are included in this PR?
* Adds `AggregateMetric` and `AggregateMetrics` interfaces for
aggregate-owned internal metrics, plus `AggregateMetricRecorder` for
recording one or more timed intervals into a submetric.
* Adds optional metric injection to `Accumulator` and
`GroupsAccumulator` while preserving default behavior for
implementations that do not expose submetrics.
* Adds grouped update/merge hooks so `GroupsAccumulatorAdapter` can time
aggregate-owned work once across grouped dispatch rather than recording
a timer for every group.
* Registers aggregate submetrics lazily using stable names of the form
`agg_expr_{index}_internal_{subphase}_time` and retains the aggregate
label and partition identity.
* Propagates submetric factories through hash, ordered, and streaming
aggregate execution paths, including newly created/empty accumulators.
* Instruments `DistinctArrayAggAccumulator` with a `distinct` submetric
covering its distinct-value handling during updates and merges.
* Keeps adapter-owned work such as slicing/filtering, size accounting,
and state materialization outside the aggregate-owned submetric.
* Updates grouped adapter allocation accounting so retained group-index
scratch capacity is charged when the underlying vectors grow.
* Adds a high-cardinality `array_agg(DISTINCT)` SQL benchmark with 1M
groups and two rows per group.
* Adds focused `GroupsAccumulatorAdapter`, `nth_value`, and
`approx_percentile_cont` benchmarks to measure the grouped adapter path.
* Adds `Time::add_duration_exact` so aggregate submetrics can accumulate
durations without rounding every individual recording up to one
nanosecond.
## Are these changes tested?
Yes. The patch adds tests covering:
* metric propagation to grouped accumulators;
* grouped update and `convert_to_state` metric recording;
* exclusion of adapter-owned size accounting and state materialization
from internal timing;
* metric recording on update errors;
* allocation accounting and cleanup after grouped updates and errors;
* `array_agg(DISTINCT)` update and merge metric recording;
* metric identity reuse and support for multiple subphases;
* zero-duration metric handling;
* aggregate submetric naming and labels for repeated aggregate
expressions;
* aggregation and partition identity across multiple partitions;
* aggregate execution with `array_agg(DISTINCT)` submetrics;
* preservation of existing update/evaluate aggregate metrics;
* compatibility with aggregates that do not expose internal submetrics.
The patch also adds benchmarks for the uninstrumented grouped
accumulator adapter path, ordered `nth_value`, `approx_percentile_cont`,
and a 1M-group `array_agg(DISTINCT)` SQL workload.
## Are there any user-facing changes?
`EXPLAIN ANALYZE`/aggregate execution metrics can now include optional
aggregate-owned internal timing metrics for aggregates that expose them.
In this PR, `array_agg(DISTINCT ...)` exposes an `internal_distinct`
timing metric.
Aggregates that do not expose internal submetrics retain their existing
behavior and metrics.
This PR also extends the public accumulator interfaces with optional
metric-related methods that have default implementations.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.
/// [Aggregating Millions of Groups Fast blog]: https://arrow.apache.org/blog/2023/08/05/datafusion_fast_grouping/
190
193
pubtraitGroupsAccumulator:Send + std::any::Any{
194
+
/// Supplies optional metrics owned by this aggregate expression.
195
+
///
196
+
/// Call this immediately after construction and before the accumulator is
197
+
/// used; replacing metrics after use is unsupported. The default preserves
198
+
/// compatibility for accumulators without internal submetrics.
0 commit comments