Repository navigation
Commit cd8cef9
authored
Add per-aggregate accumulator phase metrics (apache#24423)
## Which issue does this PR close?
* Part of apache#23570
## Rationale for this change
Aggregate argument evaluation metrics only measure the cost of producing
input arrays and do not show how much time individual aggregate
expressions spend performing accumulator work.
This PR adds per-aggregate-expression timing for accumulator phases in
migrated grouped hash aggregation, making it possible to attribute time
spent updating, merging, emitting state, converting input to state, and
evaluating accumulators to the aggregate expression performing that
work. Existing shared group-by aggregation and emission timing remains
in place.
## What changes are included in this PR?
* Adds `AggregateAccumulatorMetrics`, which creates per-aggregate timers
for accumulator phases.
* Adds phase-specific metrics for `update`, `merge`, `state`,
`convert_to_state`, and `evaluate`.
* Creates accumulator metrics once per aggregate table and shares them
with the migrated hash table implementations.
* Instruments accumulator operations at the per-aggregate/per-batch
boundary rather than per row or group.
* Adds instrumentation to both unordered and ordered migrated aggregate
tables.
* Defines the accumulator phases expected for each `AggregateMode`.
* Preserves the existing `GroupByMetrics` and aggregate argument
metrics.
* Instruments the skip-partial path's `convert_to_state` operation.
* Carries the accumulator metrics through partial-to-skip table
transitions and spill/replay paths.
## Are these changes tested?
Yes.
The patch adds or updates tests that:
* Verify the accumulator phases associated with each `AggregateMode`.
* Verify two aggregate expressions (`SUM(a)` / `SUM(b)` and `SUM(b)` /
`SUM(c)` in the respective tests) receive separate metric names and
`aggregate` labels.
* Verify partial aggregation records positive `update_time` and
`state_time` metrics.
* Verify final aggregation records positive `merge_time` and
`evaluate_time` metrics.
* Verify migrated ordered aggregation paths record the expected
accumulator phase timings.
* Verify skip-partial aggregation records `convert_to_state` timing.
* Verify a migrated single aggregate that spills records `update`,
`state`, `merge`, and `evaluate` timings.
* Explicitly enable `datafusion.execution.enable_migration_aggregate` in
the relevant tests.
The issue suggests the targeted verification command:
```bash
cargo test -p datafusion-physical-plan aggregates
```
The patch itself does not show that command being run, so this PR
description does not claim that it passed.
## Are there any user-facing changes?
Yes. Execution metrics for migrated grouped hash aggregation now expose
per-aggregate-expression accumulator timing, with metrics such as:
```text
agg_expr_0_update_time
agg_expr_0_merge_time
agg_expr_0_state_time
agg_expr_0_convert_to_state_time
agg_expr_0_evaluate_time
```
Each metric also carries the corresponding `aggregate` label.
There are no public accumulator API or aggregate semantics changes shown
in this patch.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.1 parent 0fc962f commit cd8cef9
13 files changed
Lines changed: 487 additions & 80 deletions
File tree
- datafusion/physical-plan/src/aggregates
- aggregate_hash_table
- group_values
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
36 | | - | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| |||
81 | 83 | | |
82 | 84 | | |
83 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
84 | 89 | | |
85 | 90 | | |
86 | 91 | | |
| |||
140 | 145 | | |
141 | 146 | | |
142 | 147 | | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
148 | | - | |
149 | | - | |
| 148 | + | |
150 | 149 | | |
151 | 150 | | |
152 | | - | |
153 | | - | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
| |||
208 | 208 | | |
209 | 209 | | |
210 | 210 | | |
| 211 | + | |
211 | 212 | | |
212 | 213 | | |
| 214 | + | |
213 | 215 | | |
214 | 216 | | |
215 | 217 | | |
| |||
220 | 222 | | |
221 | 223 | | |
222 | 224 | | |
223 | | - | |
| 225 | + | |
224 | 226 | | |
225 | 227 | | |
226 | 228 | | |
| 229 | + | |
227 | 230 | | |
228 | | - | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
229 | 234 | | |
230 | 235 | | |
231 | 236 | | |
| |||
244 | 249 | | |
245 | 250 | | |
246 | 251 | | |
| 252 | + | |
247 | 253 | | |
248 | 254 | | |
249 | 255 | | |
| 256 | + | |
250 | 257 | | |
251 | 258 | | |
252 | 259 | | |
| |||
260 | 267 | | |
261 | 268 | | |
262 | 269 | | |
263 | | - | |
264 | | - | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
265 | 276 | | |
266 | 277 | | |
267 | 278 | | |
| |||
322 | 333 | | |
323 | 334 | | |
324 | 335 | | |
| 336 | + | |
325 | 337 | | |
326 | 338 | | |
327 | 339 | | |
328 | 340 | | |
329 | 341 | | |
330 | 342 | | |
331 | | - | |
332 | | - | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
333 | 349 | | |
334 | 350 | | |
335 | 351 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
40 | | - | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
| |||
49 | 51 | | |
50 | 52 | | |
51 | 53 | | |
| 54 | + | |
52 | 55 | | |
53 | 56 | | |
54 | 57 | | |
55 | 58 | | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
| 59 | + | |
63 | 60 | | |
64 | | - | |
65 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
66 | 64 | | |
67 | 65 | | |
68 | 66 | | |
| |||
72 | 70 | | |
73 | 71 | | |
74 | 72 | | |
| 73 | + | |
75 | 74 | | |
76 | 75 | | |
77 | 76 | | |
| |||
136 | 135 | | |
137 | 136 | | |
138 | 137 | | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
139 | 141 | | |
140 | 142 | | |
141 | 143 | | |
| |||
222 | 224 | | |
223 | 225 | | |
224 | 226 | | |
| 227 | + | |
225 | 228 | | |
226 | 229 | | |
227 | 230 | | |
| |||
304 | 307 | | |
305 | 308 | | |
306 | 309 | | |
| 310 | + | |
307 | 311 | | |
308 | 312 | | |
309 | 313 | | |
| |||
321 | 325 | | |
322 | 326 | | |
323 | 327 | | |
| 328 | + | |
324 | 329 | | |
325 | | - | |
326 | | - | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
327 | 336 | | |
328 | 337 | | |
329 | 338 | | |
| |||
369 | 378 | | |
370 | 379 | | |
371 | 380 | | |
| 381 | + | |
372 | 382 | | |
| 383 | + | |
373 | 384 | | |
374 | 385 | | |
375 | 386 | | |
| |||
385 | 396 | | |
386 | 397 | | |
387 | 398 | | |
388 | | - | |
| 399 | + | |
389 | 400 | | |
390 | 401 | | |
391 | 402 | | |
392 | 403 | | |
| 404 | + | |
393 | 405 | | |
394 | | - | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
395 | 414 | | |
396 | 415 | | |
397 | 416 | | |
| |||
409 | 428 | | |
410 | 429 | | |
411 | 430 | | |
| 431 | + | |
412 | 432 | | |
413 | 433 | | |
414 | 434 | | |
| |||
420 | 440 | | |
421 | 441 | | |
422 | 442 | | |
| 443 | + | |
423 | 444 | | |
424 | 445 | | |
425 | 446 | | |
| |||
432 | 453 | | |
433 | 454 | | |
434 | 455 | | |
435 | | - | |
436 | | - | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
437 | 460 | | |
438 | 461 | | |
439 | 462 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
61 | | - | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
62 | 66 | | |
63 | 67 | | |
64 | 68 | | |
| |||
67 | 71 | | |
68 | 72 | | |
69 | 73 | | |
70 | | - | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
71 | 79 | | |
72 | 80 | | |
73 | 81 | | |
| |||
0 commit comments