Repository navigation
Commit ad16004
authored
fix: emit completed groups when fully sorted final and single aggrega… (apache#26637)
## Which issue does this PR close?
- Closes apache#26628.
## Rationale for this change
A `Final` or `Single` aggregate over input sorted on all grouping keys
fails with `ResourcesExhausted` under a memory limit that the legacy
`GroupedHashAggregateStream` handles. After each input batch,
`ClusteredFinalAggregateStream` and `ClusteredSingleAggregateStream`
reserve memory for the whole table before emitting completed groups. If
that reservation is refused and the stream cannot spill (full group
clustering, or temporary files disabled), it returns the error, even
though emitting the completed groups would leave only the active group
in memory.
This is the same problem as apache#26575 for the partial stream, fixed in
apache#26576.
## What changes are included in this PR?
- `ClusteredFinalAggregateStream`: when the reservation is refused and
there is no spill context, take the completed groups and move to the
output stage, then resume aggregation. The error is returned only when
no group is complete.
- `ClusteredSingleAggregateStream`: the same behavior, emitting through
the existing `start_outputting` path and resuming `ReadingInput`.
- Updated doc comments to describe the new behavior.
The spilling path (partial group clustering with temporary files
enabled) is unchanged.
## What is the testing strategy for this PR?
New unit tests, modeled on the one in apache#26576:
-
`clustered_final_stream::tests::completed_groups_under_memory_pressure`
-
`clustered_single_stream::tests::fully_sorted_completed_groups_under_memory_pressure`
Each runs `SUM(k) GROUP BY k` over two batches sorted on `k` that share
a boundary key. It first runs with an unbounded pool to measure the most
memory the stream holds when it hands off output, then reruns with a
`GreedyMemoryPool` of that size and checks that the results match. Both
tests fail on `main` with `ResourcesExhausted` and pass with this
change. All `aggregates` unit tests in `datafusion-physical-plan` pass,
and `cargo fmt` and `cargo clippy --all-targets --all-features -D
warnings` on the crate are clean.
## Are there any user-facing changes?
No API changes. Queries that used to fail with `ResourcesExhausted` in
this situation now complete.
---
_This solution was developed with AI assistance; I reviewed and tested
the changes._1 parent f362a82 commit ad16004
5 files changed
Lines changed: 311 additions & 113 deletions
File tree
- datafusion/physical-plan/src/aggregates
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
264 | | - | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
265 | 267 | | |
266 | 268 | | |
267 | 269 | | |
| |||
336 | 338 | | |
337 | 339 | | |
338 | 340 | | |
339 | | - | |
340 | | - | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
341 | 346 | | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
347 | 364 | | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
356 | | - | |
357 | 365 | | |
358 | 366 | | |
359 | | - | |
| 367 | + | |
360 | 368 | | |
361 | 369 | | |
362 | 370 | | |
363 | 371 | | |
364 | 372 | | |
365 | 373 | | |
366 | 374 | | |
367 | | - | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
368 | 379 | | |
369 | 380 | | |
370 | 381 | | |
| |||
460 | 471 | | |
461 | 472 | | |
462 | 473 | | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
463 | 479 | | |
464 | 480 | | |
465 | 481 | | |
| |||
732 | 748 | | |
733 | 749 | | |
734 | 750 | | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
735 | 768 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
422 | 422 | | |
423 | 423 | | |
424 | 424 | | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
430 | | - | |
431 | | - | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
432 | 428 | | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | 429 | | |
440 | | - | |
441 | | - | |
442 | | - | |
443 | | - | |
444 | | - | |
445 | | - | |
446 | | - | |
447 | | - | |
448 | | - | |
449 | | - | |
450 | | - | |
451 | | - | |
452 | | - | |
453 | | - | |
454 | | - | |
455 | | - | |
456 | | - | |
457 | | - | |
458 | | - | |
459 | | - | |
460 | | - | |
461 | | - | |
462 | | - | |
463 | | - | |
464 | | - | |
465 | | - | |
466 | | - | |
467 | | - | |
468 | | - | |
469 | | - | |
470 | | - | |
471 | | - | |
472 | | - | |
473 | | - | |
474 | | - | |
475 | | - | |
476 | | - | |
477 | | - | |
478 | | - | |
479 | | - | |
480 | | - | |
481 | | - | |
482 | | - | |
483 | | - | |
484 | | - | |
485 | | - | |
486 | | - | |
487 | | - | |
488 | | - | |
489 | | - | |
490 | | - | |
491 | | - | |
492 | 430 | | |
493 | | - | |
494 | | - | |
495 | | - | |
496 | | - | |
497 | | - | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
498 | 435 | | |
499 | 436 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
86 | | - | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
87 | 88 | | |
88 | 89 | | |
89 | 90 | | |
| |||
329 | 330 | | |
330 | 331 | | |
331 | 332 | | |
332 | | - | |
333 | | - | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
334 | 338 | | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
344 | 356 | | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
349 | | - | |
350 | | - | |
351 | 357 | | |
352 | 358 | | |
353 | 359 | | |
354 | 360 | | |
355 | | - | |
| 361 | + | |
356 | 362 | | |
357 | 363 | | |
358 | 364 | | |
| |||
368 | 374 | | |
369 | 375 | | |
370 | 376 | | |
371 | | - | |
| 377 | + | |
372 | 378 | | |
373 | 379 | | |
374 | | - | |
| 380 | + | |
375 | 381 | | |
376 | 382 | | |
377 | 383 | | |
| |||
385 | 391 | | |
386 | 392 | | |
387 | 393 | | |
| 394 | + | |
388 | 395 | | |
389 | | - | |
| 396 | + | |
390 | 397 | | |
391 | 398 | | |
392 | 399 | | |
393 | 400 | | |
394 | 401 | | |
395 | | - | |
| 402 | + | |
396 | 403 | | |
397 | 404 | | |
398 | 405 | | |
| |||
762 | 769 | | |
763 | 770 | | |
764 | 771 | | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
765 | 777 | | |
766 | 778 | | |
767 | 779 | | |
| |||
851 | 863 | | |
852 | 864 | | |
853 | 865 | | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
854 | 883 | | |
0 commit comments