Repository navigation
Commit 22eb8ff
authored
fix: spill final hash aggregate output when the pool refuses it (#26734)
## Which issue does this PR close?
- Closes #26701.
## Rationale for this change
`FinalHashAggregateStream` fails with `ResourcesExhausted` when the
materialized output is bigger than the table it came from (e.g.
`string_agg`) and the pool won't grow the reservation. The other streams
carry on.
## What changes are included in this PR?
When that resize is refused and temp files are enabled, the stream
writes the materialized output to one spill file, drops it, frees the
reservation and streams the batches back. Without temp files it still
errors, now with a message saying it couldn't spill the output.
## What is the testing strategy for this PR?
Updated the `Final` case in `output_materialization_refused` (from
#26562): it now spills once and returns all 256 rows in 4 batches.
Aggregate unit tests and the aggregate slt files pass.
## Are there any user-facing changes?
No, queries that hit this now succeed instead of erroring.
---
This was AI-assisted, I've reviewed the changes myself.1 parent 70c2e4c commit 22eb8ff
3 files changed
Lines changed: 83 additions & 12 deletions
File tree
- datafusion/physical-plan
- src/aggregates
- tests/cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
619 | 619 | | |
620 | 620 | | |
621 | 621 | | |
622 | | - | |
| 622 | + | |
623 | 623 | | |
624 | | - | |
| 624 | + | |
625 | 625 | | |
626 | 626 | | |
627 | 627 | | |
628 | 628 | | |
629 | | - | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
630 | 633 | | |
631 | 634 | | |
632 | 635 | | |
| |||
806 | 809 | | |
807 | 810 | | |
808 | 811 | | |
| 812 | + | |
809 | 813 | | |
810 | 814 | | |
811 | 815 | | |
| |||
825 | 829 | | |
826 | 830 | | |
827 | 831 | | |
828 | | - | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
829 | 859 | | |
830 | 860 | | |
831 | 861 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
236 | 236 | | |
237 | 237 | | |
238 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
239 | 260 | | |
240 | 261 | | |
241 | 262 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
| 170 | + | |
170 | 171 | | |
171 | 172 | | |
172 | 173 | | |
| |||
326 | 327 | | |
327 | 328 | | |
328 | 329 | | |
329 | | - | |
| 330 | + | |
330 | 331 | | |
331 | 332 | | |
332 | 333 | | |
| |||
335 | 336 | | |
336 | 337 | | |
337 | 338 | | |
| 339 | + | |
338 | 340 | | |
339 | 341 | | |
340 | 342 | | |
| |||
578 | 580 | | |
579 | 581 | | |
580 | 582 | | |
581 | | - | |
582 | | - | |
583 | | - | |
584 | | - | |
585 | | - | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
586 | 587 | | |
587 | 588 | | |
588 | 589 | | |
| |||
597 | 598 | | |
598 | 599 | | |
599 | 600 | | |
600 | | - | |
| 601 | + | |
| 602 | + | |
601 | 603 | | |
602 | 604 | | |
603 | 605 | | |
604 | 606 | | |
605 | 607 | | |
606 | 608 | | |
607 | 609 | | |
608 | | - | |
| 610 | + | |
609 | 611 | | |
610 | 612 | | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
611 | 631 | | |
612 | 632 | | |
613 | 633 | | |
| |||
0 commit comments