Repository navigation
Commit 039c0b9
authored
test: cover the memory accounting paths of the grouped aggregate streams that no test reaches (#26562)
This PR is the safety net for #26563, which moves the memory rules of
the seven grouped aggregate streams into one shared type. On main, 14 of
the 26 ways I found to break those rules fail no test, so that refactor
(or any later change to these streams) could silently change how much
memory a query reserves and still pass CI. These tests record every pool
call each stream makes on those paths, so #26563 can show it changes
nothing by leaving them unchanged. Please review and merge this one
first.
## Which issue does this PR close?
- None. This adds tests only.
## Rationale for this change
The grouped aggregate streams decide how much memory to reserve in
several places: when they spill, when they replay their spill files,
when they hand out a materialized batch in slices, and when the pool
refuses a request. To find out which of these decisions the existing
tests protect, I broke them one at a time on main and ran the
physical-plan tests, the `memory_limit` and aggregate fuzz integration
tests, and the sqllogictests. Of 26 such changes, 14 broke no test:
- Final, Single and ClusteredFinal: dropping the 4 bytes per group
reserved for the sort indices of a spill.
- Final: no longer shrinking the reservation after a spill, or no longer
releasing it before the replay.
- ClusteredSingle: no longer releasing the reservation before the
replay.
- Single with temporary files disabled: ignoring a refused reservation
instead of returning the error.
- Final: registering as spillable even when temporary files are
disabled.
- Final returning, and Single ignoring, a refusal while materializing
the final output (each reacts the other way on main).
- ClusteredPartial: returning an error instead of emitting the completed
groups whole when they do not fit next to the table.
- ClusteredPartial with full clustering: emitting the incomplete groups
on a refusal instead of only the completed ones.
- ClusteredFinal and ClusteredSingle: no longer reserving the completed
groups while they are handed out.
A pool refusal that is wrongly ignored or wrongly returned is a query
that runs past its memory limit or fails when it should succeed, so
these paths deserve tests before anyone changes them.
## What changes are included in this PR?
One test file, `tests/cases/aggregate_memory_call_sequence.rs`, in a new
`physical_plan_integration` test binary, with 8 tests. Each runs one
`AggregateExec` under a pool that records every call in order (register
with its `can_spill`, grow, try_grow and whether it was refused, shrink,
unregister), and snapshots that sequence together with the query result.
Each test also asserts through metrics or output batch sizes that it
really took its path (it spilled, or emitted whole batches, and so on).
The snapshots hide byte counts. They follow the accounting of hash
tables and arrays, so they would change with every hashbrown or arrow
upgrade, and they differ between x86_64 and aarch64. A change in
accounting still shows up as a missing or extra call, a flipped refusal
or `can_spill`, or a different outcome. The one byte count that is a
rule of its own, the 4 bytes per group reserved for the sort indices of
a spill, has its own test, which compares runs with temporary files
enabled and disabled. The Final and Single streams account in the same
way, so their tests run both and assert that they make the same calls
with the same byte counts.
Refused output materialization needs an aggregate whose output is larger
than the memory it accounts for, which built-in aggregates only reach by
a small margin (`string_agg`, see #26701), so the file defines a small
test UDAF for it.
The early emit of the unclustered Partial and PartialReduce streams is
left out: their own unit tests already catch every change I made there.
## What is the testing strategy for this PR?
The PR is the tests. Each of the 14 changes above makes at least one of
them fail, and all of them pass on main.
## Are there any user-facing changes?
No.1 parent 7032b23 commit 039c0b9
3 files changed
Lines changed: 934 additions & 0 deletions
File tree
- datafusion/physical-plan/tests
- cases
0 commit comments