Skip to content

Commit 039c0b9

Browse files
authored
test: cover the memory accounting paths of the grouped aggregate streams that no test reaches (#26562)
This PR is the safety net for #26563, which moves the memory rules of the seven grouped aggregate streams into one shared type. On main, 14 of the 26 ways I found to break those rules fail no test, so that refactor (or any later change to these streams) could silently change how much memory a query reserves and still pass CI. These tests record every pool call each stream makes on those paths, so #26563 can show it changes nothing by leaving them unchanged. Please review and merge this one first. ## Which issue does this PR close? - None. This adds tests only. ## Rationale for this change The grouped aggregate streams decide how much memory to reserve in several places: when they spill, when they replay their spill files, when they hand out a materialized batch in slices, and when the pool refuses a request. To find out which of these decisions the existing tests protect, I broke them one at a time on main and ran the physical-plan tests, the `memory_limit` and aggregate fuzz integration tests, and the sqllogictests. Of 26 such changes, 14 broke no test: - Final, Single and ClusteredFinal: dropping the 4 bytes per group reserved for the sort indices of a spill. - Final: no longer shrinking the reservation after a spill, or no longer releasing it before the replay. - ClusteredSingle: no longer releasing the reservation before the replay. - Single with temporary files disabled: ignoring a refused reservation instead of returning the error. - Final: registering as spillable even when temporary files are disabled. - Final returning, and Single ignoring, a refusal while materializing the final output (each reacts the other way on main). - ClusteredPartial: returning an error instead of emitting the completed groups whole when they do not fit next to the table. - ClusteredPartial with full clustering: emitting the incomplete groups on a refusal instead of only the completed ones. - ClusteredFinal and ClusteredSingle: no longer reserving the completed groups while they are handed out. A pool refusal that is wrongly ignored or wrongly returned is a query that runs past its memory limit or fails when it should succeed, so these paths deserve tests before anyone changes them. ## What changes are included in this PR? One test file, `tests/cases/aggregate_memory_call_sequence.rs`, in a new `physical_plan_integration` test binary, with 8 tests. Each runs one `AggregateExec` under a pool that records every call in order (register with its `can_spill`, grow, try_grow and whether it was refused, shrink, unregister), and snapshots that sequence together with the query result. Each test also asserts through metrics or output batch sizes that it really took its path (it spilled, or emitted whole batches, and so on). The snapshots hide byte counts. They follow the accounting of hash tables and arrays, so they would change with every hashbrown or arrow upgrade, and they differ between x86_64 and aarch64. A change in accounting still shows up as a missing or extra call, a flipped refusal or `can_spill`, or a different outcome. The one byte count that is a rule of its own, the 4 bytes per group reserved for the sort indices of a spill, has its own test, which compares runs with temporary files enabled and disabled. The Final and Single streams account in the same way, so their tests run both and assert that they make the same calls with the same byte counts. Refused output materialization needs an aggregate whose output is larger than the memory it accounts for, which built-in aggregates only reach by a small margin (`string_agg`, see #26701), so the file defines a small test UDAF for it. The early emit of the unclustered Partial and PartialReduce streams is left out: their own unit tests already catch every change I made there. ## What is the testing strategy for this PR? The PR is the tests. Each of the 14 changes above makes at least one of them fail, and all of them pass on main. ## Are there any user-facing changes? No.
1 parent 7032b23 commit 039c0b9

3 files changed

Lines changed: 934 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)