Repository navigation
Commit 3ab72e6
authored
fix: leave memory for aggregate spill replay (apache#25383)
## Which issue does this PR close?
Prerequisite for apache#25172, which fixes `FairSpillPool` accounting across
reservations belonging to the same memory consumer. This PR prepares
aggregate spill replay to work within that shared allowance and should
merge first.
## Rationale for this change
**Spilling an aggregation to disk only helps if there is enough memory
to read it back and finish the aggregation.** During spill replay,
DataFusion merges sorted spill files and feeds their rows into an
aggregate that combines the intermediate states into final results. The
merge needs buffers for its inputs and output, while the aggregate needs
memory for group keys and accumulators. Those allocations coexist:
producing the next batch and processing it are both part of the same
operator's memory budget.
For example, consider:
```sql
SELECT customer_id, ARRAY_AGG(event_id)
FROM events
GROUP BY customer_id;
```
With many customers and a small memory limit, the aggregate can spill
several times. A customer's events may then be spread across several
spill files. Merging those files brings the customer's intermediate
states together, but the aggregate still has to build the final array.
Even if the merge delivers one row at a time, that array grows until the
customer's group is complete. Small input batches therefore do not
eliminate the need for replay memory.
The current merge tries to merge as many spill files as its reservation
permits, without leaving room for this downstream work. As a simplified
example, suppose the operator has a 100 MiB allowance and no other
reservations. An 80 MiB merge fits by itself, but if the aggregate then
needs 24 MiB to process its output, their combined 104 MiB does not fit.
A pool enforcing the shared allowance must reject that growth, even
though using a smaller merge could let the same aggregation make
progress.
This is particularly relevant to apache#25172. The merge and aggregate use
separate reservations under the same memory consumer. Today,
`FairSpillPool` checks those reservations individually, which can hide
their combined usage exceeding the consumer's allowance. Once apache#25172
enforces that allowance across both reservations, replay must already
leave space for the aggregate. This PR addresses that requirement before
the accounting fix lands.
## What changes are included in this PR?
**Aggregate spill merges now leave room for the aggregate consuming
their output.** When choosing how many spill files to merge and how much
to buffer, the merge asks the configured memory pool to admit its
buffers plus an equal amount of spare capacity for replay. Once
admitted, it releases the spare reservation before replay starts,
retaining only the reservation for the merge buffers. This uses the
pool's existing allocation checks, so the decision follows its admission
policy, including any fair-share restrictions, without adding a public
memory-pool API.
In the example above, the 80 MiB merge would first need approval for 160
MiB and would be rejected. A smaller 40 MiB merge would need approval
for 80 MiB; after returning the 40 MiB of temporary headroom, it would
leave 60 MiB available, enough for the illustrative 24 MiB of aggregate
state. The equal-sized headroom is a practical budgeting policy, not an
estimate of the exact accumulator size or a guarantee that every
aggregation will fit.
The existing multi-level merge adapts by merging fewer files at once,
reducing read-ahead, or splitting oversized spill batches into smaller
batches. This PR makes those adjustments account for replay as well. It
also handles the point where a batch cannot shrink further: a single
wide row may still fit the real pool even when equal headroom does not.
In that case, only the minimum merge may retry without the extra
headroom, with read-ahead disabled and all merge memory still subject to
the pool's checks. Inspecting decoded batches before rewriting a spill
file avoids unnecessary writes when the rows already fit or cannot be
split, including when the original spill files fill the disk quota.
All supported aggregate replay implementations use this policy,
including hash and ordered aggregation and the legacy implementation
selected by `enable_migration_aggregate=false`. The legacy path also
releases unused initial grouping capacity before replay. The headroom
policy is private to aggregate replay; ordinary sort callers retain
their existing admission behavior.
## Are these changes tested?
Regression tests exercise replay across the supported aggregate
implementations; the migrated hash and ordered paths also run with
another registered spilling consumer. In particular, the legacy
`ARRAY_AGG` test uses 64 groups with 64 values per group, one-row
batches, and an 8 KiB pool to check that accumulator state can grow
across replay batches. The tests verify exact aggregate results,
reservation bounds, and release of memory and spill files.
Merge-level coverage checks oversized batches, short and odd-sized
batches, indivisible rows, decoded string-view sizes, and spill files
that already fill the disk quota. It also checks that temporary headroom
is released after a candidate merge is rejected and that the fallback
still fails when the actual merge cannot fit.
<details>
<summary>Validation results and environment</summary>
Validated replay alone, with the existing `FairSpillPool`
implementation:
- 2,263 physical-plan tests and 2,281 core/CLI tests passed.
- The existing permanent-pressure regression passed with its original
success expectation.
- All 520 SQL logic test files passed.
- `cargo fmt --all` and strict all-target/all-feature Clippy passed.
- Full `./dev/rust_lint.sh` passed, including private Rust documentation
and local Markdown links.
Local validation uses Rust 1.98.1 and upstream revision `22651d24` with
its unchanged dependency lockfile; newer main's dependency versions are
unavailable in the local registry. Merge compatibility with current main
is checked separately, and GitHub CI validates its merged revision.
</details>
## Are there any user-facing changes?
Aggregations that spill leave memory available for processing the merged
rows, reducing avoidable replay failures under constrained memory.
Achieving this can require smaller batches or additional merge passes.
SQL semantics and public APIs are unchanged.
The temporary headroom reservations can increase recorded reservation
peaks without allocating additional data buffers. That headroom is
released before replay, so other concurrent allocations or aggregate
state that outgrows the available memory can still cause
`ResourcesExhausted`.1 parent 98b26c5 commit 3ab72e6
8 files changed
Lines changed: 964 additions & 82 deletions
File tree
- datafusion/physical-plan/src
- aggregates
- sorts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1376 | 1376 | | |
1377 | 1377 | | |
1378 | 1378 | | |
| 1379 | + | |
1379 | 1380 | | |
1380 | 1381 | | |
1381 | 1382 | | |
| |||
1403 | 1404 | | |
1404 | 1405 | | |
1405 | 1406 | | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
1406 | 1410 | | |
1407 | 1411 | | |
1408 | 1412 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
348 | 348 | | |
349 | 349 | | |
350 | 350 | | |
| 351 | + | |
351 | 352 | | |
352 | 353 | | |
353 | 354 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
243 | 243 | | |
244 | 244 | | |
245 | 245 | | |
| 246 | + | |
246 | 247 | | |
247 | 248 | | |
248 | 249 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
303 | 303 | | |
304 | 304 | | |
305 | 305 | | |
| 306 | + | |
306 | 307 | | |
307 | 308 | | |
308 | 309 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
329 | 329 | | |
330 | 330 | | |
331 | 331 | | |
| 332 | + | |
332 | 333 | | |
333 | 334 | | |
334 | 335 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
| 99 | + | |
| 100 | + | |
99 | 101 | | |
100 | 102 | | |
101 | 103 | | |
| |||
161 | 163 | | |
162 | 164 | | |
163 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
164 | 173 | | |
165 | 174 | | |
166 | 175 | | |
| |||
194 | 203 | | |
195 | 204 | | |
196 | 205 | | |
| 206 | + | |
197 | 207 | | |
198 | 208 | | |
199 | 209 | | |
| |||
235 | 245 | | |
236 | 246 | | |
237 | 247 | | |
| 248 | + | |
238 | 249 | | |
239 | 250 | | |
240 | 251 | | |
| |||
0 commit comments