Skip to content

Commit 3ab72e6

Browse files
authored
fix: leave memory for aggregate spill replay (apache#25383)
## Which issue does this PR close? Prerequisite for apache#25172, which fixes `FairSpillPool` accounting across reservations belonging to the same memory consumer. This PR prepares aggregate spill replay to work within that shared allowance and should merge first. ## Rationale for this change **Spilling an aggregation to disk only helps if there is enough memory to read it back and finish the aggregation.** During spill replay, DataFusion merges sorted spill files and feeds their rows into an aggregate that combines the intermediate states into final results. The merge needs buffers for its inputs and output, while the aggregate needs memory for group keys and accumulators. Those allocations coexist: producing the next batch and processing it are both part of the same operator's memory budget. For example, consider: ```sql SELECT customer_id, ARRAY_AGG(event_id) FROM events GROUP BY customer_id; ``` With many customers and a small memory limit, the aggregate can spill several times. A customer's events may then be spread across several spill files. Merging those files brings the customer's intermediate states together, but the aggregate still has to build the final array. Even if the merge delivers one row at a time, that array grows until the customer's group is complete. Small input batches therefore do not eliminate the need for replay memory. The current merge tries to merge as many spill files as its reservation permits, without leaving room for this downstream work. As a simplified example, suppose the operator has a 100 MiB allowance and no other reservations. An 80 MiB merge fits by itself, but if the aggregate then needs 24 MiB to process its output, their combined 104 MiB does not fit. A pool enforcing the shared allowance must reject that growth, even though using a smaller merge could let the same aggregation make progress. This is particularly relevant to apache#25172. The merge and aggregate use separate reservations under the same memory consumer. Today, `FairSpillPool` checks those reservations individually, which can hide their combined usage exceeding the consumer's allowance. Once apache#25172 enforces that allowance across both reservations, replay must already leave space for the aggregate. This PR addresses that requirement before the accounting fix lands. ## What changes are included in this PR? **Aggregate spill merges now leave room for the aggregate consuming their output.** When choosing how many spill files to merge and how much to buffer, the merge asks the configured memory pool to admit its buffers plus an equal amount of spare capacity for replay. Once admitted, it releases the spare reservation before replay starts, retaining only the reservation for the merge buffers. This uses the pool's existing allocation checks, so the decision follows its admission policy, including any fair-share restrictions, without adding a public memory-pool API. In the example above, the 80 MiB merge would first need approval for 160 MiB and would be rejected. A smaller 40 MiB merge would need approval for 80 MiB; after returning the 40 MiB of temporary headroom, it would leave 60 MiB available, enough for the illustrative 24 MiB of aggregate state. The equal-sized headroom is a practical budgeting policy, not an estimate of the exact accumulator size or a guarantee that every aggregation will fit. The existing multi-level merge adapts by merging fewer files at once, reducing read-ahead, or splitting oversized spill batches into smaller batches. This PR makes those adjustments account for replay as well. It also handles the point where a batch cannot shrink further: a single wide row may still fit the real pool even when equal headroom does not. In that case, only the minimum merge may retry without the extra headroom, with read-ahead disabled and all merge memory still subject to the pool's checks. Inspecting decoded batches before rewriting a spill file avoids unnecessary writes when the rows already fit or cannot be split, including when the original spill files fill the disk quota. All supported aggregate replay implementations use this policy, including hash and ordered aggregation and the legacy implementation selected by `enable_migration_aggregate=false`. The legacy path also releases unused initial grouping capacity before replay. The headroom policy is private to aggregate replay; ordinary sort callers retain their existing admission behavior. ## Are these changes tested? Regression tests exercise replay across the supported aggregate implementations; the migrated hash and ordered paths also run with another registered spilling consumer. In particular, the legacy `ARRAY_AGG` test uses 64 groups with 64 values per group, one-row batches, and an 8 KiB pool to check that accumulator state can grow across replay batches. The tests verify exact aggregate results, reservation bounds, and release of memory and spill files. Merge-level coverage checks oversized batches, short and odd-sized batches, indivisible rows, decoded string-view sizes, and spill files that already fill the disk quota. It also checks that temporary headroom is released after a candidate merge is rejected and that the fallback still fails when the actual merge cannot fit. <details> <summary>Validation results and environment</summary> Validated replay alone, with the existing `FairSpillPool` implementation: - 2,263 physical-plan tests and 2,281 core/CLI tests passed. - The existing permanent-pressure regression passed with its original success expectation. - All 520 SQL logic test files passed. - `cargo fmt --all` and strict all-target/all-feature Clippy passed. - Full `./dev/rust_lint.sh` passed, including private Rust documentation and local Markdown links. Local validation uses Rust 1.98.1 and upstream revision `22651d24` with its unchanged dependency lockfile; newer main's dependency versions are unavailable in the local registry. Merge compatibility with current main is checked separately, and GitHub CI validates its merged revision. </details> ## Are there any user-facing changes? Aggregations that spill leave memory available for processing the merged rows, reducing avoidable replay failures under constrained memory. Achieving this can require smaller batches or additional merge passes. SQL semantics and public APIs are unchanged. The temporary headroom reservations can increase recorded reservation peaks without allocating additional data buffers. That headroom is released before replay, so other concurrent allocations or aggregate state that outgrows the available memory can still cause `ResourcesExhausted`.
1 parent 98b26c5 commit 3ab72e6

8 files changed

Lines changed: 964 additions & 82 deletions

File tree

‎datafusion/physical-plan/src/aggregates/grouped_hash_stream.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1376,6 +1376,7 @@ impl GroupedHashAggregateStream {
13761376
.with_metrics(self.baseline_metrics.clone())
13771377
.with_batch_size(self.batch_size)
13781378
.with_reservation(self.reservation.new_empty())
1379+
.with_replay_headroom()
13791380
.build()?;
13801381
self.input_done = false;
13811382

@@ -1403,6 +1404,9 @@ impl GroupedHashAggregateStream {
14031404
// to ensure we don't spill the spilled data to disk again.
14041405
self.oom_mode = OutOfMemoryMode::ReportError;
14051406

1407+
// Release unused initial capacity from recreated group values so it
1408+
// does not consume the memory available for spill replay.
1409+
self.group_values.clear_shrink(0);
14061410
self.update_memory_reservation()?;
14071411

14081412
ExecutionState::ReadingInput

‎datafusion/physical-plan/src/aggregates/hash_stream.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,7 @@ impl FinalSpillContext {
348348
.with_metrics(baseline_metrics.intermediate())
349349
.with_batch_size(batch_size)
350350
.with_reservation(merge_reservation)
351+
.with_replay_headroom()
351352
.build()?;
352353
let replay = OrderedFinalAggregateStream::new_with_input_and_metrics(
353354
&final_agg,

‎datafusion/physical-plan/src/aggregates/mod.rs‎

Lines changed: 437 additions & 49 deletions
Large diffs are not rendered by default.

‎datafusion/physical-plan/src/aggregates/ordered_final_stream.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,7 @@ impl OrderedFinalSpillContext {
243243
.with_metrics(baseline_metrics.intermediate())
244244
.with_batch_size(batch_size)
245245
.with_reservation(merge_reservation)
246+
.with_replay_headroom()
246247
.build()?;
247248
let replay = OrderedFinalAggregateStream::new_with_input_and_metrics(
248249
&agg,

‎datafusion/physical-plan/src/aggregates/ordered_single_stream.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,7 @@ impl OrderedSingleSpillContext {
303303
.with_metrics(baseline_metrics.intermediate())
304304
.with_batch_size(batch_size)
305305
.with_reservation(merge_reservation)
306+
.with_replay_headroom()
306307
.build()?;
307308
let replay = OrderedFinalAggregateStream::new_with_input_and_metrics(
308309
&final_agg,

‎datafusion/physical-plan/src/aggregates/single_stream.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -329,6 +329,7 @@ impl SingleSpillContext {
329329
.with_metrics(baseline_metrics.intermediate())
330330
.with_batch_size(batch_size)
331331
.with_reservation(merge_reservation)
332+
.with_replay_headroom()
332333
.build()?;
333334
let replay = OrderedFinalAggregateStream::new_with_input_and_metrics(
334335
&final_agg,

‎datafusion/physical-plan/src/sorts/multi_level_merge.rs‎

Lines changed: 508 additions & 33 deletions
Large diffs are not rendered by default.

‎datafusion/physical-plan/src/sorts/streaming_merge.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@ pub struct StreamingMergeBuilder<'a> {
9696
fetch: Option<usize>,
9797
reservation: Option<MemoryReservation>,
9898
merge_pool: Option<Arc<MergeMemoryPool>>,
99+
/// Leave memory for the aggregate consuming the merged spill rows.
100+
reserve_replay_headroom: bool,
99101
enable_round_robin_tie_breaker: bool,
100102
}
101103

@@ -161,6 +163,13 @@ impl<'a> StreamingMergeBuilder<'a> {
161163
self
162164
}
163165

166+
/// Leave room for aggregate replay by checking that the pool can admit
167+
/// merge buffers plus equal headroom. Release the headroom before replay.
168+
pub(crate) fn with_replay_headroom(mut self) -> Self {
169+
self.reserve_replay_headroom = true;
170+
self
171+
}
172+
164173
/// See [SortPreservingMergeExec::with_round_robin_repartition] for more
165174
/// information.
166175
///
@@ -194,6 +203,7 @@ impl<'a> StreamingMergeBuilder<'a> {
194203
batch_size,
195204
reservation,
196205
merge_pool,
206+
reserve_replay_headroom,
197207
fetch,
198208
expressions,
199209
enable_round_robin_tie_breaker,
@@ -235,6 +245,7 @@ impl<'a> StreamingMergeBuilder<'a> {
235245
enable_round_robin_tie_breaker,
236246
)
237247
.with_merge_pool(merge_pool)
248+
.with_replay_headroom(reserve_replay_headroom)
238249
.create_spillable_merge_stream());
239250
}
240251

0 commit comments

Comments
 (0)