Skip to content

Commit b464997

Browse files
committed
fix: remove the spill replay pool now that DataFusion 56 has #25383
DataFusion 56 includes apache/datafusion#25383, which leaves room for a final aggregate to read its spill files back, so SpillReplayPool is no longer needed. It is also now harmful: #25383 holds replay headroom on a sibling reservation while the spill merge picks its files, so the pool took every refused read-buffer request for a replay and granted it. The merge then opened every spill file and starved the next aggregate, which failed the #6880 PartialMerge test with a 2 MiB pool. The #6254 and #6857 read-back tests, which the pool was added for, pass without it. The sort-merge join #6880 test now uses an 8 MiB pool, because with DataFusion 56 its PartialMerge fits in 12 MiB or more without spilling. Closes #6583.
1 parent 9156480 commit b464997

6 files changed

Lines changed: 18 additions & 428 deletions

File tree

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 3 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -300,15 +300,12 @@ the inside out, a Comet plan in the default configuration sees:
300300
```text
301301
[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
302302
[TaskSharedMemoryPool] <- RAII handle for the per-task registry
303-
[SpillReplayPool] <- lets a spilled final aggregate read its spill files back
304-
[TrackConsumersPool] <- DataFusion; names the top 10 consumers in error messages
305-
[CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
303+
[TrackConsumersPool] <- DataFusion; names the top 10 consumers in error messages
304+
[CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
306305
```
307306

308307
Each decorator forwards every `MemoryPool` method to its inner pool, so `reserved()` at any level
309-
reports the base pool's number. `SpillReplayPool` also turns one kind of refused `try_grow` into a
310-
`grow`; see
311-
[Final aggregates reading their spill files back](#final-aggregates-reading-their-spill-files-back).
308+
reports the base pool's number.
312309

313310
### The unified pools
314311

@@ -463,18 +460,6 @@ hands every native plan in a task the same one and drops it when the task comple
463460
covers the whole task, so `CometExecIterator.close()` warns about memory still in use only when
464461
the task's last open native plan closes.
465462

466-
### Final aggregates reading their spill files back
467-
468-
`SpillReplayPool` (`spill_replay.rs`) wraps both Comet pools to work around
469-
[issue #6254](https://github.com/apache/datafusion-comet/issues/6254). Once one of DataFusion 55's
470-
final aggregates has spilled, it reads its spill files back through an aggregate that cannot spill,
471-
so a refused `try_grow` there fails the task. When the pool refuses such a request,
472-
`SpillReplayPool` records it with the pool's `grow` instead, which skips `CometFairMemoryPool`'s
473-
local checks and carries what Spark does not grant as overcommit. Every other refusal is passed on
474-
unchanged. `spill_replay.rs` describes how the wrapper recognizes these requests and what that relies
475-
on in DataFusion. [Issue #6583](https://github.com/apache/datafusion-comet/issues/6583) tracks
476-
removing it.
477-
478463
## How DataFusion consumes the pool
479464

480465
Native operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservation` API:

‎native/core/src/execution/memory_pools/fair_pool.rs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ use std::{
2121
};
2222

2323
use super::spark_memory::SparkMemory;
24-
use super::spill_replay::CLUSTERED_FINAL_AGGREGATE;
2524
use super::task_shared::RunningTasks;
2625
use datafusion::common::resources_err;
2726
use datafusion::execution::memory_pool::MemoryConsumer;
@@ -154,9 +153,13 @@ impl Display for CometFairMemoryPool {
154153
}
155154
}
156155

156+
/// The prefix of the name DataFusion gives the consumer of a `ClusteredFinalAggregateStream`, the
157+
/// final aggregate over input sorted on some or all of its keys.
158+
const CLUSTERED_FINAL_AGGREGATE: &str = "ClusteredFinalAggregateStream[";
159+
157160
/// Whether a request from `consumer` is held to its fair limit and the task's share.
158161
///
159-
/// That is the case for every consumer that can spill, with one exception. DataFusion 55.2
162+
/// That is the case for every consumer that can spill, with one exception. DataFusion
160163
/// registers `ClusteredFinalAggregateStream` as able to spill even when its input is sorted on every
161164
/// grouping key, to work around apache/datafusion#17334, but it builds a spill path only for
162165
/// partially sorted input. Fully sorted, as in a `CometSortAggregateExec` over a native sort, a

‎native/core/src/execution/memory_pools/mod.rs‎

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ mod fair_pool;
2020
pub mod logging_pool;
2121
mod plan_pool;
2222
mod spark_memory;
23-
mod spill_replay;
2423
mod task_shared;
2524
#[cfg(test)]
2625
pub(crate) mod testing;
@@ -30,7 +29,6 @@ use datafusion::execution::memory_pool::{MemoryPool, TrackConsumersPool, Unbound
3029
use fair_pool::CometFairMemoryPool;
3130
use jni::objects::{Global, JObject};
3231
use spark_memory::SparkMemory;
33-
use spill_replay::SpillReplayPool;
3432
use std::num::NonZeroUsize;
3533
use std::sync::Arc;
3634
use unified_pool::CometUnifiedMemoryPool;
@@ -94,22 +92,16 @@ fn create_pool(
9492
match pool_type {
9593
MemoryPoolType::GreedyUnified => {
9694
acquire_task_shared_pool(task_attempt_id, running_tasks, || {
97-
Arc::new(SpillReplayPool::new(
98-
task_attempt_id,
99-
tracked(CometUnifiedMemoryPool::with_spark(spark())),
100-
))
95+
tracked(CometUnifiedMemoryPool::with_spark(spark()))
10196
})
10297
}
10398
MemoryPoolType::FairUnified => {
10499
acquire_task_shared_pool(task_attempt_id, running_tasks, || {
105-
Arc::new(SpillReplayPool::new(
106-
task_attempt_id,
107-
tracked(CometFairMemoryPool::with_spark(
108-
spark(),
109-
pool_size,
110-
off_heap_size,
111-
running_tasks.clone(),
112-
)),
100+
tracked(CometFairMemoryPool::with_spark(
101+
spark(),
102+
pool_size,
103+
off_heap_size,
104+
running_tasks.clone(),
113105
))
114106
})
115107
}
@@ -123,7 +115,6 @@ fn create_pool(
123115
/// takes nothing from Spark or that the function did not create.
124116
pub(crate) fn overcommit(pool: &Arc<dyn MemoryPool>) -> usize {
125117
let pool = task_shared::unwrap_task_shared(pool).unwrap_or(pool);
126-
let pool = spill_replay::unwrap_spill_replay(pool).unwrap_or(pool);
127118
if let Some(tracked) = pool.downcast_ref::<TrackConsumersPool<CometUnifiedMemoryPool>>() {
128119
tracked.inner().overcommit()
129120
} else if let Some(tracked) = pool.downcast_ref::<TrackConsumersPool<CometFairMemoryPool>>() {

0 commit comments

Comments
 (0)