Skip to content

Commit 68a0148

Browse files
adriangbdiscord9
authored andcommitted
fix: RepartitionExec deadlock when its spill pool holds two open files (apache#24891)
Closes apache#24883 Under a memory limit, a grouped aggregation on two or more partitions can stop and never complete: when `RepartitionExec` cannot reserve memory for a batch it writes the batch to its spill pool and sends a marker, and the reader then blocks on the pool until it gets a batch. When two input tasks spill at the same time the pool holds two open files. The reader read only the oldest file and, when that file had no unread batch, waited for it, while the batch it needed was in the newer file. Once every channel had data the gate closed and the writers waited too, so nothing could wake the reader. `SpillPoolReader` now keeps every file it received (oldest first) and returns the first available batch. A file that has no unread batch is skipped; the reader still waits on the oldest file that has unread batches whose read is not complete, so a single writer keeps FIFO order. The writer side is unchanged. Also backported: - `test: reproduce the RepartitionExec spill pool deadlock with a SQL query` (`datafusion/core/tests/memory_limit/repartition_mem_limit.rs`), which sums the `spill_count` metric and fails if no attempt spilled. - `test: add a scenario fuzzer for the spill pool contract`, including `test_read_error_is_reported_to_the_reader`, a test that a second writer's batch is readable while the first writer is held, and `spill_pool_scenario_fuzz` (default 50 seeds). - the extended-CI job sets `DATAFUSION_SPILL_POOL_FUZZ_ITERATIONS: 1000`. Backport notes: applied by hand because `git cherry-pick -x` is not usable here (the worktree's git directory is not writable from this sandbox). All three files applied byte-identically; the only fork-specific deltas in the touched regions are pre-existing cosmetic differences (`#[deprecated]` without `since`, `sort_unstable` vs `sort`), which this commit does not touch. (cherry picked from commit 09a2aff) (backport of apache#24891, 09a2aff) Signed-off-by: discord9 <[email protected]> (cherry picked from commit ac60580950ee42b7358a74fdf1203a0dc5e84992)
1 parent a710e34 commit 68a0148

3 files changed

Lines changed: 874 additions & 93 deletions

File tree

‎.github/workflows/extended.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,9 @@ jobs:
8888
- name: Run tests (excluding doctests)
8989
env:
9090
RUST_BACKTRACE: 1
91+
# Run more random scenarios of the spill pool fuzzer than the
92+
# default suite does (about a minute).
93+
DATAFUSION_SPILL_POOL_FUZZ_ITERATIONS: 1000
9194
run: |
9295
cargo test \
9396
--profile ci \

‎datafusion/core/tests/memory_limit/repartition_mem_limit.rs‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,3 +117,157 @@ async fn test_repartition_memory_limit() {
117117
];
118118
assert_batches_sorted_eq!(expected, &all_batches);
119119
}
120+
121+
/// Regression test for <https://github.com/apache/datafusion/issues/24883>.
122+
///
123+
/// In non-preserve-order mode every input task of a `RepartitionExec` shares one
124+
/// spill pool per output partition. When two input tasks spilled concurrently the
125+
/// pool could end up with two open spill files while the reader only ever drained
126+
/// the head file: it parked on a drained-but-unfinished head file even though the
127+
/// batch it was waiting for had been written to the second file. Once every
128+
/// distributor channel was non-empty the gate closed, both input tasks parked in
129+
/// `send`, no sink was dropped, and nothing could wake anybody.
130+
///
131+
/// This module is not compiled with `force_hash_collisions`. That feature makes
132+
/// every hash 0, so the hash `RepartitionExec` sends all rows to one output
133+
/// partition. The gate closes only when every channel holds data, so the
134+
/// deadlock cannot occur and the test cannot test anything. The feature also
135+
/// makes the group-by hash table degenerate, which made one attempt of this
136+
/// query approximately 36 times slower and thus longer than the limit below.
137+
#[cfg(not(feature = "force_hash_collisions"))]
138+
mod spill_pool_deadlock {
139+
use super::*;
140+
141+
use std::time::Duration;
142+
143+
use datafusion::execution::session_state::SessionStateBuilder;
144+
use datafusion_execution::disk_manager::{DiskManagerBuilder, DiskManagerMode};
145+
use datafusion_execution::memory_pool::{GreedyMemoryPool, MemoryPool};
146+
use datafusion_physical_plan::collect;
147+
148+
/// Memory limit that puts `RepartitionExec` into its spilling path for this
149+
/// query without failing the aggregation outright.
150+
const MEMORY_LIMIT: usize = 4 * 1024 * 1024;
151+
152+
/// Number of attempts. On an unfixed tree the first attempt deadlocks; the
153+
/// budget is only there so a fix cannot pass by luck.
154+
const ATTEMPTS: usize = 12;
155+
156+
/// Generous per-attempt budget: a healthy run of this query takes well under
157+
/// a second.
158+
const PER_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(20);
159+
160+
/// Runs the query once. Returns the number of output rows and the number
161+
/// of spills the `RepartitionExec`s did.
162+
async fn run_once() -> datafusion::error::Result<(usize, usize)> {
163+
let pool: Arc<dyn MemoryPool> = Arc::new(GreedyMemoryPool::new(MEMORY_LIMIT));
164+
let runtime = RuntimeEnvBuilder::new()
165+
// Spilling must be possible: with the disk manager disabled the query
166+
// fails with a resource error instead of deadlocking.
167+
.with_disk_manager_builder(
168+
DiskManagerBuilder::default().with_mode(DiskManagerMode::OsTmpDirectory),
169+
)
170+
.with_memory_pool(pool)
171+
.build_arc()?;
172+
173+
let config = SessionConfig::new()
174+
// Two input partitions feeding one hash RepartitionExec is the
175+
// smallest configuration with two concurrent writers per spill pool.
176+
.with_target_partitions(2)
177+
// Small batches so many small batches reach the repartition spill
178+
// path while the spill file stays far below
179+
// `max_spill_file_size_bytes` and so never rotates.
180+
.with_batch_size(64);
181+
182+
let state = SessionStateBuilder::new()
183+
.with_config(config)
184+
.with_runtime_env(runtime)
185+
.with_default_features()
186+
.build();
187+
let ctx = SessionContext::new_with_state(state);
188+
189+
ctx.sql(
190+
"create table trace_events as
191+
select v % 13 as g,
192+
case when v % 29 = 0 then null
193+
else md5(cast(v % 337 as varchar)) end as trace_id
194+
from generate_series(1, 40000) as t(v)",
195+
)
196+
.await?
197+
.collect()
198+
.await?;
199+
200+
ctx.sql(
201+
"create view tv as
202+
select g, arrow_cast(trace_id, 'Utf8View') as trace_id from trace_events",
203+
)
204+
.await?
205+
.collect()
206+
.await?;
207+
208+
let logical = ctx
209+
.state()
210+
.create_logical_plan(
211+
"select g, count(distinct trace_id) as n from tv group by g",
212+
)
213+
.await?;
214+
let plan = ctx.state().create_physical_plan(&logical).await?;
215+
let batches = collect(Arc::clone(&plan), ctx.task_ctx()).await?;
216+
217+
// Count the spills of the `RepartitionExec`s. The caller uses this to
218+
// make sure the query still goes through the spill path: if a change
219+
// stops it spilling, this test keeps passing but no longer covers the
220+
// deadlock, and that must fail loudly instead.
221+
let mut spills = 0;
222+
plan.transform_down(|node| {
223+
if node.is::<RepartitionExec>() {
224+
spills += node.metrics().and_then(|m| m.spill_count()).unwrap_or(0);
225+
}
226+
Ok(Transformed::no(node))
227+
})?;
228+
229+
Ok((batches.iter().map(|b| b.num_rows()).sum(), spills))
230+
}
231+
232+
#[tokio::test(flavor = "multi_thread", worker_threads = 8)]
233+
async fn repartition_spill_pool_does_not_deadlock() {
234+
let mut attempts_that_spilled = 0;
235+
for attempt in 0..ATTEMPTS {
236+
match tokio::time::timeout(PER_ATTEMPT_TIMEOUT, run_once()).await {
237+
Err(elapsed) => panic!(
238+
"attempt {attempt}: grouped COUNT(DISTINCT Utf8View) never completed \
239+
within {PER_ATTEMPT_TIMEOUT:?} ({elapsed}); RepartitionExec spill \
240+
pool deadlock"
241+
),
242+
Ok(Ok((rows, spills))) => {
243+
assert_eq!(rows, 13, "attempt {attempt}: wrong row count");
244+
if spills > 0 {
245+
attempts_that_spilled += 1;
246+
}
247+
}
248+
// The budget is deliberately far too small for the query, and the
249+
// greedy pool hands memory out first come first served, so once in
250+
// a few thousand attempts an aggregate's allocation is refused
251+
// while the repartition reservations hold the pool. That is a
252+
// legitimate outcome of the limit, not the hang this test guards
253+
// against.
254+
Ok(Err(e))
255+
if matches!(
256+
e.find_root(),
257+
datafusion::error::DataFusionError::ResourcesExhausted(_)
258+
) => {}
259+
Ok(Err(e)) => panic!("attempt {attempt}: query failed: {e}"),
260+
}
261+
}
262+
263+
// The memory limit must be tight enough that `RepartitionExec` spills,
264+
// because the deadlock is in its spill pool. Timing decides how many
265+
// attempts spill, but if none of them do, the limit no longer forces
266+
// the spill path and this test has stopped testing anything.
267+
assert!(
268+
attempts_that_spilled > 0,
269+
"no attempt reached the RepartitionExec spill path in {ATTEMPTS} \
270+
attempts; the memory limit no longer forces a spill"
271+
);
272+
}
273+
}

0 commit comments

Comments
 (0)