Repository navigation
Commit 9b382fd
fix(exec): backport the spill-pool deadlock fix (apache/datafusion#24891) into the vendored crate (Part of #1985)
Port of apache/datafusion#24891 (upstream 09a2aff7) to the vendored
datafusion-physical-plan 54.1.0 spill pool. RepartitionExec deadlocks under a
memory limit when its spill pool holds two open files (apache/datafusion#24883):
the reader waited on a drained, still-open file while the batch it was owed sat
in a newer file, and nothing could wake it.
The reader now owns the list of files it has picked up (new files are published
through a `new_files` queue), polls them oldest first, skips a file that is only
caught up with its writer, and waits only on a file whose unread batches are
still being read from disk. Only spill_pool.rs changes; names and docs are
adapted to the 54.1.0 API (SpillFile, active_writer_count, RefCountedTempFile,
one cloneable SpillPoolWriter and one channel function).
Tests added to the same file: the two-open-files drain case adapted from
upstream (writer 1 is simulated mid-push by checking its file out of
open_write_files), a wake-path case, and a poll-time read-error case. Upstream's
scenario fuzz test needs 55-only I/O hooks and is not ported.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>1 parent b10326d commit 9b382fd
3 files changed
Lines changed: 338 additions & 94 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
31 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
9 | | - | |
10 | | - | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | | - | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
20 | 31 | | |
21 | 32 | | |
22 | 33 | | |
| |||
29 | 40 | | |
30 | 41 | | |
31 | 42 | | |
32 | | - | |
33 | | - | |
| 43 | + | |
| 44 | + | |
34 | 45 | | |
35 | 46 | | |
36 | 47 | | |
| |||
0 commit comments