Repository navigation
Commit 00d4d40
[Rust]: patch DataFusion RepartitionExec spill-pool deadlock
Under a memory limit, lakesoul-console hung on TPC-H q17 over
tpch_sf1_pkmonth (2 CPU, LAKESOUL_MEMORY_LIMIT=1Gi, LAKESOUL_CACHE=1) in
about one run in four: every task parked, no error. EXPLAIN ANALYZE showed
the hash RepartitionExec spilling. This is apache/datafusion#24883: two
input tasks spilling at once leave the shared spill pool with two open
files, the reader waits on the drained older file while the batch it needs
is in the newer one, and the writers park on the closed distributor gate.
The fix (apache/datafusion#24891, merge commit 09a2aff7) is only in
DataFusion 56. Vendor datafusion-physical-plan 55.2.0 under rust/third_party
with that commit's src/spill/spill_pool.rs change and use it via
[patch.crates-io]. With the patch the same configuration still spills in
RepartitionExec and completed 20 of 20 runs; the crate's 20 spill_pool
tests, including the upstream scenario fuzzer, pass.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Change-Id: I6f24b7436d4bcd3ee74bfd3cccaff5ae6a6a6964
Co-Authored-By: Codex <[email protected]>
Signed-off-by: mag1cian <[email protected]>1 parent 08bf235 commit 00d4d40
5 files changed
Lines changed: 21 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
175 | 175 | | |
176 | 176 | | |
177 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
269 | 269 | | |
270 | 270 | | |
271 | 271 | | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
272 | 278 | | |
273 | 279 | | |
274 | 280 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
288 | 288 | | |
289 | 289 | | |
290 | 290 | | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
291 | 295 | | |
292 | 296 | | |
293 | 297 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
0 commit comments