Skip to content

Commit 9b382fd

Browse files
DecisionNerdclaude
andcommitted
fix(exec): backport the spill-pool deadlock fix (apache/datafusion#24891) into the vendored crate (Part of #1985)
Port of apache/datafusion#24891 (upstream 09a2aff7) to the vendored datafusion-physical-plan 54.1.0 spill pool. RepartitionExec deadlocks under a memory limit when its spill pool holds two open files (apache/datafusion#24883): the reader waited on a drained, still-open file while the batch it was owed sat in a newer file, and nothing could wake it. The reader now owns the list of files it has picked up (new files are published through a `new_files` queue), polls them oldest first, skips a file that is only caught up with its writer, and waits only on a file whose unread batches are still being read from disk. Only spill_pool.rs changes; names and docs are adapted to the 54.1.0 API (SpillFile, active_writer_count, RefCountedTempFile, one cloneable SpillPoolWriter and one channel function). Tests added to the same file: the two-open-files drain case adapted from upstream (writer 1 is simulated mid-push by checking its file out of open_write_files), a wake-path case, and a poll-time read-error case. Upstream's scenario fuzz test needs 55-only I/O hooks and is not ported. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent b10326d commit 9b382fd

3 files changed

Lines changed: 338 additions & 94 deletions

File tree

‎Cargo.toml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,10 @@ members = [
2727
]
2828
# Vendored third-party crates are path dependencies through [patch.crates-io]
2929
# below. Excluding them keeps them from becoming members, so fmt, clippy and the
30-
# workspace test run never lint or test code we do not own.
31-
exclude = ["vendor/*"]
30+
# workspace test run never lint or test code we do not own. Cargo matches each
31+
# `exclude` entry as a literal path (a `vendor/*` glob is not recognised), so list
32+
# every vendored crate here.
33+
exclude = ["vendor/datafusion-physical-plan"]
3234

3335
[workspace.package]
3436
version = "0.6.0-rc.4"

‎vendor/datafusion-physical-plan/GRAPHFORGE.md‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,10 @@ crates.io (the registry crate `datafusion-physical-plan-54.1.0`, every file
55
except cargo's `.cargo-ok` marker). It is third-party code under the Apache
66
License 2.0; `LICENSE.txt` and `NOTICE.txt` are the upstream files, unchanged.
77

8-
The root `Cargo.toml` selects it with `[patch.crates-io]` and lists
9-
`vendor/*` under the workspace `exclude`, so fmt, clippy and the workspace test
10-
run never lint or test it.
8+
The root `Cargo.toml` selects it with `[patch.crates-io]` and lists it under
9+
the workspace `exclude`, so fmt, clippy and the workspace test run never lint or
10+
test it. Cargo matches each `exclude` entry as a literal path (a `vendor/*` glob
11+
is not recognised), so each vendored crate is named there.
1112

1213
## The one applied change
1314

@@ -16,7 +17,17 @@ Apache DataFusion pull request
1617
`09a2aff72d88c16499ae3266a943a4cd9841b618`, ported to the 54.1.0 spill-pool
1718
API. It changes only `src/spill/spill_pool.rs`; every other file is identical
1819
to the 54.1.0 crate. `git diff` of that file against the first commit that added
19-
this directory is the whole delta.
20+
this directory is the whole delta. The port keeps the upstream semantics and
21+
adapts only names and docs to the 54.1.0 API (one cloneable `SpillPoolWriter`
22+
and one `channel` function instead of `SpillPoolSink`, `spsc_channel` and
23+
`mpsc_channel`). The upstream tests that need 55-only I/O hooks are not ported;
24+
the tests added in the same file cover the fix on the 54.1.0 API.
25+
26+
To run the vendored crate's spill tests, use a checkout whose root is not nested
27+
inside another cargo workspace:
28+
`cargo test --locked --manifest-path vendor/datafusion-physical-plan/Cargo.toml --lib spill`.
29+
The regression test that CI runs is in `graphforge-exec`
30+
(`session::spill_pool_regression_tests`).
2031

2132
## Why
2233

@@ -29,8 +40,8 @@ GraphForge cannot take it by upgrading, so it carries the fix here.
2940

3041
## Removal
3142

32-
Delete this directory, the `[patch.crates-io]` entry and the `vendor/*`
33-
workspace exclude in the root `Cargo.toml` when GraphForge adopts a DataFusion
43+
Delete this directory, the `[patch.crates-io]` entry and the
44+
`vendor/datafusion-physical-plan` workspace exclude in the root `Cargo.toml` when GraphForge adopts a DataFusion
3445
version that contains #24891. Re-verify with `cargo tree -i
3546
datafusion-physical-plan` that a single registry copy resolves.
3647

0 commit comments

Comments
 (0)