Skip to content

Commit 00d4d40

Browse files
mag1c1an1claudecodex
committed
[Rust]: patch DataFusion RepartitionExec spill-pool deadlock
Under a memory limit, lakesoul-console hung on TPC-H q17 over tpch_sf1_pkmonth (2 CPU, LAKESOUL_MEMORY_LIMIT=1Gi, LAKESOUL_CACHE=1) in about one run in four: every task parked, no error. EXPLAIN ANALYZE showed the hash RepartitionExec spilling. This is apache/datafusion#24883: two input tasks spilling at once leave the shared spill pool with two open files, the reader waits on the drained older file while the batch it needs is in the newer one, and the writers park on the closed distributor gate. The fix (apache/datafusion#24891, merge commit 09a2aff7) is only in DataFusion 56. Vendor datafusion-physical-plan 55.2.0 under rust/third_party with that commit's src/spill/spill_pool.rs change and use it via [patch.crates-io]. With the patch the same configuration still spills in RepartitionExec and completed 20 of 20 runs; the crate's 20 spill_pool tests, including the upstream scenario fuzzer, pass. Co-Authored-By: Claude Opus 5.5 <[email protected]> Change-Id: I6f24b7436d4bcd3ee74bfd3cccaff5ae6a6a6964 Co-Authored-By: Codex <[email protected]> Signed-off-by: mag1cian <[email protected]>
1 parent 08bf235 commit 00d4d40

5 files changed

Lines changed: 21 additions & 2 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,3 +175,11 @@ parking_lot = "0.12"
175175

176176
[workspace.lints.rust]
177177
deprecated = "allow"
178+
179+
# datafusion-physical-plan 55.2.0 plus apache/datafusion#24891 (merge commit
180+
# 09a2aff7), which fixes the RepartitionExec spill-pool deadlock
181+
# (apache/datafusion#24883): under a memory limit, a query whose hash
182+
# repartition spills can hang forever. Only src/spill/spill_pool.rs differs
183+
# from crates.io. Remove when DataFusion is upgraded to 56.
184+
[patch.crates-io]
185+
datafusion-physical-plan = { git = "https://github.com/mag1c1an1/datafusion.git", rev = "e668c7ab1ca6f4e563060d3d1013a03a7e7e2296" }

‎flake.nix‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,12 @@
269269
// {
270270
# Only `cargo build` artifacts are reused; skip the `cargo check` pass.
271271
cargoCheckCommand = "true";
272+
# The dummy source stubs every path crate; the vendored DataFusion
273+
# crate is a dependency, not workspace code, so keep it real.
274+
extraDummyScript = ''
275+
rm -rf $out/rust/third_party
276+
cp -r ${./rust/third_party} $out/rust/third_party
277+
'';
272278
});
273279

274280
lakesoulBins = craneLib.buildPackage (lakesoulCraneArgs

‎rust/postgres-lakesoul/k8s/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -288,6 +288,10 @@ share the `CoreArgs` CLI (`--warehouse-prefix`, `--endpoint`, …).
288288
These budgets govern DataFusion execution reservations, not process RSS,
289289
LakeSoul IO pools or writer-internal buffers. Spill files use DataFusion's
290290
default OS temporary directory (`TMPDIR`, `/var/scratch` in these deployments).
291+
Builds use a vendored `datafusion-physical-plan` 55.2.0 with the upstream fix
292+
for apache/datafusion#24883 (`rust/third_party/`, root `Cargo.toml`
293+
`[patch.crates-io]`): without it, a query whose hash `RepartitionExec` spills
294+
under a bounded pool can park every task and hang without an error.
291295
The runtime still registers the warehouse object store independently per
292296
coordinator session. Pools report the five largest consumers on bounded
293297
allocation errors. Prometheus exports `lakesoul_memory_pool_reserved_bytes`,

‎treefmt.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ excludes = [
2323
# Generated code
2424
"lakesoul-flink/src/main/java/org/apache/flink/fnexecution/v1/FlinkFnApi.java",
2525
"python/src/lakesoul/metadata/generated/**",
26+
# Vendored upstream crates
27+
"rust/third_party/**",
2628
]
2729

2830
# Rust ─────────────────────────────────────────────────

0 commit comments

Comments
 (0)