Skip to content

Commit 18ddc8f

Browse files
committed
fix: return a native plan's memory before its Spark task ends
A plan with no JVM input runs on a Tokio task that sends its batches to the Spark task thread. releasePlan only dropped the receiving end of the channel, so when the JVM consumer stopped early or threw, the task kept the plan's stream, and the memory the stream had reserved, after the Spark task ended. Spark handed that memory to other tasks while native code still used it, and logged "release called on N bytes but task only has 0 bytes" when the stream was finally dropped. releasePlan now aborts the task and waits for it to finish before flushing the final metrics. Tasks that DataFusion operators spawn, such as the ones a sort's merge reads its sorted runs through, are only aborted when the plan is dropped, so releasePlan also waits, for up to a second, until every reservation the plan made has been returned. PlanMemoryPool counts the bytes each plan holds. Closes #2453
1 parent bc4be39 commit 18ddc8f

5 files changed

Lines changed: 607 additions & 48 deletions

File tree

‎docs/source/contributor-guide/development.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,14 @@ On the async I/O path, DataFusion operators execute on **tokio worker threads**.
5757
source path, `block_on()` polls them on the Spark executor task thread, and any tasks they spawn
5858
run on the shared runtime. All Spark tasks on an executor share one tokio runtime.
5959

60+
When Spark closes a plan, `releasePlan` stops the task that runs it on the async I/O path and
61+
waits for that task to finish, then drops the plan and waits until every memory reservation the
62+
plan made has been returned. Tasks that operators spawn, such as the ones a sort's merge reads its
63+
sorted runs through, are only aborted when the plan is dropped, and they return what they hold the
64+
next time they yield. The waits matter because Spark frees whatever a task still holds when the
65+
task ends and can hand that memory to another task, so memory a plan returns later was still in
66+
use while Spark counted it as free.
67+
6068
### Rules for native code
6169

6270
**Do not use `thread_local!` or assume thread identity.** Tokio may run your operator's `poll`

0 commit comments

Comments
 (0)