Repository navigation
Commit 18ddc8f
committed
fix: return a native plan's memory before its Spark task ends
A plan with no JVM input runs on a Tokio task that sends its batches to
the Spark task thread. releasePlan only dropped the receiving end of the
channel, so when the JVM consumer stopped early or threw, the task kept
the plan's stream, and the memory the stream had reserved, after the
Spark task ended. Spark handed that memory to other tasks while native
code still used it, and logged "release called on N bytes but task only
has 0 bytes" when the stream was finally dropped.
releasePlan now aborts the task and waits for it to finish before
flushing the final metrics. Tasks that DataFusion operators spawn, such
as the ones a sort's merge reads its sorted runs through, are only
aborted when the plan is dropped, so releasePlan also waits, for up to a
second, until every reservation the plan made has been returned.
PlanMemoryPool counts the bytes each plan holds.
Closes #24531 parent bc4be39 commit 18ddc8f
5 files changed
Lines changed: 607 additions & 48 deletions
File tree
- docs/source/contributor-guide
- native/core/src/execution
- memory_pools
- spark/src/test/scala/org/apache/spark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
60 | 68 | | |
61 | 69 | | |
62 | 70 | | |
| |||
0 commit comments