You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7e3cbc0
Browse filesBrowse the repository at this point in the historyBrowse files
fix: retry a shuffle page allocation that Spark failed after dropping the task's entry (#6403)
* fix: retry a shuffle page allocation that Spark failed after dropping the task's entry
Spark's ExecutionMemoryPool registers a task's memoryForTask entry once, before
its wait loop, and reads it on every pass, while releaseMemory removes the
entry when the task's balance reaches zero. When the JVM shuffle allocator's
page or pointer array request was parked below the task's minimum share and
Comet's native memory consumer released the task's last bytes, the request
woke to NoSuchElementException "key not found: <taskAttemptId>" and failed the
task, because the shuffle writers only expect SparkOutOfMemoryError.
CometUnifiedShuffleMemoryAllocator now retries allocate and allocateArray when
Spark throws that exception from ExecutionMemoryPool. The retry registers the
task again and waits as the first attempt would have. Nothing is counted or
allocated before the exception, so retrying cannot leak. After three attempts
it refuses the request with the same SparkOutOfMemoryError it uses for a
refused page, with the last exception as its cause. The Spark issue is
SPARK-59827.
* docs: say which callers the page allocation retry covers
Only the shuffle allocator's callers are guarded; Spark's own operators in the
task can still hit the removed entry until Spark re-registers a waiting task
(apache/spark#59103).
* docs: point the shuffle allocator retry at SPARK-59444
apache/spark#59103 (SPARK-59827) was closed as a duplicate of
apache/spark#58747 (SPARK-59444), which fixes the same race in
ExecutionMemoryPool, so link that issue and PR instead.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/jvm_shuffle.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -197,7 +197,9 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
197
197
depends on Spark's memory mode.
198
198
- Off-heap mode gets `CometUnifiedShuffleMemoryAllocator`, an ordinary Spark `MemoryConsumer`
199
199
drawing from `spark.memory.offHeap.size`. When it cannot acquire a page the writer spills to
200
-
disk.
200
+
disk. An allocation that loses the task's entry in Spark's execution pool while it waits
201
+
([SPARK-59444](https://issues.apache.org/jira/browse/SPARK-59444)) is retried a few times before
202
+
it counts as a page that could not be acquired.
201
203
- On-heap mode gets `CometUnboundedShuffleMemoryAllocator`, which keeps no budget and so never
202
204
refuses a page. Nothing bounds these allocations, and memory pressure never triggers a spill.
203
205
That mode exists only so the Spark SQL tests can run against Comet. See
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/memory_management.md
+10-1Lines changed: 10 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -136,7 +136,16 @@ allocator before reading anything into the split.
136
136
**The JVM shuffle allocator is an ordinary Spark consumer.**`CometShuffleMemoryAllocator.getInstance`
137
137
returns `CometUnifiedShuffleMemoryAllocator`, a Spark `MemoryConsumer` drawing from
138
138
`spark.memory.offHeap.size`, so shuffle pages are arbitrated against Spark's other consumers in the
139
-
same task like any other allocation.
139
+
same task like any other allocation. A page or pointer array allocation that waits in Spark for
140
+
memory can wake up to find the task's entry gone from Spark's execution pool, because another
141
+
consumer of the task released its last bytes in the meantime, and Spark then throws a
142
+
`NoSuchElementException` ("key not found") instead of a grant
143
+
([SPARK-59444](https://issues.apache.org/jira/browse/SPARK-59444)). The allocator retries that
144
+
case, since the failed call was granted nothing, and after three attempts throws
145
+
`SparkOutOfMemoryError`, which the shuffle writers handle like any other refused page. Only the
146
+
shuffle allocator's callers are guarded this way. Spark's operators in the same task, such as its
147
+
sorters and aggregates, can still hit the exception until Spark re-registers a waiting task in
0 commit comments