Skip to content

Commit d386eeb

Browse files
committed
docs: point the shuffle allocator retry at SPARK-59444
apache/spark#59103 (SPARK-59827) was closed as a duplicate of apache/spark#58747 (SPARK-59444), which fixes the same race in ExecutionMemoryPool, so link that issue and PR instead.
1 parent 840cefc commit d386eeb

3 files changed

Lines changed: 4 additions & 4 deletions

File tree

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
198198
- Off-heap mode gets `CometUnifiedShuffleMemoryAllocator`, an ordinary Spark `MemoryConsumer`
199199
drawing from `spark.memory.offHeap.size`. When it cannot acquire a page the writer spills to
200200
disk. An allocation that loses the task's entry in Spark's execution pool while it waits
201-
([SPARK-59827](https://issues.apache.org/jira/browse/SPARK-59827)) is retried a few times before
201+
([SPARK-59444](https://issues.apache.org/jira/browse/SPARK-59444)) is retried a few times before
202202
it counts as a page that could not be acquired.
203203
- On-heap mode gets `CometUnboundedShuffleMemoryAllocator`, which keeps no budget and so never
204204
refuses a page. Nothing bounds these allocations, and memory pressure never triggers a spill.

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,12 +140,12 @@ same task like any other allocation. A page or pointer array allocation that wai
140140
memory can wake up to find the task's entry gone from Spark's execution pool, because another
141141
consumer of the task released its last bytes in the meantime, and Spark then throws a
142142
`NoSuchElementException` ("key not found") instead of a grant
143-
([SPARK-59827](https://issues.apache.org/jira/browse/SPARK-59827)). The allocator retries that
143+
([SPARK-59444](https://issues.apache.org/jira/browse/SPARK-59444)). The allocator retries that
144144
case, since the failed call was granted nothing, and after three attempts throws
145145
`SparkOutOfMemoryError`, which the shuffle writers handle like any other refused page. Only the
146146
shuffle allocator's callers are guarded this way. Spark's operators in the same task, such as its
147147
sorters and aggregates, can still hit the exception until Spark re-registers a waiting task in
148-
`ExecutionMemoryPool` ([apache/spark#59103](https://github.com/apache/spark/pull/59103)).
148+
`ExecutionMemoryPool` ([apache/spark#58747](https://github.com/apache/spark/pull/58747)).
149149

150150
Which allocator each call site uses, and who ends up charged for the bytes:
151151

‎spark/src/main/java/org/apache/spark/shuffle/comet/CometUnifiedShuffleMemoryAllocator.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ public LongArray allocateArray(long size) {
9191
* drops the entry at a zero balance, so trying again is safe: it registers the task again and
9292
* waits for its share as Spark would have. After a few failed attempts the allocation fails with
9393
* a SparkOutOfMemoryError, which the shuffle writers handle like any other refused page, caused
94-
* by the last attempt's exception. The Spark issue is SPARK-59827.
94+
* by the last attempt's exception. The Spark issue is SPARK-59444.
9595
*/
9696
private <T> T retryMissingTaskEntry(long required, Supplier<T> allocation) {
9797
for (int attempt = 1; ; attempt++) {

0 commit comments

Comments
 (0)