Skip to content

Commit 7e3cbc0

Browse files
authored
fix: retry a shuffle page allocation that Spark failed after dropping the task's entry (#6403)
* fix: retry a shuffle page allocation that Spark failed after dropping the task's entry Spark's ExecutionMemoryPool registers a task's memoryForTask entry once, before its wait loop, and reads it on every pass, while releaseMemory removes the entry when the task's balance reaches zero. When the JVM shuffle allocator's page or pointer array request was parked below the task's minimum share and Comet's native memory consumer released the task's last bytes, the request woke to NoSuchElementException "key not found: <taskAttemptId>" and failed the task, because the shuffle writers only expect SparkOutOfMemoryError. CometUnifiedShuffleMemoryAllocator now retries allocate and allocateArray when Spark throws that exception from ExecutionMemoryPool. The retry registers the task again and waits as the first attempt would have. Nothing is counted or allocated before the exception, so retrying cannot leak. After three attempts it refuses the request with the same SparkOutOfMemoryError it uses for a refused page, with the last exception as its cause. The Spark issue is SPARK-59827. * docs: say which callers the page allocation retry covers Only the shuffle allocator's callers are guarded; Spark's own operators in the task can still hit the removed entry until Spark re-registers a waiting task (apache/spark#59103). * docs: point the shuffle allocator retry at SPARK-59444 apache/spark#59103 (SPARK-59827) was closed as a duplicate of apache/spark#58747 (SPARK-59444), which fixes the same race in ExecutionMemoryPool, so link that issue and PR instead.
1 parent e8c7ee7 commit 7e3cbc0

6 files changed

Lines changed: 386 additions & 3 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -505,6 +505,7 @@ jobs:
505505
org.apache.comet.exec.CometShuffleManagerSuite
506506
org.apache.comet.exec.DisableAQECometShuffleSuite
507507
org.apache.spark.shuffle.comet.CometUnboundedShuffleMemoryAllocatorSuite
508+
org.apache.spark.shuffle.comet.CometUnifiedShuffleMemoryAllocatorSuite
508509
org.apache.spark.shuffle.sort.SpillSorterSuite
509510
- name: "exec"
510511
value: |

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,7 @@ jobs:
209209
org.apache.comet.exec.CometShuffleManagerSuite
210210
org.apache.comet.exec.DisableAQECometShuffleSuite
211211
org.apache.spark.shuffle.comet.CometUnboundedShuffleMemoryAllocatorSuite
212+
org.apache.spark.shuffle.comet.CometUnifiedShuffleMemoryAllocatorSuite
212213
org.apache.spark.shuffle.sort.SpillSorterSuite
213214
- name: "exec"
214215
value: |

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,9 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
197197
depends on Spark's memory mode.
198198
- Off-heap mode gets `CometUnifiedShuffleMemoryAllocator`, an ordinary Spark `MemoryConsumer`
199199
drawing from `spark.memory.offHeap.size`. When it cannot acquire a page the writer spills to
200-
disk.
200+
disk. An allocation that loses the task's entry in Spark's execution pool while it waits
201+
([SPARK-59444](https://issues.apache.org/jira/browse/SPARK-59444)) is retried a few times before
202+
it counts as a page that could not be acquired.
201203
- On-heap mode gets `CometUnboundedShuffleMemoryAllocator`, which keeps no budget and so never
202204
refuses a page. Nothing bounds these allocations, and memory pressure never triggers a spill.
203205
That mode exists only so the Spark SQL tests can run against Comet. See

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,16 @@ allocator before reading anything into the split.
136136
**The JVM shuffle allocator is an ordinary Spark consumer.** `CometShuffleMemoryAllocator.getInstance`
137137
returns `CometUnifiedShuffleMemoryAllocator`, a Spark `MemoryConsumer` drawing from
138138
`spark.memory.offHeap.size`, so shuffle pages are arbitrated against Spark's other consumers in the
139-
same task like any other allocation.
139+
same task like any other allocation. A page or pointer array allocation that waits in Spark for
140+
memory can wake up to find the task's entry gone from Spark's execution pool, because another
141+
consumer of the task released its last bytes in the meantime, and Spark then throws a
142+
`NoSuchElementException` ("key not found") instead of a grant
143+
([SPARK-59444](https://issues.apache.org/jira/browse/SPARK-59444)). The allocator retries that
144+
case, since the failed call was granted nothing, and after three attempts throws
145+
`SparkOutOfMemoryError`, which the shuffle writers handle like any other refused page. Only the
146+
shuffle allocator's callers are guarded this way. Spark's operators in the same task, such as its
147+
sorters and aggregates, can still hit the exception until Spark re-registers a waiting task in
148+
`ExecutionMemoryPool` ([apache/spark#58747](https://github.com/apache/spark/pull/58747)).
140149

141150
Which allocator each call site uses, and who ends up charged for the bytes:
142151

‎spark/src/main/java/org/apache/spark/shuffle/comet/CometUnifiedShuffleMemoryAllocator.java‎

Lines changed: 86 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,18 @@
2020
package org.apache.spark.shuffle.comet;
2121

2222
import java.io.IOException;
23+
import java.util.Map;
24+
import java.util.NoSuchElementException;
25+
import java.util.function.Supplier;
26+
27+
import org.slf4j.Logger;
28+
import org.slf4j.LoggerFactory;
2329

2430
import org.apache.spark.memory.MemoryConsumer;
2531
import org.apache.spark.memory.MemoryMode;
32+
import org.apache.spark.memory.SparkOutOfMemoryError;
2633
import org.apache.spark.memory.TaskMemoryManager;
34+
import org.apache.spark.unsafe.array.LongArray;
2735
import org.apache.spark.unsafe.memory.MemoryBlock;
2836

2937
/**
@@ -38,6 +46,12 @@
3846
*/
3947
public final class CometUnifiedShuffleMemoryAllocator extends CometShuffleMemoryAllocatorTrait {
4048

49+
private static final Logger logger =
50+
LoggerFactory.getLogger(CometUnifiedShuffleMemoryAllocator.class);
51+
52+
/** Attempts at an allocation whose task entry Spark keeps losing, see retryMissingTaskEntry. */
53+
private static final int MAX_ALLOCATE_ATTEMPTS = 3;
54+
4155
CometUnifiedShuffleMemoryAllocator(TaskMemoryManager taskMemoryManager, long pageSize) {
4256
super(taskMemoryManager, pageSize, MemoryMode.OFF_HEAP);
4357
if (taskMemoryManager.getTungstenMemoryMode() != MemoryMode.OFF_HEAP) {
@@ -54,7 +68,78 @@ public long spill(long l, MemoryConsumer memoryConsumer) throws IOException {
5468
}
5569

5670
public synchronized MemoryBlock allocate(long required) {
57-
return this.allocatePage(required);
71+
return retryMissingTaskEntry(required, () -> this.allocatePage(required));
72+
}
73+
74+
/**
75+
* Allocates a pointer array, retrying like {@link #allocate}. The inherited method asks Spark for
76+
* the page directly rather than through allocate, and the sorters' pointer arrays come through
77+
* here.
78+
*/
79+
@Override
80+
public LongArray allocateArray(long size) {
81+
return retryMissingTaskEntry(size * 8L, () -> super.allocateArray(size));
82+
}
83+
84+
/**
85+
* Runs an allocation through Spark, retrying when an allocation waiting in Spark lost the task's
86+
* entry. Spark removes a task's entry from its execution pool when the task's balance reaches
87+
* zero, and an allocation that was waiting for memory then fails when it wakes up with a
88+
* NoSuchElementException ("key not found: " and the task id). Releases take no task monitor, so a
89+
* native consumer or another JVM consumer of the same task can empty the balance while an
90+
* allocation waits. The failed call was granted nothing and took no page, because Spark only
91+
* drops the entry at a zero balance, so trying again is safe: it registers the task again and
92+
* waits for its share as Spark would have. After a few failed attempts the allocation fails with
93+
* a SparkOutOfMemoryError, which the shuffle writers handle like any other refused page, caused
94+
* by the last attempt's exception. The Spark issue is SPARK-59444.
95+
*/
96+
private <T> T retryMissingTaskEntry(long required, Supplier<T> allocation) {
97+
for (int attempt = 1; ; attempt++) {
98+
try {
99+
return allocation.get();
100+
} catch (NoSuchElementException e) {
101+
if (!isMissingTaskEntry(e)) {
102+
throw e;
103+
}
104+
if (attempt >= MAX_ALLOCATE_ATTEMPTS) {
105+
logger.warn(
106+
"Lost the task's execution memory entry in Spark on {} attempts to allocate {} "
107+
+ "bytes, refusing the allocation",
108+
attempt,
109+
required,
110+
e);
111+
SparkOutOfMemoryError error =
112+
new SparkOutOfMemoryError(
113+
"UNABLE_TO_ACQUIRE_MEMORY",
114+
Map.of(
115+
"requestedBytes", String.valueOf(required),
116+
"receivedBytes", String.valueOf(0)));
117+
error.initCause(e);
118+
throw error;
119+
}
120+
logger.info(
121+
"Lost the task's execution memory entry in Spark while waiting to allocate {} bytes, "
122+
+ "trying again",
123+
required);
124+
}
125+
}
126+
}
127+
128+
/**
129+
* Whether {@code e} is Spark's execution pool failing to find the task's entry. The task id is
130+
* not available here, so this matches the message prefix and the pool's stack frame.
131+
*/
132+
private static boolean isMissingTaskEntry(NoSuchElementException e) {
133+
String message = e.getMessage();
134+
if (message == null || !message.startsWith("key not found: ")) {
135+
return false;
136+
}
137+
for (StackTraceElement frame : e.getStackTrace()) {
138+
if ("org.apache.spark.memory.ExecutionMemoryPool".equals(frame.getClassName())) {
139+
return true;
140+
}
141+
}
142+
return false;
58143
}
59144

60145
public synchronized long free(MemoryBlock block) {

0 commit comments

Comments
 (0)