Skip to content

Commit d334434

Browse files
committed
docs: qualify the memoryOverhead recommendation so it cannot shrink the container
Setting spark.executor.memoryOverhead replaces Spark's derived value rather than adding to it (ResourceProfile.calculateOverHeadMemory uses getOrElse), so a flat 2g recommendation reduces the allowance for executors large enough that the factor already derives more than that. Describe the derivation accurately, including the 0.4 factor for non-JVM apps on Kubernetes in cluster mode and the configurable floor on Spark 4.0+, scope 2g to small executors, and point larger deployments at spark.executor.memoryOverheadFactor.
1 parent db0ea2e commit d334434

1 file changed

Lines changed: 30 additions & 9 deletions

File tree

‎docs/source/user-guide/latest/tuning.md‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -140,12 +140,8 @@ need to spill or have a single spillable operator.
140140

141141
### Configuring Executor Memory Overhead
142142

143-
Enabling off-heap memory is not sufficient on its own. It is recommended to also set
144-
`spark.executor.memoryOverhead` when running Comet:
145-
146-
```
147-
spark.executor.memoryOverhead=2g
148-
```
143+
Enabling off-heap memory is not sufficient on its own. Comet also needs room in
144+
`spark.executor.memoryOverhead`.
149145

150146
`spark.memory.offHeap.size` is a budget, and the cluster manager already sizes the executor
151147
container to include it, so the memory that Comet's operators explicitly reserve has room. What does
@@ -156,9 +152,34 @@ the container sizing accounts for them. The same applies to Comet's JVM-side Arr
156152

157153
`spark.executor.memoryOverhead` is the only slack the container has for this, and the JVM's own
158154
non-heap usage — metaspace, code cache, thread stacks, GC structures — is already drawing on it.
159-
Spark's default of `max(0.1 * spark.executor.memory, 384 MiB)` leaves little behind. Raise the value
160-
further if executors are killed by the cluster manager (on Kubernetes, `ExecutorLostFailure` with
161-
exit code 137) rather than failing with a task-level out-of-memory error.
155+
156+
Work out what the executor already gets before choosing a value. When
157+
`spark.executor.memoryOverhead` is unset, Spark derives the overhead as
158+
`max(spark.executor.memoryOverheadFactor * spark.executor.memory, 384 MiB)`. The factor defaults to
159+
`0.1`, except for PySpark and SparkR applications submitted to Kubernetes in cluster mode, where it
160+
defaults to `0.4`. On Spark 4.0 and later the floor is configurable through
161+
`spark.executor.minMemoryOverhead`. Setting `spark.executor.memoryOverhead` **replaces** the derived
162+
value rather than adding to it, so a value below what is derived today shrinks the container instead
163+
of growing it.
164+
165+
For a small executor, `2g` is a reasonable starting point. A 4 GiB executor derives only 409 MiB, so
166+
this is a real increase:
167+
168+
```
169+
spark.executor.memoryOverhead=2g
170+
```
171+
172+
A 32 GiB executor, on the other hand, already derives 3276 MiB, and the same setting would take away
173+
1228 MiB. For executors that large, either pick an absolute value above what is derived today, or
174+
raise `spark.executor.memoryOverheadFactor` instead so that the overhead keeps scaling with executor
175+
size:
176+
177+
```
178+
spark.executor.memoryOverheadFactor=0.2
179+
```
180+
181+
Raise the value further if executors are killed by the cluster manager (on Kubernetes,
182+
`ExecutorLostFailure` with exit code 137) rather than failing with a task-level out-of-memory error.
162183

163184
Note that on Kubernetes and YARN the overhead is added to the container size, so raising it reduces
164185
how many executors fit on a node.

0 commit comments

Comments
 (0)