Skip to content

Commit 81f2574

Browse files
authored
docs: add a simple diagram of how Comet uses executor memory (apache#6237)
Draw the executor container as the JVM heap, the off-heap memory pool and the memory overhead, showing that Comet shares the off-heap pool with Spark and needs room in the memory overhead for the native memory the pool does not track. The diagram goes in the Memory Tuning section of the tuning guide and in the overview of the contributor guide's memory management page.
1 parent 36ab57c commit 81f2574

3 files changed

Lines changed: 99 additions & 0 deletions

File tree

Lines changed: 88 additions & 0 deletions
Loading

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,15 @@ and no JVM metric measures them, yet they land squarely in container RSS. Comet
6161
its own budget that is meant to shadow the physical one, and declares it to Spark so that the two
6262
compete for a single number. The accuracy of that shadow is the central problem this page is about.
6363

64+
The picture the [Tuning Guide](../user-guide/latest/tuning.md#memory-tuning) gives users is
65+
deliberately simple:
66+
67+
![Spark and Comet both use the JVM heap and share the off-heap memory pool, and the rest of Comet's native memory has to fit in the executor's memory overhead](../_static/images/comet-executor-memory.svg)
68+
69+
Most of this page is about the line between Comet's share of the off-heap pool and its share of the
70+
memory overhead: which allocations are declared to the pool, and which land in the overhead with
71+
nothing tracking them.
72+
6473
## Who allocates what
6574

6675
Enabling Comet does not add one new memory consumer, it adds several, and they are not all

‎docs/source/user-guide/latest/tuning.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,8 @@ Comet needs two things configured: an off-heap pool for it to draw its reservati
8080
overhead to cover the part of its footprint that no pool tracks. See [Configuring Comet Memory] and
8181
[Configuring Executor Memory Overhead].
8282

83+
![Spark and Comet both use the JVM heap and share the off-heap memory pool, and the rest of Comet's native memory has to fit in the executor's memory overhead](../../_static/images/comet-executor-memory.svg)
84+
8385
[Determining How Much Memory to Allocate]: #determining-how-much-memory-to-allocate
8486
[Configuring Comet Memory]: #configuring-comet-memory
8587
[Configuring Executor Memory Overhead]: #configuring-executor-memory-overhead

0 commit comments

Comments
 (0)