Skip to content

Commit defa254

Browse files
authored
docs: split the tuning guide into multiple pages (#6244)
The tuning guide had grown to about 600 lines on one page. Give it its own Tuning section in the user guide with a page per topic: memory, shuffle, Celeborn, scans, operators, and row/columnar transitions. tuning.md stays as the overview and lists the other pages, so the URL that Comet's runtime messages and config descriptions link to keeps working. The text is moved as-is apart from link and image path fixes.
1 parent df8e153 commit defa254

15 files changed

Lines changed: 718 additions & 579 deletions

File tree

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ under the License.
2222
This page describes how memory is budgeted, accounted, and enforced across the JVM/native
2323
boundary. It is aimed at contributors working on memory pools, operators that reserve memory, or
2424
anyone debugging an out-of-memory report. For user-facing tuning advice, see the
25-
[Tuning Guide](../user-guide/latest/tuning.md).
25+
[Memory Tuning](../user-guide/latest/tuning/memory.md) guide.
2626

2727
This page covers off-heap mode (`spark.memory.offHeap.enabled=true`) only. Comet also has an
2828
on-heap mode, but it exists so that the Spark SQL test suite can run against Comet without changing
@@ -61,7 +61,7 @@ and no JVM metric measures them, yet they land squarely in container RSS. Comet
6161
its own budget that is meant to shadow the physical one, and declares it to Spark so that the two
6262
compete for a single number. The accuracy of that shadow is the central problem this page is about.
6363

64-
The picture the [Tuning Guide](../user-guide/latest/tuning.md#memory-tuning) gives users is
64+
The picture the [Memory Tuning](../user-guide/latest/tuning/memory.md) guide gives users is
6565
deliberately simple:
6666

6767
![Spark and Comet both use the JVM heap and share the off-heap memory pool, and the rest of Comet's native memory has to fit in the executor's memory overhead](../_static/images/comet-executor-memory.svg)
@@ -435,7 +435,7 @@ reports the bytes Rust's allocator has handed out next to the pools' reservation
435435
than per interval, enable tracing and compare `native_allocated` against
436436
`comet_memory_reserved_total`; see [Tracing](tracing.md#analyzing-memory-usage).
437437

438-
[memory-usage-log]: ../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log
438+
[memory-usage-log]: ../user-guide/latest/tuning/memory.md#sizing-the-overhead-from-the-memory-usage-log
439439

440440
## What the container sees
441441

@@ -531,14 +531,14 @@ much they matter:
531531

532532
## Debugging memory issues
533533

534-
| Tool | What it gives you |
535-
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
536-
| `spark.comet.debug.memory=true` | `LoggingMemoryPool` logs every register/grow/shrink with the consumer name |
537-
| `spark.comet.explain.native.enabled=true` | Native plan with per-operator metrics, including spill counts |
538-
| [Memory usage log](../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log) | Executor-wide native allocation vs pool reservations, logged every 10 seconds by default |
539-
| [Tracing](tracing.md#analyzing-memory-usage) | `native_allocated` vs `comet_memory_reserved_total` per event; the accounting gap over time |
540-
| `TrackConsumersPool` | Names the top 10 consumers in `ResourcesExhausted` messages (always on) |
541-
| [`thresher`](https://github.com/cetra3/thresher) | Third-party crate that dumps a jemalloc heap profile at a threshold |
534+
| Tool | What it gives you |
535+
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
536+
| `spark.comet.debug.memory=true` | `LoggingMemoryPool` logs every register/grow/shrink with the consumer name |
537+
| `spark.comet.explain.native.enabled=true` | Native plan with per-operator metrics, including spill counts |
538+
| [Memory usage log](../user-guide/latest/tuning/memory.md#sizing-the-overhead-from-the-memory-usage-log) | Executor-wide native allocation vs pool reservations, logged every 10 seconds by default |
539+
| [Tracing](tracing.md#analyzing-memory-usage) | `native_allocated` vs `comet_memory_reserved_total` per event; the accounting gap over time |
540+
| `TrackConsumersPool` | Names the top 10 consumers in `ResourcesExhausted` messages (always on) |
541+
| [`thresher`](https://github.com/cetra3/thresher) | Third-party crate that dumps a jemalloc heap profile at a threshold |
542542

543543
A checklist for triaging an executor OOM kill:
544544

‎docs/source/contributor-guide/plugin_overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ and skips the remaining steps. Otherwise it:
5252
`spark.sql.queryExecutionListeners` when `spark.comet.metrics.enabled=true`.
5353
- Logs a warning for settings that are likely to cause problems, such as an unset `spark.executor.memoryOverhead`.
5454

55-
The plugin does not change any executor memory setting. The [Tuning Guide](../user-guide/latest/tuning.md) covers how
55+
The plugin does not change any executor memory setting. The [Memory Tuning](../user-guide/latest/tuning/memory.md) guide covers how
5656
to size them.
5757

5858
When the driver or an executor stops, the plugin shuts down Comet's native tokio runtime in that JVM.

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Native memory is traced as `native_allocated`. Comet wraps whichever global allo
3131
selected and counts the bytes it has handed out. It counts only what Rust code allocated, so it can be compared against the
3232
memory pool's reservations without the allocator's own caching in the way. The same figure appears in
3333
the executor's periodic memory usage log, which does not need tracing; see
34-
[Sizing the Overhead from the Memory Usage Log](../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log).
34+
[Sizing the Overhead from the Memory Usage Log](../user-guide/latest/tuning/memory.md#sizing-the-overhead-from-the-memory-usage-log).
3535

3636
Enabling the `jemalloc` feature adds a second measure, `jemalloc_allocated`, which also includes
3737
jemalloc's own metadata and fragmentation:

‎docs/source/user-guide/latest/index.rst‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,11 +72,23 @@ to read more.
7272
:hidden:
7373

7474
Understanding Comet Plans <understanding-comet-plans>
75-
Tuning Guide <tuning>
7675
Metrics Guide <metrics>
7776
In-Memory Cache <in-memory-cache>
7877
PyArrow UDF Acceleration <pyarrow-udfs>
7978

79+
.. toctree::
80+
:maxdepth: 1
81+
:caption: Tuning
82+
:hidden:
83+
84+
Overview <tuning>
85+
Memory <tuning/memory>
86+
Shuffle <tuning/shuffle>
87+
Celeborn Shuffle <tuning/celeborn>
88+
Scans <tuning/scans>
89+
Operators <tuning/operators>
90+
Row/Columnar Transitions <tuning/transitions>
91+
8092
.. toctree::
8193
:maxdepth: 1
8294
:caption: Integrations

‎docs/source/user-guide/latest/installation.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,7 +260,7 @@ Some cluster managers may require additional configuration, see <https://spark.a
260260
### Memory tuning
261261

262262
In addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory
263-
allocation for native execution. See [Comet Memory Tuning](./tuning.md) for details.
263+
allocation for native execution. See [Comet Memory Tuning](./tuning/memory.md) for details.
264264

265265
### Kryo serialization
266266

‎docs/source/user-guide/latest/metrics.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -76,21 +76,21 @@ the value would always be 0.
7676

7777
Native aggregates with grouping keys report these additional metrics:
7878

79-
| Metric | Description |
80-
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
81-
| `rows bypassing partial aggregation` | Input rows passed through without partial aggregation. See [Adaptive Partial Aggregation](tuning.md#adaptive-partial-aggregation). |
82-
| `number of spills` | Number of times the aggregate spilled to disk. |
83-
| `total spilled bytes` | Bytes written to aggregate spill files. |
84-
| `number of spilled rows` | Rows written to aggregate spill files. |
85-
| `peak native aggregate memory` | Peak memory used by the native aggregate. |
79+
| Metric | Description |
80+
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
81+
| `rows bypassing partial aggregation` | Input rows passed through without partial aggregation. See [Adaptive Partial Aggregation](tuning/operators.md#adaptive-partial-aggregation). |
82+
| `number of spills` | Number of times the aggregate spilled to disk. |
83+
| `total spilled bytes` | Bytes written to aggregate spill files. |
84+
| `number of spilled rows` | Rows written to aggregate spill files. |
85+
| `peak native aggregate memory` | Peak memory used by the native aggregate. |
8686

8787
Spill bytes from native sorts, aggregates, and sort-merge joins are also added to Spark's task-level
8888
`diskBytesSpilled` metric in every stage, not only in shuffle stages.
8989

9090
### Hash Joins
9191

9292
With `spark.comet.exec.join.dynamicFilter.enabled=true`, native broadcast and shuffled hash joins
93-
report these additional metric keys. See [Join Runtime Filters](tuning.md#join-runtime-filters) for
93+
report these additional metric keys. See [Join Runtime Filters](tuning/operators.md#join-runtime-filters) for
9494
eligibility and reader restrictions.
9595

9696
| Metric | Description |

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ omitted from the tables below and may be reconsidered based on demand:
8080
| Operator | Status | Notes |
8181
| ------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
8282
| `HashAggregateExec` | ✅ | |
83-
| `ObjectHashAggregateExec` | ✅ | Runs the object-buffer aggregates Comet supports, such as `collect_list`, `collect_set`, `percentile`, `approx_percentile`, `mode`, `bloom_filter_agg`, and (Spark 4.0+) `listagg`. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See [Shuffle](tuning.md#shuffle). |
83+
| `ObjectHashAggregateExec` | ✅ | Runs the object-buffer aggregates Comet supports, such as `collect_list`, `collect_set`, `percentile`, `approx_percentile`, `mode`, `bloom_filter_agg`, and (Spark 4.0+) `listagg`. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See [Shuffle](tuning/shuffle.md). |
8484
| `SortAggregateExec` | 🔜 | Falls back today; Comet currently accelerates hash aggregates. |
8585

8686
## Joins

0 commit comments

Comments
 (0)