You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit defa254
Browse filesBrowse the repository at this point in the historyBrowse files
docs: split the tuning guide into multiple pages (#6244)
The tuning guide had grown to about 600 lines on one page. Give it its own
Tuning section in the user guide with a page per topic: memory, shuffle,
Celeborn, scans, operators, and row/columnar transitions.
tuning.md stays as the overview and lists the other pages, so the URL that
Comet's runtime messages and config descriptions link to keeps working. The
text is moved as-is apart from link and image path fixes.
This page covers off-heap mode (`spark.memory.offHeap.enabled=true`) only. Comet also has an
28
28
on-heap mode, but it exists so that the Spark SQL test suite can run against Comet without changing
@@ -61,7 +61,7 @@ and no JVM metric measures them, yet they land squarely in container RSS. Comet
61
61
its own budget that is meant to shadow the physical one, and declares it to Spark so that the two
62
62
compete for a single number. The accuracy of that shadow is the central problem this page is about.
63
63
64
-
The picture the [Tuning Guide](../user-guide/latest/tuning.md#memory-tuning) gives users is
64
+
The picture the [Memory Tuning](../user-guide/latest/tuning/memory.md) guide gives users is
65
65
deliberately simple:
66
66
67
67

@@ -435,7 +435,7 @@ reports the bytes Rust's allocator has handed out next to the pools' reservation
435
435
than per interval, enable tracing and compare `native_allocated` against
436
436
`comet_memory_reserved_total`; see [Tracing](tracing.md#analyzing-memory-usage).
|`spark.comet.debug.memory=true`|`LoggingMemoryPool` logs every register/grow/shrink with the consumer name |
537
-
|`spark.comet.explain.native.enabled=true`| Native plan with per-operator metrics, including spill counts |
538
-
|[Memory usage log](../user-guide/latest/tuning.md#sizing-the-overhead-from-the-memory-usage-log)| Executor-wide native allocation vs pool reservations, logged every 10 seconds by default |
539
-
|[Tracing](tracing.md#analyzing-memory-usage)|`native_allocated` vs `comet_memory_reserved_total` per event; the accounting gap over time |
540
-
|`TrackConsumersPool`| Names the top 10 consumers in `ResourcesExhausted` messages (always on) |
541
-
|[`thresher`](https://github.com/cetra3/thresher)| Third-party crate that dumps a jemalloc heap profile at a threshold |
|`spark.comet.debug.memory=true`|`LoggingMemoryPool` logs every register/grow/shrink with the consumer name |
537
+
|`spark.comet.explain.native.enabled=true`| Native plan with per-operator metrics, including spill counts |
538
+
|[Memory usage log](../user-guide/latest/tuning/memory.md#sizing-the-overhead-from-the-memory-usage-log)| Executor-wide native allocation vs pool reservations, logged every 10 seconds by default |
539
+
|[Tracing](tracing.md#analyzing-memory-usage)|`native_allocated` vs `comet_memory_reserved_total` per event; the accounting gap over time |
540
+
|`TrackConsumersPool`| Names the top 10 consumers in `ResourcesExhausted` messages (always on) |
541
+
|[`thresher`](https://github.com/cetra3/thresher)| Third-party crate that dumps a jemalloc heap profile at a threshold |
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/installation.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -260,7 +260,7 @@ Some cluster managers may require additional configuration, see <https://spark.a
260
260
### Memory tuning
261
261
262
262
In addition to Apache Spark memory configuration parameters, Comet introduces additional parameters to configure memory
263
-
allocation for native execution. See [Comet Memory Tuning](./tuning.md) for details.
263
+
allocation for native execution. See [Comet Memory Tuning](./tuning/memory.md) for details.
|`ObjectHashAggregateExec`| ✅ | Runs the object-buffer aggregates Comet supports, such as `collect_list`, `collect_set`, `percentile`, `approx_percentile`, `mode`, `bloom_filter_agg`, and (Spark 4.0+) `listagg`. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See [Shuffle](tuning.md#shuffle). |
83
+
|`ObjectHashAggregateExec`| ✅ | Runs the object-buffer aggregates Comet supports, such as `collect_list`, `collect_set`, `percentile`, `approx_percentile`, `mode`, `bloom_filter_agg`, and (Spark 4.0+) `listagg`. Falls back when Comet shuffle is disabled, which would otherwise split the aggregate across Comet and Spark. See [Shuffle](tuning/shuffle.md). |
84
84
|`SortAggregateExec`| 🔜 | Falls back today; Comet currently accelerates hash aggregates. |
0 commit comments