Skip to content

Commit 3c982c8

Browse files
docs: qualify direct read, explain.rules, and shuffle child rules
- plugin_overview.md: a shuffle input is a ShuffleScanExec only when it is serialized as a ShuffleScan leaf, which needs direct read, an AQE query stage, and supported output types. Without AQE, the shuffle is read through ScanExec. - understanding-comet-plans.md: spark.comet.explain.rules logs the plan before and after CometScanRule, CometExecRule and EliminateRedundantTransitions, not stage reverts. Point to spark.comet.explain.fallback.log.enabled for those. - jvm_shuffle.md: scope the columnar-to-row diagram to children that produce Arrow batches. A CometSparkToColumnarExec child returns its input rows directly. - native_shuffle.md: with the Celeborn shuffle manager, native shuffle needs a CometNativeExec child, not any CometPlan. Part of #4419. Generated-by: OpenCode
1 parent 56396e3 commit 3c982c8

4 files changed

Lines changed: 19 additions & 11 deletions

File tree

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,10 @@ returns an `RDD[InternalRow]`. The rows flow directly to the JVM shuffle writers
6565

6666
### Comet Columnar Input
6767

68-
When the child plan is a Comet operator (e.g., `CometHashAggregate`) but JVM shuffle is selected
69-
(due to shuffle mode setting or unsupported partitioning), `CometColumnarExchange` still calls
70-
`child.execute()`. Comet operators implement `doExecute()` by wrapping themselves with `ColumnarToRowExec`:
68+
When the child plan is a Comet operator that produces Arrow batches (e.g., `CometHashAggregate`) but
69+
JVM shuffle is selected (due to shuffle mode setting or unsupported partitioning),
70+
`CometColumnarExchange` still calls `child.execute()`. Comet operators that extend `CometExec`
71+
implement `doExecute()` by wrapping themselves with `ColumnarToRowExec`:
7172

7273
```scala
7374
// In CometExec base class
@@ -81,7 +82,11 @@ This means the data path becomes:
8182
Comet operator (Arrow batches) → ColumnarToRowExec → rows → JVM Shuffle → Arrow IPC → columnar
8283
```
8384

84-
This is less efficient than native shuffle which avoids the columnar-to-row conversion:
85+
`CometSparkToColumnarExec` is different: its `doExecute()` returns its child's rows directly, so a
86+
JVM shuffle over it reads those rows without any Arrow conversion.
87+
88+
For a child that produces Arrow batches, this is less efficient than native shuffle, which avoids
89+
the columnar-to-row conversion:
8590

8691
```
8792
Comet operator (Arrow batches) → Native Shuffle → Arrow IPC → columnar

‎docs/source/contributor-guide/native_shuffle.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,9 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
5454
1. **Shuffle mode allows native shuffle**: `spark.comet.shuffle.mode` is `native` or `auto`.
5555

5656
2. **Child plan is a Comet operator**: The child must be a `CometPlan` that produces
57-
columnar output. Row-based Spark operators require JVM shuffle, unless
57+
columnar output. With the Celeborn shuffle manager, the child must be a `CometNativeExec`, and
58+
when it is not, Spark's shuffle is used, because Celeborn has no JVM columnar shuffle to fall
59+
back to. Row-based Spark operators require JVM shuffle, unless
5860
`spark.comet.convert.shuffleInput.enabled` is set. Then a shuffle that JVM shuffle would take
5961
uses native shuffle instead, provided native shuffle supports the partitioning and the
6062
columns, and a `CometSparkToColumnarExec` converts the child's rows to Arrow for it. A shuffle

‎docs/source/contributor-guide/plugin_overview.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -155,11 +155,12 @@ A leaf node in the physical plan that takes Arrow batches from the JVM is a `Sca
155155
[Arrow C Stream](https://arrow.apache.org/docs/format/CStreamInterface.html) (`org.apache.arrow.c.ArrowArrayStream`),
156156
and `ScanExec` pulls each input batch through a single C callback rather than making a JNI call per batch. The
157157
input could be a broadcast, a shuffle read decoded through JNI and imported into the JVM, the output of
158-
`CometSparkToColumnarExec`, or the output of an earlier Comet block. When direct shuffle read is enabled and the input
159-
is a Comet shuffle whose output types Comet supports, the input is instead a `ShuffleScanExec`, which pulls compressed
160-
Arrow IPC blocks from the JVM and decodes them in Rust (see [Shuffle Reads](#shuffle-reads)). Operators that run their
161-
own Rust plans, such as `CometTakeOrderedAndProjectExec`, read their shuffle input as batches through `ScanExec`. The
162-
Parquet, Iceberg, and CSV scans read their files in Rust.
158+
`CometSparkToColumnarExec`, or the output of an earlier Comet block. A shuffle input is a `ShuffleScanExec` instead
159+
only when it is serialized as a `ShuffleScan` leaf: direct shuffle read is enabled, Adaptive Query Execution has made
160+
the Comet shuffle a query stage, and Comet supports its output types. `ShuffleScanExec` pulls compressed Arrow IPC
161+
blocks from the JVM and decodes them in Rust (see [Shuffle Reads](#shuffle-reads)). Otherwise, including without
162+
Adaptive Query Execution and for operators that run their own Rust plans such as `CometTakeOrderedAndProjectExec`, the
163+
shuffle is read as batches through `ScanExec`. The Parquet, Iceberg, and CSV scans read their files in Rust.
163164

164165
`CometNativeExec` creates a `CometExecIterator` and applies this iterator to the input RDD
165166
partitions. Each call to `CometExecIterator.next()` will invoke `Native.executePlan`. Once the plan finishes

‎docs/source/user-guide/latest/understanding-comet-plans.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ They serve different purposes and produce output in different places.
9595
| `spark.comet.explain.format` | Spark SQL UI (Spark 4.0 and newer) | Annotated plan or fallback-reason list, depending on `verbose` (default) or `fallback` value. |
9696
| `spark.comet.explain.native.enabled` | Executor logs, per task | The DataFusion plan with metrics, useful for inspecting Rust execution. |
9797
| `spark.comet.explain.planOnly.enabled` | Driver log | The Comet plan and coverage summary, prefixed `[Comet plan-only]`, while Spark runs the original plan. |
98-
| `spark.comet.explain.rules` | Driver log | Each plan transformation that Comet's planner rules make. |
98+
| `spark.comet.explain.rules` | Driver log | The plan before and after `CometScanRule`, `CometExecRule`, and `EliminateRedundantTransitions`. For stage reverts, use `spark.comet.explain.fallback.log.enabled`. |
9999

100100
### `spark.comet.explain.fallback.enabled`
101101

0 commit comments

Comments
 (0)