Skip to content

Commit ef1d8ec

Browse files
committed
docs: describe the shuffle input conversion in the contributor guide
Also corrects the revert's log message in the tuning guide.
1 parent ddb428b commit ef1d8ec

3 files changed

Lines changed: 22 additions & 10 deletions

File tree

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,10 @@ JVM shuffle (`CometColumnarExchange`) is used instead of native shuffle (`CometE
4343
1. **Shuffle mode is explicitly set to "jvm"**: When `spark.comet.shuffle.mode` is set to `jvm`.
4444

4545
2. **Child plan is not a Comet native operator**: When the child plan is a Spark row-based operator
46-
(not a `CometPlan`), JVM shuffle is the only option since native shuffle requires columnar input
47-
from Comet operators.
46+
(not a `CometPlan`), JVM shuffle is used, since native shuffle requires columnar input from
47+
Comet operators. The exception is `spark.comet.convert.shuffleInput.enabled`, which converts
48+
the child's rows to Arrow with `CometSparkToColumnarExec` so that native shuffle can take the
49+
shuffle instead. See [When Native Shuffle is Used](native_shuffle.md#when-native-shuffle-is-used).
4850

4951
3. **Unsupported partition key types**: `RangePartitioning` keys must be primitive, so a complex
5052
range key always falls back here. `HashPartitioning` keys must be primitive only by default:

‎docs/source/contributor-guide/native_shuffle.md‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,14 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
4646
1. **Shuffle mode allows native**: `spark.comet.shuffle.mode` is `native` or `auto`.
4747

4848
2. **Child plan is a Comet native operator**: The child must be a `CometPlan` that produces
49-
columnar output. Row-based Spark operators require JVM shuffle.
49+
columnar output. Row-based Spark operators require JVM shuffle, except with
50+
`spark.comet.convert.shuffleInput.enabled`. Then `CometExecRule` puts a
51+
`CometSparkToColumnarExec` over the child of a shuffle that JVM shuffle would take, which
52+
converts the child's rows to Arrow, and uses native shuffle, provided native shuffle supports
53+
the partitioning and the columns. A shuffle that hashes a string, a value computed from a
54+
string, or a decimal wider than 18 digits stays on JVM shuffle, because native shuffle would
55+
not put every row in the partition Spark's partitioner does
56+
(`CometShuffleExchangeExec.convertsInputForNativeShuffle`).
5057

5158
3. **Supported partitioning type**: Native shuffle supports:
5259
- `HashPartitioning`
@@ -154,11 +161,14 @@ The native shuffle implementation is its own workspace crate, `datafusion-comet-
154161
- The child plan's `nativeOp` directly, when `CometShuffleExchangeExec`'s child is a
155162
`CometNativeExec` subtree. The upstream operators run inside the same `CometExecIterator`
156163
as the writer, with no JVM-to-native batch boundary between them.
157-
- A synthetic `Scan("ShuffleWriterInput")` placeholder, when the dep was built via the
158-
convenience `prepareShuffleDependency(rdd, ...)` overload (used by
159-
`CometCollectLimitExec` and `CometTakeOrderedAndProjectExec`, or when the
160-
exchange's child is a non-native `CometPlan` such as `CometSparkToColumnarExec`). Native
161-
code reads `ColumnarBatch`es from the JVM input iterator via Arrow C Stream Interface.
164+
- A synthetic `Scan("ShuffleWriterInput")` placeholder, when the exchange's child is a
165+
non-native `CometPlan`, and for the dependencies that `CometCollectLimitExec` and
166+
`CometTakeOrderedAndProjectExec` build. Native code reads the input through the Arrow C
167+
Stream Interface. A `CometNativeArrowSource` child, such as `CometSparkToColumnarExec`,
168+
exports its own Arrow stream from `doExecuteAsArrowStream()`, which
169+
`prepareArrowStreamShuffleDependency` hands to native. Any other input is an
170+
`RDD[ColumnarBatch]`, which the convenience `prepareShuffleDependency(rdd, ...)` overload
171+
wraps in a stream.
162172

163173
2. **Native execution**: A single `CometExecIterator` per partition runs the unified plan.
164174

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,8 @@ This shifts the affected shuffles from Comet's off-heap memory pool back to the
8787
tuned for a small JVM heap may see `ExternalSorter` spills on queries where this revert fires. Shuffle I/O may also
8888
grow marginally because Spark's row-based serializer generally compresses less well than Comet's Arrow IPC format.
8989

90-
Each revert is logged at `INFO` level on the driver as `Reverting Comet columnar shuffle to Spark shuffle between
91-
<parent> and <child>`, which lets you correlate any unexpected behavior with this optimization.
90+
Each revert is logged at `INFO` level on the driver as `Reverting Comet shuffle to Spark shuffle between <parent>
91+
and <child>`, which lets you correlate any unexpected behavior with this optimization.
9292

9393
This optimization is enabled by default and can be disabled by setting
9494
`spark.comet.shuffle.revertRedundantColumnar.enabled=false`, in which case Comet will keep the columnar shuffle

0 commit comments

Comments
 (0)