Skip to content

Commit 80253c8

Browse files
committed
docs: say that the conversion gains with hash partitioning and comes out even with range partitioning
Also rewords the contributor guide's rule for when native shuffle takes a Spark child.
1 parent 2d5e9cf commit 80253c8

2 files changed

Lines changed: 15 additions & 15 deletions

File tree

‎docs/source/contributor-guide/native_shuffle.md‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -46,14 +46,13 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
4646
1. **Shuffle mode allows native**: `spark.comet.shuffle.mode` is `native` or `auto`.
4747

4848
2. **Child plan is a Comet native operator**: The child must be a `CometPlan` that produces
49-
columnar output. Row-based Spark operators require JVM shuffle, except with
50-
`spark.comet.convert.shuffleInput.enabled`. Then `CometExecRule` puts a
51-
`CometSparkToColumnarExec` over the child of a shuffle that JVM shuffle would take, which
52-
converts the child's rows to Arrow, and uses native shuffle, provided native shuffle supports
53-
the partitioning and the columns. A shuffle that hashes a string, a value computed from a
54-
string, or a decimal wider than 18 digits stays on JVM shuffle, because native shuffle would
55-
not put every row in the partition Spark's partitioner does
56-
(`CometShuffleExchangeExec.convertsInputForNativeShuffle`).
49+
columnar output. Row-based Spark operators require JVM shuffle, unless
50+
`spark.comet.convert.shuffleInput.enabled` is set. Then a shuffle that JVM shuffle would take
51+
uses native shuffle instead, provided native shuffle supports the partitioning and the
52+
columns, and a `CometSparkToColumnarExec` converts the child's rows to Arrow for it. A shuffle
53+
that hashes a string, a value computed from a string, or a decimal wider than 18 digits stays
54+
on JVM shuffle, because native shuffle would not put every row in the partition Spark's
55+
partitioner does (`CometShuffleExchangeExec.convertsInputForNativeShuffle`).
5756

5857
3. **Supported partitioning type**: Native shuffle supports:
5958
- `HashPartitioning`

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -66,13 +66,14 @@ write side. To restrict columnar shuffle to cases where the child is already a C
6666
as native Spark shuffles, which avoids the row to columnar conversion but means the downstream stage will also start
6767
on Spark.
6868

69-
Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use native shuffle instead. Comet converts
70-
the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the batches
71-
natively, which is usually faster than columnar shuffle. This applies only where native shuffle supports the
72-
partitioning and Comet can convert all of the child's columns, which rules out calendar intervals, and arrays and maps
73-
other than `array<string>` and `map<string,string>`. A shuffle that hashes a decimal wider than 18 digits, or a string
74-
or a value computed from a string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted
75-
batches are allocated on the JVM outside Comet's memory pools. The setting is disabled by default.
69+
Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use native shuffle instead. Comet
70+
converts the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the
71+
batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning
72+
the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert
73+
all of the child's columns, which rules out calendar intervals, and arrays and maps other than `array<string>` and
74+
`map<string,string>`. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a
75+
string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on
76+
the JVM outside Comet's memory pools. The setting is disabled by default.
7677

7778
### Automatic Revert to Spark Shuffle
7879

0 commit comments

Comments
 (0)