@@ -46,7 +46,14 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
46461 . ** Shuffle mode allows native** : ` spark.comet.shuffle.mode ` is ` native ` or ` auto ` .
4747
48482 . ** Child plan is a Comet native operator** : The child must be a ` CometPlan ` that produces
49- columnar output. Row-based Spark operators require JVM shuffle.
49+ columnar output. Row-based Spark operators require JVM shuffle, except with
50+ ` spark.comet.convert.shuffleInput.enabled ` . Then ` CometExecRule ` puts a
51+ ` CometSparkToColumnarExec ` over the child of a shuffle that JVM shuffle would take, which
52+ converts the child's rows to Arrow, and uses native shuffle, provided native shuffle supports
53+ the partitioning and the columns. A shuffle that hashes a string, a value computed from a
54+ string, or a decimal wider than 18 digits stays on JVM shuffle, because native shuffle would
55+ not put every row in the partition Spark's partitioner does
56+ (` CometShuffleExchangeExec.convertsInputForNativeShuffle ` ).
5057
51583 . ** Supported partitioning type** : Native shuffle supports:
5259 - ` HashPartitioning `
@@ -154,11 +161,14 @@ The native shuffle implementation is its own workspace crate, `datafusion-comet-
154161 - The child plan's ` nativeOp ` directly, when ` CometShuffleExchangeExec ` 's child is a
155162 ` CometNativeExec ` subtree. The upstream operators run inside the same ` CometExecIterator `
156163 as the writer, with no JVM-to-native batch boundary between them.
157- - A synthetic ` Scan("ShuffleWriterInput") ` placeholder, when the dep was built via the
158- convenience ` prepareShuffleDependency(rdd, ...) ` overload (used by
159- ` CometCollectLimitExec ` and ` CometTakeOrderedAndProjectExec ` , or when the
160- exchange's child is a non-native ` CometPlan ` such as ` CometSparkToColumnarExec ` ). Native
161- code reads ` ColumnarBatch ` es from the JVM input iterator via Arrow C Stream Interface.
164+ - A synthetic ` Scan("ShuffleWriterInput") ` placeholder, when the exchange's child is a
165+ non-native ` CometPlan ` , and for the dependencies that ` CometCollectLimitExec ` and
166+ ` CometTakeOrderedAndProjectExec ` build. Native code reads the input through the Arrow C
167+ Stream Interface. A ` CometNativeArrowSource ` child, such as ` CometSparkToColumnarExec ` ,
168+ exports its own Arrow stream from ` doExecuteAsArrowStream() ` , which
169+ ` prepareArrowStreamShuffleDependency ` hands to native. Any other input is an
170+ ` RDD[ColumnarBatch] ` , which the convenience ` prepareShuffleDependency(rdd, ...) ` overload
171+ wraps in a stream.
162172
1631732 . ** Native execution** : A single ` CometExecIterator ` per partition runs the unified plan.
164174
0 commit comments