@@ -66,13 +66,14 @@ write side. To restrict columnar shuffle to cases where the child is already a C
6666as native Spark shuffles, which avoids the row to columnar conversion but means the downstream stage will also start
6767on Spark.
6868
69- Setting ` spark.comet.convert.shuffleInput.enabled=true ` makes such a shuffle use native shuffle instead. Comet converts
70- the child's rows to Arrow, as the ` spark.comet.convert.* ` settings do for Spark scans, and shuffles the batches
71- natively, which is usually faster than columnar shuffle. This applies only where native shuffle supports the
72- partitioning and Comet can convert all of the child's columns, which rules out calendar intervals, and arrays and maps
73- other than ` array<string> ` and ` map<string,string> ` . A shuffle that hashes a decimal wider than 18 digits, or a string
74- or a value computed from a string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted
75- batches are allocated on the JVM outside Comet's memory pools. The setting is disabled by default.
69+ Setting ` spark.comet.convert.shuffleInput.enabled=true ` makes such a shuffle use native shuffle instead. Comet
70+ converts the child's rows to Arrow, as the ` spark.comet.convert.* ` settings do for Spark scans, and shuffles the
71+ batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning
72+ the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert
73+ all of the child's columns, which rules out calendar intervals, and arrays and maps other than ` array<string> ` and
74+ ` map<string,string> ` . A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a
75+ string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on
76+ the JVM outside Comet's memory pools. The setting is disabled by default.
7677
7778### Automatic Revert to Spark Shuffle
7879
0 commit comments