Skip to content

Commit 9795b61

Browse files
committed
fix: keep a shuffle that hashes a string on the JVM columnar shuffle
Spark's partitioner hashes a string's bytes as they are. Once CometSparkToColumnarExec has converted the rows, the import into native replaces invalid UTF-8 before native shuffle hashes the string, so such a key can go to a different partition than Spark's partitioner sends it to. A join whose other input stays on the JVM columnar shuffle then loses the matches for that key. The conversion now leaves a shuffle that hashes a string on the JVM columnar shuffle, as it does one that hashes a wide decimal.
1 parent cd105b1 commit 9795b61

5 files changed

Lines changed: 66 additions & 32 deletions

File tree

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,9 @@ Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use
7070
the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the batches
7171
natively, which is usually faster than columnar shuffle. This applies only where native shuffle supports the
7272
partitioning and Comet can convert all of the child's columns, which rules out arrays and maps other than
73-
`array<string>` and `map<string,string>`. A shuffle that hashes a decimal wider than 18 digits stays on columnar shuffle,
74-
so that it partitions rows as Spark does. The converted batches are allocated on the JVM outside Comet's memory pools.
75-
The setting is disabled by default.
73+
`array<string>` and `map<string,string>`. A shuffle that hashes a string or a decimal wider than 18 digits stays on
74+
columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on the JVM outside
75+
Comet's memory pools. The setting is disabled by default.
7676

7777
### Automatic Revert to Spark Shuffle
7878

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -210,9 +210,9 @@ object CometConf extends ShimCometConf {
210210
"When enabled, a shuffle that would use Comet's JVM columnar shuffle because its input " +
211211
"comes from a Spark operator converts that input to Arrow format and uses Comet " +
212212
"native shuffle instead. This applies only where native shuffle supports the " +
213-
"partitioning and the input's columns. A shuffle that hashes a decimal wider than 18 " +
214-
"digits stays on the JVM columnar shuffle, because native shuffle does not hash " +
215-
"those as Spark does.")
213+
"partitioning and the input's columns. A shuffle that hashes a string or a decimal " +
214+
"wider than 18 digits stays on the JVM columnar shuffle, so that it partitions rows " +
215+
"as Spark does.")
216216
.booleanConf
217217
.createWithDefault(false)
218218

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala‎

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -547,22 +547,30 @@ object CometShuffleExchangeExec
547547
!stageContainsDPPScan(s) &&
548548
CometSparkToColumnarExec.isSchemaSupported(s.child.schema, ListBuffer.empty) &&
549549
nativeShuffleFailureReasons(s).isEmpty &&
550-
!hashesWideDecimal(s)
550+
!hashesDifferentlyFromSpark(s)
551551
}
552552

553553
/**
554-
* Whether the shuffle hashes a decimal wider than 18 digits into more than one partition.
555-
* Native shuffle hashes those differently from Spark's partitioner, so it must not take over
556-
* such a shuffle from the JVM columnar shuffle: a join with an input that is still partitioned
557-
* by Spark would put matching keys in different partitions. TODO: remove once native hashing
558-
* matches Spark for wide decimals (#5994).
554+
* Whether the shuffle hashes into more than one partition a key that native shuffle, reading
555+
* converted rows, may put in a different partition from Spark's partitioner. Native shuffle
556+
* must not take over such a shuffle from the JVM columnar shuffle: a join with an input that is
557+
* still partitioned by Spark would put matching keys in different partitions. The keys are:
558+
*
559+
* - A decimal wider than 18 digits, which native shuffle hashes differently (#5994).
560+
* - A string. Spark hashes the string's bytes as they are, but the import of the converted
561+
* batch into native replaces invalid UTF-8 before native shuffle hashes it.
562+
*
563+
* TODO: allow wide decimals once native hashing matches Spark for them.
559564
*/
560-
private def hashesWideDecimal(s: ShuffleExchangeExec): Boolean = s.outputPartitioning match {
561-
case HashPartitioning(expressions, numPartitions) =>
562-
numPartitions > 1 &&
563-
expressions.exists(_.dataType.existsRecursively(DecimalType.isByteArrayDecimalType))
564-
case _ => false
565-
}
565+
private def hashesDifferentlyFromSpark(s: ShuffleExchangeExec): Boolean =
566+
s.outputPartitioning match {
567+
case HashPartitioning(expressions, numPartitions) =>
568+
numPartitions > 1 && expressions.exists(_.dataType.existsRecursively {
569+
case _: StringType => true
570+
case dt => DecimalType.isByteArrayDecimalType(dt)
571+
})
572+
case _ => false
573+
}
566574

567575
/**
568576
* Reasons the native shuffle path cannot handle this shuffle. Empty means native is supported.

‎spark/src/test/scala/org/apache/comet/exec/CometShuffleInputConversionSuite.scala‎

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,9 @@ import org.apache.spark.sql.{CometTestBase, DataFrame, Row}
2323
import org.apache.spark.sql.comet.CometSparkToColumnarExec
2424
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometNativeShuffle, CometShuffleExchangeExec}
2525
import org.apache.spark.sql.execution.{ColumnarToRowExec, ColumnarToRowTransition, SparkPlan}
26-
import org.apache.spark.sql.functions.{avg, col, count, length, max, min, size, sum}
26+
import org.apache.spark.sql.functions.{array, avg, col, count, length, max, min, size, sum}
2727
import org.apache.spark.sql.internal.SQLConf
28-
import org.apache.spark.sql.types.{ArrayType, DataTypes, DecimalType, IntegerType, LongType, StructType}
28+
import org.apache.spark.sql.types.{ArrayType, BinaryType, DataTypes, DecimalType, IntegerType, LongType, StringType, StructType}
2929

3030
import org.apache.comet.{CometConf, ExtendedExplainInfo}
3131

@@ -145,17 +145,17 @@ class CometShuffleInputConversionSuite extends CometTestBase {
145145
val ds = spark.sql("SELECT _1 AS a, _2 AS b FROM tbl").as[ShuffleInputRec]
146146
// The shuffle reads the typed operation's SerializeFromObject.
147147
val (_, repartitioned) = checkSparkAnswer(
148-
ds.map(r => ShuffleInputRec(r.a + 1, r.b))
149-
.repartition(7, col("b"))
150-
.groupBy("b")
151-
.agg(sum("a")))
148+
ds.map(r => ShuffleInputRec(r.a % 13, r.b))
149+
.repartition(7, col("a"))
150+
.groupBy("a")
151+
.agg(count("b")))
152152
assert(convertedShuffles(repartitioned).length == 1, s"AQE $aqe:\n$repartitioned")
153153
assertRowOperatorsReadThroughTransitions(repartitioned)
154154
// The shuffle reads a Spark partial aggregate over the typed operation. The final
155155
// aggregate stays on Spark too, so the shuffle would go back to Spark's own.
156156
withSQLConf(CometConf.COMET_SHUFFLE_REVERT_REDUNDANT_COLUMNAR_ENABLED.key -> "false") {
157157
val (_, aggregated) =
158-
checkSparkAnswer(ds.map(r => ShuffleInputRec(r.a * 2, r.b)).groupBy("b").count())
158+
checkSparkAnswer(ds.map(r => ShuffleInputRec(r.a % 7, r.b)).groupBy("a").count())
159159
assert(convertedShuffles(aggregated).length == 1, s"AQE $aqe:\n$aggregated")
160160
assertRowOperatorsReadThroughTransitions(aggregated)
161161
}
@@ -215,6 +215,31 @@ class CometShuffleInputConversionSuite extends CometTestBase {
215215
}
216216
}
217217

218+
convertTest("a shuffle that hashes a string stays on the JVM columnar shuffle") {
219+
// Spark's partitioner hashes a string's bytes as they are, but native shuffle hashes them
220+
// after the import into native has replaced invalid UTF-8. The right input, with its
221+
// array<int> column, stays on the JVM columnar shuffle, so the left one has to as well, or
222+
// matching keys with invalid UTF-8 land in different partitions. Both shuffles write the rows
223+
// with invalid UTF-8 replaced, so the four keys stay apart only because they hash to four
224+
// different partitions.
225+
withSQLConf(
226+
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
227+
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
228+
SQLConf.SHUFFLE_PARTITIONS.key -> "10") {
229+
val schema = new StructType().add("b", BinaryType).add("v", LongType)
230+
val data = (0 until 40).map(i => Row(Array((0x80 + i % 4).toByte), i.toLong))
231+
val keys = spark.createDataFrame(spark.sparkContext.parallelize(data, 4), schema)
232+
val left = keys.select(col("b").cast(StringType).as("lk"), col("v"))
233+
val right = keys.select(
234+
col("b").cast(StringType).as("rk"),
235+
array(col("v").cast(IntegerType)).as("xs"))
236+
val df = left.join(right, col("lk") === col("rk"))
237+
val (_, plan) = checkSparkAnswer(df)
238+
assert(conversions(plan).isEmpty, plan)
239+
checkCometExchange(df, 2, native = false)
240+
}
241+
}
242+
218243
convertTest("columns the conversion does not support keep the JVM columnar shuffle") {
219244
val df = arraysDf().repartition(5, col("k")).groupBy("k").agg(sum(size(col("xs"))))
220245
val (_, plan) = checkSparkAnswer(df)

‎spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleInputConversionBenchmark.scala‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,10 @@ case class ShuffleInputBenchRec(a: Long, b: String)
4242
* - Comet, converted: `spark.comet.convert.shuffleInput.enabled`, which converts the rows with
4343
* `CometSparkToColumnarExec` and uses native shuffle.
4444
*
45-
* Each case hash-partitions its rows into 200 partitions and aggregates them. Every arm's result
46-
* and plan are checked before it is timed, and the Comet arm runs again at the end of each case
47-
* to show the noise. To run this benchmark:
45+
* Each case hash-partitions its rows into 200 partitions and aggregates them. The keys are
46+
* numbers, because the conversion leaves a shuffle that hashes a string on the JVM columnar
47+
* shuffle. Every arm's result and plan are checked before it is timed, and the Comet arm runs
48+
* again at the end of each case to show the noise. To run this benchmark:
4849
* {{{
4950
* SPARK_GENERATE_BENCHMARK_FILES=1 make benchmark-org.apache.spark.sql.benchmark.CometShuffleInputConversionBenchmark
5051
* }}}
@@ -136,10 +137,10 @@ object CometShuffleInputConversionBenchmark extends CometBenchmarkBase {
136137
.table("parquetV1Table")
137138
.select(col("id").as("a"), col("key").as("b"))
138139
.as[ShuffleInputBenchRec]
139-
.map(r => ShuffleInputBenchRec(r.a + 1, r.b))
140-
.repartition(numPartitions, col("b"))
141-
.groupBy("b")
142-
.agg(sum("a").as("s")))))
140+
.map(r => ShuffleInputBenchRec(r.a % 1000, r.b))
141+
.repartition(numPartitions, col("a"))
142+
.groupBy("a")
143+
.agg(sum(length(col("b"))).as("s")))))
143144

144145
private def runArm(arm: Arm, query: () => DataFrame): (Seq[Row], SparkPlan) = {
145146
var result: (Seq[Row], SparkPlan) = null

0 commit comments

Comments
 (0)