Skip to content

Commit a20a4a5

Browse files
committed
fix: keep a shuffle whose key is computed from a string on the JVM columnar shuffle
The string guard looked only at the type of each hash key, so a key that is a number computed from a string, such as hash(s), passed it. Native shuffle evaluates such a key after the import into native has replaced invalid UTF-8 in the string, so it can still send a row to a different partition than Spark's partitioner does, and a join whose other input stays on the JVM columnar shuffle loses the matches. The guard now looks for a string anywhere in the key expression. A wide decimal still matters only as the type of the key itself, because Comet's hash of a wide decimal already falls back to Spark. The string join test also joins on the hash of the strings, and uses two keys instead of four. Both shuffles write the strings with invalid UTF-8 replaced, so the keys stay apart only while each key expression sends them to different partitions, and two of the four hashes share one.
1 parent d597480 commit a20a4a5

4 files changed

Lines changed: 28 additions & 20 deletions

File tree

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,9 @@ Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use
7070
the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the batches
7171
natively, which is usually faster than columnar shuffle. This applies only where native shuffle supports the
7272
partitioning and Comet can convert all of the child's columns, which rules out calendar intervals, and arrays and maps
73-
other than `array<string>` and `map<string,string>`. A shuffle that hashes a string or a decimal wider than 18 digits
74-
stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on the JVM
75-
outside Comet's memory pools. The setting is disabled by default.
73+
other than `array<string>` and `map<string,string>`. A shuffle that hashes a decimal wider than 18 digits, or a string
74+
or a value computed from a string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted
75+
batches are allocated on the JVM outside Comet's memory pools. The setting is disabled by default.
7676

7777
### Automatic Revert to Spark Shuffle
7878

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -247,8 +247,9 @@ object CometConf extends ShimCometConf {
247247
"comes from a Spark operator converts that input to Arrow format and uses Comet " +
248248
"native shuffle instead. This applies only where native shuffle supports the " +
249249
"partitioning and Comet can convert all of the input's columns, which rules out " +
250-
"calendar intervals. A shuffle that hashes a string or a decimal wider than 18 " +
251-
"digits stays on the JVM columnar shuffle, so that it partitions rows as Spark does.")
250+
"calendar intervals. A shuffle that hashes a decimal wider than 18 digits, or a " +
251+
"string or a value computed from a string, stays on the JVM columnar shuffle, so " +
252+
"that it partitions rows as Spark does.")
252253
.booleanConf
253254
.createWithDefault(false)
254255

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -584,18 +584,19 @@ object CometShuffleExchangeExec
584584
* still partitioned by Spark would put matching keys in different partitions. The keys are:
585585
*
586586
* - A decimal wider than 18 digits, which native shuffle hashes differently (#5994).
587-
* - A string. Spark hashes the string's bytes as they are, but the import of the converted
588-
* batch into native replaces invalid UTF-8 before native shuffle hashes it.
587+
* - A string, or any value computed from one, such as `hash(s)`. Spark reads the string's
588+
* bytes as they are, but the import of the converted batch into native replaces invalid
589+
* UTF-8 before native shuffle evaluates the key.
589590
*
590591
* TODO: allow wide decimals once native hashing matches Spark for them.
591592
*/
592593
private def hashesDifferentlyFromSpark(s: ShuffleExchangeExec): Boolean =
593594
s.outputPartitioning match {
594595
case HashPartitioning(expressions, numPartitions) =>
595-
numPartitions > 1 && expressions.exists(_.dataType.existsRecursively {
596-
case _: StringType => true
597-
case dt => DecimalType.isByteArrayDecimalType(dt)
598-
})
596+
numPartitions > 1 && expressions.exists { key =>
597+
key.dataType.existsRecursively(DecimalType.isByteArrayDecimalType) ||
598+
key.exists(_.dataType.existsRecursively(_.isInstanceOf[StringType]))
599+
}
599600
case _ => false
600601
}
601602

‎spark/src/test/scala/org/apache/comet/exec/CometShuffleInputConversionSuite.scala‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ import org.apache.spark.sql.{CometTestBase, DataFrame, Row}
2323
import org.apache.spark.sql.comet.CometSparkToColumnarExec
2424
import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, CometNativeShuffle, CometShuffleExchangeExec}
2525
import org.apache.spark.sql.execution.{ColumnarToRowExec, ColumnarToRowTransition, SparkPlan}
26-
import org.apache.spark.sql.functions.{array, avg, col, count, length, max, min, size, sum}
26+
import org.apache.spark.sql.functions.{array, avg, col, count, hash, length, max, min, size, sum}
2727
import org.apache.spark.sql.internal.SQLConf
2828
import org.apache.spark.sql.types.{ArrayType, BinaryType, CalendarIntervalType, DataTypes, DecimalType, IntegerType, LongType, StringType, StructType}
2929
import org.apache.spark.unsafe.types.CalendarInterval
@@ -221,24 +221,30 @@ class CometShuffleInputConversionSuite extends CometTestBase {
221221
// Spark's partitioner hashes a string's bytes as they are, but native shuffle hashes them
222222
// after the import into native has replaced invalid UTF-8. The right input, with its
223223
// array<int> column, stays on the JVM columnar shuffle, so the left one has to as well, or
224-
// matching keys with invalid UTF-8 land in different partitions. Both shuffles write the rows
225-
// with invalid UTF-8 replaced, so the four keys stay apart only because they hash to four
226-
// different partitions.
224+
// matching keys with invalid UTF-8 land in different partitions. A key computed from a
225+
// string, such as its hash, is no different: native shuffle would compute it from the
226+
// replaced string. Both shuffles write the rows with invalid UTF-8 replaced, and the join
227+
// compares what they wrote, so the two keys stay apart only because each key expression puts
228+
// them in different partitions.
227229
withSQLConf(
228230
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
229231
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
230232
SQLConf.SHUFFLE_PARTITIONS.key -> "10") {
231233
val schema = new StructType().add("b", BinaryType).add("v", LongType)
232-
val data = (0 until 40).map(i => Row(Array((0x80 + i % 4).toByte), i.toLong))
234+
val data = (0 until 40).map { i =>
235+
Row(Array((if (i % 2 == 0) 0x80 else 0x83).toByte), i.toLong)
236+
}
233237
val keys = spark.createDataFrame(spark.sparkContext.parallelize(data, 4), schema)
234238
val left = keys.select(col("b").cast(StringType).as("lk"), col("v"))
235239
val right = keys.select(
236240
col("b").cast(StringType).as("rk"),
237241
array(col("v").cast(IntegerType)).as("xs"))
238-
val df = left.join(right, col("lk") === col("rk"))
239-
val (_, plan) = checkSparkAnswer(df)
240-
assert(conversions(plan).isEmpty, plan)
241-
checkCometExchange(df, 2, native = false)
242+
Seq(col("lk") === col("rk"), hash(col("lk")) === hash(col("rk"))).foreach { condition =>
243+
val df = left.join(right, condition)
244+
val (_, plan) = checkSparkAnswer(df)
245+
assert(conversions(plan).isEmpty, s"$condition:\n$plan")
246+
checkCometExchange(df, 2, native = false)
247+
}
242248
}
243249
}
244250

0 commit comments

Comments
 (0)