Skip to content

Commit d8d7ce4

Browse files
committed
fix: leave calendar intervals out of the shuffle input conversion
Arrow holds the time part of an interval in nanoseconds, so IntervalMonthDayNanoWriter overflows on a calendar interval with more microseconds than that can hold, which Spark accepts. The JVM columnar shuffle leaves calendar intervals to Spark's shuffle, and the conversion now does too.
1 parent 16a1f73 commit d8d7ce4

4 files changed

Lines changed: 26 additions & 9 deletions

File tree

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,10 +69,10 @@ on Spark.
6969
Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use native shuffle instead. Comet converts
7070
the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the batches
7171
natively, which is usually faster than columnar shuffle. This applies only where native shuffle supports the
72-
partitioning and Comet can convert all of the child's columns, which rules out arrays and maps other than
73-
`array<string>` and `map<string,string>`. A shuffle that hashes a string or a decimal wider than 18 digits stays on
74-
columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on the JVM outside
75-
Comet's memory pools. The setting is disabled by default.
72+
partitioning and Comet can convert all of the child's columns, which rules out calendar intervals, and arrays and maps
73+
other than `array<string>` and `map<string,string>`. A shuffle that hashes a string or a decimal wider than 18 digits
74+
stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on the JVM
75+
outside Comet's memory pools. The setting is disabled by default.
7676

7777
### Automatic Revert to Spark Shuffle
7878

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -210,9 +210,9 @@ object CometConf extends ShimCometConf {
210210
"When enabled, a shuffle that would use Comet's JVM columnar shuffle because its input " +
211211
"comes from a Spark operator converts that input to Arrow format and uses Comet " +
212212
"native shuffle instead. This applies only where native shuffle supports the " +
213-
"partitioning and the input's columns. A shuffle that hashes a string or a decimal " +
214-
"wider than 18 digits stays on the JVM columnar shuffle, so that it partitions rows " +
215-
"as Spark does.")
213+
"partitioning and Comet can convert all of the input's columns, which rules out " +
214+
"calendar intervals. A shuffle that hashes a string or a decimal wider than 18 " +
215+
"digits stays on the JVM columnar shuffle, so that it partitions rows as Spark does.")
216216
.booleanConf
217217
.createWithDefault(false)
218218

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ import org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec
4343
import org.apache.spark.sql.execution.exchange.{ENSURE_REQUIREMENTS, ShuffleExchangeExec, ShuffleExchangeLike, ShuffleOrigin}
4444
import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics, SQLShuffleReadMetricsReporter, SQLShuffleWriteMetricsReporter}
4545
import org.apache.spark.sql.internal.SQLConf
46-
import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType, ByteType, DataType, DateType, DecimalType, DoubleType, FloatType, IntegerType, LongType, MapType, ShortType, StringType, StructField, StructType, TimestampNTZType, TimestampType}
46+
import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType, ByteType, CalendarIntervalType, DataType, DateType, DecimalType, DoubleType, FloatType, IntegerType, LongType, MapType, ShortType, StringType, StructField, StructType, TimestampNTZType, TimestampType}
4747
import org.apache.spark.sql.vectorized.ColumnarBatch
4848
import org.apache.spark.util.MutablePair
4949
import org.apache.spark.util.collection.unsafe.sort.{PrefixComparators, RecordComparator}
@@ -569,6 +569,10 @@ object CometShuffleExchangeExec
569569
!isShuffleOperator(s.child) &&
570570
!stageContainsDPPScan(s) &&
571571
CometSparkToColumnarExec.isSchemaSupported(s.child.schema, ListBuffer.empty) &&
572+
// Arrow holds the time part of an interval in nanoseconds, so the conversion overflows on a
573+
// calendar interval with more microseconds than that can hold. The JVM columnar shuffle
574+
// leaves calendar intervals to Spark's shuffle as well.
575+
!s.child.schema.existsRecursively(_.isInstanceOf[CalendarIntervalType]) &&
572576
nativeShuffleFailureReasons(s).isEmpty &&
573577
!hashesDifferentlyFromSpark(s)
574578
}

‎spark/src/test/scala/org/apache/comet/exec/CometShuffleInputConversionSuite.scala‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ import org.apache.spark.sql.comet.execution.shuffle.{CometColumnarShuffle, Comet
2525
import org.apache.spark.sql.execution.{ColumnarToRowExec, ColumnarToRowTransition, SparkPlan}
2626
import org.apache.spark.sql.functions.{array, avg, col, count, length, max, min, size, sum}
2727
import org.apache.spark.sql.internal.SQLConf
28-
import org.apache.spark.sql.types.{ArrayType, BinaryType, DataTypes, DecimalType, IntegerType, LongType, StringType, StructType}
28+
import org.apache.spark.sql.types.{ArrayType, BinaryType, CalendarIntervalType, DataTypes, DecimalType, IntegerType, LongType, StringType, StructType}
29+
import org.apache.spark.unsafe.types.CalendarInterval
2930

3031
import org.apache.comet.{CometConf, ExtendedExplainInfo}
3132

@@ -260,6 +261,18 @@ class CometShuffleInputConversionSuite extends CometTestBase {
260261
}
261262
}
262263

264+
convertTest("calendar intervals keep Spark's shuffle") {
265+
// Arrow holds an interval's time part in nanoseconds, which can't hold every number of
266+
// microseconds Spark can. The JVM columnar shuffle doesn't take calendar intervals either.
267+
val schema = new StructType().add("k", IntegerType).add("i", CalendarIntervalType)
268+
val data = Seq(Row(1, new CalendarInterval(0, 0, 10800000000000000L)))
269+
val df = spark
270+
.createDataFrame(spark.sparkContext.parallelize(data, 1), schema)
271+
.repartition(2, col("k"))
272+
val (_, plan) = checkSparkAnswer(df)
273+
assert(cometShuffles(plan).isEmpty, plan)
274+
}
275+
263276
convertTest("columns the conversion does not support keep the JVM columnar shuffle") {
264277
val df = arraysDf().repartition(5, col("k")).groupBy("k").agg(sum(size(col("xs"))))
265278
val (_, plan) = checkSparkAnswer(df)

0 commit comments

Comments
 (0)