Skip to content

Commit 15305af

Browse files
compheadandygrove
andauthored
fix: apply the session's Comet configs in map stages of jobs outside a SQL execution (#7302)
* fix: apply the session's Comet configs in map stages of jobs outside a SQL execution On an executor, SQLConf.get reads the task's local properties, which Spark fills with the session's configs only for jobs that run inside a SQL execution. A job launched from a query's RDD (Dataset.rdd with AQE off, QueryExecution.toRdd) runs its map stages outside one, and Spark's SQLExecutionRDD re-applies the session's configs only in the final stage. Comet reads its settings on the executor when it builds a native plan, so those map stages ran with every spark.comet.* setting at its default, including the memory pool and the configs passed to native. Capture the session's configs on the driver where Comet builds the RDDs and native shuffle that run native plans, and make them what SQLConf.get returns on the task thread while the plan is built, as SQLExecutionRDD does. Tasks inside a SQL execution are unchanged. Closes #7301. * refactor: capture only Comet configs once per shuffle and let configs Spark passed win Capture only spark.comet.* keys, and apply them only when no conf is set on the thread and the task got none of the session's Comet configs from Spark, so SQLExecution.withSQLConfPropagated still takes effect. Carry the native shuffle's one captured conf from its input RDD to the writer, apply it inside the serialized-plan getCometIterator, and drop the test workarounds this change makes redundant. * fix: apply the session's Comet configs to shuffle reads and test each native plan site A Comet shuffle read with no native plan above it in its stage, such as one feeding a Spark operator in an intermediate stage, still read Comet defaults. Capture the configs in CometShuffledBatchRDD and apply them in compute and computeAsShuffleBlockIterator, which makes the wrap in CometNativeShuffleInputRDD.compute redundant, so drop it. Make that RDD's sqlConf required so each call site captures on the driver explicitly. Replace the single test with one per site: the native shuffle writer, a native plan in a map stage, the native limit, the native top-K, each checking the memory pool every native plan resolves, and a shuffle read under spark.comet.shuffle.readBufferSize=0. * fix: apply the captured configs to the Celeborn frame size and capture DEBUG logs in tests The native Celeborn writer read spark.comet.shuffle.rss.maxFrameBytes in getWriter, before the writer applies the captured configs. Carry the configs on NativeShuffleSpec, which getWriter can reach through the dependency, and apply them to that read as well as in the writer, in place of CometNativeShuffleInputRDD and its iterator. Spark's LogAppender keeps INFO and above by default, so set its threshold to DEBUG for the memory pool line the tests read. --------- Co-authored-by: Andy Grove <[email protected]>
1 parent 962db53 commit 15305af

15 files changed

Lines changed: 219 additions & 41 deletions
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.spark.sql.comet
21+
22+
import org.apache.spark.TaskContext
23+
import org.apache.spark.sql.internal.{ReadOnlySQLConf, SQLConf}
24+
25+
import org.apache.comet.CometConf
26+
27+
/**
28+
* The session's `spark.comet.*` configs, captured on the driver. Spark passes them to tasks as
29+
* local properties only inside a SQL execution, so jobs launched from `Dataset.rdd` or
30+
* `QueryExecution.toRdd` read Comet defaults in every stage before the last, which
31+
* `SQLExecutionRDD` covers. [[withConf]] fills that gap the same way.
32+
*/
33+
class CapturedSQLConf private (configs: Map[String, String]) extends Serializable {
34+
35+
@transient private lazy val executorSideConf: SQLConf = {
36+
val conf = new SQLConf()
37+
configs.foreach { case (key, value) => conf.setConfString(key, value) }
38+
conf
39+
}
40+
41+
/**
42+
* Runs `f` with the captured configs as `SQLConf.get`, unless this is not a task thread, a conf
43+
* is already set on it, or Spark passed the task the session's configs (it passes all or none).
44+
*/
45+
def withConf[T](f: => T): T = {
46+
val context = TaskContext.get()
47+
if (configs.isEmpty || context == null || !SQLConf.get.isInstanceOf[ReadOnlySQLConf] ||
48+
configs.keys.exists(context.getLocalProperty(_) != null)) {
49+
f
50+
} else {
51+
SQLConf.withExistingConf(executorSideConf)(f)
52+
}
53+
}
54+
}
55+
56+
object CapturedSQLConf {
57+
58+
/** Captures the active session's Comet configs. Call it on the driver. */
59+
def capture(): CapturedSQLConf =
60+
new CapturedSQLConf(SQLConf.get.getAllConfs.filter(_._1.startsWith(CometConf.COMET_PREFIX)))
61+
}

‎spark/src/main/scala/org/apache/spark/sql/comet/CometExecRDD.scala‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ private[spark] class CometExecRDD(
9393
s"All per-partition arrays must have length $numPartitions, but found: " +
9494
perPartitionByKey.map { case (key, arr) => s"$key -> ${arr.length}" }.mkString(", "))
9595

96+
private val sqlConf = CapturedSQLConf.capture()
97+
9698
override protected def getPartitions: Array[Partition] = {
9799
(0 until numPartitions).map { idx =>
98100
val inputParts = inputRDDs.map(_.partitions(idx)).toArray
@@ -103,7 +105,10 @@ private[spark] class CometExecRDD(
103105
}.toArray
104106
}
105107

106-
override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = {
108+
override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] =
109+
sqlConf.withConf(computeNative(split, context))
110+
111+
private def computeNative(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = {
107112
// Must precede resolveInputObjects and the CometExecIterator: completion listeners run in
108113
// reverse registration order, so registering first means this listener runs last, after
109114
// nested native blocks and the iterator have published their final metric values.

‎spark/src/main/scala/org/apache/spark/sql/comet/CometExecUtils.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,13 +72,15 @@ object CometExecUtils {
7272
val limitOp = CometExecUtils.getLimitNativePlan(outputAttribute, limit, offset).get
7373
val serializedPlan = CometExec.serializeNativePlan(limitOp)
7474
val inputSchema = Utils.fromAttributes(outputAttribute)
75+
val sqlConf = CapturedSQLConf.capture()
7576
childPlan.mapPartitionsWithIndexInternal { case (idx, iter) =>
7677
CometExec.getCometIterator(
7778
CometArrowStream.inputObjects(iter, inputSchema, "CometExecUtils-getNativeLimit"),
7879
outputAttribute.length,
7980
serializedPlan,
8081
numParts,
8182
idx,
83+
sqlConf,
8284
outputAttribute.map(_.dataType))
8385
}
8486
}

‎spark/src/main/scala/org/apache/spark/sql/comet/CometTakeOrderedAndProjectExec.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ case class CometTakeOrderedAndProjectExec(
155155
if (childRDD.getNumPartitions == 0 || limit == 0) {
156156
new ParallelCollectionRDD(sparkContext, Seq.empty[ColumnarBatch], 1, Map.empty)
157157
} else {
158+
val sqlConf = CapturedSQLConf.capture()
158159
val singlePartitionRDD = if (childRDD.getNumPartitions == 1) {
159160
childRDD
160161
} else {
@@ -178,6 +179,7 @@ case class CometTakeOrderedAndProjectExec(
178179
serializedTopK,
179180
numParts,
180181
idx,
182+
sqlConf,
181183
child.output.map(_.dataType))
182184
}
183185
}
@@ -207,6 +209,7 @@ case class CometTakeOrderedAndProjectExec(
207209
serializedFinalPlan,
208210
1,
209211
0,
212+
sqlConf,
210213
finalOutputTypes)
211214
setSubqueries(it.id, this)
212215

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometCelebornShuffleManager.scala‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -170,10 +170,11 @@ class CometCelebornShuffleManager private[shuffle] (
170170
.computeIfAbsent(handle.shuffleId, _ => new ConcurrentHashMap[Int, AnyRef]())
171171
.put(resolved.celebornShuffleId, resolved.client)
172172

173+
val spec = dependency.nativeShuffleSpec.getOrElse {
174+
throw new IllegalStateException("Native Comet shuffle has no execution plan")
175+
}
173176
new CometNativeShuffleWriter[K, V](
174-
dependency.nativeShuffleSpec.getOrElse {
175-
throw new IllegalStateException("Native Comet shuffle has no execution plan")
176-
},
177+
spec,
177178
dependency.outputPartitioning.getOrElse {
178179
throw new IllegalStateException("Native Comet shuffle has no output partitioning")
179180
},
@@ -189,7 +190,7 @@ class CometCelebornShuffleManager private[shuffle] (
189190
CelebornNativeShuffleDestination(
190191
resolved.pusher,
191192
CometCelebornShuffleManager.maxNativeFrameBytes(
192-
CometConf.COMET_SHUFFLE_RSS_MAX_FRAME_BYTES.get().toInt,
193+
spec.sqlConf.withConf(CometConf.COMET_SHUFFLE_RSS_MAX_FRAME_BYTES.get().toInt),
193194
resolved.pusher),
194195
dependency.partitioner.numPartitions,
195196
commitAuthorized = true,

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativeShuffleWriter.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ class CometNativeShuffleWriter[K, V](
9595

9696
override def write(inputs: Iterator[Product2[K, V]]): Unit = {
9797
try {
98-
writeInternal(inputs)
98+
spec.sqlConf.withConf(writeInternal(inputs))
9999
} catch {
100100
case failure: Throwable =>
101101
remoteDestination.foreach { destination =>

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleDependency.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ import org.apache.spark.shuffle.ShuffleWriteProcessor
2828
import org.apache.spark.sql.catalyst.InternalRow
2929
import org.apache.spark.sql.catalyst.expressions.Attribute
3030
import org.apache.spark.sql.catalyst.plans.physical.Partitioning
31-
import org.apache.spark.sql.comet.{CometMetricNode, NativeExecContext}
31+
import org.apache.spark.sql.comet.{CapturedSQLConf, CometMetricNode, NativeExecContext}
3232
import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
3333
import org.apache.spark.sql.execution.metric.SQLMetric
3434
import org.apache.spark.sql.types.StructType
@@ -44,6 +44,8 @@ case class NativeShuffleSpec(
4444
childNativeOp: OperatorOuterClass.Operator,
4545
childMetricNode: CometMetricNode,
4646
execContext: NativeExecContext,
47+
/** The session's Comet configs, captured on the driver; see [[CapturedSQLConf]]. */
48+
sqlConf: CapturedSQLConf,
4749
/**
4850
* Set when round-robin placement is positional rather than content-hashed. Both the decision
4951
* and the group size are resolved once on the driver: the decision because it depends on the

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ import org.apache.spark.sql.catalyst.expressions.{Attribute, BoundReference, Exp
3636
import org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering
3737
import org.apache.spark.sql.catalyst.plans.logical.Statistics
3838
import org.apache.spark.sql.catalyst.plans.physical._
39-
import org.apache.spark.sql.comet.{CometFilterExec, CometMetricNode, CometNativeExec, CometNativeScanExec, CometPlan, CometProjectExec, CometSinkPlaceHolder, CometSparkToColumnarExec, NativeExecContext}
39+
import org.apache.spark.sql.comet.{CapturedSQLConf, CometFilterExec, CometMetricNode, CometNativeExec, CometNativeScanExec, CometPlan, CometProjectExec, CometSinkPlaceHolder, CometSparkToColumnarExec, NativeExecContext}
4040
import org.apache.spark.sql.comet.execution.arrow.{CometArrowStream, CometNativeArrowSource}
4141
import org.apache.spark.sql.execution._
4242
import org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec
@@ -235,6 +235,7 @@ case class CometShuffleExchangeExec(
235235
nativeChild.nativeOp,
236236
nativeChildMetricNode,
237237
ctx,
238+
CapturedSQLConf.capture(),
238239
positionalRoundRobin))
239240
case None =>
240241
child match {
@@ -968,7 +969,7 @@ object CometShuffleExchangeExec
968969
outputPartitioning,
969970
serializer,
970971
metrics,
971-
NativeShuffleSpec(scanOp, childMetricNode, ctx))
972+
NativeShuffleSpec(scanOp, childMetricNode, ctx, CapturedSQLConf.capture()))
972973
}
973974

974975
/**

‎spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffledRowRDD.scala‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ package org.apache.spark.sql.comet.execution.shuffle
2222
import org.apache.spark._
2323
import org.apache.spark.rdd.RDD
2424
import org.apache.spark.shuffle.sort.SortShuffleManager
25+
import org.apache.spark.sql.comet.CapturedSQLConf
2526
import org.apache.spark.sql.execution._
2627
import org.apache.spark.sql.execution.metric.{SQLMetric, SQLShuffleReadMetricsReporter}
2728
import org.apache.spark.sql.internal.SQLConf
@@ -47,6 +48,8 @@ class CometShuffledBatchRDD(
4748
Array.tabulate(dependency.partitioner.numPartitions)(i => CoalescedPartitionSpec(i, i + 1)))
4849
}
4950

51+
private val sqlConf = CapturedSQLConf.capture()
52+
5053
dependency.rdd.context.setLocalProperty(
5154
SortShuffleManager.FETCH_SHUFFLE_BLOCKS_IN_BATCH_ENABLED_KEY,
5255
SQLConf.get.fetchShuffleBlocksInBatch.toString)
@@ -166,7 +169,7 @@ class CometShuffledBatchRDD(
166169
*/
167170
def computeAsShuffleBlockIterator(
168171
split: Partition,
169-
context: TaskContext): CometShuffleBlockIterator = {
172+
context: TaskContext): CometShuffleBlockIterator = sqlConf.withConf {
170173
val readerAndMetrics: (CometShuffleReader[_, _], SQLShuffleReadMetricsReporter) =
171174
createReader(split, context)
172175
// Register before CometExecIterator so Spark's LIFO completion order closes the native stream
@@ -180,11 +183,12 @@ class CometShuffledBatchRDD(
180183
CometConf.COMET_SHUFFLE_READ_BUFFER_SIZE.get().toInt)
181184
}
182185

183-
override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] = {
184-
val reader: CometShuffleReader[_, _] = createReader(split, context)._1
185-
// TODO: Reads IPC by native code
186-
reader.read().asInstanceOf[Iterator[Product2[Int, ColumnarBatch]]].map(_._2)
187-
}
186+
override def compute(split: Partition, context: TaskContext): Iterator[ColumnarBatch] =
187+
sqlConf.withConf {
188+
val reader: CometShuffleReader[_, _] = createReader(split, context)._1
189+
// TODO: Reads IPC by native code
190+
reader.read().asInstanceOf[Iterator[Product2[Int, ColumnarBatch]]].map(_._2)
191+
}
188192

189193
override def clearDependencies(): Unit = {
190194
super.clearDependencies()

‎spark/src/main/scala/org/apache/spark/sql/comet/operators.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -810,14 +810,16 @@ object CometExec {
810810
/**
811811
* Create a CometExecIterator with a pre-serialized native plan. Use this overload when
812812
* executing the same plan across multiple partitions to avoid serializing the plan repeatedly.
813+
* `sqlConf` is captured on the driver; see [[CapturedSQLConf]].
813814
*/
814815
def getCometIterator(
815816
inputObjects: Array[Object],
816817
numOutputCols: Int,
817818
serializedPlan: Array[Byte],
818819
numParts: Int,
819820
partitionIdx: Int,
820-
outputTypes: Seq[DataType] = Seq.empty): CometExecIterator = {
821+
sqlConf: CapturedSQLConf,
822+
outputTypes: Seq[DataType]): CometExecIterator = sqlConf.withConf {
821823
new CometExecIterator(
822824
newIterId,
823825
inputObjects,

0 commit comments

Comments
 (0)