Skip to content

Commit 02e8404

Browse files
fix: read shuffle write buffer, spill limit and off-heap sizes in bytes (apache#6191)
* fix: read shuffle write buffer, spill limit and off-heap sizes in bytes spark.comet.shuffle.native.writeBufferSize was declared in MiB but its value was sent to native code as a byte count, so the native shuffle writer ran with a 1-byte write buffer by default. Declare it in bytes with a 1 MiB default; a bare number keeps its meaning. Native code parsed spark.comet.maxTempDirectorySize and the boolean flags it reads from the raw session strings, so a size with a unit or an upper-case boolean silently fell back to the native default. Resolve every config that native code reads on the JVM before it crosses JNI, and document that the spill limit applies per native plan rather than per task. getMemoryConfig read spark.memory.offHeap.size as MiB, while Spark reads a bare number as bytes. Closes apache#6183, apache#6184, apache#6185. * docs: give a real example of a task running several native plans A Spark operator that Comet does not support does not split a stage into two native plans: Comet does not resume native execution above it. The native operators on either side of a union or a coalesce do run as separate plans in the same task. * Update spark/src/main/scala/org/apache/comet/CometConf.scala Co-authored-by: Matt Butrovich <[email protected]> --------- Co-authored-by: Matt Butrovich <[email protected]>
1 parent 67803a7 commit 02e8404

9 files changed

Lines changed: 155 additions & 23 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -507,6 +507,7 @@ jobs:
507507
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleReaderSuite
508508
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShufflePlanningSuite
509509
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleInputRDDSuite
510+
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriterSuite
510511
org.apache.spark.sql.comet.execution.shuffle.CometNativePositionalRoundRobinSuite
511512
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
512513
org.apache.comet.exec.CometShuffleEncryptionSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ jobs:
155155
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleReaderSuite
156156
org.apache.spark.sql.comet.execution.shuffle.CometCelebornShufflePlanningSuite
157157
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleInputRDDSuite
158+
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriterSuite
158159
org.apache.spark.sql.comet.execution.shuffle.CometNativePositionalRoundRobinSuite
159160
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
160161
org.apache.comet.exec.CometShuffleEncryptionSuite

‎docs/source/user-guide/latest/tuning.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -283,12 +283,14 @@ flushes sorted spill files. It must not exceed `spark.comet.batchSize`.
283283

284284
### Limiting Spill Disk Usage
285285

286-
Native operators that spill to disk (aggregate, sort, shuffle) are collectively bounded by
287-
`spark.comet.maxTempDirectorySize` (default 100 GB). The limit is applied per Spark task, so an
288-
executor running `N` concurrent tasks may use up to `N` times this value on shared local disks.
289-
If the limit is reached, further spills fail and the query errors out. Raise this on workloads
290-
with large sort/aggregate/shuffle spills, or lower it to protect executors on shared disks
291-
(remembering to divide by task concurrency to reason about the aggregate).
286+
Native operators that spill to disk (aggregate, sort, shuffle) are bounded by
287+
`spark.comet.maxTempDirectorySize` (default 100 GB). The operators of one Comet native plan share
288+
the limit. A Spark task can run more than one native plan at a time, for example the native
289+
operators on either side of a union or a coalesce, so an executor running `N` concurrent tasks may
290+
use more than `N` times this value on shared local disks. If the limit is reached, further spills
291+
fail and the query errors out. Raise this on workloads with large sort/aggregate/shuffle spills, or
292+
lower it to protect executors on shared disks, remembering that the total across an executor is a
293+
multiple of this value.
292294

293295
## Parquet Reader Tuning
294296

‎native/core/src/execution/spark_config.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ pub(crate) const COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED: &str =
2626
"spark.comet.parquet.rowFilterPushdown.enabled";
2727
pub(crate) const SPARK_EXECUTOR_CORES: &str = "spark.executor.cores";
2828

29+
/// Comet configs read through this trait must be resolved by the JVM first:
30+
/// `CometExecIterator.serializeCometSQLConfs` sends booleans as `true` or `false` and sizes as a
31+
/// bare byte count. A config missing from its list arrives exactly as the user wrote it, and a
32+
/// value such as `10g` or `TRUE` then silently parses as the default.
2933
pub(crate) trait SparkConfig {
3034
fn get_bool(&self, name: &str) -> bool;
3135
fn get_u64(&self, name: &str, default_value: u64) -> u64;

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -727,9 +727,11 @@ object CometConf extends ShimCometConf {
727727
"shuffle data to disk. Larger values may improve write performance by reducing " +
728728
"the number of system calls, but will use more memory. " +
729729
"The default is 1MB which provides a good balance between performance and memory usage.")
730-
.bytesConf(ByteUnit.MiB)
731-
.checkValue(v => v > 0, "Write buffer size must be positive")
732-
.createWithDefault(1)
730+
.bytesConf(ByteUnit.BYTE)
731+
.checkValue(
732+
v => v > 0 && v <= Int.MaxValue,
733+
s"Write buffer size must be between 1 and ${Int.MaxValue} bytes")
734+
.createWithDefault(1024 * 1024)
733735

734736
val COMET_SHUFFLE_JVM_PREFER_DICTIONARY_RATIO: ConfigEntry[Double] = conf(
735737
"spark.comet.shuffle.jvm.preferDictionary.ratio")
@@ -1102,9 +1104,11 @@ object CometConf extends ShimCometConf {
11021104
.category(CATEGORY_TUNING)
11031105
.doc(
11041106
"The maximum amount of data (in bytes) stored inside the temporary directories " +
1105-
"used by native operators when spilling. Applied per Spark task, so an executor " +
1106-
"running N concurrent tasks may use up to N times this value on shared local disks. " +
1107-
"Once the limit is reached, further spills will fail and the query will error out.")
1107+
"used by native operators when spilling. Applied to each Comet native plan " +
1108+
"separately, and a Spark task can run more than one native plan at a time, so an " +
1109+
"executor running N concurrent tasks may use more than N times this value on shared " +
1110+
"local disks. Once the limit is reached, further spills will fail and the query will " +
1111+
"error out.")
11081112
.bytesConf(ByteUnit.BYTE)
11091113
.createWithDefault(100L * 1024 * 1024 * 1024) // 100 GB
11101114

‎spark/src/main/scala/org/apache/comet/CometExecIterator.scala‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -597,12 +597,20 @@ object CometExecIterator extends Logging {
597597
val executorCores = numDriverOrExecutorCores(SparkEnv.get.conf)
598598
builder.putEntries("spark.executor.cores", executorCores.toString)
599599

600-
// Any Comet config that the native side reads must be added here manually.
601-
// `cometSqlConfs` only carries values that were explicitly set, so defaults
602-
// from `createWithDefault(...)` would otherwise not cross JNI.
603-
builder.putEntries(
604-
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key,
605-
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.get(SQLConf.get).toString)
600+
// Any Comet config that the native side reads must be added here manually, resolved.
601+
// `cometSqlConfs` only carries values that were explicitly set, exactly as they were
602+
// written, so defaults from `createWithDefault(...)` would otherwise not cross JNI, and
603+
// native code, which parses only a bare number or a lowercase boolean, would silently fall
604+
// back to its own default for a value such as `10g` or `TRUE`.
605+
Seq[ConfigEntry[_]](
606+
CometConf.COMET_DEBUG_ENABLED,
607+
CometConf.COMET_DEBUG_MEMORY_ENABLED,
608+
CometConf.COMET_EXPLAIN_NATIVE_ENABLED,
609+
CometConf.COMET_MAX_TEMP_DIRECTORY_SIZE,
610+
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED,
611+
CometConf.COMET_TRACING_ENABLED).foreach { entry =>
612+
builder.putEntries(entry.key, entry.get(SQLConf.get).toString)
613+
}
606614

607615
builder.build().toByteArray
608616
}
@@ -614,7 +622,7 @@ object CometExecIterator extends Logging {
614622
val offHeapMode = CometSparkSessionExtensions.isOffHeapEnabled(conf)
615623
if (offHeapMode) {
616624
// in off-heap mode, Comet uses unified memory management to share off-heap memory with Spark
617-
val offHeapSize = ByteUnit.MiB.toBytes(conf.getSizeAsMb("spark.memory.offHeap.size"))
625+
val offHeapSize = conf.getSizeAsBytes("spark.memory.offHeap.size")
618626
val memoryFraction = CometConf.COMET_OFFHEAP_MEMORY_POOL_FRACTION.get()
619627
val memoryLimit = (offHeapSize * memoryFraction).toLong
620628
val memoryLimitPerTask = (memoryLimit.toDouble * coresPerTask / numCores).toLong

‎spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,10 +76,6 @@ class CometExecSuite extends CometTestBase {
7676
ConfigMap.parseFrom(protobuf)
7777
}
7878

79-
// test not setting the config
80-
val deserialized: ConfigMap = roundtrip
81-
assert(null == deserialized.getEntriesMap.get(CometConf.COMET_EXPLAIN_NATIVE_ENABLED.key))
82-
8379
// test explicitly setting the config
8480
for (value <- Seq("true", "false")) {
8581
withSQLConf(CometConf.COMET_EXPLAIN_NATIVE_ENABLED.key -> value) {
@@ -90,6 +86,29 @@ class CometExecSuite extends CometTestBase {
9086
}
9187
}
9288

89+
test("SQLConf serde resolves the configs that native code parses") {
90+
def entries = ConfigMap.parseFrom(CometExecIterator.serializeCometSQLConfs()).getEntriesMap
91+
val flags = Seq(
92+
CometConf.COMET_DEBUG_ENABLED,
93+
CometConf.COMET_DEBUG_MEMORY_ENABLED,
94+
CometConf.COMET_EXPLAIN_NATIVE_ENABLED,
95+
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED,
96+
CometConf.COMET_TRACING_ENABLED)
97+
98+
// Native code parses only a bare byte count or a lowercase boolean and silently falls back
99+
// to its own default otherwise, so these cross JNI resolved, defaults included.
100+
val defaults = entries
101+
assert(defaults.get(CometConf.COMET_MAX_TEMP_DIRECTORY_SIZE.key) == "107374182400")
102+
flags.foreach(flag => assert(defaults.get(flag.key) == "false", flag.key))
103+
104+
withSQLConf(
105+
(CometConf.COMET_MAX_TEMP_DIRECTORY_SIZE.key -> "10g") +: flags.map(_.key -> "TRUE"): _*) {
106+
val resolved = entries
107+
assert(resolved.get(CometConf.COMET_MAX_TEMP_DIRECTORY_SIZE.key) == "10737418240")
108+
flags.foreach(flag => assert(resolved.get(flag.key) == "true", flag.key))
109+
}
110+
}
111+
93112
test("sample without replacement") {
94113
withParquetTable((0 until 1000).map(i => (i, i + 1)), "tbl") {
95114
val df = sql("SELECT * FROM tbl").sample(withReplacement = false, fraction = 0.3, seed = 42)

‎spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,20 @@ class CometExecIteratorLifecycleSuite extends CometTestBase {
289289
assert(nativeMemoryLimitWarning(Array(100 * mib, reserved, 1L, 1L), reserved, limit).isEmpty)
290290
}
291291

292+
test("the memory pool limit reads a bare off-heap size as bytes, as Spark does") {
293+
import CometExecIterator.getMemoryConfig
294+
val fourGiB = 4L * 1024 * 1024 * 1024
295+
val offHeap = new SparkConf(false)
296+
.set("spark.master", "local[4]")
297+
.set("spark.memory.offHeap.enabled", "true")
298+
assert(
299+
getMemoryConfig(offHeap.clone.set("spark.memory.offHeap.size", "4294967296")).memoryLimit
300+
== fourGiB)
301+
assert(
302+
getMemoryConfig(offHeap.clone.set("spark.memory.offHeap.size", "4g")).memoryLimit
303+
== fourGiB)
304+
}
305+
292306
test("the native memory limit is the off-heap size plus the memory overhead") {
293307
import CometExecIterator.nativeMemoryLimit
294308
val mib = 1024L * 1024
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.spark.sql.comet.execution.shuffle
21+
22+
import org.apache.spark.TaskContext
23+
import org.apache.spark.sql.CometTestBase
24+
import org.apache.spark.sql.vectorized.ColumnarBatch
25+
26+
import org.apache.comet.CometConf
27+
28+
/**
29+
* Checks the plan [[CometNativeShuffleWriter]] hands to native code. Lives in the
30+
* `execution.shuffle` package so it can call the `private[shuffle]` `buildUnifiedPlan`.
31+
*/
32+
class CometNativeShuffleWriterSuite extends CometTestBase {
33+
34+
import testImplicits._
35+
36+
test("the write buffer size reaches native code in bytes") {
37+
withSQLConf(
38+
CometConf.COMET_SHUFFLE_MODE.key -> "native",
39+
"spark.sql.adaptive.enabled" -> "false") {
40+
withParquetTable((0 until 10).map(i => (i, s"row-$i")), "tbl") {
41+
val exchange = sql("SELECT * FROM tbl")
42+
.repartition(3, $"_1")
43+
.queryExecution
44+
.executedPlan
45+
.collectFirst { case value: CometShuffleExchangeExec => value }
46+
.getOrElse(fail("Expected a native Comet shuffle exchange"))
47+
val dependency = exchange.shuffleDependency
48+
.asInstanceOf[CometShuffleDependency[Int, ColumnarBatch, ColumnarBatch]]
49+
val context: TaskContext = TaskContext.empty()
50+
51+
// Native code sizes the writers for the data and spill files from this field, in bytes.
52+
def writeBufferSize: Int =
53+
new CometNativeShuffleWriter[Int, ColumnarBatch](
54+
dependency.nativeShuffleSpec.get,
55+
dependency.outputPartitioning.get,
56+
dependency.outputAttributes,
57+
dependency.shuffleWriteMetrics,
58+
dependency.numParts,
59+
dependency.shuffleId,
60+
context.taskAttemptId(),
61+
context,
62+
context.taskMetrics().shuffleWriteMetrics,
63+
dependency.rangePartitionBounds)
64+
.buildUnifiedPlan("unused.data")
65+
.getShuffleWriter
66+
.getWriteBufferSize
67+
68+
assert(writeBufferSize == 1024 * 1024)
69+
withSQLConf(CometConf.COMET_SHUFFLE_NATIVE_WRITE_BUFFER_SIZE.key -> "8m") {
70+
assert(writeBufferSize == 8 * 1024 * 1024)
71+
}
72+
// A bare number is a byte count.
73+
withSQLConf(CometConf.COMET_SHUFFLE_NATIVE_WRITE_BUFFER_SIZE.key -> "65536") {
74+
assert(writeBufferSize == 65536)
75+
}
76+
}
77+
}
78+
}
79+
}

0 commit comments

Comments
 (0)