Skip to content

Commit 975b4e5

Browse files
committed
[SPARK-60984][CORE][TEST] Fix HighlyCompressedMapStatusBenchmark failure when mockito is not on the classpath
### What changes were proposed in this pull request? This PR removes the Mockito dependency from `HighlyCompressedMapStatusBenchmark`. - Instead of mocking `SparkEnv` for each case, the benchmark now starts a single `local[1]` `SparkContext` and changes `spark.shuffle.accurateBlockSkewedFactor` on `SparkEnv.get.conf` before each case. `HighlyCompressedMapStatus` reads this value from `SparkEnv.get.conf` on every call, so the behavior is the same as the mock. - The `SparkContext` is stopped before the "Driver heap retained by MapOutputTracker" report, which creates its own `SparkContext`. ### Why are the changes needed? The benchmark job fails with: ``` java.lang.NoClassDefFoundError: org/mockito/Mockito at org.apache.spark.HighlyCompressedMapStatusBenchmark$.envWithSkewedFactor(HighlyCompressedMapStatusBenchmark.scala:52) ``` (https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686502284) The GitHub Actions benchmark workflow runs `Benchmarks` through `spark-submit --jars <test jars>`. `mockito-core` is a test-scope dependency of `core`, so it is not on that classpath. `build/sbt "core/Test/runMain ..."` does include test dependencies, which is why the failure does not reproduce locally. The benchmark was added by SPARK-48290 (a125337, 2026-10-19). Other benchmarks do not depend on Mockito. - #57528 ### Does this PR introduce _any_ user-facing change? No. This only changes a benchmark. ### How was this patch tested? Ran the benchmark locally and confirmed that it completes successfully and that the reported numbers (for example, the bytes retained per map task) are the same as before the change: ``` build/sbt "core/Test/runMain org.apache.spark.HighlyCompressedMapStatusBenchmark" ``` The benchmark has not been run through `spark-submit --jars` yet, so the GitHub Actions benchmark job needs to confirm the fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5.5 Closes #60160 from dongjoon-hyun/SPARK-60984. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 7af0d94) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 3b25e35 commit 975b4e5

1 file changed

Lines changed: 35 additions & 33 deletions

File tree

‎core/src/test/scala/org/apache/spark/HighlyCompressedMapStatusBenchmark.scala‎

Lines changed: 35 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ package org.apache.spark
2020
import java.io.PrintStream
2121

2222
import org.apache.commons.io.output.TeeOutputStream
23-
import org.mockito.Mockito.{doReturn, mock}
2423

2524
import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
2625
import org.apache.spark.internal.config
@@ -46,12 +45,15 @@ object HighlyCompressedMapStatusBenchmark extends BenchmarkBase {
4645

4746
private val loc = BlockManagerId("a", "host", 1000)
4847

49-
private def envWithSkewedFactor(skewedFactor: Double): SparkEnv = {
50-
val conf = new SparkConf()
51-
.set(config.SHUFFLE_ACCURATE_BLOCK_SKEWED_FACTOR.key, skewedFactor.toString)
52-
val env = mock(classOf[SparkEnv])
53-
doReturn(conf, Seq.empty: _*).when(env).conf
54-
env
48+
// HighlyCompressedMapStatus reads the skewed factor from SparkEnv.get.conf on every call.
49+
private def setSkewedFactor(skewedFactor: Double): Unit = {
50+
SparkEnv.get.conf.set(config.SHUFFLE_ACCURATE_BLOCK_SKEWED_FACTOR.key, skewedFactor.toString)
51+
}
52+
53+
private def withLocalSparkContext(f: => Unit): Unit = {
54+
val sc = new SparkContext(
55+
new SparkConf().setMaster("local[1]").setAppName("HighlyCompressedMapStatusBenchmark"))
56+
try f finally sc.stop()
5557
}
5658

5759
private def skewedBlockSizes(numPartitions: Int): Array[Long] = {
@@ -88,10 +90,10 @@ object HighlyCompressedMapStatusBenchmark extends BenchmarkBase {
8890
* adaptive query execution plans on, so the lookup cost is paid numPartitions * numMapTasks
8991
* times per shuffle.
9092
*/
91-
private def runLookupBenchmark(envWithSkewedSizes: SparkEnv): Unit = {
93+
private def runLookupBenchmark(): Unit = {
9294
val numMapTasks = 100
9395
Seq(2048, 50000).foreach { numPartitions =>
94-
SparkEnv.set(envWithSkewedSizes)
96+
setSkewedFactor(5.0)
9597
val sizes = tiedSkewedBlockSizes(numPartitions)
9698
val statuses =
9799
Array.tabulate(numMapTasks)(id => HighlyCompressedMapStatus(loc, sizes, id.toLong))
@@ -182,33 +184,33 @@ object HighlyCompressedMapStatusBenchmark extends BenchmarkBase {
182184
}
183185

184186
override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
185-
val envWithoutSkewedSizes = envWithSkewedFactor(-1.0)
186-
val envWithSkewedSizes = envWithSkewedFactor(5.0)
187-
Seq("skewed" -> skewedBlockSizes _, "organ pipe" -> organPipeBlockSizes _).foreach {
188-
case (distribution, blockSizes) =>
189-
runBenchmark(s"Build HighlyCompressedMapStatus, $distribution block sizes") {
190-
Seq(2048, 10000, 50000).foreach { numPartitions =>
191-
val benchmark = new Benchmark(s"$numPartitions shuffle partitions", 1, output = output)
192-
val sizes = blockSizes(numPartitions)
193-
194-
benchmark.addCase("skewed block sizes not recorded", 10) { _ =>
195-
SparkEnv.set(envWithoutSkewedSizes)
196-
HighlyCompressedMapStatus(loc, sizes, 0)
197-
}
198-
199-
benchmark.addCase("skewed block sizes recorded accurately", 10) { _ =>
200-
SparkEnv.set(envWithSkewedSizes)
201-
HighlyCompressedMapStatus(loc, sizes, 0)
187+
withLocalSparkContext {
188+
Seq("skewed" -> skewedBlockSizes _, "organ pipe" -> organPipeBlockSizes _).foreach {
189+
case (distribution, blockSizes) =>
190+
runBenchmark(s"Build HighlyCompressedMapStatus, $distribution block sizes") {
191+
Seq(2048, 10000, 50000).foreach { numPartitions =>
192+
val benchmark =
193+
new Benchmark(s"$numPartitions shuffle partitions", 1, output = output)
194+
val sizes = blockSizes(numPartitions)
195+
196+
benchmark.addCase("skewed block sizes not recorded", 10) { _ =>
197+
setSkewedFactor(-1.0)
198+
HighlyCompressedMapStatus(loc, sizes, 0)
199+
}
200+
201+
benchmark.addCase("skewed block sizes recorded accurately", 10) { _ =>
202+
setSkewedFactor(5.0)
203+
HighlyCompressedMapStatus(loc, sizes, 0)
204+
}
205+
206+
benchmark.run()
202207
}
203-
204-
benchmark.run()
205208
}
206-
}
207-
}
208-
runBenchmark("Read block sizes out of HighlyCompressedMapStatus") {
209-
runLookupBenchmark(envWithSkewedSizes)
209+
}
210+
runBenchmark("Read block sizes out of HighlyCompressedMapStatus") {
211+
runLookupBenchmark()
212+
}
210213
}
211-
SparkEnv.set(null)
212214
runBenchmark("Driver heap retained by MapOutputTracker") {
213215
runDriverHeapReport()
214216
}

0 commit comments

Comments
 (0)