Repository navigation
Commit 975b4e5
committed
[SPARK-60984][CORE][TEST] Fix
### What changes were proposed in this pull request?
This PR removes the Mockito dependency from `HighlyCompressedMapStatusBenchmark`.
- Instead of mocking `SparkEnv` for each case, the benchmark now starts a single `local[1]` `SparkContext` and changes `spark.shuffle.accurateBlockSkewedFactor` on `SparkEnv.get.conf` before each case. `HighlyCompressedMapStatus` reads this value from `SparkEnv.get.conf` on every call, so the behavior is the same as the mock.
- The `SparkContext` is stopped before the "Driver heap retained by MapOutputTracker" report, which creates its own `SparkContext`.
### Why are the changes needed?
The benchmark job fails with:
```
java.lang.NoClassDefFoundError: org/mockito/Mockito
at org.apache.spark.HighlyCompressedMapStatusBenchmark$.envWithSkewedFactor(HighlyCompressedMapStatusBenchmark.scala:52)
```
(https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686502284)
The GitHub Actions benchmark workflow runs `Benchmarks` through `spark-submit --jars <test jars>`. `mockito-core` is a test-scope dependency of `core`, so it is not on that classpath. `build/sbt "core/Test/runMain ..."` does include test dependencies, which is why the failure does not reproduce locally.
The benchmark was added by SPARK-48290 (a125337, 2026-10-19). Other benchmarks do not depend on Mockito.
- #57528
### Does this PR introduce _any_ user-facing change?
No. This only changes a benchmark.
### How was this patch tested?
Ran the benchmark locally and confirmed that it completes successfully and that the reported numbers (for example, the bytes retained per map task) are the same as before the change:
```
build/sbt "core/Test/runMain org.apache.spark.HighlyCompressedMapStatusBenchmark"
```
The benchmark has not been run through `spark-submit --jars` yet, so the GitHub Actions benchmark job needs to confirm the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #60160 from dongjoon-hyun/SPARK-60984.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 7af0d94)
Signed-off-by: Dongjoon Hyun <[email protected]>HighlyCompressedMapStatusBenchmark failure when mockito is not on the classpath1 parent 3b25e35 commit 975b4e5
1 file changed
Lines changed: 35 additions & 33 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
24 | 23 | | |
25 | 24 | | |
26 | 25 | | |
| |||
46 | 45 | | |
47 | 46 | | |
48 | 47 | | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| |||
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
91 | | - | |
| 93 | + | |
92 | 94 | | |
93 | 95 | | |
94 | | - | |
| 96 | + | |
95 | 97 | | |
96 | 98 | | |
97 | 99 | | |
| |||
182 | 184 | | |
183 | 185 | | |
184 | 186 | | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
202 | 207 | | |
203 | | - | |
204 | | - | |
205 | 208 | | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
210 | 213 | | |
211 | | - | |
212 | 214 | | |
213 | 215 | | |
214 | 216 | | |
| |||
0 commit comments