Repository navigation
[SPARK-60984][CORE][TESTS] Fix HighlyCompressedMapStatusBenchmark failure when mockito is not on the classpath - #60160
Closed
dongjoon-hyun wants to merge 1 commit into
Closed
dongjoon-hyun wants to merge 1 commit into
dongjoon-hyun wants to merge 1 commit into
Conversation
…ailure when `mockito` is not on the classpath
Member
Author
|
cc @Vivek1106-04 , @sunchao , @uros-b |
Member
Author
|
Could you review this PR to unblock Java 21 migration, @HyukjinKwon ? |
HyukjinKwon
added
the
ai-review-requested
HyukjinKwon
approved these changes
Nov 6, 2026
Member
Author
|
Thank you, @HyukjinKwon ~ |
dongjoon-hyun
added a commit
that referenced
this pull request
Nov 6, 2026
…ilure when `mockito` is not on the classpath ### What changes were proposed in this pull request? This PR removes the Mockito dependency from `HighlyCompressedMapStatusBenchmark`. - Instead of mocking `SparkEnv` for each case, the benchmark now starts a single `local[1]` `SparkContext` and changes `spark.shuffle.accurateBlockSkewedFactor` on `SparkEnv.get.conf` before each case. `HighlyCompressedMapStatus` reads this value from `SparkEnv.get.conf` on every call, so the behavior is the same as the mock. - The `SparkContext` is stopped before the "Driver heap retained by MapOutputTracker" report, which creates its own `SparkContext`. ### Why are the changes needed? The benchmark job fails with: ``` java.lang.NoClassDefFoundError: org/mockito/Mockito at org.apache.spark.HighlyCompressedMapStatusBenchmark$.envWithSkewedFactor(HighlyCompressedMapStatusBenchmark.scala:52) ``` (https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686502284) The GitHub Actions benchmark workflow runs `Benchmarks` through `spark-submit --jars <test jars>`. `mockito-core` is a test-scope dependency of `core`, so it is not on that classpath. `build/sbt "core/Test/runMain ..."` does include test dependencies, which is why the failure does not reproduce locally. The benchmark was added by SPARK-48290 (a125337, 2026-10-19). Other benchmarks do not depend on Mockito. - #57528 ### Does this PR introduce _any_ user-facing change? No. This only changes a benchmark. ### How was this patch tested? Ran the benchmark locally and confirmed that it completes successfully and that the reported numbers (for example, the bytes retained per map task) are the same as before the change: ``` build/sbt "core/Test/runMain org.apache.spark.HighlyCompressedMapStatusBenchmark" ``` The benchmark has not been run through `spark-submit --jars` yet, so the GitHub Actions benchmark job needs to confirm the fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5.5 Closes #60160 from dongjoon-hyun/SPARK-60984. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 7af0d94) Signed-off-by: Dongjoon Hyun <[email protected]>
Member
Author
HyukjinKwon
removed
the
ai-review-requested
uros-b
reviewed
Nov 9, 2026
Member
There was a problem hiding this comment.
Late LGTM, thank you for following up on this @dongjoon-hyun!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR removes the Mockito dependency from
HighlyCompressedMapStatusBenchmark.SparkEnvfor each case, the benchmark now starts a singlelocal[1]SparkContextand changesspark.shuffle.accurateBlockSkewedFactoronSparkEnv.get.confbefore each case.HighlyCompressedMapStatusreads this value fromSparkEnv.get.confon every call, so the behavior is the same as the mock.SparkContextis stopped before the "Driver heap retained by MapOutputTracker" report, which creates its ownSparkContext.Why are the changes needed?
The benchmark job fails with:
(https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686502284)
The GitHub Actions benchmark workflow runs
Benchmarksthroughspark-submit --jars <test jars>.mockito-coreis a test-scope dependency ofcore, so it is not on that classpath.build/sbt "core/Test/runMain ..."does include test dependencies, which is why the failure does not reproduce locally.The benchmark was added by SPARK-48290 (a125337, 2026-10-19). Other benchmarks do not depend on Mockito.
Does this PR introduce any user-facing change?
No. This only changes a benchmark.
How was this patch tested?
Ran the benchmark locally and confirmed that it completes successfully and that the reported numbers (for example, the bytes retained per map task) are the same as before the change:
The benchmark has not been run through
spark-submit --jarsyet, so the GitHub Actions benchmark job needs to confirm the fix.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5