Skip to content

Commit f610c51

Browse files
committed
[SPARK-60987][INFRA] Fix NoClassDefFoundError when running Kafka benchmarks in the benchmark workflow
### What changes were proposed in this pull request? This PR adds the Kafka-related jars to the `--jars` option of `spark-submit` in the `Run benchmarks` step of `.github/workflows/benchmark.yml`: - `spark-*kafka-0-10*-SNAPSHOT.jar` (the main jars of `spark-sql-kafka-0-10` and `spark-token-provider-kafka-0-10`, built by `Test/package`) - From the coursier cache (the same way as `curator-test` is handled): `kafka*.jar` (`kafka-clients`, `kafka_2.13`, `kafka-server`, `kafka-metadata`, `kafka-raft`, `kafka-storage`, ...), `metrics-core-2.2.0.jar`, `jopt-simple-*`, `scala-java8-compat_*`, `scala-logging_*`, `jackson-dataformat-csv-*`, and `commons-pool2-*`. `-sources` and `-javadoc` jars are excluded. The list was derived from `sql-kafka-0-10/Test/dependencyClasspath`, keeping only the jars that are not part of the Spark distribution. ### Why are the changes needed? `RTMKafkaKafkaBenchmark` and the other `RTMKafka*Benchmark` classes cannot run in the benchmark workflow. Their classes are discovered through `*-SNAPSHOT-tests.jar`, but `kafka-clients` and the other Kafka dependencies are neither in `--jars` nor in the Spark distribution, so the JVM fails while loading the class: ``` Running org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Callback at org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark.main(RTMKafkaKafkaBenchmark.scala) ``` Because `SPARK_BENCHMARK_FAILFAST` defaults to `true`, this fails the whole split. For example, the "Run benchmarks: * (JDK 21, Scala 2.13, 1 out of 64 splits)" job fails this way: - https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686555631 - https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686511345 ### Does this PR introduce _any_ user-facing change? No. This only changes the GitHub Actions benchmark workflow. ### How was this patch tested? Manually verified that each `find` pattern matches the required jars (e.g., `kafka-clients-3.9.2.jar`, `kafka_2.13-3.9.2.jar`, `spark-sql-kafka-0-10_2.13-5.0.0-SNAPSHOT.jar`) against the local build output and Maven repository. The benchmark itself needs to be verified with the `Run benchmarks` workflow in GitHub Actions on `org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5.5 Closes #60161 from dongjoon-hyun/SPARK-60987. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 7af0d94 commit f610c51

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎.github/workflows/benchmark.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ jobs:
193193
# In benchmark, we use local as master so set driver memory only. Note that GitHub Actions has 7 GB memory limit.
194194
bin/spark-submit \
195195
--driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
196-
--jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`,`find ~/.cache/coursier -name 'curator-test-*.jar'`" \
196+
--jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' -o -name 'spark-*kafka-0-10*-SNAPSHOT.jar' | paste -sd ',' -`,`find ~/.cache/coursier -name 'curator-test-*.jar'`,`find ~/.cache/coursier \( -name 'kafka*.jar' -o -name 'metrics-core-2.2.0.jar' -o -name 'jopt-simple-*.jar' -o -name 'scala-java8-compat_*.jar' -o -name 'scala-logging_*.jar' -o -name 'jackson-dataformat-csv-*.jar' -o -name 'commons-pool2-*.jar' \) ! -name '*-sources.jar' ! -name '*-javadoc.jar' | paste -sd ',' -`" \
197197
"`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
198198
"${{ inputs.class }}"
199199
# To keep the directory structure and file permissions, tar them

0 commit comments

Comments
 (0)