Repository navigation
Commit f610c51
committed
[SPARK-60987][INFRA] Fix
### What changes were proposed in this pull request?
This PR adds the Kafka-related jars to the `--jars` option of `spark-submit` in the `Run benchmarks` step of `.github/workflows/benchmark.yml`:
- `spark-*kafka-0-10*-SNAPSHOT.jar` (the main jars of `spark-sql-kafka-0-10` and `spark-token-provider-kafka-0-10`, built by `Test/package`)
- From the coursier cache (the same way as `curator-test` is handled): `kafka*.jar` (`kafka-clients`, `kafka_2.13`, `kafka-server`, `kafka-metadata`, `kafka-raft`, `kafka-storage`, ...), `metrics-core-2.2.0.jar`, `jopt-simple-*`, `scala-java8-compat_*`, `scala-logging_*`, `jackson-dataformat-csv-*`, and `commons-pool2-*`. `-sources` and `-javadoc` jars are excluded.
The list was derived from `sql-kafka-0-10/Test/dependencyClasspath`, keeping only the jars that are not part of the Spark distribution.
### Why are the changes needed?
`RTMKafkaKafkaBenchmark` and the other `RTMKafka*Benchmark` classes cannot run in the benchmark workflow. Their classes are discovered through `*-SNAPSHOT-tests.jar`, but `kafka-clients` and the other Kafka dependencies are neither in `--jars` nor in the Spark distribution, so the JVM fails while loading the class:
```
Running org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Callback
at org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark.main(RTMKafkaKafkaBenchmark.scala)
```
Because `SPARK_BENCHMARK_FAILFAST` defaults to `true`, this fails the whole split. For example, the "Run benchmarks: * (JDK 21, Scala 2.13, 1 out of 64 splits)" job fails this way:
- https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686555631
- https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686511345
### Does this PR introduce _any_ user-facing change?
No. This only changes the GitHub Actions benchmark workflow.
### How was this patch tested?
Manually verified that each `find` pattern matches the required jars (e.g., `kafka-clients-3.9.2.jar`, `kafka_2.13-3.9.2.jar`, `spark-sql-kafka-0-10_2.13-5.0.0-SNAPSHOT.jar`) against the local build output and Maven repository.
The benchmark itself needs to be verified with the `Run benchmarks` workflow in GitHub Actions on `org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #60161 from dongjoon-hyun/SPARK-60987.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>NoClassDefFoundError when running Kafka benchmarks in the benchmark workflow1 parent 7af0d94 commit f610c51
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
196 | | - | |
| 196 | + | |
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
| |||
0 commit comments