[SPARK-60987][INFRA] Fix NoClassDefFoundError when running Kafka benchmarks in the benchmark workflow - #60161

What changes were proposed in this pull request?

This PR adds the Kafka-related jars to the --jars option of spark-submit in the Run benchmarks step of .github/workflows/benchmark.yml:

• spark-*kafka-0-10*-SNAPSHOT.jar (the main jars of spark-sql-kafka-0-10 and spark-token-provider-kafka-0-10, built by Test/package)
• From the coursier cache (the same way as curator-test is handled): kafka*.jar (kafka-clients, kafka_2.13, kafka-server, kafka-metadata, kafka-raft, kafka-storage, ...), metrics-core-2.2.0.jar, jopt-simple-*, scala-java8-compat_*, scala-logging_*, jackson-dataformat-csv-*, and commons-pool2-*. -sources and -javadoc jars are excluded.

The list was derived from sql-kafka-0-10/Test/dependencyClasspath, keeping only the jars that are not part of the Spark distribution.

Why are the changes needed?

RTMKafkaKafkaBenchmark and the other RTMKafka*Benchmark classes cannot run in the benchmark workflow. Their classes are discovered through *-SNAPSHOT-tests.jar, but kafka-clients and the other Kafka dependencies are neither in --jars nor in the Spark distribution, so the JVM fails while loading the class:

Running org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Callback
	at org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark.main(RTMKafkaKafkaBenchmark.scala)

Because SPARK_BENCHMARK_FAILFAST defaults to true, this fails the whole split. For example, the "Run benchmarks: * (JDK 21, Scala 2.13, 1 out of 64 splits)" job fails this way:

• https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686555631
• https://github.com/dongjoon-hyun/spark/actions/runs/41684094207/job/127686511345

Does this PR introduce any user-facing change?

No. This only changes the GitHub Actions benchmark workflow.

How was this patch tested?

Manually verified that each find pattern matches the required jars (e.g., kafka-clients-3.9.2.jar, kafka_2.13-3.9.2.jar, spark-sql-kafka-0-10_2.13-5.0.0-SNAPSHOT.jar) against the local build output and Maven repository.

The benchmark itself needs to be verified with the Run benchmarks workflow in GitHub Actions on org.apache.spark.sql.kafka010.benchmark.RTMKafkaKafkaBenchmark.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Sonnet 5.5