Skip to content

Commit f2f0c70

Browse files
committed
fix: keep the Iceberg benchmark's row collection compatible with Spark 3.x
Spark 3.4 and 3.5 declare `SQLHelper.withSQLConf` as `(pairs: (String, String)*)(f: => Unit): Unit`; only Spark 4 has the generic result-returning form. Returning the wrapper from a method typed `Array[Row]` therefore failed to compile on both 3.x profiles and on the two Celeborn jobs, before any test could run. Collect into a local inside the configuration scope and return it after the block. Also drop the claim that the two arms scan identically. Enabling Comet replaces the Parquet scan and the projection with native operators as well as the transform, so each ratio is a query-level scan-plus-projection result, not a measurement of the transform on its own; the criterion benchmark in native/spark-expr/benches is what isolates the kernel.
1 parent 29f69db commit f2f0c70

1 file changed

Lines changed: 15 additions & 4 deletions

File tree

‎spark/src/test/scala/org/apache/spark/sql/benchmark/CometIcebergSystemFunctionBenchmark.scala‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,11 @@ import org.apache.comet.iceberg.IcebergReflection
3636
* output is compared between the two engines over the same corpus that is then timed, so a timing
3737
* cannot come from an engine that computed something else.
3838
*
39+
* The two cases differ in more than the transform: enabling Comet also replaces the Parquet scan
40+
* and the projection with native operators. Each ratio here is therefore a query-level
41+
* scan-plus-projection result and does not isolate the cost of the transform.
42+
* `native/spark-expr/benches/iceberg_transforms.rs` is the kernel-level measurement.
43+
*
3944
* To run this benchmark:
4045
* {{{
4146
* SPARK_GENERATE_BENCHMARK_FILES=1 make benchmark-org.apache.spark.sql.benchmark.CometIcebergSystemFunctionBenchmark
@@ -91,13 +96,19 @@ object CometIcebergSystemFunctionBenchmark extends CometBenchmarkBase {
9196
* order is the same. The confs match the ones the benchmark times.
9297
*/
9398
private def verifyOutputsMatch(name: String, query: String): Unit = {
94-
def collect(cometEnabled: Boolean): Array[Row] =
99+
// The rows are assigned to a local rather than returned from the `withSQLConf` block, because
100+
// Spark 3.4 and 3.5 declare `SQLHelper.withSQLConf` as returning `Unit`; only Spark 4 has the
101+
// generic result-returning form.
102+
def collect(cometEnabled: Boolean): Array[Row] = {
103+
var rows: Array[Row] = Array.empty
95104
withSQLConf(
96105
SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> excludedRulesWith(ConstantFolding.ruleName),
97106
CometConf.COMET_ENABLED.key -> cometEnabled.toString,
98107
CometConf.COMET_EXEC_ENABLED.key -> cometEnabled.toString) {
99-
spark.sql(query).collect()
108+
rows = spark.sql(query).collect()
100109
}
110+
rows
111+
}
101112

102113
// `Row.equals` compares binary columns by reference, so normalize before comparing.
103114
def comparable(row: Row): String =
@@ -128,8 +139,8 @@ object CometIcebergSystemFunctionBenchmark extends CometBenchmarkBase {
128139
return
129140
}
130141
// The Iceberg system functions are resolved through a v2 catalog, so one has to be
131-
// registered. No Iceberg table is read: the data stays in Parquet so both cases scan
132-
// identically and the only difference is who evaluates the transform.
142+
// registered. No Iceberg table is read: the data stays in Parquet, which keeps the Iceberg
143+
// reader out of the Spark case and lets both cases read the same files.
133144
withTempPath { warehouse =>
134145
spark.conf.set(s"spark.sql.catalog.$catalog", "org.apache.iceberg.spark.SparkCatalog")
135146
spark.conf.set(s"spark.sql.catalog.$catalog.type", "hadoop")

0 commit comments

Comments
 (0)