Repository navigation
Commit f2f0c70
committed
fix: keep the Iceberg benchmark's row collection compatible with Spark 3.x
Spark 3.4 and 3.5 declare `SQLHelper.withSQLConf` as
`(pairs: (String, String)*)(f: => Unit): Unit`; only Spark 4 has the
generic result-returning form. Returning the wrapper from a method typed
`Array[Row]` therefore failed to compile on both 3.x profiles and on the
two Celeborn jobs, before any test could run. Collect into a local inside
the configuration scope and return it after the block.
Also drop the claim that the two arms scan identically. Enabling Comet
replaces the Parquet scan and the projection with native operators as
well as the transform, so each ratio is a query-level scan-plus-projection
result, not a measurement of the transform on its own; the criterion
benchmark in native/spark-expr/benches is what isolates the kernel.1 parent 29f69db commit f2f0c70
1 file changed
Lines changed: 15 additions & 4 deletions
File tree
- spark/src/test/scala/org/apache/spark/sql/benchmark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
39 | 44 | | |
40 | 45 | | |
41 | 46 | | |
| |||
91 | 96 | | |
92 | 97 | | |
93 | 98 | | |
94 | | - | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
95 | 104 | | |
96 | 105 | | |
97 | 106 | | |
98 | 107 | | |
99 | | - | |
| 108 | + | |
100 | 109 | | |
| 110 | + | |
| 111 | + | |
101 | 112 | | |
102 | 113 | | |
103 | 114 | | |
| |||
128 | 139 | | |
129 | 140 | | |
130 | 141 | | |
131 | | - | |
132 | | - | |
| 142 | + | |
| 143 | + | |
133 | 144 | | |
134 | 145 | | |
135 | 146 | | |
| |||
0 commit comments