Skip to content

Commit 20dfd08

Browse files
committed
review: benchmark the cache codec, test an unknown codec, fix non-clean shading
The codec table in the docs had zstd beating `none` on write and read speed as well as footprint, and it did not come from a committed benchmark. CometInMemoryCacheBenchmark now has a codec axis over the flat relation -- materialize time, footprint, and narrow and full reads -- and with each case warmed up, `none` is faster at everything but footprint: 1091 ms against 1503 ms to materialize and 63 ms against 296 ms for a full read, at 315 MiB against 55 MiB. zstd stays the default, on footprint grounds, and the docs and the compression.codec doc string now say so. Every table on the docs page is regenerated from one run. The unknown body-compression byte branch in readCodec had no test. A helper now assembles a payload whose body records a byte no Arrow codec owns, and a test asserts the read fails naming it. Any build without `clean` in which no class had changed failed in shade with a duplicate META-INF/services entry: the jar plugin leaves the previously shaded jar in place, and the services transformer then writes the relocated codec registration twice. Exclude the already relocated copy on the way in. Also fixes a collation-test comment that no longer described its assertion.
1 parent 14bf845 commit 20dfd08

6 files changed

Lines changed: 257 additions & 37 deletions

File tree

‎docs/source/user-guide/latest/in-memory-cache.md‎

Lines changed: 28 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -70,19 +70,25 @@ column out of six does roughly a sixth of the decompression work, and a `SELECT
7070
selects no columns at all, answers from the row count stored beside the payload without touching
7171
it.
7272

73-
Compression defaults to `zstd`, which is faster than storing cached batches uncompressed: the
74-
bytes it saves cost more to copy and store than compressing them costs. Measured over a 200k-row,
75-
six-column relation:
73+
Compression defaults to `zstd`, for footprint rather than for speed. Over the same 5M-row,
74+
six-column relation the tables under [Performance](#performance) use — and measured by the same
75+
benchmark — it holds the data in a sixth of the memory and pays for that on both sides: about 40%
76+
longer to materialize, and, on a read wide enough to inflate everything, close to five times
77+
longer. A narrow projection pays far less, because it only inflates the columns it asked for.
7678

7779
| Codec | Materialize | Footprint | Read 1 of 6 | Read 6 of 6 |
7880
| ------ | ----------: | --------: | ----------: | ----------: |
79-
| `zstd` | 363 ms | 2 MiB | 56 ms | 62 ms |
80-
| `none` | 1776 ms | 13 MiB | 78 ms | 81 ms |
81+
| `zstd` | 1503 ms | 55 MiB | 47 ms | 296 ms |
82+
| `none` | 1091 ms | 315 MiB | 36 ms | 63 ms |
8183

82-
Arrow's other IPC codec, LZ4, is deliberately not offered. It is commons-compress's pure-Java
83-
implementation and is unrelated to the JNI-accelerated lz4-java behind `spark.io.compression.codec`;
84-
it measured three orders of magnitude slower to write than `zstd` while also producing larger
85-
output, so no workload prefers it.
84+
`none` is the better setting for a relation that fits in memory uncompressed and is read at close
85+
to full width. The default is the other way round because a cache that does not fit costs more than
86+
one that is slower to read, and Spark's own cache format compresses by default too.
87+
88+
Arrow's other IPC codec, LZ4, is deliberately not offered and the config rejects it. It is
89+
commons-compress's pure-Java implementation, unrelated to the JNI-accelerated lz4-java behind
90+
`spark.io.compression.codec`, and is orders of magnitude slower to write than `zstd` while also
91+
producing larger output.
8692

8793
Dictionary-encoded columns are decoded before they are stored. A payload with no schema message has
8894
nowhere to record either that a column is dictionary encoded or the dictionary itself.
@@ -109,25 +115,27 @@ On a 5M-row relation of six flat columns:
109115

110116
| Query shape | Spark cache scan + convert | `CometInMemoryTableScan` | Relative |
111117
| ------------------------------ | -------------------------: | -----------------------: | -------: |
112-
| Repeated scan (3 of 6 columns) | 201 ms | 167 ms | 1.2x |
113-
| Selective filter | 69 ms | 61 ms | 1.1x |
114-
| Row count only (0 of 6) | 45 ms | 47 ms | 1.0x |
115-
| Narrow projection (1 of 6) | 70 ms | 57 ms | 1.2x |
116-
| Full projection (6 of 6) | 556 ms | 290 ms | 1.9x |
118+
| Repeated scan (3 of 6 columns) | 180 ms | 147 ms | 1.2x |
119+
| Selective filter | 58 ms | 50 ms | 1.2x |
120+
| Row count only (0 of 6) | 37 ms | 34 ms | 1.1x |
121+
| Narrow projection (1 of 6) | 52 ms | 48 ms | 1.1x |
122+
| Full projection (6 of 6) | 547 ms | 291 ms | 1.9x |
117123

118124
And on a 1M-row relation of six columns whose middle three are structs, one of them nested two
119125
levels deep:
120126

121127
| Query shape | Spark cache scan + convert | `CometInMemoryTableScan` | Relative |
122128
| -------------------------- | -------------------------: | -----------------------: | -------: |
123-
| Row count only (0 of 6) | 39 ms | 35 ms | 1.1x |
124-
| Narrow projection (1 of 6) | 109 ms | 61 ms | 1.8x |
125-
| Full projection (6 of 6) | 282 ms | 126 ms | 2.2x |
129+
| Row count only (0 of 6) | 30 ms | 29 ms | 1.0x |
130+
| Narrow projection (1 of 6) | 98 ms | 53 ms | 1.8x |
131+
| Full projection (6 of 6) | 250 ms | 125 ms | 2.0x |
132+
133+
Both columns read the cache at the default codec, `zstd`. The codec table above shows what `none`
134+
changes, and it is the full projection that moves most: nothing has to be inflated, so it runs
135+
several times faster, at six times the memory.
126136

127137
The two relations are not comparable to each other — different row counts, and a struct column
128-
carries several values per row. Within the struct relation the gap is wider than the flat one at
129-
every width, because the conversion the left column pays scales with the values per row rather than
130-
with the columns.
138+
carries several values per row.
131139

132140
Array and map columns are deliberately absent from the benchmark, not from the format — the cache
133141
stores and projects them, and `CometInMemoryCacheSuite` covers them. They cannot be measured _here_

‎spark/pom.xml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -625,6 +625,14 @@ under the License.
625625
<exclude>log4j2.properties</exclude>
626626
<exclude>arrow-git.properties</exclude>
627627
<exclude>**/SparkFilterApi.*</exclude>
628+
<!-- The jar plugin leaves this module's jar alone when no class has changed,
629+
and shade has already replaced that jar with its own output, so a build
630+
without `clean` shades an already shaded jar. That jar holds the service
631+
file the transformer below relocated last time, arrow-compression's
632+
original relocates to the same name, and the transformer keys the two
633+
separately and writes the name twice, failing the build. Excluding the
634+
already relocated copy on the way in leaves exactly one. -->
635+
<exclude>META-INF/services/${comet.shade.packageName}.arrow.*</exclude>
628636
</excludes>
629637
</filter>
630638
<filter>

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -287,14 +287,16 @@ object CometConf extends ShimCometConf {
287287
"The Arrow IPC compression codec used when Comet's cache serializer writes cached " +
288288
"data. Unlike spark.io.compression.codec, this compresses each Arrow buffer " +
289289
"separately rather than the batch as a whole, which is what lets a projected scan " +
290-
"decompress only the columns it selected. Set to none to store cached batches " +
291-
"uncompressed, which is both slower to write and larger than zstd because the extra " +
292-
"bytes cost more to move and store than compressing them costs. Only affects newly " +
293-
"cached data; the codec a batch was written with is recorded in the batch itself and " +
294-
"is what the read path uses. Arrow's lz4 is deliberately not offered: it is a " +
295-
"pure-Java implementation, unrelated to the JNI-accelerated lz4 behind " +
296-
"spark.io.compression.codec, and is orders of magnitude slower to write than zstd " +
297-
"while also producing larger output.")
290+
"decompress only the columns it selected. zstd is the default for footprint rather " +
291+
"than speed: it stores cached data in a fraction of the memory, but makes " +
292+
"materializing slower, and a read slower in proportion to how many columns it " +
293+
"selects. Set to none when a relation fits in memory uncompressed and is read at " +
294+
"close to full width. " +
295+
"Only affects newly cached data; the codec a batch was written with is recorded in " +
296+
"the batch itself and is what the read path uses. Arrow's lz4 is deliberately not " +
297+
"offered: it is a pure-Java implementation, unrelated to the JNI-accelerated lz4 " +
298+
"behind spark.io.compression.codec, and is orders of magnitude slower to write than " +
299+
"zstd while also producing larger output.")
298300
.stringConf
299301
.checkValues(Set("none", "zstd"))
300302
.createWithDefault("zstd")

‎spark/src/test/scala/org/apache/comet/exec/CometInMemoryCacheSuite.scala‎

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ import org.apache.spark.sql.{CometTestBase, Row}
3232
import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, Expression, GreaterThanOrEqual, LessThan, Literal}
3333
import org.apache.spark.sql.columnar.{CachedBatch, SimpleMetricsCachedBatch}
3434
import org.apache.spark.sql.comet.{CometBroadcastHashJoinExec, CometInMemoryTableScanExec, CometSortExec, CometSortMergeJoinExec}
35-
import org.apache.spark.sql.comet.execution.arrow.CometCachedBatchHelper
35+
import org.apache.spark.sql.comet.execution.arrow.{ArrowCachedBatchSerializer, CometCachedBatchHelper}
3636
import org.apache.spark.sql.comet.util.Utils
3737
import org.apache.spark.sql.execution.SortExec
3838
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, AQEShuffleReadExec, QueryStageExec, ShuffleQueryStageExec}
@@ -1056,7 +1056,7 @@ class CometInMemoryCacheSuite extends CometTestBase {
10561056
spark.sql("SELECT id FROM collated_case_cache WHERE s = 'x1'").collect().length == 1,
10571057
"a case-insensitive match must survive pruning")
10581058

1059-
// Null-count based pruning stays available for columns without bounds.
1059+
// IsNotNull is pushed down through the null count.
10601060
assert(
10611061
spark.sql("SELECT id FROM collated_cache WHERE s IS NOT NULL").collect().length == 100)
10621062
}
@@ -1498,6 +1498,43 @@ class CometInMemoryCacheSuite extends CometTestBase {
14981498
}
14991499
}
15001500

1501+
test("Comet in-memory cache rejects a payload that records an unknown compression codec") {
1502+
// `CodecType.fromCompressionType` answers NO_COMPRESSION for any byte outside its enum, so a
1503+
// reader that took its word for it would read a compressed body as plain bytes and hand back
1504+
// garbage values instead of failing. No other test reaches this branch: the writer can only
1505+
// record a byte one of Arrow's three CodecTypes owns, so the payload has to be assembled with
1506+
// a byte none of them does.
1507+
val unknownCodec = 99.toByte
1508+
val rows = 64
1509+
val ints = new IntVector("i", CometArrowAllocator)
1510+
try {
1511+
ints.allocateNew(rows)
1512+
(0 until rows).foreach(i => ints.set(i, i))
1513+
ints.setValueCount(rows)
1514+
val batch = new ColumnarBatch(Array[ColumnVector](new CometPlainVector(ints)), rows)
1515+
val cached = CometCachedBatchHelper.cachedBatchWithBodyCompression(batch, unknownCodec)
1516+
1517+
val attrs = Seq(AttributeReference("i", IntegerType)())
1518+
val thrown = interceptDecodeFailure {
1519+
new ArrowCachedBatchSerializer()
1520+
.convertCachedBatchToColumnarBatch(
1521+
spark.sparkContext.parallelize(Seq(cached), 1),
1522+
attrs,
1523+
attrs,
1524+
spark.sessionState.conf)
1525+
.mapPartitions(it => Iterator.single(it.map(_.numRows().toLong).sum))
1526+
.collect()
1527+
}
1528+
assert(
1529+
causeChain(thrown).exists(t =>
1530+
Option(t.getMessage)
1531+
.exists(_.contains(s"unknown Arrow compression codec: $unknownCodec"))),
1532+
s"an unrecognized codec must be reported as itself: $thrown")
1533+
} finally {
1534+
ints.close()
1535+
}
1536+
}
1537+
15011538
test("Comet in-memory cache stores no schema message per cached batch") {
15021539
// The reader rebuilds the schema from the cached relation's attributes, so storing one in
15031540
// every batch would repeat the same bytes for as many batches as the relation was cached in.

‎spark/src/test/scala/org/apache/spark/sql/benchmark/CometInMemoryCacheBenchmark.scala‎

Lines changed: 123 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import org.apache.spark.sql.SparkSession
2727
import org.apache.spark.sql.catalyst.expressions.AttributeReference
2828
import org.apache.spark.sql.comet.CometInMemoryTableScanExec
2929
import org.apache.spark.sql.comet.execution.arrow.ArrowCachedBatchSerializer
30-
import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
30+
import org.apache.spark.sql.execution.columnar.{InMemoryRelation, InMemoryTableScanExec}
3131
import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector
3232
import org.apache.spark.sql.internal.SQLConf
3333
import org.apache.spark.sql.types.{DataType, LongType, StringType}
@@ -70,6 +70,12 @@ object CometInMemoryCacheBenchmark extends CometBenchmarkBase {
7070
Seq("id", "sc", "deep", "wide", "tail", "d"),
7171
nestedNumRows)
7272

73+
// Every value spark.comet.exec.inMemoryCache.compression.codec accepts, default first. Arrow's
74+
// other IPC codec, LZ4_FRAME, is not one of them: it is commons-compress's pure-Java LZ4 rather
75+
// than the JNI-accelerated lz4-java behind spark.io.compression.codec, and the write path
76+
// rejects it.
77+
private val codecs = Seq("zstd", "none")
78+
7379
@volatile private var statsResult: (Array[Any], Array[Any], Array[Int]) = _
7480

7581
override def getSparkSession: SparkSession = {
@@ -224,9 +230,122 @@ object CometInMemoryCacheBenchmark extends CometBenchmarkBase {
224230
"SELECT count(id), count(sc.a), count(deep.n.v), count(wide.p), count(tail), count(d) " +
225231
s"FROM $nestedCacheTable",
226232
scanned = 6)
233+
234+
runCodecBenchmark(flatRelation)
235+
}
236+
}
237+
238+
/**
239+
* Every write codec the config accepts, over the same relation.
240+
*
241+
* The codec decides both how long materializing a relation takes and how much of a read is
242+
* decompression, and it is chosen once per relation, so it is a separate axis from the
243+
* projection widths above rather than another case alongside them.
244+
*
245+
* Only one copy is cached at a time. Two views over the same query would not give two cached
246+
* relations: the cache manager keys on the plan rather than the name, so the second would find
247+
* the first's copy and every codec would end up measuring whichever wrote last.
248+
*
249+
* Every case reads the cache natively. What moves between them is the codec, not the scan.
250+
*/
251+
private def runCodecBenchmark(relation: CachedRelation): Unit = {
252+
val view = s"${relation.table}_codec"
253+
254+
spark.catalog.clearCache()
255+
withTempTable(view) {
256+
spark
257+
.sql(s"SELECT ${relation.columns.mkString(", ")} FROM ${relation.source}")
258+
.createOrReplaceTempView(view)
259+
260+
var cached: String = null
261+
def cacheUnder(codec: String): Unit = if (cached != codec) {
262+
spark.catalog.uncacheTable(view)
263+
cached = null
264+
withSQLConf(cacheConf(nativeCacheEnabled = true) ++ codecConf(codec): _*) {
265+
spark.catalog.cacheTable(view)
266+
spark.table(view).count()
267+
}
268+
cached = codec
269+
}
270+
271+
// Measured before the cases below, while each codec's copy is the one freshly materialized:
272+
// the accumulator behind it belongs to whichever copy is cached now.
273+
val footprints = codecs.map { codec =>
274+
cacheUnder(codec)
275+
codec -> cachedBytes(view)
276+
}
277+
278+
val materialize =
279+
new Benchmark("in-memory cache materialize by codec", relation.rows, output = output)
280+
codecs.foreach { codec =>
281+
// Timed around the caching alone: dropping the previous copy is setup, and a plain
282+
// addCase would charge it to whichever codec happens to be running.
283+
materialize.addTimerCase(codec) { timer =>
284+
spark.catalog.uncacheTable(view)
285+
cached = null
286+
withSQLConf(cacheConf(nativeCacheEnabled = true) ++ codecConf(codec): _*) {
287+
timer.startTiming()
288+
spark.catalog.cacheTable(view)
289+
spark.table(view).count()
290+
timer.stopTiming()
291+
}
292+
cached = codec
293+
}
294+
}
295+
materialize.run()
296+
297+
// Footprint is not a time, so it has no column in a Benchmark table, but it is the other
298+
// half of why one codec is the default.
299+
footprints.foreach { case (codec, bytes) =>
300+
materialize.out.println(
301+
f"Cached footprint ($codec): ${bytes / (1024.0 * 1024.0)}%.1f MiB")
302+
}
303+
304+
// The two projection widths a codec can tell apart: a read decompresses only the buffers it
305+
// selected, so a narrow projection pays a proportionally smaller share of the codec's cost.
306+
// A row count decodes nothing at all and so cannot distinguish them.
307+
Seq(
308+
("narrow projection (1 of 6 columns)", s"SELECT count(k) FROM $view", 1),
309+
(
310+
"full projection (6 of 6 columns)",
311+
s"SELECT count(id), count(k), count(v), count(s1), count(s2), count(s3) FROM $view",
312+
6)).foreach { case (label, query, scanned) =>
313+
// The plan does not depend on the codec, so whichever copy is cached now will do.
314+
withSQLConf(cacheConf(nativeCacheEnabled = true): _*) {
315+
verifyPlan(query, nativeCacheEnabled = true, scanned)
316+
}
317+
val benchmark =
318+
new Benchmark(s"in-memory cache $label by codec", relation.rows, output = output)
319+
codecs.foreach { codec =>
320+
// Re-caching under this case's codec is setup, so it is outside the timer, and it only
321+
// happens on the case's first call, which is a warmup iteration.
322+
benchmark.addTimerCase(codec) { timer =>
323+
cacheUnder(codec)
324+
withSQLConf(cacheConf(nativeCacheEnabled = true): _*) {
325+
timer.startTiming()
326+
spark.sql(query).noop()
327+
timer.stopTiming()
328+
}
329+
}
330+
}
331+
benchmark.run()
332+
}
333+
334+
spark.catalog.uncacheTable(view)
227335
}
228336
}
229337

338+
/** What the cached relation behind `view` occupies, summed over its batches as written. */
339+
private def cachedBytes(view: String): Long = {
340+
val relation = spark
341+
.table(view)
342+
.queryExecution
343+
.optimizedPlan
344+
.collectFirst { case r: InMemoryRelation => r }
345+
.getOrElse(sys.error(s"$view is not cached"))
346+
relation.cacheBuilder.sizeInBytesStats.value
347+
}
348+
230349
private def runStatsBenchmark(): Unit = {
231350
val batchSize = 10000
232351
val types: Seq[DataType] = Seq.fill(3)(LongType) ++ Seq.fill(3)(StringType)
@@ -367,4 +486,7 @@ object CometInMemoryCacheBenchmark extends CometBenchmarkBase {
367486
"spark.comet.exec.onHeap.enabled" -> "true",
368487
"spark.sql.inMemoryColumnarStorage.batchSize" -> "10000")
369488
}
489+
490+
private def codecConf(codec: String): Seq[(String, String)] =
491+
Seq(CometConf.COMET_EXEC_IN_MEMORY_CACHE_COMPRESSION_CODEC.key -> codec)
370492
}

0 commit comments

Comments
 (0)