|
| 1 | +<!--- |
| 2 | + Licensed to the Apache Software Foundation (ASF) under one |
| 3 | + or more contributor license agreements. See the NOTICE file |
| 4 | + distributed with this work for additional information |
| 5 | + regarding copyright ownership. The ASF licenses this file |
| 6 | + to you under the Apache License, Version 2.0 (the |
| 7 | + "License"); you may not use this file except in compliance |
| 8 | + with the License. You may obtain a copy of the License at |
| 9 | +
|
| 10 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 11 | +
|
| 12 | + Unless required by applicable law or agreed to in writing, |
| 13 | + software distributed under the License is distributed on an |
| 14 | + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 15 | + KIND, either express or implied. See the License for the |
| 16 | + specific language governing permissions and limitations |
| 17 | + under the License. |
| 18 | +--> |
| 19 | + |
| 20 | +# In-Memory Cache |
| 21 | + |
| 22 | +Comet can store Spark's in-memory cache (`CACHE TABLE`, `df.cache()`, `df.persist()`) in an Arrow |
| 23 | +format that Comet operators read directly. Without it, a cached table is stored in Spark's own |
| 24 | +format and every scan of it has to convert each batch before Comet can continue, which shows up in |
| 25 | +the plan as a `CometSparkColumnarToColumnar` above the cache scan. |
| 26 | + |
| 27 | +This feature is **experimental and disabled by default**. Turn it on at startup, alongside the rest |
| 28 | +of Comet's configuration: |
| 29 | + |
| 30 | +```shell |
| 31 | +$SPARK_HOME/bin/spark-shell \ |
| 32 | + ... \ |
| 33 | + --conf spark.comet.exec.inMemoryCache.enabled=true |
| 34 | +``` |
| 35 | + |
| 36 | +It has to be set before the `SparkContext` starts. Comet's driver plugin chooses |
| 37 | +`spark.sql.cache.serializer` once, while the context is initializing, so a session that started |
| 38 | +with the default goes on using Spark's cache format however the config is set afterwards. |
| 39 | + |
| 40 | +## What changes when it is enabled |
| 41 | + |
| 42 | +With Comet's serializer installed as `spark.sql.cache.serializer`: |
| 43 | + |
| 44 | +- Cached data is stored as `CometCachedBatch` rather than Spark's `DefaultCachedBatch`. |
| 45 | +- Cached tables are scanned by `CometInMemoryTableScan`, which feeds Comet operators directly. |
| 46 | +- Per-batch column statistics are recorded in the layout Spark's `SimpleMetricsCachedBatchSerializer` |
| 47 | + expects, so Spark can prune whole cached batches on a predicate before any of them is decoded. |
| 48 | + |
| 49 | +Relations whose schema Comet's Arrow writer cannot store — interval types, most notably — are |
| 50 | +delegated in full to Spark's default cache format, per relation. Which format a relation uses does |
| 51 | +not depend on a runtime config, because `spark.sql.cache.serializer` is a static setting and a |
| 52 | +relation whose format could change mid-session could not be read back reliably. The compression |
| 53 | +codec is a runtime config, but each batch records the codec it was written with, so data cached |
| 54 | +under one setting stays readable after the setting changes. Turning |
| 55 | +`spark.comet.exec.inMemoryCache.enabled` off at runtime only sends cached scans back to Spark's |
| 56 | +execution path; the cached data stays readable either way. |
| 57 | + |
| 58 | +## Storage format |
| 59 | + |
| 60 | +Each cached batch is stored as a single Arrow IPC record batch message and its body. |
| 61 | + |
| 62 | +The message carries **no Arrow schema**. The reader already has one: `InMemoryRelation` knows the |
| 63 | +cached relation's attributes, and Comet maps them to exactly the Arrow fields the writer produced. |
| 64 | +Storing a schema in every batch would repeat the same bytes once per cached batch — for a wide |
| 65 | +relation cached in many batches, a large share of a payload that is not data. |
| 66 | + |
| 67 | +Compression is applied by Arrow to **each buffer separately**, rather than by wrapping the whole |
| 68 | +payload in a Spark compression codec. That is what makes a projected read cheap: the message |
| 69 | +metadata records every buffer's offset and length within the body, so a scan copies out only the |
| 70 | +byte ranges belonging to the columns it selected, and only those are decompressed. A read of one |
| 71 | +column out of six does roughly a sixth of the decompression work, and a `SELECT count(*)`, which |
| 72 | +selects no columns at all, answers from the row count stored beside the payload without touching |
| 73 | +it. |
| 74 | + |
| 75 | +Compression defaults to `zstd`, for footprint rather than for speed. Over the same 5M-row, |
| 76 | +six-column relation the tables under [Performance](#performance) use — and measured by the same |
| 77 | +benchmark — it holds the data in a sixth of the memory and pays for that on both sides: about 40% |
| 78 | +longer to materialize, and, on a read wide enough to inflate everything, close to five times |
| 79 | +longer. A narrow projection pays far less, because it only inflates the columns it asked for. |
| 80 | + |
| 81 | +| Codec | Materialize | Footprint | Read 1 of 6 | Read 6 of 6 | |
| 82 | +| ------ | ----------: | --------: | ----------: | ----------: | |
| 83 | +| `zstd` | 1507 ms | 55 MiB | 45 ms | 295 ms | |
| 84 | +| `none` | 1081 ms | 315 MiB | 35 ms | 64 ms | |
| 85 | + |
| 86 | +`none` is the better setting for a relation that fits in memory uncompressed and is read at close |
| 87 | +to full width. The default is the other way round because a cache that does not fit costs more than |
| 88 | +one that is slower to read, and Spark's own cache format compresses by default too. |
| 89 | + |
| 90 | +Arrow's other IPC codec, LZ4, is deliberately not offered and the config rejects it. It is |
| 91 | +commons-compress's pure-Java implementation, unrelated to the JNI-accelerated lz4-java behind |
| 92 | +`spark.io.compression.codec`, and is orders of magnitude slower to write than `zstd` while also |
| 93 | +producing larger output. |
| 94 | + |
| 95 | +Dictionary-encoded columns are decoded before they are stored. A payload with no schema message has |
| 96 | +nowhere to record either that a column is dictionary encoded or the dictionary itself. |
| 97 | + |
| 98 | +## Configuration |
| 99 | + |
| 100 | +| Config | Default | Description | |
| 101 | +| ------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | |
| 102 | +| `spark.comet.exec.inMemoryCache.enabled` | `false` | Whether to store and scan Spark's in-memory cache in Comet's format. Read at startup. | |
| 103 | +| `spark.comet.exec.inMemoryCache.compression.codec` | `zstd` | Arrow IPC compression codec for cached data: `zstd` or `none`. Affects newly cached data only — a batch records the codec it was written with. | |
| 104 | +| `spark.comet.exec.inMemoryCache.compression.zstd.level` | `1` | Compression level when the codec is `zstd`. Ignored otherwise. | |
| 105 | + |
| 106 | +## Performance |
| 107 | + |
| 108 | +Measured with `CometInMemoryCacheBenchmark` (Apple M3 Max, JDK 17, Spark 4.1, release build). |
| 109 | +Regenerate with: |
| 110 | + |
| 111 | +```sh |
| 112 | +SPARK_GENERATE_BENCHMARK_FILES=1 \ |
| 113 | + make benchmark-org.apache.spark.sql.benchmark.CometInMemoryCacheBenchmark |
| 114 | +``` |
| 115 | + |
| 116 | +On a 5M-row relation of six flat columns: |
| 117 | + |
| 118 | +| Query shape | Spark cache scan + convert | `CometInMemoryTableScan` | Relative | |
| 119 | +| ------------------------------ | -------------------------: | -----------------------: | -------: | |
| 120 | +| Repeated scan (3 of 6 columns) | 209 ms | 172 ms | 1.2x | |
| 121 | +| Selective filter | 72 ms | 61 ms | 1.2x | |
| 122 | +| Row count only (0 of 6) | 46 ms | 38 ms | 1.2x | |
| 123 | +| Narrow projection (1 of 6) | 70 ms | 58 ms | 1.2x | |
| 124 | +| Full projection (6 of 6) | 566 ms | 324 ms | 1.7x | |
| 125 | + |
| 126 | +And on a 1M-row relation of six columns whose middle three are structs, one of them nested two |
| 127 | +levels deep: |
| 128 | + |
| 129 | +| Query shape | Spark cache scan + convert | `CometInMemoryTableScan` | Relative | |
| 130 | +| -------------------------- | -------------------------: | -----------------------: | -------: | |
| 131 | +| Row count only (0 of 6) | 38 ms | 32 ms | 1.2x | |
| 132 | +| Narrow projection (1 of 6) | 109 ms | 58 ms | 1.9x | |
| 133 | +| Full projection (6 of 6) | 275 ms | 130 ms | 2.1x | |
| 134 | + |
| 135 | +Both columns read the cache at the default codec, `zstd`. The codec table above shows what `none` |
| 136 | +changes, and it is the full projection that moves most: nothing has to be inflated, so it runs |
| 137 | +several times faster, at six times the memory. |
| 138 | + |
| 139 | +The two relations are not comparable to each other — different row counts, and a struct column |
| 140 | +carries several values per row. |
| 141 | + |
| 142 | +Array and map columns are deliberately absent from the benchmark, not from the format — the cache |
| 143 | +stores and projects them, and `CometInMemoryCacheSuite` covers them. They cannot be measured _here_ |
| 144 | +because the left column would not exist: it needs Spark's cache scan to bridge into Comet operators, |
| 145 | +and `CometSparkToColumnarExec` declines `ArrayType` and `MapType`, so a query projecting one falls |
| 146 | +back to Spark row execution above the scan and the two columns stop measuring the same boundary. |
| 147 | + |
| 148 | +Read what this compares carefully. Comet execution is on in both columns, so the aggregation runs |
| 149 | +on Comet either way and only the cache-scan boundary moves: on the left, Spark's |
| 150 | +`InMemoryTableScanExec` feeds those same Comet operators through a `CometSparkColumnarToColumnar` |
| 151 | +bridge; on the right, `CometInMemoryTableScan` feeds them directly. Both columns read the same |
| 152 | +Comet-written `CometCachedBatch`. These numbers are therefore "keep the cached scan native" against |
| 153 | +"fall back to a Spark cache scan and convert", not Comet against Spark execution, and not a |
| 154 | +comparison with Spark's own cache format. That comparison is under [Limitations](#limitations). |
| 155 | + |
| 156 | +## Kryo |
| 157 | + |
| 158 | +Spark serializes a cached batch with `spark.serializer` whenever the block leaves the heap: the |
| 159 | +`_SER` storage levels, replication, cross-executor fetches, and the disk half of the default |
| 160 | +`MEMORY_AND_DISK`. So an ordinary `df.cache()` that spills is enough to reach it. |
| 161 | + |
| 162 | +If you run with `spark.kryo.registrationRequired=true`, register Comet's classes: |
| 163 | + |
| 164 | +``` |
| 165 | +spark.serializer=org.apache.spark.serializer.KryoSerializer |
| 166 | +spark.kryo.registrationRequired=true |
| 167 | +spark.kryo.registrator=org.apache.comet.CometKryoRegistrator |
| 168 | +``` |
| 169 | + |
| 170 | +Comet cannot set `spark.kryo.registrator` for you the way it sets `spark.sql.cache.serializer`: |
| 171 | +`KryoSerializer` reads it when `SparkEnv` builds the serializer, which happens before any plugin |
| 172 | +runs. Without it, caching fails with a "Class is not registered" error that does not name this |
| 173 | +feature. Comet's driver plugin warns at startup when it sees Kryo, `registrationRequired`, and no |
| 174 | +registrator. |
| 175 | + |
| 176 | +## Limitations |
| 177 | + |
| 178 | +Reads that feed **Spark** operators rather than Comet ones are slower than Spark's own cache |
| 179 | +format, and the narrower the read, the wider the gap. Measured by the same benchmark over the same |
| 180 | +5M-row relation, with Comet off so that Spark operators consume the cached data: |
| 181 | + |
| 182 | +| Read shape | Spark's cache format | Comet's cache format | Slowdown | |
| 183 | +| ----------------------- | -------------------: | -------------------: | -------: | |
| 184 | +| Row count only (0 of 6) | 35 ms | 183 ms | 5.2x | |
| 185 | +| 1 of 6 columns | 54 ms | 257 ms | 4.8x | |
| 186 | +| 3 of 6 columns | 98 ms | 331 ms | 3.4x | |
| 187 | +| 6 of 6 columns | 410 ms | 623 ms | 1.5x | |
| 188 | + |
| 189 | +This is why the feature is off by default. The cause is not yet established; |
| 190 | +[#5485](https://github.com/apache/datafusion-comet/issues/5485) tracks it. |
| 191 | + |
| 192 | +Comet's serializer exists because Spark's own Arrow cache format |
| 193 | +([SPARK-57268](https://issues.apache.org/jira/browse/SPARK-57268)) is only available from Spark |
| 194 | +4.3, which Comet does not yet support. |
0 commit comments