Skip to content

Commit e8eedcb

Browse files
andygrovetest
andauthored
perf: project cached batches by buffer selection, prune on collated strings (apache#5543)
* perf: project cached batches by buffer selection, prune on collated strings Follow-up to apache#5051, applying items from apache#5487. Replace the per-column Arrow IPC stream layout of `CometCachedBatch` with a single encapsulated IPC record batch message per cached batch, carrying no Schema message and no end-of-stream marker. The reader rebuilds the schema from the cached relation's attributes, so a wide relation no longer repeats the same schema bytes once per cached batch. Compression moves from a whole-payload Spark codec to Arrow's per-buffer IPC compression. That is what makes projection cheap: the message metadata records every buffer's offset and length in the body, so `CachedBatchIpc.readProjected` copies out only the byte ranges of the columns a scan selected and decompresses just those. This subsumes the separate "drop the schema message" item, since there is no longer a per-column stream to frame. Dictionary-encoded columns are decoded before being stored: a payload with no schema message cannot describe a dictionary encoding. The codec defaults to zstd, and lz4 is deliberately not offered. Arrow's lz4 is commons-compress's pure-Java implementation, unrelated to the JNI-accelerated lz4-java behind `spark.io.compression.codec`. Over a 200k-row six-column relation it measured 205s to write against 347ms for zstd, while also producing larger output, so no workload prefers it. zstd also beats storing batches uncompressed on both axes (347ms and 2 MiB against 1743ms and 13 MiB), because the bytes it saves cost more to copy and store than compressing them costs. Decompression is done here rather than left to `VectorLoader`, which leaks: `VectorLoader.loadBuffers` collects a field's decompressed buffers into a local list and releases them only after the whole field loads, so a buffer that fails to decompress strands every buffer of that field decompressed before it. A string column reaches this, its offsets buffer decompressing before its data buffer throws. Also track statistics bounds for collated string columns, comparing with the collation's own ordering through a new `CometTypeShim.compareStrings`. Matching the bare `StringType` object excluded collated columns, which then got null bounds and no pruning. Benchmark over a 5M-row six-column relation, keeping the cached scan native against falling back to a Spark cache scan and converting: 1.3x on a repeated scan, 1.3x on a narrow projection and 2.3x on a full projection. * refactor: use Spark's interpreted ordering for bounds, hoist projection layout Cleanup pass over the cache format change. No behaviour change. Drop the `compareStrings` shim in favour of `TypeUtils.getInterpretedOrdering`. That method is public with the same signature on every supported Spark version, and on Spark 4 it resolves a `StringType` through `CollationFactory.fetchCollation(collationId).comparator` -- the comparison the shim was reaching for. So the collation awareness comes from Spark itself and the shim, its Spark 3.x stub and the hand-rolled per-type `compare` all go. The ordering is now resolved once per column per partition rather than being re-dispatched on the `DataType` twice per row. Build the projection's index layout once per partition instead of per batch. The node, buffer and variadic index arithmetic is a pure function of the cached schema and the selected columns, but it walks every field of the relation, so recomputing it per batch made the bookkeeping O(total columns) against O(selected columns) of useful work -- worst in the wide-relation, narrow-projection case the format exists for. `CachedBatchIpc.Projection` now holds that layout and the projected schema, and owns the whole decode; `ProjectedBatch` is left with ownership only. This also puts the projected schema next to the code that packs buffers in the same order, an invariant that previously spanned two files unstated. Smaller cleanups: use Arrow's `DataSizeRoundingUtil.roundUpTo8Multiple` rather than open-coding IPC body alignment; size the serialization buffer from the record batch's known body length instead of growing from 32 bytes; resolve decompressors once instead of per batch; share the dictionary lookup guard between `Utils.combineDictionaryProviders` and the cache writer; read the codec config through one helper carrying the driver-vs-executor rationale; and collapse the duplicated compressed-buffer predicate and scramble loop in the test helper. Corrects two `Utils` scaladocs that still described the per-column stream format this change replaced. Benchmark and codec figures in the docs re-measured against the current code. * fix: relocate the arrow-compression service file when shading arrow-compression ships META-INF/services/org.apache.arrow.vector.compression.CompressionCodec$Factory. The shade plugin copies it verbatim without a ServicesResourceTransformer, so the jar declared a provider for Spark's own unshaded Arrow interface while naming a class that exists here only under the relocated package. Every ServiceLoader lookup Spark's Arrow made then failed with a ServiceConfigurationError, which took CompressionCodec.Factory's static initializer down with it and broke unrelated Arrow IPC reads, including mapInArrow. Add ServicesResourceTransformer so the service file name and its contents are both relocated. arrow-compression is the only bundled artifact that ships one. Also drop an unused NonFatal import that scalafix flagged. * test: drop the cache leak test that depends on zstd corruption detection "releases its vectors when a column fails part way through" zeroed the last 16 bytes of a compressed buffer and required the read to fail. Whether that fails is a property of the zstd runtime, not of Comet: the cached payload is byte-identical across Spark versions, but Comet takes zstd-jni from Spark rather than from arrow-compression, and 1.5.5 (Spark 3.4, 3.5) decodes that frame while 1.5.7 (Spark 4.x) reports it corrupt. So the test passed on 4.x and failed on 3.4 and 3.5. The scenario it claimed to cover is also unreachable: CachedBatchIpc decompresses every selected buffer before VectorLoader runs, so no content corruption can fail part way through the load. The two remaining leak tests corrupt a frame from its header onwards, which every zstd release rejects, and already cover a failure at a column's first buffer and a failure after an earlier buffer of the same column decoded. Records the constraint on scramble so a future test does not reach for a tail-only corruption again, and drops the now unused truncateColumn helper and the dictionary fixture's payload argument. * test: cover nested columns in the cached-batch projection tests and benchmark Addresses review feedback asking whether nested data should be tested and benchmarked. Nested columns were already round-tripped, but only under a full projection, which cannot see the part of the format that is nontrivial for them. A flat column always owns one field node and two or three buffers; a nested one owns a run as long as its subtree, and selecting every column covers the whole sequence however it is partitioned. So the buffer-span arithmetic was only exercised in the one shape where getting it wrong does not show. Adds two tests over a six-column relation whose middle four columns are a struct, an array, a map and a struct wrapping an array: - Each column takes its turn as the sole projection while the other five are corrupted, so a run computed short or long is caught by reaching into a corrupted neighbour. - Values are compared against the uncached query across single-column, paired and out-of-order projections. Row counts cannot catch a window that is misaligned but still decompresses, and out-of-order is the case a full projection cannot stand in for. The per-column statistics test now runs over the nested relation too, since a nested column's recorded size is the sum of its whole subtree. Both new tests fail if fieldNodeCount stops recursing into children. In the benchmark, adds the three projection widths over a relation of struct columns, and asserts the width each case claims. That assertion caught the existing "full projection (6 of 6 columns)" case reading three: count() over a non-nullable column is rewritten to count(1) by NullPropagation, which prunes the column out of the scan, and only k, s1 and s2 were nullable -- and those only incidentally, because Remainder can divide by zero. Every column of both relations is now nullable so count(c) genuinely reads c, and the documented numbers are regenerated. Array and map columns are left out of the benchmark deliberately: the baseline arm needs Spark's cache scan to bridge into Comet operators, and CometSparkToColumnarExec declines ArrayType and MapType, so for those the arm does not exist and the two cases stop measuring the same boundary. The docs say so rather than leaving it to be rediscovered. * fix: drop a redundant string interpolator flagged by scalafix RedundantSyntax * review: check the cached layout, and address the rest of the review Reader-side: a cached payload carries no schema, so `Projection` derived every node and buffer window from `Utils.toArrowSchema(cacheAttributes)` with nothing checking the writer had produced that layout. `load` now compares `nodesLength()`/`buffersLength()` against the totals `selectedRange` already computes, before any unchecked `batch.buffers(j)`. Writer-side: `isArrowBacked` accepts a `FixedSizeBinaryVector` for a `BinaryType` column, which is two buffers where the reader rebuilds three, and it answers for the top-level vector only -- so a struct of large strings passes it and is stored with 64-bit offsets. `matchesReaderLayout` compares the batch's Arrow types against the reader's recursively, and a batch that disagrees takes the conversion path instead. A dictionary column's field carries the index type, so the dictionary's field is what is compared. Also: an unrecognized body-compression byte is rejected rather than read as plain bytes, `fieldVariadicCount` and the variadic plumbing are gone (the length check covers view vectors, which the counts would not have), `columnSizes` no longer re-walks each column's subtree, the write codec is a case class rather than a bare tuple, the per-partition `Projection` is lazy so a row-count-only read never builds it, `hydrateDictionaries` is `decodeDictionaries`, `Projection` takes an `IndexedSeq`, and the stale `readProjected` links and some over-long comments are fixed. Tests: the two projection tests become one parameterized over both relations, caching once and restoring the payload between columns instead of re-caching; the two leak tests become one with two corruption points. New tests cover the reader's layout check and the writer declining a fixed-size-binary batch. * review: own the write-side compression buffers, fix the activation example Compressing through VectorUnloader leaks on the failure path: appendNodes retains each input buffer and accumulates the compressed ones into a list local to getRecordBatch, so a buffer that fails to compress strands that retain and leaves every buffer compressed before it reachable from nothing. Closing the input batch afterwards undoes neither. Unload plain and compress in CachedBatchIpc.compressed instead, mirroring what decompressed already does on the read side, so every allocation stays reachable from an error path that owns it. The docs enabled the cache with spark.conf.set, which cannot work: the driver plugin picks spark.sql.cache.serializer while the SparkContext is initializing. Show it as a startup --conf. Also drops a redundant s interpolator that the scalafix lint rejected. * fix: store cached batch payloads in chunks so a batch can exceed 2 GiB A single Array[Byte] capped each cached batch at the 2 GiB one JVM array can hold, a regression against the chunked format this replaced. Nothing bounds a batch's bytes upstream, so wide rows or a large columnar input batch could reach it. Write the IPC message into ChunkedByteBuffer heap chunks and copy each projected buffer window out chunk by chunk. An internal spark.comet.exec.inMemoryCache.chunkSize config lets the projection, corruption and leak tests run with 61-byte chunks so every window crosses a chunk boundary. * review: benchmark the cache codec, test an unknown codec, fix non-clean shading The codec table in the docs had zstd beating `none` on write and read speed as well as footprint, and it did not come from a committed benchmark. CometInMemoryCacheBenchmark now has a codec axis over the flat relation -- materialize time, footprint, and narrow and full reads -- and with each case warmed up, `none` is faster at everything but footprint: 1091 ms against 1503 ms to materialize and 63 ms against 296 ms for a full read, at 315 MiB against 55 MiB. zstd stays the default, on footprint grounds, and the docs and the compression.codec doc string now say so. Every table on the docs page is regenerated from one run. The unknown body-compression byte branch in readCodec had no test. A helper now assembles a payload whose body records a byte no Arrow codec owns, and a test asserts the read fails naming it. Any build without `clean` in which no class had changed failed in shade with a duplicate META-INF/services entry: the jar plugin leaves the previously shaded jar in place, and the services transformer then writes the relocated codec registration twice. Exclude the already relocated copy on the way in. Also fixes a collation-test comment that no longer described its assertion. * review: keep prefix pruning off collated columns, free compressed windows, measure Spark reads Spark prunes StartsWith by cutting each bound to the prefix's length in characters, which a collation can break: under UTF8_LCASE, U+0130 is one character that lowercases to two, so a batch of values beginning with it was pruned against a two-character prefix they all match. buildFilter now leaves out any predicate holding a StartsWith over a non-default collation, and keeps equality and range pruning on those columns. Arrow returns a buffer it stored raw as a slice of whatever holds it, so with every selected window in one allocation, the smallest raw buffer kept the batch's compressed bytes alive as long as the decoded vectors. Projection.load now reads each window's uncompressed-length prefix and copies the raw ones into a second allocation, and gives empty windows Arrow's shared empty buffer. The value tests collected their reference after caching, which Spark then answered from the cache too; they, and the codec round-trip test, now collect it first. New tests cover the collated prefix case, the raw-buffer case, a projection of only NullType columns, and a batch with no rows. CometInMemoryCacheBenchmark gains an axis comparing reads by Spark operators against Spark's own cache format, which is what the docs' limitation now quotes, in place of a figure measured on the previous format. The docs also no longer say the format ignores every runtime config, since the codec is one. * fix: read cached relation size via computeStats so the benchmark compiles on Spark 4.2 Spark 4.2 replaced CachedRDDBuilder.sizeInBytesStats with per-partition materialized stats, breaking test compilation in the merge queue. --------- Co-authored-by: test <[email protected]>
1 parent ccc780a commit e8eedcb

12 files changed

Lines changed: 2721 additions & 403 deletions

File tree

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# In-Memory Cache
21+
22+
Comet can store Spark's in-memory cache (`CACHE TABLE`, `df.cache()`, `df.persist()`) in an Arrow
23+
format that Comet operators read directly. Without it, a cached table is stored in Spark's own
24+
format and every scan of it has to convert each batch before Comet can continue, which shows up in
25+
the plan as a `CometSparkColumnarToColumnar` above the cache scan.
26+
27+
This feature is **experimental and disabled by default**. Turn it on at startup, alongside the rest
28+
of Comet's configuration:
29+
30+
```shell
31+
$SPARK_HOME/bin/spark-shell \
32+
... \
33+
--conf spark.comet.exec.inMemoryCache.enabled=true
34+
```
35+
36+
It has to be set before the `SparkContext` starts. Comet's driver plugin chooses
37+
`spark.sql.cache.serializer` once, while the context is initializing, so a session that started
38+
with the default goes on using Spark's cache format however the config is set afterwards.
39+
40+
## What changes when it is enabled
41+
42+
With Comet's serializer installed as `spark.sql.cache.serializer`:
43+
44+
- Cached data is stored as `CometCachedBatch` rather than Spark's `DefaultCachedBatch`.
45+
- Cached tables are scanned by `CometInMemoryTableScan`, which feeds Comet operators directly.
46+
- Per-batch column statistics are recorded in the layout Spark's `SimpleMetricsCachedBatchSerializer`
47+
expects, so Spark can prune whole cached batches on a predicate before any of them is decoded.
48+
49+
Relations whose schema Comet's Arrow writer cannot store — interval types, most notably — are
50+
delegated in full to Spark's default cache format, per relation. Which format a relation uses does
51+
not depend on a runtime config, because `spark.sql.cache.serializer` is a static setting and a
52+
relation whose format could change mid-session could not be read back reliably. The compression
53+
codec is a runtime config, but each batch records the codec it was written with, so data cached
54+
under one setting stays readable after the setting changes. Turning
55+
`spark.comet.exec.inMemoryCache.enabled` off at runtime only sends cached scans back to Spark's
56+
execution path; the cached data stays readable either way.
57+
58+
## Storage format
59+
60+
Each cached batch is stored as a single Arrow IPC record batch message and its body.
61+
62+
The message carries **no Arrow schema**. The reader already has one: `InMemoryRelation` knows the
63+
cached relation's attributes, and Comet maps them to exactly the Arrow fields the writer produced.
64+
Storing a schema in every batch would repeat the same bytes once per cached batch — for a wide
65+
relation cached in many batches, a large share of a payload that is not data.
66+
67+
Compression is applied by Arrow to **each buffer separately**, rather than by wrapping the whole
68+
payload in a Spark compression codec. That is what makes a projected read cheap: the message
69+
metadata records every buffer's offset and length within the body, so a scan copies out only the
70+
byte ranges belonging to the columns it selected, and only those are decompressed. A read of one
71+
column out of six does roughly a sixth of the decompression work, and a `SELECT count(*)`, which
72+
selects no columns at all, answers from the row count stored beside the payload without touching
73+
it.
74+
75+
Compression defaults to `zstd`, for footprint rather than for speed. Over the same 5M-row,
76+
six-column relation the tables under [Performance](#performance) use — and measured by the same
77+
benchmark — it holds the data in a sixth of the memory and pays for that on both sides: about 40%
78+
longer to materialize, and, on a read wide enough to inflate everything, close to five times
79+
longer. A narrow projection pays far less, because it only inflates the columns it asked for.
80+
81+
| Codec | Materialize | Footprint | Read 1 of 6 | Read 6 of 6 |
82+
| ------ | ----------: | --------: | ----------: | ----------: |
83+
| `zstd` | 1507 ms | 55 MiB | 45 ms | 295 ms |
84+
| `none` | 1081 ms | 315 MiB | 35 ms | 64 ms |
85+
86+
`none` is the better setting for a relation that fits in memory uncompressed and is read at close
87+
to full width. The default is the other way round because a cache that does not fit costs more than
88+
one that is slower to read, and Spark's own cache format compresses by default too.
89+
90+
Arrow's other IPC codec, LZ4, is deliberately not offered and the config rejects it. It is
91+
commons-compress's pure-Java implementation, unrelated to the JNI-accelerated lz4-java behind
92+
`spark.io.compression.codec`, and is orders of magnitude slower to write than `zstd` while also
93+
producing larger output.
94+
95+
Dictionary-encoded columns are decoded before they are stored. A payload with no schema message has
96+
nowhere to record either that a column is dictionary encoded or the dictionary itself.
97+
98+
## Configuration
99+
100+
| Config | Default | Description |
101+
| ------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
102+
| `spark.comet.exec.inMemoryCache.enabled` | `false` | Whether to store and scan Spark's in-memory cache in Comet's format. Read at startup. |
103+
| `spark.comet.exec.inMemoryCache.compression.codec` | `zstd` | Arrow IPC compression codec for cached data: `zstd` or `none`. Affects newly cached data only — a batch records the codec it was written with. |
104+
| `spark.comet.exec.inMemoryCache.compression.zstd.level` | `1` | Compression level when the codec is `zstd`. Ignored otherwise. |
105+
106+
## Performance
107+
108+
Measured with `CometInMemoryCacheBenchmark` (Apple M3 Max, JDK 17, Spark 4.1, release build).
109+
Regenerate with:
110+
111+
```sh
112+
SPARK_GENERATE_BENCHMARK_FILES=1 \
113+
make benchmark-org.apache.spark.sql.benchmark.CometInMemoryCacheBenchmark
114+
```
115+
116+
On a 5M-row relation of six flat columns:
117+
118+
| Query shape | Spark cache scan + convert | `CometInMemoryTableScan` | Relative |
119+
| ------------------------------ | -------------------------: | -----------------------: | -------: |
120+
| Repeated scan (3 of 6 columns) | 209 ms | 172 ms | 1.2x |
121+
| Selective filter | 72 ms | 61 ms | 1.2x |
122+
| Row count only (0 of 6) | 46 ms | 38 ms | 1.2x |
123+
| Narrow projection (1 of 6) | 70 ms | 58 ms | 1.2x |
124+
| Full projection (6 of 6) | 566 ms | 324 ms | 1.7x |
125+
126+
And on a 1M-row relation of six columns whose middle three are structs, one of them nested two
127+
levels deep:
128+
129+
| Query shape | Spark cache scan + convert | `CometInMemoryTableScan` | Relative |
130+
| -------------------------- | -------------------------: | -----------------------: | -------: |
131+
| Row count only (0 of 6) | 38 ms | 32 ms | 1.2x |
132+
| Narrow projection (1 of 6) | 109 ms | 58 ms | 1.9x |
133+
| Full projection (6 of 6) | 275 ms | 130 ms | 2.1x |
134+
135+
Both columns read the cache at the default codec, `zstd`. The codec table above shows what `none`
136+
changes, and it is the full projection that moves most: nothing has to be inflated, so it runs
137+
several times faster, at six times the memory.
138+
139+
The two relations are not comparable to each other — different row counts, and a struct column
140+
carries several values per row.
141+
142+
Array and map columns are deliberately absent from the benchmark, not from the format — the cache
143+
stores and projects them, and `CometInMemoryCacheSuite` covers them. They cannot be measured _here_
144+
because the left column would not exist: it needs Spark's cache scan to bridge into Comet operators,
145+
and `CometSparkToColumnarExec` declines `ArrayType` and `MapType`, so a query projecting one falls
146+
back to Spark row execution above the scan and the two columns stop measuring the same boundary.
147+
148+
Read what this compares carefully. Comet execution is on in both columns, so the aggregation runs
149+
on Comet either way and only the cache-scan boundary moves: on the left, Spark's
150+
`InMemoryTableScanExec` feeds those same Comet operators through a `CometSparkColumnarToColumnar`
151+
bridge; on the right, `CometInMemoryTableScan` feeds them directly. Both columns read the same
152+
Comet-written `CometCachedBatch`. These numbers are therefore "keep the cached scan native" against
153+
"fall back to a Spark cache scan and convert", not Comet against Spark execution, and not a
154+
comparison with Spark's own cache format. That comparison is under [Limitations](#limitations).
155+
156+
## Kryo
157+
158+
Spark serializes a cached batch with `spark.serializer` whenever the block leaves the heap: the
159+
`_SER` storage levels, replication, cross-executor fetches, and the disk half of the default
160+
`MEMORY_AND_DISK`. So an ordinary `df.cache()` that spills is enough to reach it.
161+
162+
If you run with `spark.kryo.registrationRequired=true`, register Comet's classes:
163+
164+
```
165+
spark.serializer=org.apache.spark.serializer.KryoSerializer
166+
spark.kryo.registrationRequired=true
167+
spark.kryo.registrator=org.apache.comet.CometKryoRegistrator
168+
```
169+
170+
Comet cannot set `spark.kryo.registrator` for you the way it sets `spark.sql.cache.serializer`:
171+
`KryoSerializer` reads it when `SparkEnv` builds the serializer, which happens before any plugin
172+
runs. Without it, caching fails with a "Class is not registered" error that does not name this
173+
feature. Comet's driver plugin warns at startup when it sees Kryo, `registrationRequired`, and no
174+
registrator.
175+
176+
## Limitations
177+
178+
Reads that feed **Spark** operators rather than Comet ones are slower than Spark's own cache
179+
format, and the narrower the read, the wider the gap. Measured by the same benchmark over the same
180+
5M-row relation, with Comet off so that Spark operators consume the cached data:
181+
182+
| Read shape | Spark's cache format | Comet's cache format | Slowdown |
183+
| ----------------------- | -------------------: | -------------------: | -------: |
184+
| Row count only (0 of 6) | 35 ms | 183 ms | 5.2x |
185+
| 1 of 6 columns | 54 ms | 257 ms | 4.8x |
186+
| 3 of 6 columns | 98 ms | 331 ms | 3.4x |
187+
| 6 of 6 columns | 410 ms | 623 ms | 1.5x |
188+
189+
This is why the feature is off by default. The cause is not yet established;
190+
[#5485](https://github.com/apache/datafusion-comet/issues/5485) tracks it.
191+
192+
Comet's serializer exists because Spark's own Arrow cache format
193+
([SPARK-57268](https://issues.apache.org/jira/browse/SPARK-57268)) is only available from Spark
194+
4.3, which Comet does not yet support.

‎docs/source/user-guide/latest/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ to read more.
7474
Understanding Comet Plans <understanding-comet-plans>
7575
Tuning Guide <tuning>
7676
Metrics Guide <metrics>
77+
In-Memory Cache <in-memory-cache>
7778
PyArrow UDF Acceleration <pyarrow-udfs>
7879

7980
.. toctree::

‎pom.xml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,32 @@ under the License.
226226
<artifactId>arrow-c-data</artifactId>
227227
<version>${arrow.version}</version>
228228
</dependency>
229+
<dependency>
230+
<groupId>org.apache.arrow</groupId>
231+
<artifactId>arrow-compression</artifactId>
232+
<version>${arrow.version}</version>
233+
<!-- Arrow's per-buffer IPC compression codecs. Their backing libraries ship with Spark,
234+
so take them from there rather than bundling a second copy: commons-compress (LZ4
235+
frame) and zstd-jni are both on every Spark version Comet supports. -->
236+
<exclusions>
237+
<exclusion>
238+
<groupId>org.apache.commons</groupId>
239+
<artifactId>commons-compress</artifactId>
240+
</exclusion>
241+
<exclusion>
242+
<groupId>com.github.luben</groupId>
243+
<artifactId>zstd-jni</artifactId>
244+
</exclusion>
245+
<exclusion>
246+
<groupId>io.netty</groupId>
247+
<artifactId>netty-common</artifactId>
248+
</exclusion>
249+
<exclusion>
250+
<groupId>com.google.code.findbugs</groupId>
251+
<artifactId>jsr305</artifactId>
252+
</exclusion>
253+
</exclusions>
254+
</dependency>
229255

230256
<!-- Parquet dependencies -->
231257
<dependency>

‎spark/pom.xml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,10 @@ under the License.
6060
<groupId>org.apache.arrow</groupId>
6161
<artifactId>arrow-vector</artifactId>
6262
</dependency>
63+
<dependency>
64+
<groupId>org.apache.arrow</groupId>
65+
<artifactId>arrow-compression</artifactId>
66+
</dependency>
6367
<dependency>
6468
<groupId>org.scala-lang.modules</groupId>
6569
<artifactId>scala-collection-compat_${scala.binary.version}</artifactId>
@@ -621,6 +625,14 @@ under the License.
621625
<exclude>log4j2.properties</exclude>
622626
<exclude>arrow-git.properties</exclude>
623627
<exclude>**/SparkFilterApi.*</exclude>
628+
<!-- The jar plugin leaves this module's jar alone when no class has changed,
629+
and shade has already replaced that jar with its own output, so a build
630+
without `clean` shades an already shaded jar. That jar holds the service
631+
file the transformer below relocated last time, arrow-compression's
632+
original relocates to the same name, and the transformer keys the two
633+
separately and writes the name twice, failing the build. Excluding the
634+
already relocated copy on the way in leaves exactly one. -->
635+
<exclude>META-INF/services/${comet.shade.packageName}.arrow.*</exclude>
624636
</excludes>
625637
</filter>
626638
<filter>
@@ -665,6 +677,16 @@ under the License.
665677
<shadedPattern>${comet.shade.packageName}.guava.thirdparty</shadedPattern>
666678
</relocation>
667679
</relocations>
680+
<transformers>
681+
<!-- arrow-compression ships META-INF/services/org.apache.arrow.vector.compression.CompressionCodec$Factory.
682+
Copied verbatim, that file names Spark's own unshaded Arrow interface while
683+
pointing at a provider class that only exists here under the relocated name, so
684+
Spark's ServiceLoader lookup fails and takes CompressionCodec.Factory's static
685+
initializer down with it. This transformer relocates both the service file name
686+
and its contents. -->
687+
<transformer
688+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
689+
</transformers>
668690
</configuration>
669691
</execution>
670692
</executions>

0 commit comments

Comments
 (0)