Skip to content

Commit 1e08791

Browse files
committed
docs: say where shuffle offsets are stored as lengths
The tuning guide and the config description now say that native shuffle and JVM shuffle blocks without dictionary-encoded columns store offsets as lengths with zstd, and that Celeborn shuffle blocks do not. native_shuffle.md describes the codec-tag flag, and the native and JVM shuffle config tables list spark.comet.shuffle.compression.offsetsAsLengths.enabled.
1 parent 3194f1a commit 1e08791

4 files changed

Lines changed: 22 additions & 10 deletions

File tree

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -215,9 +215,10 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
215215

216216
## Configuration
217217

218-
| Config | Description |
219-
| ---------------------------------------- | ------------------------------------------------- |
220-
| `spark.comet.shuffle.jvm.batchSize` | Rows per Arrow batch |
221-
| `spark.comet.shuffle.jvm.spillThreshold` | Row count threshold for spill |
222-
| `spark.comet.shuffle.compression.codec` | Compression codec, `lz4` by default |
223-
| `spark.comet.shuffle.directRead.enabled` | Decode blocks natively on read, `true` by default |
218+
| Config | Description |
219+
| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
220+
| `spark.comet.shuffle.jvm.batchSize` | Rows per Arrow batch |
221+
| `spark.comet.shuffle.jvm.spillThreshold` | Row count threshold for spill |
222+
| `spark.comet.shuffle.compression.codec` | Compression codec, `lz4` by default |
223+
| `spark.comet.shuffle.compression.offsetsAsLengths.enabled` | Store offsets as lengths in zstd blocks without dictionary-encoded columns, `true` by default |
224+
| `spark.comet.shuffle.directRead.enabled` | Decode blocks natively on read, `true` by default |

‎docs/source/contributor-guide/native_shuffle.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,10 @@ The native shuffle implementation is its own workspace crate, `datafusion-comet-
191191
- Writes compression type header
192192
- Writes field count header
193193
- Writes compressed IPC stream
194+
- With zstd and `spark.comet.shuffle.compression.offsetsAsLengths.enabled`, first rewrites
195+
the IPC body's offsets buffers as lengths in byte planes, and marks the block by setting the
196+
high bit of the codec tag's first byte (`native/shuffle/src/offsets_as_lengths.rs`). The
197+
reader rebuilds the offsets before decoding.
194198

195199
6. **Output**: One data file holds the concatenated partition data. The writer records the byte
196200
offset where each partition begins, plus the total length, and keeps them in memory.
@@ -270,8 +274,10 @@ leave that walk consistent with it.
270274

271275
`CometShuffleBlockIterator` reads a 16-byte header per block: an 8-byte compressed length, which
272276
includes the field count but not itself, followed by an 8-byte field count that is discarded because
273-
the schema comes from the `ShuffleScan` protobuf fields. The compressed body is read into a reused
274-
`DirectByteBuffer`, **valid only until the next `hasNext()` call**. Native must fully consume it
277+
the schema comes from the `ShuffleScan` protobuf fields. The 4-byte codec tag after it is read only by
278+
native code; the high bit of its first byte marks a block whose offsets buffers hold lengths. The
279+
compressed body is read into a reused `DirectByteBuffer`, **valid only until the next `hasNext()`
280+
call**. Native must fully consume it
275281
before pulling the next block, which `read_ipc_compressed` satisfies because it allocates fresh
276282
native memory for the decoded data.
277283

@@ -510,6 +516,7 @@ independently compressed, allowing parallel decompression during reads.
510516
| `spark.comet.shuffle.directRead.enabled` | `true` | Decode shuffle blocks in native code, bypassing Arrow FFI |
511517
| `spark.comet.shuffle.compression.codec` | `lz4` | Compression codec |
512518
| `spark.comet.shuffle.compression.zstd.level` | `1` | Zstd compression level |
519+
| `spark.comet.shuffle.compression.offsetsAsLengths.enabled` | `true` | Store offsets as lengths in zstd blocks |
513520
| `spark.comet.shuffle.native.writeBufferSize` | `1MB` | Write buffer size |
514521
| `spark.comet.shuffle.native.maxBufferBytes` | `0` | Fixed spill threshold. `0` disables it, leaving pool pressure |
515522
| `spark.comet.shuffle.native.partitioning.hash.enabled` | `true` | Allow `HashPartitioning` on the native path |

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,10 @@ controlled by `spark.comet.shuffle.compression.zstd.level` (default `1`).
117117

118118
With zstd, Comet also stores the offsets of string, binary, array and map columns as lengths before
119119
compressing each shuffle block. Lengths compress much better than offsets: in our measurements, shuffle
120-
files with such columns were 10% to 40% smaller, and zstd was no slower.
120+
files with such columns were 10% to 40% smaller, and zstd was no slower. This applies to native shuffle
121+
and to JVM shuffle blocks without dictionary-encoded columns. A JVM shuffle block in which
122+
`spark.comet.shuffle.jvm.preferDictionary.ratio` dictionary-encodes a column keeps the offsets of all
123+
its columns, and Celeborn shuffle blocks keep theirs too.
121124
`spark.comet.shuffle.compression.offsetsAsLengths.enabled` turns this off. LZ4 and snappy shuffle blocks
122125
are written as before.
123126

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -830,7 +830,8 @@ object CometConf extends ShimCometConf {
830830
"Whether Comet shuffle stores the offsets of string, binary, array and map columns as " +
831831
"lengths before compressing each block. Lengths compress much better than offsets, " +
832832
"which makes shuffle files with such columns smaller. Only takes effect when " +
833-
s"${COMET_SHUFFLE_COMPRESSION_CODEC.key} is zstd.")
833+
s"${COMET_SHUFFLE_COMPRESSION_CODEC.key} is zstd, and not for Celeborn shuffle or " +
834+
"for JVM shuffle blocks with dictionary-encoded columns, which keep their offsets.")
834835
.booleanConf
835836
.createWithDefault(true)
836837

0 commit comments

Comments
 (0)