Skip to content

Commit 67e734a

Browse files
committed
docs: correct shuffle spill, nested hash keys, direct-read fallback, and FFI export dispatch
Addresses review feedback on the new review skills and the shuffle docs. - The local shuffle writer spills every partition into one file per task, not one file per partition. PartitionedSpill records per-partition byte ranges in write order and finish_partition copies them back in that order. - HashPartitioning keys are primitive-only only by default. The nested config admits structs and arrays recursively, and maps on Spark 4.0+. Range partitioning keys are primitive unconditionally, which is a different rule. - A shuffle output type that fails supportedSinkDataType does not retry as a regular Scan. convertToShuffleScan returns None and the Spark stage is kept. - NativeUtil.exportBatch has two cases, CometVector and ConstantColumnVector. CometSelectionVector and CometDelegateVector do not exist, and ScanExec imports through pull_next with no selection-vector hook. Also brings the two shuffle docs in line with the corrected key rules, notes that round robin is disabled by default and why, and adds the partitioning and maxBufferBytes configs to the native shuffle config table.
1 parent e2f0549 commit 67e734a

4 files changed

Lines changed: 138 additions & 53 deletions

File tree

‎.ai/skills/review-comet-ffi-pr/SKILL.md‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -90,15 +90,29 @@ ownership. A new input shape needs a reader, not a special case elsewhere.
9090

9191
## 3. Vector Types and Export Dispatch
9292

93-
`NativeUtil.exportBatch()` matches on the concrete vector type. The `CometVector` hierarchy is
94-
`CometDecodedVector` with Plain, Dictionary, List, Map, and Struct subclasses, plus
95-
`CometSelectionVector` and `CometDelegateVector`.
96-
97-
- [ ] A new `CometVector` subclass has a case in `exportBatch()`
98-
- [ ] The case ordering is right. `CometSelectionVector` must be matched **before** the general
99-
`CometVector` case, or the selection is silently dropped and the exported batch has the wrong
100-
rows.
101-
- [ ] Selection vectors are applied where `scan.rs` expects them, in `ScanExec::get_next()`
93+
`NativeUtil.exportBatch()` matches on the column type and has exactly two cases. A `CometVector`
94+
exports its underlying `FieldVector` through `Data.exportVector`, passing the dictionary provider
95+
only when the field carries a dictionary. Spark's own `ConstantColumnVector` is materialized into a
96+
fresh Arrow vector first, because native takes Arrow arrays only. Anything else throws
97+
`"Comet execution only takes Arrow Arrays"`.
98+
99+
The `CometVector` hierarchy is the abstract `CometVector`, the abstract `CometDecodedVector` under
100+
it, and the concrete `CometPlainVector`, `CometDictionaryVector`, `CometListVector`,
101+
`CometMapVector`, and `CometStructVector`. All of them export through the one `CometVector` case, so
102+
a new subclass needs no new case as long as `getValueVector` returns an Arrow vector.
103+
104+
- [ ] A new column type that is **not** a `CometVector` needs its own case, and needs to say who
105+
owns the vector it materializes. The `ConstantColumnVector` case allocates a new Arrow vector
106+
per batch, which is a real cost on a hot path.
107+
- [ ] A new `CometVector` subclass whose `getValueVector` is synthesized rather than owned has a
108+
defined lifetime relative to the export.
109+
- [ ] Import is symmetric. `ScanExec::pull_next` runs every imported column through
110+
`import_column`, which decodes invalid UTF-8 to Spark's rendering before
111+
`copy_or_unpack_array` with `CopyMode::UnpackOrClone`. A PR that adds a column type has to say
112+
what that pair does to it.
113+
- [ ] The row-count check still holds. `exportBatch` requires every column to report the same value
114+
count and throws otherwise, which is the guard that catches a vector exported at the wrong
115+
length.
102116

103117
## 4. Alignment and Schema
104118

‎.ai/skills/review-comet-shuffle-pr/SKILL.md‎

Lines changed: 47 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -42,13 +42,30 @@ piece is the most common way to break the other.
4242

4343
## 1. Which Implementation
4444

45-
| Implementation | Selected when |
46-
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
47-
| Native, `CometExchange` | `shuffle.mode` is `native` or `auto`, child is a `CometPlan`, supported partitioning, primitive partition keys |
48-
| JVM columnar, `CometColumnarExchange` | `shuffle.mode` is `jvm`, or the child is row-based, or partition keys are complex types |
49-
50-
Complex types are fully supported as **data** columns in both. The primitive-only restriction
51-
applies to **partition keys** for `HashPartitioning` and `RangePartitioning` only.
45+
| Implementation | Selected when |
46+
| ------------------------------------- | --------------------------------------------------------------------------------------------------------- |
47+
| Native, `CometExchange` | `shuffle.mode` is `native` or `auto`, child is a `CometPlan`, supported partitioning, supported key types |
48+
| JVM columnar, `CometColumnarExchange` | `shuffle.mode` is `jvm`, or the child is row-based, or a partition key type native shuffle cannot handle |
49+
50+
Complex types are fully supported as **data** columns in both. The restriction is on **partition
51+
keys**, and it is not the same rule for the two partitionings:
52+
53+
- **`RangePartitioning` is primitive-only, unconditionally.**
54+
`supportedRangePartitioningDataType` rejects every nested type, because native cannot sort them,
55+
and rejects collated strings, because Comet compares raw bytes. It also rejects float and double
56+
when `spark.comet.exec.strictFloatingPoint` is on.
57+
- **`HashPartitioning` is primitive-only by default only.** With
58+
`spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),
59+
`supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark
60+
4.0+, where Spark's `mapsort` normalization makes physical entry order irrelevant. The recursion
61+
still rejects a collated string at any depth, and a map on Spark 3.x. A map key also has to
62+
clear the separate expression check on the inserted `mapsort(...)`, which `CometMapSort`
63+
(`spark/src/main/spark-4.x/`) supports for scalar map keys only.
64+
`CometNativeShuffleSuite` covers both settings of the config.
65+
66+
A native exchange on a struct or array hash key is therefore not automatically a bug. Check the
67+
config before calling it one, and do not carry the range-partitioning rule over to hash
68+
partitioning.
5269

5370
- [ ] A PR that widens what native shuffle supports updates the fallback conditions in
5471
`CometShuffleExchangeExec` **and** both docs' "When X is Used" lists
@@ -64,11 +81,13 @@ Partitioning is where shuffle silently produces wrong answers rather than failin
6481
- [ ] **Hash partitioning uses Murmur3 with seed 42** and `partition_id = hash % num_partitions`,
6582
matching Spark. Any change to the hash, the seed, or the modulo changes which rows land in
6683
which partition, which breaks a join between a Comet-shuffled side and a Spark-shuffled side.
67-
- [ ] **Round robin is hash-based on purpose.** Comet assigns partitions from a Murmur3 hash rather
68-
than cycling row by row, because determinism across task retries is required for correctness
69-
under fault tolerance. A PR that implements "true" round robin to fix skew breaks that. The
70-
known cost is that low-cardinality data distributes unevenly, and that is the accepted
71-
trade-off.
84+
- [ ] **Round robin is hash-based on purpose, and off by default.** Comet assigns partitions from a
85+
Murmur3 hash rather than cycling row by row, because determinism across task retries is
86+
required for correctness under fault tolerance. A PR that implements "true" round robin to fix
87+
skew breaks that. Two costs are accepted: low-cardinality data distributes unevenly, and
88+
unsorted rows land in different partitions than Spark's `UnsafeRow`-sorted assignment would
89+
put them, which is why `spark.comet.shuffle.native.partitioning.roundrobin.enabled` defaults
90+
to `false`. Sorted output is identical either way.
7291
- [ ] **Range partitioning bounds come from the driver.** Spark's `RangePartitioner` samples and
7392
computes boundaries, they are serialized into the native plan, and native does a binary
7493
search over comparable-row-format keys. A change to the comparison or the row encoding must
@@ -104,16 +123,26 @@ budgets.
104123

105124
**Native shuffle** uses the DataFusion memory pool. Partitions spill when the pool denies an
106125
allocation, or when buffered bytes reach `spark.comet.shuffle.native.maxBufferBytes`, which
107-
defaults to `0`, meaning the fixed limit is disabled and memory pressure is the only trigger. Each
108-
partition has its own spill file and multiple spills for a partition are concatenated when the
109-
final output is written.
126+
defaults to `0`, meaning the fixed limit is disabled and memory pressure is the only trigger.
127+
128+
**The local writer spills every partition into one file per task, not one file per partition.**
129+
`PartitionedSpill` (`native/shuffle/src/writers/local/spill.rs`) opens a single DataFusion spill
130+
file lazily on the first spill and records, per partition, the byte ranges holding that partition's
131+
blocks in write order. `finish_partition` copies a partition's ranges into the output file in that
132+
order. `spilling_every_partition_creates_one_file` asserts the file count after spilling 64
133+
partitions three times. The single-partition writer does not spill, and neither does the RSS
134+
writer, which pushes blocks to the remote service instead.
110135

111136
**JVM shuffle** allocates off-heap pages through `CometShuffleMemoryAllocator`, which is an
112137
ordinary Spark `MemoryConsumer` drawing from `spark.memory.offHeap.size`. `CometDiskBlockWriter`
113138
coordinates spilling across partition writers, largest first. `TooLargePageException` is the signal
114139
that a single record does not fit in a page.
115140

116141
- [ ] A PR that adds buffering on either path says where the reservation is
142+
- [ ] A change to native spill bookkeeping keeps the recorded ranges and the file in sync.
143+
`PartitionedSpill` sets a `failed` flag on a partial write because a mid-write failure makes
144+
every later range untrustworthy, and a range that runs past the end of a truncated file must
145+
fail the task rather than write a short partition.
117146
- [ ] A PR that changes spill thresholds has benchmark evidence, because spilling too late is an
118147
OOM and spilling too early is a throughput loss
119148
- [ ] Scratch buffers reused for partition-id computation are correctly reset between batches
@@ -164,8 +193,9 @@ Ask specifically:
164193
- [ ] Is the **other** shuffle path tested, if the change touched anything shared?
165194
- [ ] Is the Celeborn path tested, if the change touched the manager, dependency, writer, or reader?
166195
- [ ] Are multiple partitions and an actual spill exercised, rather than one small batch?
167-
- [ ] Are complex types covered as data columns, and is the primitive-key restriction tested at its
168-
boundary?
196+
- [ ] Are complex types covered as data columns, and is the partition-key restriction tested at its
197+
boundary, on **both** settings of
198+
`spark.comet.shuffle.native.partitioning.hash.nested.enabled`?
169199
- [ ] For a partitioning change, is there a test that the same input produces the same partition
170200
assignment as Spark, not just that the total row count matches?
171201

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,10 +46,13 @@ JVM shuffle (`CometColumnarExchange`) is used instead of native shuffle (`CometE
4646
(not a `CometPlan`), JVM shuffle is the only option since native shuffle requires columnar input
4747
from Comet operators.
4848

49-
3. **Unsupported partition key types**: For `HashPartitioning` and `RangePartitioning`, native shuffle
50-
only supports primitive types as partition keys. Complex types (struct, array, map) cannot be used
51-
as partition keys in native shuffle and will fall back to JVM columnar shuffle. Note that complex types are
52-
fully supported as data columns in both implementations.
49+
3. **Unsupported partition key types**: `RangePartitioning` keys must be primitive, so a complex
50+
range key always falls back here. `HashPartitioning` keys must be primitive only by default:
51+
setting `spark.comet.shuffle.native.partitioning.hash.nested.enabled` to `true` keeps struct and
52+
array keys, and map keys on Spark 4.0 and later, on the native path. The config defaults to
53+
`false`, so a complex hash key falls back to JVM columnar shuffle unless it is enabled. See
54+
[Supported partition key types](native_shuffle.md#when-native-shuffle-is-used) for the exact
55+
rules. Complex types are fully supported as data columns in both implementations.
5356

5457
## Input Handling
5558

0 commit comments

Comments
 (0)