@@ -42,13 +42,30 @@ piece is the most common way to break the other.
4242
4343## 1. Which Implementation
4444
45- | Implementation | Selected when |
46- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
47- | Native, ` CometExchange ` | ` shuffle.mode ` is ` native ` or ` auto ` , child is a ` CometPlan ` , supported partitioning, primitive partition keys |
48- | JVM columnar, ` CometColumnarExchange ` | ` shuffle.mode ` is ` jvm ` , or the child is row-based, or partition keys are complex types |
49-
50- Complex types are fully supported as ** data** columns in both. The primitive-only restriction
51- applies to ** partition keys** for ` HashPartitioning ` and ` RangePartitioning ` only.
45+ | Implementation | Selected when |
46+ | ------------------------------------- | --------------------------------------------------------------------------------------------------------- |
47+ | Native, ` CometExchange ` | ` shuffle.mode ` is ` native ` or ` auto ` , child is a ` CometPlan ` , supported partitioning, supported key types |
48+ | JVM columnar, ` CometColumnarExchange ` | ` shuffle.mode ` is ` jvm ` , or the child is row-based, or a partition key type native shuffle cannot handle |
49+
50+ Complex types are fully supported as ** data** columns in both. The restriction is on ** partition
51+ keys** , and it is not the same rule for the two partitionings:
52+
53+ - ** ` RangePartitioning ` is primitive-only, unconditionally.**
54+ ` supportedRangePartitioningDataType ` rejects every nested type, because native cannot sort them,
55+ and rejects collated strings, because Comet compares raw bytes. It also rejects float and double
56+ when ` spark.comet.exec.strictFloatingPoint ` is on.
57+ - ** ` HashPartitioning ` is primitive-only by default only.** With
58+ ` spark.comet.shuffle.native.partitioning.hash.nested.enabled=true ` (default ` false ` ),
59+ ` supportedHashPartitioningDataType ` admits structs and arrays recursively, and maps on Spark
60+ 4.0+, where Spark's ` mapsort ` normalization makes physical entry order irrelevant. The recursion
61+ still rejects a collated string at any depth, and a map on Spark 3.x. A map key also has to
62+ clear the separate expression check on the inserted ` mapsort(...) ` , which ` CometMapSort `
63+ (` spark/src/main/spark-4.x/ ` ) supports for scalar map keys only.
64+ ` CometNativeShuffleSuite ` covers both settings of the config.
65+
66+ A native exchange on a struct or array hash key is therefore not automatically a bug. Check the
67+ config before calling it one, and do not carry the range-partitioning rule over to hash
68+ partitioning.
5269
5370- [ ] A PR that widens what native shuffle supports updates the fallback conditions in
5471 ` CometShuffleExchangeExec ` ** and** both docs' "When X is Used" lists
@@ -64,11 +81,13 @@ Partitioning is where shuffle silently produces wrong answers rather than failin
6481- [ ] ** Hash partitioning uses Murmur3 with seed 42** and ` partition_id = hash % num_partitions ` ,
6582 matching Spark. Any change to the hash, the seed, or the modulo changes which rows land in
6683 which partition, which breaks a join between a Comet-shuffled side and a Spark-shuffled side.
67- - [ ] ** Round robin is hash-based on purpose.** Comet assigns partitions from a Murmur3 hash rather
68- than cycling row by row, because determinism across task retries is required for correctness
69- under fault tolerance. A PR that implements "true" round robin to fix skew breaks that. The
70- known cost is that low-cardinality data distributes unevenly, and that is the accepted
71- trade-off.
84+ - [ ] ** Round robin is hash-based on purpose, and off by default.** Comet assigns partitions from a
85+ Murmur3 hash rather than cycling row by row, because determinism across task retries is
86+ required for correctness under fault tolerance. A PR that implements "true" round robin to fix
87+ skew breaks that. Two costs are accepted: low-cardinality data distributes unevenly, and
88+ unsorted rows land in different partitions than Spark's ` UnsafeRow ` -sorted assignment would
89+ put them, which is why ` spark.comet.shuffle.native.partitioning.roundrobin.enabled ` defaults
90+ to ` false ` . Sorted output is identical either way.
7291- [ ] ** Range partitioning bounds come from the driver.** Spark's ` RangePartitioner ` samples and
7392 computes boundaries, they are serialized into the native plan, and native does a binary
7493 search over comparable-row-format keys. A change to the comparison or the row encoding must
@@ -104,16 +123,26 @@ budgets.
104123
105124** Native shuffle** uses the DataFusion memory pool. Partitions spill when the pool denies an
106125allocation, or when buffered bytes reach ` spark.comet.shuffle.native.maxBufferBytes ` , which
107- defaults to ` 0 ` , meaning the fixed limit is disabled and memory pressure is the only trigger. Each
108- partition has its own spill file and multiple spills for a partition are concatenated when the
109- final output is written.
126+ defaults to ` 0 ` , meaning the fixed limit is disabled and memory pressure is the only trigger.
127+
128+ ** The local writer spills every partition into one file per task, not one file per partition.**
129+ ` PartitionedSpill ` (` native/shuffle/src/writers/local/spill.rs ` ) opens a single DataFusion spill
130+ file lazily on the first spill and records, per partition, the byte ranges holding that partition's
131+ blocks in write order. ` finish_partition ` copies a partition's ranges into the output file in that
132+ order. ` spilling_every_partition_creates_one_file ` asserts the file count after spilling 64
133+ partitions three times. The single-partition writer does not spill, and neither does the RSS
134+ writer, which pushes blocks to the remote service instead.
110135
111136** JVM shuffle** allocates off-heap pages through ` CometShuffleMemoryAllocator ` , which is an
112137ordinary Spark ` MemoryConsumer ` drawing from ` spark.memory.offHeap.size ` . ` CometDiskBlockWriter `
113138coordinates spilling across partition writers, largest first. ` TooLargePageException ` is the signal
114139that a single record does not fit in a page.
115140
116141- [ ] A PR that adds buffering on either path says where the reservation is
142+ - [ ] A change to native spill bookkeeping keeps the recorded ranges and the file in sync.
143+ ` PartitionedSpill ` sets a ` failed ` flag on a partial write because a mid-write failure makes
144+ every later range untrustworthy, and a range that runs past the end of a truncated file must
145+ fail the task rather than write a short partition.
117146- [ ] A PR that changes spill thresholds has benchmark evidence, because spilling too late is an
118147 OOM and spilling too early is a throughput loss
119148- [ ] Scratch buffers reused for partition-id computation are correctly reset between batches
@@ -164,8 +193,9 @@ Ask specifically:
164193- [ ] Is the ** other** shuffle path tested, if the change touched anything shared?
165194- [ ] Is the Celeborn path tested, if the change touched the manager, dependency, writer, or reader?
166195- [ ] Are multiple partitions and an actual spill exercised, rather than one small batch?
167- - [ ] Are complex types covered as data columns, and is the primitive-key restriction tested at its
168- boundary?
196+ - [ ] Are complex types covered as data columns, and is the partition-key restriction tested at its
197+ boundary, on ** both** settings of
198+ ` spark.comet.shuffle.native.partitioning.hash.nested.enabled ` ?
169199- [ ] For a partitioning change, is there a test that the same input produces the same partition
170200 assignment as Spark, not just that the total row count matches?
171201
0 commit comments