Skip to content

Commit fc75952

Browse files
committed
refactor: rename SpillSorter.writeSortedFileNative and document the output buffer
Rename the spill-wide write to writeSortedPartitions so that writeSortedFileNative only names the per-partition native call, document that the encoder's output buffer is not reserved from a memory pool, and explain in the benchmark's scaladoc why it is separate from CometShuffleBenchmark.
1 parent c4ae0a8 commit fc75952

6 files changed

Lines changed: 15 additions & 6 deletions

File tree

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ Selection logic in `CometShuffleManager.shouldBypassMergeSort()`:
177177
3. Rows are serialized and buffered in off-heap memory pages
178178
4. When memory threshold or batch size is reached, the buffered rows are written. `CometDiskBlockWriter`
179179
calls `SpillWriter.doSpilling()` for its partition. `CometShuffleExternalSorter` calls
180-
`SpillSorter.writeSortedFileNative()` once for each spill and once more for the final file, and each
180+
`SpillSorter.writeSortedPartitions()` once for each spill and once more for the final file, and each
181181
call writes every partition
182182
5. Native code converts the rows to Arrow arrays and writes them as Arrow IPC blocks. `doSpilling()`
183183
passes one partition's row addresses and sizes to `Native.writeSortedFileNative()`. `SpillSorter`

‎native/shuffle/src/spark_unsafe/row.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1386,7 +1386,9 @@ struct SortedRowEncoder<'a> {
13861386
block_writer: Option<(SchemaRef, ShuffleBlockWriter)>,
13871387
codec_context: ShuffleCodecContext,
13881388
output: File,
1389-
/// Encoded blocks not yet written to `output`.
1389+
/// Encoded blocks not yet written to `output`. Native memory that no memory pool reserves, like
1390+
/// the builders. Its contents are written out as soon as they reach `OUTPUT_BUFFER_BYTES`, so
1391+
/// they never exceed that size plus one block. The `Vec` keeps the capacity it grew to.
13901392
buffer: Vec<u8>,
13911393
/// Encode and compression time across all batches.
13921394
ipc_time: Time,

‎spark/src/main/java/org/apache/spark/shuffle/sort/CometShuffleExternalSorter.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ public void spill() throws IOException {
199199

200200
activeSpillSorter.setSpillInfo(spillInfo);
201201

202-
activeSpillSorter.writeSortedFileNative(false, tracingEnabled);
202+
activeSpillSorter.writeSortedPartitions(false, tracingEnabled);
203203
updatePeakMemoryUsed();
204204
final long spillSize = activeSpillSorter.freeMemory();
205205
activeSpillSorter.reset();
@@ -338,7 +338,7 @@ public SpillInfo[] closeAndGetSpills() throws IOException {
338338
final SpillInfo spillInfo = new SpillInfo(numPartitions, file, blockId);
339339

340340
activeSpillSorter.setSpillInfo(spillInfo);
341-
activeSpillSorter.writeSortedFileNative(true, tracingEnabled);
341+
activeSpillSorter.writeSortedPartitions(true, tracingEnabled);
342342

343343
freeMemory();
344344
}

‎spark/src/main/java/org/apache/spark/shuffle/sort/SpillSorter.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ public int numRecords() {
243243
* #setSpillInfo}, every partition in one native call, recording each partition's length and
244244
* checksum.
245245
*/
246-
public void writeSortedFileNative(boolean isLastFile, boolean tracingEnabled) throws IOException {
246+
public void writeSortedPartitions(boolean isLastFile, boolean tracingEnabled) throws IOException {
247247
final int numRecords = inMemSorter.numRecords();
248248
// If there are no records, we don't need to create an empty spill file.
249249
if (numRecords == 0) {

‎spark/src/test/scala/org/apache/spark/shuffle/sort/SpillSorterSuite.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -343,7 +343,7 @@ class SpillSorterSuite extends AnyFunSuite with BeforeAndAfterEach {
343343
files += file
344344
sorter.setSpillInfo(
345345
new CometSpillInfo(numPartitions, file, TempShuffleBlockId(UUID.randomUUID())))
346-
sorter.writeSortedFileNative(isLastFile, false)
346+
sorter.writeSortedPartitions(isLastFile, false)
347347
sorter.freeMemory()
348348
if (!isLastFile) {
349349
sorter.reset()

‎spark/src/test/scala/org/apache/spark/sql/benchmark/CometColumnarShuffleWriteBenchmark.scala‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,13 @@ import org.apache.comet.CometConf
4141
* `spark.comet.shuffle.jvm.spillThreshold` makes Comet spill a fixed number of rows at a time. It
4242
* does not apply to Spark's shuffle, which does not spill in these cases.
4343
*
44+
* This is separate from `CometShuffleBenchmark`, which also covers the sort-based writer, because
45+
* it needs a session of its own. The master is fixed when the session starts, and this benchmark
46+
* uses `local[1]` so that the map tasks run one at a time and the time follows their cost, where
47+
* `CometShuffleBenchmark` uses `local[5]`. That benchmark also sets a default spill threshold, so
48+
* its no-spill cases would spill, and this one leaves the default so that its no-spill case does
49+
* not.
50+
*
4451
* To run this benchmark:
4552
* {{{
4653
* SPARK_GENERATE_BENCHMARK_FILES=1 make benchmark-org.apache.spark.sql.benchmark.CometColumnarShuffleWriteBenchmark

0 commit comments

Comments
 (0)