Skip to content

Commit 0d23cb4

Browse files
committed
docs: name the shuffle writer's zstd context as memory the pool reserves but the counters miss
The multi-partition local shuffle writer charges its zstd context to the pool, and libzstd allocates that workspace through libc malloc, so it is in `reserved` without ever being in `allocated`. Two passages said no C library memory reaches the pool; both now call this context out as the exception, and the overhead sizing step says the difference understates what has to fit by one context per such task.
1 parent f0aa284 commit 0d23cb4

2 files changed

Lines changed: 19 additions & 10 deletions

File tree

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -404,15 +404,19 @@ diverge for several structural reasons:
404404
retained/dirty page cache all add resident bytes that no layer above the allocator can see.
405405
Freeing memory does not necessarily return pages to the OS.
406406
- **Non-Rust allocations.** Memory allocated by C dependencies through libc `malloc`, and anything
407-
`mmap`ed, never passes through Rust's `GlobalAlloc`, so neither the memory pool nor the
408-
allocation counters (`native_allocated`, `jemalloc_allocated`) see it. In a default build the C
409-
dependencies are libzstd (`zstd-sys`, behind the Parquet `zstd` codec), libhdfs (`hdfs-sys`,
410-
pulled in by the default `hdfs-opendal` feature), and the TLS stack used for cloud object stores
411-
(`aws-lc-sys`). Building with the `jemalloc` or `mimalloc` feature adds the allocator itself
412-
(`tikv-jemalloc-sys`, `libmimalloc-sys`). It is worth knowing which dependencies are _not_ C,
413-
because several names suggest otherwise: the other Parquet codecs are pure Rust in this build,
414-
`snap` for Snappy, `lz4_flex` for LZ4 and `zlib-rs` for gzip, as is `libbz2-rs-sys` despite its
415-
name, so those allocations do pass through `GlobalAlloc` and are counted.
407+
`mmap`ed, never passes through Rust's `GlobalAlloc`, so the allocation counters
408+
(`native_allocated`, `jemalloc_allocated`) never see it, and the memory pool tracks it only where
409+
an operator charges it by hand. The one place that happens today is the native shuffle writer's
410+
zstd context when a task writes to more than one local partition: the writer reserves the
411+
context's size for as long as it holds one, so those bytes are in `reserved` but not in
412+
`allocated`. In a default build the C dependencies are libzstd (`zstd-sys`, behind the Parquet
413+
`zstd` codec and the shuffle writer's `zstd` codec), libhdfs (`hdfs-sys`, pulled in by the default
414+
`hdfs-opendal` feature), and the TLS stack used for cloud object stores (`aws-lc-sys`). Building
415+
with the `jemalloc` or `mimalloc` feature adds the allocator itself (`tikv-jemalloc-sys`,
416+
`libmimalloc-sys`). It is worth knowing which dependencies are _not_ C, because several names
417+
suggest otherwise: the other Parquet codecs are pure Rust in this build, `snap` for Snappy,
418+
`lz4_flex` for LZ4 and `zlib-rs` for gzip, as is `libbz2-rs-sys` despite its name, so those
419+
allocations do pass through `GlobalAlloc` and are counted.
416420
- **Batches in flight across the FFI boundary.** Reservations stop at the operator that made them.
417421
Imported JVM batches are reserved only while a reserving operator holds them, and exported native
418422
batches have usually been released by the time the JVM receives them yet stay resident until the

‎docs/source/user-guide/latest/tuning/memory.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,12 @@ JVM's own non-heap memory. To size the overhead from it:
174174
non-heap memory, and add the largest difference seen on any executor.
175175
3. Add a margin on top. The log can miss the true peak between samples, and neither figure includes
176176
the allocator's fragmentation and retained pages, memory allocated by native C libraries such as
177-
zstd, or Comet's Arrow buffers on the JVM side.
177+
zstd, or Comet's Arrow buffers on the JVM side. The zstd context of the native shuffle writer is
178+
the exception: when `spark.comet.shuffle.compression.codec` is `zstd`, a shuffle task writing to
179+
more than one local partition reserves its context while spilling or writing its output, so that
180+
memory is in `reserved` but never in `allocated`, and the difference understates what has to fit
181+
in the overhead by one context per such task (about 1.3 MiB at the default level, close to 8 MiB
182+
at levels 7 and 8).
178183

179184
For example, a 16 GiB executor derives an overhead of 1638 MiB. If the largest difference in its
180185
log is the 1522.3 MiB in the line above, the overhead needs to be at least 1638 + 1523 = 3161 MiB

0 commit comments

Comments
 (0)