You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d5faf56
Browse filesBrowse the repository at this point in the historyBrowse files
fix: narrow the imported-memory claim and keep UDF output off the import allocator
The import allocator does not hold foreign memory exclusively. Arrow's
importer allocates the owning ArrowArray struct from it, and
loadValidityBuffer allocates a validity bitmap there when an imported
vector is all-valid or all-null and carries no validity buffer. Both are
bytes the JVM allocated, so the difference between the two counters is a
close lower bound on the JVM's own Arrow memory rather than an exact
split. Say so in the tracing guide, the allocator scaladoc and the
emission scaladoc, and note that the counters are separate reads of
process-wide state, so they are neither an atomic per-query balance nor
a measure of RSS.
A UDF may also allocate its result from the allocator it finds on its
inputs, which is the import allocator, and Data.exportVector does not
re-own the buffers. That output would then be reported as imported
native memory for as long as the export holds it. Transfer the result to
the root before exporting when the UDF allocated it elsewhere;
TransferPair moves ownership without copying the payload.
Add a characterization test pinning the import path's own overhead, and
a regression test for a UDF output allocated from the input allocator.
Record the new child in the memory management guide's allocator
inventory.
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
134
-
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
135
-
| jvm_arrow_allocated | Arrow memory held by Comet's JVM allocator, including buffers imported from native over the Arrow C Data Interface |
136
-
| jvm_arrow_imported |The imported part of `jvm_arrow_allocated`. It is native memory, so it also counts in `native_allocated`; subtract it to get Arrow memory the JVM allocated itself. |
137
-
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
138
-
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
139
-
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
142
+
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
143
+
| jvm_arrow_allocated | Arrow memory held by Comet's JVM allocator, including buffers imported from native over the Arrow C Data Interface |
144
+
| jvm_arrow_imported |Arrow memory held by the FFI import path, mostly buffers imported from native and so also counted in `native_allocated`. Includes a little JVM-allocated import overhead, so the subtraction is a lower bound.|
145
+
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
146
+
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
147
+
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
0 commit comments