Skip to content

Commit d5faf56

Browse files
committed
fix: narrow the imported-memory claim and keep UDF output off the import allocator
The import allocator does not hold foreign memory exclusively. Arrow's importer allocates the owning ArrowArray struct from it, and loadValidityBuffer allocates a validity bitmap there when an imported vector is all-valid or all-null and carries no validity buffer. Both are bytes the JVM allocated, so the difference between the two counters is a close lower bound on the JVM's own Arrow memory rather than an exact split. Say so in the tracing guide, the allocator scaladoc and the emission scaladoc, and note that the counters are separate reads of process-wide state, so they are neither an atomic per-query balance nor a measure of RSS. A UDF may also allocate its result from the allocator it finds on its inputs, which is the import allocator, and Data.exportVector does not re-own the buffers. That output would then be reported as imported native memory for as long as the export holds it. Transfer the result to the root before exporting when the UDF allocated it elsewhere; TransferPair moves ownership without copying the payload. Add a characterization test pinning the import path's own overhead, and a regression test for a UDF output allocated from the input allocator. Record the new child in the memory management guide's allocator inventory.
1 parent 4a302a1 commit d5faf56

7 files changed

Lines changed: 151 additions & 28 deletions

File tree

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,12 @@ off-heap bytes in container RSS that neither Spark's `TaskMemoryManager` nor Com
103103
pool sees. In practice the volume is modest, a batch at a time per stream, but there is no
104104
ceiling and no backpressure.
105105

106+
One further child, `CometArrowImportAllocator` (`comet-ffi-imports`), owns the buffers imported
107+
from native over the C Data Interface, so that tracing can report them apart from the Arrow memory
108+
the JVM allocates itself. Unlike the others it is process-wide and never closed, because imported
109+
buffers are reference counted and routinely outlive the task that imported them. Its reservation
110+
is zero, so every byte still escalates to the root and the inventory above is unchanged by it.
111+
106112
**The JVM shuffle allocator is an ordinary Spark consumer.** `CometShuffleMemoryAllocator.getInstance`
107113
returns `CometUnifiedShuffleMemoryAllocator`, a Spark `MemoryConsumer` drawing from
108114
`spark.memory.offHeap.size`, so shuffle pages are arbitrated against Spark's other consumers in the

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -122,18 +122,26 @@ not being tracked by the pool.
122122
Arrow memory on the JVM side is reported separately, because it is off-heap and so invisible to
123123
`jvm_heap_used`. Comet imports batches from native over the Arrow C Data Interface, and Arrow
124124
charges an imported buffer to whichever allocator wraps it, so those buffers are held in a
125-
dedicated child allocator and reported as `jvm_arrow_imported`. That portion is native memory
126-
counted twice, once in `native_allocated` and once in `jvm_arrow_allocated`; the difference
127-
`jvm_arrow_allocated - jvm_arrow_imported` is the Arrow memory the JVM allocated itself.
125+
dedicated child allocator and reported as `jvm_arrow_imported`. Most of that is native memory
126+
counted twice, once in `native_allocated` and once in `jvm_arrow_allocated`.
127+
128+
`jvm_arrow_imported` measures what the import path holds, which is not exclusively foreign memory.
129+
Arrow's importer allocates the owning `ArrowArray` struct from the same allocator, and
130+
`BitVectorHelper.loadValidityBuffer` allocates a validity bitmap there when a native vector is
131+
all-valid or all-null and carries no validity buffer (512 bytes per 4096 rows). Both are bytes the
132+
JVM allocated, so `jvm_arrow_allocated - jvm_arrow_imported` slightly understates the Arrow memory
133+
the JVM allocated itself. Treat it as a close lower bound rather than an exact split, and note that
134+
the two counters are separate reads of process-wide state, so concurrent tasks can change them
135+
between samples: they are not an atomic per-query balance, and neither is a measure of RSS.
128136

129137
## Definition of Labels
130138

131-
| Label | Meaning |
132-
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
133-
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
134-
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
135-
| jvm_arrow_allocated | Arrow memory held by Comet's JVM allocator, including buffers imported from native over the Arrow C Data Interface |
136-
| jvm_arrow_imported | The imported part of `jvm_arrow_allocated`. It is native memory, so it also counts in `native_allocated`; subtract it to get Arrow memory the JVM allocated itself. |
137-
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
138-
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
139-
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
139+
| Label | Meaning |
140+
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
141+
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
142+
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
143+
| jvm_arrow_allocated | Arrow memory held by Comet's JVM allocator, including buffers imported from native over the Arrow C Data Interface |
144+
| jvm_arrow_imported | Arrow memory held by the FFI import path, mostly buffers imported from native and so also counted in `native_allocated`. Includes a little JVM-allocated import overhead, so the subtraction is a lower bound. |
145+
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
146+
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
147+
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |

‎spark/src/main/java/org/apache/comet/udf/CometUdfBridge.java‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.arrow.memory.BufferAllocator;
2828
import org.apache.arrow.vector.FieldVector;
2929
import org.apache.arrow.vector.ValueVector;
30+
import org.apache.arrow.vector.util.TransferPair;
3031
import org.apache.spark.TaskContext;
3132
import org.apache.spark.comet.CometTaskContextShim;
3233
import org.apache.spark.util.TaskCompletionListener;
@@ -216,6 +217,7 @@ private static void evaluateInternal(
216217

217218
ValueVector[] inputs = new ValueVector[inputArrayPtrs.length];
218219
ValueVector result = null;
220+
ValueVector transferred = null;
219221
try {
220222
for (int i = 0; i < inputArrayPtrs.length; i++) {
221223
ArrowArray inArr = ArrowArray.wrap(inputArrayPtrs[i]);
@@ -235,9 +237,21 @@ private static void evaluateInternal(
235237
+ " rows, expected "
236238
+ numRows);
237239
}
240+
// The UDF may allocate its result from the allocator it found on its inputs, which is
241+
// the import allocator. Data.exportVector does not re-own the buffers, so the result
242+
// would stay charged there for as long as the export holds it and be reported as
243+
// imported native memory. TransferPair moves ownership without copying the payload.
244+
FieldVector toExport = (FieldVector) result;
245+
if (result.getAllocator() != allocator) {
246+
TransferPair transferPair = result.getTransferPair(allocator);
247+
transferPair.transfer();
248+
transferred = transferPair.getTo();
249+
toExport = (FieldVector) transferred;
250+
}
251+
238252
ArrowArray outArr = ArrowArray.wrap(outArrayPtr);
239253
ArrowSchema outSch = ArrowSchema.wrap(outSchemaPtr);
240-
Data.exportVector(allocator, (FieldVector) result, null, outArr, outSch);
254+
Data.exportVector(allocator, toExport, null, outArr, outSch);
241255
} finally {
242256
for (ValueVector v : inputs) {
243257
if (v != null) {
@@ -255,6 +269,13 @@ private static void evaluateInternal(
255269
// do not mask the original throwable
256270
}
257271
}
272+
if (transferred != null) {
273+
try {
274+
transferred.close();
275+
} catch (RuntimeException ignored) {
276+
// do not mask the original throwable
277+
}
278+
}
258279
}
259280
}
260281
}

‎spark/src/main/scala/org/apache/comet/Tracing.scala‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,10 @@ object Tracing {
2424
private val nativeLib = new Native
2525

2626
/**
27-
* Emits the Arrow memory counters for the JVM side. `jvm_arrow_imported` is the part of
28-
* `jvm_arrow_allocated` that came from native over the C Data Interface, so the difference is
29-
* the Arrow memory the JVM allocated itself. See [[CometArrowImportAllocator]].
27+
* Emits the Arrow memory counters for the JVM side. `jvm_arrow_imported` is what the FFI import
28+
* path holds, which is mostly buffers imported from native over the C Data Interface, so the
29+
* difference is a close lower bound on the Arrow memory the JVM allocated itself. See
30+
* [[CometArrowImportAllocator]] for the JVM-allocated bytes that also land there.
3031
*/
3132
def logArrowMemory(): Unit = {
3233
nativeLib.logMemoryUsage("jvm_arrow_allocated", CometArrowAllocator.getAllocatedMemory)

‎spark/src/main/scala/org/apache/comet/package.scala‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,16 @@ package object comet {
4040
*
4141
* Arrow charges an imported buffer to whichever allocator wraps it, so imports taken directly
4242
* against [[CometArrowAllocator]] are indistinguishable from buffers the JVM allocated itself.
43-
* Holding them in their own child keeps the two separable for tracing. The child reserves
44-
* nothing, so every byte still escalates to the parent and the root keeps reporting the total.
45-
* Like the root, it is never closed: imported buffers are reference counted and routinely
46-
* outlive the task that imported them.
43+
* Holding them in their own child keeps the two separable for tracing.
44+
*
45+
* What lands here is not exclusively foreign memory: Arrow's importer allocates the owning
46+
* `ArrowArray` struct from this allocator, and `BitVectorHelper.loadValidityBuffer` allocates a
47+
* validity bitmap here when an imported vector is all-valid or all-null and carries no validity
48+
* buffer. Both are small and both are bytes the JVM allocated, so treat this as what the import
49+
* path holds rather than as an exact foreign-byte count. The child reserves nothing, so every
50+
* byte still escalates to the parent and the root keeps reporting the total. Like the root, it
51+
* is never closed: imported buffers are reference counted and routinely outlive the task that
52+
* imported them.
4753
*/
4854
val CometArrowImportAllocator: BufferAllocator =
4955
CometArrowAllocator.newChildAllocator("comet-ffi-imports", 0, Long.MaxValue)

‎spark/src/test/scala/org/apache/comet/udf/CometUdfBridgeSuite.scala‎

Lines changed: 50 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,27 @@ object ImportAllocatorProbeUdf {
5353
@volatile var inputAllocator: Option[BufferAllocator] = None
5454
}
5555

56+
/**
57+
* A UDF that allocates its output from the allocator that owns its inputs.
58+
*
59+
* The `CometUDF` interface permits this, and nothing about it is wrong from the UDF's side. It
60+
* matters here because that allocator is the FFI import allocator, so without intervention the
61+
* output would be charged to it and counted as imported memory.
62+
*/
63+
class InputAllocatorOutputUdf extends CometUDF {
64+
override def evaluate(inputs: Array[ValueVector], numRows: Int): ValueVector = {
65+
val out = new IntVector("out", inputs.head.getAllocator)
66+
out.allocateNew(numRows)
67+
(0 until numRows).foreach(row => out.setSafe(row, 7))
68+
out.setValueCount(numRows)
69+
out
70+
}
71+
}
72+
5673
class CometUdfBridgeSuite extends AnyFunSuite {
5774

58-
test("evaluate imports its input vectors against the FFI import allocator") {
59-
val numRows = 4
75+
/** Exports a single-column batch and invokes the bridge against it. */
76+
private def withBridgeCall(udfClassName: String, numRows: Int)(check: () => Unit): Unit = {
6077
val col = new ConstantColumnVector(numRows, IntegerType)
6178
col.setInt(42)
6279
val batch = new ColumnarBatch(Array[ColumnVector](col), numRows)
@@ -68,7 +85,7 @@ class CometUdfBridgeSuite extends AnyFunSuite {
6885

6986
try {
7087
CometUdfBridge.evaluate(
71-
classOf[ImportAllocatorProbeUdf].getName,
88+
udfClassName,
7289
inputArrayAddrs,
7390
inputSchemaAddrs,
7491
outArrays(0).memoryAddress(),
@@ -77,11 +94,9 @@ class CometUdfBridgeSuite extends AnyFunSuite {
7794
null,
7895
null)
7996

80-
assert(
81-
ImportAllocatorProbeUdf.inputAllocator.contains(CometArrowImportAllocator),
82-
"the UDF's inputs were imported against " +
83-
s"${ImportAllocatorProbeUdf.inputAllocator.map(_.getName)}, so their bytes are not " +
84-
"reported as imported memory")
97+
// Checked before releasing the exported structs: the export keeps the result's buffers
98+
// alive, so whichever allocator owns them is still charged at this point.
99+
check()
85100
} finally {
86101
outArrays(0).release()
87102
outArrays(0).close()
@@ -92,4 +107,31 @@ class CometUdfBridgeSuite extends AnyFunSuite {
92107
nativeUtil.close()
93108
}
94109
}
110+
111+
test("evaluate imports its input vectors against the FFI import allocator") {
112+
withBridgeCall(classOf[ImportAllocatorProbeUdf].getName, 4) { () =>
113+
assert(
114+
ImportAllocatorProbeUdf.inputAllocator.contains(CometArrowImportAllocator),
115+
"the UDF's inputs were imported against " +
116+
s"${ImportAllocatorProbeUdf.inputAllocator.map(_.getName)}, so their bytes are not " +
117+
"reported as imported memory")
118+
}
119+
}
120+
121+
test("a UDF output allocated from the input allocator is not left charged as imported") {
122+
// The CometUDF interface lets a UDF allocate its result from inputs.head.getAllocator, which
123+
// is the import allocator. Data.exportVector does not re-own the buffers, so without a
124+
// transfer the output stays charged to the import allocator for as long as the export holds
125+
// it, and jvm_arrow_imported counts JVM-created bytes as imported native memory.
126+
val numRows = 4096
127+
val before = CometArrowImportAllocator.getAllocatedMemory
128+
129+
withBridgeCall(classOf[InputAllocatorOutputUdf].getName, numRows) { () =>
130+
val during = CometArrowImportAllocator.getAllocatedMemory
131+
assert(
132+
during - before < numRows.toLong * 4,
133+
s"the UDF's output is still charged to the import allocator ($before -> $during bytes, " +
134+
s"output is ${numRows * 4} bytes), so it would be reported as imported native memory")
135+
}
136+
}
95137
}

‎spark/src/test/scala/org/apache/comet/vector/NativeUtilSuite.scala‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,45 @@ class NativeUtilSuite extends CometTestBase {
411411
}
412412
}
413413

414+
test("the import allocator also holds the JVM-side cost of importing") {
415+
// Characterization, not an aspiration: Arrow's importer allocates the owning ArrowArray
416+
// struct from the import allocator (ArrayImporter calls ArrowArray.allocateNew(allocator)),
417+
// and loadValidityBuffer allocates a validity bitmap there when an imported vector is
418+
// all-valid and carries no validity buffer. So the import allocator holds more than the
419+
// foreign buffers, and jvm_arrow_allocated minus jvm_arrow_imported understates the JVM's own
420+
// Arrow memory by that much. Compared against every imported buffer, not just the data one,
421+
// so the excess measured here is JVM-allocated rather than the foreign validity buffer.
422+
val numRows = 4096
423+
val col = new ConstantColumnVector(numRows, IntegerType)
424+
col.setInt(42)
425+
val batch = new ColumnarBatch(Array[ColumnVector](col), numRows)
426+
427+
val nativeUtil = new NativeUtil
428+
var imported: ColumnarBatch = null
429+
val before = CometArrowImportAllocator.getAllocatedMemory
430+
try {
431+
val (arrayAddrs, schemaAddrs, _) = nativeUtil.exportBatchToAddresses(batch)
432+
val vectors =
433+
nativeUtil.importVector(
434+
arrayAddrs.map(ArrowArray.wrap),
435+
schemaAddrs.map(ArrowSchema.wrap))
436+
imported = new ColumnarBatch(vectors.toArray, numRows)
437+
438+
val charged = CometArrowImportAllocator.getAllocatedMemory - before
439+
val foreign = vectors.head.getValueVector.getBuffers(false).map(_.capacity()).sum
440+
assert(
441+
charged > foreign,
442+
s"expected the import allocator to hold more than the $foreign bytes of imported " +
443+
s"buffers, since the importer allocates its own ArrowArray struct there, but it held " +
444+
s"$charged bytes")
445+
} finally {
446+
if (imported != null) {
447+
imported.close()
448+
}
449+
nativeUtil.close()
450+
}
451+
}
452+
414453
test("Variant schema identity round-trips through native Arrow FFI") {
415454
val variantType = Utils.variantType.getOrElse {
416455
cancel("VariantType requires Spark 4.0+")

0 commit comments

Comments
 (0)