Skip to content

Commit 51a4a21

Browse files
committed
Merge remote-tracking branch 'apache/main' into fix/analyze-trace-pool-total
# Conflicts: # docs/source/contributor-guide/tracing.md
2 parents 2923064 + 32fae06 commit 51a4a21

18 files changed

Lines changed: 499 additions & 168 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -539,8 +539,8 @@ jobs:
539539
org.apache.spark.CometTaskMemoryManagerSuite
540540
org.apache.spark.CometExecIteratorLifecycleSuite
541541
org.apache.spark.CometPluginsDefaultSuite
542-
org.apache.spark.CometPluginsNonOverrideSuite
543-
org.apache.spark.CometPluginsUnifiedModeOverrideSuite
542+
org.apache.spark.CometPluginsMemoryOverheadWarningSuite
543+
org.apache.spark.CometPluginsUnifiedModeSuite
544544
org.apache.comet.rules.CometScanRuleSuite
545545
org.apache.comet.rules.CometScanContribSuite
546546
org.apache.comet.rules.CometScanSchemeFallbackSuite
@@ -564,6 +564,7 @@ jobs:
564564
org.apache.spark.sql.comet.util.UtilsSuite
565565
org.apache.comet.vector.NativeUtilSuite
566566
org.apache.comet.vector.CometVectorUtilsSuite
567+
org.apache.comet.udf.CometUdfBridgeSuite
567568
org.apache.comet.objectstore.NativeConfigSuite
568569
org.apache.comet.serde.operator.CometIcebergNativeScanSuite
569570
org.apache.comet.serde.operator.CometNativeScanSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -187,8 +187,8 @@ jobs:
187187
org.apache.spark.CometTaskMemoryManagerSuite
188188
org.apache.spark.CometExecIteratorLifecycleSuite
189189
org.apache.spark.CometPluginsDefaultSuite
190-
org.apache.spark.CometPluginsNonOverrideSuite
191-
org.apache.spark.CometPluginsUnifiedModeOverrideSuite
190+
org.apache.spark.CometPluginsMemoryOverheadWarningSuite
191+
org.apache.spark.CometPluginsUnifiedModeSuite
192192
org.apache.comet.rules.CometScanRuleSuite
193193
org.apache.comet.rules.CometScanContribSuite
194194
org.apache.comet.rules.CometScanSchemeFallbackSuite
@@ -212,6 +212,7 @@ jobs:
212212
org.apache.spark.sql.comet.util.UtilsSuite
213213
org.apache.comet.vector.NativeUtilSuite
214214
org.apache.comet.vector.CometVectorUtilsSuite
215+
org.apache.comet.udf.CometUdfBridgeSuite
215216
org.apache.comet.objectstore.NativeConfigSuite
216217
org.apache.comet.serde.operator.CometIcebergNativeScanSuite
217218
org.apache.comet.serde.operator.CometNativeScanSuite

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,14 @@ off-heap bytes in container RSS that neither Spark's `TaskMemoryManager` nor Com
103103
pool sees. In practice the volume is modest, a batch at a time per stream, but there is no
104104
ceiling and no backpressure.
105105

106+
One further child, `CometArrowImportAllocator` (`comet-ffi-imports`), is what the Arrow C Data
107+
Interface import path allocates from, so that tracing can report those charges apart from the rest
108+
of Comet's Arrow memory. Unlike the others it is process-wide and never closed, because imported
109+
buffers are reference counted and routinely outlive the task that imported them. Its reservation
110+
is zero, so every byte still escalates to the root and the inventory above is unchanged by it.
111+
Being charged there is not the same as having been allocated there; see the scaladoc on the
112+
allocator before reading anything into the split.
113+
106114
**The JVM shuffle allocator is an ordinary Spark consumer.** `CometShuffleMemoryAllocator.getInstance`
107115
returns `CometUnifiedShuffleMemoryAllocator`, a Spark `MemoryConsumer` drawing from
108116
`spark.memory.offHeap.size`, so shuffle pages are arbitrated against Spark's other consumers in the

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,12 +133,38 @@ instance), and `jemalloc_allocated` additionally includes the allocator's own me
133133
excess that grows over the run, as it does from 26 MB to 171 MB above, is the signal worth chasing: that is
134134
memory the pool is not accounting for.
135135

136+
Arrow memory on the JVM side is reported separately, because it is off-heap and so invisible to
137+
`jvm_heap_used`. Comet imports batches from native over the Arrow C Data Interface, and Arrow
138+
charges a buffer to whichever allocator owns it, so those imports are taken against a dedicated
139+
child allocator and reported as `jvm_arrow_imported`, within the `jvm_arrow_allocated` total.
140+
141+
Both are allocator charges. They report what each allocator is accountable for, not where the bytes
142+
were allocated, and their difference is not a bound on the Arrow memory the JVM allocated itself.
143+
Ownership and allocation come apart in both directions:
144+
145+
- Bytes the JVM allocated get charged to the import allocator. Arrow's importer allocates the
146+
owning `ArrowArray` struct there, and `BitVectorHelper.loadValidityBuffer` allocates a validity
147+
bitmap there when an imported vector is all-valid or all-null and carries no validity buffer
148+
(512 bytes per 4096 rows).
149+
- Imported bytes get charged to the root. An ownership transfer re-parents a charge without moving
150+
the payload, so a vector that shares buffers with an import, such as a slice of a UDF input, can
151+
leave the root accountable for memory the producer allocated.
152+
153+
For the same reason neither counter is a count of unique physical bytes, and `jvm_arrow_imported`
154+
is not guaranteed to be included in `native_allocated`. Usually the producer is Rust and the bytes
155+
are counted in both, but a buffer that Comet exported to native and that native passed back by
156+
reference is imported without the Rust allocator ever having handed it out. Finally, the two
157+
counters are separate reads of process-wide state, so concurrent tasks can change them between
158+
samples: they are not an atomic per-query balance, and neither is a measure of RSS.
159+
136160
## Definition of Labels
137161

138162
| Label | Meaning |
139163
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
140164
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
141165
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
166+
| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator tree on the JVM, including buffers imported from native over the Arrow C Data Interface |
167+
| jvm_arrow_imported | Bytes charged to the Arrow C Data Interface import allocator, a subset of `jvm_arrow_allocated`. An allocator charge, not a measure of where the bytes were allocated; see above. |
142168
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
143169
| comet_memory_reserved_total | Total memory reserved across every live Comet memory pool, process-wide, whatever the configured pool type. Counts a pool shared between execution contexts once, so unlike the per-thread counters it can be compared directly against an allocation counter. |
144170
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. Do not sum these across threads: a shared pool reports its full reservation on every thread that references it. |

‎spark/src/main/java/org/apache/comet/udf/CometUdfBridge.java‎

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.arrow.memory.BufferAllocator;
2828
import org.apache.arrow.vector.FieldVector;
2929
import org.apache.arrow.vector.ValueVector;
30+
import org.apache.arrow.vector.util.TransferPair;
3031
import org.apache.spark.TaskContext;
3132
import org.apache.spark.comet.CometTaskContextShim;
3233
import org.apache.spark.util.TaskCompletionListener;
@@ -210,17 +211,26 @@ private static void evaluateInternal(
210211
assert udf != null : "reflective instantiation returned null for " + udfClassName;
211212

212213
BufferAllocator allocator = org.apache.comet.package$.MODULE$.CometArrowAllocator();
214+
// See CometArrowImportAllocator: inputs are imported against that allocator, so that tracing
215+
// can report the import path's charges apart from the rest of Comet's Arrow memory.
216+
BufferAllocator importAllocator = org.apache.comet.package$.MODULE$.CometArrowImportAllocator();
213217

214218
ValueVector[] inputs = new ValueVector[inputArrayPtrs.length];
215219
ValueVector result = null;
220+
// Whether the UDF handed back one of the vectors it was given. Such a result is closed by the
221+
// input loop below, so the result branch there must leave it alone.
222+
boolean resultIsInput = false;
223+
ValueVector transferred = null;
216224
try {
217225
for (int i = 0; i < inputArrayPtrs.length; i++) {
218226
ArrowArray inArr = ArrowArray.wrap(inputArrayPtrs[i]);
219227
ArrowSchema inSch = ArrowSchema.wrap(inputSchemaPtrs[i]);
220-
inputs[i] = Data.importVector(allocator, inArr, inSch, null);
228+
inputs[i] = Data.importVector(importAllocator, inArr, inSch, null);
221229
}
222230

223231
result = udf.evaluate(inputs, numRows);
232+
// Recorded before the checks below, so the invariant holds however this exits.
233+
resultIsInput = isOneOf(result, inputs);
224234
if (!(result instanceof FieldVector)) {
225235
throw new RuntimeException(
226236
"CometUDF.evaluate() must return a FieldVector, got: " + result.getClass().getName());
@@ -232,9 +242,27 @@ private static void evaluateInternal(
232242
+ " rows, expected "
233243
+ numRows);
234244
}
245+
// The UDF may allocate its result from the allocator it found on its inputs, which is
246+
// the import allocator. Data.exportVector does not re-own the buffers, so the result
247+
// would stay charged there for as long as the export holds it and be reported as
248+
// imported memory. TransferPair moves ownership without copying the payload.
249+
//
250+
// A result that *is* one of the inputs is left alone: those buffers were imported, so the
251+
// import allocator is the right place for them, and transferring would move a foreign
252+
// charge onto the root. The check is reference identity, so a result that merely shares
253+
// buffers with an input (a slice, say) is still transferred. That is a limit of allocator
254+
// accounting rather than something this can close; see CometArrowImportAllocator.
255+
FieldVector toExport = (FieldVector) result;
256+
if (!resultIsInput && result.getAllocator() != allocator) {
257+
TransferPair transferPair = result.getTransferPair(allocator);
258+
transferPair.transfer();
259+
transferred = transferPair.getTo();
260+
toExport = (FieldVector) transferred;
261+
}
262+
235263
ArrowArray outArr = ArrowArray.wrap(outArrayPtr);
236264
ArrowSchema outSch = ArrowSchema.wrap(outSchemaPtr);
237-
Data.exportVector(allocator, (FieldVector) result, null, outArr, outSch);
265+
Data.exportVector(allocator, toExport, null, outArr, outSch);
238266
} finally {
239267
for (ValueVector v : inputs) {
240268
if (v != null) {
@@ -245,13 +273,30 @@ private static void evaluateInternal(
245273
}
246274
}
247275
}
248-
if (result != null) {
276+
if (result != null && !resultIsInput) {
249277
try {
250278
result.close();
251279
} catch (RuntimeException ignored) {
252280
// do not mask the original throwable
253281
}
254282
}
283+
if (transferred != null) {
284+
try {
285+
transferred.close();
286+
} catch (RuntimeException ignored) {
287+
// do not mask the original throwable
288+
}
289+
}
290+
}
291+
}
292+
293+
/** Whether the UDF handed back one of the vectors it was given, rather than a new one. */
294+
private static boolean isOneOf(ValueVector result, ValueVector[] inputs) {
295+
for (ValueVector input : inputs) {
296+
if (result == input) {
297+
return true;
298+
}
255299
}
300+
return false;
256301
}
257302
}

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ object CometConf extends ShimCometConf {
4848
val COMPAT_GUIDE: String = "For more information, refer to the Comet Compatibility " +
4949
"Guide (https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html)"
5050

51-
private val TUNING_GUIDE = "For more information, refer to the Comet Tuning " +
51+
val TUNING_GUIDE: String = "For more information, refer to the Comet Tuning " +
5252
"Guide (https://datafusion.apache.org/comet/user-guide/latest/tuning.html)"
5353

5454
private val TRACING_GUIDE = "For more information, refer to the Comet Tracing " +

‎spark/src/main/scala/org/apache/comet/CometExecIterator.scala‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,7 @@ class CometExecIterator(
357357

358358
private def traceMemoryUsage(): Unit = {
359359
nativeLib.logMemoryUsage("jvm_heap_used", memoryMXBean.getHeapMemoryUsage.getUsed)
360+
Tracing.logArrowMemory()
360361
}
361362
}
362363

‎spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala‎

Lines changed: 2 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -247,27 +247,14 @@ object CometSparkSessionExtensions extends Logging {
247247
org.apache.spark.SPARK_VERSION >= "4.2"
248248
}
249249

250-
/**
251-
* Whether we should override Spark memory configuration for Comet. This only returns true when
252-
* Comet native execution is enabled and/or Comet shuffle is enabled and Comet doesn't use
253-
* off-heap mode (unified memory manager).
254-
*/
255-
def shouldOverrideMemoryConf(conf: SparkConf): Boolean = {
256-
val cometEnabled = getBooleanConf(conf, CometConf.COMET_ENABLED)
257-
val cometShuffleEnabled = getBooleanConf(conf, CometConf.COMET_SHUFFLE_ENABLED)
258-
val cometExecEnabled = getBooleanConf(conf, CometConf.COMET_EXEC_ENABLED)
259-
val offHeapMode = CometSparkSessionExtensions.isOffHeapEnabled(conf)
260-
cometEnabled && (cometShuffleEnabled || cometExecEnabled) && !offHeapMode
261-
}
262-
263250
/**
264251
* Determines required memory overhead in MB per executor process for Comet when running in
265252
* on-heap mode.
266253
*/
267254
def getCometMemoryOverheadInMiB(sparkConf: SparkConf): Long = {
268255
if (isOffHeapEnabled(sparkConf)) {
269-
// when running in off-heap mode we use unified memory management to share
270-
// off-heap memory with Spark so do not add overhead
256+
// off-heap mode sizes the native memory pool from spark.memory.offHeap.size instead
257+
// (see CometExecIterator.getMemoryConfig), so this value does not apply
271258
return 0
272259
}
273260
ConfigHelpers.byteFromString(
@@ -277,9 +264,6 @@ object CometSparkSessionExtensions extends Logging {
277264
ByteUnit.MiB)
278265
}
279266

280-
private def getBooleanConf(conf: SparkConf, entry: ConfigEntry[Boolean]) =
281-
conf.getBoolean(entry.key, entry.defaultValue.get)
282-
283267
/**
284268
* Calculates required memory overhead in bytes per executor process for Comet when running in
285269
* on-heap mode.

‎spark/src/main/scala/org/apache/comet/Tracing.scala‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,18 @@ object Tracing {
2323

2424
private val nativeLib = new Native
2525

26+
/**
27+
* Emits the Arrow memory counters for the JVM side: what Comet's Arrow allocator tree is
28+
* charged for in total, and how much of that the C Data Interface import path is charged for.
29+
*
30+
* Both are allocator charges rather than allocation origin, so their difference is not a bound
31+
* on the Arrow memory the JVM allocated itself. See [[CometArrowImportAllocator]].
32+
*/
33+
def logArrowMemory(): Unit = {
34+
nativeLib.logMemoryUsage("jvm_arrow_allocated", CometArrowAllocator.getAllocatedMemory)
35+
nativeLib.logMemoryUsage("jvm_arrow_imported", CometArrowImportAllocator.getAllocatedMemory)
36+
}
37+
2638
def withTrace[T](label: String, tracingEnabled: Boolean, fun: => T): T = {
2739
try {
2840
if (tracingEnabled) {

‎spark/src/main/scala/org/apache/comet/package.scala‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ package org.apache
2121

2222
import java.util.Properties
2323

24-
import org.apache.arrow.memory.RootAllocator
24+
import org.apache.arrow.memory.{BufferAllocator, RootAllocator}
2525
import org.apache.spark.internal.Logging
2626

2727
package object comet {
@@ -35,6 +35,30 @@ package object comet {
3535
*/
3636
val CometArrowAllocator = new RootAllocator(Long.MaxValue)
3737

38+
/**
39+
* The allocator that the Arrow C Data Interface import path allocates from.
40+
*
41+
* Arrow charges a buffer to whichever allocator owns it, so imports taken directly against
42+
* [[CometArrowAllocator]] are indistinguishable from buffers the JVM allocated itself. Giving
43+
* the import path its own child keeps the two separable for tracing. The child reserves
44+
* nothing, so every byte still escalates to the parent and the root keeps reporting the total.
45+
* Like the root, it is never closed: imported buffers are reference counted and routinely
46+
* outlive the task that imported them.
47+
*
48+
* What this counts is what the import path is charged for, not where the bytes were allocated.
49+
* Ownership and allocation come apart in both directions. Bytes the JVM allocated land here:
50+
* Arrow's importer allocates the owning `ArrowArray` struct from this allocator, and
51+
* `BitVectorHelper.loadValidityBuffer` allocates a validity bitmap here when an imported vector
52+
* is all-valid or all-null and carries no validity buffer. Imported bytes land elsewhere: an
53+
* ownership transfer re-parents a charge without moving the payload, so a vector that shares
54+
* buffers with an import can leave the root accountable for memory the producer allocated.
55+
*
56+
* So read this and the root's total as allocator charges. Their difference is not a bound on
57+
* the Arrow memory the JVM allocated itself, and neither is a count of unique physical bytes.
58+
*/
59+
val CometArrowImportAllocator: BufferAllocator =
60+
CometArrowAllocator.newChildAllocator("comet-ffi-imports", 0, Long.MaxValue)
61+
3862
/**
3963
* Provides access to build information about the Comet libraries. This will be used by the
4064
* benchmarking software to provide the source revision and repository. In addition, the build

0 commit comments

Comments
 (0)