Skip to content

Commit 6b504af

Browse files
committed
fix: count untraced plans' pools in the process-wide total
`createPlan` registered a plan's memory pool only when that plan had `spark.comet.tracing.enabled` set, but `tracing.enabled` is a session config and the allocation counter the total is compared against is process-wide. An executor running an untraced plan holding 100 MiB alongside a traced plan holding 20 MiB reported 20 MiB against 120 MiB of live reservations, and the analyzer called the difference untracked allocation. Register unconditionally; only the counter emission stays gated on tracing. Read both the per-thread and the process-wide total under one lock, so the two counters emitted as a pair describe the same instant, and so tracing takes the registry mutex once per sample rather than twice. Move the counter name to `datafusion_comet_common::tracing::POOL_TOTAL_METRIC` so the producer and `analyze_trace` cannot drift apart: renaming it on one side alone would silently put every new trace on the warned fallback path.
1 parent 090e944 commit 6b504af

4 files changed

Lines changed: 132 additions & 44 deletions

File tree

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -79,16 +79,19 @@ cargo run --bin analyze_trace -- /path/to/comet-event-trace.json
7979
The tool reads counter events from the trace log. Because tracing logs metrics per thread, `native_allocated`
8080
and `jemalloc_allocated` are process-wide values (the same global allocation reported from whichever thread
8181
logs it). The total tracked memory comes from `comet_memory_reserved_total`, which covers every pool type and
82-
counts each pool once process-wide. The per-thread `thread_NNN_comet_memory_reserved` values must not be
83-
summed to obtain it: a shared pool reports its full reservation on every thread that references it, so
84-
summing multiplies it by the thread count. The allocation counter and the total are emitted back to back on
85-
one thread when a traced plan finishes executing, and the tool compares only samples paired that way, so a
86-
fresh allocation is never measured against a stale reservation. Traces recorded before that counter existed
87-
are still analyzed from the per-thread sum, and the tool warns that the sum is not the same measure. The tool
88-
analyzes
89-
`native_allocated` when the trace contains it, since that counts only what Rust code holds from the
90-
allocator, and otherwise falls back to `jemalloc_allocated`. The output names the counter it used. A trace
91-
with neither counter is rejected.
82+
counts each pool once process-wide. Every plan registers its pool for this total, including plans running
83+
with tracing disabled, because the allocation counter it is compared against covers them too. The per-thread
84+
`thread_NNN_comet_memory_reserved` values must not be summed to obtain it: a shared pool reports its full
85+
reservation on every thread that references it, so summing multiplies it by the thread count.
86+
87+
The allocation counter and the total are emitted back to back on one thread when a traced plan finishes
88+
executing, and the tool compares only samples paired that way, so a fresh allocation is never measured
89+
against a stale reservation. Traces recorded before that counter existed are still analyzed from the
90+
per-thread sum, and the tool warns that the sum is not the same measure.
91+
92+
The tool analyzes `native_allocated` when the trace contains it, since that counts only what Rust code holds
93+
from the allocator, and otherwise falls back to `jemalloc_allocated`. The output names the counter it used. A
94+
trace with neither counter is rejected.
9295

9396
Sample output, with the violation table and the per-thread list elided:
9497

‎native/common/src/bin/analyze_trace.rs‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
//! Usage:
2424
//! cargo run --bin analyze_trace -- <path-to-comet-event-trace.json>
2525
26+
use datafusion_comet_common::tracing::POOL_TOTAL_METRIC as POOL_TOTAL_COUNTER;
2627
use serde::Deserialize;
2728
use std::collections::HashMap;
2829
use std::io::{BufRead, BufReader};
@@ -37,13 +38,13 @@ use std::{env, fs::File};
3738
/// analyzed.
3839
const ALLOCATED_COUNTERS: [&str; 2] = ["native_allocated", "jemalloc_allocated"];
3940

40-
/// The process-wide total of Comet's memory pool reservations.
41-
///
42-
/// Preferred over summing the per-thread `thread_NNN_comet_memory_reserved` counters. Those report
43-
/// the full reservation of a task-shared pool once per thread that references it, so adding them
44-
/// across threads multiplies a shared pool by its thread count. A trace without this counter is
45-
/// still analyzed from the per-thread sum, with a warning, so older traces remain readable.
46-
const POOL_TOTAL_COUNTER: &str = "comet_memory_reserved_total";
41+
// The process-wide total of Comet's memory pool reservations, imported from the producer so the
42+
// two cannot drift apart.
43+
//
44+
// Preferred over summing the per-thread `thread_NNN_comet_memory_reserved` counters. Those report
45+
// the full reservation of a task-shared pool once per thread that references it, so adding them
46+
// across threads multiplies a shared pool by its thread count. A trace without this counter is
47+
// still analyzed from the per-thread sum, with a warning, so older traces remain readable.
4748

4849
/// A single Chrome trace event (only the fields we care about).
4950
#[derive(Deserialize)]

‎native/common/src/tracing.rs‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,13 @@ pub fn log_memory_usage(name: &str, value: u64) {
110110
RECORDER.log_memory_usage(name, value);
111111
}
112112

113+
/// Counter name for the process-wide total of Comet's memory pool reservations.
114+
///
115+
/// Shared between the producer in `jni_api` and `analyze_trace`, which recognises this exact name
116+
/// and falls back to a warned, over-counting estimate when a trace does not carry it. Renaming it
117+
/// in one place only would silently put every new trace on the fallback path.
118+
pub const POOL_TOTAL_METRIC: &str = "comet_memory_reserved_total";
119+
113120
pub fn with_trace<T, F>(label: &str, tracing_enabled: bool, f: F) -> T
114121
where
115122
F: FnOnce() -> T,

‎native/core/src/execution/jni_api.rs‎

Lines changed: 104 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ use crate::execution::shuffle::{
108108
use crate::execution::spark_plan::SparkPlan;
109109

110110
use crate::execution::tracing::{
111-
get_thread_id, log_memory_usage, trace_begin, trace_end, with_trace,
111+
get_thread_id, log_memory_usage, trace_begin, trace_end, with_trace, POOL_TOTAL_METRIC,
112112
};
113113

114114
use crate::execution::memory_pools::logging_pool::LoggingMemoryPool;
@@ -232,21 +232,52 @@ fn total_reserved_for_thread(thread_id: u64) -> usize {
232232
.unwrap_or(0)
233233
}
234234

235-
/// Total bytes reserved across every pool registered for tracing, process-wide.
235+
/// Reservation totals read from the pool registry in one pass, under one lock.
236+
struct ReservedTotals {
237+
/// Bytes reserved by the pools registered on the requested thread, deduplicated within it.
238+
/// Zero when no thread was requested.
239+
for_thread: usize,
240+
/// Bytes reserved across every live Comet memory pool in the process.
241+
///
242+
/// This is the figure to compare against a process-wide allocation counter, so it has to
243+
/// account for every live pool rather than only the ones a traced plan created. Allocation is
244+
/// process-wide and a plan running with tracing off still holds memory, which is why
245+
/// `createPlan` registers its pool unconditionally: with two concurrent plans configured
246+
/// differently, counting only the traced one would report the untraced plan's reservation as
247+
/// untracked allocation. It covers every pool type, because each plan registers whichever pool
248+
/// `create_memory_pool` gave it, and it counts a pool once however many contexts or threads
249+
/// hold it.
250+
///
251+
/// The per-thread `thread_NNN_comet_memory_reserved` counters must not be summed to obtain it:
252+
/// a shared pool reports its full reservation on every thread that references it, so adding
253+
/// them across threads multiplies that pool by its thread count.
254+
across_threads: usize,
255+
}
256+
257+
/// Reads both totals under a single lock.
236258
///
237-
/// This is the figure to compare against a process-wide allocation counter. It covers every pool
238-
/// type, because each plan registers whichever pool `create_memory_pool` gave it, and it counts a
239-
/// pool once however many contexts or threads hold it.
259+
/// They are emitted as a pair, and a pair that straddles two acquisitions describes two different
260+
/// instants. Taking the lock once also halves the tracing traffic through a mutex the executor
261+
/// needs in order to register and release pools.
240262
///
241-
/// The per-thread `thread_NNN_comet_memory_reserved` counters must not be summed to obtain it: a
242-
/// shared pool reports its full reservation on every thread that references it, so adding them
243-
/// across threads multiplies that pool by its thread count.
244-
fn total_reserved_across_threads() -> usize {
263+
/// `thread_id` of `None` skips the per-thread figure; the caller is only after the process total.
264+
fn total_reserved(thread_id: Option<u64>) -> ReservedTotals {
245265
let map = get_thread_memory_pools().lock();
266+
let for_thread = thread_id
267+
.and_then(|id| map.get(&id))
268+
.map(|pools| sum_distinct_pools(pools.values(), &mut HashSet::new()))
269+
.unwrap_or(0);
270+
// Deduplicated across the whole map, not within each thread: the same pool is registered by
271+
// every context that holds it, and a task-shared or global pool spans threads.
246272
let mut seen = HashSet::new();
247-
map.values()
273+
let across_threads = map
274+
.values()
248275
.map(|pools| sum_distinct_pools(pools.values(), &mut seen))
249-
.sum()
276+
.sum();
277+
ReservedTotals {
278+
for_thread,
279+
across_threads,
280+
}
250281
}
251282

252283
fn parse_usize_env_var(name: &str) -> Option<usize> {
@@ -551,10 +582,18 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan(
551582

552583
// Register the shared base pool before wrapping it for per-plan debug logging. The
553584
// guard removes the entry if any later plan setup step fails.
585+
//
586+
// Registration is not conditional on this plan's tracing setting. `tracing.enabled` is
587+
// a session config, so an executor can run a traced and an untraced plan at once,
588+
// while the allocation counter a trace compares against is process-wide. Registering
589+
// only traced plans would leave the untraced plan's reservation out of the total and
590+
// report it as allocation held outside any pool.
554591
let rust_thread_id = get_thread_id();
555-
let memory_pool_registration = tracing_enabled.then(|| {
556-
ThreadMemoryPoolRegistration::new(rust_thread_id, id, Arc::clone(&memory_pool))
557-
});
592+
let memory_pool_registration = Some(ThreadMemoryPoolRegistration::new(
593+
rust_thread_id,
594+
id,
595+
Arc::clone(&memory_pool),
596+
));
558597

559598
let memory_pool = if logging_memory_pool {
560599
Arc::new(LoggingMemoryPool::new(task_attempt_id as u64, memory_pool))
@@ -1126,17 +1165,17 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan(
11261165
log_jemalloc_usage();
11271166
#[cfg(feature = "alloc-accounting")]
11281167
log_native_allocated();
1168+
// Both totals come from one read of the registry, so the pair describes a single
1169+
// instant, and both are emitted next to the allocation counter above so a trace can
1170+
// compare them. The per-thread counter cannot be summed across threads to obtain the
1171+
// process-wide one: it reports a shared pool's full reservation once per referencing
1172+
// thread.
1173+
let totals = total_reserved(Some(exec_context.rust_thread_id));
11291174
log_memory_usage(
11301175
&exec_context.tracing_memory_metric_name,
1131-
total_reserved_for_thread(exec_context.rust_thread_id) as u64,
1132-
);
1133-
// Process-wide total, emitted here so it is true at the same instant as the
1134-
// allocation counter above. The per-thread counter cannot be summed across threads:
1135-
// it reports a shared pool's full reservation once per referencing thread.
1136-
log_memory_usage(
1137-
"comet_memory_reserved_total",
1138-
total_reserved_across_threads() as u64,
1176+
totals.for_thread as u64,
11391177
);
1178+
log_memory_usage(POOL_TOTAL_METRIC, totals.across_threads as u64);
11401179
}
11411180

11421181
result
@@ -1165,13 +1204,16 @@ pub extern "system" fn Java_org_apache_comet_Native_releasePlan(
11651204
let mut execution_context: Box<ExecutionContext> =
11661205
Box::from_raw(exec_context as *mut ExecutionContext);
11671206

1168-
// Unregister this context's pool and emit the remaining total for the thread
1207+
// Unregister this context's pool and, when tracing, emit the remaining total for the
1208+
// thread. Every context registers, but only a traced one writes counters.
11691209
if let Some(memory_pool_registration) = execution_context.memory_pool_registration.take() {
11701210
let remaining = memory_pool_registration.unregister_and_total();
1171-
log_memory_usage(
1172-
&execution_context.tracing_memory_metric_name,
1173-
remaining as u64,
1174-
);
1211+
if execution_context.tracing_enabled {
1212+
log_memory_usage(
1213+
&execution_context.tracing_memory_metric_name,
1214+
remaining as u64,
1215+
);
1216+
}
11751217
}
11761218

11771219
// Flush metrics last, as it is the only fallible step here.
@@ -1920,6 +1962,10 @@ mod tests {
19201962
(pool, reservation)
19211963
}
19221964

1965+
fn total_reserved_across_threads() -> usize {
1966+
total_reserved(None).across_threads
1967+
}
1968+
19231969
/// The property that makes this total comparable against a process-wide allocation counter,
19241970
/// and the one summing the per-thread counters gets wrong: a pool shared by several contexts
19251971
/// on several threads contributes its reservation once.
@@ -1986,6 +2032,37 @@ mod tests {
19862032
drop(reservation);
19872033
}
19882034

2035+
/// What the traced thread sees when a plan on another thread holds memory it knows nothing
2036+
/// about, which is the case a tracing-gated registry got wrong: `tracing.enabled` is a session
2037+
/// config, so an untraced plan can run alongside a traced one, and its reservation is part of
2038+
/// the process-wide allocation the trace compares against. The per-thread figure stays local —
2039+
/// that counter is for attribution — while the process total has to include the other thread.
2040+
#[test]
2041+
fn the_process_total_includes_pools_registered_on_other_threads() {
2042+
let _guard = serial();
2043+
let before = total_reserved_across_threads();
2044+
let (traced_pool, traced_reservation) = reserving(20 * 1024 * 1024);
2045+
let (untraced_pool, untraced_reservation) = reserving(100 * 1024 * 1024);
2046+
2047+
let _traced = ThreadMemoryPoolRegistration::new(15, 1, traced_pool);
2048+
let _untraced = ThreadMemoryPoolRegistration::new(16, 2, untraced_pool);
2049+
2050+
let totals = total_reserved(Some(15));
2051+
assert_eq!(
2052+
totals.for_thread,
2053+
20 * 1024 * 1024,
2054+
"the per-thread counter attributes only this thread's pools"
2055+
);
2056+
assert_eq!(
2057+
totals.across_threads - before,
2058+
120 * 1024 * 1024,
2059+
"the process total must cover the pool held by the other thread"
2060+
);
2061+
2062+
drop(traced_reservation);
2063+
drop(untraced_reservation);
2064+
}
2065+
19892066
#[test]
19902067
fn length_resolves_to_spark_length_for_string_and_binary() {
19912068
use datafusion::physical_expr::expressions::{CastExpr, Column};

0 commit comments

Comments
 (0)