Skip to content

Commit c23468e

Browse files
committed
fix: cover every pool type in the total, and pair trace samples
Addresses review feedback on the process-wide reservation counter. Source the total from the per-thread tracing registry instead of the task-shared one. The task-shared registry only holds pools created by `acquire_task_shared_pool`, so the `greedy`, `fair_spill`, the `_global` variants and `unbounded` modes reported zero while holding live reservations, and the analyzer then called those reservations untracked. Every plan registers whichever pool `create_memory_pool` gave it, so deduplicating that registry by pool identity covers all of them and still counts a shared pool once. That also removes the `Weak::upgrade` under the registry lock. An upgraded reference could become a pool's last owner if the owning task finished mid-scan, and `TaskSharedMemoryPool::drop` re-locks the registry, so the sampling thread would have deadlocked holding it. The thread registry holds strong references that the scan only borrows. In the analyzer, compare an allocation sample only against the pool total emitted next to it on the same thread. `executePlan` emits the two back to back, but a busy-poll reservation event in between meant the previous comparison paired a fresh allocation with a stale total and recorded excess that was never held. Traces without the total counter still take the legacy path, whose numbers are unchanged. A trace that carries the counter but never pairs a sample now says so rather than reporting OK.
1 parent 83cd46f commit c23468e

4 files changed

Lines changed: 256 additions & 168 deletions

File tree

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 25 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ Example trace visualization:
6666
## Analyzing Memory Usage
6767

6868
The `analyze_trace` tool parses a trace log and compares the process-wide native allocation counter against
69-
the sum of per-thread Comet memory pool reservations. This is useful for detecting untracked native memory
69+
the total memory reserved by Comet's memory pools. This is useful for detecting untracked native memory
7070
growth where native allocations exceed what the memory pools account for.
7171

7272
Build and run:
@@ -78,22 +78,28 @@ cargo run --bin analyze_trace -- /path/to/comet-event-trace.json
7878

7979
The tool reads counter events from the trace log. Because tracing logs metrics per thread, `native_allocated`
8080
and `jemalloc_allocated` are process-wide values (the same global allocation reported from whichever thread
81-
logs it). The total tracked memory comes from `comet_memory_reserved_total`, which counts each pool once
82-
process-wide. The per-thread `thread_NNN_comet_memory_reserved` values must not be summed to obtain it: a
83-
task-shared pool reports its full reservation on every thread that references it, so summing multiplies it by
84-
the thread count. Traces recorded before that counter existed are still analyzed from the per-thread sum, and
85-
the tool warns that the total over-counts. The tool analyzes `native_allocated` when the trace contains it,
86-
since that counts only what Rust code holds from the allocator, and otherwise falls back to
87-
`jemalloc_allocated`. The output names the counter it used. A trace with neither counter is rejected.
88-
89-
Sample output:
81+
logs it). The total tracked memory comes from `comet_memory_reserved_total`, which covers every pool type and
82+
counts each pool once process-wide. The per-thread `thread_NNN_comet_memory_reserved` values must not be
83+
summed to obtain it: a shared pool reports its full reservation on every thread that references it, so
84+
summing multiplies it by the thread count. The allocation counter and the total are emitted back to back on
85+
one thread when a traced plan finishes executing, and the tool compares only samples paired that way, so a
86+
fresh allocation is never measured against a stale reservation. Traces recorded before that counter existed
87+
are still analyzed from the per-thread sum, and the tool warns that the total over-counts. The tool analyzes
88+
`native_allocated` when the trace contains it, since that counts only what Rust code holds from the
89+
allocator, and otherwise falls back to `jemalloc_allocated`. The output names the counter it used. A trace
90+
with neither counter is rejected.
91+
92+
Sample output, here for a trace recorded before `comet_memory_reserved_total` existed:
9093

9194
```
9295
=== Comet Trace Memory Analysis ===
9396
9497
Counter events parsed: 193104
9598
Allocation counter: jemalloc_allocated
96-
Threads with memory pools: 8
99+
Pool total source: sum of 8 per-thread counters
100+
WARNING: this trace predates comet_memory_reserved_total. Summing the per-thread counters
101+
over-counts: a task-shared pool reports its full reservation on every thread that
102+
references it, so the totals below are upper bounds and the excess is understated.
97103
Peak jemalloc_allocated: 3068.2 MB
98104
Peak pool total: 2864.6 MB
99105
Peak excess (jemalloc_allocated - pool): 364.6 MB
@@ -124,11 +130,11 @@ not being tracked by the pool.
124130

125131
## Definition of Labels
126132

127-
| Label | Meaning |
128-
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
129-
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
130-
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
131-
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
132-
| comet_memory_reserved_total | Total memory reserved across every live Comet memory pool, process-wide. Counts each task-shared pool once, so unlike the per-thread counters it can be compared directly against an allocation counter. |
133-
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. Do not sum these across threads: a task-shared pool reports its full reservation on every thread that references it. |
134-
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
133+
| Label | Meaning |
134+
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
135+
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
136+
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
137+
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
138+
| comet_memory_reserved_total | Total memory reserved across every live Comet memory pool, process-wide, whatever the configured pool type. Counts a pool shared between execution contexts once, so unlike the per-thread counters it can be compared directly against an allocation counter. |
139+
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. Do not sum these across threads: a shared pool reports its full reservation on every thread that references it. |
140+
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |

‎native/common/src/bin/analyze_trace.rs‎

Lines changed: 106 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,9 @@
1616
// under the License.
1717

1818
//! Analyzes a Comet chrome trace event log (`comet-event-trace.json`) and
19-
//! compares the process-wide native allocation counter against the sum of
20-
//! per-thread Comet memory pool reservations. Reports any points where the
21-
//! allocated bytes exceed the total pool size.
19+
//! compares the process-wide native allocation counter against the total memory
20+
//! reserved by Comet's memory pools. Reports any points where the allocated
21+
//! bytes exceed the total pool size.
2222
//!
2323
//! Usage:
2424
//! cargo run --bin analyze_trace -- <path-to-comet-event-trace.json>
@@ -50,7 +50,6 @@ const POOL_TOTAL_COUNTER: &str = "comet_memory_reserved_total";
5050
struct TraceEvent {
5151
name: String,
5252
ph: String,
53-
#[allow(dead_code)]
5453
tid: u64,
5554
ts: u64,
5655
#[serde(default)]
@@ -64,6 +63,50 @@ struct MemorySnapshot {
6463
pool_total: u64,
6564
}
6665

66+
/// What one pool-total source says about the trace.
67+
///
68+
/// Both sources are accumulated in the same pass and one is reported, because whether the trace
69+
/// carries [`POOL_TOTAL_COUNTER`] is only known once a line containing it has been read.
70+
#[derive(Default)]
71+
struct Analysis {
72+
/// Highest pool total this source observed.
73+
peak_pool_total: u64,
74+
/// Largest `allocated - pool_total` over the comparisons this source could make.
75+
peak_excess: u64,
76+
/// A sample of the points where allocation exceeded the total.
77+
violations: Vec<MemorySnapshot>,
78+
/// How many comparisons this source made. Zero means it never produced a usable pair, which
79+
/// is worth reporting rather than passing off as "allocation never exceeded the total".
80+
comparisons: u64,
81+
}
82+
83+
impl Analysis {
84+
fn observe_total(&mut self, pool_total: u64) {
85+
self.peak_pool_total = self.peak_pool_total.max(pool_total);
86+
}
87+
88+
fn compare(&mut self, ts: u64, allocated: u64, pool_total: u64) {
89+
self.comparisons += 1;
90+
self.observe_total(pool_total);
91+
92+
let Some(excess) = allocated.checked_sub(pool_total).filter(|e| *e > 0) else {
93+
return;
94+
};
95+
self.peak_excess = self.peak_excess.max(excess);
96+
// Sample the violations rather than listing every one, but always keep the worst.
97+
if self.violations.is_empty()
98+
|| ts.saturating_sub(self.violations.last().unwrap().ts) > 1_000_000
99+
|| excess == self.peak_excess
100+
{
101+
self.violations.push(MemorySnapshot {
102+
ts,
103+
allocated,
104+
pool_total,
105+
});
106+
}
107+
}
108+
}
109+
67110
fn format_bytes(bytes: u64) -> String {
68111
const MB: f64 = 1024.0 * 1024.0;
69112
format!("{:.1} MB", bytes as f64 / MB)
@@ -83,17 +126,22 @@ fn main() {
83126
let mut source: Option<usize> = None;
84127
// Latest allocated value (global, not per-thread)
85128
let mut latest_allocated: u64 = 0;
86-
// Per-thread pool reservations: thread_NNN -> bytes. Only used when the trace lacks the
87-
// process-wide total, since summing these over-counts task-shared pools.
129+
// Per-thread pool reservations: thread_NNN -> bytes. Reported at the end, and summed as the
130+
// pool total only for traces that predate the process-wide counter.
88131
let mut pool_by_thread: HashMap<String, u64> = HashMap::new();
89-
// Process-wide pool total, when the trace carries it.
132+
// Whether the trace carries the process-wide total, and its last value.
90133
let mut pool_total_counter: Option<u64> = None;
91-
// Points where allocated exceeded pool total
92-
let mut violations: Vec<MemorySnapshot> = Vec::new();
134+
// The allocation sample most recently emitted on each thread, waiting for that thread's
135+
// matching pool total. `executePlan` emits the allocation counter and the process-wide total
136+
// back to back on one thread, so pairing them by thread compares two values sampled at the
137+
// same instant. Comparing the latest allocation against whatever total arrived last instead
138+
// pairs a fresh allocation with a stale reservation and invents excess that was never held.
139+
let mut allocated_awaiting_total: HashMap<u64, u64> = HashMap::new();
140+
// Comparisons against the process-wide counter, and against the legacy per-thread sum.
141+
let mut paired = Analysis::default();
142+
let mut legacy = Analysis::default();
93143
// Track peak values
94144
let mut peak_allocated: u64 = 0;
95-
let mut peak_pool_total: u64 = 0;
96-
let mut peak_excess: u64 = 0;
97145
let mut counter_events: u64 = 0;
98146

99147
// Each line is one JSON event, possibly with a trailing comma.
@@ -147,21 +195,29 @@ fn main() {
147195
source = Some(rank);
148196
latest_allocated = 0;
149197
peak_allocated = 0;
150-
peak_excess = 0;
151-
violations.clear();
198+
allocated_awaiting_total.clear();
199+
paired = Analysis::default();
200+
legacy = Analysis::default();
152201
}
153202
}
154203
if let Some(val) = event.args.get(&event.name) {
155204
latest_allocated = val.as_u64().unwrap_or(0);
156-
if latest_allocated > peak_allocated {
157-
peak_allocated = latest_allocated;
158-
}
205+
peak_allocated = peak_allocated.max(latest_allocated);
206+
allocated_awaiting_total.insert(event.tid, latest_allocated);
159207
}
160208
} else if event.name == POOL_TOTAL_COUNTER {
161209
// Must be matched before the per-thread branch below, whose `contains` check would
162210
// otherwise also match this name and fold the process-wide total into the map.
163211
if let Some(val) = event.args.get(&event.name) {
164-
pool_total_counter = Some(val.as_u64().unwrap_or(0));
212+
let pool_total = val.as_u64().unwrap_or(0);
213+
pool_total_counter = Some(pool_total);
214+
paired.observe_total(pool_total);
215+
// Only compare against the allocation sampled in this thread's own group. An
216+
// observed zero reservation is a real value that allocation can exceed, so a
217+
// paired zero is a genuine comparison, not a missing sample.
218+
if let Some(allocated) = allocated_awaiting_total.remove(&event.tid) {
219+
paired.compare(event.ts, allocated, pool_total);
220+
}
165221
}
166222
} else if event.name.contains("comet_memory_reserved") {
167223
// Name format: thread_NNN_comet_memory_reserved
@@ -175,31 +231,14 @@ fn main() {
175231
continue;
176232
}
177233

178-
// After each allocated or pool update, check the current state. A comparison needs one
179-
// sample of each side: an observed zero reservation is a real value that allocation can
180-
// exceed, so only the absence of any pool sample defers the check.
181-
let pool_total: u64 = pool_total_counter.unwrap_or_else(|| pool_by_thread.values().sum());
182-
if pool_total > peak_pool_total {
183-
peak_pool_total = pool_total;
184-
}
185-
186-
let have_pool_sample = pool_total_counter.is_some() || !pool_by_thread.is_empty();
187-
if source.is_some() && have_pool_sample && latest_allocated > pool_total {
188-
let excess = latest_allocated - pool_total;
189-
if excess > peak_excess {
190-
peak_excess = excess;
191-
}
192-
// Record violation (sample - don't record every single one)
193-
if violations.is_empty()
194-
|| event.ts.saturating_sub(violations.last().unwrap().ts) > 1_000_000
195-
|| excess == peak_excess
196-
{
197-
violations.push(MemorySnapshot {
198-
ts: event.ts,
199-
allocated: latest_allocated,
200-
pool_total,
201-
});
202-
}
234+
// Legacy association, for traces recorded before the process-wide total existed: compare
235+
// the latest allocation against the running per-thread sum after every counter event.
236+
// There is nothing to pair on in those traces, so this keeps them analyzable on the terms
237+
// the tool always used, over-count and all.
238+
let per_thread_sum: u64 = pool_by_thread.values().sum();
239+
legacy.observe_total(per_thread_sum);
240+
if latest_allocated > 0 && per_thread_sum > 0 {
241+
legacy.compare(event.ts, latest_allocated, per_thread_sum);
203242
}
204243
}
205244

@@ -212,11 +251,18 @@ fn main() {
212251
std::process::exit(1);
213252
};
214253

254+
let have_pool_total_counter = pool_total_counter.is_some();
255+
let analysis = if have_pool_total_counter {
256+
&paired
257+
} else {
258+
&legacy
259+
};
260+
215261
// Print summary
216262
println!("=== Comet Trace Memory Analysis ===\n");
217263
println!("Counter events parsed: {counter_events}");
218264
println!("Allocation counter: {source}");
219-
if pool_total_counter.is_some() {
265+
if have_pool_total_counter {
220266
println!("Pool total source: {POOL_TOTAL_COUNTER} (process-wide)");
221267
} else {
222268
println!(
@@ -232,31 +278,39 @@ fn main() {
232278
println!("Peak {source}: {}", format_bytes(peak_allocated));
233279
println!(
234280
"Peak pool total: {}",
235-
format_bytes(peak_pool_total)
281+
format_bytes(analysis.peak_pool_total)
236282
);
237283
println!(
238284
"Peak excess ({source} - pool): {}",
239-
format_bytes(peak_excess)
285+
format_bytes(analysis.peak_excess)
240286
);
241287
println!();
242288

243-
if pool_total_counter.is_none() && pool_by_thread.is_empty() {
244-
println!(
245-
"No pool reservation samples in the trace, so there is nothing to compare against."
246-
);
247-
} else if violations.is_empty() {
289+
if analysis.comparisons == 0 {
290+
if have_pool_total_counter {
291+
println!(
292+
"No allocation sample was emitted next to a {POOL_TOTAL_COUNTER} sample on the\n\
293+
same thread, so there was nothing to compare. The two are emitted together when\n\
294+
a traced plan finishes executing."
295+
);
296+
} else {
297+
println!(
298+
"No pool reservation samples in the trace, so there is nothing to compare against."
299+
);
300+
}
301+
} else if analysis.violations.is_empty() {
248302
println!("OK: {source} never exceeded the total pool reservation.");
249303
} else {
250304
println!(
251305
"WARNING: {source} exceeded pool reservation at {} sampled points:\n",
252-
violations.len()
306+
analysis.violations.len()
253307
);
254308
println!(
255309
"{:>14} {:>18} {:>14} {:>14}",
256310
"Time (us)", source, "pool_total", "excess"
257311
);
258312
println!("{}", "-".repeat(66));
259-
for snap in &violations {
313+
for snap in &analysis.violations {
260314
let excess = snap.allocated - snap.pool_total;
261315
println!(
262316
"{:>14} {:>18} {:>14} {:>14}",

0 commit comments

Comments
 (0)