Skip to content

Commit 7ca27dc

Browse files
committed
fix: compare observed zero reservations in legacy traces
Gate the legacy comparison on whether an allocation sample and a reservation sample have been observed, not on whether their values are positive. A reservation recorded as zero is a real value, and allocation standing above it is the signal the tool exists to find: memory still held after the pool released it. The previous gate dropped those samples, and a trace whose only reservation sample was zero was reported as having no pool samples at all. The legacy warning said the excess was understated. That is only half of it now: summing inflates the total when a shared pool is counted on every referencing thread, and deflates it when a thread has not reported yet, so the excess can err either way. Say that instead. The doc's sample output predated the counter and its numbers no longer match what the tool prints, so replace it with real output from a trace that carries the counter, elided the way the per-thread list already was.
1 parent c23468e commit 7ca27dc

2 files changed

Lines changed: 39 additions & 31 deletions

File tree

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -84,49 +84,51 @@ summed to obtain it: a shared pool reports its full reservation on every thread
8484
summing multiplies it by the thread count. The allocation counter and the total are emitted back to back on
8585
one thread when a traced plan finishes executing, and the tool compares only samples paired that way, so a
8686
fresh allocation is never measured against a stale reservation. Traces recorded before that counter existed
87-
are still analyzed from the per-thread sum, and the tool warns that the total over-counts. The tool analyzes
87+
are still analyzed from the per-thread sum, and the tool warns that the sum is not the same measure. The tool
88+
analyzes
8889
`native_allocated` when the trace contains it, since that counts only what Rust code holds from the
8990
allocator, and otherwise falls back to `jemalloc_allocated`. The output names the counter it used. A trace
9091
with neither counter is rejected.
9192

92-
Sample output, here for a trace recorded before `comet_memory_reserved_total` existed:
93+
Sample output, with the violation table and the per-thread list elided:
9394

9495
```
9596
=== Comet Trace Memory Analysis ===
9697
97-
Counter events parsed: 193104
98-
Allocation counter: jemalloc_allocated
99-
Pool total source: sum of 8 per-thread counters
100-
WARNING: this trace predates comet_memory_reserved_total. Summing the per-thread counters
101-
over-counts: a task-shared pool reports its full reservation on every thread that
102-
references it, so the totals below are upper bounds and the excess is understated.
103-
Peak jemalloc_allocated: 3068.2 MB
104-
Peak pool total: 2864.6 MB
105-
Peak excess (jemalloc_allocated - pool): 364.6 MB
98+
Counter events parsed: 2946
99+
Allocation counter: native_allocated
100+
Pool total source: comet_memory_reserved_total (process-wide)
101+
Peak native_allocated: 395.9 MB
102+
Peak pool total: 250.9 MB
103+
Peak excess (native_allocated - pool): 171.2 MB
106104
107-
WARNING: jemalloc_allocated exceeded pool reservation at 138 sampled points:
105+
WARNING: native_allocated exceeded pool reservation at 87 sampled points:
108106
109-
Time (us) jemalloc_allocated pool_total excess
107+
Time (us) native_allocated pool_total excess
110108
------------------------------------------------------------------
111-
179578 210.8 MB 0.1 MB 210.7 MB
112-
429663 420.5 MB 145.1 MB 275.5 MB
113-
1304969 2122.5 MB 1797.2 MB 325.2 MB
114-
21974838 407.0 MB 42.3 MB 364.6 MB
115-
33543599 5.5 MB 0.1 MB 5.3 MB
109+
14662 19.0 MB 0.7 MB 18.3 MB
110+
109895 48.2 MB 22.4 MB 25.8 MB
111+
1623833 94.0 MB 71.4 MB 22.6 MB
112+
3315951 193.8 MB 167.8 MB 26.0 MB
113+
4107671 233.7 MB 199.9 MB 33.8 MB
114+
...
115+
5441547 389.3 MB 218.1 MB 171.2 MB
116+
6445070 235.6 MB 209.8 MB 25.8 MB
116117
117118
--- Final per-thread pool reservations ---
118119
119-
thread_60_comet_memory_reserved: 0.0 MB
120-
thread_95_comet_memory_reserved: 0.0 MB
121-
thread_96_comet_memory_reserved: 0.0 MB
120+
thread_60_comet_memory_reserved: 39.2 MB
121+
thread_61_comet_memory_reserved: 48.3 MB
122+
thread_62_comet_memory_reserved: 27.2 MB
122123
...
123124
124-
Total: 0.0 MB
125+
Total: 245.8 MB
125126
```
126127

127-
Some excess is expected (allocator metadata and fragmentation for `jemalloc_allocated`, and non-pool
128-
allocations like Arrow IPC buffers for either counter). Large or growing excess may indicate memory that is
129-
not being tracked by the pool.
128+
A steady excess is expected, since not every native allocation goes through a pool (Arrow IPC buffers, for
129+
instance), and `jemalloc_allocated` additionally includes the allocator's own metadata and fragmentation. An
130+
excess that grows over the run, as it does from 26 MB to 171 MB above, is the signal worth chasing: that is
131+
memory the pool is not accounting for.
130132

131133
## Definition of Labels
132134

‎native/common/src/bin/analyze_trace.rs‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -233,11 +233,15 @@ fn main() {
233233

234234
// Legacy association, for traces recorded before the process-wide total existed: compare
235235
// the latest allocation against the running per-thread sum after every counter event.
236-
// There is nothing to pair on in those traces, so this keeps them analyzable on the terms
237-
// the tool always used, over-count and all.
236+
// There is nothing to pair on in those traces, so this is the best they support.
237+
//
238+
// A comparison needs one sample of each side, not a positive one. A reservation that has
239+
// been observed at zero is a real value, and allocation standing above it is exactly the
240+
// signal worth finding: memory still held after the pool released it. Requiring a positive
241+
// total instead would drop those samples silently.
238242
let per_thread_sum: u64 = pool_by_thread.values().sum();
239243
legacy.observe_total(per_thread_sum);
240-
if latest_allocated > 0 && per_thread_sum > 0 {
244+
if source.is_some() && !pool_by_thread.is_empty() {
241245
legacy.compare(event.ts, latest_allocated, per_thread_sum);
242246
}
243247
}
@@ -270,9 +274,11 @@ fn main() {
270274
pool_by_thread.len()
271275
);
272276
println!(
273-
"WARNING: this trace predates {POOL_TOTAL_COUNTER}. Summing the per-thread counters\n\
274-
over-counts: a task-shared pool reports its full reservation on every thread that\n\
275-
references it, so the totals below are upper bounds and the excess is understated."
277+
"WARNING: this trace predates {POOL_TOTAL_COUNTER}, so the total below is the sum of\n\
278+
the per-thread counters. That is not the same measure: a shared pool reports its full\n\
279+
reservation on every thread referencing it, which inflates the total, while a thread\n\
280+
that has not reported yet contributes nothing, which deflates it. The excess below can\n\
281+
err in either direction."
276282
);
277283
}
278284
println!("Peak {source}: {}", format_bytes(peak_allocated));

0 commit comments

Comments
 (0)