Skip to content

Commit 5a4334a

Browse files
committed
fix: compare against observed zero reservations in analyze_trace and format tracing.md
A zero pool reservation is a real sample that allocation can exceed, so the analyzer now defers the comparison only until it has seen at least one pool sample, and says so when it never does instead of reporting that allocation never exceeded reservations. Reflow the tracing.md label table with prettier, which the Preflight check enforces.
1 parent 4b71ad9 commit 5a4334a

2 files changed

Lines changed: 16 additions & 10 deletions

File tree

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -121,10 +121,10 @@ not being tracked by the pool.
121121

122122
## Definition of Labels
123123

124-
| Label | Meaning |
125-
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
126-
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
127-
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
128-
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate: each live thread holds up to 64 KiB of un-flushed delta, so the value can lag the true balance by that much per thread. |
129-
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
130-
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |
124+
| Label | Meaning |
125+
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
126+
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
127+
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
128+
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate to within 64 KiB of un-flushed delta per live thread. |
129+
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
130+
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |

‎native/common/src/bin/analyze_trace.rs‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -158,13 +158,15 @@ fn main() {
158158
continue;
159159
}
160160

161-
// After each allocated or pool update, check the current state
161+
// After each allocated or pool update, check the current state. A comparison needs one
162+
// sample of each side: an observed zero reservation is a real value that allocation can
163+
// exceed, so only the absence of any pool sample defers the check.
162164
let pool_total: u64 = pool_by_thread.values().sum();
163165
if pool_total > peak_pool_total {
164166
peak_pool_total = pool_total;
165167
}
166168

167-
if latest_allocated > 0 && pool_total > 0 && latest_allocated > pool_total {
169+
if source.is_some() && !pool_by_thread.is_empty() && latest_allocated > pool_total {
168170
let excess = latest_allocated - pool_total;
169171
if excess > peak_excess {
170172
peak_excess = excess;
@@ -208,7 +210,11 @@ fn main() {
208210
);
209211
println!();
210212

211-
if violations.is_empty() {
213+
if pool_by_thread.is_empty() {
214+
println!(
215+
"No pool reservation samples in the trace, so there is nothing to compare against."
216+
);
217+
} else if violations.is_empty() {
212218
println!("OK: {source} never exceeded the total pool reservation.");
213219
} else {
214220
println!(

0 commit comments

Comments
 (0)