Skip to content

Commit 4b71ad9

Browse files
committed
fix: teach analyze_trace about native_allocated and cover the accounting feature in CI
Address the third review round on the alloc-accounting feature. - analyze_trace only matched jemalloc_allocated, so a trace from a build with alloc-accounting and no jemalloc reported zero allocated and no excess. It now analyzes native_allocated when present, falls back to jemalloc_allocated otherwise, names the counter in its output, and rejects a trace with neither. tracing.md describes the selection. - Nothing in CI compiled the feature. The rust-test action now lints the jemalloc,alloc-accounting build with --all-targets, runs the accounting tests with the wrapper installed, and checks the system-allocator arm. - lib.rs: the three backend modules only name the type; the two global_allocator statics sit together, so the default build now installs System explicitly. ALLOCATOR_BACKEND is doc(hidden). - alloc_accounting.rs: inline the clamp, table-drive the settle tests, add a realloc test that pins the size-difference accounting and the after-delegating order, and confine the thread-exit test to the default build, where a missing destructor is actually caught. Trim the prose. - alloc_overhead bench: drop the redundant extern crate, run both liveness guards from a single Once that every bench function calls, merge the two alloc/free loops, and use iter instead of iter_batched. Benchmark IDs are unchanged.
1 parent ac51c5c commit 4b71ad9

6 files changed

Lines changed: 339 additions & 292 deletions

File tree

‎.github/actions/rust-test/action.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,3 +70,16 @@ runs:
7070
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
7171
RUST_BACKTRACE=1 cargo nextest run
7272
73+
# The `alloc-accounting` feature is off by default, so nothing else in CI compiles the
74+
# allocator wrapper, its backend selection, or the benchmark's liveness guards. Lint them with
75+
# the accounting wrapper over jemalloc, run the accounting tests with the wrapper installed, and
76+
# check the system-allocator arm.
77+
- name: Check and test the alloc-accounting feature
78+
shell: bash
79+
run: |
80+
cd native
81+
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
82+
cargo clippy --color=never -p datafusion-comet --all-targets --features jemalloc,alloc-accounting -- -D warnings
83+
RUST_BACKTRACE=1 cargo nextest run -p datafusion-comet --lib --features jemalloc,alloc-accounting alloc_accounting
84+
cargo check -p datafusion-comet --features alloc-accounting
85+

‎docs/source/contributor-guide/tracing.md‎

Lines changed: 24 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,9 @@ Example trace visualization:
6565

6666
## Analyzing Memory Usage
6767

68-
The `analyze_trace` tool parses a trace log and compares jemalloc usage against the sum of per-thread
69-
Comet memory pool reservations. This is useful for detecting untracked native memory growth where jemalloc
70-
allocations exceed what the memory pools account for.
68+
The `analyze_trace` tool parses a trace log and compares the process-wide native allocation counter against
69+
the sum of per-thread Comet memory pool reservations. This is useful for detecting untracked native memory
70+
growth where native allocations exceed what the memory pools account for.
7171

7272
Build and run:
7373

@@ -76,31 +76,34 @@ cd native
7676
cargo run --bin analyze_trace -- /path/to/comet-event-trace.json
7777
```
7878

79-
The tool reads counter events from the trace log. Because tracing logs metrics per thread, `jemalloc_allocated`
80-
is a process-wide value (the same global allocation reported from whichever thread logs it), while
81-
`thread_NNN_comet_memory_reserved` values are per-thread pool reservations that are summed to get the total
82-
tracked memory.
79+
The tool reads counter events from the trace log. Because tracing logs metrics per thread, `native_allocated`
80+
and `jemalloc_allocated` are process-wide values (the same global allocation reported from whichever thread
81+
logs it), while `thread_NNN_comet_memory_reserved` values are per-thread pool reservations that are summed to
82+
get the total tracked memory. The tool analyzes `native_allocated` when the trace contains it, since that
83+
counts only what Rust code holds from the allocator, and otherwise falls back to `jemalloc_allocated`. The
84+
output names the counter it used. A trace with neither counter is rejected.
8385

8486
Sample output:
8587

8688
```
8789
=== Comet Trace Memory Analysis ===
8890
8991
Counter events parsed: 193104
92+
Allocation counter: jemalloc_allocated
9093
Threads with memory pools: 8
91-
Peak jemalloc allocated: 3068.2 MB
94+
Peak jemalloc_allocated: 3068.2 MB
9295
Peak pool total: 2864.6 MB
93-
Peak excess (jemalloc - pool): 364.6 MB
96+
Peak excess (jemalloc_allocated - pool): 364.6 MB
9497
95-
WARNING: jemalloc exceeded pool reservation at 138 sampled points:
98+
WARNING: jemalloc_allocated exceeded pool reservation at 138 sampled points:
9699
97-
Time (us) jemalloc pool_total excess
98-
--------------------------------------------------------------
99-
179578 210.8 MB 0.1 MB 210.7 MB
100-
429663 420.5 MB 145.1 MB 275.5 MB
101-
1304969 2122.5 MB 1797.2 MB 325.2 MB
102-
21974838 407.0 MB 42.3 MB 364.6 MB
103-
33543599 5.5 MB 0.1 MB 5.3 MB
100+
Time (us) jemalloc_allocated pool_total excess
101+
------------------------------------------------------------------
102+
179578 210.8 MB 0.1 MB 210.7 MB
103+
429663 420.5 MB 145.1 MB 275.5 MB
104+
1304969 2122.5 MB 1797.2 MB 325.2 MB
105+
21974838 407.0 MB 42.3 MB 364.6 MB
106+
33543599 5.5 MB 0.1 MB 5.3 MB
104107
105108
--- Final per-thread pool reservations ---
106109
@@ -112,15 +115,16 @@ WARNING: jemalloc exceeded pool reservation at 138 sampled points:
112115
Total: 0.0 MB
113116
```
114117

115-
Some excess is expected (jemalloc metadata, fragmentation, non-pool allocations like Arrow IPC buffers).
116-
Large or growing excess may indicate memory that is not being tracked by the pool.
118+
Some excess is expected (allocator metadata and fragmentation for `jemalloc_allocated`, and non-pool
119+
allocations like Arrow IPC buffers for either counter). Large or growing excess may indicate memory that is
120+
not being tracked by the pool.
117121

118122
## Definition of Labels
119123

120124
| Label | Meaning |
121125
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
122126
| jvm_heap_used | JVM heap memory usage of live objects for the executor process |
123127
| jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) |
124-
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature) |
128+
| native_allocated | Bytes handed out by the Rust global allocator, process-wide (requires `alloc-accounting` feature). Approximate: each live thread holds up to 64 KiB of un-flushed delta, so the value can lag the true balance by that much per thread. |
125129
| thread_NNN_comet_memory_reserved | Memory reserved by Comet's DataFusion memory pool (summed across all contexts on the thread). NNN is the Rust thread ID. |
126130
| thread_NNN_comet_jvm_shuffle | Off-heap memory allocated by Comet for columnar shuffle. NNN is the Rust thread ID. |

‎native/common/src/bin/analyze_trace.rs‎

Lines changed: 65 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@
1616
// under the License.
1717

1818
//! Analyzes a Comet chrome trace event log (`comet-event-trace.json`) and
19-
//! compares jemalloc usage against the sum of per-thread Comet memory pool
20-
//! reservations. Reports any points where jemalloc exceeds the total pool size.
19+
//! compares the process-wide native allocation counter against the sum of
20+
//! per-thread Comet memory pool reservations. Reports any points where the
21+
//! allocated bytes exceed the total pool size.
2122
//!
2223
//! Usage:
2324
//! cargo run --bin analyze_trace -- <path-to-comet-event-trace.json>
@@ -27,6 +28,15 @@ use std::collections::HashMap;
2728
use std::io::{BufRead, BufReader};
2829
use std::{env, fs::File};
2930

31+
/// The process-wide allocation counters the tool understands, most preferred first.
32+
///
33+
/// `native_allocated` (the `alloc-accounting` feature) counts only the bytes Rust code holds from
34+
/// the global allocator, so it is the tighter comparison against pool reservations.
35+
/// `jemalloc_allocated` (the `jemalloc` feature) also includes jemalloc's own metadata. A trace
36+
/// that carries both is analyzed against `native_allocated` alone; a trace with neither cannot be
37+
/// analyzed.
38+
const ALLOCATED_COUNTERS: [&str; 2] = ["native_allocated", "jemalloc_allocated"];
39+
3040
/// A single Chrome trace event (only the fields we care about).
3141
#[derive(Deserialize)]
3242
struct TraceEvent {
@@ -42,7 +52,7 @@ struct TraceEvent {
4252
/// Snapshot of memory state at a given timestamp.
4353
struct MemorySnapshot {
4454
ts: u64,
45-
jemalloc: u64,
55+
allocated: u64,
4656
pool_total: u64,
4757
}
4858

@@ -61,14 +71,16 @@ fn main() {
6171
let file = File::open(&args[1]).expect("Failed to open trace file");
6272
let reader = BufReader::new(file);
6373

64-
// Latest jemalloc value (global, not per-thread)
65-
let mut latest_jemalloc: u64 = 0;
74+
// Index into ALLOCATED_COUNTERS of the counter being analyzed, once one has been seen
75+
let mut source: Option<usize> = None;
76+
// Latest allocated value (global, not per-thread)
77+
let mut latest_allocated: u64 = 0;
6678
// Per-thread pool reservations: thread_NNN -> bytes
6779
let mut pool_by_thread: HashMap<String, u64> = HashMap::new();
68-
// Points where jemalloc exceeded pool total
80+
// Points where allocated exceeded pool total
6981
let mut violations: Vec<MemorySnapshot> = Vec::new();
7082
// Track peak values
71-
let mut peak_jemalloc: u64 = 0;
83+
let mut peak_allocated: u64 = 0;
7284
let mut peak_pool_total: u64 = 0;
7385
let mut peak_excess: u64 = 0;
7486
let mut counter_events: u64 = 0;
@@ -110,11 +122,28 @@ fn main() {
110122

111123
counter_events += 1;
112124

113-
if event.name == "jemalloc_allocated" {
114-
if let Some(val) = event.args.get("jemalloc_allocated") {
115-
latest_jemalloc = val.as_u64().unwrap_or(0);
116-
if latest_jemalloc > peak_jemalloc {
117-
peak_jemalloc = latest_jemalloc;
125+
if let Some(rank) = ALLOCATED_COUNTERS
126+
.iter()
127+
.position(|name| *name == event.name)
128+
{
129+
match source {
130+
// A preferred counter is present in this trace; ignore the other one.
131+
Some(current) if current < rank => continue,
132+
Some(current) if current == rank => {}
133+
// First sighting of a more preferred counter. Start over so the peaks and
134+
// violations reported all come from a single source.
135+
_ => {
136+
source = Some(rank);
137+
latest_allocated = 0;
138+
peak_allocated = 0;
139+
peak_excess = 0;
140+
violations.clear();
141+
}
142+
}
143+
if let Some(val) = event.args.get(&event.name) {
144+
latest_allocated = val.as_u64().unwrap_or(0);
145+
if latest_allocated > peak_allocated {
146+
peak_allocated = latest_allocated;
118147
}
119148
}
120149
} else if event.name.contains("comet_memory_reserved") {
@@ -129,14 +158,14 @@ fn main() {
129158
continue;
130159
}
131160

132-
// After each jemalloc or pool update, check the current state
161+
// After each allocated or pool update, check the current state
133162
let pool_total: u64 = pool_by_thread.values().sum();
134163
if pool_total > peak_pool_total {
135164
peak_pool_total = pool_total;
136165
}
137166

138-
if latest_jemalloc > 0 && pool_total > 0 && latest_jemalloc > pool_total {
139-
let excess = latest_jemalloc - pool_total;
167+
if latest_allocated > 0 && pool_total > 0 && latest_allocated > pool_total {
168+
let excess = latest_allocated - pool_total;
140169
if excess > peak_excess {
141170
peak_excess = excess;
142171
}
@@ -147,46 +176,56 @@ fn main() {
147176
{
148177
violations.push(MemorySnapshot {
149178
ts: event.ts,
150-
jemalloc: latest_jemalloc,
179+
allocated: latest_allocated,
151180
pool_total,
152181
});
153182
}
154183
}
155184
}
156185

186+
let Some(source) = source.map(|rank| ALLOCATED_COUNTERS[rank]) else {
187+
eprintln!(
188+
"No process-wide allocation counter found in the trace: expected one of {}. \
189+
Build the native library with the `alloc-accounting` or `jemalloc` feature.",
190+
ALLOCATED_COUNTERS.join(", ")
191+
);
192+
std::process::exit(1);
193+
};
194+
157195
// Print summary
158196
println!("=== Comet Trace Memory Analysis ===\n");
159197
println!("Counter events parsed: {counter_events}");
198+
println!("Allocation counter: {source}");
160199
println!("Threads with memory pools: {}", pool_by_thread.len());
161-
println!("Peak jemalloc allocated: {}", format_bytes(peak_jemalloc));
200+
println!("Peak {source}: {}", format_bytes(peak_allocated));
162201
println!(
163202
"Peak pool total: {}",
164203
format_bytes(peak_pool_total)
165204
);
166205
println!(
167-
"Peak excess (jemalloc - pool): {}",
206+
"Peak excess ({source} - pool): {}",
168207
format_bytes(peak_excess)
169208
);
170209
println!();
171210

172211
if violations.is_empty() {
173-
println!("OK: jemalloc never exceeded the total pool reservation.");
212+
println!("OK: {source} never exceeded the total pool reservation.");
174213
} else {
175214
println!(
176-
"WARNING: jemalloc exceeded pool reservation at {} sampled points:\n",
215+
"WARNING: {source} exceeded pool reservation at {} sampled points:\n",
177216
violations.len()
178217
);
179218
println!(
180-
"{:>14} {:>14} {:>14} {:>14}",
181-
"Time (us)", "jemalloc", "pool_total", "excess"
219+
"{:>14} {:>18} {:>14} {:>14}",
220+
"Time (us)", source, "pool_total", "excess"
182221
);
183-
println!("{}", "-".repeat(62));
222+
println!("{}", "-".repeat(66));
184223
for snap in &violations {
185-
let excess = snap.jemalloc - snap.pool_total;
224+
let excess = snap.allocated - snap.pool_total;
186225
println!(
187-
"{:>14} {:>14} {:>14} {:>14}",
226+
"{:>14} {:>18} {:>14} {:>14}",
188227
snap.ts,
189-
format_bytes(snap.jemalloc),
228+
format_bytes(snap.allocated),
190229
format_bytes(snap.pool_total),
191230
format_bytes(excess),
192231
);

0 commit comments

Comments
 (0)