Skip to content

Commit 2d78f87

Browse files
committed
fix: install the accounting wrapper for every allocator feature combination
Select the allocator backend once, in a `backend` module whose three cfgs partition every feature combination, and let the single accounting `#[global_allocator]` refer to whatever that resolved to. Previously each backend predicate was repeated per arm and the accounting fallback to the system allocator excluded `mimalloc`, so `jemalloc,mimalloc,alloc-accounting` on a non-MSVC target matched no arm at all: the process ran on the unwrapped default allocator while `native_allocated` stayed enabled and read zero. A build without the feature is unchanged: the unwrapped allocator lives in the backend module that owns it, and no explicit allocator is installed when the selection is the system allocator. Make the accounting tests immune to parallel test noise. `BALANCE` is process-wide, so with the wrapper installed the crate's other tests move it concurrently and the margin-based assertions failed intermittently. The tests that observe the balance now serialize against each other, the real-allocation probe uses an untouched 256 MiB block that nothing else in the crate can mask, and the thread-exit test injects its drift directly so it no longer needs the feature and runs in the default CI build. Add `threshold_churn` to the `alloc_overhead` benchmark: alloc/free loops at 32 KiB (never flushes) and 64 KiB (flushes on every call), single threaded and from every core at once, so shared-counter contention can be quantified rather than inferred.
1 parent ce9b13a commit 2d78f87

3 files changed

Lines changed: 173 additions & 99 deletions

File tree

‎native/core/benches/alloc_overhead.rs‎

Lines changed: 64 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,13 +28,19 @@
2828
//! through the `rlib`. `assert_accounting_is_live` fails the run if the wrapper is somehow not in
2929
//! effect — without it, a "with the feature" run could silently be a second baseline.
3030
//!
31-
//! `small_churn` is the worst case: allocations so small that the wrapper's bookkeeping is a
32-
//! meaningful fraction of the allocator's own work. `arrow_sized_churn` is closer to what Comet
33-
//! actually does, where a batch-sized buffer dwarfs the bookkeeping. Real query workloads sit at
34-
//! or below `arrow_sized_churn`, because they do actual work between allocations.
31+
//! `small_churn` is the worst case for the thread-local path: allocations so small that the
32+
//! wrapper's bookkeeping is a meaningful fraction of the allocator's own work. `threshold_churn` is
33+
//! the worst case for the shared counter: an alloc/free loop at exactly the 64 KiB settle threshold
34+
//! flushes to the process-wide atomic on every call, and the parallel variant does that from every
35+
//! core at once, so the gap between the single-threaded and parallel numbers is the cost of
36+
//! contention on that cacheline. `arrow_sized_churn` is closer to what Comet actually does, where
37+
//! a batch-sized buffer dwarfs the bookkeeping. Real query workloads sit at or below
38+
//! `arrow_sized_churn`, because they do actual work between allocations.
3539
3640
use criterion::{criterion_group, criterion_main, BatchSize, Criterion, Throughput};
3741
use std::hint::black_box;
42+
use std::thread;
43+
use std::time::Instant;
3844

3945
/// Guards against measuring nothing. If the wrapper were not actually installed in the benchmark
4046
/// binary, every "with the feature" number would silently be a second baseline run.
@@ -108,5 +114,58 @@ fn growth_churn(c: &mut Criterion) {
108114
group.finish();
109115
}
110116

111-
criterion_group!(benches, small_churn, arrow_sized_churn, growth_churn);
117+
/// Alloc/free of an untouched block, so the allocator call itself is most of the work and the
118+
/// wrapper's share is largest.
119+
fn alloc_free(size: usize) {
120+
let v: Vec<u8> = Vec::with_capacity(black_box(size));
121+
black_box(&v);
122+
}
123+
124+
/// Alloc/free loops either side of the 64 KiB settle threshold, single-threaded and from every
125+
/// core at once.
126+
///
127+
/// A loop of one size never accumulates drift, so the threshold decides everything: at 32 KiB the
128+
/// alloc and the free cancel inside the thread-local cell and the shared counter is never touched,
129+
/// while at 64 KiB every alloc and every free flushes. The 64 KiB parallel case is therefore the
130+
/// upper bound on shared-counter contention: `available_parallelism()` threads each doing two
131+
/// atomic read-modify-writes per iteration on the same cacheline, with nothing else in between.
132+
///
133+
/// Times are reported per alloc/free pair per thread, so a parallel number equal to its
134+
/// single-threaded counterpart means the threads did not slow each other down at all.
135+
fn threshold_churn(c: &mut Criterion) {
136+
assert_accounting_is_live();
137+
let threads = thread::available_parallelism().map_or(4, |n| n.get());
138+
let mut group = c.benchmark_group("alloc_overhead");
139+
group.throughput(Throughput::Elements(1));
140+
for size in [32 * 1024usize, 64 * 1024] {
141+
let kb = size / 1024;
142+
group.bench_function(format!("alloc_free_{kb}kb"), |b| {
143+
b.iter(|| alloc_free(size));
144+
});
145+
group.bench_function(format!("parallel_alloc_free_{kb}kb_x{threads}"), |b| {
146+
b.iter_custom(|iters| {
147+
let start = Instant::now();
148+
thread::scope(|scope| {
149+
for _ in 0..threads {
150+
scope.spawn(move || {
151+
for _ in 0..iters {
152+
alloc_free(size);
153+
}
154+
});
155+
}
156+
});
157+
start.elapsed()
158+
});
159+
});
160+
}
161+
group.finish();
162+
}
163+
164+
criterion_group!(
165+
benches,
166+
small_churn,
167+
threshold_churn,
168+
arrow_sized_churn,
169+
growth_churn
170+
);
112171
criterion_main!(benches);

‎native/core/src/alloc_accounting.rs‎

Lines changed: 53 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -196,14 +196,28 @@ unsafe impl<A: GlobalAlloc> GlobalAlloc for AccountingAllocator<A> {
196196
#[cfg(test)]
197197
mod tests {
198198
use super::*;
199+
use std::sync::{Mutex, MutexGuard};
200+
201+
/// `BALANCE` is process-wide and the crate's tests run in parallel, so a test that reads it
202+
/// sees every other test's allocations. The tests that move it by tens of megabytes take this
203+
/// lock so they cannot land inside each other's windows; the rest of the crate is kept out by
204+
/// making each window microseconds wide and each expected move far larger than anything else
205+
/// allocates in that time.
206+
static SERIAL: Mutex<()> = Mutex::new(());
207+
208+
fn serial() -> MutexGuard<'static, ()> {
209+
SERIAL
210+
.lock()
211+
.unwrap_or_else(|poisoned| poisoned.into_inner())
212+
}
199213

200214
#[test]
201215
fn settle_accumulates_below_the_threshold() {
202216
let drift = Cell::new(0);
203-
let before = BALANCE.load(Ordering::Relaxed);
204217
settle(&drift, 1024);
218+
// A flush would have reset the drift to zero, so this alone shows the shared balance was
219+
// not touched. Reading `BALANCE` here would race with every other test's allocations.
205220
assert_eq!(drift.get(), 1024, "small delta stays thread-local");
206-
assert_eq!(BALANCE.load(Ordering::Relaxed), before);
207221
}
208222

209223
#[test]
@@ -228,18 +242,29 @@ mod tests {
228242
assert_eq!(clamp_balance(4096), 4096);
229243
}
230244

231-
/// A real allocation must move the reported balance. Parallel test noise can only add to the
232-
/// balance, so the assertion is one-sided.
245+
/// A real allocation must move the reported balance: this is the one test that checks the
246+
/// wrapper is actually installed as the global allocator for the current feature set, rather
247+
/// than exercising it through a local instance.
248+
///
249+
/// The block is zeroed and never touched, so it costs address space rather than resident
250+
/// memory, and it is large enough that nothing else in the crate can free half of it inside the
251+
/// microseconds between the two reads.
233252
#[test]
234253
#[cfg(feature = "alloc-accounting")]
235254
fn a_real_allocation_raises_the_balance() {
255+
use std::hint::black_box;
256+
257+
const SIZE: usize = 256 * 1024 * 1024;
258+
let _guard = serial();
236259
let before = current_balance();
237-
// Well above the settle threshold, so it is guaranteed to flush.
238-
let held: Vec<u8> = vec![0u8; 8 * 1024 * 1024];
260+
// `black_box` keeps the allocation observable so it cannot be elided.
261+
let held: Vec<u8> = black_box(vec![0u8; SIZE]);
239262
let during = current_balance();
263+
black_box(&held);
240264
assert!(
241-
during >= before + 4 * 1024 * 1024,
242-
"8 MiB allocation should raise the balance (before={before}, during={during})"
265+
during >= before + SIZE / 2,
266+
"a {SIZE} byte allocation should raise the balance (before={before}, during={during}); \
267+
is the accounting wrapper installed for this feature set?"
243268
);
244269
drop(held);
245270
}
@@ -275,6 +300,7 @@ mod tests {
275300

276301
// Well above the settle threshold, so both the allocation and the free flush immediately.
277302
const SIZE: usize = 64 * 1024 * 1024;
303+
let _guard = serial();
278304
let allocator = AccountingAllocator::new(Recording {
279305
balance_at_dealloc: AtomicUsize::new(usize::MAX),
280306
});
@@ -300,49 +326,33 @@ mod tests {
300326

301327
/// Threads must settle their remaining drift on exit.
302328
///
303-
/// Each worker allocates a sub-threshold buffer — so the bytes are still sitting in its local
304-
/// drift, never flushed — and hands ownership back to this thread before exiting. The matching
305-
/// free therefore happens here, after the worker is gone, so the only way those bytes can ever
306-
/// reach the shared balance is `ThreadDrift::drop`. Without the destructor the balance does not
307-
/// move at all, and the later frees drive it *below* where it started.
329+
/// The worker writes a drift straight into its `LOCAL_DRIFT` cell and exits. Without the
330+
/// wrapper installed nothing else ever calls `track`, so the only path by which that value can
331+
/// reach the shared balance is `ThreadDrift::drop`; that is the build CI runs, and the one in
332+
/// which a missing destructor is caught. The value is far larger than any real allocation,
333+
/// which makes the check immune to whatever the rest of the crate is allocating meanwhile.
334+
/// The injected amount is taken back out afterwards so later tests see an unchanged balance.
308335
#[test]
309-
#[cfg(feature = "alloc-accounting")]
310336
fn thread_exit_settles_remaining_drift() {
311-
use std::sync::mpsc;
312337
use std::thread;
313338

314-
const THREADS: usize = 64;
315-
const PER_THREAD: usize = 32 * 1024;
316-
assert!(
317-
(PER_THREAD as isize) < SETTLE_THRESHOLD,
318-
"the per-thread buffer must stay in local drift for this test to mean anything"
319-
);
339+
const INJECTED: isize = 1 << 40;
340+
let _guard = serial();
320341

321-
let (tx, rx) = mpsc::channel();
322-
let before = current_balance() as isize;
323-
324-
for _ in 0..THREADS {
325-
let tx = tx.clone();
326-
thread::spawn(move || tx.send(vec![0u8; PER_THREAD]).unwrap())
327-
.join()
328-
.unwrap();
329-
}
330-
drop(tx);
331-
332-
let held: Vec<Vec<u8>> = rx.iter().collect();
333-
assert_eq!(held.len(), THREADS);
342+
let before = BALANCE.load(Ordering::Relaxed);
343+
thread::spawn(|| {
344+
LOCAL_DRIFT.with(|drift| drift.0.set(drift.0.get() + INJECTED));
345+
})
346+
.join()
347+
.unwrap();
348+
let moved = BALANCE.load(Ordering::Relaxed) - before;
349+
BALANCE.fetch_sub(INJECTED, Ordering::Relaxed);
334350

335-
let moved = current_balance() as isize - before;
336-
let allocated = (THREADS * PER_THREAD) as isize;
337-
// Half the expected total is a wide margin against parallel test noise while still being
338-
// far outside anything the mutation (a destructor that discards the drift) could produce.
339351
assert!(
340-
moved >= allocated / 2,
341-
"drift from exited threads never reached the shared balance: \
352+
moved >= INJECTED / 2,
353+
"drift from an exited thread never reached the shared balance: \
342354
balance moved {moved} bytes, expected at least {}",
343-
allocated / 2
355+
INJECTED / 2
344356
);
345-
346-
drop(held);
347357
}
348358
}

‎native/core/src/lib.rs‎

Lines changed: 56 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -41,19 +41,6 @@ use log4rs::{
4141
Config,
4242
};
4343

44-
#[cfg(all(
45-
not(target_env = "msvc"),
46-
feature = "jemalloc",
47-
not(feature = "mimalloc")
48-
))]
49-
use tikv_jemallocator::Jemalloc;
50-
51-
#[cfg(all(
52-
feature = "mimalloc",
53-
not(all(not(target_env = "msvc"), feature = "jemalloc"))
54-
))]
55-
use mimalloc::MiMalloc;
56-
5744
// Re-export from jvm-bridge crate for internal use
5845
pub use datafusion_comet_jni_bridge::errors;
5946
pub use datafusion_comet_jni_bridge::JAVA_VM;
@@ -73,57 +60,75 @@ pub mod parquet;
7360
#[cfg(debug_assertions)]
7461
pub mod debug;
7562

76-
// The global allocator is the selected backend (jemalloc, mimalloc, or the system allocator),
77-
// optionally wrapped in `AccountingAllocator` when the `alloc-accounting` feature is on. The cfgs
78-
// below are mutually exclusive so exactly one `#[global_allocator]` is defined; a build without
79-
// the feature is byte-for-byte the previous arrangement, with no wrapper and no per-allocation
80-
// work.
63+
// Global allocator selection.
64+
//
65+
// `backend` names the allocator the feature set asks for: jemalloc where it builds, otherwise
66+
// mimalloc, otherwise the system allocator. The three `backend` cfgs partition every feature
67+
// combination, so exactly one definition exists, and each backend predicate is written once. The
68+
// unwrapped `#[global_allocator]` lives inside the backend module that owns it, so a build without
69+
// `alloc-accounting` is byte-for-byte the previous arrangement: no wrapper, no per-allocation work,
70+
// and no explicit allocator at all when the selection is the system allocator.
71+
//
72+
// With `alloc-accounting`, the single wrapped `#[global_allocator]` below refers to
73+
// `backend::Backend` whatever it resolved to. That is what makes the wrapper impossible to drop
74+
// silently: a feature combination with no backend would fail to compile rather than run with the
75+
// metric enabled and reading zero.
8176

77+
/// jemalloc, on targets where it builds, unless mimalloc was also requested.
8278
#[cfg(all(
8379
not(target_env = "msvc"),
8480
feature = "jemalloc",
85-
not(feature = "mimalloc"),
86-
not(feature = "alloc-accounting")
81+
not(feature = "mimalloc")
8782
))]
88-
#[global_allocator]
89-
static GLOBAL: Jemalloc = Jemalloc;
83+
mod backend {
84+
pub type Backend = tikv_jemallocator::Jemalloc;
85+
pub const BACKEND: Backend = tikv_jemallocator::Jemalloc;
86+
87+
#[cfg(not(feature = "alloc-accounting"))]
88+
#[global_allocator]
89+
static GLOBAL: Backend = BACKEND;
90+
}
9091

92+
/// mimalloc, unless a usable jemalloc was also requested.
9193
#[cfg(all(
9294
feature = "mimalloc",
93-
not(all(not(target_env = "msvc"), feature = "jemalloc")),
94-
not(feature = "alloc-accounting")
95+
not(all(not(target_env = "msvc"), feature = "jemalloc"))
9596
))]
96-
#[global_allocator]
97-
static GLOBAL: MiMalloc = MiMalloc;
97+
mod backend {
98+
pub type Backend = mimalloc::MiMalloc;
99+
pub const BACKEND: Backend = mimalloc::MiMalloc;
98100

99-
#[cfg(all(
100-
not(target_env = "msvc"),
101-
feature = "jemalloc",
102-
not(feature = "mimalloc"),
103-
feature = "alloc-accounting"
104-
))]
105-
#[global_allocator]
106-
static GLOBAL: alloc_accounting::AccountingAllocator<Jemalloc> =
107-
alloc_accounting::AccountingAllocator::new(Jemalloc);
101+
#[cfg(not(feature = "alloc-accounting"))]
102+
#[global_allocator]
103+
static GLOBAL: Backend = BACKEND;
104+
}
108105

109-
#[cfg(all(
110-
feature = "mimalloc",
111-
not(all(not(target_env = "msvc"), feature = "jemalloc")),
112-
feature = "alloc-accounting"
113-
))]
114-
#[global_allocator]
115-
static GLOBAL: alloc_accounting::AccountingAllocator<MiMalloc> =
116-
alloc_accounting::AccountingAllocator::new(MiMalloc);
106+
/// The system allocator: the complement of the two cases above. This covers neither feature, a
107+
/// jemalloc request on MSVC, and both features together, which each backend cfg excludes in favour
108+
/// of the other.
109+
#[cfg(not(any(
110+
all(
111+
not(target_env = "msvc"),
112+
feature = "jemalloc",
113+
not(feature = "mimalloc")
114+
),
115+
all(
116+
feature = "mimalloc",
117+
not(all(not(target_env = "msvc"), feature = "jemalloc"))
118+
)
119+
)))]
120+
// Without `alloc-accounting` nothing refers to this selection: the system allocator is the
121+
// default, so no `#[global_allocator]` is installed.
122+
#[cfg_attr(not(feature = "alloc-accounting"), allow(dead_code))]
123+
mod backend {
124+
pub type Backend = std::alloc::System;
125+
pub const BACKEND: Backend = std::alloc::System;
126+
}
117127

118-
// Accounting over the system allocator: neither mimalloc nor a usable jemalloc.
119-
#[cfg(all(
120-
feature = "alloc-accounting",
121-
not(feature = "mimalloc"),
122-
any(target_env = "msvc", not(feature = "jemalloc"))
123-
))]
128+
#[cfg(feature = "alloc-accounting")]
124129
#[global_allocator]
125-
static GLOBAL: alloc_accounting::AccountingAllocator<std::alloc::System> =
126-
alloc_accounting::AccountingAllocator::new(std::alloc::System);
130+
static GLOBAL: alloc_accounting::AccountingAllocator<backend::Backend> =
131+
alloc_accounting::AccountingAllocator::new(backend::BACKEND);
127132

128133
#[no_mangle]
129134
pub extern "system" fn Java_org_apache_comet_NativeBase_init(

0 commit comments

Comments
 (0)