Skip to content

Commit ce9b13a

Browse files
committed
fix: settle the balance before delegating a free in AccountingAllocator
`dealloc` accounted after calling the inner allocator, mirroring `alloc` and `realloc`. A free cannot fail, so that ordering bought nothing, and it opened a window: jemalloc decrements `stats.allocated` at the start of a large free and then, for blocks above its 8 MiB oversize threshold, unmaps the pages eagerly, which takes milliseconds for a block of a few hundred megabytes. For that whole window the balance still carried a block the allocator had already given back, so `native_allocated` read above `jemalloc_allocated` by the size of the block in flight. On TPC-H SF100 about 2% of trace samples showed the excess, up to 160 MB, during task teardown in Q10, Q17 and Q18. Subtract before delegating. A new test wraps a recording inner allocator and asserts the balance has already dropped by the time the inner free is called; it fails with "inner dealloc saw balance 67182807, expected at most 33628375" under the previous ordering.
1 parent 93d0770 commit ce9b13a

1 file changed

Lines changed: 60 additions & 1 deletion

File tree

‎native/core/src/alloc_accounting.rs‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,8 +168,13 @@ unsafe impl<A: GlobalAlloc> GlobalAlloc for AccountingAllocator<A> {
168168
}
169169

170170
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
171-
self.inner.dealloc(ptr, layout);
171+
// Settle before delegating. A free cannot fail, so there is nothing to wait for, and the
172+
// inner free can be slow: jemalloc returns oversize blocks to the OS eagerly, and unmapping
173+
// a few hundred megabytes takes milliseconds. Accounting afterwards would keep the block on
174+
// the balance for that whole window, after the allocator's own statistics had already
175+
// dropped it.
172176
track(-(layout.size() as isize));
177+
self.inner.dealloc(ptr, layout);
173178
}
174179

175180
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
@@ -239,6 +244,60 @@ mod tests {
239244
drop(held);
240245
}
241246

247+
/// The balance must drop before the inner allocator is asked to free the block.
248+
///
249+
/// jemalloc decrements its own `stats.allocated` at the start of a large free and then, for
250+
/// blocks above its oversize threshold, unmaps the pages eagerly, which takes milliseconds for
251+
/// a block of a few hundred megabytes. If the subtraction happened after delegating, the balance
252+
/// would keep reporting a block the allocator had already given back for that whole window,
253+
/// and `native_allocated` would read above `jemalloc_allocated`.
254+
#[test]
255+
fn dealloc_settles_before_delegating() {
256+
use std::alloc::System;
257+
use std::sync::atomic::AtomicUsize;
258+
259+
/// Records the reported balance at the moment the inner free is called.
260+
struct Recording {
261+
balance_at_dealloc: AtomicUsize,
262+
}
263+
264+
unsafe impl GlobalAlloc for Recording {
265+
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
266+
System.alloc(layout)
267+
}
268+
269+
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
270+
self.balance_at_dealloc
271+
.store(current_balance(), Ordering::Relaxed);
272+
System.dealloc(ptr, layout)
273+
}
274+
}
275+
276+
// Well above the settle threshold, so both the allocation and the free flush immediately.
277+
const SIZE: usize = 64 * 1024 * 1024;
278+
let allocator = AccountingAllocator::new(Recording {
279+
balance_at_dealloc: AtomicUsize::new(usize::MAX),
280+
});
281+
let layout = Layout::from_size_align(SIZE, 8).unwrap();
282+
283+
// SAFETY: the layout is valid and non-zero, and the block is freed below through the same
284+
// allocator that produced it.
285+
let ptr = unsafe { allocator.alloc(layout) };
286+
assert!(!ptr.is_null());
287+
let after_alloc = current_balance();
288+
unsafe { allocator.dealloc(ptr, layout) };
289+
290+
let seen = allocator.inner.balance_at_dealloc.load(Ordering::Relaxed);
291+
// Half the block is a wide margin against parallel test noise while still being far
292+
// outside anything the mutation (subtracting after delegating) could produce.
293+
assert!(
294+
seen + SIZE / 2 <= after_alloc,
295+
"inner dealloc saw balance {seen}, expected at most {} (balance after alloc was \
296+
{after_alloc})",
297+
after_alloc - SIZE / 2
298+
);
299+
}
300+
242301
/// Threads must settle their remaining drift on exit.
243302
///
244303
/// Each worker allocates a sub-threshold buffer — so the bytes are still sitting in its local

0 commit comments

Comments
 (0)