Skip to content

Commit cf01af5

Browse files
sqllogictest: account before alloc to avoid panic-after-alloc hazards (apache#22742)
## Which issue does this PR close? - Closes #. Follow-up defect fix in the allocator-level accounting framework added by apache#22626. ## Rationale for this change `AccountingAllocator::realloc` (introduced in apache#22626) called `inner.realloc` first — which frees the caller's old pointer on success — then called `track`, which can `panic_any` on overdraft. The unwind starts with the caller's `Vec` (or similar growing container) still holding the now-freed old pointer; the next `Drop` produces glibc's `double free or corruption (out)` and SIGABRT, masking the underlying untracked-allocation panic the framework was trying to surface. The same hazard does not apply to `alloc` / `alloc_zeroed` / `dealloc`: - `alloc` / `alloc_zeroed` — the caller has no preexisting pointer to be invalidated; panicking after the inner alloc just leaks the new allocation, which is fine on the kill path. - `dealloc` — credits the bank, never panics. Only `realloc` has a live caller-side pointer that gets invalidated by the inner operation before the panic decision. ## What changes are included in this PR? Reorder `AccountingAllocator::realloc` to account first, forward to `inner.realloc` second: - `track(delta)` first — on overdraft we panic with the caller's pointer still valid; unwind drops the live container cleanly, no abort. - If `inner.realloc` returns null after a successful track, refund the delta so the bank stays consistent with what actually got allocated. ## Are these changes tested? Manually verified using a `GroupedHashAggregateStream` + `List<Utf8>` group-key reproducer (routes through `GroupValuesRows::intern` → `RowConverter::append` → `Vec::resize` → `__rust_realloc`, which is one of the untracked-allocation sites this framework is meant to catch): - before: exit 101, `double free or corruption (out)`, signal 6 SIGABRT - after: exit 1, clean `allocator overdraft: account balance at panic = -1.7 MB` at the same `RowConverter::append` stack frame A regression test for "panic from inside `realloc` does not double-free" would require constructing the precise realloc-mid-grow scenario plus a `catch_unwind` harness; deferred as follow-up. ## Are there any user-facing changes? No. Confined to `sqllogictest`'s `memory-accounting` feature, which is internal CI tooling. Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent d88ab6a commit cf01af5

1 file changed

Lines changed: 26 additions & 9 deletions

File tree

‎datafusion/sqllogictest/src/accounting.rs‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -276,37 +276,54 @@ impl AccountingAllocator<System> {
276276

277277
unsafe impl<A: GlobalAlloc> GlobalAlloc for AccountingAllocator<A> {
278278
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
279+
// Account BEFORE the inner alloc. If we panicked AFTER `inner.alloc`
280+
// succeeded, the bytes are physically allocated but no caller ever
281+
// sees the pointer → unwind leaks the very bytes that pushed us
282+
// over the budget — the opposite of what the kill panic is for.
283+
let delta = -(layout.size() as isize);
284+
track(delta);
279285
// SAFETY: layout is forwarded unchanged.
280286
let ptr = unsafe { self.inner.alloc(layout) };
281-
if !ptr.is_null() {
282-
// Allocation debits the bank.
283-
track(-(layout.size() as isize));
287+
if ptr.is_null() {
288+
// Allocator refused — refund so the bank matches reality.
289+
track(-delta);
284290
}
285291
ptr
286292
}
287293

288294
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
289295
// SAFETY: caller upholds GlobalAlloc invariants; we forward unchanged.
290296
unsafe { self.inner.dealloc(ptr, layout) };
291-
// Free credits the bank.
297+
// Credit only; `track()` short-circuits on `delta >= 0` and never
298+
// panics, so ordering relative to `inner.dealloc` doesn't matter.
292299
track(layout.size() as isize);
293300
}
294301

295302
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
303+
// Same panic-then-leak hazard as `alloc`; account first.
304+
let delta = -(layout.size() as isize);
305+
track(delta);
296306
// SAFETY: layout is forwarded unchanged.
297307
let ptr = unsafe { self.inner.alloc_zeroed(layout) };
298-
if !ptr.is_null() {
299-
track(-(layout.size() as isize));
308+
if ptr.is_null() {
309+
track(-delta);
300310
}
301311
ptr
302312
}
303313

304314
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
315+
// Account BEFORE the inner realloc so a kill panic doesn't strand the
316+
// caller with a freed `ptr`. `inner.realloc` frees `ptr` on success;
317+
// if we panicked after that, the caller's `Vec`-or-similar would
318+
// still hold the old pointer and double-free on unwind (glibc
319+
// "double free or corruption (out)" + SIGABRT).
320+
let delta = layout.size() as isize - new_size as isize;
321+
track(delta);
305322
// SAFETY: caller upholds GlobalAlloc invariants; we forward unchanged.
306323
let new_ptr = unsafe { self.inner.realloc(ptr, layout, new_size) };
307-
if !new_ptr.is_null() {
308-
// Growth debits, shrink credits.
309-
track(layout.size() as isize - new_size as isize);
324+
if new_ptr.is_null() {
325+
// Allocator refused — refund so the bank matches reality.
326+
track(-delta);
310327
}
311328
new_ptr
312329
}

0 commit comments

Comments
 (0)