Skip to content

Commit 0d79252

Browse files
authored
Fix retained-capacity accounting for grouped byte MIN/MAX (#26154)
## Which issue does this PR close? - Part of #23393 ## Rationale for this change Grouped byte MIN/MAX accumulators can underreport memory usage because their accounting is based on the current data length and number of group slots rather than the memory allocations they retain. For example, replacing a 128-byte string with a one-byte string reuses the existing vector allocation, but the previous accounting reports only the new string length. Similarly, unused capacity in the outer group vector is not counted. This change makes memory accounting reflect retained allocations, providing more accurate memory usage estimates for grouped MIN/MAX operations. ## What changes are included in this PR? - Add `total_data_capacity` to `MinMaxBytesState` to track retained inner `Vec<u8>` capacities independently of `total_data_bytes`, which remains responsible for output preallocation. - Update `set_value()` to maintain capacity accounting when values are inserted, replaced, or cause an existing vector to grow. - Update `size()` to account for the outer `min_max` vector's capacity and the cached inner vector capacities, without double-counting the inline `Option<Vec<u8>>` descriptors. - Keep `size()` O(1) by maintaining the inner capacity total incrementally. - Update `emit_to()` to reset capacity accounting on full emission and subtract emitted inner vector capacities on partial emission. ## Are these changes tested? Three regression tests are added: - `size_counts_outer_capacity`: Verifies that memory accounting includes allocated outer vector capacity, even when fewer group slots are initialized. - `size_retains_inner_capacity`: Tests both MIN and MAX accumulators, verifying that replacing a long string with a shorter one preserves retained-capacity accounting and that partial and full emission update it correctly. - `size_tracks_growth_and_emit_reuse`: Verifies capacity accounting when an inner vector grows, groups are partially emitted, and the state is reused after full emission. ## Are there any user-facing changes? No changes to SQL MIN/MAX results or public APIs are intended. The change corrects internal memory accounting for grouped byte MIN/MAX accumulators, allowing retained allocations to be reported more accurately. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 92323f7 commit 0d79252

1 file changed

Lines changed: 83 additions & 11 deletions

File tree

‎datafusion/functions-aggregate/src/min_max/min_max_bytes.rs‎

Lines changed: 83 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -416,9 +416,10 @@ struct MinMaxBytesState {
416416
min_max: Vec<Option<Vec<u8>>>,
417417
/// The data type of the array
418418
data_type: DataType,
419-
/// The total bytes of the string data (for pre-allocating the final array,
420-
/// and tracking memory usage)
419+
/// The total length of the string data, for pre-allocating the final array.
421420
total_data_bytes: usize,
421+
/// The retained inner vector capacities, cached to keep `size()` O(1).
422+
total_data_capacity: usize,
422423
}
423424

424425
/// Implement the MinMaxBytesAccumulator with a comparison function
@@ -433,22 +434,27 @@ impl MinMaxBytesState {
433434
min_max: vec![],
434435
data_type,
435436
total_data_bytes: 0,
437+
total_data_capacity: 0,
436438
}
437439
}
438440

439441
/// Set the specified group to the given value, updating memory usage appropriately
440442
fn set_value(&mut self, group_index: usize, new_val: &[u8]) {
441443
match self.min_max[group_index].as_mut() {
442444
None => {
443-
self.min_max[group_index] = Some(new_val.to_vec());
445+
let value = new_val.to_vec();
446+
self.total_data_capacity += value.capacity();
447+
self.min_max[group_index] = Some(value);
444448
self.total_data_bytes += new_val.len();
445449
}
446450
Some(existing_val) => {
447451
// Copy data over to avoid re-allocating
448452
self.total_data_bytes -= existing_val.len();
449453
self.total_data_bytes += new_val.len();
454+
self.total_data_capacity -= existing_val.capacity();
450455
existing_val.clear();
451456
existing_val.extend_from_slice(new_val);
457+
self.total_data_capacity += existing_val.capacity();
452458
}
453459
}
454460
}
@@ -509,32 +515,38 @@ impl MinMaxBytesState {
509515

510516
/// Emits the specified min_max values
511517
///
512-
/// Returns (data_capacity, min_maxes), updating the current value of total_data_bytes
518+
/// Returns (data_bytes, min_maxes), updating the current value of total_data_bytes
513519
///
514-
/// - `data_capacity`: the total length of all strings and their contents,
520+
/// - `data_bytes`: the total byte length of the emitted values,
515521
/// - `min_maxes`: the actual min/max values for each group
516522
fn emit_to(&mut self, emit_to: EmitTo) -> (usize, Vec<Option<Vec<u8>>>) {
517523
match emit_to {
518524
EmitTo::All => {
525+
self.total_data_capacity = 0;
519526
(
520527
std::mem::take(&mut self.total_data_bytes), // reset total bytes and min_max
521528
std::mem::take(&mut self.min_max),
522529
)
523530
}
524531
EmitTo::First(n) => {
525532
let first_min_maxes = split_vec_min_alloc(&mut self.min_max, n);
526-
let first_data_capacity: usize = first_min_maxes
533+
let (first_data_bytes, first_data_capacity) = first_min_maxes
527534
.iter()
528-
.map(|opt| opt.as_ref().map(|s| s.len()).unwrap_or(0))
529-
.sum();
530-
self.total_data_bytes -= first_data_capacity;
531-
(first_data_capacity, first_min_maxes)
535+
.flatten()
536+
.fold((0, 0), |(bytes, capacity), value| {
537+
(bytes + value.len(), capacity + value.capacity())
538+
});
539+
self.total_data_bytes -= first_data_bytes;
540+
self.total_data_capacity -= first_data_capacity;
541+
(first_data_bytes, first_min_maxes)
532542
}
533543
}
534544
}
535545

536546
fn size(&self) -> usize {
537-
self.total_data_bytes + self.min_max.len() * size_of::<Option<Vec<u8>>>()
547+
// Only retained heap allocations: the outer slots already include the
548+
// inline Option<Vec<u8>> descriptors.
549+
self.total_data_capacity + self.min_max.capacity() * size_of::<Option<Vec<u8>>>()
538550
}
539551
}
540552

@@ -543,6 +555,66 @@ mod tests {
543555
use super::*;
544556
use arrow::array::StringArray;
545557

558+
fn expected_size(state: &MinMaxBytesState) -> usize {
559+
state.min_max.capacity() * size_of::<Option<Vec<u8>>>()
560+
+ state
561+
.min_max
562+
.iter()
563+
.flatten()
564+
.map(Vec::capacity)
565+
.sum::<usize>()
566+
}
567+
568+
#[test]
569+
fn size_tracks_growth_and_emit_reuse() -> Result<()> {
570+
let mut state = MinMaxBytesState::new(DataType::Binary);
571+
assert_eq!(state.size(), 0);
572+
573+
state.min_max = Vec::with_capacity(16);
574+
assert_eq!(state.size(), expected_size(&state));
575+
state.min_max.resize(4, None);
576+
assert_eq!(state.size(), expected_size(&state));
577+
578+
state.set_value(0, b"a");
579+
state.set_value(0, &[b'z'; 128]);
580+
state.set_value(1, b"b");
581+
state.set_value(2, b"c");
582+
state.set_value(3, b"d");
583+
assert_eq!(state.size(), expected_size(&state));
584+
let size_before_replacement = state.size();
585+
586+
state.set_value(0, b"x");
587+
assert_eq!(state.size(), size_before_replacement);
588+
assert!(state.min_max[0].as_ref().unwrap().capacity() >= 128);
589+
assert_eq!(state.total_data_bytes, 4);
590+
assert_eq!(state.size(), expected_size(&state));
591+
592+
let (bytes, emitted) = state.emit_to(EmitTo::First(1));
593+
assert_eq!(emitted[0].as_deref(), Some(b"x".as_slice()));
594+
assert_eq!(bytes, 1);
595+
assert_eq!(state.size(), expected_size(&state));
596+
assert_eq!(state.total_data_bytes, 3);
597+
598+
let (bytes, emitted) = state.emit_to(EmitTo::First(2));
599+
assert_eq!(emitted[0].as_deref(), Some(b"b".as_slice()));
600+
assert_eq!(emitted[1].as_deref(), Some(b"c".as_slice()));
601+
assert_eq!(bytes, 2);
602+
assert_eq!(state.size(), expected_size(&state));
603+
assert_eq!(state.total_data_bytes, 1);
604+
605+
let (bytes, emitted) = state.emit_to(EmitTo::All);
606+
assert_eq!(emitted[0].as_deref(), Some(b"d".as_slice()));
607+
assert_eq!(bytes, 1);
608+
assert_eq!(state.size(), 0);
609+
assert_eq!(state.total_data_bytes, 0);
610+
611+
state.update_batch([Some(b"x".as_slice())], &[0], 1, |a, b| a < b)?;
612+
assert_eq!(state.min_max[0].as_deref(), Some(b"x".as_slice()));
613+
assert_eq!(state.total_data_bytes, 1);
614+
assert_eq!(state.size(), expected_size(&state));
615+
Ok(())
616+
}
617+
546618
#[test]
547619
fn preserving_selected_min_values() -> Result<()> {
548620
let mut accumulator = MinMaxBytesAccumulator::new_min(DataType::Utf8);

0 commit comments

Comments
 (0)