Skip to content

Commit 6a4e997

Browse files
committed
Fix aggregate accumulator capacity accounting (apache#24099)
## Which issue does this PR close? * Part of apache#23393 ## Rationale for this change Two aggregate accumulators underreport owned memory in their `size()` implementations: * `CountGroupsAccumulator` calculates vector backing storage using the wrong element type. * `SlidingDistinctSumAccumulator` reports only the accumulator struct size and omits its owned hash map allocation. This change corrects those capacity-based estimates while preserving the different `Accumulator::size()` and `GroupsAccumulator::size()` self-inclusion contracts. ## What changes are included in this PR? * Add `vec_capacity_bytes` to calculate a vector’s backing allocation using its actual element type. * Update `CountGroupsAccumulator::size()` to report the capacity-based allocation of its counts vector while remaining self-excluded. * Update `SlidingDistinctSumAccumulator::size()` to include `size_of_val(self)` and an estimate of its hash map’s allocated entry capacity. * Document that the hash map estimate excludes implementation-specific control bytes. ## Are these changes tested? Yes. This PR adds the following unit tests: * `count_groups_size_includes_vec_capacity` * `vec_capacity_bytes_uses_element_type` * `sliding_distinct_sum_size_includes_hash_map_capacity` The tests verify that: * grouped count size grows when vector backing storage is allocated; * vector capacity accounting uses the vector’s element type; * sliding distinct-sum size includes hash map capacity, grows after the map expands, and continues to reflect retained capacity after entries are retracted. ## Are there any user-facing changes? No user-facing API or aggregate result changes are intended. This PR only corrects internal memory-size estimates used for aggregate accounting. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent e05aa1f commit 6a4e997

2 files changed

Lines changed: 60 additions & 4 deletions

File tree

‎datafusion/functions-aggregate/src/count.rs‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ use arrow::{
2929
},
3030
};
3131
use datafusion_common::hash_utils::RandomState;
32+
use datafusion_common::heap_size::{DFHeapSize, DFHeapSizeCtx};
3233
use datafusion_common::{
3334
HashMap, Result, ScalarValue, downcast_value, exec_err, internal_err, not_impl_err,
3435
stats::Precision, utils::expr::COUNT_STAR_EXPANSION,
@@ -774,7 +775,7 @@ impl GroupsAccumulator for CountGroupsAccumulator {
774775
Ok(vec![state_array])
775776
}
776777
fn size(&self) -> usize {
777-
self.counts.capacity() * size_of::<usize>()
778+
self.counts.heap_size(&mut DFHeapSizeCtx::default())
778779
}
779780
}
780781

@@ -932,6 +933,23 @@ mod tests {
932933
)?)
933934
}
934935

936+
#[test]
937+
fn count_groups_size_includes_vec_capacity() -> Result<()> {
938+
let mut acc = CountGroupsAccumulator::new();
939+
let empty_size = acc.size();
940+
assert_eq!(empty_size, 0);
941+
let values: ArrayRef = Arc::new(Int64Array::from(vec![1, 2, 3]));
942+
acc.update_batch(&[values], &[0, 1, 2], None, 3)?;
943+
944+
assert!(acc.counts.capacity() > 0);
945+
let allocated_size = acc.counts.heap_size(&mut DFHeapSizeCtx::default());
946+
assert_eq!(allocated_size, acc.counts.capacity() * size_of::<i64>());
947+
assert_eq!(acc.size(), allocated_size);
948+
assert!(acc.size() > empty_size);
949+
950+
Ok(())
951+
}
952+
935953
#[test]
936954
fn count_accumulator_nulls() -> Result<()> {
937955
let mut accumulator = CountAccumulator::new();

‎datafusion/functions-aggregate/src/sum.rs‎

Lines changed: 41 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ use datafusion_functions_aggregate_common::aggregate::groups_accumulator::prim_o
4848
use datafusion_functions_aggregate_common::aggregate::sum_distinct::DistinctSumAccumulator;
4949
use datafusion_macros::user_doc;
5050
use datafusion_physical_expr::expressions::{CastExpr, Column};
51-
use std::mem::size_of_val;
51+
use std::mem::{size_of, size_of_val};
5252

5353
make_udaf_expr_and_func!(
5454
Sum,
@@ -671,7 +671,8 @@ impl Accumulator for SlidingDistinctSumAccumulator {
671671
}
672672

673673
fn size(&self) -> usize {
674-
size_of_val(self)
674+
// Estimate the owned map buckets; implementation-specific control bytes are excluded.
675+
size_of_val(self) + self.counts.capacity() * size_of::<(i64, usize)>()
675676
}
676677

677678
fn state(&mut self) -> Result<Vec<ScalarValue>> {
@@ -722,7 +723,10 @@ mod tests {
722723
array::{Decimal128Array, Int64Array},
723724
buffer::{NullBuffer, ScalarBuffer},
724725
};
725-
use std::sync::Arc;
726+
use std::{
727+
mem::{size_of, size_of_val},
728+
sync::Arc,
729+
};
726730

727731
#[test]
728732
fn sliding_distinct_sum_ignores_null_slots() -> Result<()> {
@@ -750,6 +754,40 @@ mod tests {
750754
Ok(())
751755
}
752756

757+
fn expected_sliding_distinct_sum_size(acc: &SlidingDistinctSumAccumulator) -> usize {
758+
size_of_val(acc) + acc.counts.capacity() * size_of::<(i64, usize)>()
759+
}
760+
761+
#[test]
762+
fn sliding_distinct_sum_size_includes_hash_map_capacity() -> Result<()> {
763+
let mut acc = SlidingDistinctSumAccumulator::try_new(&DataType::Int64)?;
764+
let empty_size = acc.size();
765+
let values: ArrayRef = Arc::new(Int64Array::from(vec![1, 2, 3]));
766+
acc.update_batch(&[Arc::clone(&values)])?;
767+
768+
let expected = expected_sliding_distinct_sum_size(&acc);
769+
assert!(acc.counts.capacity() > 0);
770+
assert_eq!(acc.size(), expected);
771+
assert!(acc.size() > empty_size);
772+
773+
let initial_capacity = acc.counts.capacity();
774+
let additional_values: ArrayRef =
775+
Arc::new(Int64Array::from_iter(4..4 + initial_capacity as i64 + 1));
776+
acc.update_batch(&[Arc::clone(&additional_values)])?;
777+
778+
let grown_size = expected_sliding_distinct_sum_size(&acc);
779+
assert!(acc.counts.capacity() > initial_capacity);
780+
assert_eq!(acc.size(), grown_size);
781+
assert!(acc.size() > expected);
782+
783+
acc.retract_batch(&[values])?;
784+
acc.retract_batch(&[additional_values])?;
785+
assert!(acc.counts.is_empty());
786+
assert_eq!(acc.size(), grown_size);
787+
788+
Ok(())
789+
}
790+
753791
#[test]
754792
fn sliding_distinct_sum_returns_null_for_all_null_frame() -> Result<()> {
755793
let mut acc = SlidingDistinctSumAccumulator::try_new(&DataType::Int64)?;

0 commit comments

Comments
 (0)