Skip to content

Commit 6263df2

Browse files
authored
Fix AvgGroupsAccumulator::size() self-overcount (apache#25117)
## Which issue does this PR close? * Part of apache#23393 ## Rationale for this change `GroupsAccumulator::size()` is expected to report retained state owned by the accumulator without counting the accumulator object itself. `AvgGroupsAccumulator::size()` was also charging for the inline `Vec` descriptors for `counts` and `sums`, even though those descriptors are part of the accumulator object rather than separately retained heap state. This caused AVG group accumulators to over-report their memory usage. ## What changes are included in this PR? This PR: * Removes the `size_of::<Vec<_>>()` charges for the inline `counts` and `sums` vector descriptors from `AvgGroupsAccumulator::size()`. * Keeps accounting for the heap capacities of `counts` and `sums`. * Keeps accounting for retained null-tracking state through `null_state.size()`. * Adds coverage verifying that the sum buffer is accounted for using the actual sum native type, including the `Decimal128` input / `Decimal256` sum case. ## Are these changes tested? Yes. This PR adds the following focused unit tests in `datafusion/functions-aggregate/src/average.rs`: * `avg_groups_size_excludes_accumulator_storage` * `avg_groups_size_uses_sum_native_type` The first verifies that an empty accumulator reports zero retained state and that, after group state is allocated, `size()` equals the capacities of `counts` and `sums` plus `null_state.size()`. The second verifies that memory accounting uses the sum accumulator's native type by checking a `Decimal128` AVG whose sum state uses `Decimal256`. No test execution results are shown in the patch. ## Are there any user-facing changes? No direct user-facing API or query-result changes are included. This changes the memory-size accounting reported by `AvgGroupsAccumulator` so that it follows the `GroupsAccumulator::size()` contract and no longer includes inline vector descriptors. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 1fb8980 commit 6263df2

1 file changed

Lines changed: 53 additions & 6 deletions

File tree

‎datafusion/functions-aggregate/src/average.rs‎

Lines changed: 53 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1246,12 +1246,9 @@ where
12461246
fn size(&self) -> usize {
12471247
// Heap buffers
12481248
self.counts.capacity() * size_of::<u64>()
1249-
+ self.sums.capacity() * size_of::<S::Native>()
1250-
// Vec struct overhead (ptr, len, cap) for each field
1251-
+ size_of::<Vec<u64>>()
1252-
+ size_of::<Vec<S::Native>>()
1253-
// Null tracking buffers
1254-
+ self.null_state.size()
1249+
+ self.sums.capacity() * size_of::<S::Native>()
1250+
// Null tracking buffers
1251+
+ self.null_state.size()
12551252
}
12561253
}
12571254

@@ -1643,6 +1640,56 @@ mod tests {
16431640
Ok(())
16441641
}
16451642

1643+
#[test]
1644+
fn avg_groups_size_excludes_accumulator_storage() -> Result<()> {
1645+
let mut accumulator = AvgGroupsAccumulator::<Float64Type, _>::new(
1646+
&DataType::Float64,
1647+
&DataType::Float64,
1648+
|sum, count| Ok(sum / count as f64),
1649+
);
1650+
1651+
assert_eq!(accumulator.size(), 0);
1652+
1653+
let values = Arc::new(Float64Array::from(vec![Some(2.0), None, Some(4.0)]));
1654+
accumulator.update_batch(&[values], &[0, 1, 2], None, 3)?;
1655+
1656+
let expected_size = accumulator.counts.capacity() * size_of::<u64>()
1657+
+ accumulator.sums.capacity() * size_of::<f64>()
1658+
+ accumulator.null_state.size();
1659+
assert_eq!(accumulator.size(), expected_size);
1660+
assert!(accumulator.size() > 0);
1661+
1662+
Ok(())
1663+
}
1664+
1665+
#[test]
1666+
fn avg_groups_size_uses_sum_native_type() -> Result<()> {
1667+
let input_type = DataType::Decimal128(26, 0);
1668+
let sum_type = avg_sum_data_type(&input_type);
1669+
let return_type = Avg::new().return_type(std::slice::from_ref(&input_type))?;
1670+
assert_eq!(sum_type, DataType::Decimal256(76, 0));
1671+
assert_eq!(return_type, DataType::Decimal128(30, 4));
1672+
let mut accumulator = AvgGroupsAccumulator::<
1673+
Decimal128Type,
1674+
_,
1675+
Decimal256Type,
1676+
Decimal128Type,
1677+
>::new(&sum_type, &return_type, |_, _| Ok(0_i128));
1678+
1679+
let values = Arc::new(
1680+
Decimal128Array::from(vec![Some(2), None, Some(4)])
1681+
.with_precision_and_scale(26, 0)?,
1682+
);
1683+
accumulator.update_batch(&[values], &[0, 1, 2], None, 3)?;
1684+
1685+
let expected_size = accumulator.counts.capacity() * size_of::<u64>()
1686+
+ accumulator.sums.capacity() * size_of::<i256>()
1687+
+ accumulator.null_state.size();
1688+
assert_eq!(accumulator.size(), expected_size);
1689+
1690+
Ok(())
1691+
}
1692+
16461693
#[test]
16471694
fn average_groups_preserving_reads() -> Result<()> {
16481695
let mut accumulator = AvgGroupsAccumulator::<Float64Type, _>::new(

0 commit comments

Comments
 (0)