Repository navigation
Commit 6263df2
authored
Fix AvgGroupsAccumulator::size() self-overcount (apache#25117)
## Which issue does this PR close?
* Part of apache#23393
## Rationale for this change
`GroupsAccumulator::size()` is expected to report retained state owned
by the accumulator without counting the accumulator object itself.
`AvgGroupsAccumulator::size()` was also charging for the inline `Vec`
descriptors for `counts` and `sums`, even though those descriptors are
part of the accumulator object rather than separately retained heap
state. This caused AVG group accumulators to over-report their memory
usage.
## What changes are included in this PR?
This PR:
* Removes the `size_of::<Vec<_>>()` charges for the inline `counts` and
`sums` vector descriptors from `AvgGroupsAccumulator::size()`.
* Keeps accounting for the heap capacities of `counts` and `sums`.
* Keeps accounting for retained null-tracking state through
`null_state.size()`.
* Adds coverage verifying that the sum buffer is accounted for using the
actual sum native type, including the `Decimal128` input / `Decimal256`
sum case.
## Are these changes tested?
Yes. This PR adds the following focused unit tests in
`datafusion/functions-aggregate/src/average.rs`:
* `avg_groups_size_excludes_accumulator_storage`
* `avg_groups_size_uses_sum_native_type`
The first verifies that an empty accumulator reports zero retained state
and that, after group state is allocated, `size()` equals the capacities
of `counts` and `sums` plus `null_state.size()`.
The second verifies that memory accounting uses the sum accumulator's
native type by checking a `Decimal128` AVG whose sum state uses
`Decimal256`.
No test execution results are shown in the patch.
## Are there any user-facing changes?
No direct user-facing API or query-result changes are included.
This changes the memory-size accounting reported by
`AvgGroupsAccumulator` so that it follows the
`GroupsAccumulator::size()` contract and no longer includes inline
vector descriptors.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.1 parent 1fb8980 commit 6263df2
1 file changed
Lines changed: 53 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1246 | 1246 | | |
1247 | 1247 | | |
1248 | 1248 | | |
1249 | | - | |
1250 | | - | |
1251 | | - | |
1252 | | - | |
1253 | | - | |
1254 | | - | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
1255 | 1252 | | |
1256 | 1253 | | |
1257 | 1254 | | |
| |||
1643 | 1640 | | |
1644 | 1641 | | |
1645 | 1642 | | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
1646 | 1693 | | |
1647 | 1694 | | |
1648 | 1695 | | |
| |||
0 commit comments