You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6b5b27b
Browse filesBrowse the repository at this point in the historyBrowse files
fix: charge retained scratch indices capacity in GroupsAccumulatorAdapter
`GroupsAccumulatorAdapter` tracks per-group memory in `allocation_bytes` by
measuring each `AccumulatorState::size()` before and after accumulator work and
applying the delta. `size()` includes the scratch `indices` vector's capacity,
but that capacity is never charged, because:
1. `indices` grows in the per-row push loop, which runs before `sizes_pre` is
measured;
2. `indices.clear()` after the accumulator call retains the capacity.
So `sizes_pre` and `sizes_post` observe the identical `allocated_size()` on
every batch and the delta is always zero. The capacity is charged exactly zero
times, permanently, while `size()` is what the aggregate stream reports to the
`MemoryPool`, so the pool under-counts and memory-pressure handling is delayed.
The same asymmetry has a second effect at emit time: `evaluate` and `state`
call `free_allocation(state.size())`, which releases capacity that was never
charged, so `allocation_bytes` drifts down (and saturates at zero) across
partial emits.
Charge the growth explicitly. `indices_allocation_bytes` records the capacity
already charged; each batch totals the current capacity in the pass that
already visits every group and charges only the difference, so a group whose
`indices` grew once and was then cleared stays charged without being charged
again. Emitting a state drops its capacity from that total. The invariant is
now that `allocation_bytes` equals the sum of `AccumulatorState::size()` plus
the `states` vector allocation, which is what the added tests assert.
No new per-row work: the push loop is untouched, and no `size()` call is added
(`size()` was historically a bottleneck with many distinct groups, which is why
deltas are used). The added cost is one `usize` addition per group per batch in
an existing loop.
Measured on a 16384-row batch across 1000 groups, with 8192 rows in group 0 and
the rest spread evenly over the remaining 999, using a 16-byte accumulator:
168,096 bytes truly retained, 96,960 reported before, so 71,136 bytes (42%) went
unaccounted. Query results are unchanged.
0 commit comments