You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1bfc59e
Browse filesBrowse the repository at this point in the historyBrowse files
fix: release clustered aggregate table capacity after emitting completed groups on memory pressure (apache#26726)
## Which issue does this PR close?
- Closesapache#26725.
## Rationale for this change
When a reservation is refused, the clustered aggregate streams emit
their completed groups instead of failing. apache#26576 did this for the
partial stream under full group clustering, and apache#26637 did it for the
final and single streams. The remaining table keeps the capacity it grew
for the batch, so re-reserving it can still fail with
`ResourcesExhausted`. With 8192 distinct `Int64` keys per batch, the
partial table still needs 344 KB, while a table holding only the active
group needs 5.6 KB. The issue has the details.
## What changes are included in this PR?
- `GroupValues::shrink_to_fit` (new, does nothing by default) releases
capacity that the stored groups do not need. It is implemented for:
- `GroupValuesPrimitive`: the hash table and values.
- `GroupValuesClustered`: `new_groups`.
- `GroupValuesColumn`: the hash table and the scratch buffers of one
`intern` call.
- `GroupValuesRows`: the hash table, hashes, and rows buffers.
`GroupValuesBytes` and `GroupValuesBytesView` already rebuild their map
on `EmitTo::First`.
- `ClusteredAggregateTable::shrink_to_fit` (new) calls it and also
releases `group_indices`.
- After a refused reservation, `ClusteredPartialAggregateStream` (under
full group clustering), `ClusteredFinalAggregateStream`, and
`ClusteredSingleAggregateStream` emit the completed groups as before and
then call `shrink_to_fit`. The output does not change, and each stream
still returns the refusal when no group is complete. The path without
memory pressure does not call it.
Accumulator scratch, such as the `APPROX_DISTINCT` hashes, is not
released yet; see apache#26781.
## What is the testing strategy for this PR?
-
`clustered_partial_stream::tests::completed_groups_under_memory_pressure`
runs three batches of 8192 distinct keys whose boundaries share a key,
under a 100 KiB limit, and compares the output exactly with an unlimited
run. It covers one key each for `GroupValuesPrimitive` (`Int64`),
`GroupValuesClustered` (`Int64, Int64`), `GroupValuesColumn` (`Float64,
Float64`), and `GroupValuesRows` (`Int64, List`). All four cases fail on
`main` with `ResourcesExhausted`. The `Int64`, `Float64, Float64`, and
`Int64, List` cases also fail if only `group_indices` is released.
- `completed_groups_release_batch_capacity` in `clustered_final_stream`
and `clustered_single_stream` runs the same input with an `Int64` key
under the same limit, using the new `distinct_key_batches` and
`assert_output_unchanged_under_limit` helpers in `clustered_test_util`.
Both fail on `main` with `ResourcesExhausted`.
## Are there any user-facing changes?
No. `GroupValues` gains a method with a default implementation.
This pull request and its description were written by Isaac.
---------
Co-authored-by: Isaac <[email protected]>
0 commit comments