You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0add046
Browse filesBrowse the repository at this point in the historyBrowse files
perf: hoist split_vec_min_alloc to datafusion-common and shrink the emitted prefix (apache#22416)
## Which issue does this PR close?
Related to apache#22164 and apache#22165.
## Rationale for this change
apache#22165 added a `split_vec_min_alloc` helper so that `EmitTo::First(n)`
allocates `min(n, len - n)` instead of always copying `n` elements. This
PR finishes and corrects that work, following review:
- The helper was `pub(super)` inside `datafusion-physical-plan`, so it
could not be reused. It moves to `datafusion_common::utils` as the
single shared copy.
- The `split_off` branch returned the original allocation as the emitted
prefix: short length, but original (larger) capacity. datafusion
accounts memory by capacity rather than length, so that prefix did not
actually release memory under pressure. The helper now calls
`shrink_to_fit` on it.
- Two further `EmitTo::First(n)` paths still used the
`drain(..n).collect()` idiom: the min/max accumulators and
`ByteViewGroupValueBuilder::take_n`. Both now route through the shared
helper.
## What changes are included in this PR?
- `datafusion_common::utils::split_vec_min_alloc`: the shared helper,
with `shrink_to_fit` on the emitted prefix.
- `datafusion-physical-plan`: the duplicate helper is removed;
`bytes.rs`, `primitive.rs` and `bytes_view.rs` use the shared one.
- `datafusion-functions-aggregate`: `MinMaxStructAccumulator` and
`MinMaxBytesAccumulator` use it in `emit_to`.
This supersedes apache#22205, whose `ByteViewGroupValueBuilder::take_n` change
is included here.
## Are these changes tested?
Yes. Unit tests for the helper (both branches and the boundaries) plus
the existing `take_n` and `min_max` tests pass.
## Are there any user-facing changes?
No.
---------
Co-authored-by: RyanJamesStewart <[email protected]>
Co-authored-by: Claude Opus 4.7 <[email protected]>
0 commit comments