Repository navigation
fix: account for struct memory when emitting min/max groups - #26894
Closed
efegokdemir wants to merge 1 commit into
Closed
efegokdemir wants to merge 1 commit into
efegokdemir wants to merge 1 commit into
Conversation
Signed-off-by: Efe Gökdemir <[email protected]>
alamb
added
the
Prolific Author
Contributor
|
Thanks for the contribution! I am closing this under DataFusion's AI Dump policy. Review capacity is very limited and this is a PR opened within hours of its issue with a testing section built around command runs and pass counts, from an author flagged for opening multiple PRs in a short time. The policy asks new contributors to work with maintainers on one PR at a time rather than opening several at once. If this fix matters to you, please pick the single change you care most about, reopen or resubmit it with a short description in your own words, and we are happy to help with review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Ordered aggregation emits groups incrementally, but
MinMaxStructState::emit_to(EmitTo::First(n))subtracts row counts from a byte counter. Emitted struct memory therefore remains counted, which can cause valid queries to exhaust the memory pool.What changes are included in this PR?
Subtract the memory size of each emitted struct and add a regression test for the emitted byte count and remaining state size.
What is the testing strategy for this PR?
cargo test --locked -p datafusion-functions-aggregate --lib(267 passed).cargo fmt --all -- --checkand workspace Clippy with all targets and features passed.Are there any user-facing changes?
Yes. Ordered aggregation no longer overcounts memory for emitted struct values. There are no public API changes.