Repository navigation
refactor(hash-aggr): Migrate ordered partial/final aggregation #23181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
0173f80
refactor(hash-aggr): Migrate ordered partial/final aggregation
2010YOUY01 ce800d5
remove memory limit check, ordered case uses bounded memory
2010YOUY01 14623b4
Merge branch 'main' into split-aggr-ordered
2010YOUY01 59864f2
fix clippy
2010YOUY01 8bb64f0
fix: respect batch size in ordered aggregates
alamb add797a
fix: fall back for ordered aggregates with memory limits
alamb e5af273
Merge pull request #3 from alamb/codex/ordered-aggregate-batch-size
2010YOUY01 7559ad4
review: cleanup redundant marker types
2010YOUY01 674234b
review: reuse common aggregate/output code
2010YOUY01 6e65855
review: more comments
2010YOUY01 71ef114
Merge pull request #4 from alamb/codex/ordered-aggregate-memory-fallback
2010YOUY01 29abb12
review: simplify
2010YOUY01 c2ae7dc
Merge remote-tracking branch 'origin/split-aggr-ordered' into split-a…
2010YOUY01 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
codex also notes that the exising code doesn't use the partial group by operator when memory limited, as the new tables don't seem to handle the OOM path (though maybe I missed that)
This is somewhat unobviously implemented by
OutOfMemoryModechoice heredatafusion/datafusion/physical-plan/src/aggregates/row_hash.rs
Lines 492 to 511 in 59864f2
And a proposed PR to fix
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the existing behavior is, if there is memory limit, ordered aggregation case fallback to regular hash aggregation path (no ordering optimization) for the spilling.
This is intended for partially ordered case, since it might OOM, and we haven't implement spilling for that.
For fully ordered cases, this is not necessary, since it ensures always use bounded memory, but there is no existing testing coverage for that, probably in a follow-up PR, we can enable ordered fast path even there is a memory limit.
Patch applied in c2ae7dc