Repository navigation
Implement GroupsAccumulator for array_agg aggregation function #10145
Description
Is your feature request related to a problem or challenge?
Based on the article Aggregating Millions of Groups Fast in Apache Arrow DataFusion it's possible to optimise aggregation functions using GroupsAccumulator.
Describe the solution you'd like
Implement GroupsAccumulator for array_agg aggregation function
Describe alternatives you've considered
No response
Additional context
No response
Activity
alamb commentedon Sep 29, 2025
Also related is the memory use in
vegarsti commentedon Sep 29, 2025
take
alamb commentedon Sep 29, 2025
I believe @duongcongtoai may be working on this too -- see #17446 (comment)
It would probably be a good idea to coordinate with him
vegarsti commentedon Sep 29, 2025
Oh oops, I didn't realize that's what he was doing. Thanks!
vegarsti commentedon Sep 29, 2025
I can't see a way to unassign myself. Sorry about that. Feel free to unassign.
Are there any others you can think of that are missing GroupsAccumulator? I couldn't find any on the issue tracker.
vegarsti commentedon Sep 29, 2025
Indeed he is! #17446 (comment) Sorry!
alamb commentedon Oct 1, 2025
You can look for anything that implements
AggregateUDFImplthat doesn't return GroupsAccumulatorhttps://github.com/search?q=repo%3Aapache%2Fdatafusion+%22impl+AggregateUDFImpl%22&type=code
There are a bunch of them, but given the complexity of implementing and testing GroupsAccumulator I think it would make sense to pick aggregate functions that we think are important to invest in (both code and review). For example, any aggregate functions that show up in benchmarks
alamb commentedon Feb 24, 2026
array_agg, Polars is 4 times faster for equal query #17446array_agg, Polars is 4 times faster for equal query #17446