Repository navigation
Support RangePartitioning with native shuffle #458
Description
What is the problem the feature request solves?
We do not currently support RangePartitioning with native shuffle.
Adding this support would allow us to use native shuffle for more queries, including queries where there is a global sort.
Describe the potential solution
No response
Additional context
No response
Activity
viirya commentedon May 22, 2024
Since the early development of Comet shuffle, this is one item on to do list. After we finish columnar shuffle which has higher coverage and support main partitioning types, we turn our focus to other more urgent items. Because we want to increase query coverage as much as possible. Currently we can run TPC-H queries mostly native, seems we can go back to look at this as it can give us more performance gain on shuffle (because native shuffle is faster than columnar shuffle).
viirya commentedon May 22, 2024
I will work on this after finishing HashJoin build right support.
andygrove commentedon Oct 3, 2024
This would help with TPC-H performance, so I plan on taking a look at this issue soon
jinwenjie123 commentedon Mar 28, 2025
Hi @andygrove
May I ask why we decide not support RangePartitioning ? and will it be supported in the near future ?
Thanks
andygrove commentedon Mar 29, 2025
We didn't decide not to support it.
It just hasn't been implemented yet because it is more complex than other partitioning schemes. It would be great if someone could submit a PR to add support for this. It could really help with benchmark results and performance overall.
andygrove commentedon Mar 29, 2025
I discussed this feature with @mbutrovich recently and he may have additional thoughts on this topic.
viirya commentedon Mar 29, 2025
The implementation issue or difference for RangePartitioning other than other partitioning like HashPartitioning, is that it involves some sampling operations that perform with RDD in Spark. That's said Comet lib at each executor cannot determine the sampling results separately like we did for general query execution.
chore: Change defaults to enable all execs (apache#458)
andygrove commentedon Jan 30, 2026
Implemented in #1862, with follow-up fixes in #1907, #2258, and #2716. Closing.