Repository navigation
[EPIC] Add support for all join operators #12
Description
This is an umbrella ticket for adding Join support to Comet. In Spark, there are basically three types of Join operators: BroadcastJoin, HashJoin, SortMergeJoin. In DataFusion, two Join operators are supported: HashJoin, SortMergeJoin (experimental).
We are going to delegate Spark Join operators to correspond DataFusion Join operators. We will also go to improve DataFusion Join operators if needed.
SortMergeJoin
- Relax join keys constraint from Column to any physical expression for physical join operators Relax join keys constraint from Column to any physical expression for physical join operators datafusion#8991
- Support join filter for SortMergeJoin Support join filter for
SortMergeJoindatafusion#9080 - Add spilling in SortMergeJoin Add spilling in SortMergeJoin datafusion#9359
- Support SortMergeJoin operator - Support sort merge join #177
...
HashJoin
- Support HashJoin operator - Support HashJoin operator #193
BroadcastJoin
- Add CometBroadcastExchangeExec support broadcasting the result of Comet native operator - Add CometBroadcastExchangeExec to support broadcasting the result of Comet native operator #81
- Support BroadcastHashJoinExec Support BroadcastHashJoinExec #202
- Enable Comet broadcast by default Enable Comet broadcast by default #212
Other Join operators (Cross Join)
- Support CartesianProductExec - Support CartesianProductExec #199
- Support BroadcastNestedLoopJoinExec - Support BroadcastNestedLoopJoinExec #198
Reacted by Daniël Heres, Prashant Singh, Sonny and Peter Lee
Activity
mwiewior commentedon Feb 15, 2024
Hey @viirya - it's really awesome ! Do you happen to know when approx these BHJ and shuffle components will be open sourced?
viirya commentedon Feb 15, 2024
Hi @mwiewior
I will open source shuffle components soon. For join support, it is in development now. We already have working code but get some test failure on Spark tests. Once these test failures are resolved, we will open source it.
viirya commentedon Feb 15, 2024
For shuffle, we just remove it from the initial open source code. So the code is ready, we only need to add it back to Comet.
mwiewior commentedon Feb 15, 2024
Great news ! Thx !
[-]Join support[/-][+][EPIC] Add support for all join operators[/+]fix: address all remaining code review items
coderfender commentedon May 6, 2026
#198 #199
coderfender commentedon Jun 12, 2026
I will be working on #199 next
coderfender commentedon Jul 13, 2026
Working on cartesian join. Fixing errors at the moment
judahrand commentedon Oct 8, 2026
Could we add Datafusion's new ASOF join to the list? apache/datafusion#318
This would, I suppose, replace Spark's
SortMergeAsOfJoinExecoperator.