You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
NestedLoopJoin::swap_inputs should check number of right partitions before swapping inputs #17358
NestedLoopJoin left side has to have single partition, there is a check for this:
fnexecute(&self,partition:usize,context:Arc<TaskContext>,) -> Result<SendableRecordBatchStream>{ifself.left.output_partitioning().partition_count() != 1{returninternal_err!("Invalid NestedLoopJoinExec, the output partition count of the left child must be 1,\ consider using CoalescePartitionsExec or the EnforceDistribution rule");}
...
}
but no check has been performed when join is created with NestedLoopJoinExec::try_new nor this is checked in NestedLoopJoin::swap_inputs.
I'm not sure if it is expected that additional PhysicalOptimizerRule which will insert CoalescePartitionsExec automatically (like EnforceDistribution) or this is a bug in NLJ .
To Reproduce
No response
Expected behavior
I believe this check should be performed when NLJ is created. Delaying check until execute(...) may be a bit too late.
Also, NLJ::swap_inputs should fail to swap left and right input if it will break this rule
Lastly, JoinSelection should not perform join re-order if right input has more than single partition.
Ballista performs join re-ordering based on shuffle file information, before stage runs.
At that point all stages have been created so partitioning can't really change. Swapping join inputs and inserting CoalescePartitionsExec wont work for ballista. If this is expected behaviour, ballista should disable this rule
EDIT: I can't find any physical rule which would "fix" left side with CoalescePartitionsExec
changed the title [-]`NestedLoopJoin::swap_inputs` should check right partition number before swapping[/-][+]`NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs[/+]on Aug 30, 2025
After looking at this issue I believe, this is NPT a datafusion bug, EnforceDistribution will add CoalescePartitionsExec so, proposed checks can't be done before execute. This is ballista issue and how it uses JoinSelection
Activity
[-]`NestedLoopJoin::swap_inputs` should check right partition number before swapping[/-][+]`NestedLoopJoin::swap_inputs` should check number of right partitions before swapping inputs[/+]jonathanc-n commentedon Aug 30, 2025
take
milenkovicm commentedon Aug 31, 2025
@jonathanc-n before you start working on this, we need to establish if this is a bug or not, I'm not sure
milenkovicm commentedon Aug 31, 2025
After looking at this issue I believe, this is NPT a datafusion bug,
EnforceDistributionwill addCoalescePartitionsExecso, proposed checks can't be done beforeexecute. This is ballista issue and how it usesJoinSelectionI apologise for the noise!