Repository navigation
support join conditions in where clause #892
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
tpch query 8 is taking forever to execute in ballista because its subquery is planned as a cross join across 8 tables.
Describe the solution you'd like
The join conditions for query 8 are specified in the where clause. Our optimizer should pick this up and rewrite the part of the cross join into inner join.
Describe alternatives you've considered
Handle the rewrite in sql planner, but this will won't benefit dataframe users.
Additional context
Surfaced from #891.
Activity
xudong963 commentedon Oct 9, 2021
@houqp I want to try it!
xudong963 commentedon Nov 13, 2021
If I read right, the issue has been solved? https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/sql/planner.rs#L691
houqp commentedon Nov 13, 2021
You are right @xudong963. I think there is a bug in that rewrite that resulted in inefficient cross join in an edge-case. I filed #1293 to track that problem instead.
perf: Optimize CometSparkToColumnar for columnar input (apache#892)
Add physical and logical plan conversion to and from protobuf (apache…