Repository navigation
Fix push_down_filter for children with non-empty fetch fields #21057
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
475de9f
0e83891
db84bf8
79bdcac
cf5ab10
b87fd67
eee6542
2a42cf0
6601e44
2ac6b09
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -796,6 +796,12 @@ impl OptimizerRule for PushDownFilter { | |
| filter.predicate = new_predicate; | ||
| } | ||
|
|
||
| // If the child has a fetch (limit), pushing a filter below it would | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
| // change semantics: the limit should apply before the filter, not after. | ||
| if filter.input.fetch().is_some() { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we should also either explicitly check here for Limit or (better yet) change
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this makes a lot of sense, the reason I initially skipped |
||
| return Ok(Transformed::no(LogicalPlan::Filter(filter))); | ||
| } | ||
|
|
||
| match Arc::unwrap_or_clone(filter.input) { | ||
| LogicalPlan::Filter(child_filter) => { | ||
| let parents_predicates = split_conjunction_owned(filter.predicate); | ||
|
|
@@ -832,13 +838,6 @@ impl OptimizerRule for PushDownFilter { | |
| insert_below(LogicalPlan::Distinct(distinct), new_filter) | ||
| } | ||
| LogicalPlan::Sort(sort) => { | ||
| // If the sort has a fetch (limit), pushing a filter below | ||
| // it would change semantics: the limit should apply before | ||
| // the filter, not after. | ||
| if sort.fetch.is_some() { | ||
| filter.input = Arc::new(LogicalPlan::Sort(sort)); | ||
| return Ok(Transformed::no(LogicalPlan::Filter(filter))); | ||
| } | ||
| let new_filter = | ||
| Filter::try_new(filter.predicate, Arc::clone(&sort.input)) | ||
| .map(LogicalPlan::Filter)?; | ||
|
|
@@ -1137,13 +1136,6 @@ impl OptimizerRule for PushDownFilter { | |
| } | ||
| LogicalPlan::Join(join) => push_down_join(join, Some(&filter.predicate)), | ||
| LogicalPlan::TableScan(scan) => { | ||
| // If the scan has a fetch (limit), pushing filters into it | ||
| // would change semantics: the limit should apply before the | ||
| // filter, not after. | ||
| if scan.fetch.is_some() { | ||
| filter.input = Arc::new(LogicalPlan::TableScan(scan)); | ||
| return Ok(Transformed::no(LogicalPlan::Filter(filter))); | ||
| } | ||
| let filter_predicates = split_conjunction(&filter.predicate); | ||
|
|
||
| let (volatile_filters, non_volatile_filters): (Vec<&Expr>, Vec<&Expr>) = | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am surprised LogicalPlan::Limit does not return a value here as well;
I looked into the code a bit to try and understand why Limits were needed. Initially I thought it was because the following code claims that we only push down Filters after Limits
datafusion/datafusion/optimizer/src/optimizer.rs
Lines 297 to 299 in ee24f3c
However, at some point we started running multiple optimizer passes so
PushdownFiltercan (effectively) run afterPushDownLimitdatafusion/datafusion/optimizer/src/optimizer.rs
Lines 386 to 402 in ee24f3c
So this bug was probably introduced when we started running multiple optimizer passes