Repository navigation
Add the missing equivalence info for filter pushdown #16686
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,8 +35,11 @@ use crate::file_scan_config::FileScanConfig; | |
| use datafusion_common::config::ConfigOptions; | ||
| use datafusion_common::{Constraints, Result, Statistics}; | ||
| use datafusion_execution::{SendableRecordBatchStream, TaskContext}; | ||
| use datafusion_physical_expr::{EquivalenceProperties, Partitioning, PhysicalExpr}; | ||
| use datafusion_physical_expr::{ | ||
| conjunction, EquivalenceProperties, Partitioning, PhysicalExpr, | ||
| }; | ||
| use datafusion_physical_expr_common::sort_expr::LexOrdering; | ||
| use datafusion_physical_plan::filter::collect_columns_from_predicate; | ||
| use datafusion_physical_plan::filter_pushdown::{ | ||
| ChildPushdownResult, FilterPushdownPhase, FilterPushdownPropagation, | ||
| }; | ||
|
|
@@ -325,6 +328,9 @@ impl ExecutionPlan for DataSourceExec { | |
| new_node.data_source = data_source; | ||
| new_node.cache = | ||
| Self::compute_properties(Arc::clone(&new_node.data_source)); | ||
| // Add the missing filters' equivalence info when filters pushdown is applied | ||
| let filter = conjunction(res.filters.collect_supported()); | ||
| new_node = new_node.add_filter_equivalence_info(filter)?; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What happens if the node already had equivalence info / old filters? Practically speaking this shouldn't happen, but it is technically possible for a data source to have some filters passed in directly (e.g. in a TableProvider) and some filters passed in during pushdown. I think this is correct because it's adding the new filters and when we built the new node we did a conjunction.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this will add the eq info from filter pushdown into the existing eq info group from Maybe it's better if we have a test case for it, but I'm not sure how to create a |
||
| Ok(FilterPushdownPropagation { | ||
| filters: res.filters, | ||
| updated_node: Some(Arc::new(new_node)), | ||
|
|
@@ -372,6 +378,20 @@ impl DataSourceExec { | |
| self | ||
| } | ||
|
|
||
| /// Add filters' equivalence info | ||
| fn add_filter_equivalence_info( | ||
| mut self, | ||
| filter: Arc<dyn PhysicalExpr>, | ||
| ) -> Result<Self> { | ||
| let (equal_pairs, _) = collect_columns_from_predicate(&filter); | ||
| for (lhs, rhs) in equal_pairs { | ||
| self.cache | ||
| .eq_properties | ||
| .add_equal_conditions(Arc::clone(lhs), Arc::clone(rhs))? | ||
| } | ||
| Ok(self) | ||
| } | ||
|
Comment on lines
+382
to
+393
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This part I'm not familiar with and would appreciate if someone else took a look at and confirm is correct.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Basically, it's enough, lgtm |
||
|
|
||
| fn compute_properties(data_source: Arc<dyn DataSource>) -> PlanProperties { | ||
| PlanProperties::new( | ||
| data_source.eq_properties(), | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -148,10 +148,8 @@ logical_plan | |
| 03)----Filter: t_pushdown.b = Int32(2) | ||
| 04)------TableScan: t_pushdown projection=[a, b], partial_filters=[t_pushdown.b = Int32(2)] | ||
| physical_plan | ||
| 01)ProjectionExec: expr=[a@0 as a] | ||
| 02)--SortPreservingMergeExec: [b@1 ASC NULLS LAST] | ||
| 03)----SortExec: expr=[b@1 ASC NULLS LAST], preserve_partitioning=[true] | ||
| 04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
| 01)CoalescePartitionsExec | ||
| 02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
|
|
||
| # If we set the setting to `true` it override's the table's setting | ||
| statement ok | ||
|
|
@@ -220,10 +218,8 @@ logical_plan | |
| 03)----Filter: t.b = Int32(2) | ||
| 04)------TableScan: t projection=[a, b], partial_filters=[t.b = Int32(2)] | ||
| physical_plan | ||
| 01)ProjectionExec: expr=[a@0 as a] | ||
| 02)--SortPreservingMergeExec: [b@1 ASC NULLS LAST] | ||
| 03)----SortExec: expr=[b@1 ASC NULLS LAST], preserve_partitioning=[true] | ||
| 04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
| 01)CoalescePartitionsExec | ||
| 02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
|
|
||
| query TT | ||
| EXPLAIN select a from t_pushdown where b = 2 ORDER BY b; | ||
|
|
@@ -234,10 +230,8 @@ logical_plan | |
| 03)----Filter: t_pushdown.b = Int32(2) | ||
| 04)------TableScan: t_pushdown projection=[a, b], partial_filters=[t_pushdown.b = Int32(2)] | ||
| physical_plan | ||
| 01)ProjectionExec: expr=[a@0 as a] | ||
| 02)--SortPreservingMergeExec: [b@1 ASC NULLS LAST] | ||
| 03)----SortExec: expr=[b@1 ASC NULLS LAST], preserve_partitioning=[true] | ||
| 04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
| 01)CoalescePartitionsExec | ||
| 02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
|
|
||
| # If we reset the default the table created without pushdown goes back to disabling it | ||
| statement ok | ||
|
|
@@ -325,10 +319,8 @@ logical_plan | |
| 03)----Filter: t_pushdown.b = Int32(2) | ||
| 04)------TableScan: t_pushdown projection=[a, b], partial_filters=[t_pushdown.b = Int32(2)] | ||
| physical_plan | ||
| 01)ProjectionExec: expr=[a@0 as a] | ||
| 02)--SortPreservingMergeExec: [b@1 ASC NULLS LAST] | ||
| 03)----SortExec: expr=[b@1 ASC NULLS LAST], preserve_partitioning=[true] | ||
| 04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
| 01)CoalescePartitionsExec | ||
| 02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
|
|
||
| # When filter pushdown *is* enabled, ParquetExec can filter exactly, | ||
| # not just metadata, so we expect to see no FilterExec | ||
|
|
@@ -371,10 +363,8 @@ logical_plan | |
| 03)----Filter: t_pushdown.b = Int32(2) | ||
| 04)------TableScan: t_pushdown projection=[a, b], partial_filters=[t_pushdown.b = Int32(2)] | ||
| physical_plan | ||
| 01)ProjectionExec: expr=[a@0 as a] | ||
| 02)--SortPreservingMergeExec: [b@1 ASC NULLS LAST] | ||
| 03)----SortExec: expr=[b@1 ASC NULLS LAST], preserve_partitioning=[true] | ||
| 04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a, b], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
| 01)CoalescePartitionsExec | ||
| 02)--DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/parquet_filter_pushdown/parquet_table/2.parquet]]}, projection=[a], file_type=parquet, predicate=b@1 = 2, pruning_predicate=b_null_count@2 != row_count@3 AND b_min@0 <= 2 AND 2 <= b_max@1, required_guarantees=[b in (2)] | ||
|
|
||
|
Comment on lines
+352
to
+368
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @liamzwbao could you help me understand what I should be looking for in this output / what proves that this fix works / what would have been different before this fix?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The 1st commit reproduce the issue, and the 2nd commit reflect the fix. Basically, the SortExec which matches the filter equivalence gets removed after the fix. |
||
| # also test querying on columns that are not in all the files | ||
| query T | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.