Repository navigation
Commit 2bfdd4a
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Informs: apache#23814
- Informs:
datafusion-contrib/datafusion-distributed#584
This change does not close the above issues because it does not
implement a way to tell if a node is a producer dynamic filters.
## Rationale for this change
See apache#23814 and
datafusion-contrib/datafusion-distributed#553.
To send dynamic filter updates across the network, there needs to be a
way to get access to `PhysicalExpr` from `ExecutionPlan`. As discussed
in apache#23814, the cleanest way
to do this is to add `ExecutionPlan::apply_expressions`, which mirrors a
similar method for logical plan nodes.
## What changes are included in this PR?
There's 3 commits in this PR:
Firstly, commit 1 re-applies the changes in
apache#20337 (reverted in
apache#22437).
Some of the reasons for why the original PR was reverted include
(a) `apply_expressions` is too complicated to implement and there's no
concrete need to justify this complexity
(b) there was no usage of `apply_expressions` inside this repo
To address (a)
- justification for adding this method is provided in
apache#23814
- commit 2 in this PR adds helper methods `apply_expression_roots` and
`apply_no_expressions` which abstract away the `TreeNodeRecursion`
complexity from implementors. Now, `apply_expressions` very trivial to
implement ex.
```rust
fn apply_expressions(
&self,
f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
) -> Result<TreeNodeRecursion> {
apply_expression_roots([&self.predicate_1], f)
apply_expression_roots([&self.predicate_2], f)
apply_expression_roots([&self.other_expression], f)
}
```
- the method traverses over `&Arc<dyn PhysicalExpr>` rather than `&dyn
PhysicalExpr` to reduce complexity around lifetimes
To address (b):
- commit 3 adds a usage of `apply_expressions` in
`physical-plan/src/aggregates/mod.rs`. Previously, there was a hack that
checked if a filter was pushed down using `Arc::strong_count(dyn_filter)
> 1`. Now it uses `apply_expressions`
- similarly, commit 4 removes `is_used` from dynamic filters which used
to check Arc references counts to see if a filter was pushed down. Now,
the hash join uses `apply_expressions` to find pushed down filters.
## Are these changes tested?
Yes.
## Are there any user-facing changes?
There's a new mandatory method `ExecutionPlan::apply_expressions()`. See
the upgrading guide and documentation for details.
---------
Co-authored-by: Andrew Lamb <[email protected]>
1 parent 308e212 commit 2bfdd4a
81 files changed
Lines changed: 1781 additions & 96 deletions
File tree
- datafusion-examples/examples
- custom_data_source
- execution_monitoring
- proto
- relation_planner
- datafusion
- catalog/src/memory
- core
- src
- tests
- custom_sources_cases
- fuzz_cases
- physical_optimizer
- user_defined
- datasource-arrow/src
- datasource-avro/src
- datasource-csv/src
- datasource-json/src
- datasource-parquet/src
- datasource/src
- file_scan_config
- ffi
- src
- physical_expr
- tests
- tests
- physical-expr/src/expressions/dynamic_filters
- physical-optimizer/src
- physical-plan
- benches
- src
- aggregates
- joins
- hash_join
- piecewise_merge_join
- sort_merge_join
- operator_statistics
- repartition
- sorts
- test
- windows
- proto
- src/physical_plan
- tests/cases
- docs/source/library-user-guide
- upgrading
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
314 | 315 | | |
315 | 316 | | |
316 | 317 | | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
317 | 327 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
291 | 292 | | |
292 | 293 | | |
293 | 294 | | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
294 | 304 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| |||
124 | 125 | | |
125 | 126 | | |
126 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
127 | 137 | | |
128 | 138 | | |
129 | 139 | | |
| |||
202 | 212 | | |
203 | 213 | | |
204 | 214 | | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
205 | 224 | | |
206 | 225 | | |
207 | 226 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
| 121 | + | |
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
| |||
750 | 750 | | |
751 | 751 | | |
752 | 752 | | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
753 | 762 | | |
754 | 763 | | |
755 | 764 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
42 | | - | |
| 43 | + | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
47 | | - | |
48 | | - | |
| 48 | + | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
597 | 598 | | |
598 | 599 | | |
599 | 600 | | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
600 | 608 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4950 | 4950 | | |
4951 | 4951 | | |
4952 | 4952 | | |
| 4953 | + | |
| 4954 | + | |
| 4955 | + | |
| 4956 | + | |
| 4957 | + | |
| 4958 | + | |
| 4959 | + | |
4953 | 4960 | | |
4954 | 4961 | | |
4955 | 4962 | | |
| |||
5117 | 5124 | | |
5118 | 5125 | | |
5119 | 5126 | | |
| 5127 | + | |
| 5128 | + | |
| 5129 | + | |
| 5130 | + | |
| 5131 | + | |
| 5132 | + | |
5120 | 5133 | | |
5121 | 5134 | | |
5122 | 5135 | | |
| |||
5163 | 5176 | | |
5164 | 5177 | | |
5165 | 5178 | | |
| 5179 | + | |
| 5180 | + | |
| 5181 | + | |
| 5182 | + | |
| 5183 | + | |
| 5184 | + | |
5166 | 5185 | | |
5167 | 5186 | | |
5168 | 5187 | | |
| |||
5287 | 5306 | | |
5288 | 5307 | | |
5289 | 5308 | | |
| 5309 | + | |
| 5310 | + | |
| 5311 | + | |
| 5312 | + | |
| 5313 | + | |
| 5314 | + | |
5290 | 5315 | | |
5291 | 5316 | | |
5292 | 5317 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| |||
209 | 210 | | |
210 | 211 | | |
211 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
212 | 222 | | |
213 | 223 | | |
214 | 224 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
148 | 149 | | |
149 | 150 | | |
150 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
151 | 161 | | |
152 | 162 | | |
153 | 163 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
185 | 186 | | |
186 | 187 | | |
187 | 188 | | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
188 | 198 | | |
189 | 199 | | |
190 | 200 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
105 | 106 | | |
106 | 107 | | |
107 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
108 | 118 | | |
0 commit comments