Skip to content

Commit bf9042a

Browse files
jayshrivastavaalamb
authored andcommitted
Reapply "Add ExecutionPlan::apply_expressions() (apache#20337)" (apache#22437) (apache#24018)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Informs: apache#23814 - Informs: datafusion-contrib/datafusion-distributed#584 This change does not close the above issues because it does not implement a way to tell if a node is a producer dynamic filters. ## Rationale for this change See apache#23814 and datafusion-contrib/datafusion-distributed#553. To send dynamic filter updates across the network, there needs to be a way to get access to `PhysicalExpr` from `ExecutionPlan`. As discussed in apache#23814, the cleanest way to do this is to add `ExecutionPlan::apply_expressions`, which mirrors a similar method for logical plan nodes. ## What changes are included in this PR? There's 3 commits in this PR: Firstly, commit 1 re-applies the changes in apache#20337 (reverted in apache#22437). Some of the reasons for why the original PR was reverted include (a) `apply_expressions` is too complicated to implement and there's no concrete need to justify this complexity (b) there was no usage of `apply_expressions` inside this repo To address (a) - justification for adding this method is provided in apache#23814 - commit 2 in this PR adds helper methods `apply_expression_roots` and `apply_no_expressions` which abstract away the `TreeNodeRecursion` complexity from implementors. Now, `apply_expressions` very trivial to implement ex. ```rust fn apply_expressions( &self, f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>, ) -> Result<TreeNodeRecursion> { apply_expression_roots([&self.predicate_1], f) apply_expression_roots([&self.predicate_2], f) apply_expression_roots([&self.other_expression], f) } ``` - the method traverses over `&Arc<dyn PhysicalExpr>` rather than `&dyn PhysicalExpr` to reduce complexity around lifetimes To address (b): - commit 3 adds a usage of `apply_expressions` in `physical-plan/src/aggregates/mod.rs`. Previously, there was a hack that checked if a filter was pushed down using `Arc::strong_count(dyn_filter) > 1`. Now it uses `apply_expressions` - similarly, commit 4 removes `is_used` from dynamic filters which used to check Arc references counts to see if a filter was pushed down. Now, the hash join uses `apply_expressions` to find pushed down filters. ## Are these changes tested? Yes. ## Are there any user-facing changes? There's a new mandatory method `ExecutionPlan::apply_expressions()`. See the upgrading guide and documentation for details. --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent b9027fe commit bf9042a

81 files changed

Lines changed: 1781 additions & 96 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎datafusion-examples/examples/custom_data_source/custom_datasource.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ use datafusion::arrow::array::{UInt8Builder, UInt64Builder};
2727
use datafusion::arrow::datatypes::{DataType, Field, Schema, SchemaRef};
2828
use datafusion::arrow::record_batch::RecordBatch;
2929
use datafusion::common::assert_batches_eq;
30+
use datafusion::common::tree_node::TreeNodeRecursion;
3031
use datafusion::datasource::{TableProvider, TableType, provider_as_source};
3132
use datafusion::error::Result;
3233
use datafusion::execution::context::TaskContext;
@@ -314,4 +315,13 @@ impl ExecutionPlan for CustomExec {
314315
None,
315316
)?))
316317
}
318+
319+
fn apply_expressions(
320+
&self,
321+
_f: &mut dyn FnMut(
322+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
323+
) -> Result<TreeNodeRecursion>,
324+
) -> Result<TreeNodeRecursion> {
325+
Ok(TreeNodeRecursion::Continue)
326+
}
317327
}

‎datafusion-examples/examples/execution_monitoring/memory_pool_execution_plan.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
use arrow::array::record_batch;
3030
use arrow::record_batch::RecordBatch;
3131
use arrow_schema::SchemaRef;
32+
use datafusion::common::tree_node::TreeNodeRecursion;
3233
use datafusion::common::{exec_datafusion_err, internal_err};
3334
use datafusion::datasource::{DefaultTableSource, memory::MemTable};
3435
use datafusion::error::Result;
@@ -291,4 +292,13 @@ impl ExecutionPlan for BufferingExecutionPlan {
291292
}),
292293
)))
293294
}
295+
296+
fn apply_expressions(
297+
&self,
298+
_f: &mut dyn FnMut(
299+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
300+
) -> Result<TreeNodeRecursion>,
301+
) -> Result<TreeNodeRecursion> {
302+
Ok(TreeNodeRecursion::Continue)
303+
}
294304
}

‎datafusion-examples/examples/proto/composed_extension_codec.rs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ use std::sync::Arc;
3737

3838
use datafusion::common::Result;
3939
use datafusion::common::internal_err;
40+
use datafusion::common::tree_node::TreeNodeRecursion;
4041
use datafusion::execution::TaskContext;
4142
use datafusion::physical_plan::{DisplayAs, ExecutionPlan};
4243
use datafusion::prelude::SessionContext;
@@ -124,6 +125,15 @@ impl ExecutionPlan for ParentExec {
124125
) -> Result<datafusion::physical_plan::SendableRecordBatchStream> {
125126
unreachable!()
126127
}
128+
129+
fn apply_expressions(
130+
&self,
131+
_f: &mut dyn FnMut(
132+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
133+
) -> Result<TreeNodeRecursion>,
134+
) -> Result<TreeNodeRecursion> {
135+
Ok(TreeNodeRecursion::Continue)
136+
}
127137
}
128138

129139
/// A PhysicalExtensionCodec that can serialize and deserialize ParentExec
@@ -202,6 +212,15 @@ impl ExecutionPlan for ChildExec {
202212
) -> Result<datafusion::physical_plan::SendableRecordBatchStream> {
203213
unreachable!()
204214
}
215+
216+
fn apply_expressions(
217+
&self,
218+
_f: &mut dyn FnMut(
219+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
220+
) -> Result<TreeNodeRecursion>,
221+
) -> Result<TreeNodeRecursion> {
222+
Ok(TreeNodeRecursion::Continue)
223+
}
205224
}
206225

207226
/// A PhysicalExtensionCodec that can serialize and deserialize ChildExec

‎datafusion-examples/examples/relation_planner/table_sample.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ use datafusion::{
118118
};
119119
use datafusion_common::{
120120
DFSchemaRef, DataFusionError, Result, Statistics, internal_err, not_impl_err,
121-
plan_datafusion_err, plan_err,
121+
plan_datafusion_err, plan_err, tree_node::TreeNodeRecursion,
122122
};
123123
use datafusion_expr::physical_planning_context::PhysicalPlanningContext;
124124
use datafusion_expr::{
@@ -750,6 +750,15 @@ impl ExecutionPlan for SampleExec {
750750

751751
Ok(Arc::new(stats))
752752
}
753+
754+
fn apply_expressions(
755+
&self,
756+
_f: &mut dyn FnMut(
757+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
758+
) -> Result<TreeNodeRecursion>,
759+
) -> Result<TreeNodeRecursion> {
760+
Ok(TreeNodeRecursion::Continue)
761+
}
753762
}
754763

755764
/// Bernoulli sampler: includes each row with probability `(upper - lower)`.

‎datafusion/catalog/src/memory/table.rs‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ use arrow::compute::{and, filter_record_batch};
3131
use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
3232
use arrow::record_batch::RecordBatch;
3333
use datafusion_common::error::Result;
34+
use datafusion_common::tree_node::TreeNodeRecursion;
3435
use datafusion_common::{Constraints, DFSchema, SchemaExt, not_impl_err, plan_err};
3536
use datafusion_datasource::memory::{MemSink, MemorySourceConfig};
3637
use datafusion_datasource::sink::DataSinkExec;
@@ -39,13 +40,13 @@ use datafusion_expr::dml::InsertOp;
3940
use datafusion_expr::physical_planning_context::PhysicalPlanningContext;
4041
use datafusion_expr::{Expr, SortExpr, TableType};
4142
use datafusion_physical_expr::{
42-
LexOrdering, PhysicalExpr, create_physical_expr, create_physical_sort_exprs,
43+
LexOrdering, create_physical_expr, create_physical_sort_exprs,
4344
};
4445
use datafusion_physical_plan::repartition::RepartitionExec;
4546
use datafusion_physical_plan::stream::RecordBatchStreamAdapter;
4647
use datafusion_physical_plan::{
47-
DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
48-
collect_partitioned,
48+
DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PhysicalExpr,
49+
PlanProperties, collect_partitioned,
4950
};
5051
use datafusion_session::Session;
5152

@@ -597,4 +598,11 @@ impl ExecutionPlan for DmlResultExec {
597598
stream,
598599
)))
599600
}
601+
602+
fn apply_expressions(
603+
&self,
604+
_f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
605+
) -> Result<TreeNodeRecursion> {
606+
Ok(TreeNodeRecursion::Continue)
607+
}
600608
}

‎datafusion/core/src/physical_planner.rs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4950,6 +4950,13 @@ mod tests {
49504950
) -> Result<SendableRecordBatchStream> {
49514951
unimplemented!("NoOpExecutionPlan::execute");
49524952
}
4953+
4954+
fn apply_expressions(
4955+
&self,
4956+
_f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
4957+
) -> Result<TreeNodeRecursion> {
4958+
Ok(TreeNodeRecursion::Continue)
4959+
}
49534960
}
49544961

49554962
struct ExpressionExtensionPlanner;
@@ -5117,6 +5124,12 @@ digraph {
51175124
) -> Result<SendableRecordBatchStream> {
51185125
unimplemented!()
51195126
}
5127+
fn apply_expressions(
5128+
&self,
5129+
_f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
5130+
) -> Result<TreeNodeRecursion> {
5131+
Ok(TreeNodeRecursion::Continue)
5132+
}
51205133
}
51215134
impl DisplayAs for OkExtensionNode {
51225135
fn fmt_as(&self, _t: DisplayFormatType, f: &mut fmt::Formatter) -> fmt::Result {
@@ -5163,6 +5176,12 @@ digraph {
51635176
) -> Result<SendableRecordBatchStream> {
51645177
unimplemented!()
51655178
}
5179+
fn apply_expressions(
5180+
&self,
5181+
_f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
5182+
) -> Result<TreeNodeRecursion> {
5183+
Ok(TreeNodeRecursion::Continue)
5184+
}
51665185
}
51675186
impl DisplayAs for InvariantFailsExtensionNode {
51685187
fn fmt_as(&self, _t: DisplayFormatType, f: &mut fmt::Formatter) -> fmt::Result {
@@ -5287,6 +5306,12 @@ digraph {
52875306
) -> Result<SendableRecordBatchStream> {
52885307
unimplemented!()
52895308
}
5309+
fn apply_expressions(
5310+
&self,
5311+
_f: &mut dyn FnMut(&Arc<dyn PhysicalExpr>) -> Result<TreeNodeRecursion>,
5312+
) -> Result<TreeNodeRecursion> {
5313+
Ok(TreeNodeRecursion::Continue)
5314+
}
52905315
}
52915316
impl DisplayAs for ExecutableInvariantFails {
52925317
fn fmt_as(&self, _t: DisplayFormatType, f: &mut fmt::Formatter) -> fmt::Result {

‎datafusion/core/tests/custom_sources_cases/mod.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ use datafusion_catalog::Session;
3838
use datafusion_common::cast::as_primitive_array;
3939
use datafusion_common::project_schema;
4040
use datafusion_common::stats::Precision;
41+
use datafusion_common::tree_node::TreeNodeRecursion;
4142
use datafusion_physical_expr::EquivalenceProperties;
4243
use datafusion_physical_plan::PlanProperties;
4344
use datafusion_physical_plan::StatisticsArgs;
@@ -209,6 +210,15 @@ impl ExecutionPlan for CustomExecutionPlan {
209210
.collect(),
210211
}))
211212
}
213+
214+
fn apply_expressions(
215+
&self,
216+
_f: &mut dyn FnMut(
217+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
218+
) -> Result<TreeNodeRecursion>,
219+
) -> Result<TreeNodeRecursion> {
220+
Ok(TreeNodeRecursion::Continue)
221+
}
212222
}
213223

214224
#[async_trait]

‎datafusion/core/tests/custom_sources_cases/provider_filter_pushdown.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ use datafusion::prelude::*;
3535
use datafusion::scalar::ScalarValue;
3636
use datafusion_catalog::Session;
3737
use datafusion_common::cast::as_primitive_array;
38+
use datafusion_common::tree_node::TreeNodeRecursion;
3839
use datafusion_common::{DataFusionError, internal_err, not_impl_err};
3940
use datafusion_expr::expr::{BinaryExpr, Cast};
4041
use datafusion_functions_aggregate::expr_fn::count;
@@ -148,6 +149,15 @@ impl ExecutionPlan for CustomPlan {
148149
})),
149150
)))
150151
}
152+
153+
fn apply_expressions(
154+
&self,
155+
_f: &mut dyn FnMut(
156+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
157+
) -> Result<TreeNodeRecursion>,
158+
) -> Result<TreeNodeRecursion> {
159+
Ok(TreeNodeRecursion::Continue)
160+
}
151161
}
152162

153163
#[derive(Clone, Debug)]

‎datafusion/core/tests/custom_sources_cases/statistics.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ use datafusion::{
3333
scalar::ScalarValue,
3434
};
3535
use datafusion_catalog::Session;
36+
use datafusion_common::tree_node::TreeNodeRecursion;
3637
use datafusion_common::{project_schema, stats::Precision};
3738
use datafusion_physical_expr::EquivalenceProperties;
3839
use datafusion_physical_plan::execution_plan::{Boundedness, EmissionType};
@@ -185,6 +186,15 @@ impl ExecutionPlan for StatisticsValidation {
185186
Ok(Arc::new(self.stats.clone()))
186187
}
187188
}
189+
190+
fn apply_expressions(
191+
&self,
192+
_f: &mut dyn FnMut(
193+
&Arc<dyn datafusion::physical_plan::PhysicalExpr>,
194+
) -> Result<TreeNodeRecursion>,
195+
) -> Result<TreeNodeRecursion> {
196+
Ok(TreeNodeRecursion::Continue)
197+
}
188198
}
189199

190200
fn init_ctx(stats: Statistics, schema: Schema) -> Result<SessionContext> {

‎datafusion/core/tests/fuzz_cases/once_exec.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
use arrow_schema::SchemaRef;
1919
use datafusion_common::internal_datafusion_err;
20+
use datafusion_common::tree_node::TreeNodeRecursion;
2021
use datafusion_execution::{SendableRecordBatchStream, TaskContext};
2122
use datafusion_physical_expr::{EquivalenceProperties, Partitioning};
2223
use datafusion_physical_plan::execution_plan::{Boundedness, EmissionType};
@@ -105,4 +106,13 @@ impl ExecutionPlan for OnceExec {
105106

106107
stream.ok_or_else(|| internal_datafusion_err!("Stream already consumed"))
107108
}
109+
110+
fn apply_expressions(
111+
&self,
112+
_f: &mut dyn FnMut(
113+
&Arc<dyn datafusion_physical_plan::PhysicalExpr>,
114+
) -> datafusion_common::Result<TreeNodeRecursion>,
115+
) -> datafusion_common::Result<TreeNodeRecursion> {
116+
Ok(TreeNodeRecursion::Continue)
117+
}
108118
}

0 commit comments

Comments
 (0)