Skip to content

Commit caa3cb4

Browse files
JSOD11alamb
andauthored
refactor(physical-plan): Simplify ExecutionPlan API with replace_children (apache#23903)
## Which issue does this PR close? - Closes apache#23441 ## User-facing changes: Deprecating `with_new_children` and `with_new_children_and_same_properties` in favor of `replace_children` As noted [here](apache#23332 (comment)), while the addition of `with_new_children_and_same_properties` has the benefit of skipping potentially expensive computation in the case that replacement children have the same properties as the original children, it widens the API surface area of `ExecutionPlan` in a way that could be confusing for users. Thus, to rectify this, we unify these methods by introducing `replace_children`, and we shift towards `with_new_children_if_necessary` as the universal entry point for replacing the children of an `ExecutionPlan`. `replace_children` simplifies the interface for users by taking an enum called `ChildrenPropertiesHint` as an argument. The enum has two variants, `SameProperties` and `Recompute`, which function as a hint to `replace_children` from the caller as to whether or not the properties need to be recomputed. ## Trait implementation migration To migrate from `with_new_children` and `with_new_children_and_same_properties` to `replace_children`, I went through all 93 implementations of `with_new_children` and implemented `replace_children` with a `match` statement matching on the `ChildrenPropertiesHint`. In the case that the properties match, `ChildrenPropertiesHint::SameProperties`, and we have an implementation of `with_new_children_and_same_properties`, then we follow the body of `with_new_children_and_same_properties`. In the case that the properties do not match, `ChildrenPropertiesHint::Recompute`, we follow the body of `with_new_children`. In the cases in which there was no implementation of `with_new_children_and_same_properties`, I simply move the body of `with_new_children` into `replace_children` and ignore the hint. I mark `with_new_children` and `with_new_children_and_same_properties` as deprecated with a migration note pointing to `replace_children`. After a couple releases, we'll drop the deprecated methods. ## Example For example, here is what the implementation looks like for `FilterExec` after this change: ``` fn replace_children( self: Arc<Self>, mut children: Vec<Arc<dyn ExecutionPlan>>, hint: ChildrenPropertiesHint, ) -> Result<Arc<dyn ExecutionPlan>> { validate_child_count!(self, children); match hint { ChildrenPropertiesHint::SameProperties => Ok(Arc::new(Self { input: children.swap_remove(0), metrics: ExecutionPlanMetricsSet::new(), ..Self::clone(&*self) })), ChildrenPropertiesHint::Recompute => { let new_input = children.swap_remove(0); FilterExecBuilder::from(&*self) .with_input(new_input) .build() .map(|e| Arc::new(e) as _) } } } ``` We see here that in the case that the hint suggests the properties are the same, we can simply swap the children without having to recompute the properties. In the case that the properties are not the same, we create a new node from scratch. We achieve this functionality by moving the hint calculations definitively into `with_new_children_if_necessary` rather than having them scattered around many methods. However, for this to all work we must ensure that users actually do use `with_new_children_if_necessary` by making it obvious to them somehow. I feel `replace_children` is a step in the right direction, but it could still be easy for a user to miss `with_new_children_if_necessary` and just jump to using `replace_children` instead. ## Usage Migration `replace_children` is called from `with_new_children_if_necessary`, which is the standard entry point that should be used for replacing the children of a node. To model the intended behavior for our users, I took the time here to migrate usages of `with_new_children` and `with_children_and_same_properties` to `with_new_children_if_necessary` where it made sense to do so, and I migrated `with_new_children_if_necessary` to use `replace_children` with the correct hint filled in at each branch. ## Testing - `cargo fmt --all` - `cargo check -p datafusion-physical-plan` - `cargo check -p datafusion-physical-optimizer` - `cargo check -p datafusion --lib` - `cargo check -p datafusion-ffi` - CI passing --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent 6459afe commit caa3cb4

76 files changed

Lines changed: 2371 additions & 764 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎datafusion-examples/examples/custom_data_source/custom_datasource.rs‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,8 @@ use datafusion::physical_expr::EquivalenceProperties;
3636
use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
3737
use datafusion::physical_plan::memory::MemoryStream;
3838
use datafusion::physical_plan::{
39-
DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
40-
SendableRecordBatchStream, project_schema,
39+
ChildrenPropertiesMode, DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning,
40+
PlanProperties, ReplaceChildrenOptions, SendableRecordBatchStream, project_schema,
4141
};
4242
use datafusion::prelude::*;
4343

@@ -268,13 +268,24 @@ impl ExecutionPlan for CustomExec {
268268
vec![]
269269
}
270270

271-
fn with_new_children(
271+
fn replace_children(
272272
self: Arc<Self>,
273273
_: Vec<Arc<dyn ExecutionPlan>>,
274+
_: ReplaceChildrenOptions,
274275
) -> Result<Arc<dyn ExecutionPlan>> {
275276
Ok(self)
276277
}
277278

279+
fn with_new_children(
280+
self: Arc<Self>,
281+
children: Vec<Arc<dyn ExecutionPlan>>,
282+
) -> Result<Arc<dyn ExecutionPlan>> {
283+
self.replace_children(
284+
children,
285+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
286+
)
287+
}
288+
278289
fn execute(
279290
&self,
280291
_partition: usize,

‎datafusion-examples/examples/execution_monitoring/memory_pool_execution_plan.rs‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,8 @@ use datafusion::execution::{SendableRecordBatchStream, TaskContext};
3939
use datafusion::logical_expr::LogicalPlanBuilder;
4040
use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
4141
use datafusion::physical_plan::{
42-
DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties,
42+
ChildrenPropertiesMode, DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties,
43+
ReplaceChildrenOptions,
4344
};
4445
use datafusion::prelude::*;
4546
use futures::stream::{StreamExt, TryStreamExt};
@@ -237,9 +238,10 @@ impl ExecutionPlan for BufferingExecutionPlan {
237238
vec![&self.input]
238239
}
239240

240-
fn with_new_children(
241+
fn replace_children(
241242
self: Arc<Self>,
242243
children: Vec<Arc<dyn ExecutionPlan>>,
244+
_options: ReplaceChildrenOptions,
243245
) -> Result<Arc<dyn ExecutionPlan>> {
244246
if children.len() == 1 {
245247
Ok(Arc::new(BufferingExecutionPlan::new(
@@ -251,6 +253,16 @@ impl ExecutionPlan for BufferingExecutionPlan {
251253
}
252254
}
253255

256+
fn with_new_children(
257+
self: Arc<Self>,
258+
children: Vec<Arc<dyn ExecutionPlan>>,
259+
) -> Result<Arc<dyn ExecutionPlan>> {
260+
self.replace_children(
261+
children,
262+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
263+
)
264+
}
265+
254266
fn execute(
255267
&self,
256268
partition: usize,

‎datafusion-examples/examples/proto/composed_extension_codec.rs‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ use datafusion::common::Result;
3939
use datafusion::common::internal_err;
4040
use datafusion::common::tree_node::TreeNodeRecursion;
4141
use datafusion::execution::TaskContext;
42+
use datafusion::physical_plan::{ChildrenPropertiesMode, ReplaceChildrenOptions};
4243
use datafusion::physical_plan::{DisplayAs, ExecutionPlan};
4344
use datafusion::prelude::SessionContext;
4445
use datafusion_proto::physical_plan::{
@@ -111,13 +112,24 @@ impl ExecutionPlan for ParentExec {
111112
vec![&self.input]
112113
}
113114

114-
fn with_new_children(
115+
fn replace_children(
115116
self: Arc<Self>,
116-
_children: Vec<Arc<dyn ExecutionPlan>>,
117+
_: Vec<Arc<dyn ExecutionPlan>>,
118+
_: ReplaceChildrenOptions,
117119
) -> Result<Arc<dyn ExecutionPlan>> {
118120
unreachable!()
119121
}
120122

123+
fn with_new_children(
124+
self: Arc<Self>,
125+
children: Vec<Arc<dyn ExecutionPlan>>,
126+
) -> Result<Arc<dyn ExecutionPlan>> {
127+
self.replace_children(
128+
children,
129+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
130+
)
131+
}
132+
121133
fn execute(
122134
&self,
123135
_partition: usize,
@@ -198,13 +210,24 @@ impl ExecutionPlan for ChildExec {
198210
vec![]
199211
}
200212

201-
fn with_new_children(
213+
fn replace_children(
202214
self: Arc<Self>,
203-
_children: Vec<Arc<dyn ExecutionPlan>>,
215+
_: Vec<Arc<dyn ExecutionPlan>>,
216+
_: ReplaceChildrenOptions,
204217
) -> Result<Arc<dyn ExecutionPlan>> {
205218
unreachable!()
206219
}
207220

221+
fn with_new_children(
222+
self: Arc<Self>,
223+
children: Vec<Arc<dyn ExecutionPlan>>,
224+
) -> Result<Arc<dyn ExecutionPlan>> {
225+
self.replace_children(
226+
children,
227+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
228+
)
229+
}
230+
208231
fn execute(
209232
&self,
210233
_partition: usize,

‎datafusion-examples/examples/relation_planner/table_sample.rs‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,6 @@ use futures::{
100100
use rand::{Rng, SeedableRng, rngs::StdRng};
101101
use tonic::async_trait;
102102

103-
use datafusion::optimizer::simplify_expressions::simplify_literal::parse_literal;
104103
use datafusion::{
105104
catalog::Session,
106105
execution::{
@@ -116,6 +115,10 @@ use datafusion::{
116115
physical_planner::{DefaultPhysicalPlanner, ExtensionPlanner, PhysicalPlanner},
117116
prelude::*,
118117
};
118+
use datafusion::{
119+
optimizer::simplify_expressions::simplify_literal::parse_literal,
120+
physical_plan::{ChildrenPropertiesMode, ReplaceChildrenOptions},
121+
};
119122
use datafusion_common::{
120123
DFSchemaRef, DataFusionError, Result, Statistics, internal_err, not_impl_err,
121124
plan_datafusion_err, plan_err, tree_node::TreeNodeRecursion,
@@ -698,9 +701,10 @@ impl ExecutionPlan for SampleExec {
698701
vec![&self.input]
699702
}
700703

701-
fn with_new_children(
704+
fn replace_children(
702705
self: Arc<Self>,
703706
mut children: Vec<Arc<dyn ExecutionPlan>>,
707+
_: ReplaceChildrenOptions,
704708
) -> Result<Arc<dyn ExecutionPlan>> {
705709
Ok(Arc::new(Self::try_new(
706710
children.swap_remove(0),
@@ -710,6 +714,16 @@ impl ExecutionPlan for SampleExec {
710714
)?))
711715
}
712716

717+
fn with_new_children(
718+
self: Arc<Self>,
719+
children: Vec<Arc<dyn ExecutionPlan>>,
720+
) -> Result<Arc<dyn ExecutionPlan>> {
721+
self.replace_children(
722+
children,
723+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
724+
)
725+
}
726+
713727
fn execute(
714728
&self,
715729
partition: usize,

‎datafusion/catalog/src/memory/table.rs‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,8 @@ use datafusion_physical_expr::{
4545
use datafusion_physical_plan::repartition::RepartitionExec;
4646
use datafusion_physical_plan::stream::RecordBatchStreamAdapter;
4747
use datafusion_physical_plan::{
48-
DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PhysicalExpr,
49-
PlanProperties, collect_partitioned,
48+
ChildrenPropertiesMode, DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning,
49+
PhysicalExpr, PlanProperties, ReplaceChildrenOptions, collect_partitioned,
5050
};
5151
use datafusion_session::Session;
5252

@@ -572,13 +572,24 @@ impl ExecutionPlan for DmlResultExec {
572572
vec![]
573573
}
574574

575-
fn with_new_children(
575+
fn replace_children(
576576
self: Arc<Self>,
577-
_children: Vec<Arc<dyn ExecutionPlan>>,
577+
_: Vec<Arc<dyn ExecutionPlan>>,
578+
_: ReplaceChildrenOptions,
578579
) -> Result<Arc<dyn ExecutionPlan>> {
579580
Ok(self)
580581
}
581582

583+
fn with_new_children(
584+
self: Arc<Self>,
585+
children: Vec<Arc<dyn ExecutionPlan>>,
586+
) -> Result<Arc<dyn ExecutionPlan>> {
587+
self.replace_children(
588+
children,
589+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
590+
)
591+
}
592+
582593
fn execute(
583594
&self,
584595
_partition: usize,

‎datafusion/core/src/physical_planner.rs‎

Lines changed: 78 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3323,6 +3323,7 @@ mod tests {
33233323
use datafusion_functions_aggregate::expr_fn::sum;
33243324
use datafusion_physical_expr::EquivalenceProperties;
33253325
use datafusion_physical_plan::execution_plan::{Boundedness, EmissionType};
3326+
use datafusion_physical_plan::{ChildrenPropertiesMode, ReplaceChildrenOptions};
33263327
use datafusion_session::QueryPlanner;
33273328

33283329
#[derive(Debug)]
@@ -4932,9 +4933,10 @@ mod tests {
49324933
vec![]
49334934
}
49344935

4935-
fn with_new_children(
4936+
fn replace_children(
49364937
self: Arc<Self>,
49374938
children: Vec<Arc<dyn ExecutionPlan>>,
4939+
_: ReplaceChildrenOptions,
49384940
) -> Result<Arc<dyn ExecutionPlan>> {
49394941
if children.is_empty() {
49404942
Ok(self)
@@ -4943,6 +4945,16 @@ mod tests {
49434945
}
49444946
}
49454947

4948+
fn with_new_children(
4949+
self: Arc<Self>,
4950+
children: Vec<Arc<dyn ExecutionPlan>>,
4951+
) -> Result<Arc<dyn ExecutionPlan>> {
4952+
self.replace_children(
4953+
children,
4954+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
4955+
)
4956+
}
4957+
49464958
fn execute(
49474959
&self,
49484960
_partition: usize,
@@ -5102,12 +5114,22 @@ digraph {
51025114
fn name(&self) -> &str {
51035115
"always ok"
51045116
}
5105-
fn with_new_children(
5117+
fn replace_children(
51065118
self: Arc<Self>,
51075119
children: Vec<Arc<dyn ExecutionPlan>>,
5120+
_: ReplaceChildrenOptions,
51085121
) -> Result<Arc<dyn ExecutionPlan>> {
51095122
Ok(Arc::new(Self(children)))
51105123
}
5124+
fn with_new_children(
5125+
self: Arc<Self>,
5126+
children: Vec<Arc<dyn ExecutionPlan>>,
5127+
) -> Result<Arc<dyn ExecutionPlan>> {
5128+
self.replace_children(
5129+
children,
5130+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5131+
)
5132+
}
51115133
fn schema(&self) -> SchemaRef {
51125134
Arc::new(Schema::empty())
51135135
}
@@ -5157,12 +5179,22 @@ digraph {
51575179
fn schema(&self) -> SchemaRef {
51585180
Arc::new(Schema::empty())
51595181
}
5160-
fn with_new_children(
5182+
fn replace_children(
51615183
self: Arc<Self>,
5162-
_children: Vec<Arc<dyn ExecutionPlan>>,
5184+
_: Vec<Arc<dyn ExecutionPlan>>,
5185+
_: ReplaceChildrenOptions,
51635186
) -> Result<Arc<dyn ExecutionPlan>> {
51645187
unimplemented!()
51655188
}
5189+
fn with_new_children(
5190+
self: Arc<Self>,
5191+
children: Vec<Arc<dyn ExecutionPlan>>,
5192+
) -> Result<Arc<dyn ExecutionPlan>> {
5193+
self.replace_children(
5194+
children,
5195+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5196+
)
5197+
}
51665198
fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
51675199
unimplemented!()
51685200
}
@@ -5216,10 +5248,16 @@ digraph {
52165248
// ok plan
52175249
let ok_node: Arc<dyn ExecutionPlan> = Arc::new(OkExtensionNode(vec![]));
52185250
let child = Arc::clone(&ok_node);
5219-
let ok_plan = Arc::clone(&ok_node).with_new_children(vec![
5220-
Arc::clone(&child).with_new_children(vec![Arc::clone(&child)])?,
5221-
Arc::clone(&child),
5222-
])?;
5251+
let ok_plan = Arc::clone(&ok_node).replace_children(
5252+
vec![
5253+
Arc::clone(&child).replace_children(
5254+
vec![Arc::clone(&child)],
5255+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5256+
)?,
5257+
Arc::clone(&child),
5258+
],
5259+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5260+
)?;
52235261

52245262
// Test: check should pass with same schema
52255263
let equal_schema = ok_plan.schema();
@@ -5251,10 +5289,16 @@ digraph {
52515289

52525290
// Test: should fail when descendent extension node fails
52535291
let failing_node: Arc<dyn ExecutionPlan> = Arc::new(InvariantFailsExtensionNode);
5254-
let invalid_plan = ok_node.with_new_children(vec![
5255-
Arc::clone(&child).with_new_children(vec![Arc::clone(&failing_node)])?,
5256-
Arc::clone(&child),
5257-
])?;
5292+
let invalid_plan = ok_node.replace_children(
5293+
vec![
5294+
Arc::clone(&child).replace_children(
5295+
vec![Arc::clone(&failing_node)],
5296+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5297+
)?,
5298+
Arc::clone(&child),
5299+
],
5300+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5301+
)?;
52585302
let result = OptimizationInvariantChecker::new(&rule)
52595303
.check(&invalid_plan, &ok_plan.schema());
52605304
if cfg!(debug_assertions) {
@@ -5287,12 +5331,22 @@ digraph {
52875331
fn schema(&self) -> SchemaRef {
52885332
Arc::new(Schema::empty())
52895333
}
5290-
fn with_new_children(
5334+
fn replace_children(
52915335
self: Arc<Self>,
5292-
_children: Vec<Arc<dyn ExecutionPlan>>,
5336+
_: Vec<Arc<dyn ExecutionPlan>>,
5337+
_: ReplaceChildrenOptions,
52935338
) -> Result<Arc<dyn ExecutionPlan>> {
52945339
unimplemented!()
52955340
}
5341+
fn with_new_children(
5342+
self: Arc<Self>,
5343+
children: Vec<Arc<dyn ExecutionPlan>>,
5344+
) -> Result<Arc<dyn ExecutionPlan>> {
5345+
self.replace_children(
5346+
children,
5347+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5348+
)
5349+
}
52965350
fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>> {
52975351
vec![]
52985352
}
@@ -5339,10 +5393,16 @@ digraph {
53395393
let failing_node: Arc<dyn ExecutionPlan> = Arc::new(ExecutableInvariantFails);
53405394
let ok_node: Arc<dyn ExecutionPlan> = Arc::new(OkExtensionNode(vec![]));
53415395
let child = Arc::clone(&ok_node);
5342-
let plan = ok_node.with_new_children(vec![
5343-
Arc::clone(&child).with_new_children(vec![Arc::clone(&failing_node)])?,
5344-
Arc::clone(&child),
5345-
])?;
5396+
let plan = ok_node.replace_children(
5397+
vec![
5398+
Arc::clone(&child).replace_children(
5399+
vec![Arc::clone(&failing_node)],
5400+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5401+
)?,
5402+
Arc::clone(&child),
5403+
],
5404+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
5405+
)?;
53465406
let expected_err = InvariantChecker(InvariantLevel::Executable)
53475407
.check(&plan)
53485408
.unwrap_err();

0 commit comments

Comments
 (0)