|
15 | 15 | // specific language governing permissions and limitations |
16 | 16 | // under the License. |
17 | 17 |
|
18 | | -//! Physical optimizer traits |
| 18 | +//! [`PhysicalOptimizer`] for optimizing [`ExecutionPlan`]s |
19 | 19 | //! |
20 | | -//! # Physical Optimizer Contract |
21 | | -//! |
22 | | -//! This section explains the contract for extending or replacing the default |
23 | | -//! list of optimizer rules. |
24 | | -//! |
25 | | -//! [`PhysicalOptimizer::new`] defines the default rule sequence: |
26 | | -//! |
27 | | -//! ```text |
28 | | -//! // Default rules |
29 | | -//! let rules = vec![ |
30 | | -//! rule1, |
31 | | -//! rule2, |
32 | | -//! rule3, |
33 | | -//! rule4, |
34 | | -//! // ... |
35 | | -//! ]; |
36 | | -//! ``` |
37 | | -//! |
38 | | -//! It's recommended to keep the default rule order and using supported configuration |
39 | | -//! options to disable optimizations. These variations preserve correctness. For |
40 | | -//! example, `SET datafusion.optimizer.enable_distinct_aggregation_soft_limit = false` |
41 | | -//! disables the distinct aggregation soft-limit optimization. Custom pipelines |
42 | | -//! require more care, as described below. |
43 | | -//! |
44 | | -//! ## Custom Optimizer Pipelines |
45 | | -//! |
46 | | -//! The following extension pipeline can expect built-in optimizer rules to preserve |
47 | | -//! plan validity, but not to provide complete optimization coverage. |
48 | | -//! |
49 | | -//! ```text |
50 | | -//! // Custom rules |
51 | | -//! let rules = vec![ |
52 | | -//! rule3, |
53 | | -//! extension_rule_1, |
54 | | -//! rule1, |
55 | | -//! ]; |
56 | | -//! ``` |
57 | | -//! |
58 | | -//! - **Plan Validity:** Applying `rule1` after `rule3` should preserve query |
59 | | -//! semantics and return a valid plan without internal errors or invalid |
60 | | -//! plan shapes. |
61 | | -//! |
62 | | -//! The pipeline must still satisfy requirements needed for correctness. For |
63 | | -//! example, [`EnsureRequirements`] enforces input distribution and ordering. |
64 | | -//! Some constraints remain implicit or are documented only in individual rules. |
65 | | -//! |
66 | | -//! - **Optimization Coverage:** A non-default rule order may expose rules to |
67 | | -//! plan shapes they do not recognize, causing them to miss optimizations |
68 | | -//! that would apply in the default pipeline. |
69 | | -//! |
70 | | -//! For example, [`LimitedDistinctAggregation`] runs before [`EnsureRequirements`] |
71 | | -//! and [`CombinePartialFinalAggregate`], so it only targets the canonical shape. To keep |
72 | | -//! its implementation simple, it does not guarantee optimization coverage for |
73 | | -//! non-canonical shapes produced by extension pipelines. |
74 | | -//! |
75 | | -//! ```txt |
76 | | -//! -- Canonical (default pipeline): |
77 | | -//! -- Built-in rule recognize and optimize this shape |
78 | | -//! Limit |
79 | | -//! Aggregate(Final) |
80 | | -//! Aggregate(Partial) |
81 | | -//! |
82 | | -//! -- Non-canonical (repartitioning inserted first): |
83 | | -//! -- Built-in rule skip the optimization |
84 | | -//! Limit |
85 | | -//! Aggregate(Final) |
86 | | -//! Repartition |
87 | | -//! Aggregate(Partial) |
88 | | -//! ``` |
89 | | -//! |
90 | | -//! For custom optimizer pipelines: |
91 | | -//! |
92 | | -//! - Reordering optimizer rules must still result in a valid plan, subject to the |
93 | | -//! correctness requirements above. Downstream projects may add unit tests for |
94 | | -//! built-in rules to guard against correctness regressions. |
95 | | -//! - Extending built-in rules to match plan shapes that the default rule order |
96 | | -//! cannot produce is discouraged. Prefer adding a custom optimizer rule that |
97 | | -//! transforms the plan into a standard shape recognized by built-in rules. |
98 | | -//! If extending a built-in rule is still necessary, please open a DataFusion |
99 | | -//! issue to discuss it. |
| 20 | +//! [`ExecutionPlan`]: datafusion_physical_plan::ExecutionPlan |
100 | 21 |
|
101 | 22 | use std::fmt::Debug; |
102 | 23 | use std::sync::Arc; |
@@ -162,7 +83,96 @@ impl PhysicalOptimizerContext for ConfigOnlyContext<'_> { |
162 | 83 |
|
163 | 84 | /// A rule-based physical optimizer. |
164 | 85 | /// |
165 | | -/// Applies its rule list in order in a single pass. |
| 86 | +/// Applies a *Pipeline* (an ordered list) of [`PhysicalOptimizerRule`]s in a single pass. |
| 87 | +/// |
| 88 | +/// # Physical Optimizer Contract |
| 89 | +/// |
| 90 | +/// [`PhysicalOptimizer::new`] defines the default sequence of |
| 91 | +/// [`PhysicalOptimizerRule`]s. The optimizer applies them in order, in a single |
| 92 | +/// pass. This section describes the contract between the optimizer rules, both |
| 93 | +/// for projects that extend or reorder the default rule list and for |
| 94 | +/// contributors adding new built-in rules. |
| 95 | +/// |
| 96 | +/// ## Invariants |
| 97 | +/// |
| 98 | +/// The following properties hold for all rules (built-in and custom): |
| 99 | +/// |
| 100 | +/// 1. **Correctness.** The returned plan must a) be valid (it can be executed |
| 101 | +/// without internal errors) and b) produce the same results as the input |
| 102 | +/// plan. A rule may change the operators in the plan, such as changing |
| 103 | +/// which operator implements a join, but it may never change the query's |
| 104 | +/// semantics. |
| 105 | +/// |
| 106 | +/// 2. **Reorderable.** Because each rule preserves correctness, running the |
| 107 | +/// rules in a different order must still yield a valid plan that produces |
| 108 | +/// the same results. |
| 109 | +/// |
| 110 | +/// ### Exceptions |
| 111 | +/// |
| 112 | +/// Some built-in rules do not yet fully satisfy the invariants above because |
| 113 | +/// they are responsible for making an initial plan *executable* rather than for |
| 114 | +/// optimizing an existing plan: |
| 115 | +/// |
| 116 | +/// - [`EnsureRequirements`] inserts the repartitioning and sorting needed to |
| 117 | +/// satisfy the distribution and ordering requirements. |
| 118 | +/// - [`OutputRequirements`] records output requirements. |
| 119 | +/// - [`SanityCheckPlan`] rejects plans whose requirements are not satisfied. |
| 120 | +/// It should run last. |
| 121 | +/// |
| 122 | +/// Custom pipelines must keep these rules in a compatible relative order. We |
| 123 | +/// are working to remove these exceptions, so that all rules can be reordered |
| 124 | +/// freely. |
| 125 | +/// |
| 126 | +/// ## Optimization Coverage Is Not Guaranteed |
| 127 | +/// |
| 128 | +/// The invariants above guarantee *correctness*, not that every optimization is |
| 129 | +/// applied. Built-in rules often match specific plan shapes that may only occur |
| 130 | +/// at a certain point in the default pipeline. Adding, removing, or reordering |
| 131 | +/// rules can change plan shapes, so a built-in rule may silently miss an |
| 132 | +/// optimization that would apply in the default order. To keep their |
| 133 | +/// implementations simple, built-in rules typically only recognize shapes that |
| 134 | +/// the default pipeline produces. |
| 135 | +/// |
| 136 | +/// For example, [`CombinePartialFinalAggregate`] runs after |
| 137 | +/// [`EnsureRequirements`] and only merges a `Partial` aggregate that is the |
| 138 | +/// direct child of a `Final` aggregate. |
| 139 | +/// |
| 140 | +/// ```text |
| 141 | +/// -- Canonical shape (default pipeline): |
| 142 | +/// -- built-in rules recognize and optimize this |
| 143 | +/// Limit |
| 144 | +/// Aggregate(Final) |
| 145 | +/// Aggregate(Partial) |
| 146 | +/// |
| 147 | +/// -- Non-canonical shape (repartitioning inserted first): |
| 148 | +/// -- built-in rules skip the optimization |
| 149 | +/// Limit |
| 150 | +/// Aggregate(Final) |
| 151 | +/// Repartition |
| 152 | +/// Aggregate(Partial) |
| 153 | +/// ``` |
| 154 | +/// |
| 155 | +/// ## Guidance for Custom Pipelines |
| 156 | +/// |
| 157 | +/// We recommend keeping the default rule order and using the supported |
| 158 | +/// configuration options to disable individual optimizations. For example, `SET |
| 159 | +/// datafusion.optimizer.enable_distinct_aggregation_soft_limit = false` |
| 160 | +/// disables the distinct aggregation soft-limit optimization. These variations |
| 161 | +/// are tested and preserve both correctness and optimization coverage. |
| 162 | +/// |
| 163 | +/// If you do build a custom pipeline: |
| 164 | +/// |
| 165 | +/// - **Reordering must still yield a valid plan.** Downstream projects are |
| 166 | +/// encouraged to add tests that run their pipeline end to end and check |
| 167 | +/// results, as DataFusion's own tests only cover the default order. |
| 168 | +/// - **Test for reduced optimization coverage,** as described above, and verify |
| 169 | +/// the plans you care about with `EXPLAIN`. |
| 170 | +/// - **Prefer adding a rule over extending a built-in one.** We discourage |
| 171 | +/// extending a built-in rule to match plan shapes it will not encounter in |
| 172 | +/// the default pipeline, because such extensions are difficult to test and |
| 173 | +/// maintain. Instead, add a custom rule that rewrites your plan into the |
| 174 | +/// canonical shape recognized by the built-in rules. If extending a built-in |
| 175 | +/// rule still seems necessary, please open a DataFusion issue to discuss it. |
166 | 176 | #[derive(Clone, Debug)] |
167 | 177 | pub struct PhysicalOptimizer { |
168 | 178 | /// All rules to apply |
@@ -291,8 +301,8 @@ impl PhysicalOptimizer { |
291 | 301 |
|
292 | 302 | /// Create a new optimizer with the given rules |
293 | 303 | /// |
294 | | - /// See the [module documentation](self) for the contract governing non-default |
295 | | - /// optimizer pipelines. |
| 304 | + /// See the [`PhysicalOptimizer`](Self) documentation for the contract |
| 305 | + /// governing non-default optimizer pipelines. |
296 | 306 | pub fn with_rules(rules: Vec<Arc<dyn PhysicalOptimizerRule + Send + Sync>>) -> Self { |
297 | 307 | Self { rules } |
298 | 308 | } |
|
0 commit comments