Skip to content

Commit 862843f

Browse files
2010YOUY01alamb
andauthored
doc: Document the physical optimizer contract (apache#26351)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#26171 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> See issue for rationale, this PR documents what has been agreed on ## What changes are included in this PR? doc only <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent d38c545 commit 862843f

2 files changed

Lines changed: 103 additions & 4 deletions

File tree

‎datafusion/physical-optimizer/src/optimizer.rs‎

Lines changed: 97 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,9 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18-
//! Physical optimizer traits
18+
//! [`PhysicalOptimizer`] for optimizing [`ExecutionPlan`]s
19+
//!
20+
//! [`ExecutionPlan`]: datafusion_physical_plan::ExecutionPlan
1921
2022
use std::fmt::Debug;
2123
use std::sync::Arc;
@@ -81,7 +83,97 @@ impl PhysicalOptimizerContext for ConfigOnlyContext<'_> {
8183

8284
/// A rule-based physical optimizer.
8385
///
84-
/// Applies its rule list in order in a single pass.
86+
/// Applies a *Pipeline* (an ordered list) of [`PhysicalOptimizerRule`]s in a single pass.
87+
///
88+
/// # Physical Optimizer Contract
89+
///
90+
/// [`PhysicalOptimizer::new`] defines the default sequence of
91+
/// [`PhysicalOptimizerRule`]s. The optimizer applies them in order, in a single
92+
/// pass. This section describes the contract between the optimizer rules, both
93+
/// for projects that extend or reorder the default rule list and for
94+
/// contributors adding new built-in rules.
95+
///
96+
/// ## Invariants
97+
///
98+
/// The following properties hold for all rules (built-in and custom):
99+
///
100+
/// 1. **Correctness.** The returned plan must a) be valid (it can be executed
101+
/// without internal errors) and b) produce the same results as the input
102+
/// plan. A rule may change the operators in the plan, such as changing
103+
/// which operator implements a join, but it may never change the query's
104+
/// semantics.
105+
///
106+
/// 2. **Reorderable.** Because each rule preserves correctness, running the
107+
/// rules in a different order must still yield a valid plan that produces
108+
/// the same results.
109+
///
110+
/// ### Exceptions
111+
///
112+
/// Some built-in rules do not yet fully satisfy the invariants above because
113+
/// they are responsible for making an initial plan *executable* rather than for
114+
/// optimizing an existing plan:
115+
///
116+
/// - [`EnsureRequirements`] inserts the repartitioning and sorting needed to
117+
/// satisfy the distribution and ordering requirements.
118+
/// - [`OutputRequirements`] records output requirements.
119+
/// - [`SanityCheckPlan`] rejects plans whose requirements are not satisfied.
120+
/// It should run last.
121+
///
122+
/// Custom pipelines must keep these rules in a compatible relative order. We
123+
/// are working to remove these exceptions, so that all rules can be reordered
124+
/// freely.
125+
///
126+
/// ## Optimization Coverage Is Not Guaranteed
127+
///
128+
/// The invariants above guarantee *correctness*, not that every optimization is
129+
/// applied. Built-in rules often match specific plan shapes that may only occur
130+
/// at a certain point in the default pipeline. Adding, removing, or reordering
131+
/// rules can change plan shapes, so a built-in rule may silently miss an
132+
/// optimization that would apply in the default order. To keep their
133+
/// implementations simple, built-in rules typically only recognize shapes that
134+
/// the default pipeline produces.
135+
///
136+
/// For example, [`CombinePartialFinalAggregate`] runs after
137+
/// [`EnsureRequirements`] and only merges a `Partial` aggregate that is the
138+
/// direct child of a `Final` aggregate.
139+
///
140+
/// ```text
141+
/// -- Canonical shape (default pipeline):
142+
/// -- built-in rules recognize and optimize this
143+
/// Limit
144+
/// Aggregate(Final)
145+
/// Aggregate(Partial)
146+
///
147+
/// -- Non-canonical shape (repartitioning inserted first):
148+
/// -- built-in rules skip the optimization
149+
/// Limit
150+
/// Aggregate(Final)
151+
/// Repartition
152+
/// Aggregate(Partial)
153+
/// ```
154+
///
155+
/// ## Guidance for Custom Pipelines
156+
///
157+
/// We recommend keeping the default rule order and using the supported
158+
/// configuration options to disable individual optimizations. For example, `SET
159+
/// datafusion.optimizer.enable_distinct_aggregation_soft_limit = false`
160+
/// disables the distinct aggregation soft-limit optimization. These variations
161+
/// are tested and preserve both correctness and optimization coverage.
162+
/// See <https://github.com/apache/datafusion/issues/26481> for more details.
163+
///
164+
/// If you do build a custom pipeline:
165+
///
166+
/// - **Reordering must still yield a valid plan.** Downstream projects are
167+
/// encouraged to add tests that run their pipeline end to end and check
168+
/// results, as DataFusion's own tests only cover the default order.
169+
/// - **Test for reduced optimization coverage,** as described above, and verify
170+
/// the plans you care about with `EXPLAIN`.
171+
/// - **Prefer adding a rule over extending a built-in one.** We discourage
172+
/// extending a built-in rule to match plan shapes it will not encounter in
173+
/// the default pipeline, because such extensions are difficult to test and
174+
/// maintain. Instead, add a custom rule that rewrites your plan into the
175+
/// canonical shape recognized by the built-in rules. If extending a built-in
176+
/// rule still seems necessary, please open a DataFusion issue to discuss it.
85177
#[derive(Clone, Debug)]
86178
pub struct PhysicalOptimizer {
87179
/// All rules to apply
@@ -211,6 +303,9 @@ impl PhysicalOptimizer {
211303
}
212304

213305
/// Create a new optimizer with the given rules
306+
///
307+
/// See the [`PhysicalOptimizer`](Self) documentation for the contract
308+
/// governing non-default optimizer pipelines.
214309
pub fn with_rules(rules: Vec<Arc<dyn PhysicalOptimizerRule + Send + Sync>>) -> Self {
215310
Self { rules }
216311
}

‎datafusion/session/src/physical_optimizer.rs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,10 +120,14 @@ where
120120
/// `PhysicalOptimizerRule` transforms one [`ExecutionPlan`] into another which
121121
/// computes the same results, but in a potentially more efficient way.
122122
///
123-
/// Use [`SessionState::add_physical_optimizer_rule`] to register additional
123+
/// The returned plan must be valid and produce the same results as the input
124+
/// plan. See [`PhysicalOptimizer`] for more details.
125+
///
126+
/// Use [`SessionStateBuilder::with_physical_optimizer_rule`] to register additional
124127
/// `PhysicalOptimizerRule`s.
125128
///
126-
/// [`SessionState::add_physical_optimizer_rule`]: https://docs.rs/datafusion/latest/datafusion/execution/session_state/struct.SessionState.html#method.add_physical_optimizer_rule
129+
/// [`PhysicalOptimizer`]: https://docs.rs/datafusion-physical-optimizer/latest/datafusion_physical_optimizer/optimizer/struct.PhysicalOptimizer.html
130+
/// [`SessionStateBuilder::with_physical_optimizer_rule`]: https://docs.rs/datafusion/latest/datafusion/execution/session_state/struct.SessionStateBuilder.html#method.with_physical_optimizer_rule
127131
pub trait PhysicalOptimizerRule: Debug + std::any::Any {
128132
/// Rewrite `plan` to an optimized form.
129133
///

0 commit comments

Comments
 (0)