Skip to content

Commit cc2ab97

Browse files
committed
docs: proposed update to optimizer doc
1 parent 748e3b7 commit cc2ab97

2 files changed

Lines changed: 100 additions & 86 deletions

File tree

‎datafusion/physical-optimizer/src/optimizer.rs‎

Lines changed: 94 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -15,88 +15,9 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18-
//! Physical optimizer traits
18+
//! [`PhysicalOptimizer`] for optimizing [`ExecutionPlan`]s
1919
//!
20-
//! # Physical Optimizer Contract
21-
//!
22-
//! This section explains the contract for extending or replacing the default
23-
//! list of optimizer rules.
24-
//!
25-
//! [`PhysicalOptimizer::new`] defines the default rule sequence:
26-
//!
27-
//! ```text
28-
//! // Default rules
29-
//! let rules = vec![
30-
//! rule1,
31-
//! rule2,
32-
//! rule3,
33-
//! rule4,
34-
//! // ...
35-
//! ];
36-
//! ```
37-
//!
38-
//! It's recommended to keep the default rule order and using supported configuration
39-
//! options to disable optimizations. These variations preserve correctness. For
40-
//! example, `SET datafusion.optimizer.enable_distinct_aggregation_soft_limit = false`
41-
//! disables the distinct aggregation soft-limit optimization. Custom pipelines
42-
//! require more care, as described below.
43-
//!
44-
//! ## Custom Optimizer Pipelines
45-
//!
46-
//! The following extension pipeline can expect built-in optimizer rules to preserve
47-
//! plan validity, but not to provide complete optimization coverage.
48-
//!
49-
//! ```text
50-
//! // Custom rules
51-
//! let rules = vec![
52-
//! rule3,
53-
//! extension_rule_1,
54-
//! rule1,
55-
//! ];
56-
//! ```
57-
//!
58-
//! - **Plan Validity:** Applying `rule1` after `rule3` should preserve query
59-
//! semantics and return a valid plan without internal errors or invalid
60-
//! plan shapes.
61-
//!
62-
//! The pipeline must still satisfy requirements needed for correctness. For
63-
//! example, [`EnsureRequirements`] enforces input distribution and ordering.
64-
//! Some constraints remain implicit or are documented only in individual rules.
65-
//!
66-
//! - **Optimization Coverage:** A non-default rule order may expose rules to
67-
//! plan shapes they do not recognize, causing them to miss optimizations
68-
//! that would apply in the default pipeline.
69-
//!
70-
//! For example, [`LimitedDistinctAggregation`] runs before [`EnsureRequirements`]
71-
//! and [`CombinePartialFinalAggregate`], so it only targets the canonical shape. To keep
72-
//! its implementation simple, it does not guarantee optimization coverage for
73-
//! non-canonical shapes produced by extension pipelines.
74-
//!
75-
//! ```txt
76-
//! -- Canonical (default pipeline):
77-
//! -- Built-in rule recognize and optimize this shape
78-
//! Limit
79-
//! Aggregate(Final)
80-
//! Aggregate(Partial)
81-
//!
82-
//! -- Non-canonical (repartitioning inserted first):
83-
//! -- Built-in rule skip the optimization
84-
//! Limit
85-
//! Aggregate(Final)
86-
//! Repartition
87-
//! Aggregate(Partial)
88-
//! ```
89-
//!
90-
//! For custom optimizer pipelines:
91-
//!
92-
//! - Reordering optimizer rules must still result in a valid plan, subject to the
93-
//! correctness requirements above. Downstream projects may add unit tests for
94-
//! built-in rules to guard against correctness regressions.
95-
//! - Extending built-in rules to match plan shapes that the default rule order
96-
//! cannot produce is discouraged. Prefer adding a custom optimizer rule that
97-
//! transforms the plan into a standard shape recognized by built-in rules.
98-
//! If extending a built-in rule is still necessary, please open a DataFusion
99-
//! issue to discuss it.
20+
//! [`ExecutionPlan`]: datafusion_physical_plan::ExecutionPlan
10021
10122
use std::fmt::Debug;
10223
use std::sync::Arc;
@@ -162,7 +83,96 @@ impl PhysicalOptimizerContext for ConfigOnlyContext<'_> {
16283

16384
/// A rule-based physical optimizer.
16485
///
165-
/// Applies its rule list in order in a single pass.
86+
/// Applies a *Pipeline* (an ordered list) of [`PhysicalOptimizerRule`]s in a single pass.
87+
///
88+
/// # Physical Optimizer Contract
89+
///
90+
/// [`PhysicalOptimizer::new`] defines the default sequence of
91+
/// [`PhysicalOptimizerRule`]s. The optimizer applies them in order, in a single
92+
/// pass. This section describes the contract between the optimizer rules, both
93+
/// for projects that extend or reorder the default rule list and for
94+
/// contributors adding new built-in rules.
95+
///
96+
/// ## Invariants
97+
///
98+
/// The following properties hold for all rules (built-in and custom):
99+
///
100+
/// 1. **Correctness.** The returned plan must a) be valid (it can be executed
101+
/// without internal errors) and b) produce the same results as the input
102+
/// plan. A rule may change the operators in the plan, such as changing
103+
/// which operator implements a join, but it may never change the query's
104+
/// semantics.
105+
///
106+
/// 2. **Reorderable.** Because each rule preserves correctness, running the
107+
/// rules in a different order must still yield a valid plan that produces
108+
/// the same results.
109+
///
110+
/// ### Exceptions
111+
///
112+
/// Some built-in rules do not yet fully satisfy the invariants above because
113+
/// they are responsible for making an initial plan *executable* rather than for
114+
/// optimizing an existing plan:
115+
///
116+
/// - [`EnsureRequirements`] inserts the repartitioning and sorting needed to
117+
/// satisfy the distribution and ordering requirements.
118+
/// - [`OutputRequirements`] records output requirements.
119+
/// - [`SanityCheckPlan`] rejects plans whose requirements are not satisfied.
120+
/// It should run last.
121+
///
122+
/// Custom pipelines must keep these rules in a compatible relative order. We
123+
/// are working to remove these exceptions, so that all rules can be reordered
124+
/// freely.
125+
///
126+
/// ## Optimization Coverage Is Not Guaranteed
127+
///
128+
/// The invariants above guarantee *correctness*, not that every optimization is
129+
/// applied. Built-in rules often match specific plan shapes that may only occur
130+
/// at a certain point in the default pipeline. Adding, removing, or reordering
131+
/// rules can change plan shapes, so a built-in rule may silently miss an
132+
/// optimization that would apply in the default order. To keep their
133+
/// implementations simple, built-in rules typically only recognize shapes that
134+
/// the default pipeline produces.
135+
///
136+
/// For example, [`CombinePartialFinalAggregate`] runs after
137+
/// [`EnsureRequirements`] and only merges a `Partial` aggregate that is the
138+
/// direct child of a `Final` aggregate.
139+
///
140+
/// ```text
141+
/// -- Canonical shape (default pipeline):
142+
/// -- built-in rules recognize and optimize this
143+
/// Limit
144+
/// Aggregate(Final)
145+
/// Aggregate(Partial)
146+
///
147+
/// -- Non-canonical shape (repartitioning inserted first):
148+
/// -- built-in rules skip the optimization
149+
/// Limit
150+
/// Aggregate(Final)
151+
/// Repartition
152+
/// Aggregate(Partial)
153+
/// ```
154+
///
155+
/// ## Guidance for Custom Pipelines
156+
///
157+
/// We recommend keeping the default rule order and using the supported
158+
/// configuration options to disable individual optimizations. For example, `SET
159+
/// datafusion.optimizer.enable_distinct_aggregation_soft_limit = false`
160+
/// disables the distinct aggregation soft-limit optimization. These variations
161+
/// are tested and preserve both correctness and optimization coverage.
162+
///
163+
/// If you do build a custom pipeline:
164+
///
165+
/// - **Reordering must still yield a valid plan.** Downstream projects are
166+
/// encouraged to add tests that run their pipeline end to end and check
167+
/// results, as DataFusion's own tests only cover the default order.
168+
/// - **Test for reduced optimization coverage,** as described above, and verify
169+
/// the plans you care about with `EXPLAIN`.
170+
/// - **Prefer adding a rule over extending a built-in one.** We discourage
171+
/// extending a built-in rule to match plan shapes it will not encounter in
172+
/// the default pipeline, because such extensions are difficult to test and
173+
/// maintain. Instead, add a custom rule that rewrites your plan into the
174+
/// canonical shape recognized by the built-in rules. If extending a built-in
175+
/// rule still seems necessary, please open a DataFusion issue to discuss it.
166176
#[derive(Clone, Debug)]
167177
pub struct PhysicalOptimizer {
168178
/// All rules to apply
@@ -291,8 +301,8 @@ impl PhysicalOptimizer {
291301

292302
/// Create a new optimizer with the given rules
293303
///
294-
/// See the [module documentation](self) for the contract governing non-default
295-
/// optimizer pipelines.
304+
/// See the [`PhysicalOptimizer`](Self) documentation for the contract
305+
/// governing non-default optimizer pipelines.
296306
pub fn with_rules(rules: Vec<Arc<dyn PhysicalOptimizerRule + Send + Sync>>) -> Self {
297307
Self { rules }
298308
}

‎datafusion/session/src/physical_optimizer.rs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,10 +120,14 @@ where
120120
/// `PhysicalOptimizerRule` transforms one [`ExecutionPlan`] into another which
121121
/// computes the same results, but in a potentially more efficient way.
122122
///
123-
/// Use [`SessionState::add_physical_optimizer_rule`] to register additional
123+
/// The returned plan must be valid and produce the same results as the input
124+
/// plan. See [`PhysicalOptimizer`] for more details.
125+
///
126+
/// Use [`SessionStateBuilder::with_physical_optimizer_rule`] to register additional
124127
/// `PhysicalOptimizerRule`s.
125128
///
126-
/// [`SessionState::add_physical_optimizer_rule`]: https://docs.rs/datafusion/latest/datafusion/execution/session_state/struct.SessionState.html#method.add_physical_optimizer_rule
129+
/// [`PhysicalOptimizer`]: https://docs.rs/datafusion-physical-optimizer/latest/datafusion_physical_optimizer/optimizer/struct.PhysicalOptimizer.html
130+
/// [`SessionStateBuilder::with_physical_optimizer_rule`]: https://docs.rs/datafusion/latest/datafusion/execution/session_state/struct.SessionStateBuilder.html#method.with_physical_optimizer_rule
127131
pub trait PhysicalOptimizerRule: Debug + std::any::Any {
128132
/// Rewrite `plan` to an optimized form.
129133
///

0 commit comments

Comments
 (0)