Skip to content

Commit ab53c9f

Browse files
committed
docs: list who is affected by the physical analyzer phase and what to do
1 parent 19aaa2c commit ab53c9f

1 file changed

Lines changed: 58 additions & 27 deletions

File tree

  • docs/source/library-user-guide/upgrading

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 58 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1352,34 +1352,65 @@ analyzer list is the former optimizer list up to and including
13521352
`EnsureRequirements` (`OutputRequirements` add mode, `aggregate_statistics`,
13531353
`EliminateConstantJoin`, `join_selection`, `LimitedDistinctAggregation`,
13541354
`FilterPushdown`, `WindowTopN`, `EnsureRequirements`), in the same order, so
1355-
default plans do not change. The
1356-
`PhysicalOptimizerRule`s then run on that plan. `EXPLAIN VERBOSE` shows the
1357-
analyzer passes as well.
1358-
1359-
**Migration guide:**
1360-
1361-
- A type that implements the `Session` trait and overrides
1355+
default plans do not change. `EXPLAIN VERBOSE` and the observer passed to
1356+
`DefaultPhysicalPlanner::optimize_physical_plan` show the analyzer rules as
1357+
well.
1358+
1359+
**Who is affected and what to do:**
1360+
1361+
- **Rules placed in front of the default optimizer list.** A list built as
1362+
your rules followed by `PhysicalOptimizer::new().rules` used to run your
1363+
rules before `EnsureRequirements`. They now run after the whole analyzer
1364+
phase, enforcement included, and nothing enforces the plan again after them.
1365+
A rule that has to run before enforcement belongs in the analyzer list, in
1366+
front of the default rules and wrapped in `RunInAnalyzer`:
1367+
1368+
```rust,ignore
1369+
use datafusion::physical_optimizer::analyzer::{PhysicalAnalyzer, RunInAnalyzer};
1370+
1371+
let mut analyzer_rules: Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> =
1372+
vec![Arc::new(RunInAnalyzer::new(Arc::new(MyRule::new())))];
1373+
analyzer_rules.extend(PhysicalAnalyzer::new().rules);
1374+
let state = SessionStateBuilder::new()
1375+
.with_default_features()
1376+
.with_physical_analyzer_rules(analyzer_rules)
1377+
.build();
1378+
```
1379+
1380+
A rule added with `with_physical_analyzer_rule` runs after
1381+
`EnsureRequirements`, so it must leave the plan valid.
1382+
1383+
- **An empty optimizer list no longer skips enforcement.**
1384+
`with_physical_optimizer_rules(vec![])` still runs the analyzer phase, so the
1385+
plan has the repartitions and sorts enforcement adds. To plan without them
1386+
as before, also call `with_physical_analyzer_rules(vec![])`.
1387+
1388+
- **Code that wraps, lists or runs the rules itself.**
1389+
`SessionState::physical_optimizers()` no longer contains the analyzer rules.
1390+
Tools that wrap every rule, for tracing or for `EXPLAIN` output, or that
1391+
list rule names, should handle `physical_analyzers()` too. Code that runs the
1392+
rules itself should call `DefaultPhysicalPlanner::optimize_physical_plan`, or
1393+
run the `physical_analyzers()` rules first.
1394+
1395+
- **Custom optimizer lists that still contain analyzer rules.** A list copied
1396+
from the old default runs `join_selection`, `EnsureRequirements` and the
1397+
other analyzer rules twice. Results stay correct, but planning takes longer
1398+
and plans can change. Drop those rules from the list.
1399+
1400+
- **Custom `Session` implementations.** A type that overrides
13621401
`physical_optimizers` should also override `physical_analyzers`, for example
1363-
by returning `SessionState::physical_analyzers`. The default implementation
1364-
returns no rules, so such a session would otherwise run no enforcement.
1365-
- The default analyzer rules run even when a custom optimizer list is set with
1366-
`SessionStateBuilder::with_physical_optimizer_rules`. A custom list that still
1367-
contains a rule now in the analyzer (for example `EnsureRequirements`) runs it
1368-
twice; drop it from the list, or pass `with_physical_analyzer_rules(vec![])`
1369-
to keep the previous single-list behavior.
1370-
- Code that runs the `physical_optimizers()` rules itself, instead of calling
1371-
`DefaultPhysicalPlanner::optimize_physical_plan`, must run the
1372-
`physical_analyzers()` rules first, the same way. Without them the plan has
1373-
no enforcement and no record of the query's output ordering.
1374-
- To run a built-in `PhysicalOptimizerRule` in a custom analyzer list, wrap it
1375-
in `RunInAnalyzer`.
1376-
- In debug builds, planning fails where the analyzer phase ends if the plan is
1377-
not executable or leaves a distribution or ordering requirement unmet. A
1378-
custom analyzer list must therefore end with `EnsureRequirements`, or repair
1379-
the plan with
1402+
by returning `SessionState::physical_analyzers`. The default returns no
1403+
rules, so the session plans without enforcement and fails at planning or at
1404+
execution.
1405+
1406+
- **Debug builds** check where the analyzer phase ends that the plan is
1407+
executable and meets every distribution and ordering requirement. A custom
1408+
analyzer list must end with `EnsureRequirements`, or repair the plan with
13801409
`datafusion_physical_optimizer::ensure_requirements::satisfy_requirements`,
13811410
which inserts only the missing repartitions and sorts. A session with no
13821411
analyzer rules is not checked.
1383-
- The observer passed to `DefaultPhysicalPlanner::optimize_physical_plan` now
1384-
receives the rule name: `FnMut(&dyn ExecutionPlan, &str)` instead of
1385-
`FnMut(&dyn ExecutionPlan, &dyn PhysicalOptimizerRule)`.
1412+
1413+
- **`EnsureRequirements` on a plan that already has repartitions and sorts**
1414+
now removes them first and places them again. Nothing changes in the default
1415+
pipeline, but code that runs it more than once, or on a plan whose
1416+
enforcement is already in place, can get different plans.

0 commit comments

Comments
 (0)