Skip to content

Commit d4d2767

Browse files
committed
docs: describe the physical analyzer phase and who is affected
1 parent 1eb88e0 commit d4d2767

1 file changed

Lines changed: 82 additions & 23 deletions

File tree

  • docs/source/library-user-guide/upgrading

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 82 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1337,29 +1337,88 @@ Code that reads `FileScanConfig::preserve_order` to decide whether files may be
13371337
reordered should call `FileScanConfig::preserves_file_order()` instead, which
13381338
also accounts for the output ordering.
13391339

1340-
### Physical planning gained an analyzer phase
1341-
1342-
Physical planning now runs two rule lists, mirroring the logical `Analyzer` /
1343-
`Optimizer` split. The new `PhysicalAnalyzerRule`s run first. The default
1344-
analyzer list is the former optimizer list up to and including
1345-
`EnsureRequirements` (`OutputRequirements` add mode, `aggregate_statistics`,
1346-
`EliminateConstantJoin`, `join_selection`, `LimitedDistinctAggregation`,
1347-
`FilterPushdown`, `WindowTopN`, `EnsureRequirements`), in the same order, so
1348-
default plans do not change. The
1349-
`PhysicalOptimizerRule`s then run on that plan. `EXPLAIN VERBOSE` shows the
1350-
analyzer passes as well.
1340+
### `EnsureRequirements` and the rules before it moved to a new physical analyzer list
1341+
1342+
Physical planning now runs two rule lists: the physical analyzer rules
1343+
(`PhysicalAnalyzerRule`), which run before distribution and ordering
1344+
requirements are enforced and end with `EnsureRequirements`, then the physical
1345+
optimizer rules, which receive a valid plan and must return one. The default
1346+
analyzer list is the start of the old optimizer list, up to and including
1347+
`EnsureRequirements`, in the same order (`AddOutputRequirements`,
1348+
`aggregate_statistics`, `EliminateConstantJoin`, `join_selection`,
1349+
`LimitedDistinctAggregation`, `FilterPushdown`, `WindowTopN`,
1350+
`EnsureRequirements`), so default plans do not change. Every optimizer rule now receives a plan that meets its
1351+
distribution and ordering requirements. `EXPLAIN VERBOSE` and the observer
1352+
passed to `DefaultPhysicalPlanner::optimize_physical_plan` show the analyzer
1353+
rules too; the observer now receives the rule name (`&str`) instead of a
1354+
`&dyn PhysicalOptimizerRule`.
13511355

1352-
**Migration guide:**
1356+
**Who is affected:**
13531357

1354-
- A type that implements the `Session` trait and overrides
1358+
- **Rules placed in front of the default optimizer list.** A list built as
1359+
your rules followed by `PhysicalOptimizer::new().rules` used to run your
1360+
rules before `EnsureRequirements`. They now run after the whole analyzer
1361+
phase, enforcement included, and nothing enforces the plan again after them.
1362+
A rule that has to run before enforcement is an analyzer rule: implement
1363+
`PhysicalAnalyzerRule` (`analyze` instead of `optimize`) and put it in front
1364+
of the default analyzer rules:
1365+
1366+
```rust,ignore
1367+
use datafusion::physical_optimizer::analyzer::{PhysicalAnalyzer, PhysicalAnalyzerRule};
1368+
1369+
let mut analyzer_rules: Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> =
1370+
vec![Arc::new(MyRule::new())];
1371+
analyzer_rules.extend(PhysicalAnalyzer::new().rules);
1372+
let state = SessionStateBuilder::new()
1373+
.with_default_features()
1374+
.with_physical_analyzer_rules(analyzer_rules)
1375+
.build();
1376+
```
1377+
1378+
A rule added with `with_physical_analyzer_rule` runs after
1379+
`EnsureRequirements`, so it must leave the plan valid.
1380+
1381+
- **An empty optimizer list no longer skips enforcement.**
1382+
`with_physical_optimizer_rules(vec![])` still runs the analyzer phase, so the
1383+
plan has the repartitions and sorts enforcement adds. To plan without them
1384+
as before, also call `with_physical_analyzer_rules(vec![])`.
1385+
1386+
- **Code that wraps, lists or runs the rules itself.**
1387+
`SessionState::physical_optimizers()` no longer contains the analyzer rules.
1388+
Tools that wrap every rule, for tracing or for `EXPLAIN` output, or that
1389+
list rule names, should handle `physical_analyzers()` too. Code that runs the
1390+
rules itself should call `DefaultPhysicalPlanner::optimize_physical_plan`, or
1391+
run the `physical_analyzers()` rules first.
1392+
1393+
- **Code that uses the moved rules directly.** The rules in the analyzer list
1394+
implement `PhysicalAnalyzerRule` instead of `PhysicalOptimizerRule`: call
1395+
`analyze` instead of `optimize` (for example
1396+
`EnsureRequirements::new().analyze(plan, config)`), and keep them in a
1397+
`Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>>`. Rules that ran in both
1398+
lists are now two types: `OutputRequirements::new_add_mode()` is
1399+
`AddOutputRequirements::new()` (analyzer) and `new_remove_mode()` is
1400+
`RemoveOutputRequirements::new()` (optimizer);
1401+
`FilterPushdown::new_post_optimization()` is `DynamicFilterPushdown::new()`
1402+
(optimizer), and `FilterPushdown::new()` is the analyzer rule.
1403+
1404+
- **Custom optimizer lists copied from the old default.** They no longer
1405+
compile, because the analyzer rules are not `PhysicalOptimizerRule`s. Drop
1406+
those rules from the list; the analyzer list runs them.
1407+
1408+
- **Custom `Session` implementations.** A type that overrides
13551409
`physical_optimizers` should also override `physical_analyzers`, for example
1356-
by returning `SessionState::physical_analyzers`. The default implementation
1357-
returns no rules, so such a session would otherwise run no enforcement.
1358-
- The default analyzer rules run even when a custom optimizer list is set with
1359-
`SessionStateBuilder::with_physical_optimizer_rules`. A custom list that still
1360-
contains a rule now in the analyzer (for example `EnsureRequirements`) runs it
1361-
twice; drop it from the list, or pass `with_physical_analyzer_rules(vec![])`
1362-
to keep the previous single-list behavior.
1363-
- The observer passed to `DefaultPhysicalPlanner::optimize_physical_plan` now
1364-
receives the rule name: `FnMut(&dyn ExecutionPlan, &str)` instead of
1365-
`FnMut(&dyn ExecutionPlan, &dyn PhysicalOptimizerRule)`.
1410+
by returning `SessionState::physical_analyzers`. The default returns no
1411+
rules, so the session plans without enforcement and fails at planning or at
1412+
execution.
1413+
1414+
- **Tests that match error messages.** When a rule in the analyzer list fails
1415+
the schema or invariant check, the error now names the phase, for example
1416+
`PhysicalAnalyzer rule 'join_selection' failed. Schema mismatch.` instead of
1417+
`PhysicalOptimizer rule 'join_selection' failed. Schema mismatch.`
1418+
1419+
- **Tests that match `EXPLAIN VERBOSE` output.** The rules that ran in both
1420+
lists now have their own names: `OutputRequirements` is
1421+
`AddOutputRequirements` in the analyzer and `RemoveOutputRequirements` in the
1422+
optimizer, and `FilterPushdown(Post)` is `DynamicFilterPushdown`.
1423+
1424+
See [PR #26757](https://github.com/apache/datafusion/pull/26757) for details.

0 commit comments

Comments
 (0)