Skip to content

Commit 820cd17

Browse files
committed
docs: describe the physical analyzer phase and who is affected
1 parent 72a5d8c commit 820cd17

1 file changed

Lines changed: 95 additions & 27 deletions

File tree

  • docs/source/library-user-guide/upgrading

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 95 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -249,16 +249,18 @@ SELECT ts_tz - ts FROM timestamps;
249249
owns the session. It now returns a `NotImplemented` error because forwarding can
250250
re-enter an installed foreign planner, and the execution-plan handle returned by
251251
the old callback cannot restore local Rust type identities for downcasting.
252-
The original `FFI_SessionRef` callback slot remains in place for ABI compatibility
253-
with DataFusion 55 consumers, but calling that callback returns the same error.
252+
The original `FFI_SessionRef` callback slot remains in place, but calling that
253+
callback returns the same error.
254254

255255
The session-owning library should instead export its original planner as a
256256
`datafusion_ffi::query_planner::FFI_QueryPlanner` before installing a foreign
257257
planner. The foreign planner can retain and invoke that handle to receive a
258258
serialized physical plan reconstructed with local type identities. See the
259259
`datafusion_ffi::query_planner` module documentation for the complete delegation
260-
pattern. `ForeignSession::query_planner`, `optimize`, and `physical_optimizers`
261-
continue to forward to the owning session across the FFI boundary.
260+
pattern. `ForeignSession::query_planner`, `optimize`, `physical_optimizers` and
261+
`physical_analyzers` forward to the owning session across the FFI boundary.
262+
`FFI_SessionRef` has a new field for the physical analyzer rules, so FFI
263+
libraries must be rebuilt against this release.
262264

263265
### Range partitioning supports retained samples and scaling
264266

@@ -1337,29 +1339,95 @@ Code that reads `FileScanConfig::preserve_order` to decide whether files may be
13371339
reordered should call `FileScanConfig::preserves_file_order()` instead, which
13381340
also accounts for the output ordering.
13391341

1340-
### Physical planning gained an analyzer phase
1342+
### `EnsureRequirements` and the rules before it moved to a new physical analyzer list
1343+
1344+
Physical planning now runs two rule lists: the physical analyzer rules
1345+
(`PhysicalAnalyzerRule`), which run before distribution and ordering
1346+
requirements are enforced and end with `EnsureRequirements`, then the physical
1347+
optimizer rules, which receive a valid plan and must return one. The default
1348+
analyzer list is the start of the old optimizer list, up to and including
1349+
`EnsureRequirements`, in the same order (`AddOutputRequirements`,
1350+
`aggregate_statistics`, `EliminateConstantJoin`, `join_selection`,
1351+
`LimitedDistinctAggregation`, `FilterPushdown`, `WindowTopN`,
1352+
`EnsureRequirements`), so default plans do not change. Every optimizer rule now receives a plan that meets its
1353+
distribution and ordering requirements. `EXPLAIN VERBOSE` and the observer
1354+
passed to `DefaultPhysicalPlanner::optimize_physical_plan` show the analyzer
1355+
rules too; the observer now receives the rule name (`&str`) instead of a
1356+
`&dyn PhysicalOptimizerRule`.
13411357

1342-
Physical planning now runs two rule lists, mirroring the logical `Analyzer` /
1343-
`Optimizer` split. The new `PhysicalAnalyzerRule`s run first. The default
1344-
analyzer list is the former optimizer list up to and including
1345-
`EnsureRequirements` (`OutputRequirements` add mode, `aggregate_statistics`,
1346-
`EliminateConstantJoin`, `join_selection`, `LimitedDistinctAggregation`,
1347-
`FilterPushdown`, `WindowTopN`, `EnsureRequirements`), in the same order, so
1348-
default plans do not change. The
1349-
`PhysicalOptimizerRule`s then run on that plan. `EXPLAIN VERBOSE` shows the
1350-
analyzer passes as well.
1351-
1352-
**Migration guide:**
1358+
**Who is affected:**
13531359

1354-
- A type that implements the `Session` trait and overrides
1360+
- **Rules placed in front of the default optimizer list.** A list built as
1361+
your rules followed by `PhysicalOptimizer::new().rules` used to run your
1362+
rules before `EnsureRequirements`. They now run after the whole analyzer
1363+
phase, enforcement included, and nothing enforces the plan again after them.
1364+
A rule that has to run before enforcement is an analyzer rule: implement
1365+
`PhysicalAnalyzerRule` (`analyze` instead of `optimize`) and put it in front
1366+
of the default analyzer rules:
1367+
1368+
```rust,ignore
1369+
use datafusion::physical_optimizer::analyzer::{PhysicalAnalyzer, PhysicalAnalyzerRule};
1370+
1371+
let mut analyzer_rules: Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> =
1372+
vec![Arc::new(MyRule::new())];
1373+
analyzer_rules.extend(PhysicalAnalyzer::new().rules);
1374+
let state = SessionStateBuilder::new()
1375+
.with_default_features()
1376+
.with_physical_analyzer_rules(analyzer_rules)
1377+
.build();
1378+
```
1379+
1380+
- **An empty optimizer list no longer skips enforcement.**
1381+
`with_physical_optimizer_rules(vec![])` still runs the analyzer phase, so the
1382+
plan has the repartitions and sorts enforcement adds. To plan without them
1383+
as before, also call `with_physical_analyzer_rules(vec![])`.
1384+
1385+
- **Code that wraps, lists or runs the rules itself.**
1386+
`SessionState::physical_optimizers()` no longer contains the analyzer rules.
1387+
Tools that wrap every rule, for tracing or for `EXPLAIN` output, or that
1388+
list rule names, should handle `physical_analyzers()` too. Code that runs the
1389+
rules itself should call `DefaultPhysicalPlanner::optimize_physical_plan`, or
1390+
run the `physical_analyzers()` rules first.
1391+
1392+
- **Code that removes or places rules by name.** Filters on
1393+
`PhysicalOptimizer::new().rules` still compile, but the analyzer rules are no
1394+
longer in that list, so a filter on `join_selection`, `EnsureRequirements`,
1395+
`FilterPushdown`, `WindowTopN`, `aggregate_statistics`,
1396+
`EliminateConstantJoin` or `LimitedDistinctAggregation` matches nothing and
1397+
the rule keeps running. Filter `PhysicalAnalyzer::new().rules` instead. The
1398+
renamed rules are `OutputRequirements`, now `AddOutputRequirements` and
1399+
`RemoveOutputRequirements`, and `FilterPushdown(Post)`, now
1400+
`DynamicFilterPushdown`.
1401+
1402+
- **Code that uses the moved rules directly.** The rules in the analyzer list
1403+
implement `PhysicalAnalyzerRule` instead of `PhysicalOptimizerRule`: call
1404+
`analyze` instead of `optimize` (for example
1405+
`EnsureRequirements::new().analyze(plan, config)`), and keep them in a
1406+
`Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>>`. Rules that ran in both
1407+
lists are now two types: `OutputRequirements::new_add_mode()` is
1408+
`AddOutputRequirements::new()` (analyzer) and `new_remove_mode()` is
1409+
`RemoveOutputRequirements::new()` (optimizer);
1410+
`FilterPushdown::new_post_optimization()` is `DynamicFilterPushdown::new()`
1411+
(optimizer), and `FilterPushdown::new()` is the analyzer rule.
1412+
1413+
- **Custom optimizer lists copied from the old default.** They no longer
1414+
compile, because the analyzer rules are not `PhysicalOptimizerRule`s. Drop
1415+
those rules from the list; the analyzer list runs them.
1416+
1417+
- **Custom `Session` implementations.** A type that overrides
13551418
`physical_optimizers` should also override `physical_analyzers`, for example
1356-
by returning `SessionState::physical_analyzers`. The default implementation
1357-
returns no rules, so such a session would otherwise run no enforcement.
1358-
- The default analyzer rules run even when a custom optimizer list is set with
1359-
`SessionStateBuilder::with_physical_optimizer_rules`. A custom list that still
1360-
contains a rule now in the analyzer (for example `EnsureRequirements`) runs it
1361-
twice; drop it from the list, or pass `with_physical_analyzer_rules(vec![])`
1362-
to keep the previous single-list behavior.
1363-
- The observer passed to `DefaultPhysicalPlanner::optimize_physical_plan` now
1364-
receives the rule name: `FnMut(&dyn ExecutionPlan, &str)` instead of
1365-
`FnMut(&dyn ExecutionPlan, &dyn PhysicalOptimizerRule)`.
1419+
by returning `SessionState::physical_analyzers`. The default returns no
1420+
rules, so the session plans without enforcement and fails at planning or at
1421+
execution.
1422+
1423+
- **Tests that match error messages.** When a rule in the analyzer list fails
1424+
the schema or invariant check, the error now names the phase, for example
1425+
`PhysicalAnalyzer rule 'join_selection' failed. Schema mismatch.` instead of
1426+
`PhysicalOptimizer rule 'join_selection' failed. Schema mismatch.`
1427+
1428+
- **Tests that match `EXPLAIN VERBOSE` output.** The rules that ran in both
1429+
lists now have their own names: `OutputRequirements` is
1430+
`AddOutputRequirements` in the analyzer and `RemoveOutputRequirements` in the
1431+
optimizer, and `FilterPushdown(Post)` is `DynamicFilterPushdown`.
1432+
1433+
See [PR #26757](https://github.com/apache/datafusion/pull/26757) for details.

0 commit comments

Comments
 (0)