@@ -1337,29 +1337,88 @@ Code that reads `FileScanConfig::preserve_order` to decide whether files may be
13371337reordered should call ` FileScanConfig::preserves_file_order() ` instead, which
13381338also accounts for the output ordering.
13391339
1340- ### Physical planning gained an analyzer phase
1341-
1342- Physical planning now runs two rule lists, mirroring the logical ` Analyzer ` /
1343- ` Optimizer ` split. The new ` PhysicalAnalyzerRule ` s run first. The default
1344- analyzer list is the former optimizer list up to and including
1345- ` EnsureRequirements ` (` OutputRequirements ` add mode, ` aggregate_statistics ` ,
1346- ` EliminateConstantJoin ` , ` join_selection ` , ` LimitedDistinctAggregation ` ,
1347- ` FilterPushdown ` , ` WindowTopN ` , ` EnsureRequirements ` ), in the same order, so
1348- default plans do not change. The
1349- ` PhysicalOptimizerRule ` s then run on that plan. ` EXPLAIN VERBOSE ` shows the
1350- analyzer passes as well.
1340+ ### ` EnsureRequirements ` and the rules before it moved to a new physical analyzer list
1341+
1342+ Physical planning now runs two rule lists: the physical analyzer rules
1343+ (` PhysicalAnalyzerRule ` ), which run before distribution and ordering
1344+ requirements are enforced and end with ` EnsureRequirements ` , then the physical
1345+ optimizer rules, which receive a valid plan and must return one. The default
1346+ analyzer list is the start of the old optimizer list, up to and including
1347+ ` EnsureRequirements ` , in the same order (` AddOutputRequirements ` ,
1348+ ` aggregate_statistics ` , ` EliminateConstantJoin ` , ` join_selection ` ,
1349+ ` LimitedDistinctAggregation ` , ` FilterPushdown ` , ` WindowTopN ` ,
1350+ ` EnsureRequirements ` ), so default plans do not change. Every optimizer rule now receives a plan that meets its
1351+ distribution and ordering requirements. ` EXPLAIN VERBOSE ` and the observer
1352+ passed to ` DefaultPhysicalPlanner::optimize_physical_plan ` show the analyzer
1353+ rules too; the observer now receives the rule name (` &str ` ) instead of a
1354+ ` &dyn PhysicalOptimizerRule ` .
13511355
1352- ** Migration guide :**
1356+ ** Who is affected :**
13531357
1354- - A type that implements the ` Session ` trait and overrides
1358+ - ** Rules placed in front of the default optimizer list.** A list built as
1359+ your rules followed by ` PhysicalOptimizer::new().rules ` used to run your
1360+ rules before ` EnsureRequirements ` . They now run after the whole analyzer
1361+ phase, enforcement included, and nothing enforces the plan again after them.
1362+ A rule that has to run before enforcement is an analyzer rule: implement
1363+ ` PhysicalAnalyzerRule ` (` analyze ` instead of ` optimize ` ) and put it in front
1364+ of the default analyzer rules:
1365+
1366+ ``` rust,ignore
1367+ use datafusion::physical_optimizer::analyzer::{PhysicalAnalyzer, PhysicalAnalyzerRule};
1368+
1369+ let mut analyzer_rules: Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> =
1370+ vec![Arc::new(MyRule::new())];
1371+ analyzer_rules.extend(PhysicalAnalyzer::new().rules);
1372+ let state = SessionStateBuilder::new()
1373+ .with_default_features()
1374+ .with_physical_analyzer_rules(analyzer_rules)
1375+ .build();
1376+ ```
1377+
1378+ A rule added with ` with_physical_analyzer_rule ` runs after
1379+ ` EnsureRequirements ` , so it must leave the plan valid.
1380+
1381+ - ** An empty optimizer list no longer skips enforcement.**
1382+ ` with_physical_optimizer_rules(vec![]) ` still runs the analyzer phase, so the
1383+ plan has the repartitions and sorts enforcement adds. To plan without them
1384+ as before, also call ` with_physical_analyzer_rules(vec![]) ` .
1385+
1386+ - ** Code that wraps, lists or runs the rules itself.**
1387+ ` SessionState::physical_optimizers() ` no longer contains the analyzer rules.
1388+ Tools that wrap every rule, for tracing or for ` EXPLAIN ` output, or that
1389+ list rule names, should handle ` physical_analyzers() ` too. Code that runs the
1390+ rules itself should call ` DefaultPhysicalPlanner::optimize_physical_plan ` , or
1391+ run the ` physical_analyzers() ` rules first.
1392+
1393+ - ** Code that uses the moved rules directly.** The rules in the analyzer list
1394+ implement ` PhysicalAnalyzerRule ` instead of ` PhysicalOptimizerRule ` : call
1395+ ` analyze ` instead of ` optimize ` (for example
1396+ ` EnsureRequirements::new().analyze(plan, config) ` ), and keep them in a
1397+ ` Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> ` . Rules that ran in both
1398+ lists are now two types: ` OutputRequirements::new_add_mode() ` is
1399+ ` AddOutputRequirements::new() ` (analyzer) and ` new_remove_mode() ` is
1400+ ` RemoveOutputRequirements::new() ` (optimizer);
1401+ ` FilterPushdown::new_post_optimization() ` is ` DynamicFilterPushdown::new() `
1402+ (optimizer), and ` FilterPushdown::new() ` is the analyzer rule.
1403+
1404+ - ** Custom optimizer lists copied from the old default.** They no longer
1405+ compile, because the analyzer rules are not ` PhysicalOptimizerRule ` s. Drop
1406+ those rules from the list; the analyzer list runs them.
1407+
1408+ - ** Custom ` Session ` implementations.** A type that overrides
13551409 ` physical_optimizers ` should also override ` physical_analyzers ` , for example
1356- by returning ` SessionState::physical_analyzers ` . The default implementation
1357- returns no rules, so such a session would otherwise run no enforcement.
1358- - The default analyzer rules run even when a custom optimizer list is set with
1359- ` SessionStateBuilder::with_physical_optimizer_rules ` . A custom list that still
1360- contains a rule now in the analyzer (for example ` EnsureRequirements ` ) runs it
1361- twice; drop it from the list, or pass ` with_physical_analyzer_rules(vec![]) `
1362- to keep the previous single-list behavior.
1363- - The observer passed to ` DefaultPhysicalPlanner::optimize_physical_plan ` now
1364- receives the rule name: ` FnMut(&dyn ExecutionPlan, &str) ` instead of
1365- ` FnMut(&dyn ExecutionPlan, &dyn PhysicalOptimizerRule) ` .
1410+ by returning ` SessionState::physical_analyzers ` . The default returns no
1411+ rules, so the session plans without enforcement and fails at planning or at
1412+ execution.
1413+
1414+ - ** Tests that match error messages.** When a rule in the analyzer list fails
1415+ the schema or invariant check, the error now names the phase, for example
1416+ ` PhysicalAnalyzer rule 'join_selection' failed. Schema mismatch. ` instead of
1417+ ` PhysicalOptimizer rule 'join_selection' failed. Schema mismatch. `
1418+
1419+ - ** Tests that match ` EXPLAIN VERBOSE ` output.** The rules that ran in both
1420+ lists now have their own names: ` OutputRequirements ` is
1421+ ` AddOutputRequirements ` in the analyzer and ` RemoveOutputRequirements ` in the
1422+ optimizer, and ` FilterPushdown(Post) ` is ` DynamicFilterPushdown ` .
1423+
1424+ See [ PR #26757 ] ( https://github.com/apache/datafusion/pull/26757 ) for details.
0 commit comments