@@ -249,16 +249,18 @@ SELECT ts_tz - ts FROM timestamps;
249249owns the session. It now returns a ` NotImplemented ` error because forwarding can
250250re-enter an installed foreign planner, and the execution-plan handle returned by
251251the old callback cannot restore local Rust type identities for downcasting.
252- The original ` FFI_SessionRef ` callback slot remains in place for ABI compatibility
253- with DataFusion 55 consumers, but calling that callback returns the same error.
252+ The original ` FFI_SessionRef ` callback slot remains in place, but calling that
253+ callback returns the same error.
254254
255255The session-owning library should instead export its original planner as a
256256` datafusion_ffi::query_planner::FFI_QueryPlanner ` before installing a foreign
257257planner. The foreign planner can retain and invoke that handle to receive a
258258serialized physical plan reconstructed with local type identities. See the
259259` datafusion_ffi::query_planner ` module documentation for the complete delegation
260- pattern. ` ForeignSession::query_planner ` , ` optimize ` , and ` physical_optimizers `
261- continue to forward to the owning session across the FFI boundary.
260+ pattern. ` ForeignSession::query_planner ` , ` optimize ` , ` physical_optimizers ` and
261+ ` physical_analyzers ` forward to the owning session across the FFI boundary.
262+ ` FFI_SessionRef ` has a new field for the physical analyzer rules, so FFI
263+ libraries must be rebuilt against this release.
262264
263265### Range partitioning supports retained samples and scaling
264266
@@ -1337,29 +1339,95 @@ Code that reads `FileScanConfig::preserve_order` to decide whether files may be
13371339reordered should call ` FileScanConfig::preserves_file_order() ` instead, which
13381340also accounts for the output ordering.
13391341
1340- ### Physical planning gained an analyzer phase
1342+ ### ` EnsureRequirements ` and the rules before it moved to a new physical analyzer list
1343+
1344+ Physical planning now runs two rule lists: the physical analyzer rules
1345+ (` PhysicalAnalyzerRule ` ), which run before distribution and ordering
1346+ requirements are enforced and end with ` EnsureRequirements ` , then the physical
1347+ optimizer rules, which receive a valid plan and must return one. The default
1348+ analyzer list is the start of the old optimizer list, up to and including
1349+ ` EnsureRequirements ` , in the same order (` AddOutputRequirements ` ,
1350+ ` aggregate_statistics ` , ` EliminateConstantJoin ` , ` join_selection ` ,
1351+ ` LimitedDistinctAggregation ` , ` FilterPushdown ` , ` WindowTopN ` ,
1352+ ` EnsureRequirements ` ), so default plans do not change. Every optimizer rule now receives a plan that meets its
1353+ distribution and ordering requirements. ` EXPLAIN VERBOSE ` and the observer
1354+ passed to ` DefaultPhysicalPlanner::optimize_physical_plan ` show the analyzer
1355+ rules too; the observer now receives the rule name (` &str ` ) instead of a
1356+ ` &dyn PhysicalOptimizerRule ` .
13411357
1342- Physical planning now runs two rule lists, mirroring the logical ` Analyzer ` /
1343- ` Optimizer ` split. The new ` PhysicalAnalyzerRule ` s run first. The default
1344- analyzer list is the former optimizer list up to and including
1345- ` EnsureRequirements ` (` OutputRequirements ` add mode, ` aggregate_statistics ` ,
1346- ` EliminateConstantJoin ` , ` join_selection ` , ` LimitedDistinctAggregation ` ,
1347- ` FilterPushdown ` , ` WindowTopN ` , ` EnsureRequirements ` ), in the same order, so
1348- default plans do not change. The
1349- ` PhysicalOptimizerRule ` s then run on that plan. ` EXPLAIN VERBOSE ` shows the
1350- analyzer passes as well.
1351-
1352- ** Migration guide:**
1358+ ** Who is affected:**
13531359
1354- - A type that implements the ` Session ` trait and overrides
1360+ - ** Rules placed in front of the default optimizer list.** A list built as
1361+ your rules followed by ` PhysicalOptimizer::new().rules ` used to run your
1362+ rules before ` EnsureRequirements ` . They now run after the whole analyzer
1363+ phase, enforcement included, and nothing enforces the plan again after them.
1364+ A rule that has to run before enforcement is an analyzer rule: implement
1365+ ` PhysicalAnalyzerRule ` (` analyze ` instead of ` optimize ` ) and put it in front
1366+ of the default analyzer rules:
1367+
1368+ ``` rust,ignore
1369+ use datafusion::physical_optimizer::analyzer::{PhysicalAnalyzer, PhysicalAnalyzerRule};
1370+
1371+ let mut analyzer_rules: Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> =
1372+ vec![Arc::new(MyRule::new())];
1373+ analyzer_rules.extend(PhysicalAnalyzer::new().rules);
1374+ let state = SessionStateBuilder::new()
1375+ .with_default_features()
1376+ .with_physical_analyzer_rules(analyzer_rules)
1377+ .build();
1378+ ```
1379+
1380+ - ** An empty optimizer list no longer skips enforcement.**
1381+ ` with_physical_optimizer_rules(vec![]) ` still runs the analyzer phase, so the
1382+ plan has the repartitions and sorts enforcement adds. To plan without them
1383+ as before, also call ` with_physical_analyzer_rules(vec![]) ` .
1384+
1385+ - ** Code that wraps, lists or runs the rules itself.**
1386+ ` SessionState::physical_optimizers() ` no longer contains the analyzer rules.
1387+ Tools that wrap every rule, for tracing or for ` EXPLAIN ` output, or that
1388+ list rule names, should handle ` physical_analyzers() ` too. Code that runs the
1389+ rules itself should call ` DefaultPhysicalPlanner::optimize_physical_plan ` , or
1390+ run the ` physical_analyzers() ` rules first.
1391+
1392+ - ** Code that removes or places rules by name.** Filters on
1393+ ` PhysicalOptimizer::new().rules ` still compile, but the analyzer rules are no
1394+ longer in that list, so a filter on ` join_selection ` , ` EnsureRequirements ` ,
1395+ ` FilterPushdown ` , ` WindowTopN ` , ` aggregate_statistics ` ,
1396+ ` EliminateConstantJoin ` or ` LimitedDistinctAggregation ` matches nothing and
1397+ the rule keeps running. Filter ` PhysicalAnalyzer::new().rules ` instead. The
1398+ renamed rules are ` OutputRequirements ` , now ` AddOutputRequirements ` and
1399+ ` RemoveOutputRequirements ` , and ` FilterPushdown(Post) ` , now
1400+ ` DynamicFilterPushdown ` .
1401+
1402+ - ** Code that uses the moved rules directly.** The rules in the analyzer list
1403+ implement ` PhysicalAnalyzerRule ` instead of ` PhysicalOptimizerRule ` : call
1404+ ` analyze ` instead of ` optimize ` (for example
1405+ ` EnsureRequirements::new().analyze(plan, config) ` ), and keep them in a
1406+ ` Vec<Arc<dyn PhysicalAnalyzerRule + Send + Sync>> ` . Rules that ran in both
1407+ lists are now two types: ` OutputRequirements::new_add_mode() ` is
1408+ ` AddOutputRequirements::new() ` (analyzer) and ` new_remove_mode() ` is
1409+ ` RemoveOutputRequirements::new() ` (optimizer);
1410+ ` FilterPushdown::new_post_optimization() ` is ` DynamicFilterPushdown::new() `
1411+ (optimizer), and ` FilterPushdown::new() ` is the analyzer rule.
1412+
1413+ - ** Custom optimizer lists copied from the old default.** They no longer
1414+ compile, because the analyzer rules are not ` PhysicalOptimizerRule ` s. Drop
1415+ those rules from the list; the analyzer list runs them.
1416+
1417+ - ** Custom ` Session ` implementations.** A type that overrides
13551418 ` physical_optimizers ` should also override ` physical_analyzers ` , for example
1356- by returning ` SessionState::physical_analyzers ` . The default implementation
1357- returns no rules, so such a session would otherwise run no enforcement.
1358- - The default analyzer rules run even when a custom optimizer list is set with
1359- ` SessionStateBuilder::with_physical_optimizer_rules ` . A custom list that still
1360- contains a rule now in the analyzer (for example ` EnsureRequirements ` ) runs it
1361- twice; drop it from the list, or pass ` with_physical_analyzer_rules(vec![]) `
1362- to keep the previous single-list behavior.
1363- - The observer passed to ` DefaultPhysicalPlanner::optimize_physical_plan ` now
1364- receives the rule name: ` FnMut(&dyn ExecutionPlan, &str) ` instead of
1365- ` FnMut(&dyn ExecutionPlan, &dyn PhysicalOptimizerRule) ` .
1419+ by returning ` SessionState::physical_analyzers ` . The default returns no
1420+ rules, so the session plans without enforcement and fails at planning or at
1421+ execution.
1422+
1423+ - ** Tests that match error messages.** When a rule in the analyzer list fails
1424+ the schema or invariant check, the error now names the phase, for example
1425+ ` PhysicalAnalyzer rule 'join_selection' failed. Schema mismatch. ` instead of
1426+ ` PhysicalOptimizer rule 'join_selection' failed. Schema mismatch. `
1427+
1428+ - ** Tests that match ` EXPLAIN VERBOSE ` output.** The rules that ran in both
1429+ lists now have their own names: ` OutputRequirements ` is
1430+ ` AddOutputRequirements ` in the analyzer and ` RemoveOutputRequirements ` in the
1431+ optimizer, and ` FilterPushdown(Post) ` is ` DynamicFilterPushdown ` .
1432+
1433+ See [ PR #26757 ] ( https://github.com/apache/datafusion/pull/26757 ) for details.
0 commit comments