Repository navigation
Commit 0d3aac0
Check the final physical plan in the planner and name the rule that broke it (apache#26461)
## Which issue does this PR close?
- Closes apache#26460.
Related: apache#26171 (option 3), apache#26351, apache#26360, apache#25688.
## Rationale for this change
`SanityCheckPlan` is the last rule in `PhysicalOptimizer::new()`, so
rules appended with `SessionStateBuilder::with_physical_optimizer_rule`
run after it and are never checked. On `main`, an appended rule that
removes a window's sort silently returns wrong `row_number()` values.
Custom pipelines have to keep the rule last by hand, and a custom list
without it runs plans whose requirements do not hold. When the check
does fail, the error only says `SanityCheckPlan`, so finding the rule
that broke the plan has needed a manual bisect each time (for example
apache#26391, apache#17077, apache#25921).
@alamb suggested this change in the review of apache#26107: remove
`SanityCheckPlan` as an optimizer pass and always run it after the
optimizer is done
([comment](apache#26107 (comment))).
apache#26171 agreed (option 3) that a custom pipeline must keep the plan
valid, while optimization coverage is not guaranteed. This PR makes that
validity part checked for every pipeline. The review on apache#26360 asked for
the same shape: one rule, `EnsureRequirements`, makes the plan valid,
and every later rule keeps it valid.
## What changes are included in this PR?
- `SanityCheckPlan` leaves the default rule list.
`DefaultPhysicalPlanner::optimize_physical_plan` runs the same check
(unbounded inputs, ordering, distribution,
`check_invariants(Executable)`) after the last rule, including appended
rules. It replaces the planner's own `InvariantChecker(Executable)`
walk, which `SanityCheckPlan` repeated.
- When the ordering or distribution requirements fail, the planner runs
the rules again from the input plan to name the rule that broke them, so
planning that succeeds keeps no intermediate plans:
```text
SanityCheckPlan: order and distribution requirements stopped holding
after physical optimizer rule 'ProjectionPushdown'
caused by
Error during planning: Plan: [...] does not satisfy order requirements:
[...]
```
or, when the plan was already invalid as `EnsureRequirements` produced
it:
```text
SanityCheckPlan: the plan produced by physical optimizer rule
'EnsureRequirements' does not satisfy the order and distribution
requirements
```
Plans before `EnsureRequirements` need not satisfy the requirements, so
only plans from its last run onwards are checked. The rule is recognized
by its name; a separate enforcement phase (apache#25688) would give a cleaner
boundary. Errors about unbounded inputs keep the plain `SanityCheckPlan`
context, so existing messages do not change.
- `sanity_checker` exposes `check_plan_tree_sanity`,
`check_input_requirements` and `input_requirements_hold`, which the
planner uses.
- `SanityCheckPlan` is deprecated. Tests that used it to validate a plan
call `check_plan_tree_sanity` instead.
- The upgrade guide, the optimizer rule reference and `explain.slt` are
updated.
## What is the testing strategy for this PR?
- `sanity_checker.rs`: a rule appended after the default rules that
removes sorts is reported by name; a custom list without
`EnsureRequirements` reports that no rule produced a valid plan.
- `memory_limit/mod.rs`: the two sort merge join tests ran a
`SortMergeJoinExec` over unsorted streams through a custom rule list
without `EnsureRequirements`. They now sort the generated data by the
join keys and declare that order, so the plan is valid and the tests
still measure the join's memory use.
- The extended workspace suite passes.
## Are there any user-facing changes?
- Plans produced by rules added with `with_physical_optimizer_rule`, or
by a custom rule list, are now checked. A plan whose requirements do not
hold, such as a sort merge join over unsorted input, fails to plan
instead of running and possibly returning wrong results.
- `PhysicalOptimizer::new()` no longer contains `SanityCheckPlan`, and
`EXPLAIN VERBOSE` no longer prints `physical_plan after
SanityCheckPlan`. Code that looks the rule up in the default list no
longer finds it. `SanityCheckPlan` itself is deprecated; code that runs
optimizer rules without the planner can call
`sanity_checker::check_plan_tree_sanity` on the final plan.
- Errors for broken ordering or distribution requirements name the rule
that broke them.
This changes the public behavior of the default rule list, so it needs
the `api change` label.
---------
Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>1 parent 503fde8 commit 0d3aac0
11 files changed
Lines changed: 349 additions & 99 deletions
File tree
- datafusion
- core
- src
- tests
- memory_limit
- physical_optimizer
- physical-optimizer/src
- sqllogictest/test_files
- docs/source/library-user-guide/upgrading
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
97 | 101 | | |
98 | 102 | | |
99 | 103 | | |
| |||
3481 | 3485 | | |
3482 | 3486 | | |
3483 | 3487 | | |
3484 | | - | |
3485 | | - | |
3486 | | - | |
| 3488 | + | |
| 3489 | + | |
| 3490 | + | |
| 3491 | + | |
| 3492 | + | |
| 3493 | + | |
| 3494 | + | |
| 3495 | + | |
| 3496 | + | |
| 3497 | + | |
| 3498 | + | |
| 3499 | + | |
3487 | 3500 | | |
3488 | 3501 | | |
3489 | 3502 | | |
| |||
3693 | 3706 | | |
3694 | 3707 | | |
3695 | 3708 | | |
| 3709 | + | |
| 3710 | + | |
| 3711 | + | |
| 3712 | + | |
| 3713 | + | |
| 3714 | + | |
| 3715 | + | |
| 3716 | + | |
| 3717 | + | |
| 3718 | + | |
| 3719 | + | |
| 3720 | + | |
| 3721 | + | |
| 3722 | + | |
| 3723 | + | |
| 3724 | + | |
| 3725 | + | |
| 3726 | + | |
| 3727 | + | |
| 3728 | + | |
| 3729 | + | |
| 3730 | + | |
| 3731 | + | |
| 3732 | + | |
| 3733 | + | |
| 3734 | + | |
| 3735 | + | |
| 3736 | + | |
| 3737 | + | |
| 3738 | + | |
| 3739 | + | |
| 3740 | + | |
| 3741 | + | |
| 3742 | + | |
| 3743 | + | |
| 3744 | + | |
| 3745 | + | |
| 3746 | + | |
| 3747 | + | |
| 3748 | + | |
| 3749 | + | |
| 3750 | + | |
| 3751 | + | |
| 3752 | + | |
| 3753 | + | |
| 3754 | + | |
| 3755 | + | |
| 3756 | + | |
| 3757 | + | |
| 3758 | + | |
| 3759 | + | |
| 3760 | + | |
| 3761 | + | |
| 3762 | + | |
| 3763 | + | |
| 3764 | + | |
| 3765 | + | |
| 3766 | + | |
| 3767 | + | |
| 3768 | + | |
| 3769 | + | |
| 3770 | + | |
| 3771 | + | |
| 3772 | + | |
| 3773 | + | |
| 3774 | + | |
| 3775 | + | |
3696 | 3776 | | |
3697 | 3777 | | |
3698 | 3778 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
578 | 578 | | |
579 | 579 | | |
580 | 580 | | |
581 | | - | |
| 581 | + | |
582 | 582 | | |
583 | 583 | | |
584 | 584 | | |
| |||
597 | 597 | | |
598 | 598 | | |
599 | 599 | | |
600 | | - | |
| 600 | + | |
601 | 601 | | |
602 | 602 | | |
603 | 603 | | |
| |||
1461 | 1461 | | |
1462 | 1462 | | |
1463 | 1463 | | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
1464 | 1469 | | |
1465 | 1470 | | |
1466 | 1471 | | |
| |||
1522 | 1527 | | |
1523 | 1528 | | |
1524 | 1529 | | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
1525 | 1547 | | |
1526 | 1548 | | |
1527 | 1549 | | |
| |||
1570 | 1592 | | |
1571 | 1593 | | |
1572 | 1594 | | |
1573 | | - | |
| 1595 | + | |
1574 | 1596 | | |
1575 | 1597 | | |
1576 | 1598 | | |
| |||
1646 | 1668 | | |
1647 | 1669 | | |
1648 | 1670 | | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
1649 | 1691 | | |
1650 | 1692 | | |
1651 | 1693 | | |
| |||
0 commit comments