Repository navigation
Commit 13ea24a
authored
## Which issue does this PR close?
- Closes apache#21826. Builds on apache#24959, which handled the
`EnsureRequirements` side.
## Rationale for this change
`InterleaveExec` is only valid while every child shares the same hash
(or
range) partitioning. `EnsureRequirements` creates it from a `UnionExec`
when
that happens to hold. Any later rule that rewrites a child and changes
its
output partitioning, for example `JoinSelection` swapping a join's
sides,
makes the optimizer's tree walk rebuild the interleave over children
that no
longer match. `InterleaveExec::replace_children` asserted on that, so
the
whole rule failed:
```
join_selection
caused by
Internal error: Assertion failed: can_interleave(children.iter()):
Can not create InterleaveExec: new children can not be interleaved.
```
This is the chain reported in apache#21826: distribution enforcement builds
the
interleave, `JoinSelection` breaks it, and no rule can repair it because
the
failure happens inside the rebuild, before any rule's closure runs.
apache#24959 made `EnsureRequirements` itself immune by normalizing
interleaves back
to unions before it runs, but that does not help when the rebuild
happens
inside another rule.
The root cause is that `InterleaveExec` enforces a cross-child
distribution
property one level earlier than the rest of the framework does.
Co-partitioning
for joins is validated by
`InputDistributionRequirements::check_invariants`
only at `InvariantLevel::Executable`; a hash join rebuilt over children
that
lost their partitioning does not error at rebuild time, it becomes
temporarily
unsatisfied and is either repaired by a later distribution pass or
rejected by
`SanityCheckPlan`. This PR makes `InterleaveExec` behave the same way.
This is not the fallback proposed in apache#21827. Nothing is substituted for
a
different node and no error is downgraded to a log. The node stays an
`InterleaveExec`, the error is the same, and it is raised at the
executable
checkpoint instead of at rebuild time. The decision on how to recover is
left
to the optimizer rule, which apache#24959 already implements.
## What changes are included in this PR?
In `datafusion/physical-plan/src/union.rs`:
- `InterleaveExec::try_new` still asserts interleavability, so explicit
construction and proto decoding keep the strict check.
- A private `try_new_unchecked` builds the node without the check.
`replace_children` in `Recompute` mode now uses it.
- `compute_properties` only claims the shared hash / range partitioning
when
every child still has it, and reports `UnknownPartitioning` otherwise,
so
nothing downstream can rely on a layout the node does not have.
- `InterleaveExec::check_invariants` runs the `can_interleave` check at
`InvariantLevel::Executable`, with the same message as before.
The physical planner already checks `Always` after each rule and
`Executable`
at the end, so an unrepaired interleave is still rejected, with the same
error, by `SanityCheckPlan`.
## What is the testing strategy for this PR?
- `test_interleave_rebuild_defers_partitioning_invariant` (unit test in
`union.rs`): rebuilding over one round-robin child succeeds, `try_new`
on
the same children still fails, the node reports unknown partitioning,
the
`Always` check passes, and the `Executable` check fails with the
original
message.
- `issue_21826_join_selection_after_distribution_pass`
(`core/tests/physical_optimizer/enforce_distribution.rs`): the exact
reported chain. A union of two partitioned hash joins with statistics
chosen
so `JoinSelection` swaps only one of them. On `main` this fails inside
`JoinSelection` with the error above. Here `JoinSelection` completes,
the
interleave reports unknown partitioning, `SanityCheckPlan` rejects the
unrepaired plan, and a second `EnsureRequirements` pass produces a valid
union that the sanity checker accepts (snapshotted).
- `interleave_broken_by_later_rewrite_is_repaired_on_next_pass`: the
same
mechanism where the disruption is a removable `CoalescePartitionsExec`,
so
the repair restores the interleave rather than demoting it, showing the
optimization is not permanently lost.
## Are there any user-facing changes?
No API changes. Behaviorally, an `InterleaveExec` whose children lose
their
shared partitioning during optimization is now reported as an invariant
failure by `SanityCheckPlan` at the end of the pipeline instead of as an
error
from the rule that rebuilt it. The diagnostic is therefore coarser (it
names
the node, not the rule), which is the trade-off for letting rules
complete and
a later distribution pass repair the plan. Custom pipelines that run
rules
between two distribution passes no longer need a workaround.
1 parent c98fe99 commit 13ea24a
2 files changed
Lines changed: 274 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
| 44 | + | |
43 | 45 | | |
44 | 46 | | |
45 | 47 | | |
| |||
56 | 58 | | |
57 | 59 | | |
58 | 60 | | |
| 61 | + | |
59 | 62 | | |
| 63 | + | |
60 | 64 | | |
61 | 65 | | |
62 | 66 | | |
| |||
2856 | 2860 | | |
2857 | 2861 | | |
2858 | 2862 | | |
| 2863 | + | |
| 2864 | + | |
| 2865 | + | |
| 2866 | + | |
| 2867 | + | |
| 2868 | + | |
| 2869 | + | |
| 2870 | + | |
| 2871 | + | |
| 2872 | + | |
| 2873 | + | |
| 2874 | + | |
| 2875 | + | |
| 2876 | + | |
| 2877 | + | |
| 2878 | + | |
| 2879 | + | |
| 2880 | + | |
| 2881 | + | |
| 2882 | + | |
| 2883 | + | |
| 2884 | + | |
| 2885 | + | |
| 2886 | + | |
| 2887 | + | |
| 2888 | + | |
| 2889 | + | |
| 2890 | + | |
| 2891 | + | |
| 2892 | + | |
| 2893 | + | |
| 2894 | + | |
| 2895 | + | |
| 2896 | + | |
| 2897 | + | |
| 2898 | + | |
| 2899 | + | |
| 2900 | + | |
| 2901 | + | |
| 2902 | + | |
| 2903 | + | |
| 2904 | + | |
| 2905 | + | |
| 2906 | + | |
| 2907 | + | |
| 2908 | + | |
| 2909 | + | |
| 2910 | + | |
| 2911 | + | |
| 2912 | + | |
| 2913 | + | |
| 2914 | + | |
| 2915 | + | |
| 2916 | + | |
| 2917 | + | |
| 2918 | + | |
| 2919 | + | |
| 2920 | + | |
| 2921 | + | |
| 2922 | + | |
| 2923 | + | |
| 2924 | + | |
| 2925 | + | |
| 2926 | + | |
| 2927 | + | |
| 2928 | + | |
| 2929 | + | |
| 2930 | + | |
| 2931 | + | |
| 2932 | + | |
| 2933 | + | |
| 2934 | + | |
| 2935 | + | |
| 2936 | + | |
| 2937 | + | |
| 2938 | + | |
| 2939 | + | |
| 2940 | + | |
| 2941 | + | |
| 2942 | + | |
| 2943 | + | |
| 2944 | + | |
| 2945 | + | |
| 2946 | + | |
| 2947 | + | |
| 2948 | + | |
| 2949 | + | |
| 2950 | + | |
| 2951 | + | |
| 2952 | + | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
| 2956 | + | |
| 2957 | + | |
| 2958 | + | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
| 2969 | + | |
| 2970 | + | |
| 2971 | + | |
| 2972 | + | |
| 2973 | + | |
| 2974 | + | |
| 2975 | + | |
| 2976 | + | |
| 2977 | + | |
| 2978 | + | |
| 2979 | + | |
| 2980 | + | |
| 2981 | + | |
| 2982 | + | |
| 2983 | + | |
| 2984 | + | |
| 2985 | + | |
| 2986 | + | |
| 2987 | + | |
| 2988 | + | |
| 2989 | + | |
| 2990 | + | |
| 2991 | + | |
| 2992 | + | |
| 2993 | + | |
| 2994 | + | |
| 2995 | + | |
| 2996 | + | |
| 2997 | + | |
| 2998 | + | |
2859 | 2999 | | |
2860 | 3000 | | |
2861 | 3001 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
670 | 670 | | |
671 | 671 | | |
672 | 672 | | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
673 | 690 | | |
674 | 691 | | |
675 | 692 | | |
| |||
694 | 711 | | |
695 | 712 | | |
696 | 713 | | |
697 | | - | |
698 | | - | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
699 | 731 | | |
700 | 732 | | |
701 | 733 | | |
| |||
758 | 790 | | |
759 | 791 | | |
760 | 792 | | |
761 | | - | |
762 | | - | |
763 | | - | |
764 | | - | |
765 | | - | |
766 | | - | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
767 | 796 | | |
768 | 797 | | |
769 | 798 | | |
770 | 799 | | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
771 | 812 | | |
772 | 813 | | |
773 | 814 | | |
| |||
1751 | 1792 | | |
1752 | 1793 | | |
1753 | 1794 | | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
| 1819 | + | |
| 1820 | + | |
| 1821 | + | |
| 1822 | + | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
| 1875 | + | |
| 1876 | + | |
| 1877 | + | |
| 1878 | + | |
| 1879 | + | |
1754 | 1880 | | |
1755 | 1881 | | |
1756 | 1882 | | |
| |||
0 commit comments