Repository navigation
Commit 483a48b
[branch-54] fix: keep a CoalescePartitionsExec required by a SinglePartition child (#26961)
Backport of #23948 to `branch-54`. Credits: @adriangb
## Which issue does this PR close?
- Related to #23948
- Related to #23805
## Rationale for this change
A valid query can be planned into a physical plan that `SanityCheckPlan`
then rejects with `does not satisfy distribution requirements:
SinglePartition`. A `CollectLeft` `HashJoinExec` requires
`SinglePartition` on its build (left) child. `EnforceDistribution` adds
a `CoalescePartitionsExec` there, but the sort-parallelization phase of
`EnforceSorting` (`parallelize_sorts`) removes it again:
`remove_bottleneck_in_subplan` drops a coalesce at `children[0]` based
on its position, without checking the parent's distribution requirement
for that child. See #23948 for the full analysis and reproducer.
## What changes are included in this PR?
The fix from #23948: `remove_bottleneck_in_subplan` now checks the
parent's per-child distribution requirement before removing a coalesce.
The node being rewritten (the root of the call) is exempt.
Differences from the original, needed for `branch-54`:
- The distribution requirements are read with
`required_input_distribution()` / `.get(idx)` instead of
`input_distribution_requirements()` / `child_distribution(idx)`, which
don't exist on this branch. The behavior is the same.
- On this branch the logic lives in `EnforceSorting`
(`datafusion/physical-optimizer/src/enforce_sorting/mod.rs`) rather than
`EnsureRequirements`.
## Are these changes tested?
Yes:
- The end-to-end sqllogictest in `joins.slt` is the same as in the
original (only a comment changes, to name `EnforceSorting`).
- `ensure_requirements.rs` doesn't exist on this branch, so the two unit
tests are in
`datafusion/core/tests/physical_optimizer/enforce_sorting.rs`. They
cover the same two build-side shapes (`UnknownPartitioning(n)` and
`Hash([k], n)`). Each test builds the plan shape that
`parallelize_sorts` receives, runs that phase directly, and then checks
the result with `SanityCheckPlan`. Running the full rule would rewrite
the plan before it reaches the code path under test.
## Are there any user-facing changes?
No API changes. Plans that `SanityCheckPlan` used to reject now plan and
execute.
---------
Co-authored-by: Adrian Garcia Badaracco <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>1 parent 1e317c7 commit 483a48b
3 files changed
Lines changed: 245 additions & 4 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-optimizer/src/enforce_sorting
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| 59 | + | |
59 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
60 | 64 | | |
61 | 65 | | |
62 | 66 | | |
| |||
2850 | 2854 | | |
2851 | 2855 | | |
2852 | 2856 | | |
| 2857 | + | |
| 2858 | + | |
| 2859 | + | |
| 2860 | + | |
| 2861 | + | |
| 2862 | + | |
| 2863 | + | |
| 2864 | + | |
| 2865 | + | |
| 2866 | + | |
| 2867 | + | |
| 2868 | + | |
| 2869 | + | |
| 2870 | + | |
| 2871 | + | |
| 2872 | + | |
| 2873 | + | |
| 2874 | + | |
| 2875 | + | |
| 2876 | + | |
| 2877 | + | |
| 2878 | + | |
| 2879 | + | |
| 2880 | + | |
| 2881 | + | |
| 2882 | + | |
| 2883 | + | |
| 2884 | + | |
| 2885 | + | |
| 2886 | + | |
| 2887 | + | |
| 2888 | + | |
| 2889 | + | |
| 2890 | + | |
| 2891 | + | |
| 2892 | + | |
| 2893 | + | |
| 2894 | + | |
| 2895 | + | |
| 2896 | + | |
| 2897 | + | |
| 2898 | + | |
| 2899 | + | |
| 2900 | + | |
| 2901 | + | |
| 2902 | + | |
| 2903 | + | |
| 2904 | + | |
| 2905 | + | |
| 2906 | + | |
| 2907 | + | |
| 2908 | + | |
| 2909 | + | |
| 2910 | + | |
| 2911 | + | |
| 2912 | + | |
| 2913 | + | |
| 2914 | + | |
| 2915 | + | |
| 2916 | + | |
| 2917 | + | |
| 2918 | + | |
| 2919 | + | |
| 2920 | + | |
| 2921 | + | |
| 2922 | + | |
| 2923 | + | |
| 2924 | + | |
| 2925 | + | |
| 2926 | + | |
| 2927 | + | |
| 2928 | + | |
| 2929 | + | |
| 2930 | + | |
| 2931 | + | |
| 2932 | + | |
| 2933 | + | |
| 2934 | + | |
| 2935 | + | |
| 2936 | + | |
| 2937 | + | |
| 2938 | + | |
| 2939 | + | |
| 2940 | + | |
| 2941 | + | |
| 2942 | + | |
| 2943 | + | |
| 2944 | + | |
| 2945 | + | |
| 2946 | + | |
| 2947 | + | |
| 2948 | + | |
| 2949 | + | |
| 2950 | + | |
| 2951 | + | |
| 2952 | + | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
| 2956 | + | |
| 2957 | + | |
| 2958 | + | |
| 2959 | + | |
| 2960 | + | |
| 2961 | + | |
| 2962 | + | |
| 2963 | + | |
| 2964 | + | |
| 2965 | + | |
| 2966 | + | |
| 2967 | + | |
| 2968 | + | |
| 2969 | + | |
| 2970 | + | |
| 2971 | + | |
| 2972 | + | |
| 2973 | + | |
| 2974 | + | |
| 2975 | + | |
| 2976 | + | |
| 2977 | + | |
| 2978 | + | |
| 2979 | + | |
| 2980 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
676 | 676 | | |
677 | 677 | | |
678 | 678 | | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
679 | 689 | | |
| 690 | + | |
680 | 691 | | |
681 | 692 | | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
682 | 716 | | |
683 | | - | |
| 717 | + | |
684 | 718 | | |
685 | 719 | | |
686 | 720 | | |
| |||
694 | 728 | | |
695 | 729 | | |
696 | 730 | | |
697 | | - | |
698 | | - | |
699 | | - | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
700 | 739 | | |
701 | 740 | | |
702 | 741 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5571 | 5571 | | |
5572 | 5572 | | |
5573 | 5573 | | |
| 5574 | + | |
| 5575 | + | |
| 5576 | + | |
| 5577 | + | |
| 5578 | + | |
| 5579 | + | |
| 5580 | + | |
| 5581 | + | |
| 5582 | + | |
| 5583 | + | |
| 5584 | + | |
| 5585 | + | |
| 5586 | + | |
| 5587 | + | |
| 5588 | + | |
| 5589 | + | |
| 5590 | + | |
| 5591 | + | |
| 5592 | + | |
| 5593 | + | |
| 5594 | + | |
| 5595 | + | |
| 5596 | + | |
| 5597 | + | |
| 5598 | + | |
| 5599 | + | |
| 5600 | + | |
| 5601 | + | |
| 5602 | + | |
| 5603 | + | |
| 5604 | + | |
| 5605 | + | |
| 5606 | + | |
| 5607 | + | |
| 5608 | + | |
| 5609 | + | |
| 5610 | + | |
| 5611 | + | |
| 5612 | + | |
| 5613 | + | |
| 5614 | + | |
| 5615 | + | |
| 5616 | + | |
| 5617 | + | |
| 5618 | + | |
| 5619 | + | |
| 5620 | + | |
| 5621 | + | |
| 5622 | + | |
| 5623 | + | |
| 5624 | + | |
| 5625 | + | |
| 5626 | + | |
| 5627 | + | |
| 5628 | + | |
| 5629 | + | |
| 5630 | + | |
| 5631 | + | |
| 5632 | + | |
| 5633 | + | |
| 5634 | + | |
| 5635 | + | |
| 5636 | + | |
| 5637 | + | |
| 5638 | + | |
| 5639 | + | |
| 5640 | + | |
| 5641 | + | |
| 5642 | + | |
| 5643 | + | |
| 5644 | + | |
| 5645 | + | |
| 5646 | + | |
| 5647 | + | |
0 commit comments