Repository navigation
Commit c14633e
authored
## Which issue does this PR close?
- Part of apache#21826.
## Rationale for this change
`EnsureRequirements` fails with an internal error when the plan it is
given
already contains an `InterleaveExec` whose children no longer share a
hash
(or range) partitioning:
```
Internal error: Assertion failed: can_interleave(children.iter()):
Can not create InterleaveExec: new children can not be interleaved.
```
`InterleaveExec` declares no distribution requirement of its own. It is
only
valid while its children happen to share the same partitioning, which is
why
`ensure_distribution` creates it from a `UnionExec` in the first place.
Once
it exists, several rewrites can take that partitioning away: this rule
strips
`RepartitionExec`s that nothing requires, join key reordering changes
the hash
expressions, `JoinSelection` side swaps change output partitioning, and
so on.
`PlanContext::with_new_children` rebuilds every parent from its updated
children while walking the tree, so the interleave is rebuilt over the
changed
children *before* the rule's closure sees the node, and the rebuild
fails.
This affects any pipeline that runs the rule more than once,
re-optimizes a
deserialized plan, or runs `EnforceDistribution` after other rewrites
(the
setup reported in apache#21826). It also breaks the idempotency the rule
advertises.
The discussion on apache#21827 concluded that the fix belongs at the
`EnforceDistribution` call site rather than as a silent fallback in
`InterleaveExec::with_new_children`; this PR is that fix.
## What changes are included in this PR?
- New `replace_interleave_with_union` helper in
`ensure_requirements/enforce_distribution.rs`, which turns an
`InterleaveExec` back into the equivalent `UnionExec`.
- `EnsureRequirements::optimize` runs it as a top-down "Phase 0" before
join
key reordering. Phase 2 then re-derives interleaves from unions wherever
the
children still qualify, exactly as it already re-derives
`RepartitionExec`,
`CoalescePartitionsExec` and `SortPreservingMergeExec`.
- The pass is top-down on purpose: demoting a nested interleave first
would
change its output partitioning and make the parent interleave fail to
rebuild.
- Module docs updated with the new phase.
A minimal reproducer that fails on `main` and passes here:
```rust
// Interleave over two hash repartitions that nothing above requires.
let plan = InterleaveExec::try_new(vec![
RepartitionExec::try_new(parquet_exec(), Partitioning::Hash(vec![col_a], 10))?,
RepartitionExec::try_new(parquet_exec(), Partitioning::Hash(vec![col_a], 10))?,
])?;
EnsureRequirements::new().optimize(Arc::new(plan), &config)?;
```
## What is the testing strategy for this PR?
Three new tests in
`datafusion/core/tests/physical_optimizer/enforce_distribution.rs`, each
run
through the existing multi-pass harness (`DISTRIB_DISTRIB_SORT` and
`SORT_DISTRIB_DISTRIB`) so idempotency is checked as well:
- `interleave_falls_back_to_union_when_children_lose_partitioning`: the
reproducer above; the result is a `UnionExec` over the scans.
- `interleave_fallback_still_satisfies_parent_hash_requirement`: same
input
under a `FinalPartitioned` aggregate; the aggregate gets a single hash
repartition above the union.
- `existing_interleave_is_kept_when_children_stay_interleavable`: an
interleave over hash partitioned aggregates is re-derived as an
interleave,
and stays a union with `prefer_existing_union = true`.
The test harness's tree-node integrity block now applies the same
pre-pass so
it mirrors the rule. Existing `union_to_interleave` and
`union_not_to_interleave` tests are unchanged, and `union.slt` passes.
## Are there any user-facing changes?
No API changes. One behavioral note: with
`datafusion.optimizer.prefer_existing_union = true`, an `InterleaveExec`
present in the input plan is now emitted as a `UnionExec`. The rule
never
creates interleaves under that setting, and plans produced by the
default
pipeline are unaffected because `EnsureRequirements` is the only creator
of
`InterleaveExec` and runs once.
1 parent 1d9a982 commit c14633e
3 files changed
Lines changed: 207 additions & 1 deletion
File tree
- datafusion
- core/tests/physical_optimizer
- physical-optimizer/src/ensure_requirements
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| 74 | + | |
74 | 75 | | |
75 | | - | |
| 76 | + | |
76 | 77 | | |
77 | 78 | | |
78 | 79 | | |
| |||
731 | 732 | | |
732 | 733 | | |
733 | 734 | | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
734 | 741 | | |
735 | 742 | | |
736 | 743 | | |
| |||
2694 | 2701 | | |
2695 | 2702 | | |
2696 | 2703 | | |
| 2704 | + | |
| 2705 | + | |
| 2706 | + | |
| 2707 | + | |
| 2708 | + | |
| 2709 | + | |
| 2710 | + | |
| 2711 | + | |
| 2712 | + | |
| 2713 | + | |
| 2714 | + | |
| 2715 | + | |
| 2716 | + | |
| 2717 | + | |
| 2718 | + | |
| 2719 | + | |
| 2720 | + | |
| 2721 | + | |
| 2722 | + | |
| 2723 | + | |
| 2724 | + | |
| 2725 | + | |
| 2726 | + | |
| 2727 | + | |
| 2728 | + | |
| 2729 | + | |
| 2730 | + | |
| 2731 | + | |
| 2732 | + | |
| 2733 | + | |
| 2734 | + | |
| 2735 | + | |
| 2736 | + | |
| 2737 | + | |
| 2738 | + | |
| 2739 | + | |
| 2740 | + | |
| 2741 | + | |
| 2742 | + | |
| 2743 | + | |
| 2744 | + | |
| 2745 | + | |
| 2746 | + | |
| 2747 | + | |
| 2748 | + | |
| 2749 | + | |
| 2750 | + | |
| 2751 | + | |
| 2752 | + | |
| 2753 | + | |
| 2754 | + | |
| 2755 | + | |
| 2756 | + | |
| 2757 | + | |
| 2758 | + | |
| 2759 | + | |
| 2760 | + | |
| 2761 | + | |
| 2762 | + | |
| 2763 | + | |
| 2764 | + | |
| 2765 | + | |
| 2766 | + | |
| 2767 | + | |
| 2768 | + | |
| 2769 | + | |
| 2770 | + | |
| 2771 | + | |
| 2772 | + | |
| 2773 | + | |
| 2774 | + | |
| 2775 | + | |
| 2776 | + | |
| 2777 | + | |
| 2778 | + | |
| 2779 | + | |
| 2780 | + | |
| 2781 | + | |
| 2782 | + | |
| 2783 | + | |
| 2784 | + | |
| 2785 | + | |
| 2786 | + | |
| 2787 | + | |
| 2788 | + | |
| 2789 | + | |
| 2790 | + | |
| 2791 | + | |
| 2792 | + | |
| 2793 | + | |
| 2794 | + | |
| 2795 | + | |
| 2796 | + | |
| 2797 | + | |
| 2798 | + | |
| 2799 | + | |
| 2800 | + | |
| 2801 | + | |
| 2802 | + | |
| 2803 | + | |
| 2804 | + | |
| 2805 | + | |
| 2806 | + | |
| 2807 | + | |
| 2808 | + | |
| 2809 | + | |
| 2810 | + | |
| 2811 | + | |
| 2812 | + | |
| 2813 | + | |
| 2814 | + | |
| 2815 | + | |
| 2816 | + | |
| 2817 | + | |
| 2818 | + | |
| 2819 | + | |
| 2820 | + | |
| 2821 | + | |
| 2822 | + | |
| 2823 | + | |
| 2824 | + | |
| 2825 | + | |
| 2826 | + | |
| 2827 | + | |
| 2828 | + | |
| 2829 | + | |
| 2830 | + | |
| 2831 | + | |
| 2832 | + | |
| 2833 | + | |
| 2834 | + | |
| 2835 | + | |
| 2836 | + | |
| 2837 | + | |
| 2838 | + | |
| 2839 | + | |
| 2840 | + | |
| 2841 | + | |
| 2842 | + | |
| 2843 | + | |
| 2844 | + | |
| 2845 | + | |
| 2846 | + | |
| 2847 | + | |
| 2848 | + | |
| 2849 | + | |
| 2850 | + | |
| 2851 | + | |
| 2852 | + | |
| 2853 | + | |
| 2854 | + | |
| 2855 | + | |
| 2856 | + | |
| 2857 | + | |
| 2858 | + | |
2697 | 2859 | | |
2698 | 2860 | | |
2699 | 2861 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
662 | 662 | | |
663 | 663 | | |
664 | 664 | | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
665 | 696 | | |
666 | 697 | | |
667 | 698 | | |
| |||
1708 | 1739 | | |
1709 | 1740 | | |
1710 | 1741 | | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
1711 | 1746 | | |
1712 | 1747 | | |
1713 | 1748 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
| |||
178 | 180 | | |
179 | 181 | | |
180 | 182 | | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
181 | 190 | | |
182 | 191 | | |
183 | 192 | | |
| |||
0 commit comments