Repository navigation
Commit 872df4d
authored
fix: keep join filter columns when pushing a projection below SortMergeJoinExec (#24593)
## Rationale for this change
A sort merge join with a non-equi filter panics (`index out of bounds`
in `get_filter_columns`) when the query does not select a column the
filter references:
```sql
SET datafusion.optimizer.prefer_hash_join = false;
SET datafusion.execution.target_partitions = 2;
CREATE TABLE l (k VARCHAR, a INT, b INT, c INT) AS VALUES ('x', 1, 1, 1), ('y', 2, 2, 2), ('x', 3, 3, 3);
CREATE TABLE r (k VARCHAR, a INT, b INT, c INT) AS VALUES ('x', 1, 1, 1), ('y', 2, 2, 2), ('z', 3, 3, 3);
SELECT l.c, l.k, l.b, r.k, r.b, r.a FROM l RIGHT JOIN r ON l.k = r.k AND l.c >= r.c;
```
```
thread 'tokio-rt-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join/filter.rs:169:34:
index out of bounds: the len is 3 but the index is 3
```
The projection pushdown added in #24517 pushes the projection into the
join's children but passes `self.filter.clone()` through unchanged. The
right child no longer produces `c`, but the filter's `ColumnIndex` still
points at it.
## What changes are included in this PR?
`SortMergeJoinExec::try_swapping_with_projection` now uses the shared
`try_pushdown_through_join_with_column_indices` helper, as
`HashJoinExec` and `NestedLoopJoinExec` already do. It remaps both the
`on` keys and the filter's column indices to the projected children, and
declines the pushdown — embedding the projection in the join instead —
when the filter needs a column the projection drops.
One behavioral side effect: projections containing expressions (e.g.
`length(w.p)`) are now also embedded into the join, so the join only
emits the columns they reference. This matches the existing behavior of
the other join operators; the affected `EXPLAIN ANALYZE` expectations in
`sort_merge_join_spill.slt` are updated accordingly (`projection=[...]`
on the `SortMergeJoinExec` line).
## Are these changes tested?
Yes. A regression test in `joins.slt` covers the panicking case (explain
+ results). Existing SMJ unit tests, projection tests, proto round-trip
tests, and `joins.slt` / `range_partitioning.slt` /
`sort_merge_join_spill.slt` pass.
## Are there any user-facing changes?
No API changes. Plans for `SortMergeJoinExec` under an expression
projection now show `projection=[...]` on the join node.1 parent e1eaeb1 commit 872df4d
4 files changed
Lines changed: 131 additions & 83 deletions
File tree
- datafusion
- physical-plan/src/joins/sort_merge_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
38 | | - | |
39 | | - | |
| 37 | + | |
| 38 | + | |
40 | 39 | | |
41 | 40 | | |
42 | 41 | | |
| |||
707 | 706 | | |
708 | 707 | | |
709 | 708 | | |
710 | | - | |
711 | | - | |
712 | | - | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
713 | 712 | | |
714 | 713 | | |
715 | 714 | | |
716 | 715 | | |
717 | 716 | | |
718 | 717 | | |
719 | 718 | | |
720 | | - | |
721 | | - | |
722 | | - | |
723 | | - | |
724 | | - | |
725 | 719 | | |
726 | | - | |
727 | | - | |
728 | | - | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
729 | 725 | | |
730 | 726 | | |
731 | | - | |
732 | | - | |
733 | | - | |
734 | | - | |
735 | | - | |
736 | | - | |
737 | | - | |
738 | | - | |
739 | | - | |
740 | | - | |
741 | | - | |
742 | | - | |
743 | | - | |
744 | | - | |
745 | | - | |
746 | | - | |
747 | | - | |
748 | | - | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
749 | 739 | | |
750 | | - | |
751 | | - | |
752 | | - | |
753 | | - | |
754 | | - | |
755 | | - | |
756 | | - | |
757 | | - | |
758 | | - | |
759 | | - | |
760 | | - | |
761 | | - | |
762 | | - | |
763 | | - | |
764 | | - | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
765 | 753 | | |
766 | | - | |
767 | | - | |
768 | | - | |
769 | | - | |
770 | | - | |
771 | | - | |
772 | | - | |
773 | | - | |
774 | | - | |
775 | | - | |
776 | | - | |
777 | | - | |
778 | | - | |
779 | | - | |
780 | | - | |
781 | | - | |
782 | | - | |
783 | | - | |
784 | | - | |
785 | | - | |
| 754 | + | |
| 755 | + | |
786 | 756 | | |
787 | 757 | | |
788 | 758 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
365 | 365 | | |
366 | 366 | | |
367 | 367 | | |
368 | | - | |
369 | | - | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
370 | 378 | | |
371 | 379 | | |
372 | 380 | | |
| |||
406 | 414 | | |
407 | 415 | | |
408 | 416 | | |
409 | | - | |
410 | | - | |
411 | | - | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
412 | 420 | | |
413 | 421 | | |
414 | 422 | | |
| |||
428 | 436 | | |
429 | 437 | | |
430 | 438 | | |
431 | | - | |
432 | | - | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
433 | 446 | | |
434 | 447 | | |
435 | 448 | | |
| |||
458 | 471 | | |
459 | 472 | | |
460 | 473 | | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
461 | 509 | | |
462 | 510 | | |
463 | 511 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2857 | 2857 | | |
2858 | 2858 | | |
2859 | 2859 | | |
| 2860 | + | |
| 2861 | + | |
| 2862 | + | |
| 2863 | + | |
| 2864 | + | |
| 2865 | + | |
| 2866 | + | |
| 2867 | + | |
| 2868 | + | |
| 2869 | + | |
| 2870 | + | |
| 2871 | + | |
| 2872 | + | |
| 2873 | + | |
| 2874 | + | |
| 2875 | + | |
| 2876 | + | |
| 2877 | + | |
| 2878 | + | |
| 2879 | + | |
| 2880 | + | |
| 2881 | + | |
| 2882 | + | |
| 2883 | + | |
| 2884 | + | |
| 2885 | + | |
| 2886 | + | |
| 2887 | + | |
| 2888 | + | |
| 2889 | + | |
2860 | 2890 | | |
2861 | 2891 | | |
2862 | 2892 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
143 | | - | |
| 143 | + | |
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| |||
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
155 | | - | |
| 155 | + | |
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
167 | | - | |
| 167 | + | |
168 | 168 | | |
169 | 169 | | |
170 | 170 | | |
| |||
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
179 | | - | |
| 179 | + | |
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| |||
192 | 192 | | |
193 | 193 | | |
194 | 194 | | |
195 | | - | |
| 195 | + | |
196 | 196 | | |
197 | 197 | | |
198 | 198 | | |
| |||
206 | 206 | | |
207 | 207 | | |
208 | 208 | | |
209 | | - | |
| 209 | + | |
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
| |||
220 | 220 | | |
221 | 221 | | |
222 | 222 | | |
223 | | - | |
| 223 | + | |
224 | 224 | | |
225 | 225 | | |
226 | 226 | | |
| |||
234 | 234 | | |
235 | 235 | | |
236 | 236 | | |
237 | | - | |
| 237 | + | |
238 | 238 | | |
239 | 239 | | |
240 | 240 | | |
| |||
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
252 | | - | |
| 252 | + | |
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
| |||
0 commit comments