Repository navigation
Commit a57ed5e
fix: prevent incorrect results when pushing filters through anti joins (apache#24045)
## Which issue does this PR close?
- Closes apache#24002.
## Rationale for this change
Pushing filters to an anti join's non-output side can produce incorrect
results.
## What changes are included in this PR?
Restrict anti-join filter pushdown to the output side while preserving
two-sided join-key pushdown for semi joins.
## Are these changes tested?
Yes, with updated unit and SQL logic tests.
## Are there any user-facing changes?
no API changes but some downstream expected plans may change
(cherry picked from commit a3f0f93)
Conflict resolved when applying to spiceai-54 (DataFusion 54.1):
- datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt: the
expected physical plan for the NOT IN + TopK query keeps the 54.1 display
(no null_aware marker on HashJoinExec, no dynamic_rg_pruning field). With
this change the TopK filter no longer reaches the non-output side, so the
right-side scan drops from two dynamic filters to one; the one that remains
is the hash join's own probe-side filter, which upstream no longer pushes
for a null-aware anti join with a nullable build key (5975a2d, apache#23173,
not on 54.1). Query results are unchanged from upstream.1 parent 260b243 commit a57ed5e
3 files changed
Lines changed: 77 additions & 34 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1587 | 1587 | | |
1588 | 1588 | | |
1589 | 1589 | | |
1590 | | - | |
1591 | | - | |
1592 | | - | |
1593 | | - | |
| 1590 | + | |
| 1591 | + | |
1594 | 1592 | | |
1595 | 1593 | | |
1596 | 1594 | | |
| |||
1620 | 1618 | | |
1621 | 1619 | | |
1622 | 1620 | | |
1623 | | - | |
1624 | | - | |
| 1621 | + | |
| 1622 | + | |
1625 | 1623 | | |
1626 | 1624 | | |
1627 | 1625 | | |
| |||
1660 | 1658 | | |
1661 | 1659 | | |
1662 | 1660 | | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
1663 | 1679 | | |
1664 | 1680 | | |
1665 | 1681 | | |
| |||
1898 | 1914 | | |
1899 | 1915 | | |
1900 | 1916 | | |
1901 | | - | |
| 1917 | + | |
1902 | 1918 | | |
1903 | 1919 | | |
1904 | 1920 | | |
1905 | 1921 | | |
1906 | 1922 | | |
1907 | | - | |
| 1923 | + | |
1908 | 1924 | | |
1909 | 1925 | | |
1910 | 1926 | | |
| |||
2233 | 2249 | | |
2234 | 2250 | | |
2235 | 2251 | | |
2236 | | - | |
| 2252 | + | |
2237 | 2253 | | |
2238 | 2254 | | |
2239 | 2255 | | |
| |||
2270 | 2286 | | |
2271 | 2287 | | |
2272 | 2288 | | |
2273 | | - | |
| 2289 | + | |
2274 | 2290 | | |
2275 | 2291 | | |
2276 | 2292 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1639 | 1639 | | |
1640 | 1640 | | |
1641 | 1641 | | |
1642 | | - | |
1643 | | - | |
1644 | | - | |
1645 | | - | |
1646 | | - | |
1647 | | - | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
1648 | 1645 | | |
1649 | | - | |
| 1646 | + | |
1650 | 1647 | | |
1651 | 1648 | | |
1652 | 1649 | | |
| |||
1660 | 1657 | | |
1661 | 1658 | | |
1662 | 1659 | | |
1663 | | - | |
| 1660 | + | |
1664 | 1661 | | |
1665 | 1662 | | |
1666 | 1663 | | |
| |||
1766 | 1763 | | |
1767 | 1764 | | |
1768 | 1765 | | |
1769 | | - | |
1770 | | - | |
| 1766 | + | |
| 1767 | + | |
1771 | 1768 | | |
1772 | 1769 | | |
1773 | 1770 | | |
1774 | 1771 | | |
1775 | 1772 | | |
1776 | 1773 | | |
1777 | | - | |
1778 | | - | |
1779 | | - | |
1780 | | - | |
1781 | | - | |
1782 | | - | |
1783 | | - | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
1784 | 1778 | | |
1785 | 1779 | | |
1786 | 1780 | | |
| |||
6406 | 6400 | | |
6407 | 6401 | | |
6408 | 6402 | | |
6409 | | - | |
| 6403 | + | |
6410 | 6404 | | |
6411 | 6405 | | |
6412 | | - | |
| 6406 | + | |
6413 | 6407 | | |
6414 | 6408 | | |
6415 | 6409 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
457 | 457 | | |
458 | 458 | | |
459 | 459 | | |
460 | | - | |
461 | | - | |
462 | | - | |
463 | | - | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
464 | 463 | | |
465 | 464 | | |
466 | 465 | | |
| |||
479 | 478 | | |
480 | 479 | | |
481 | 480 | | |
482 | | - | |
| 481 | + | |
483 | 482 | | |
484 | 483 | | |
485 | 484 | | |
| |||
491 | 490 | | |
492 | 491 | | |
493 | 492 | | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
494 | 527 | | |
495 | 528 | | |
496 | 529 | | |
| |||
0 commit comments