Repository navigation
Commit c2c0bac
fix: keep null-aware anti-join NULLs in the pushed dynamic filter (apache#23104)
## Which issue does this close?
Closes apache#23103.
## Rationale for this change
A hash join pushes a build-side dynamic filter (`key IN build_keys`)
down to the probe scan. For a null-aware anti join (`NOT IN`), that
filter drops the probe's NULL rows. But `NOT IN` three-valued logic
needs a probe-side NULL to collapse the whole result to zero rows. With
the NULL filtered away at the scan, before the join's null-aware check
runs, the join returns rows that shouldn't be there.
## What changes are included in this PR?
`SharedBuildAccumulator::build_filter` now ORs `probe_key IS NULL` into
the pushed predicate when the join is `null_aware`. Non-NULL probe rows
still get filtered, so the optimization stays. `HashJoinExec`'s
`null_aware` validation already guarantees a single probe key.
## Are these changes tested?
Yes. Added a parquet-backed case to `null_aware_anti_join.slt`. The
existing cases use in-memory `VALUES`, whose scans never apply the
pushed filter, so they passed despite the bug. The new one sets
`parquet.pushdown_filters = true` so the filter runs row-level. Without
the fix it returns `1, 3`; with it, zero rows.
## Are there any user-facing changes?
A `NOT IN` over a NULL-bearing inner now returns zero rows instead of
leaking rows, when join dynamic filter pushdown and row-level scan
filtering are both on.
---------
Co-authored-by: Andrew Lamb <[email protected]>
Co-authored-by: Adrian Garcia Badaracco <[email protected]>
Co-authored-by: rjhallsted <[email protected]>
(cherry picked from commit 8d680db)
Conflicts resolved when applying to spiceai-54 (DataFusion 54.1):
- datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs: both hunks
are in test scaffolding. Upstream adds null_aware: false to the
make_partitioned_accumulator_for_test and completed_partitions_for_test
helpers, which 54.1 does not have, so they are not added. In the tests
module helper, 54.1 builds probe_schema locally where upstream calls
test_probe_schema(); kept the 54.1 form and added null_aware: false.
The accumulator changes themselves applied as upstream wrote them; the
fork's CollectLeftAccumulator seam only computes bounds, and the pushed
filter is still published only at the two update sites this wraps.1 parent 7f292c2 commit c2c0bac
3 files changed
Lines changed: 104 additions & 3 deletions
File tree
- datafusion
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1394 | 1394 | | |
1395 | 1395 | | |
1396 | 1396 | | |
| 1397 | + | |
1397 | 1398 | | |
1398 | 1399 | | |
1399 | 1400 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| |||
308 | 308 | | |
309 | 309 | | |
310 | 310 | | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
311 | 314 | | |
312 | 315 | | |
313 | 316 | | |
| |||
411 | 414 | | |
412 | 415 | | |
413 | 416 | | |
| 417 | + | |
414 | 418 | | |
415 | 419 | | |
416 | 420 | | |
| |||
457 | 461 | | |
458 | 462 | | |
459 | 463 | | |
| 464 | + | |
460 | 465 | | |
461 | 466 | | |
462 | 467 | | |
| |||
632 | 637 | | |
633 | 638 | | |
634 | 639 | | |
635 | | - | |
| 640 | + | |
| 641 | + | |
636 | 642 | | |
637 | 643 | | |
638 | 644 | | |
| |||
738 | 744 | | |
739 | 745 | | |
740 | 746 | | |
741 | | - | |
| 747 | + | |
| 748 | + | |
742 | 749 | | |
743 | 750 | | |
744 | 751 | | |
745 | 752 | | |
746 | 753 | | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
747 | 781 | | |
748 | 782 | | |
749 | 783 | | |
| |||
776 | 810 | | |
777 | 811 | | |
778 | 812 | | |
| 813 | + | |
779 | 814 | | |
780 | 815 | | |
781 | 816 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
451 | 451 | | |
452 | 452 | | |
453 | 453 | | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
0 commit comments