Repository navigation
Commit e1f8393
fix: handle dictionary nulls in null-aware left mark joins (apache#25693)
## Which issue does this PR close?
- Closes apache#25691.
## Rationale for this change
Null-aware `LeftMark` joins use physical null checks for dictionary
keys. They miss valid dictionary keys that point to `NULL` dictionary
values and return `FALSE` marks where SQL three-valued logic requires
`NULL`.
## What changes are included in this PR?
Null-aware `LeftMark` joins now use Arrow's logical-null APIs on both
probe and build keys. The build-side logical-null bitmap is computed
once during build collection, retained with memory accounting, and
reused by final output chunks. Probe-side null summaries are computed
once per input batch even when hash lookup resumes across several
chunks. Correlated joins continue using their precomputed per-row null
bitmap.
## What is the testing strategy for this PR?
The diff adds three `HashJoinExec` test functions covering logical
dictionary nulls on the build and probe sides, including nested
dictionary build keys. `#[apply(hash_join_exec_configs)]` expands each
function across five batch sizes, so Cargo reports 15 new test
instances. The probe-side function also checks both probe-partition
completion orders inside each instance.
Ablation testing confirmed that the original build- and probe-side
functions fail at every batch size without the correctness fix. With the
fix, all 109 null-aware tests and the complete 2,409-test
`datafusion-physical-plan` all-features suite pass.
For 524,288-row debug diagnostics, bounding the original flat-dictionary
scan reduced runtime from about 693ms to 141ms. The follow-up
nested-dictionary diagnostic dropped from about 561ms to 177ms, and the
oversized resumed-probe diagnostic dropped from about 1.29s to 910ms.
## Are there any user-facing changes?
Null-aware mark joins over dictionary-encoded keys now return the
correct nullable marks. There are no public API changes.1 parent 9bc5cd9 commit e1f8393
3 files changed
Lines changed: 254 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
358 | 358 | | |
359 | 359 | | |
360 | 360 | | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
361 | 366 | | |
362 | 367 | | |
363 | 368 | | |
| |||
438 | 443 | | |
439 | 444 | | |
440 | 445 | | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
441 | 450 | | |
442 | 451 | | |
443 | 452 | | |
| |||
3586 | 3595 | | |
3587 | 3596 | | |
3588 | 3597 | | |
| 3598 | + | |
| 3599 | + | |
| 3600 | + | |
| 3601 | + | |
| 3602 | + | |
| 3603 | + | |
| 3604 | + | |
| 3605 | + | |
| 3606 | + | |
| 3607 | + | |
| 3608 | + | |
| 3609 | + | |
| 3610 | + | |
| 3611 | + | |
| 3612 | + | |
| 3613 | + | |
| 3614 | + | |
| 3615 | + | |
| 3616 | + | |
| 3617 | + | |
| 3618 | + | |
| 3619 | + | |
| 3620 | + | |
| 3621 | + | |
| 3622 | + | |
| 3623 | + | |
| 3624 | + | |
3589 | 3625 | | |
3590 | 3626 | | |
3591 | 3627 | | |
| |||
3635 | 3671 | | |
3636 | 3672 | | |
3637 | 3673 | | |
| 3674 | + | |
3638 | 3675 | | |
3639 | 3676 | | |
3640 | 3677 | | |
| |||
9695 | 9732 | | |
9696 | 9733 | | |
9697 | 9734 | | |
| 9735 | + | |
| 9736 | + | |
| 9737 | + | |
| 9738 | + | |
| 9739 | + | |
| 9740 | + | |
| 9741 | + | |
| 9742 | + | |
| 9743 | + | |
| 9744 | + | |
| 9745 | + | |
| 9746 | + | |
| 9747 | + | |
| 9748 | + | |
| 9749 | + | |
| 9750 | + | |
| 9751 | + | |
| 9752 | + | |
| 9753 | + | |
| 9754 | + | |
| 9755 | + | |
| 9756 | + | |
| 9757 | + | |
| 9758 | + | |
| 9759 | + | |
| 9760 | + | |
| 9761 | + | |
| 9762 | + | |
| 9763 | + | |
| 9764 | + | |
| 9765 | + | |
| 9766 | + | |
| 9767 | + | |
| 9768 | + | |
| 9769 | + | |
| 9770 | + | |
| 9771 | + | |
| 9772 | + | |
| 9773 | + | |
| 9774 | + | |
| 9775 | + | |
| 9776 | + | |
| 9777 | + | |
| 9778 | + | |
| 9779 | + | |
| 9780 | + | |
| 9781 | + | |
| 9782 | + | |
| 9783 | + | |
| 9784 | + | |
| 9785 | + | |
| 9786 | + | |
| 9787 | + | |
9698 | 9788 | | |
9699 | 9789 | | |
9700 | 9790 | | |
| |||
10615 | 10705 | | |
10616 | 10706 | | |
10617 | 10707 | | |
| 10708 | + | |
| 10709 | + | |
| 10710 | + | |
| 10711 | + | |
| 10712 | + | |
| 10713 | + | |
| 10714 | + | |
| 10715 | + | |
| 10716 | + | |
| 10717 | + | |
| 10718 | + | |
| 10719 | + | |
| 10720 | + | |
| 10721 | + | |
| 10722 | + | |
| 10723 | + | |
| 10724 | + | |
| 10725 | + | |
| 10726 | + | |
| 10727 | + | |
| 10728 | + | |
| 10729 | + | |
| 10730 | + | |
| 10731 | + | |
| 10732 | + | |
| 10733 | + | |
| 10734 | + | |
| 10735 | + | |
| 10736 | + | |
| 10737 | + | |
| 10738 | + | |
| 10739 | + | |
| 10740 | + | |
| 10741 | + | |
| 10742 | + | |
| 10743 | + | |
| 10744 | + | |
| 10745 | + | |
| 10746 | + | |
| 10747 | + | |
| 10748 | + | |
| 10749 | + | |
| 10750 | + | |
| 10751 | + | |
| 10752 | + | |
| 10753 | + | |
| 10754 | + | |
| 10755 | + | |
| 10756 | + | |
| 10757 | + | |
| 10758 | + | |
| 10759 | + | |
| 10760 | + | |
| 10761 | + | |
| 10762 | + | |
| 10763 | + | |
| 10764 | + | |
| 10765 | + | |
| 10766 | + | |
| 10767 | + | |
| 10768 | + | |
| 10769 | + | |
| 10770 | + | |
| 10771 | + | |
| 10772 | + | |
| 10773 | + | |
| 10774 | + | |
| 10775 | + | |
| 10776 | + | |
| 10777 | + | |
| 10778 | + | |
| 10779 | + | |
| 10780 | + | |
| 10781 | + | |
| 10782 | + | |
| 10783 | + | |
| 10784 | + | |
| 10785 | + | |
| 10786 | + | |
| 10787 | + | |
| 10788 | + | |
| 10789 | + | |
| 10790 | + | |
| 10791 | + | |
| 10792 | + | |
| 10793 | + | |
| 10794 | + | |
| 10795 | + | |
| 10796 | + | |
| 10797 | + | |
| 10798 | + | |
| 10799 | + | |
| 10800 | + | |
| 10801 | + | |
| 10802 | + | |
| 10803 | + | |
| 10804 | + | |
| 10805 | + | |
| 10806 | + | |
| 10807 | + | |
| 10808 | + | |
| 10809 | + | |
| 10810 | + | |
| 10811 | + | |
| 10812 | + | |
| 10813 | + | |
| 10814 | + | |
| 10815 | + | |
| 10816 | + | |
| 10817 | + | |
| 10818 | + | |
| 10819 | + | |
| 10820 | + | |
| 10821 | + | |
| 10822 | + | |
| 10823 | + | |
| 10824 | + | |
| 10825 | + | |
| 10826 | + | |
| 10827 | + | |
| 10828 | + | |
| 10829 | + | |
| 10830 | + | |
| 10831 | + | |
| 10832 | + | |
| 10833 | + | |
| 10834 | + | |
| 10835 | + | |
| 10836 | + | |
| 10837 | + | |
| 10838 | + | |
| 10839 | + | |
| 10840 | + | |
| 10841 | + | |
| 10842 | + | |
| 10843 | + | |
| 10844 | + | |
| 10845 | + | |
| 10846 | + | |
10618 | 10847 | | |
10619 | 10848 | | |
10620 | 10849 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1346 | 1346 | | |
1347 | 1347 | | |
1348 | 1348 | | |
1349 | | - | |
1350 | | - | |
1351 | 1349 | | |
1352 | | - | |
1353 | | - | |
1354 | | - | |
1355 | | - | |
1356 | | - | |
1357 | | - | |
1358 | | - | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
1359 | 1358 | | |
1360 | | - | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
1361 | 1362 | | |
1362 | 1363 | | |
1363 | 1364 | | |
| |||
1445 | 1446 | | |
1446 | 1447 | | |
1447 | 1448 | | |
1448 | | - | |
1449 | 1449 | | |
1450 | 1450 | | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
1451 | 1458 | | |
1452 | 1459 | | |
1453 | 1460 | | |
1454 | | - | |
| 1461 | + | |
1455 | 1462 | | |
1456 | 1463 | | |
1457 | 1464 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1389 | 1389 | | |
1390 | 1390 | | |
1391 | 1391 | | |
| 1392 | + | |
| 1393 | + | |
1392 | 1394 | | |
1393 | 1395 | | |
1394 | 1396 | | |
1395 | 1397 | | |
1396 | 1398 | | |
1397 | | - | |
| 1399 | + | |
1398 | 1400 | | |
1399 | 1401 | | |
1400 | 1402 | | |
| |||
1404 | 1406 | | |
1405 | 1407 | | |
1406 | 1408 | | |
1407 | | - | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
1408 | 1412 | | |
1409 | 1413 | | |
1410 | 1414 | | |
| |||
0 commit comments