Repository navigation
Commit 509725f
fix: reject null-aware
A null-aware `LeftAnti` join emits build rows in the final stage, and the
"a probe partition saw a NULL `NOT IN` key" fact is recorded on the build
data (`JoinLeftData::probe_completion`). `PartitionMode::Partitioned`
gives every output partition its own build side, so a NULL probe key -
routed by hash to a single partition - only silenced that partition's
build rows; the others still emitted their unmatched ones. With build
`c1` = 1..8 and probe `c2` = {1, NULL} over four hash partitions the join
returned `[3, 6, 8]` where `CollectLeft` correctly returned nothing.
`NullAwareMode::try_new` already rejected `Partitioned` for the
correlated `LeftAnti` and for `LeftMark`, which share the per-build-row
null bitmap for the same reason. Widen that arm to cover the uncorrelated
`LeftAnti` too, so the whole build-driven family is rejected
consistently.
The DataFusion planner never produced this plan - `physical_planner` does
not repartition null-aware joins and `JoinSelection` pins them to
`CollectLeft` - so the shape was only reachable from hand-built plans and
from protobuf, which decodes the partition mode as written and goes
through the same builder.
`test_null_aware_left_anti_swaps_to_right_anti` loses its `Partitioned`
case: that input plan can no longer be constructed, and the swap it
checks is still covered by the `CollectLeft` and `Auto` cases.
Closes apache#26496
Co-Authored-By: Claude Opus 5 <[email protected]>LeftAnti hash join in Partitioned mode1 parent e12901d commit 509725f
2 files changed
Lines changed: 83 additions & 7 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-plan/src/joins/hash_join
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
469 | 469 | | |
470 | 470 | | |
471 | 471 | | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
472 | 476 | | |
473 | 477 | | |
474 | 478 | | |
475 | | - | |
476 | 479 | | |
477 | 480 | | |
478 | 481 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
300 | 300 | | |
301 | 301 | | |
302 | 302 | | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
309 | 311 | | |
310 | 312 | | |
311 | 313 | | |
| |||
2743 | 2745 | | |
2744 | 2746 | | |
2745 | 2747 | | |
| 2748 | + | |
| 2749 | + | |
| 2750 | + | |
2746 | 2751 | | |
2747 | 2752 | | |
2748 | 2753 | | |
| 2754 | + | |
| 2755 | + | |
| 2756 | + | |
| 2757 | + | |
| 2758 | + | |
| 2759 | + | |
| 2760 | + | |
| 2761 | + | |
| 2762 | + | |
| 2763 | + | |
| 2764 | + | |
| 2765 | + | |
| 2766 | + | |
| 2767 | + | |
| 2768 | + | |
| 2769 | + | |
| 2770 | + | |
| 2771 | + | |
| 2772 | + | |
| 2773 | + | |
| 2774 | + | |
| 2775 | + | |
| 2776 | + | |
| 2777 | + | |
| 2778 | + | |
2749 | 2779 | | |
2750 | 2780 | | |
2751 | 2781 | | |
| |||
10326 | 10356 | | |
10327 | 10357 | | |
10328 | 10358 | | |
| 10359 | + | |
| 10360 | + | |
| 10361 | + | |
| 10362 | + | |
| 10363 | + | |
| 10364 | + | |
| 10365 | + | |
| 10366 | + | |
| 10367 | + | |
| 10368 | + | |
| 10369 | + | |
| 10370 | + | |
| 10371 | + | |
| 10372 | + | |
| 10373 | + | |
| 10374 | + | |
| 10375 | + | |
| 10376 | + | |
| 10377 | + | |
| 10378 | + | |
| 10379 | + | |
| 10380 | + | |
| 10381 | + | |
| 10382 | + | |
| 10383 | + | |
| 10384 | + | |
| 10385 | + | |
| 10386 | + | |
| 10387 | + | |
| 10388 | + | |
| 10389 | + | |
| 10390 | + | |
| 10391 | + | |
| 10392 | + | |
| 10393 | + | |
| 10394 | + | |
| 10395 | + | |
| 10396 | + | |
| 10397 | + | |
| 10398 | + | |
| 10399 | + | |
| 10400 | + | |
| 10401 | + | |
10329 | 10402 | | |
10330 | 10403 | | |
10331 | 10404 | | |
| |||
0 commit comments