Repository navigation
Commit 6e673b0
fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key (apache#23173)
## Which issue does this close?
- Closes apache#23126.
## Rationale for this change
`x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build
= outer `x`, probe = subquery). Join dynamic filter pushdown pushes a
bounds + membership filter, built from the build keys, onto the probe
scan. That filter can prune every probe row. A null-aware `LeftAnti`
reads an empty probe as a genuinely-empty subquery, so it emits
build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is
UNKNOWN, not TRUE.
The result is scan-dependent, so it's a silent correctness bug. A
`VALUES` scan ignores the pushed filter and stays correct; a parquet
scan applies it and is wrong.
apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side
NULL.
## What changes are included in this PR?
Skip join dynamic filter pushdown for a null-aware anti join when the
build key can be NULL. The build-side NULL emission depends on whether
the probe is truly empty, which the pushed filter can change by emptying
it. A NOT NULL build key has no such NULL, so it keeps the pushdown.
The check is static: a schema-nullable build key disables the pushdown
even when the data contains no NULLs. A runtime alternative (keep the
pushdown and neutralize the filter only when the build actually holds a
NULL key) would restore the optimization for those cases. I'd leave that
as a follow-up.
## Are these changes tested?
Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side
NULL, a non-matching parquet probe) and asserts the single correct row.
Without the change it returns the extra NULL. In addition, unit tests
pin both directions of the guard: a nullable build key rejects the
pushdown and a NOT NULL build key keeps it.
## Are there any user-facing changes?
`NOT IN` over a parquet (or otherwise prunable) scan with a nullable
outer key now returns correct results. Such joins lose the dynamic
filter pushdown.
(cherry picked from commit 5975a2d)
Conflicts resolved when applying to spiceai-54 (DataFusion 54.1):
- datafusion/physical-plan/src/joins/hash_join/exec.rs: added only this
commit's null-aware gate to allow_join_dynamic_filter_pushdown and its two
unit tests. The surrounding upstream context (the NullEqualsNull gate and
its test from baa497d, apache#22965, and the swap-inputs and range-partitioning
tests) is not on 54.1 and is not added.
- datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt: kept
the 54.1 plan display (no null_aware, no dynamic_rg_pruning) and removed the
hash join's own dynamic filter from the probe scan of both NOT IN plans.
For the NOT IN + TopK plan that leaves the right-side scan with no
predicate: the TopK filter was already removed there by a3f0f93 (apache#24045),
which is upstream's state once both changes are in.
- datafusion/sqllogictest/test_files/explain_tree.slt: dropped. The change
edits a NOT IN record that 127731b (apache#22913) added and 54.1 does not have.
- datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: added only
this commit's regression block, not the null-equal join block from
baa497d (apache#22965) that precedes it upstream.1 parent c2c0bac commit 6e673b0
3 files changed
Lines changed: 153 additions & 2 deletions
File tree
- datafusion
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
905 | 905 | | |
906 | 906 | | |
907 | 907 | | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
908 | 920 | | |
909 | 921 | | |
910 | 922 | | |
| |||
6444 | 6456 | | |
6445 | 6457 | | |
6446 | 6458 | | |
| 6459 | + | |
| 6460 | + | |
| 6461 | + | |
| 6462 | + | |
| 6463 | + | |
| 6464 | + | |
| 6465 | + | |
| 6466 | + | |
| 6467 | + | |
| 6468 | + | |
| 6469 | + | |
| 6470 | + | |
| 6471 | + | |
| 6472 | + | |
| 6473 | + | |
| 6474 | + | |
| 6475 | + | |
| 6476 | + | |
| 6477 | + | |
| 6478 | + | |
| 6479 | + | |
| 6480 | + | |
| 6481 | + | |
| 6482 | + | |
| 6483 | + | |
| 6484 | + | |
| 6485 | + | |
| 6486 | + | |
| 6487 | + | |
| 6488 | + | |
| 6489 | + | |
| 6490 | + | |
| 6491 | + | |
| 6492 | + | |
| 6493 | + | |
| 6494 | + | |
| 6495 | + | |
| 6496 | + | |
| 6497 | + | |
| 6498 | + | |
| 6499 | + | |
| 6500 | + | |
| 6501 | + | |
| 6502 | + | |
| 6503 | + | |
| 6504 | + | |
| 6505 | + | |
| 6506 | + | |
| 6507 | + | |
| 6508 | + | |
| 6509 | + | |
| 6510 | + | |
| 6511 | + | |
| 6512 | + | |
| 6513 | + | |
| 6514 | + | |
| 6515 | + | |
| 6516 | + | |
| 6517 | + | |
| 6518 | + | |
| 6519 | + | |
| 6520 | + | |
| 6521 | + | |
| 6522 | + | |
| 6523 | + | |
| 6524 | + | |
| 6525 | + | |
| 6526 | + | |
| 6527 | + | |
| 6528 | + | |
| 6529 | + | |
| 6530 | + | |
| 6531 | + | |
6447 | 6532 | | |
6448 | 6533 | | |
6449 | 6534 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
383 | 383 | | |
384 | 384 | | |
385 | 385 | | |
386 | | - | |
| 386 | + | |
387 | 387 | | |
388 | 388 | | |
389 | 389 | | |
| |||
478 | 478 | | |
479 | 479 | | |
480 | 480 | | |
481 | | - | |
| 481 | + | |
482 | 482 | | |
483 | 483 | | |
484 | 484 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1023 | 1023 | | |
1024 | 1024 | | |
1025 | 1025 | | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
1026 | 1092 | | |
1027 | 1093 | | |
1028 | 1094 | | |
| |||
0 commit comments