Repository navigation
Commit 5975a2d
authored
fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key (apache#23173)
## Which issue does this close?
- Closes apache#23126.
## Rationale for this change
`x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build
= outer `x`, probe = subquery). Join dynamic filter pushdown pushes a
bounds + membership filter, built from the build keys, onto the probe
scan. That filter can prune every probe row. A null-aware `LeftAnti`
reads an empty probe as a genuinely-empty subquery, so it emits
build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is
UNKNOWN, not TRUE.
The result is scan-dependent, so it's a silent correctness bug. A
`VALUES` scan ignores the pushed filter and stays correct; a parquet
scan applies it and is wrong.
apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side
NULL.
## What changes are included in this PR?
Skip join dynamic filter pushdown for a null-aware anti join when the
build key can be NULL. The build-side NULL emission depends on whether
the probe is truly empty, which the pushed filter can change by emptying
it. A NOT NULL build key has no such NULL, so it keeps the pushdown.
The check is static: a schema-nullable build key disables the pushdown
even when the data contains no NULLs. A runtime alternative (keep the
pushdown and neutralize the filter only when the build actually holds a
NULL key) would restore the optimization for those cases. I'd leave that
as a follow-up.
## Are these changes tested?
Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side
NULL, a non-matching parquet probe) and asserts the single correct row.
Without the change it returns the extra NULL. In addition, unit tests
pin both directions of the guard: a nullable build key rejects the
pushdown and a NOT NULL build key keeps it.
## Are there any user-facing changes?
`NOT IN` over a parquet (or otherwise prunable) scan with a nullable
outer key now returns correct results. Such joins lose the dynamic
filter pushdown.1 parent bb670fb commit 5975a2d
4 files changed
Lines changed: 154 additions & 6 deletions
File tree
- datafusion
- physical-plan/src/joins/hash_join
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
871 | 871 | | |
872 | 872 | | |
873 | 873 | | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
874 | 886 | | |
875 | 887 | | |
876 | 888 | | |
| |||
6949 | 6961 | | |
6950 | 6962 | | |
6951 | 6963 | | |
| 6964 | + | |
| 6965 | + | |
| 6966 | + | |
| 6967 | + | |
| 6968 | + | |
| 6969 | + | |
| 6970 | + | |
| 6971 | + | |
| 6972 | + | |
| 6973 | + | |
| 6974 | + | |
| 6975 | + | |
| 6976 | + | |
| 6977 | + | |
| 6978 | + | |
| 6979 | + | |
| 6980 | + | |
| 6981 | + | |
| 6982 | + | |
| 6983 | + | |
| 6984 | + | |
| 6985 | + | |
| 6986 | + | |
| 6987 | + | |
| 6988 | + | |
| 6989 | + | |
| 6990 | + | |
| 6991 | + | |
| 6992 | + | |
| 6993 | + | |
| 6994 | + | |
| 6995 | + | |
| 6996 | + | |
| 6997 | + | |
| 6998 | + | |
| 6999 | + | |
| 7000 | + | |
| 7001 | + | |
| 7002 | + | |
| 7003 | + | |
| 7004 | + | |
| 7005 | + | |
| 7006 | + | |
| 7007 | + | |
| 7008 | + | |
| 7009 | + | |
| 7010 | + | |
| 7011 | + | |
| 7012 | + | |
| 7013 | + | |
| 7014 | + | |
| 7015 | + | |
| 7016 | + | |
| 7017 | + | |
| 7018 | + | |
| 7019 | + | |
| 7020 | + | |
| 7021 | + | |
| 7022 | + | |
| 7023 | + | |
| 7024 | + | |
| 7025 | + | |
| 7026 | + | |
| 7027 | + | |
| 7028 | + | |
| 7029 | + | |
| 7030 | + | |
| 7031 | + | |
| 7032 | + | |
| 7033 | + | |
| 7034 | + | |
| 7035 | + | |
| 7036 | + | |
6952 | 7037 | | |
6953 | 7038 | | |
6954 | 7039 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
383 | 383 | | |
384 | 384 | | |
385 | 385 | | |
386 | | - | |
| 386 | + | |
387 | 387 | | |
388 | 388 | | |
389 | 389 | | |
| |||
479 | 479 | | |
480 | 480 | | |
481 | 481 | | |
482 | | - | |
| 482 | + | |
483 | 483 | | |
484 | 484 | | |
485 | 485 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1120 | 1120 | | |
1121 | 1121 | | |
1122 | 1122 | | |
1123 | | - | |
1124 | | - | |
1125 | | - | |
1126 | | - | |
| 1123 | + | |
1127 | 1124 | | |
1128 | 1125 | | |
1129 | 1126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1066 | 1066 | | |
1067 | 1067 | | |
1068 | 1068 | | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
1069 | 1135 | | |
1070 | 1136 | | |
1071 | 1137 | | |
| |||
0 commit comments