Skip to content

Commit 9e0ea6c

Browse files
mdashtikosiew
authored andcommitted
fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key (apache#23173)
## Which issue does this close? - Closes apache#23126. ## Rationale for this change `x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build = outer `x`, probe = subquery). Join dynamic filter pushdown pushes a bounds + membership filter, built from the build keys, onto the probe scan. That filter can prune every probe row. A null-aware `LeftAnti` reads an empty probe as a genuinely-empty subquery, so it emits build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is UNKNOWN, not TRUE. The result is scan-dependent, so it's a silent correctness bug. A `VALUES` scan ignores the pushed filter and stays correct; a parquet scan applies it and is wrong. apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side NULL. ## What changes are included in this PR? Skip join dynamic filter pushdown for a null-aware anti join when the build key can be NULL. The build-side NULL emission depends on whether the probe is truly empty, which the pushed filter can change by emptying it. A NOT NULL build key has no such NULL, so it keeps the pushdown. The check is static: a schema-nullable build key disables the pushdown even when the data contains no NULLs. A runtime alternative (keep the pushdown and neutralize the filter only when the build actually holds a NULL key) would restore the optimization for those cases. I'd leave that as a follow-up. ## Are these changes tested? Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side NULL, a non-matching parquet probe) and asserts the single correct row. Without the change it returns the extra NULL. In addition, unit tests pin both directions of the guard: a nullable build key rejects the pushdown and a NOT NULL build key keeps it. ## Are there any user-facing changes? `NOT IN` over a parquet (or otherwise prunable) scan with a nullable outer key now returns correct results. Such joins lose the dynamic filter pushdown.
1 parent a96bc77 commit 9e0ea6c

4 files changed

Lines changed: 154 additions & 6 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -871,6 +871,18 @@ impl HashJoinExec {
871871
return false;
872872
}
873873

874+
// A null-aware anti join emits a build-side NULL only when the probe
875+
// is truly empty. The pushed filter can empty the probe by pruning
876+
// every row, which would surface that NULL wrongly. A NOT NULL build
877+
// key cannot produce such a NULL, so the filter stays there.
878+
if self.null_aware
879+
&& self.on.iter().any(|(build_key, _)| {
880+
build_key.nullable(&self.left.schema()).unwrap_or(true)
881+
})
882+
{
883+
return false;
884+
}
885+
874886
// `preserve_file_partitions` can report Hash partitioning for Hive-style
875887
// file groups, but those partitions are not actually hash-distributed.
876888
// Partitioned dynamic filters rely on hash routing, so disable them in
@@ -6949,6 +6961,79 @@ mod tests {
69496961
Ok(())
69506962
}
69516963

6964+
#[test]
6965+
fn test_dynamic_filter_pushdown_rejects_null_aware_nullable_build_key() -> Result<()>
6966+
{
6967+
let left = build_table_two_cols(
6968+
("a1", &vec![Some(1), None]),
6969+
("b1", &vec![Some(1), Some(2)]),
6970+
);
6971+
let right = build_table_two_cols(
6972+
("a2", &vec![Some(2), Some(3)]),
6973+
("b2", &vec![Some(1), Some(2)]),
6974+
);
6975+
let on = vec![(
6976+
Arc::new(Column::new_with_schema("a1", &left.schema())?) as _,
6977+
Arc::new(Column::new_with_schema("a2", &right.schema())?) as _,
6978+
)];
6979+
6980+
let mut session_config = SessionConfig::default();
6981+
session_config
6982+
.options_mut()
6983+
.optimizer
6984+
.enable_join_dynamic_filter_pushdown = true;
6985+
6986+
let join = HashJoinExec::try_new(
6987+
left,
6988+
right,
6989+
on,
6990+
None,
6991+
&JoinType::LeftAnti,
6992+
None,
6993+
PartitionMode::CollectLeft,
6994+
NullEquality::NullEqualsNothing,
6995+
true,
6996+
)?;
6997+
6998+
assert!(!join.allow_join_dynamic_filter_pushdown(session_config.options()));
6999+
7000+
Ok(())
7001+
}
7002+
7003+
#[test]
7004+
fn test_dynamic_filter_pushdown_allows_null_aware_non_null_build_key() -> Result<()> {
7005+
// A NOT NULL build key cannot surface a build-side NULL, so the
7006+
// pushdown must stay enabled.
7007+
let left = build_table(("a1", &vec![1]), ("b1", &vec![1]), ("c1", &vec![1]));
7008+
let right = build_table(("a2", &vec![2]), ("b2", &vec![2]), ("c2", &vec![2]));
7009+
let on = vec![(
7010+
Arc::new(Column::new_with_schema("a1", &left.schema())?) as _,
7011+
Arc::new(Column::new_with_schema("a2", &right.schema())?) as _,
7012+
)];
7013+
7014+
let mut session_config = SessionConfig::default();
7015+
session_config
7016+
.options_mut()
7017+
.optimizer
7018+
.enable_join_dynamic_filter_pushdown = true;
7019+
7020+
let join = HashJoinExec::try_new(
7021+
left,
7022+
right,
7023+
on,
7024+
None,
7025+
&JoinType::LeftAnti,
7026+
None,
7027+
PartitionMode::CollectLeft,
7028+
NullEquality::NullEqualsNothing,
7029+
true,
7030+
)?;
7031+
7032+
assert!(join.allow_join_dynamic_filter_pushdown(session_config.options()));
7033+
7034+
Ok(())
7035+
}
7036+
69527037
#[test]
69537038
fn test_partitioned_dynamic_filter_pushdown_rejects_range_partitioning() -> Result<()>
69547039
{

‎datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -383,7 +383,7 @@ logical_plan
383383
physical_plan
384384
01)HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(id@0, id@0)], null_aware
385385
02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_left.parquet]]}, projection=[id, data], file_type=parquet
386-
03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
386+
03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet
387387

388388
# LEFT MARK JOIN: the OR prevents decorrelation to LeftSemi, so the optimizer
389389
# uses LeftMark. Self-generated dynamic filter pushes to the probe side.
@@ -479,7 +479,7 @@ physical_plan
479479
01)SortExec: TopK(fetch=2), expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false]
480480
02)--HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(id@0, id@0)], null_aware
481481
03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_left.parquet]]}, projection=[id, data], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
482-
04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ] AND DynamicFilter [ empty ], dynamic_rg_pruning=eligible
482+
04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ], dynamic_rg_pruning=eligible
483483

484484
# Correctness check
485485
query IT

‎datafusion/sqllogictest/test_files/explain_tree.slt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1120,10 +1120,7 @@ physical_plan
11201120
13)│ -------------------- ││ -------------------- │
11211121
14)│ files: 1 ││ files: 1 │
11221122
15)│ format: csv ││ format: parquet │
1123-
16)│ ││ │
1124-
17)│ ││ predicate: │
1125-
18)│ ││ DynamicFilter [ empty ] │
1126-
19)└───────────────────────────┘└───────────────────────────┘
1123+
16)└───────────────────────────┘└───────────────────────────┘
11271124

11281125
# Query with nested loop join.
11291126
query TT

‎datafusion/sqllogictest/test_files/push_down_filter_parquet.slt‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1066,6 +1066,72 @@ statement ok
10661066
drop table nej_probe;
10671067

10681068

1069+
########
1070+
# Regression test for build-NULL + emptied-probe interaction in null-aware LeftAnti joins.
1071+
#
1072+
# `x NOT IN (subquery)` plans as a null-aware LeftAnti hash join where `x` is
1073+
# the build (left) side. The dynamic-filter pushdown derives a bounds/membership
1074+
# filter from the build keys and pushes it onto the probe scan. When the build
1075+
# contains a NULL key and the filter prunes every probe row, the probe looks
1076+
# empty to the join. A null-aware LeftAnti treats an empty probe as a genuinely-
1077+
# absent subquery, so it emits the build-side NULL as a matching row. That is
1078+
# wrong: `NULL NOT IN (non-empty set)` must be UNKNOWN, not TRUE.
1079+
#
1080+
# The fix: suppress dynamic-filter pushdown whenever the build key is nullable
1081+
# and the join is null-aware, so the probe is never artificially emptied.
1082+
########
1083+
1084+
statement ok
1085+
set datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
1086+
1087+
statement ok
1088+
set datafusion.execution.parquet.pushdown_filters = true;
1089+
1090+
# Build side: `ao` has a nullable `id` column; the NULL row is the one that
1091+
# must NOT appear in the output.
1092+
query I
1093+
COPY (SELECT * FROM (VALUES (5), (NULL)) v(id))
1094+
TO 'test_files/scratch/push_down_filter_parquet/ao_p.parquet'
1095+
STORED AS PARQUET;
1096+
----
1097+
2
1098+
1099+
# Probe / subquery side: `i_disj` has two non-NULL values that don't match 5,
1100+
# and no NULLs. The subquery is non-empty, so `NULL NOT IN (...)` is UNKNOWN.
1101+
query I
1102+
COPY (SELECT * FROM (VALUES (2), (3)) v(eid))
1103+
TO 'test_files/scratch/push_down_filter_parquet/i_disj_p.parquet'
1104+
STORED AS PARQUET;
1105+
----
1106+
2
1107+
1108+
statement ok
1109+
CREATE EXTERNAL TABLE ao_p (id INT) STORED AS PARQUET
1110+
LOCATION 'test_files/scratch/push_down_filter_parquet/ao_p.parquet';
1111+
1112+
statement ok
1113+
CREATE EXTERNAL TABLE i_disj_p (eid INT) STORED AS PARQUET
1114+
LOCATION 'test_files/scratch/push_down_filter_parquet/i_disj_p.parquet';
1115+
1116+
# Must return only `5`. `NULL NOT IN (2, 3)` is UNKNOWN, so that row is dropped.
1117+
query I
1118+
SELECT id FROM ao_p WHERE id NOT IN (SELECT eid FROM i_disj_p) ORDER BY id;
1119+
----
1120+
5
1121+
1122+
statement ok
1123+
drop table ao_p;
1124+
1125+
statement ok
1126+
drop table i_disj_p;
1127+
1128+
statement ok
1129+
RESET datafusion.optimizer.enable_join_dynamic_filter_pushdown;
1130+
1131+
statement ok
1132+
RESET datafusion.execution.parquet.pushdown_filters;
1133+
1134+
10691135
# Config reset
10701136
statement ok
10711137
RESET datafusion.explain.physical_plan_only;

0 commit comments

Comments
 (0)