Skip to content

Commit 6e673b0

Browse files
mdashtilukekim
authored andcommitted
fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key (apache#23173)
## Which issue does this close? - Closes apache#23126. ## Rationale for this change `x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build = outer `x`, probe = subquery). Join dynamic filter pushdown pushes a bounds + membership filter, built from the build keys, onto the probe scan. That filter can prune every probe row. A null-aware `LeftAnti` reads an empty probe as a genuinely-empty subquery, so it emits build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is UNKNOWN, not TRUE. The result is scan-dependent, so it's a silent correctness bug. A `VALUES` scan ignores the pushed filter and stays correct; a parquet scan applies it and is wrong. apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side NULL. ## What changes are included in this PR? Skip join dynamic filter pushdown for a null-aware anti join when the build key can be NULL. The build-side NULL emission depends on whether the probe is truly empty, which the pushed filter can change by emptying it. A NOT NULL build key has no such NULL, so it keeps the pushdown. The check is static: a schema-nullable build key disables the pushdown even when the data contains no NULLs. A runtime alternative (keep the pushdown and neutralize the filter only when the build actually holds a NULL key) would restore the optimization for those cases. I'd leave that as a follow-up. ## Are these changes tested? Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side NULL, a non-matching parquet probe) and asserts the single correct row. Without the change it returns the extra NULL. In addition, unit tests pin both directions of the guard: a nullable build key rejects the pushdown and a NOT NULL build key keeps it. ## Are there any user-facing changes? `NOT IN` over a parquet (or otherwise prunable) scan with a nullable outer key now returns correct results. Such joins lose the dynamic filter pushdown. (cherry picked from commit 5975a2d) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/exec.rs: added only this commit's null-aware gate to allow_join_dynamic_filter_pushdown and its two unit tests. The surrounding upstream context (the NullEqualsNull gate and its test from baa497d, apache#22965, and the swap-inputs and range-partitioning tests) is not on 54.1 and is not added. - datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt: kept the 54.1 plan display (no null_aware, no dynamic_rg_pruning) and removed the hash join's own dynamic filter from the probe scan of both NOT IN plans. For the NOT IN + TopK plan that leaves the right-side scan with no predicate: the TopK filter was already removed there by a3f0f93 (apache#24045), which is upstream's state once both changes are in. - datafusion/sqllogictest/test_files/explain_tree.slt: dropped. The change edits a NOT IN record that 127731b (apache#22913) added and 54.1 does not have. - datafusion/sqllogictest/test_files/push_down_filter_parquet.slt: added only this commit's regression block, not the null-equal join block from baa497d (apache#22965) that precedes it upstream.
1 parent c2c0bac commit 6e673b0

3 files changed

Lines changed: 153 additions & 2 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -905,6 +905,18 @@ impl<A: CollectLeftAccumulator + 'static> HashJoinExec<A> {
905905
return false;
906906
}
907907

908+
// A null-aware anti join emits a build-side NULL only when the probe
909+
// is truly empty. The pushed filter can empty the probe by pruning
910+
// every row, which would surface that NULL wrongly. A NOT NULL build
911+
// key cannot produce such a NULL, so the filter stays there.
912+
if self.null_aware
913+
&& self.on.iter().any(|(build_key, _)| {
914+
build_key.nullable(&self.left.schema()).unwrap_or(true)
915+
})
916+
{
917+
return false;
918+
}
919+
908920
// `preserve_file_partitions` can report Hash partitioning for Hive-style
909921
// file groups, but those partitions are not actually hash-distributed.
910922
// Partitioned dynamic filters rely on hash routing, so disable them in
@@ -6444,6 +6456,79 @@ mod tests {
64446456
Ok(())
64456457
}
64466458

6459+
#[test]
6460+
fn test_dynamic_filter_pushdown_rejects_null_aware_nullable_build_key() -> Result<()>
6461+
{
6462+
let left = build_table_two_cols(
6463+
("a1", &vec![Some(1), None]),
6464+
("b1", &vec![Some(1), Some(2)]),
6465+
);
6466+
let right = build_table_two_cols(
6467+
("a2", &vec![Some(2), Some(3)]),
6468+
("b2", &vec![Some(1), Some(2)]),
6469+
);
6470+
let on = vec![(
6471+
Arc::new(Column::new_with_schema("a1", &left.schema())?) as _,
6472+
Arc::new(Column::new_with_schema("a2", &right.schema())?) as _,
6473+
)];
6474+
6475+
let mut session_config = SessionConfig::default();
6476+
session_config
6477+
.options_mut()
6478+
.optimizer
6479+
.enable_join_dynamic_filter_pushdown = true;
6480+
6481+
let join = HashJoinExec::try_new(
6482+
left,
6483+
right,
6484+
on,
6485+
None,
6486+
&JoinType::LeftAnti,
6487+
None,
6488+
PartitionMode::CollectLeft,
6489+
NullEquality::NullEqualsNothing,
6490+
true,
6491+
)?;
6492+
6493+
assert!(!join.allow_join_dynamic_filter_pushdown(session_config.options()));
6494+
6495+
Ok(())
6496+
}
6497+
6498+
#[test]
6499+
fn test_dynamic_filter_pushdown_allows_null_aware_non_null_build_key() -> Result<()> {
6500+
// A NOT NULL build key cannot surface a build-side NULL, so the
6501+
// pushdown must stay enabled.
6502+
let left = build_table(("a1", &vec![1]), ("b1", &vec![1]), ("c1", &vec![1]));
6503+
let right = build_table(("a2", &vec![2]), ("b2", &vec![2]), ("c2", &vec![2]));
6504+
let on = vec![(
6505+
Arc::new(Column::new_with_schema("a1", &left.schema())?) as _,
6506+
Arc::new(Column::new_with_schema("a2", &right.schema())?) as _,
6507+
)];
6508+
6509+
let mut session_config = SessionConfig::default();
6510+
session_config
6511+
.options_mut()
6512+
.optimizer
6513+
.enable_join_dynamic_filter_pushdown = true;
6514+
6515+
let join = HashJoinExec::try_new(
6516+
left,
6517+
right,
6518+
on,
6519+
None,
6520+
&JoinType::LeftAnti,
6521+
None,
6522+
PartitionMode::CollectLeft,
6523+
NullEquality::NullEqualsNothing,
6524+
true,
6525+
)?;
6526+
6527+
assert!(join.allow_join_dynamic_filter_pushdown(session_config.options()));
6528+
6529+
Ok(())
6530+
}
6531+
64476532
#[test]
64486533
fn test_with_dynamic_filter_rejects_invalid_columns() -> Result<()> {
64496534
let (_, _, on) = build_schema_and_on()?;

‎datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -383,7 +383,7 @@ logical_plan
383383
physical_plan
384384
01)HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(id@0, id@0)]
385385
02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_left.parquet]]}, projection=[id, data], file_type=parquet
386-
03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ]
386+
03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet
387387

388388
# LEFT MARK JOIN: the OR prevents decorrelation to LeftSemi, so the optimizer
389389
# uses LeftMark. Self-generated dynamic filter pushes to the probe side.
@@ -478,7 +478,7 @@ physical_plan
478478
01)SortExec: TopK(fetch=2), expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false]
479479
02)--HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(id@0, id@0)]
480480
03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_left.parquet]]}, projection=[id, data], file_type=parquet, predicate=DynamicFilter [ empty ]
481-
04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet, predicate=DynamicFilter [ empty ]
481+
04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/join_right.parquet]]}, projection=[id], file_type=parquet
482482

483483
# Correctness check
484484
query IT

‎datafusion/sqllogictest/test_files/push_down_filter_parquet.slt‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1023,6 +1023,72 @@ statement ok
10231023
drop table int_probe;
10241024

10251025

1026+
########
1027+
# Regression test for build-NULL + emptied-probe interaction in null-aware LeftAnti joins.
1028+
#
1029+
# `x NOT IN (subquery)` plans as a null-aware LeftAnti hash join where `x` is
1030+
# the build (left) side. The dynamic-filter pushdown derives a bounds/membership
1031+
# filter from the build keys and pushes it onto the probe scan. When the build
1032+
# contains a NULL key and the filter prunes every probe row, the probe looks
1033+
# empty to the join. A null-aware LeftAnti treats an empty probe as a genuinely-
1034+
# absent subquery, so it emits the build-side NULL as a matching row. That is
1035+
# wrong: `NULL NOT IN (non-empty set)` must be UNKNOWN, not TRUE.
1036+
#
1037+
# The fix: suppress dynamic-filter pushdown whenever the build key is nullable
1038+
# and the join is null-aware, so the probe is never artificially emptied.
1039+
########
1040+
1041+
statement ok
1042+
set datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
1043+
1044+
statement ok
1045+
set datafusion.execution.parquet.pushdown_filters = true;
1046+
1047+
# Build side: `ao` has a nullable `id` column; the NULL row is the one that
1048+
# must NOT appear in the output.
1049+
query I
1050+
COPY (SELECT * FROM (VALUES (5), (NULL)) v(id))
1051+
TO 'test_files/scratch/push_down_filter_parquet/ao_p.parquet'
1052+
STORED AS PARQUET;
1053+
----
1054+
2
1055+
1056+
# Probe / subquery side: `i_disj` has two non-NULL values that don't match 5,
1057+
# and no NULLs. The subquery is non-empty, so `NULL NOT IN (...)` is UNKNOWN.
1058+
query I
1059+
COPY (SELECT * FROM (VALUES (2), (3)) v(eid))
1060+
TO 'test_files/scratch/push_down_filter_parquet/i_disj_p.parquet'
1061+
STORED AS PARQUET;
1062+
----
1063+
2
1064+
1065+
statement ok
1066+
CREATE EXTERNAL TABLE ao_p (id INT) STORED AS PARQUET
1067+
LOCATION 'test_files/scratch/push_down_filter_parquet/ao_p.parquet';
1068+
1069+
statement ok
1070+
CREATE EXTERNAL TABLE i_disj_p (eid INT) STORED AS PARQUET
1071+
LOCATION 'test_files/scratch/push_down_filter_parquet/i_disj_p.parquet';
1072+
1073+
# Must return only `5`. `NULL NOT IN (2, 3)` is UNKNOWN, so that row is dropped.
1074+
query I
1075+
SELECT id FROM ao_p WHERE id NOT IN (SELECT eid FROM i_disj_p) ORDER BY id;
1076+
----
1077+
5
1078+
1079+
statement ok
1080+
drop table ao_p;
1081+
1082+
statement ok
1083+
drop table i_disj_p;
1084+
1085+
statement ok
1086+
RESET datafusion.optimizer.enable_join_dynamic_filter_pushdown;
1087+
1088+
statement ok
1089+
RESET datafusion.execution.parquet.pushdown_filters;
1090+
1091+
10261092
# Config reset
10271093
statement ok
10281094
RESET datafusion.explain.physical_plan_only;

0 commit comments

Comments
 (0)