Skip to content

Commit c2c0bac

Browse files
mdashtilukekim
authored andcommitted
fix: keep null-aware anti-join NULLs in the pushed dynamic filter (apache#23104)
## Which issue does this close? Closes apache#23103. ## Rationale for this change A hash join pushes a build-side dynamic filter (`key IN build_keys`) down to the probe scan. For a null-aware anti join (`NOT IN`), that filter drops the probe's NULL rows. But `NOT IN` three-valued logic needs a probe-side NULL to collapse the whole result to zero rows. With the NULL filtered away at the scan, before the join's null-aware check runs, the join returns rows that shouldn't be there. ## What changes are included in this PR? `SharedBuildAccumulator::build_filter` now ORs `probe_key IS NULL` into the pushed predicate when the join is `null_aware`. Non-NULL probe rows still get filtered, so the optimization stays. `HashJoinExec`'s `null_aware` validation already guarantees a single probe key. ## Are these changes tested? Yes. Added a parquet-backed case to `null_aware_anti_join.slt`. The existing cases use in-memory `VALUES`, whose scans never apply the pushed filter, so they passed despite the bug. The new one sets `parquet.pushdown_filters = true` so the filter runs row-level. Without the fix it returns `1, 3`; with it, zero rows. ## Are there any user-facing changes? A `NOT IN` over a NULL-bearing inner now returns zero rows instead of leaking rows, when join dynamic filter pushdown and row-level scan filtering are both on. --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Adrian Garcia Badaracco <[email protected]> Co-authored-by: rjhallsted <[email protected]> (cherry picked from commit 8d680db) Conflicts resolved when applying to spiceai-54 (DataFusion 54.1): - datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs: both hunks are in test scaffolding. Upstream adds null_aware: false to the make_partitioned_accumulator_for_test and completed_partitions_for_test helpers, which 54.1 does not have, so they are not added. In the tests module helper, 54.1 builds probe_schema locally where upstream calls test_probe_schema(); kept the 54.1 form and added null_aware: false. The accumulator changes themselves applied as upstream wrote them; the fork's CollectLeftAccumulator seam only computes bounds, and the pushed filter is still published only at the two update sites this wraps.
1 parent 7f292c2 commit c2c0bac

3 files changed

Lines changed: 104 additions & 3 deletions

File tree

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1394,6 +1394,7 @@ impl<A: CollectLeftAccumulator + 'static> ExecutionPlan for HashJoinExec<A> {
13941394
filter,
13951395
on_right,
13961396
repartition_random_state,
1397+
self.null_aware,
13971398
))
13981399
})))
13991400
})

‎datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs‎

Lines changed: 38 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ use datafusion_common::{DataFusionError, Result, ScalarValue, SharedResult};
3838
use datafusion_expr::Operator;
3939
use datafusion_functions::core::r#struct as struct_func;
4040
use datafusion_physical_expr::expressions::{
41-
BinaryExpr, CaseExpr, DynamicFilterPhysicalExpr, InListExpr, lit,
41+
BinaryExpr, CaseExpr, DynamicFilterPhysicalExpr, InListExpr, IsNullExpr, lit,
4242
};
4343
use datafusion_physical_expr::{PhysicalExpr, PhysicalExprRef, ScalarFunctionExpr};
4444

@@ -308,6 +308,9 @@ pub(crate) struct SharedBuildAccumulator {
308308
repartition_random_state: SeededRandomState,
309309
/// Schema of the probe (right) side for evaluating filter expressions
310310
probe_schema: Arc<Schema>,
311+
/// Null-aware anti join (`NOT IN`). A probe-side NULL must reach the join so its
312+
/// three-valued logic can collapse the result, so the pushed filter keeps NULL rows.
313+
null_aware: bool,
311314
}
312315

313316
/// Strategy for filter pushdown (decided at collection time)
@@ -411,6 +414,7 @@ impl SharedBuildAccumulator {
411414
dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
412415
on_right: Vec<PhysicalExprRef>,
413416
repartition_random_state: SeededRandomState,
417+
null_aware: bool,
414418
) -> Self {
415419
// Troubleshooting: If partition counts are incorrect, verify this logic matches
416420
// the actual execution pattern in collect_build_side()
@@ -457,6 +461,7 @@ impl SharedBuildAccumulator {
457461
on_right,
458462
repartition_random_state,
459463
probe_schema: right_child.schema(),
464+
null_aware,
460465
}
461466
}
462467

@@ -632,7 +637,8 @@ impl SharedBuildAccumulator {
632637
if let Some(filter_expr) =
633638
combine_membership_and_bounds(membership_expr, bounds_expr)
634639
{
635-
self.dynamic_filter.update(filter_expr)?;
640+
self.dynamic_filter
641+
.update(self.null_aware_filter(filter_expr))?;
636642
}
637643
}
638644
PartitionStatus::Pending => {
@@ -738,12 +744,40 @@ impl SharedBuildAccumulator {
738744
)?) as Arc<dyn PhysicalExpr>
739745
};
740746

741-
self.dynamic_filter.update(filter_expr)?;
747+
self.dynamic_filter
748+
.update(self.null_aware_filter(filter_expr))?;
742749
}
743750
}
744751

745752
Ok(())
746753
}
754+
755+
/// Wraps a pushdown filter so a null-aware anti join keeps its probe-side NULL rows.
756+
///
757+
/// The build-side predicate drops probe rows whose key is NULL, but `NOT IN` three-valued
758+
/// logic needs that NULL to reach the join. OR-ing `probe_key IS NULL` preserves the dynamic
759+
/// filter's selectivity for non-NULL rows while letting the NULL through.
760+
fn null_aware_filter(
761+
&self,
762+
filter_expr: Arc<dyn PhysicalExpr>,
763+
) -> Arc<dyn PhysicalExpr> {
764+
if !self.null_aware {
765+
return filter_expr;
766+
}
767+
debug_assert_eq!(
768+
self.on_right.len(),
769+
1,
770+
"null_aware anti join must have exactly one probe key"
771+
);
772+
let probe_key_is_null: Arc<dyn PhysicalExpr> =
773+
Arc::new(IsNullExpr::new(Arc::clone(&self.on_right[0])));
774+
// Cheap null check first short-circuits before the costlier dynamic filter.
775+
Arc::new(BinaryExpr::new(
776+
probe_key_is_null,
777+
Operator::Or,
778+
filter_expr,
779+
))
780+
}
747781
}
748782

749783
impl Debug for SharedBuildAccumulator {
@@ -776,6 +810,7 @@ mod tests {
776810
on_right: vec![],
777811
repartition_random_state: SeededRandomState::with_seed(1),
778812
probe_schema,
813+
null_aware: false,
779814
}
780815
}
781816

‎datafusion/sqllogictest/test_files/null_aware_anti_join.slt‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -451,3 +451,68 @@ DROP TABLE customers_test;
451451

452452
statement ok
453453
DROP TABLE all_null_banned;
454+
455+
#############
456+
## Test: dynamic filter pushdown must not drop inner (probe-side) NULLs.
457+
## With join dynamic filter pushdown on, the build-side filter pushed to the probe scan would drop
458+
## inner NULLs, but NOT IN three-valued logic needs them to collapse the result to zero rows. The
459+
## in-memory VALUES scans above never apply the pushed filter, so this case needs a parquet scan.
460+
#############
461+
462+
statement ok
463+
set datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
464+
465+
# Row-level parquet filtering, so the pushed filter actually drops matching rows instead of only
466+
# pruning row groups. Without this the single row group is read whole and the NULL never gets dropped.
467+
statement ok
468+
set datafusion.execution.parquet.pushdown_filters = true;
469+
470+
statement ok
471+
CREATE TABLE asa_outer(id INT) AS VALUES (1), (2), (3);
472+
473+
statement ok
474+
CREATE TABLE asa_inner(eid INT) AS VALUES (2), (NULL);
475+
476+
query I
477+
COPY asa_outer TO 'test_files/scratch/null_aware_anti_join/asa_outer.parquet' STORED AS PARQUET;
478+
----
479+
3
480+
481+
query I
482+
COPY asa_inner TO 'test_files/scratch/null_aware_anti_join/asa_inner.parquet' STORED AS PARQUET;
483+
----
484+
2
485+
486+
statement ok
487+
CREATE EXTERNAL TABLE asa_outer_parquet(id INT)
488+
STORED AS PARQUET
489+
LOCATION 'test_files/scratch/null_aware_anti_join/asa_outer.parquet';
490+
491+
statement ok
492+
CREATE EXTERNAL TABLE asa_inner_parquet(eid INT)
493+
STORED AS PARQUET
494+
LOCATION 'test_files/scratch/null_aware_anti_join/asa_inner.parquet';
495+
496+
# Expected: zero rows. Before the fix the pushed dynamic filter dropped inner NULLs, so the join
497+
# wrongly returned id = 1 and id = 3.
498+
query I
499+
SELECT id FROM asa_outer_parquet WHERE id NOT IN (SELECT eid FROM asa_inner_parquet) ORDER BY id;
500+
----
501+
502+
statement ok
503+
DROP TABLE asa_outer;
504+
505+
statement ok
506+
DROP TABLE asa_inner;
507+
508+
statement ok
509+
DROP TABLE asa_outer_parquet;
510+
511+
statement ok
512+
DROP TABLE asa_inner_parquet;
513+
514+
statement ok
515+
RESET datafusion.execution.parquet.pushdown_filters;
516+
517+
statement ok
518+
RESET datafusion.optimizer.enable_join_dynamic_filter_pushdown;

0 commit comments

Comments
 (0)