You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9e0ea6c
Browse filesBrowse the repository at this point in the historyBrowse files
fix: skip dynamic filter pushdown for null-aware anti joins with a nullable build key (apache#23173)
## Which issue does this close?
- Closesapache#23126.
## Rationale for this change
`x NOT IN (subquery)` plans to a null-aware `LeftAnti` hash join (build
= outer `x`, probe = subquery). Join dynamic filter pushdown pushes a
bounds + membership filter, built from the build keys, onto the probe
scan. That filter can prune every probe row. A null-aware `LeftAnti`
reads an empty probe as a genuinely-empty subquery, so it emits
build-side NULL rows that should drop: `NULL NOT IN (non-empty)` is
UNKNOWN, not TRUE.
The result is scan-dependent, so it's a silent correctness bug. A
`VALUES` scan ignores the pushed filter and stays correct; a parquet
scan applies it and is wrong.
apache#23103 (the probe-side NULL drop) is orthogonal; this is the build-side
NULL.
## What changes are included in this PR?
Skip join dynamic filter pushdown for a null-aware anti join when the
build key can be NULL. The build-side NULL emission depends on whether
the probe is truly empty, which the pushed filter can change by emptying
it. A NOT NULL build key has no such NULL, so it keeps the pushdown.
The check is static: a schema-nullable build key disables the pushdown
even when the data contains no NULLs. A runtime alternative (keep the
pushdown and neutralize the filter only when the build actually holds a
NULL key) would restore the optimization for those cases. I'd leave that
as a follow-up.
## Are these changes tested?
Yes. A `push_down_filter_parquet.slt` case reproduces it (build-side
NULL, a non-matching parquet probe) and asserts the single correct row.
Without the change it returns the extra NULL. In addition, unit tests
pin both directions of the guard: a nullable build key rejects the
pushdown and a NOT NULL build key keeps it.
## Are there any user-facing changes?
`NOT IN` over a parquet (or otherwise prunable) scan with a nullable
outer key now returns correct results. Such joins lose the dynamic
filter pushdown.
0 commit comments