You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8733e2e
Browse filesBrowse the repository at this point in the historyBrowse files
authored
fix: simplify IN lists containing NULL literals in EliminateFilter (apache#26548)
## Which issue does this PR close?
- Closesapache#26542.
## Rationale for this change
Under filter semantics only rows for which the predicate is `TRUE` are
kept, so `NULL` rejects a row exactly like `FALSE`. That makes two
rewrites sound for `IN` lists that hold `NULL` literals:
- `x IN (a, NULL)` is `TRUE` exactly when `x` matches `a`. A `NULL`
member can only turn `FALSE` into `NULL`, and both reject. The `NULL`
members can be dropped, and a list of nothing but `NULL`s rejects every
row.
- `x NOT IN (a, NULL)` is `FALSE` when `x` matches `a` and `NULL`
otherwise, so it is never `TRUE` and rejects every row.
Leaving the `NULL`s in place keeps a filter that `PruningPredicate` does
not handle well, which is the pruning degradation described in the
issue. apache#24711 fixed this only for the compact path (lists over 20
members) and apache#25187 made short lists on several types reach the same
path.
## What changes are included in this PR?
An `Expr::InList` arm in `simplify_filter_predicate`, plus a
`simplify_filter_in_list` helper and an `is_null_literal` predicate, in
`datafusion/optimizer/src/eliminate_filter.rs`.
The rewrite stays in `EliminateFilter` deliberately. It is **not** sound
in `ExprSimplifier`: in a general expression context `SELECT 3 IN (1,
NULL)` must remain `NULL`, and dropping the `NULL` would make it
`FALSE`.
The negated case keeps the original `InList` inside
`FilterPredicate::RejectsAll` rather than substituting a literal, so
nothing is discarded that the `AND`/`OR` reconstruction may still need.
## Are these changes tested?
Yes — five tests in `eliminate_filter.rs`:
| test | asserts |
|---|---|
| `filter_in_list_with_null_drops_null` | `a IN (1, 2, NULL)` → `a IN
(1, 2)` |
| `filter_in_list_of_only_nulls_rejects_all` | `a IN (NULL)` →
`EmptyRelation` |
| `filter_not_in_list_with_null_rejects_all` | `a NOT IN (1, NULL)` →
`EmptyRelation` |
| `filter_in_list_without_null_is_untouched` | no `NULL` → unchanged |
| `filter_not_in_list_without_null_is_untouched` | no `NULL` → unchanged
|
I wrote the tests first and confirmed the three `NULL` cases **fail on
`main`** before the change and pass after, while the two control tests
pass in both states.
Also run: the full `datafusion-optimizer` suite (948 + 29 + 5 passing),
`cargo fmt --all -- --check`, and `cargo clippy -p datafusion-optimizer
--all-targets -- -D warnings`, all clean. `in_list.slt` and the
`null_aware_*` slt files pass. `expr`, `predicates`, `scalar` and
`aggregate` slt files each fail identically with and without this change
on my machine (a local test-data issue), so they are unaffected.
## Are there any user-facing changes?
Explain output for filters whose `IN` list contains `NULL` literals: the
`NULL` members no longer appear, and a filter that can never be `TRUE`
becomes an empty relation. Query results are unchanged.
---------
Signed-off-by: Raghvendra Yadav <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: Andrew Lamb <[email protected]>
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/simplify_expr.slt
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -296,10 +296,10 @@ explain select price from decimal_prices where cast(price as decimal(38,15)) in
296
296
);
297
297
----
298
298
logical_plan
299
-
01)Filter: CAST(decimal_prices.price AS Decimal128(38, 15)) IN ([Decimal128(4.000000000000000,38,15), Decimal128(4.000000000000001,38,15), Decimal128(NULL,38,15)])
299
+
01)Filter: CAST(decimal_prices.price AS Decimal128(38, 15)) IN ([Decimal128(4.000000000000000,38,15), Decimal128(4.000000000000001,38,15)])
300
300
02)--TableScan: decimal_prices projection=[price]
301
301
physical_plan
302
-
01)FilterExec: CAST(price@0 AS Decimal128(38, 15)) IN (SET) ([4.000000000000000, 4.000000000000001, NULL])
302
+
01)FilterExec: CAST(price@0 AS Decimal128(38, 15)) IN (SET) ([4.000000000000000, 4.000000000000001])
0 commit comments