You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fc503ee
Browse filesBrowse the repository at this point in the historyBrowse files
Optimize NULL-containing string IN list pruning (apache#24954)
## Which issue does this PR close?
- Closesapache#24711.
## Rationale for this change
apache#24781 added compact pruning for non-NULL `NOT IN` lists. Issue apache#24711
still needs support for NULL-containing lists.
Large string lists can create expensive per-value pruning expressions.
The compact path reduces this cost when `max_in_list_size` permits the
rewrite.
NULL values require special handling. Filter pruning treats FALSE and
UNKNOWN equally, but full-match detection must keep these results
separate.
## What changes are included in this PR?
This PR makes these changes:
- Supports compact pruning for large string `IN` lists that contain
NULL.
- Treats `NOT IN` with NULL as unable to match rows.
- Treats all-NULL `IN` lists as unable to match rows.
- Records whether a pruning predicate is safe for full-match inversion.
- Uses the same configured list-size cap when it builds the inverse
predicate.
- Adds unit tests, Parquet integration tests, documentation, and
benchmarks.
## What is the testing strategy for this PR?
The tests cover positive and negative lists, all-NULL lists, nested `OR`
expressions, configured caps, and Parquet full-match detection.
These commands passed:
```shell
cargo test -p datafusion-pruning
cargo test -p datafusion-datasource-parquet
cargo test -p datafusion --test parquet_integration string_in_list_pruning
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
```
Focused benchmarks also cover compact `IN` and constant-false `NOT IN`
expressions with NULL.
## Are there any user-facing changes?
Large string `IN` and `NOT IN` lists that contain NULL can now use
compact pruning when the cap permits it.
Query results and configuration defaults do not change.
This PR adds two non-breaking `PruningPredicate` accessors:
`max_in_list_size` and `can_be_inverted_for_full_match`.
Backport provenance:
- Upstream PR: apache#24954
- Upstream commit: 33028d5
- Upstream self-tests and all additions/removals retained; only unavoidable fork patch context resolved.
- Any fork-specific adaptation is kept in a separate following commit.
Signed-off-by: discord9 <[email protected]>
0 commit comments