Repository navigation
Commit c120619
authored
fix: use null counts to estimate IS NULL and IS NOT NULL (apache#25648)
## Which issue does this PR close?
Closes apache#25620.
## Rationale for this change
Bare `IS NULL` and `IS NOT NULL` filters use the default selectivity
even when the input has a null count. For five rows containing two
NULLs, both filters estimate one row instead of two and three.
## What changes are included in this PR?
Use available null counts to estimate bare column null checks, while
retaining the existing fallback for missing statistics and other
predicates. Keep the output column statistics consistent with all-null,
non-null, and empty results. Preserve the checked column's distinct
count through the statistics provider when all its non-null values
survive.
Add regression coverage for exact and estimated counts, missing
statistics, projection, fetch, and null filters above outer joins.
The supporting changes in apache#25861, apache#25862, and apache#25863 have merged. This
PR now contains only the null-filter change on current `main`.
## What is the testing strategy for this PR?
The null-count and statistics-provider regressions fail against
unchanged `main` production code.
Compared 21 optimized `compute_statistics` coalesce, filter, and
mixed-chain cases against `main` at
`2717e25625ff0c51c1805f8df52d317979db74ec`, using the `release-nonlto`
profile and 30 samples per case. Median time changes were +0.4% for
coalesce controls, -1.1% for filter chains, and -5.1% for mixed chains.
These local measurements cover statistics computation.
## Are there any user-facing changes?
Null-check row estimates and related EXPLAIN statistics improve. Plan
choices can change as a result. No public API changes.1 parent caaf0fa commit c120619
3 files changed
Lines changed: 533 additions & 8 deletions
File tree
- datafusion
- physical-plan/src
- operator_statistics
- sqllogictest/test_files
0 commit comments