Repository navigation
Commit e12901d
authored
fix: account for row duplication and padding in join null counts (apache#25861)
## Which issue does this PR close?
Related to apache#25620. Split from apache#25648 following [the review
request](apache#25648 (comment)).
This PR can be reviewed and merged independently.
## Rationale for this change
Joins repeat or drop input rows, and outer joins add NULL padding.
Copying each input column's null count unchanged can therefore produce
inconsistent output statistics. For example, a left join can repeat a
NULL in a non-key left column many times, while an unmatched left row
introduces NULLs in every right column.
## What changes are included in this PR?
- Estimate inner and outer join null counts from the rows carrying each
side's values and the rows added as NULL padding.
- Estimate unmatched input rows using the existing semi-join estimator.
Inner-join row counts count matching pairs, so they cannot alone
determine how many input rows are unmatched.
- Account for NULL keys under ordinary and null-safe equality, while
retaining unknown contributions and conservative precision.
- Rescale null counts when a statistics provider replaces the row
estimate. Cover projected joins and swapped inputs.
The semi/anti-join estimator itself is unchanged. The rules follow join
semantics and do not contain benchmark-specific cases.
## What is the testing strategy for this PR?
Added unit tests cover NULL padding, repeated matches, unknown counts,
null-safe keys, projection, input swapping, and provider rescaling.
Existing cardinality assertions remain in place.
Passed on this independent branch:
- `cargo test --profile ci -p datafusion-physical-plan --lib` (2,369
tests)
- `cargo clippy --profile ci -p datafusion-physical-plan --all-targets
--all-features -- -D warnings`
- `cargo fmt --all` and `git diff --check`
Benchmarks were not rerun for the split. This extracts the join
implementation already present in apache#25648; its earlier combined
measurements are not isolated measurements of this PR.
## Are there any user-facing changes?
Join row and null-count estimates can change, which can affect EXPLAIN
output and plan choices. No public API changes.1 parent 0cba642 commit e12901d
2 files changed
Lines changed: 766 additions & 67 deletions
0 commit comments