Skip to content

Commit e12901d

Browse files
authored
fix: account for row duplication and padding in join null counts (apache#25861)
## Which issue does this PR close? Related to apache#25620. Split from apache#25648 following [the review request](apache#25648 (comment)). This PR can be reviewed and merged independently. ## Rationale for this change Joins repeat or drop input rows, and outer joins add NULL padding. Copying each input column's null count unchanged can therefore produce inconsistent output statistics. For example, a left join can repeat a NULL in a non-key left column many times, while an unmatched left row introduces NULLs in every right column. ## What changes are included in this PR? - Estimate inner and outer join null counts from the rows carrying each side's values and the rows added as NULL padding. - Estimate unmatched input rows using the existing semi-join estimator. Inner-join row counts count matching pairs, so they cannot alone determine how many input rows are unmatched. - Account for NULL keys under ordinary and null-safe equality, while retaining unknown contributions and conservative precision. - Rescale null counts when a statistics provider replaces the row estimate. Cover projected joins and swapped inputs. The semi/anti-join estimator itself is unchanged. The rules follow join semantics and do not contain benchmark-specific cases. ## What is the testing strategy for this PR? Added unit tests cover NULL padding, repeated matches, unknown counts, null-safe keys, projection, input swapping, and provider rescaling. Existing cardinality assertions remain in place. Passed on this independent branch: - `cargo test --profile ci -p datafusion-physical-plan --lib` (2,369 tests) - `cargo clippy --profile ci -p datafusion-physical-plan --all-targets --all-features -- -D warnings` - `cargo fmt --all` and `git diff --check` Benchmarks were not rerun for the split. This extracts the join implementation already present in apache#25648; its earlier combined measurements are not isolated measurements of this PR. ## Are there any user-facing changes? Join row and null-count estimates can change, which can affect EXPLAIN output and plan choices. No public API changes.
1 parent 0cba642 commit e12901d

2 files changed

Lines changed: 766 additions & 67 deletions

File tree

0 commit comments

Comments
 (0)