You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ce077f6
Browse filesBrowse the repository at this point in the historyBrowse files
authored and
Gene Bordegaray
committed
feat: allow Full joins to reuse range co-partitioning in HashJoinExec (apache#23583)
## Which issue does this PR close?
- Closesapache#23454.
## Rationale for this change
apache#23184 let compatible range-partitioned inputs satisfy inner partitioned
hash joins without repartitioning. Full partitioned equi joins still
always went through the conservative hash-repartition path, even when
both inputs were already co-partitioned by range on the join key(s). The
per-partition unmatched-row tracking in `HashJoinExec` is already
partition-local under `PartitionMode::Partitioned` (not shared globally
like in `CollectLeft`), so Full-join semantics generalize cleanly to
range co-partitioning with no additional bookkeeping required.
## What changes are included in this PR?
- Extend `HashJoinExec::input_distribution_requirements()` to opt
`JoinType::Full` in to
`allow_range_satisfaction_for_key_partitioning()`, alongside the
existing `JoinType::Inner` case. The underlying `co_partitioned` /
`compatible_co_partitioning_layout` / `co_partitioning_satisfied` logic
in `distribution_requirements.rs` was already join-type-agnostic, so no
changes were needed there.
- Add planner unit tests in `enforce_distribution.rs` covering both the
compatible-layout case (no repartition inserted) and the
incompatible-split-points case (repartition still inserted) for
`JoinType::Full`.
- Add a `range_partitioned_sparse` sqllogictest fixture table with the
same partition layout as `range_partitioned` but only partially
overlapping keys, and add sqllogictest coverage in
`range_partitioning.slt` for: a compatible Full join avoiding
repartition, an incompatible Full join still repartitioning, and
matched/left-only/right-only unmatched rows produced correctly by a
co-partitioned Full join.
## Are these changes tested?
Yes:
- Two new Rust unit tests in
`datafusion/core/tests/physical_optimizer/enforce_distribution.rs`
assert on the physical plan shape (repartition inserted or not) for
compatible and incompatible range layouts.
- Three new sqllogictest cases in
`datafusion/sqllogictest/test_files/range_partitioning.slt` exercise the
feature end-to-end against real data, including matched rows, left-only
unmatched rows, and right-only unmatched rows for a Full outer join.
- Existing `enforce_distribution` tests, `range_partitioning.slt`, and
proto roundtrip tests all continue to pass.
## Are there any user-facing changes?
Yes: `EXPLAIN` output for `Full` joins over compatible range-partitioned
inputs will no longer show a `RepartitionExec`, and such queries will
avoid the associated hash-shuffle cost at execution time. No public API
changes.
---------
Co-authored-by: Matthew Patton <[email protected]>
0 commit comments