feat: allow Full joins to reuse range co-partitioning in HashJoinExec by mattp5657 · Pull Request #23583 · apache/datafusion

mattp5657 · 2026-07-14T19:48:29Z

Which issue does this PR close?

Closes Allow co-partitioned Partitioning::Range inputs for full hash joins #23454.

Rationale for this change

#23184 let compatible range-partitioned inputs satisfy inner partitioned hash joins without repartitioning. Full partitioned equi joins still always went through the conservative hash-repartition path, even when both inputs were already co-partitioned by range on the join key(s). The per-partition unmatched-row tracking in HashJoinExec is already partition-local under PartitionMode::Partitioned (not shared globally like in CollectLeft), so Full-join semantics generalize cleanly to range co-partitioning with no additional bookkeeping required.

What changes are included in this PR?

Extend HashJoinExec::input_distribution_requirements() to opt JoinType::Full in to allow_range_satisfaction_for_key_partitioning(), alongside the existing JoinType::Inner case. The underlying co_partitioned / compatible_co_partitioning_layout / co_partitioning_satisfied logic in distribution_requirements.rs was already join-type-agnostic, so no changes were needed there.
Add planner unit tests in enforce_distribution.rs covering both the compatible-layout case (no repartition inserted) and the incompatible-split-points case (repartition still inserted) for JoinType::Full.
Add a range_partitioned_sparse sqllogictest fixture table with the same partition layout as range_partitioned but only partially overlapping keys, and add sqllogictest coverage in range_partitioning.slt for: a compatible Full join avoiding repartition, an incompatible Full join still repartitioning, and matched/left-only/right-only unmatched rows produced correctly by a co-partitioned Full join.

Are these changes tested?

Yes:

Two new Rust unit tests in datafusion/core/tests/physical_optimizer/enforce_distribution.rs assert on the physical plan shape (repartition inserted or not) for compatible and incompatible range layouts.
Three new sqllogictest cases in datafusion/sqllogictest/test_files/range_partitioning.slt exercise the feature end-to-end against real data, including matched rows, left-only unmatched rows, and right-only unmatched rows for a Full outer join.
Existing enforce_distribution tests, range_partitioning.slt, and proto roundtrip tests all continue to pass.

Are there any user-facing changes?

Yes: EXPLAIN output for Full joins over compatible range-partitioned inputs will no longer show a RepartitionExec, and such queries will avoid the associated hash-shuffle cost at execution time. No public API changes.

gene-bordegaray

this looks good, just rebase and ping me for a re-review before I approve 👍

mattp5657 · 2026-07-15T23:36:16Z

@gene-bordegaray This should be ready.

gene-bordegaray

this should be good after next addresses 👍

gene-bordegaray · 2026-07-16T00:01:17Z

-# TEST 12: Non-Inner Range Join Repartitions
-# Only inner partitioned hash joins opt in to Range satisfying KeyPartitioned
-# requirements. Non-inner joins keep using Hash repartitioning.
+# TEST 12: Non-Inner, Non-Full Range Join Repartitions


lets just have this say "Unsupported"

gene-bordegaray · 2026-07-16T00:01:51Z

+02)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
+03)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), file_type=csv, has_header=false
+04)--RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4
+05)----DataSourceExec: file_groups=<slt:ignore>, projection=[range_key, value], output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4), file_type=csv, has_header=false


can we add query results

mattp5657 · 2026-07-17T22:58:22Z

@gene-bordegaray Thank you for the comments, this should be ready when you have a chance :)

codecov-commenter · 2026-07-17T23:24:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.67%. Comparing base (4c20fa7) to head (01d6200).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23583      +/-   ##
==========================================
- Coverage   80.67%   80.67%   -0.01%     
==========================================
  Files        1087     1087              
  Lines      367541   367566      +25     
  Branches   367541   367566      +25     
==========================================
+ Hits       296520   296529       +9     
- Misses      53365    53376      +11     
- Partials    17656    17661       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gene-bordegaray

thank you 🙇

gene-bordegaray · 2026-07-19T12:08:01Z

cc: @gabotechs

gabotechs

Thanks @mattp5657 and @gene-bordegaray!

…apache#23583) ## Which issue does this PR close? - Closes apache#23454. ## Rationale for this change apache#23184 let compatible range-partitioned inputs satisfy inner partitioned hash joins without repartitioning. Full partitioned equi joins still always went through the conservative hash-repartition path, even when both inputs were already co-partitioned by range on the join key(s). The per-partition unmatched-row tracking in `HashJoinExec` is already partition-local under `PartitionMode::Partitioned` (not shared globally like in `CollectLeft`), so Full-join semantics generalize cleanly to range co-partitioning with no additional bookkeeping required. ## What changes are included in this PR? - Extend `HashJoinExec::input_distribution_requirements()` to opt `JoinType::Full` in to `allow_range_satisfaction_for_key_partitioning()`, alongside the existing `JoinType::Inner` case. The underlying `co_partitioned` / `compatible_co_partitioning_layout` / `co_partitioning_satisfied` logic in `distribution_requirements.rs` was already join-type-agnostic, so no changes were needed there. - Add planner unit tests in `enforce_distribution.rs` covering both the compatible-layout case (no repartition inserted) and the incompatible-split-points case (repartition still inserted) for `JoinType::Full`. - Add a `range_partitioned_sparse` sqllogictest fixture table with the same partition layout as `range_partitioned` but only partially overlapping keys, and add sqllogictest coverage in `range_partitioning.slt` for: a compatible Full join avoiding repartition, an incompatible Full join still repartitioning, and matched/left-only/right-only unmatched rows produced correctly by a co-partitioned Full join. ## Are these changes tested? Yes: - Two new Rust unit tests in `datafusion/core/tests/physical_optimizer/enforce_distribution.rs` assert on the physical plan shape (repartition inserted or not) for compatible and incompatible range layouts. - Three new sqllogictest cases in `datafusion/sqllogictest/test_files/range_partitioning.slt` exercise the feature end-to-end against real data, including matched rows, left-only unmatched rows, and right-only unmatched rows for a Full outer join. - Existing `enforce_distribution` tests, `range_partitioning.slt`, and proto roundtrip tests all continue to pass. ## Are there any user-facing changes? Yes: `EXPLAIN` output for `Full` joins over compatible range-partitioned inputs will no longer show a `RepartitionExec`, and such queries will avoid the associated hash-shuffle cost at execution time. No public API changes. --------- Co-authored-by: Matthew Patton <[email protected]>

github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jul 14, 2026

mattp5657 marked this pull request as ready for review July 14, 2026 19:49

mattp5657 changed the title ~~Feat/full join range copartitioning~~ feat: allow Full joins to reuse range co-partitioning in HashJoinExec Jul 14, 2026

gene-bordegaray reviewed Jul 15, 2026

View reviewed changes

Matthew Patton added 3 commits July 15, 2026 19:21

feat: allow Full joins to reuse range co-partitioning in HashJoinExec

768cd26

test: add planner tests for Full join range co-partitioning

656a7b5

test: add sqllogictest coverage for Full join range co-partitioning

8e7a4da

mattp5657 force-pushed the feat/full-join-range-copartitioning branch from d2ca127 to 8e7a4da Compare July 15, 2026 23:35

gene-bordegaray approved these changes Jul 16, 2026

View reviewed changes

Matthew Patton added 2 commits July 17, 2026 18:44

Merge branch 'main' into feat/full-join-range-copartitioning

7c08bc3

test: add Full join results and fix TEST 12 comment

01d6200

gene-bordegaray approved these changes Jul 19, 2026

View reviewed changes

gabotechs approved these changes Jul 20, 2026

View reviewed changes

gabotechs added this pull request to the merge queue Jul 20, 2026

Merged via the queue into apache:main with commit 0f5790a Jul 20, 2026
40 checks passed

alamb mentioned this pull request Aug 25, 2026

Blog post about Range Partitioning #24667

Open

nuno-faria added the v55.0.0 label Oct 3, 2026

Navigation Menu

Repository navigation

feat: allow Full joins to reuse range co-partitioning in HashJoinExec - #23583

feat: allow Full joins to reuse range co-partitioning in HashJoinExec#23583
gabotechs merged 5 commits into
apache:mainfrom
mattp5657:feat/full-join-range-copartitioning

mattp5657 commented Jul 14, 2026

Uh oh!

gene-bordegaray left a comment

Uh oh!

mattp5657 commented Jul 15, 2026

Uh oh!

gene-bordegaray left a comment

Uh oh!

gene-bordegaray Jul 16, 2026

Uh oh!

gene-bordegaray Jul 16, 2026

Uh oh!

mattp5657 commented Jul 17, 2026

Uh oh!

codecov-commenter commented Jul 17, 2026

Uh oh!

gene-bordegaray left a comment

Uh oh!

gene-bordegaray commented Jul 19, 2026

Uh oh!

gabotechs left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mattp5657 commented Jul 14, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

mattp5657 commented Jul 15, 2026

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

mattp5657 commented Jul 17, 2026

Uh oh!

codecov-commenter commented Jul 17, 2026

Codecov Report

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

gene-bordegaray commented Jul 19, 2026

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants