Skip to content

[improvement](nereids) Support OR expansion for right joins - #70471

Open
englefly wants to merge 2 commits into
apache:masterfrom
englefly:fix/or-expansion-right-joins
Open

englefly wants to merge 2 commits into
apache:masterfrom
englefly:fix/or-expansion-right-joins

Conversation

@englefly

@englefly englefly commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Background

RBO normally uses SemiJoinCommute to normalize RIGHT OUTER and RIGHT ANTI joins to their left equivalents before OR expansion. This normalization is skipped when join reordering is disabled, the join is marked LEADING, or it has a distribution hint. OR expansion replaces eligible disjunctive joins with hash join branches, using anti joins for unmatched outer/anti rows.

What problem does this PR solve?

Issue Number: None
Related PR: #67067 (join-order rewrite context; master retains SemiJoinCommute)
PR Type: Enhancement

Problem Summary:

When an eligible RIGHT OUTER or RIGHT ANTI join survives RBO normalization, OR expansion skips it because its supported-type set only includes INNER, LEFT OUTER, FULL OUTER and LEFT ANTI. For example, setting disable_join_reorder=true keeps the original right join but does not disable OR_EXPANSION. Extend both right types using the right-preserving anti expansion already used by FULL OUTER JOIN, preserving original output slots and ordering. Ordinary right joins that are already normalized to left can use the existing expansion path; the new regression explicitly prevents that normalization so it exercises the added right-join paths.

Details

  • Change map: 16 changed production lines in OrExpansion.java; 176 changed test lines across OrExpansionTest.java and the new regression suite; 223 generated expected-result lines.
  • Before and after: When a right join reaches OrExpansion, RIGHT OUTER now combines matching branches with unmatched right rows and null-pads the left output. RIGHT ANTI returns unmatched right rows using the existing left anti helper with producers and slot maps swapped together. Previously these right types were skipped.
  • Configs and defaults: No new settings. The existing OR_EXPANSION switch controls expansion; mark joins and ineligible hash disjuncts remain excluded. The regression sets disable_join_reorder=true specifically to preserve right joins through RBO and prevent CBO commute from obscuring the baseline direction.
  • Impact on existing users: Eligible right joins that survive normalization can use hash joins instead of nested loop joins. Existing left/inner/full expansion paths remain covered. JoinExtractOrFromCaseWhen shares the eligibility check, so right-side CASE extraction is also covered.
  • Behavior change: Plans for retained right joins can change; SQL result semantics remain the same. No BE operator, protocol or storage change is needed.

Testing

Run locally with the compiled FE and an existing ASAN BE:

  • MAVEN_ARGS='-Ddisable.obs=true' bash run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.OrExpansionTest: 14 tests passed for the production implementation, including direct right-join expansion, output ExprIds/order/nullability, CTE nesting, expression keys, residual predicates and non-hash rejection.
  • bash build.sh --fe --exclude-obs-dependencies -j8: passed, including FE core Checkstyle with 0 violations. The follow-up changes only the Groovy regression suite and this description.
  • bash run-regression-test.sh --conf /tmp/or-expansion-regression-conf.groovy --run -d query_p0/union -s or_expansion_right_join: passed after the coverage correction. With reordering disabled, EXPLAIN checks require RIGHT OUTER/RIGHT ANTI nested loop joins when expansion is disabled, and hash joins with no nested loop joins when expansion is enabled. CASE extraction has the same positive/negative plan checks.
  • All 30 result blocks match the existing auto-generated expected file, and all 15 enabled/disabled result pairs are identical. Coverage includes duplicates, overlapping disjuncts, NULLs, null-safe equality, residual predicates, CASE extraction, aggregation and empty inputs. No expected-output changes were necessary.
  • Earlier implementation validation also passed the existing or_expansion and join_extract_or_from_case_when suites. The temporary configuration selects the isolated local cluster.

Release note

Support OR expansion for eligible RIGHT OUTER and RIGHT ANTI joins that survive RBO join-direction normalization, allowing these queries to use hash joins.

Check List (For Author)

  • Behavior changed: Yes (see Details).
  • Does this need documentation: No.
  • Pre-submission status: Ready.

AI Disclosure

  • Tool: Codex (GPT-6).
  • Generated: Implementation, tests and this PR description.
  • Human review: Pending author review.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#67067

Problem Summary: OR expansion excludes RIGHT OUTER and RIGHT ANTI joins even when each disjunct is an eligible hash equality. Consequently, equivalent queries can miss hash join expansion depending on their join direction or earlier join swaps. Extend the supported types and reuse the existing FULL OUTER right-preserving anti expansion path. RIGHT OUTER combines matching branches with unmatched right rows, while RIGHT ANTI emits only unmatched right rows. Preserve the original output slots, order and nullability, and add unit and regression coverage for both new types and the existing paths.

### Release note

Support OR expansion for RIGHT OUTER and RIGHT ANTI joins with eligible disjunctive equality predicates, allowing these queries to use hash joins.

### Check List (For Author)

- Test:
    - Unit Test: MAVEN_ARGS='-Ddisable.obs=true' bash run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.OrExpansionTest (14 passed)
    - Regression test: or_expansion_right_join, or_expansion and join_extract_or_from_case_when (3 suites passed using run-regression-test.sh)
    - Manual test: All 15 enabled/disabled result pairs match an independent bag-semantics oracle.
    - Build and style: bash build.sh --fe --exclude-obs-dependencies -j8 (passed, 0 FE core Checkstyle violations)
- Behavior changed: Yes. Eligible right outer and right anti joins can now use OR expansion.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#70471, apache#67067

Problem Summary: With join reordering enabled, SemiJoinCommute can normalize RIGHT OUTER and RIGHT ANTI joins to LEFT joins before OR expansion. The regression's hash-join assertions could therefore pass through existing left-join expansion without exercising the new right-join dispatch. Set disable_join_reorder=true in the suite and add negative-control EXPLAIN checks that retain the corresponding RIGHT nested loop join when OR expansion is disabled. Keep the existing enabled hash-join checks and result comparisons, including CASE extraction, aggregation and empty inputs.

### Release note

None

### Check List (For Author)

- Test:
    - Regression test: bash run-regression-test.sh --conf /tmp/or-expansion-regression-conf.groovy --run -d query_p0/union -s or_expansion_right_join (passed)
    - Manual test: All 30 actual output blocks match the committed generated expectations; all 15 enabled/disabled pairs match.
    - Style: git diff --check (passed)
- Behavior changed: No. Test coverage only.
- Does this need documentation: No
@englefly
englefly marked this pull request as ready for review November 11, 2026 10:45
@englefly

Copy link
Copy Markdown
Contributor Author

run buildall

@englefly

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43006288513

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant