Skip to content

branch-4.2: [improvement](fe) Prune partition keys from partition topn sort #63379 - #68171

Merged
yiguolei merged 2 commits into
apache:branch-4.2from
CalvinKirs:backport-63379-branch-4.2
Sep 20, 2026
Merged

yiguolei merged 2 commits into
apache:branch-4.2from
CalvinKirs:backport-63379-branch-4.2

Conversation

@CalvinKirs

Copy link
Copy Markdown
Member

apupier and others added 2 commits September 18, 2026 14:09
macos-13 is no more available

https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

it is causing a lot of PR jobs to wait in queue for a long time when
there is a PR on 4.2 branch

it has been already upgraded on main branch

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Signed-off-by: Aurélien Pupier <[email protected]>
…he#63379)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: PartitionTopN still passed partition key expressions to
BE sort info when the same original slot also appeared in window order
keys. These keys are constant inside each partition, so keeping them
causes redundant comparison work and noisy explain output, especially
after slot remapping changes ExprIds.

``` before
|   1:VPartitionTopN(42)                                                     |
|   |  functions: rank                                                       |
|   |  partition by: c_region[#3]                                            |
|   |  order by: c_region[#5] ASC, c_custkey[#6] ASC                         |
|   |  has global limit: false                                               |
|   |  partition limit: 1                                                    |
|   |  partition topn phase: TWO_PHASE_LOCAL_PTOPN                           |
|   |  distribute expr lists: c_custkey[#4]
```

after

```
|   1:VPartitionTopN(42)                                                     |
|   |  functions: rank                                                       |
|   |  partition by: c_region[#3]                                            |
|   |  order by: c_custkey[#6] ASC                                           |
|   |  has global limit: false                                               |
|   |  partition limit: 1                                                    |
|   |  partition topn phase: TWO_PHASE_LOCAL_PTOPN                           |
|   |  distribute expr lists: c_custkey[#4]                                  |
|   |

```

This PR also adds a guard for self-join cases where the partition key
and order key come from the same original table column, but from
different join inputs.

  Example:

  ```sql
  EXPLAIN
  SELECT *
  FROM (
      SELECT
          l.c2 AS lc2,
          r.c2 AS rc2,
          row_number() OVER (
              PARTITION BY l.c2
              ORDER BY r.c2
          ) AS rn
      FROM test_global_partition_topn_plan l
      JOIN test_global_partition_topn_plan r
        ON l.c1 = r.c1
  ) tmp
  WHERE rn <= 1;
```
  Here l.c2 and r.c2 both come from test_global_partition_topn_plan.c2, but they are different slots after the self join. PARTITION BY l.c2 only guarantees that
  the left-side c2 is constant inside each partition. It does not guarantee that the right-side r.c2 is constant, so ORDER BY r.c2 must not be pruned.

  The expected EXPLAIN should still keep the order key in VPartitionTopN:
```
  1:VPartitionTopN
  |  functions: row_number
  |  partition by: c2[#0]
  |  order by: c2[#3] ASC
  |  has global limit: false
  |  partition limit: 1
  |  partition topn phase: TWO_PHASE_LOCAL_PTOPN

The different slot ids, c2[#0] and c2[#3], show that the partition key
and order key are different join outputs. If the order key were pruned
here, the row
  selected by rn <= 1 could become nondeterministic or incorrect.
```

(cherry picked from commit 07bcd31)
@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.21% (16/19) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit 0083a3d into apache:branch-4.2 Sep 20, 2026
28 of 30 checks passed
@CalvinKirs
CalvinKirs deleted the backport-63379-branch-4.2 branch September 20, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants