Repository navigation
branch-4.2: [improvement](fe) Prune partition keys from partition topn sort #63379 - #68171
Merged
yiguolei merged 2 commits intoSep 20, 2026
Merged
Conversation
macos-13 is no more available https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/ it is causing a lot of PR jobs to wait in queue for a long time when there is a PR on 4.2 branch it has been already upgraded on main branch ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> Signed-off-by: Aurélien Pupier <[email protected]>
…he#63379) Issue Number: close #xxx Related PR: #xxx Problem Summary: PartitionTopN still passed partition key expressions to BE sort info when the same original slot also appeared in window order keys. These keys are constant inside each partition, so keeping them causes redundant comparison work and noisy explain output, especially after slot remapping changes ExprIds. ``` before | 1:VPartitionTopN(42) | | | functions: rank | | | partition by: c_region[#3] | | | order by: c_region[#5] ASC, c_custkey[#6] ASC | | | has global limit: false | | | partition limit: 1 | | | partition topn phase: TWO_PHASE_LOCAL_PTOPN | | | distribute expr lists: c_custkey[#4] ``` after ``` | 1:VPartitionTopN(42) | | | functions: rank | | | partition by: c_region[#3] | | | order by: c_custkey[#6] ASC | | | has global limit: false | | | partition limit: 1 | | | partition topn phase: TWO_PHASE_LOCAL_PTOPN | | | distribute expr lists: c_custkey[#4] | | | ``` This PR also adds a guard for self-join cases where the partition key and order key come from the same original table column, but from different join inputs. Example: ```sql EXPLAIN SELECT * FROM ( SELECT l.c2 AS lc2, r.c2 AS rc2, row_number() OVER ( PARTITION BY l.c2 ORDER BY r.c2 ) AS rn FROM test_global_partition_topn_plan l JOIN test_global_partition_topn_plan r ON l.c1 = r.c1 ) tmp WHERE rn <= 1; ``` Here l.c2 and r.c2 both come from test_global_partition_topn_plan.c2, but they are different slots after the self join. PARTITION BY l.c2 only guarantees that the left-side c2 is constant inside each partition. It does not guarantee that the right-side r.c2 is constant, so ORDER BY r.c2 must not be pruned. The expected EXPLAIN should still keep the order key in VPartitionTopN: ``` 1:VPartitionTopN | functions: row_number | partition by: c2[#0] | order by: c2[#3] ASC | has global limit: false | partition limit: 1 | partition topn phase: TWO_PHASE_LOCAL_PTOPN The different slot ids, c2[#0] and c2[#3], show that the partition key and order key are different join outputs. If the order key were pruned here, the row selected by rn <= 1 could become nondeterministic or incorrect. ``` (cherry picked from commit 07bcd31)
Member
Author
|
run buildall |
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
FE UT Coverage ReportIncrement line coverage |
yiguolei
force-pushed
the
branch-4.2
branch
from
September 20, 2026 02:23
5799884 to
e3ad599
Compare
yiguolei
approved these changes
Sep 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#63379