Repository navigation
[fix](cost model) scale group-by-less global agg cost by BE number when partitioned - #66354
Merged
Merged
Conversation
foxtail463
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
August 1, 2026 17:20
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
foxtail463
force-pushed
the
fix/disinct-no-group-by
branch
from
August 1, 2026 17:21
dd90308 to
19f7f33
Compare
Contributor
TPC-H: Total hot run time: 28674 ms |
Contributor
TPC-DS: Total hot run time: 168752 ms |
Contributor
ClickBench: Total hot run time: 23.9 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
3 similar comments
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 28709 ms |
Contributor
TPC-DS: Total hot run time: 169538 ms |
Contributor
ClickBench: Total hot run time: 23.83 s |
morrySnow
changed the title
Contributor
|
/review |
Contributor
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
FE UT Coverage ReportIncrement line coverage |
feiniaofeiafei
approved these changes
Aug 3, 2026
Contributor
ClickBench: Total hot run time: 24 s |
Contributor
Author
|
@englefly 改好了辛苦看下 |
englefly
approved these changes
Aug 5, 2026
Contributor
|
run buildall |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
Author
|
run compile |
Contributor
Author
|
run fe_ut |
Contributor
Author
|
run feut |
Contributor
FE Regression Coverage ReportIncrement line coverage |
2 similar comments
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 28653 ms |
Contributor
TPC-DS: Total hot run time: 165841 ms |
Contributor
ClickBench: Total hot run time: 23.74 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…en partitioned (apache#66354) Problem Summary: A scalar count(distinct) runs its global aggregate hash-shuffled by the distinct key (non-empty partitionExpressions), i.e. in parallel across BEs. But the cost model set the parallelism factor from group-by keys alone, so with no group by it used factor = 1, costing the aggregate as single-point. This overestimated the parallel plan and made CBO reject it. Solution: Treat a global aggregate as partitioned when it has group by keys or non-empty partitionExpressions, and scale its cost by beNumber accordingly. --------- Co-authored-by: yangtao555 <[email protected]>
feiniaofeiafei
added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Sep 22, 2026
yiguolei
pushed a commit
that referenced
this pull request
Sep 23, 2026
…by BE number #66354 (#68368) picked from #66354 --------- Co-authored-by: foxtail463 <[email protected]> Co-authored-by: yangtao555 <[email protected]>
yiguolei
pushed a commit
that referenced
this pull request
Sep 29, 2026
…efore dereference (#67438) (#68260) (#68526) #68062 --- This PR also carries a second commit that regenerates three dphyper shape expectation files on branch-4.2, unrelated to the pick itself: cc @feiniaofeiafei @starocean999 — #68368 (pick of #66354, scaling partitioned scalar aggregate cost by BE number) and #68204 (pick of #64559, whose dphyper `.out` files were generated on branch-4.2 before #68368 merged) landed within two hours of each other on 2026-09-23. Each was green against its own pre-merge base, but combined, `shape_check/tpcds_sf1000/dphyper` query16/94/95 still expect the 4-phase DISTINCT aggregate plan (`hashAgg[DISTINCT_GLOBAL]`/`DISTINCT_LOCAL`) while the updated cost model now chooses the 2-phase multi_distinct plan (`hashAgg[GLOBAL]`) for these scalar `count(distinct ...)` queries, so the three cases fail on every branch-4.2 PR since then (the same queries in the other shape directories were already updated by #68368 itself, and master carries the GLOBAL version of the dphyper files). The second commit here regenerates the three `.out` files on current branch-4.2 via `run-regression-test.sh -d shape_check/tpcds_sf1000/dphyper -s query16,query94,query95 -forceGenOut`; the three suites pass locally afterwards. Please shout if the expected fix is different.
4 of 16 tasks
morningman
added a commit
that referenced
this pull request
Nov 5, 2026
…0163) ### What problem does this PR solve? Issue Number: None Related PR: #68206, #68636, #68368 Problem Summary: **In short.** Three branch-4.2 backports of master fixes each added a test: #68206 (of #61342), #68636 (of #57006) and #68368 (of #66354). Master has the three fixes but none of the tests. This PR ports them. Two guard a fix that has no test on master; the third keeps the ClickBench `query1` shape case from failing whenever the exact row count of `hits` is cached. The PR changes tests only. **Background, and what each test covers.** - *Old primary key metadata (#61342, #68206).* A `PrimaryKeyConstraint` written before its foreign table collections existed has no `ft`, `ftn` or `ftni` in its JSON. Gson builds the object without its field initializers, so these stay null unless `gsonPostProcess` fills them in, and an FE that replays the journal from the start (without an image, or with `metadata_failure_recovery`) failed with a NullPointerException. #61342 added the null checks without a test; #68206 added one on branch-4.2 for `ft`, the only collection there. The new `PrimaryKeyConstraintTest` removes all three from a serialized constraint, loads it, and adds, renames and removes a foreign table on it. - *Shadow index columns (#57006, #68636).* During a schema change the new schema names a changed column `__doris_shadow_<name>`. `Index.getColumnUniqueIds` compares the index's column names with the schema's after `Column.removeNamePrefix`, so the new tablet metadata keeps the index column's unique id. #57006 has no test; #68636 added one to `IndexTest`, ported here to JUnit 5. - *ClickBench query1 shape (#66354, #68368).* `shape_check/clickbench/query1` checks the plan of `SELECT COUNT(*) FROM hits`. Once `SimpleAggCacheMgr` has cached the exact row count of `hits`, `REWRITE_SIMPLE_AGG_TO_CONSTANT` turns the count into a constant and the shape into `PhysicalOneRowRelation`, so the case fails whenever that cache is warm when it runs. It did on P0 for #69830 (build 1087320: expected `--hashAgg[GLOBAL]`, real `--PhysicalOneRowRelation`), a PR that does not touch the planner. On branch-4.2, #68368 disables the rule in this case together with the cost model change it backports, which master has from #66354. This PR disables it the same way; the expected output does not change. **Results.** | Change | Test | Result | |---|---|---| | none | `PrimaryKeyConstraintTest` (new), `IndexTest` (one new case), `ConstraintManagerTest`, `ConstraintPersistTest` | pass | | `gsonPostProcess` without its null checks | `PrimaryKeyConstraintTest` | fails: NullPointerException while loading | | `getColumnUniqueIds` without `removeNamePrefix` | `IndexTest#testGetColumnUniqueIdsForShadowColumn` | fails: `expected [107] but was []` | | `query1` as on master, row count of `hits` cached (local cluster) | `shape_check/clickbench/query1` | fails: `PhysicalOneRowRelation` | | `query1` with this PR, same cache | `shape_check/clickbench/query1` | passes | On the local cluster, `explain shape plan SELECT COUNT(*) FROM hits` gives `hashAgg[GLOBAL]` the first time and `PhysicalOneRowRelation` three seconds later, once the asynchronous row count load has finished; with the rule disabled it stays `hashAgg[GLOBAL]`. Co-authored-by: minghong <[email protected]> Co-authored-by: Jack <[email protected]> Co-authored-by: feiniaofeiafei <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Summary:
A scalar count(distinct) runs its global aggregate hash-shuffled by the distinct
key (non-empty partitionExpressions), i.e. in parallel across BEs. But the cost
model set the parallelism factor from group-by keys alone, so with no group by it
used factor = 1, costing the aggregate as single-point. This overestimated the
parallel plan and made CBO reject it.
Solution:
Treat a global aggregate as partitioned when it has group by keys or non-empty
partitionExpressions, and scale its cost by beNumber accordingly.