Repository navigation
branch-4.2:[fix](cost model) scale partitioned scalar aggregate cost by BE number #66354 - #68368
Merged
Merged
Conversation
…en partitioned (apache#66354) Problem Summary: A scalar count(distinct) runs its global aggregate hash-shuffled by the distinct key (non-empty partitionExpressions), i.e. in parallel across BEs. But the cost model set the parallelism factor from group-by keys alone, so with no group by it used factor = 1, costing the aggregate as single-point. This overestimated the parallel plan and made CBO reject it. Solution: Treat a global aggregate as partitioned when it has group by keys or non-empty partitionExpressions, and scale its cost by beNumber accordingly. --------- Co-authored-by: yangtao555 <[email protected]> (cherry picked from commit 051b9ac)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
run buildall |
englefly
added a commit
to englefly/incubator-doris
that referenced
this pull request
Sep 23, 2026
…tputs after rebase Rebasing onto the current branch-4.2 brought in b58e421 [fix](cost model) scale partitioned scalar aggregate cost by BE number (apache#68368) which changes the distributed plan of tpcds q16/q94/q95. Those 25 baselines were touched by both sides, so the rebase conflicted on 22 of them and the merge left them without the table alias. Regenerated the whole shape_check tree with -forceGenOut on the rebased tree (1001 suites, 0 failures) so the baselines carry both the new cost-model plan and the table alias. Only these 25 files changed: 75 insertions / 75 deletions, every changed line a pure alias insertion. Verification: - re-run without -forceGenOut: shape_check 1001 suites, 0 failures - the 22 suites from the previous commit: 0 failures (cost model change does not affect them) - alias-stripped, all 998 shape_check .out are byte-identical to origin/branch-4.2's committed baselines, i.e. the only delta is the alias
englefly
added a commit
to englefly/incubator-doris
that referenced
this pull request
Sep 23, 2026
…tputs after rebase Rebasing onto the current branch-4.2 brought in b58e421 [fix](cost model) scale partitioned scalar aggregate cost by BE number (apache#68368) which changes the distributed plan of tpcds q16/q94/q95. Those 25 baselines were touched by both sides, so the rebase conflicted on 22 of them and the merge left them without the table alias. Regenerated the whole shape_check tree with -forceGenOut on the rebased tree (1001 suites, 0 failures) so the baselines carry both the new cost-model plan and the table alias. Only these 25 files changed: 75 insertions / 75 deletions, every changed line a pure alias insertion. Verification: - re-run without -forceGenOut: shape_check 1001 suites, 0 failures - the 22 suites from the previous commit: 0 failures (cost model change does not affect them) - alias-stripped, all 998 shape_check .out are byte-identical to origin/branch-4.2's committed baselines, i.e. the only delta is the alias
yiguolei
pushed a commit
that referenced
this pull request
Sep 29, 2026
…efore dereference (#67438) (#68260) (#68526) #68062 --- This PR also carries a second commit that regenerates three dphyper shape expectation files on branch-4.2, unrelated to the pick itself: cc @feiniaofeiafei @starocean999 — #68368 (pick of #66354, scaling partitioned scalar aggregate cost by BE number) and #68204 (pick of #64559, whose dphyper `.out` files were generated on branch-4.2 before #68368 merged) landed within two hours of each other on 2026-09-23. Each was green against its own pre-merge base, but combined, `shape_check/tpcds_sf1000/dphyper` query16/94/95 still expect the 4-phase DISTINCT aggregate plan (`hashAgg[DISTINCT_GLOBAL]`/`DISTINCT_LOCAL`) while the updated cost model now chooses the 2-phase multi_distinct plan (`hashAgg[GLOBAL]`) for these scalar `count(distinct ...)` queries, so the three cases fail on every branch-4.2 PR since then (the same queries in the other shape directories were already updated by #68368 itself, and master carries the GLOBAL version of the dphyper files). The second commit here regenerates the three `.out` files on current branch-4.2 via `run-regression-test.sh -d shape_check/tpcds_sf1000/dphyper -s query16,query94,query95 -forceGenOut`; the three suites pass locally afterwards. Please shout if the expected fix is different.
englefly
added a commit
to englefly/incubator-doris
that referenced
this pull request
Oct 8, 2026
…tputs after rebase Rebasing onto the current branch-4.2 brought in b58e421 [fix](cost model) scale partitioned scalar aggregate cost by BE number (apache#68368) which changes the distributed plan of tpcds q16/q94/q95. Those 25 baselines were touched by both sides, so the rebase conflicted on 22 of them and the merge left them without the table alias. Regenerated the whole shape_check tree with -forceGenOut on the rebased tree (1001 suites, 0 failures) so the baselines carry both the new cost-model plan and the table alias. Only these 25 files changed: 75 insertions / 75 deletions, every changed line a pure alias insertion. Verification: - re-run without -forceGenOut: shape_check 1001 suites, 0 failures - the 22 suites from the previous commit: 0 failures (cost model change does not affect them) - alias-stripped, all 998 shape_check .out are byte-identical to origin/branch-4.2's committed baselines, i.e. the only delta is the alias
4 of 16 tasks
morningman
added a commit
that referenced
this pull request
Nov 5, 2026
…0163) ### What problem does this PR solve? Issue Number: None Related PR: #68206, #68636, #68368 Problem Summary: **In short.** Three branch-4.2 backports of master fixes each added a test: #68206 (of #61342), #68636 (of #57006) and #68368 (of #66354). Master has the three fixes but none of the tests. This PR ports them. Two guard a fix that has no test on master; the third keeps the ClickBench `query1` shape case from failing whenever the exact row count of `hits` is cached. The PR changes tests only. **Background, and what each test covers.** - *Old primary key metadata (#61342, #68206).* A `PrimaryKeyConstraint` written before its foreign table collections existed has no `ft`, `ftn` or `ftni` in its JSON. Gson builds the object without its field initializers, so these stay null unless `gsonPostProcess` fills them in, and an FE that replays the journal from the start (without an image, or with `metadata_failure_recovery`) failed with a NullPointerException. #61342 added the null checks without a test; #68206 added one on branch-4.2 for `ft`, the only collection there. The new `PrimaryKeyConstraintTest` removes all three from a serialized constraint, loads it, and adds, renames and removes a foreign table on it. - *Shadow index columns (#57006, #68636).* During a schema change the new schema names a changed column `__doris_shadow_<name>`. `Index.getColumnUniqueIds` compares the index's column names with the schema's after `Column.removeNamePrefix`, so the new tablet metadata keeps the index column's unique id. #57006 has no test; #68636 added one to `IndexTest`, ported here to JUnit 5. - *ClickBench query1 shape (#66354, #68368).* `shape_check/clickbench/query1` checks the plan of `SELECT COUNT(*) FROM hits`. Once `SimpleAggCacheMgr` has cached the exact row count of `hits`, `REWRITE_SIMPLE_AGG_TO_CONSTANT` turns the count into a constant and the shape into `PhysicalOneRowRelation`, so the case fails whenever that cache is warm when it runs. It did on P0 for #69830 (build 1087320: expected `--hashAgg[GLOBAL]`, real `--PhysicalOneRowRelation`), a PR that does not touch the planner. On branch-4.2, #68368 disables the rule in this case together with the cost model change it backports, which master has from #66354. This PR disables it the same way; the expected output does not change. **Results.** | Change | Test | Result | |---|---|---| | none | `PrimaryKeyConstraintTest` (new), `IndexTest` (one new case), `ConstraintManagerTest`, `ConstraintPersistTest` | pass | | `gsonPostProcess` without its null checks | `PrimaryKeyConstraintTest` | fails: NullPointerException while loading | | `getColumnUniqueIds` without `removeNamePrefix` | `IndexTest#testGetColumnUniqueIdsForShadowColumn` | fails: `expected [107] but was []` | | `query1` as on master, row count of `hits` cached (local cluster) | `shape_check/clickbench/query1` | fails: `PhysicalOneRowRelation` | | `query1` with this PR, same cache | `shape_check/clickbench/query1` | passes | On the local cluster, `explain shape plan SELECT COUNT(*) FROM hits` gives `hashAgg[GLOBAL]` the first time and `PhysicalOneRowRelation` three seconds later, once the asynchronous row count load has finished; with the rule disabled it stays `hashAgg[GLOBAL]`. Co-authored-by: minghong <[email protected]> Co-authored-by: Jack <[email protected]> Co-authored-by: feiniaofeiafei <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
picked from #66354