Skip to content

branch-4.2:[fix](cost model) scale partitioned scalar aggregate cost by BE number #66354 - #68368

Merged
yiguolei merged 3 commits into
apache:branch-4.2from
feiniaofeiafei:codex/pick-66354-4.2
Sep 23, 2026
Merged

yiguolei merged 3 commits into
apache:branch-4.2from
feiniaofeiafei:codex/pick-66354-4.2

Conversation

@feiniaofeiafei

@feiniaofeiafei feiniaofeiafei commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

picked from #66354

foxtail463 and others added 3 commits September 22, 2026 11:03
…en partitioned (apache#66354)

Problem Summary:
A scalar count(distinct) runs its global aggregate hash-shuffled by the
distinct
key (non-empty partitionExpressions), i.e. in parallel across BEs. But
the cost
model set the parallelism factor from group-by keys alone, so with no
group by it
used factor = 1, costing the aggregate as single-point. This
overestimated the
parallel plan and made CBO reject it.

Solution:
Treat a global aggregate as partitioned when it has group by keys or
non-empty
partitionExpressions, and scale its cost by beNumber accordingly.

---------

Co-authored-by: yangtao555 <[email protected]>
(cherry picked from commit 051b9ac)
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei feiniaofeiafei changed the title [branch-4.2][fix](cost model) scale partitioned scalar aggregate cost by BE number (#66354) branch-4.2:[fix](cost model) scale partitioned scalar aggregate cost by BE number (#66354) Sep 22, 2026
@feiniaofeiafei feiniaofeiafei changed the title branch-4.2:[fix](cost model) scale partitioned scalar aggregate cost by BE number (#66354) branch-4.2:[fix](cost model) scale partitioned scalar aggregate cost by BE number #66354 Sep 22, 2026
@feiniaofeiafei

Copy link
Copy Markdown
Collaborator Author

run buildall

@yiguolei
yiguolei merged commit b58e421 into apache:branch-4.2 Sep 23, 2026
48 of 50 checks passed
englefly added a commit to englefly/incubator-doris that referenced this pull request Sep 23, 2026
…tputs after rebase

Rebasing onto the current branch-4.2 brought in
  b58e421 [fix](cost model) scale partitioned scalar aggregate cost by BE number (apache#68368)
which changes the distributed plan of tpcds q16/q94/q95. Those 25 baselines were
touched by both sides, so the rebase conflicted on 22 of them and the merge left
them without the table alias.

Regenerated the whole shape_check tree with -forceGenOut on the rebased tree
(1001 suites, 0 failures) so the baselines carry both the new cost-model plan and
the table alias. Only these 25 files changed: 75 insertions / 75 deletions, every
changed line a pure alias insertion.

Verification:
  - re-run without -forceGenOut: shape_check 1001 suites, 0 failures
  - the 22 suites from the previous commit: 0 failures (cost model change does not
    affect them)
  - alias-stripped, all 998 shape_check .out are byte-identical to
    origin/branch-4.2's committed baselines, i.e. the only delta is the alias
englefly added a commit to englefly/incubator-doris that referenced this pull request Sep 23, 2026
…tputs after rebase

Rebasing onto the current branch-4.2 brought in
  b58e421 [fix](cost model) scale partitioned scalar aggregate cost by BE number (apache#68368)
which changes the distributed plan of tpcds q16/q94/q95. Those 25 baselines were
touched by both sides, so the rebase conflicted on 22 of them and the merge left
them without the table alias.

Regenerated the whole shape_check tree with -forceGenOut on the rebased tree
(1001 suites, 0 failures) so the baselines carry both the new cost-model plan and
the table alias. Only these 25 files changed: 75 insertions / 75 deletions, every
changed line a pure alias insertion.

Verification:
  - re-run without -forceGenOut: shape_check 1001 suites, 0 failures
  - the 22 suites from the previous commit: 0 failures (cost model change does not
    affect them)
  - alias-stripped, all 998 shape_check .out are byte-identical to
    origin/branch-4.2's committed baselines, i.e. the only delta is the alias
yiguolei pushed a commit that referenced this pull request Sep 29, 2026
…efore dereference (#67438) (#68260) (#68526)

#68062

---

This PR also carries a second commit that regenerates three dphyper
shape expectation files on branch-4.2, unrelated to the pick itself:

cc @feiniaofeiafei @starocean999 — #68368 (pick of #66354, scaling
partitioned scalar aggregate cost by BE number) and #68204 (pick of
#64559, whose dphyper `.out` files were generated on branch-4.2 before
#68368 merged) landed within two hours of each other on 2026-09-23. Each
was green against its own pre-merge base, but combined,
`shape_check/tpcds_sf1000/dphyper` query16/94/95 still expect the
4-phase DISTINCT aggregate plan
(`hashAgg[DISTINCT_GLOBAL]`/`DISTINCT_LOCAL`) while the updated cost
model now chooses the 2-phase multi_distinct plan (`hashAgg[GLOBAL]`)
for these scalar `count(distinct ...)` queries, so the three cases fail
on every branch-4.2 PR since then (the same queries in the other shape
directories were already updated by #68368 itself, and master carries
the GLOBAL version of the dphyper files). The second commit here
regenerates the three `.out` files on current branch-4.2 via
`run-regression-test.sh -d shape_check/tpcds_sf1000/dphyper -s
query16,query94,query95 -forceGenOut`; the three suites pass locally
afterwards. Please shout if the expected fix is different.
englefly added a commit to englefly/incubator-doris that referenced this pull request Oct 8, 2026
…tputs after rebase

Rebasing onto the current branch-4.2 brought in
  b58e421 [fix](cost model) scale partitioned scalar aggregate cost by BE number (apache#68368)
which changes the distributed plan of tpcds q16/q94/q95. Those 25 baselines were
touched by both sides, so the rebase conflicted on 22 of them and the merge left
them without the table alias.

Regenerated the whole shape_check tree with -forceGenOut on the rebased tree
(1001 suites, 0 failures) so the baselines carry both the new cost-model plan and
the table alias. Only these 25 files changed: 75 insertions / 75 deletions, every
changed line a pure alias insertion.

Verification:
  - re-run without -forceGenOut: shape_check 1001 suites, 0 failures
  - the 22 suites from the previous commit: 0 failures (cost model change does not
    affect them)
  - alias-stripped, all 998 shape_check .out are byte-identical to
    origin/branch-4.2's committed baselines, i.e. the only delta is the alias
morningman added a commit that referenced this pull request Nov 5, 2026
…0163)

### What problem does this PR solve?

Issue Number: None

Related PR: #68206, #68636, #68368

Problem Summary:

**In short.** Three branch-4.2 backports of master fixes each added a
test: #68206 (of #61342), #68636 (of #57006) and #68368 (of #66354).
Master has the three fixes but none of the tests. This PR ports them.
Two guard a fix that has no test on master; the third keeps the
ClickBench `query1` shape case from failing whenever the exact row count
of `hits` is cached. The PR changes tests only.

**Background, and what each test covers.**

- *Old primary key metadata (#61342, #68206).* A `PrimaryKeyConstraint`
written before its foreign table collections existed has no `ft`, `ftn`
or `ftni` in its JSON. Gson builds the object without its field
initializers, so these stay null unless `gsonPostProcess` fills them in,
and an FE that replays the journal from the start (without an image, or
with `metadata_failure_recovery`) failed with a NullPointerException.
#61342 added the null checks without a test; #68206 added one on
branch-4.2 for `ft`, the only collection there. The new
`PrimaryKeyConstraintTest` removes all three from a serialized
constraint, loads it, and adds, renames and removes a foreign table on
it.
- *Shadow index columns (#57006, #68636).* During a schema change the
new schema names a changed column `__doris_shadow_<name>`.
`Index.getColumnUniqueIds` compares the index's column names with the
schema's after `Column.removeNamePrefix`, so the new tablet metadata
keeps the index column's unique id. #57006 has no test; #68636 added one
to `IndexTest`, ported here to JUnit 5.
- *ClickBench query1 shape (#66354, #68368).*
`shape_check/clickbench/query1` checks the plan of `SELECT COUNT(*) FROM
hits`. Once `SimpleAggCacheMgr` has cached the exact row count of
`hits`, `REWRITE_SIMPLE_AGG_TO_CONSTANT` turns the count into a constant
and the shape into `PhysicalOneRowRelation`, so the case fails whenever
that cache is warm when it runs. It did on P0 for #69830 (build 1087320:
expected `--hashAgg[GLOBAL]`, real `--PhysicalOneRowRelation`), a PR
that does not touch the planner. On branch-4.2, #68368 disables the rule
in this case together with the cost model change it backports, which
master has from #66354. This PR disables it the same way; the expected
output does not change.

**Results.**

| Change | Test | Result |
|---|---|---|
| none | `PrimaryKeyConstraintTest` (new), `IndexTest` (one new case),
`ConstraintManagerTest`, `ConstraintPersistTest` | pass |
| `gsonPostProcess` without its null checks | `PrimaryKeyConstraintTest`
| fails: NullPointerException while loading |
| `getColumnUniqueIds` without `removeNamePrefix` |
`IndexTest#testGetColumnUniqueIdsForShadowColumn` | fails: `expected
[107] but was []` |
| `query1` as on master, row count of `hits` cached (local cluster) |
`shape_check/clickbench/query1` | fails: `PhysicalOneRowRelation` |
| `query1` with this PR, same cache | `shape_check/clickbench/query1` |
passes |

On the local cluster, `explain shape plan SELECT COUNT(*) FROM hits`
gives `hashAgg[GLOBAL]` the first time and `PhysicalOneRowRelation`
three seconds later, once the asynchronous row count load has finished;
with the rule disabled it stays `hashAgg[GLOBAL]`.

Co-authored-by: minghong <[email protected]>
Co-authored-by: Jack <[email protected]>
Co-authored-by: feiniaofeiafei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants