Repository navigation
[fix](inverted index) remove shadow prefix when matching index columns in schema change - #57006
Merged
Merged
Conversation
…s in schema change
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
|
PR approved by anyone and no changes requested. |
TPC-DS: Total hot run time: 189868 ms |
ClickBench: Total hot run time: 30.56 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
|
PR approved by at least one committer and no changes requested. |
Merged
4 of 12 tasks
yiguolei
pushed a commit
that referenced
this pull request
Sep 29, 2026
…chema change (#57006) (#68636) ### What problem does this PR solve? Issue Number: None Related PR: #57006 Problem Summary: During schema change, an indexed column may have a shadow prefix in the working schema. `Index.getColumnUniqueIds()` previously compared that temporary name with the original index column name, so the index column ID could be omitted from the new tablet metadata. This PR separately backports the master fix to `branch-4.2` and adds a focused FE test. It does not include the SNII or gram-index backports. Source commit: `5f0a8c68ded9933b1ea7df709cdf1322bad48dea` (#57006). Backport commits: `8fde58aa539` (cherry-pick with `-x`) and `197b6e46c9b` (test). | Source PR hunk | Result on `branch-4.2` | | --- | --- | | `fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java` `@@ -283,7 +283,9` | Ported: compare the original index column name with `Column.removeNamePrefix(column.getName())`, preserving case-insensitive matching. | `git range-diff` maps the source commit to `8fde58aa539`; the source code change is unchanged. The picked commit adds the `-x` provenance line, and the separate test commit has no source PR counterpart. Validation: - `./build.sh --fe`: passed; Checkstyle reported zero violations. - `./run-fe-ut.sh --run org.apache.doris.catalog.IndexTest`: 2 tests passed, including the new shadow-column case. - `test_index_ddl_fault_injection` on an isolated cloud cluster: 1 suite passed, 0 failed. The FE was built from this PR; the BE came from a local branch-4.2 SNII/gram backport build. The isolated BE had debug points enabled, as required by this existing suite. - Full PR English check against `origin/branch-4.2`: passed with no exceptions. - Full merged-tree FE Checkstyle preflight: passed. ### Release note None ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - [ ] No need to test or manual test - Behavior changed: - [ ] No. - [x] Yes. Index column IDs are retained when schema change uses shadow column names; no protocol or on-disk format changes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --------- Co-authored-by: zzzxl <[email protected]>
4 of 16 tasks
morningman
added a commit
that referenced
this pull request
Nov 5, 2026
…0163) ### What problem does this PR solve? Issue Number: None Related PR: #68206, #68636, #68368 Problem Summary: **In short.** Three branch-4.2 backports of master fixes each added a test: #68206 (of #61342), #68636 (of #57006) and #68368 (of #66354). Master has the three fixes but none of the tests. This PR ports them. Two guard a fix that has no test on master; the third keeps the ClickBench `query1` shape case from failing whenever the exact row count of `hits` is cached. The PR changes tests only. **Background, and what each test covers.** - *Old primary key metadata (#61342, #68206).* A `PrimaryKeyConstraint` written before its foreign table collections existed has no `ft`, `ftn` or `ftni` in its JSON. Gson builds the object without its field initializers, so these stay null unless `gsonPostProcess` fills them in, and an FE that replays the journal from the start (without an image, or with `metadata_failure_recovery`) failed with a NullPointerException. #61342 added the null checks without a test; #68206 added one on branch-4.2 for `ft`, the only collection there. The new `PrimaryKeyConstraintTest` removes all three from a serialized constraint, loads it, and adds, renames and removes a foreign table on it. - *Shadow index columns (#57006, #68636).* During a schema change the new schema names a changed column `__doris_shadow_<name>`. `Index.getColumnUniqueIds` compares the index's column names with the schema's after `Column.removeNamePrefix`, so the new tablet metadata keeps the index column's unique id. #57006 has no test; #68636 added one to `IndexTest`, ported here to JUnit 5. - *ClickBench query1 shape (#66354, #68368).* `shape_check/clickbench/query1` checks the plan of `SELECT COUNT(*) FROM hits`. Once `SimpleAggCacheMgr` has cached the exact row count of `hits`, `REWRITE_SIMPLE_AGG_TO_CONSTANT` turns the count into a constant and the shape into `PhysicalOneRowRelation`, so the case fails whenever that cache is warm when it runs. It did on P0 for #69830 (build 1087320: expected `--hashAgg[GLOBAL]`, real `--PhysicalOneRowRelation`), a PR that does not touch the planner. On branch-4.2, #68368 disables the rule in this case together with the cost model change it backports, which master has from #66354. This PR disables it the same way; the expected output does not change. **Results.** | Change | Test | Result | |---|---|---| | none | `PrimaryKeyConstraintTest` (new), `IndexTest` (one new case), `ConstraintManagerTest`, `ConstraintPersistTest` | pass | | `gsonPostProcess` without its null checks | `PrimaryKeyConstraintTest` | fails: NullPointerException while loading | | `getColumnUniqueIds` without `removeNamePrefix` | `IndexTest#testGetColumnUniqueIdsForShadowColumn` | fails: `expected [107] but was []` | | `query1` as on master, row count of `hits` cached (local cluster) | `shape_check/clickbench/query1` | fails: `PhysicalOneRowRelation` | | `query1` with this PR, same cache | `shape_check/clickbench/query1` | passes | On the local cluster, `explain shape plan SELECT COUNT(*) FROM hits` gives `hashAgg[GLOBAL]` the first time and `PhysicalOneRowRelation` three seconds later, once the asynchronous row count load has finished; with the rule disabled it stays `hashAgg[GLOBAL]`. Co-authored-by: minghong <[email protected]> Co-authored-by: Jack <[email protected]> Co-authored-by: feiniaofeiafei <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
test_index_ddl_fault_injection.groovy
Fixes an issue where index columns fail to match during schema change operations due to shadow column prefix mismatch.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)