Repository navigation
[fix](constraint) Fix NPE in PrimaryKeyConstraint when loading old metadata - #61342
Merged
Merged
Conversation
…tadata When FE starts with -r (metadata_failure_recovery) or without a checkpoint image, it replays all journal logs from the beginning. Old journal entries may contain PrimaryKeyConstraint objects that were serialized before the fields foreignTables, foreignTableNameStrs, or foreignTableInfos were added to the class. Gson uses Unsafe to instantiate objects during deserialization, bypassing the constructor and its field initializers. If a field is absent from the JSON, Gson leaves it as null rather than using the declared initializer (e.g. `= new HashSet<>()`). Calling .isEmpty() or any method on a null collection then throws a NullPointerException. Fix: remove `final` from the three collection fields (so they can be reassigned in gsonPostProcess) and add null-guards at the top of gsonPostProcess() to initialize them to empty collections when absent. Impact on -r mode: when FE is started with --metadata_failure_recovery it replays every journal from journal #1, encountering the oldest serialized PrimaryKeyConstraint objects. Without this fix, FE always crashes during -r recovery if any PrimaryKeyConstraint was persisted before these fields existed. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26661 ms |
TPC-DS: Total hot run time: 168375 ms |
morrySnow
approved these changes
Mar 16, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
Author
|
run nonConcurrent |
starocean999
approved these changes
Mar 16, 2026
yiguolei
pushed a commit
that referenced
this pull request
Sep 20, 2026
…ding old metadata (#68206) Cherry-pick of #61342 to branch-4.2 [fix](constraint) Fix NPE in PrimaryKeyConstraint when loading old metadata On branch-4.2 the class only has the `foreignTables` collection (the newer `foreignTableNameStrs`/`foreignTableInfos` fields come from a later refactor that is not part of this branch), so only that field is guarded. A unit test covering deserialization of metadata without the `ft` field was added. --------- Signed-off-by: Aurélien Pupier <[email protected]> Co-authored-by: Aurélien Pupier <[email protected]>
4 of 16 tasks
morningman
added a commit
that referenced
this pull request
Nov 5, 2026
…0163) ### What problem does this PR solve? Issue Number: None Related PR: #68206, #68636, #68368 Problem Summary: **In short.** Three branch-4.2 backports of master fixes each added a test: #68206 (of #61342), #68636 (of #57006) and #68368 (of #66354). Master has the three fixes but none of the tests. This PR ports them. Two guard a fix that has no test on master; the third keeps the ClickBench `query1` shape case from failing whenever the exact row count of `hits` is cached. The PR changes tests only. **Background, and what each test covers.** - *Old primary key metadata (#61342, #68206).* A `PrimaryKeyConstraint` written before its foreign table collections existed has no `ft`, `ftn` or `ftni` in its JSON. Gson builds the object without its field initializers, so these stay null unless `gsonPostProcess` fills them in, and an FE that replays the journal from the start (without an image, or with `metadata_failure_recovery`) failed with a NullPointerException. #61342 added the null checks without a test; #68206 added one on branch-4.2 for `ft`, the only collection there. The new `PrimaryKeyConstraintTest` removes all three from a serialized constraint, loads it, and adds, renames and removes a foreign table on it. - *Shadow index columns (#57006, #68636).* During a schema change the new schema names a changed column `__doris_shadow_<name>`. `Index.getColumnUniqueIds` compares the index's column names with the schema's after `Column.removeNamePrefix`, so the new tablet metadata keeps the index column's unique id. #57006 has no test; #68636 added one to `IndexTest`, ported here to JUnit 5. - *ClickBench query1 shape (#66354, #68368).* `shape_check/clickbench/query1` checks the plan of `SELECT COUNT(*) FROM hits`. Once `SimpleAggCacheMgr` has cached the exact row count of `hits`, `REWRITE_SIMPLE_AGG_TO_CONSTANT` turns the count into a constant and the shape into `PhysicalOneRowRelation`, so the case fails whenever that cache is warm when it runs. It did on P0 for #69830 (build 1087320: expected `--hashAgg[GLOBAL]`, real `--PhysicalOneRowRelation`), a PR that does not touch the planner. On branch-4.2, #68368 disables the rule in this case together with the cost model change it backports, which master has from #66354. This PR disables it the same way; the expected output does not change. **Results.** | Change | Test | Result | |---|---|---| | none | `PrimaryKeyConstraintTest` (new), `IndexTest` (one new case), `ConstraintManagerTest`, `ConstraintPersistTest` | pass | | `gsonPostProcess` without its null checks | `PrimaryKeyConstraintTest` | fails: NullPointerException while loading | | `getColumnUniqueIds` without `removeNamePrefix` | `IndexTest#testGetColumnUniqueIdsForShadowColumn` | fails: `expected [107] but was []` | | `query1` as on master, row count of `hits` cached (local cluster) | `shape_check/clickbench/query1` | fails: `PhysicalOneRowRelation` | | `query1` with this PR, same cache | `shape_check/clickbench/query1` | passes | On the local cluster, `explain shape plan SELECT COUNT(*) FROM hits` gives `hashAgg[GLOBAL]` the first time and `PhysicalOneRowRelation` three seconds later, once the asynchronous row count load has finished; with the rule disabled it stays `hashAgg[GLOBAL]`. Co-authored-by: minghong <[email protected]> Co-authored-by: Jack <[email protected]> Co-authored-by: feiniaofeiafei <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
When FE starts with -r (metadata_failure_recovery) or without a checkpoint image, it replays all journal logs from the beginning. Old journal entries may contain PrimaryKeyConstraint objects that were serialized before the fields foreignTables, foreignTableNameStrs, or foreignTableInfos were added to the class.
Gson uses Unsafe to instantiate objects during deserialization, bypassing the constructor and its field initializers. If a field is absent from the JSON, Gson leaves it as null rather than using the declared initializer (e.g.
= new HashSet<>()). Calling .isEmpty() or any method on a null collection then throws a NullPointerException.Fix: remove
finalfrom the three collection fields (so they can be reassigned in gsonPostProcess) and add null-guards at the top of gsonPostProcess() to initialize them to empty collections when absent.Impact on -r mode: when FE is started with --metadata_failure_recovery it replays every journal from journal #1, encountering the oldest serialized PrimaryKeyConstraint objects. Without this fix, FE always crashes during -r recovery if any PrimaryKeyConstraint was persisted before these fields existed.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)