Repository navigation
Commit 4a483d4
[feature](fe) Support MTMV refresh fallback strategy (apache#14)
* [feature](fe) Support MTMV refresh fallback strategy
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Add MTMV refresh method fallback semantics for AUTO, INCREMENTAL, PARTITIONS and COMPLETE refresh execution. Persist fallback policy, route scheduled and commit jobs through MV default strategy, and make IVM delta execution failures hard failures while allowing precheck/analyze fallback.
Support MTMV refresh fallback strategy syntax and execution semantics.
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.MTMVTaskTest,org.apache.doris.mtmv.MTMVJobManagerTest,org.apache.doris.job.extensions.mtmv.MTMVTaskContextCompatTest,org.apache.doris.nereids.trees.plans.commands.RefreshMTMVCommandTest,org.apache.doris.nereids.trees.plans.commands.info.RefreshMTMVInfoAnalyzeTest,org.apache.doris.nereids.trees.plans.commands.ExplainRefreshIvmCommandTest
- ./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.mtmv.AlterMTMVTest
- Behavior changed: Yes. MTMV refresh strategy now separates refresh method from fallback policy and uses method-specific fallback chains.
- Does this need documentation: Yes
* [doc](fe) Clarify MTMV refresh fallback code paths
Issue Number: N/A
Related PR: N/A
Problem Summary: Add explanatory comments for the MTMV refresh fallback implementation, covering persisted fallback metadata, task default-policy resolution, fallback attempt boundaries, and IVM execution hard-failure recovery semantics.
None
- Test: Unit Test
- git diff --check
- ./run-fe-ut.sh --run org.apache.doris.job.extensions.mtmv.MTMVTaskContextCompatTest,org.apache.doris.nereids.trees.plans.commands.info.RefreshMTMVInfoAnalyzeTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.mtmv.AlterMTMVTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest
- Behavior changed: No
- Does this need documentation: No
* [fix](fe) Address MTMV refresh fallback review comments
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Address review comments for MTMV refresh fallback. The fallback loop now routes IVM direct-to-complete recovery through the normal COMPLETE attempt by marking partition refresh disabled, partition refresh planning is wrapped inside the partition refresh helper, unused tryIvmFastPath was removed, non-IVM IVM attempts now fail directly, MTMVTaskContext uses short serialized field names while keeping old JSON aliases, and TODOs document additional IVM recovery classifications.
None
- Test: Unit Test
- git diff --check
- Attempted ./run-fe-ut.sh --run org.apache.doris.mtmv.MTMVTaskTest,org.apache.doris.job.extensions.mtmv.MTMVTaskContextCompatTest, but it failed during FE Core testCompile with missing generated/project classes such as Patterns, CascadesContext, LogicalPlan, and HMSExternalTable before running the selected tests.
- Behavior changed: No
- Does this need documentation: No
* [fix](fe) Preserve IVM execution failure reasons
Issue Number: None
Related PR: apache#14
Problem Summary: IVM execution-time guard failures for min/max boundary hits and non-deterministic row ids were collapsed into INCREMENTAL_EXECUTION_FAILED, losing the reason needed for diagnosis and follow-up recovery strategy. MTMVTaskContext also exposed multiple public constructors for each new policy field. This change classifies known runtime guard messages back to stable IvmFailureReason values while keeping execution failures as hard failures, and narrows MTMVTaskContext explicit policy construction to static factories.
None
- Test: Manual test
- Ran git diff --check.
- Tried ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.job.extensions.mtmv.MTMVTaskContextCompatTest,org.apache.doris.mtmv.MTMVTaskTest; blocked in fe-core testCompile by missing existing generated/main classes such as CascadesContext, LogicalPlan and GeneratedMemoPatterns.
- Tried cd fe && mvn -pl fe-core -am -DskipTests -Dcheckstyle.skip=true -Dmaven.build.cache.enabled=false compile; blocked in fe-core generate-patterns because PatternDescribableProcessor was not found.
- Behavior changed: No
- Does this need documentation: No
* [fix](fe) Fix MTMV incremental fallback attempts
Issue Number: close #xxx
Related PR: apache#14
Problem Summary: Non-IVM materialized views were allowed to accept manual INCREMENTAL FALLBACK requests, but the task attempt builder still scheduled an IVM attempt first. Since the IVM attempt now fails fast for non-IVM MVs, that path failed before reaching the intended PARTITIONS/COMPLETE fallback chain. This change skips the IVM attempt for non-IVM INCREMENTAL FALLBACK while preserving strict INCREMENTAL fail-fast behavior, and fixes FE checkstyle issues reported by CI.
None
- Test: Unit Test
- git diff --check
- Behavior changed: No
- Does this need documentation: No
* [fix](fe) Remove unused IVM test import
Issue Number: close #xxx
Related PR: apache#14
Problem Summary: FE CheckStyle failed on an unused import in IvmNormalizeMtmvTest when validating the PR merge ref. Remove the unused import so the full FE checkstyle job can proceed.
None
- Test: Unit Test
- Ran targeted FE CheckStyle for IvmNormalizeMtmvTest.
- Behavior changed: No
- Does this need documentation: No
* [fix](fe) Validate MTMV refresh policy compatibility
Issue Number: None
Related PR: apache#14
Problem Summary: Manual MTMV refresh analysis still relied on IVM capability only, so refresh-mode compatibility did not consistently match the MV refresh policy. INCREMENTAL fallback also skipped the IVM attempt for non-IVM MVs, which bypassed the intended fallback path. This change validates the requested refresh mode against the MV refresh method and partition mode, keeps the IVM attempt in the incremental fallback sequence, and extends unit coverage for COMPLETE, PARTITIONS, INCREMENTAL, and AUTO policy combinations.
Refresh MATERIALIZED VIEW now validates the requested refresh mode against the materialized view refresh policy.
- Test: Unit Test
- git diff --check
- ./run-fe-ut.sh --run org.apache.doris.mtmv.MTMVTaskTest,org.apache.doris.nereids.trees.plans.commands.info.RefreshMTMVInfoAnalyzeTest attempted, but failed before tests because fe-core compile could not resolve existing generated parser/proto classes, such as DorisParser.RefreshPolicyContext and Cloud.CreateMetaSyncPointRequest.
- Behavior changed: Yes (manual MTMV refresh mode validation now follows the MV refresh policy, and incremental fallback keeps the IVM attempt before falling back)
- Does this need documentation: No
---------
Co-authored-by: seawinde <[email protected]>1 parent d6af723 commit 4a483d4
25 files changed
Lines changed: 1042 additions & 236 deletions
File tree
- fe
- fe-core/src
- main/java/org/apache/doris
- job/extensions/mtmv
- mtmv
- ivm
- agg
- nereids
- parser
- trees/plans/commands/info
- test/java/org/apache/doris
- job/extensions/mtmv
- mtmv
- ivm
- nereids/trees/plans
- commands
- info
- fe-sql-parser/src/main/antlr4/org/apache/doris/nereids
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
131 | | - | |
| 131 | + | |
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
44 | 57 | | |
45 | 58 | | |
46 | 59 | | |
| 60 | + | |
47 | 61 | | |
48 | 62 | | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
| 63 | + | |
| 64 | + | |
55 | 65 | | |
56 | 66 | | |
57 | 67 | | |
58 | 68 | | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
| 69 | + | |
72 | 70 | | |
73 | 71 | | |
74 | 72 | | |
| |||
102 | 100 | | |
103 | 101 | | |
104 | 102 | | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
105 | 142 | | |
106 | 143 | | |
107 | 144 | | |
108 | 145 | | |
109 | 146 | | |
110 | 147 | | |
111 | 148 | | |
| 149 | + | |
| 150 | + | |
112 | 151 | | |
113 | 152 | | |
114 | 153 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | 50 | | |
52 | 51 | | |
53 | 52 | | |
| |||
68 | 67 | | |
69 | 68 | | |
70 | 69 | | |
71 | | - | |
| 70 | + | |
72 | 71 | | |
73 | 72 | | |
74 | 73 | | |
| |||
159 | 158 | | |
160 | 159 | | |
161 | 160 | | |
162 | | - | |
163 | | - | |
| 161 | + | |
| 162 | + | |
164 | 163 | | |
165 | 164 | | |
166 | 165 | | |
| |||
222 | 221 | | |
223 | 222 | | |
224 | 223 | | |
225 | | - | |
226 | | - | |
| 224 | + | |
227 | 225 | | |
228 | 226 | | |
229 | 227 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
35 | 41 | | |
36 | 42 | | |
37 | 43 | | |
| |||
41 | 47 | | |
42 | 48 | | |
43 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
44 | 57 | | |
45 | 58 | | |
| 59 | + | |
46 | 60 | | |
47 | 61 | | |
48 | 62 | | |
49 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
50 | 69 | | |
51 | 70 | | |
52 | 71 | | |
| |||
64 | 83 | | |
65 | 84 | | |
66 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
67 | 104 | | |
68 | 105 | | |
69 | 106 | | |
| |||
72 | 109 | | |
73 | 110 | | |
74 | 111 | | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
75 | 116 | | |
76 | 117 | | |
77 | 118 | | |
| |||
87 | 128 | | |
88 | 129 | | |
89 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
90 | 138 | | |
91 | 139 | | |
92 | 140 | | |
| |||
101 | 149 | | |
102 | 150 | | |
103 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
104 | 155 | | |
105 | 156 | | |
106 | 157 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
| 77 | + | |
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | 60 | | |
64 | 61 | | |
65 | 62 | | |
| |||
133 | 130 | | |
134 | 131 | | |
135 | 132 | | |
136 | | - | |
| 133 | + | |
137 | 134 | | |
138 | 135 | | |
139 | | - | |
| 136 | + | |
140 | 137 | | |
141 | 138 | | |
142 | 139 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
0 commit comments