Skip to content

Commit addf0c8

Browse files
authored
[feature](ivm) Add the per-partition refresh state and its journal channel (#68193)
### What problem does this PR solve? Trace issue: #65418 **This PR adds no behaviour of its own. It adds the state that the following PRs need, and the channel that persists it, so that they can be reviewed as logic alone.** An IVM materialized view has to invalidate the MV partitions that a base-table change really affected. A partition drop / truncate / replace / recover changes the base table through metadata and emits no row binlog, so the affected MV partitions must be rebuilt; today the only answer the MV has is "rebuild all of them", which throws away partitions that are still correct. Deciding per partition needs a per-partition answer to two questions: - which generation of data does this MV partition currently hold? - which generation must it hold? That pair is `MTMVPartitionState { refreshEpoch, latestEpoch }`, one entry per MV partition, keyed by partition name. `latestEpoch` is the requirement, `refreshEpoch` is the reality, and a partition whose requirement is ahead of its reality is dirty: it holds rows read before a change that left no binlog, so it can no longer be maintained incrementally and has to be rebuilt. The requirement has to survive a restart, because an invalidation that only lives in memory is lost the moment the FE restarts, and a partition that is then refreshed incrementally keeps the stale rows forever with no error anywhere. So this PR adds 1. the state itself (`MTMVPartitionState`, plus a copy helper for taking a detached snapshot), and 2. the channel that carries it into the journal and back: a field on the MV, its own field on the alter record, a dedicated alter op with its replay branch, and the replay handling of the task result. Nothing in the FE decides anything from the state yet, and nothing but a replay ever writes it, so every MV behaves exactly as before. That is deliberate: it makes this step independently mergeable and independently testable, which is what the PR that starts using the state needs underneath it. ### Scope | | | | --- | --- | | Adds | `MTMVPartitionState`, the `partitionStates` field, the alter record field, the new alter op and its replay branch | | Does not touch | any criterion, routing or invalidation decision; `IvmInfo`; the refresh path | | Field is shared, behaviour is not | the field sits on the MV, so both kinds of MV carry it; only an IVM MV ever populates it, and the journal of a non-IVM MV stays byte-for-byte what it was | | Compatibility | an image written before this PR has no such field and loads as an empty map; an ADD_TASK journal written before it applies nothing on replay instead of clearing what is there | The state is on the MV rather than inside `IvmInfo`, and the alter op is its own rather than riding on `ALTER_IVM_INFO`, whose branch only swaps the `IvmInfo` object. Both are structural: the same state is meant to serve a non-IVM MV later, and its journal payload must not be reconstructed as a side effect of replaying some other op. ### Key changes - Add `MTMVPartitionState`, a persisted `refreshEpoch` / `latestEpoch` pair keyed by MV partition name, and `MTMVPartitionState.copyOf` for taking a detached snapshot of a state map. A partition gets a new id on every refresh, so the name is the only identity it can have. - Add `MTMV.partitionStates` with its getter and its replay setter; `gsonPostProcess` initializes it, so an image written before the field existed and a non-IVM MV both load as an empty map. - Carry the state in the ADD_TASK payload under the same condition as `ivmInfo`, which keeps the journal of a non-IVM MV byte-for-byte unchanged, and apply it on replay only when the field is present, so an old journal applies nothing rather than clearing the state. - Add `MTMVAlterOpType.ALTER_PARTITION_STATES` and its `Alter.processAlterMTMV` branch.
1 parent d93137b commit addf0c8

7 files changed

Lines changed: 442 additions & 1 deletion

File tree

‎fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1344,6 +1344,10 @@ public void processAlterMTMV(AlterMTMV alterMTMV, boolean isReplay) {
13441344
// Live IVM changes are journaled inside MTMV; this branch applies the journal snapshot.
13451345
mtmv.alterIvmInfo(alterMTMV.getIvmInfo());
13461346
break;
1347+
case ALTER_PARTITION_STATES:
1348+
// Replay only, like ALTER_IVM_INFO: a live change journals itself from inside MTMV.
1349+
mtmv.alterPartitionStates(alterMTMV.getPartitionStates());
1350+
break;
13471351
default:
13481352
throw new RuntimeException("Unknown type value: " + alterMTMV.getOpType());
13491353
}

‎fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
import org.apache.doris.mtmv.MTMVPartitionExpander;
3939
import org.apache.doris.mtmv.MTMVPartitionInfo;
4040
import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType;
41+
import org.apache.doris.mtmv.MTMVPartitionState;
4142
import org.apache.doris.mtmv.MTMVPartitionUtil;
4243
import org.apache.doris.mtmv.MTMVPlanUtil;
4344
import org.apache.doris.mtmv.MTMVPropertyUtil;
@@ -69,6 +70,7 @@
6970
import org.apache.logging.log4j.Logger;
7071

7172
import java.io.IOException;
73+
import java.util.Collections;
7274
import java.util.List;
7375
import java.util.Map;
7476
import java.util.Map.Entry;
@@ -103,6 +105,21 @@ public class MTMV extends OlapTable {
103105
private MTMVRefreshSnapshot refreshSnapshot;
104106
@SerializedName("ii")
105107
private IvmInfo ivmInfo;
108+
/**
109+
* The refresh epoch of every MV partition, keyed by MV partition name.
110+
*
111+
* <p>Deliberately on MTMV rather than inside {@link IvmInfo}: the field is shared, the behaviour is
112+
* not. Both kinds of MV carry it, but only an IVM MV ever populates it -- alignment, invalidation,
113+
* the ADD_TASK payload and ALTER_PARTITION_STATES are all no-ops for a non-IVM MV, so for one an
114+
* empty map is the complete answer.
115+
*
116+
* <p>Null means the same thing -- no state -- and has three causes: an image written before the
117+
* field existed, a non-IVM MV, and a live MV that has not been aligned yet. Only
118+
* {@link #gsonPostProcess()} turns it into an empty map, on load; nothing else needs to, because a
119+
* reader treats the two the same.
120+
*/
121+
@SerializedName("pst")
122+
private Map<String, MTMVPartitionState> partitionStates;
106123
// Should update after every fresh, not persist
107124
// Cache with SessionVarGuardExpr: used when query session variables differ from MV creation variables
108125
private MTMVCache cacheWithGuard;
@@ -290,6 +307,11 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) {
290307
// Replay the final IVM state; ADD_TASK does not change schemaChangeVersion.
291308
ivmInfo = new IvmInfo(alterMTMV.getIvmInfo());
292309
}
310+
if (isReplay && alterMTMV.getPartitionStates() != null) {
311+
// A journal written before the field existed carries no state at all: leave the
312+
// partition states alone rather than clearing them.
313+
partitionStates = MTMVPartitionState.copyOf(alterMTMV.getPartitionStates());
314+
}
293315
if (task.getStatus() == TaskStatus.SUCCESS) {
294316
this.status.setState(MTMVState.NORMAL);
295317
this.status.setSchemaChangeDetail(null);
@@ -323,6 +345,10 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) {
323345
}
324346
if (ivmInfo.isEnableIvm()) {
325347
alterMTMV.setIvmInfo(ivmInfo);
348+
// Same condition as ivmInfo, so the journal of a non-IVM MV stays byte-for-byte what it
349+
// was. The map is null until the states are first aligned, and a payload without the
350+
// member means the same as one carrying an empty map.
351+
alterMTMV.setPartitionStates(partitionStates);
326352
}
327353
editLogItem = submitAlterLog(alterMTMV);
328354
} finally {
@@ -598,6 +624,46 @@ public void alterIvmInfo(IvmInfo ivmInfo) {
598624
}
599625
}
600626

627+
/**
628+
* A snapshot of the partition states, taken under the MV read lock.
629+
*
630+
* <p>The caller gets its own map and its own state objects, not the ones the MV owns: handing those
631+
* out would let a caller add or change an entry while {@link #addTaskResult} copies the same map
632+
* into the journal, and a replay that replaces the field would leave the caller's reference
633+
* pointing at state that is no longer the MV's. Changing the states is the MV's own job, under its
634+
* write lock.
635+
*
636+
* <p>A missing map -- an image written before the field existed, or a non-IVM MV -- reads as empty.
637+
*/
638+
public Map<String, MTMVPartitionState> getPartitionStates() {
639+
readMvLock();
640+
try {
641+
if (partitionStates == null) {
642+
return Collections.emptyMap();
643+
}
644+
return Collections.unmodifiableMap(MTMVPartitionState.copyOf(partitionStates));
645+
} finally {
646+
readMvUnlock();
647+
}
648+
}
649+
650+
// ALTER_PARTITION_STATES replay applies a detached snapshot here, mirroring alterIvmInfo(). Live
651+
// invalidation changes submit their journal from the mutating method instead.
652+
//
653+
// A payload without the member carries no state at all, which is not the same as an empty map that
654+
// says the states are now empty: leaving them alone is the only answer that cannot lose state.
655+
public void alterPartitionStates(Map<String, MTMVPartitionState> partitionStates) {
656+
if (partitionStates == null) {
657+
return;
658+
}
659+
writeMvLock();
660+
try {
661+
this.partitionStates = MTMVPartitionState.copyOf(partitionStates);
662+
} finally {
663+
writeMvUnlock();
664+
}
665+
}
666+
601667
public void invalidateIvmBaseline() {
602668
EditLogItem editLogItem;
603669
writeMvLock();
@@ -997,6 +1063,11 @@ public void gsonPostProcess() throws IOException {
9971063
if (ivmInfo == null) {
9981064
ivmInfo = new IvmInfo();
9991065
}
1066+
if (partitionStates == null) {
1067+
// An image written before the field existed deserializes it as null, and so does a non-IVM MV.
1068+
// Both mean "no state", so an empty map is the whole answer.
1069+
partitionStates = Maps.newLinkedHashMap();
1070+
}
10001071
if (refreshInfo != null && refreshInfo.getRefreshMethod() == null) {
10011072
LOG.warn("MTMV {} has unknown refresh method, marking as schema change", name);
10021073
status.setState(MTMVState.SCHEMA_CHANGE);

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVAlterOpType.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,6 @@ public enum MTMVAlterOpType {
2222
ALTER_STATUS,
2323
ALTER_PROPERTY,
2424
ADD_TASK,
25-
ALTER_IVM_INFO;
25+
ALTER_IVM_INFO,
26+
ALTER_PARTITION_STATES;
2627
}
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.mtmv;
19+
20+
import com.google.gson.annotations.SerializedName;
21+
22+
import java.util.LinkedHashMap;
23+
import java.util.Map;
24+
import java.util.Map.Entry;
25+
26+
/**
27+
* The per-partition refresh state of one MV partition.
28+
*
29+
* <p>{@code refreshEpoch} says which generation of data the partition currently holds, {@code
30+
* latestEpoch} says which generation it must hold. A partition whose {@code latestEpoch} is ahead of
31+
* its {@code refreshEpoch} is dirty: it holds rows read before a metadata-only change of a base table
32+
* (a dropped / truncated / replaced / recovered partition emits no row binlog), so those rows can no
33+
* longer be removed incrementally and the partition has to be rebuilt.
34+
*
35+
* <p>Keyed by MV partition name in {@code MTMV.partitionStates}. The name is deliberately the only
36+
* identity: an MV partition is rewritten by {@code INSERT OVERWRITE} on every refresh and gets a new
37+
* partition id each time, so an id would stop matching as soon as the partition is refreshed.
38+
*
39+
* <p>The two values are plain {@code long}s rather than atomics because this is a persisted DTO: it is
40+
* serialized into the alter journal, so it has to stay a plain bean.
41+
*/
42+
public class MTMVPartitionState {
43+
/** The generation of the data this MV partition currently holds; 0 means it was never refreshed. */
44+
@SerializedName("re")
45+
private long refreshEpoch;
46+
47+
/** The generation the data must reach; starts at 1 and grows on every invalidation. */
48+
@SerializedName("le")
49+
private long latestEpoch;
50+
51+
public MTMVPartitionState() {
52+
}
53+
54+
public MTMVPartitionState(long refreshEpoch, long latestEpoch) {
55+
this.refreshEpoch = refreshEpoch;
56+
this.latestEpoch = latestEpoch;
57+
}
58+
59+
public MTMVPartitionState(MTMVPartitionState other) {
60+
this.refreshEpoch = other.refreshEpoch;
61+
this.latestEpoch = other.latestEpoch;
62+
}
63+
64+
/**
65+
* Deep-copies a state map, or returns null for null.
66+
*
67+
* <p>The journal needs this on both sides. A payload is serialized by the journal thread, which
68+
* runs after the submitting thread released the MV lock, so a payload that shared state with the
69+
* live map could be written out half-mutated. The replay path goes through the same helper so that
70+
* both sides of the journal follow one rule instead of two.
71+
*/
72+
public static Map<String, MTMVPartitionState> copyOf(Map<String, MTMVPartitionState> states) {
73+
if (states == null) {
74+
return null;
75+
}
76+
Map<String, MTMVPartitionState> copy = new LinkedHashMap<>();
77+
for (Entry<String, MTMVPartitionState> entry : states.entrySet()) {
78+
copy.put(entry.getKey(), new MTMVPartitionState(entry.getValue()));
79+
}
80+
return copy;
81+
}
82+
83+
public long getRefreshEpoch() {
84+
return refreshEpoch;
85+
}
86+
87+
public void setRefreshEpoch(long refreshEpoch) {
88+
this.refreshEpoch = refreshEpoch;
89+
}
90+
91+
public long getLatestEpoch() {
92+
return latestEpoch;
93+
}
94+
95+
public void setLatestEpoch(long latestEpoch) {
96+
this.latestEpoch = latestEpoch;
97+
}
98+
}

‎fe/fe-core/src/main/java/org/apache/doris/persist/AlterMTMV.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
import org.apache.doris.common.io.Writable;
2323
import org.apache.doris.job.extensions.mtmv.MTMVTask;
2424
import org.apache.doris.mtmv.MTMVAlterOpType;
25+
import org.apache.doris.mtmv.MTMVPartitionState;
2526
import org.apache.doris.mtmv.MTMVRefreshInfo;
2627
import org.apache.doris.mtmv.MTMVRefreshPartitionSnapshot;
2728
import org.apache.doris.mtmv.MTMVRelation;
@@ -58,6 +59,8 @@ public class AlterMTMV implements Writable {
5859
private Map<String, MTMVRefreshPartitionSnapshot> partitionSnapshots;
5960
@SerializedName("ii")
6061
private IvmInfo ivmInfo;
62+
@SerializedName("pst")
63+
private Map<String, MTMVPartitionState> partitionStates;
6164

6265
public AlterMTMV(TableNameInfo mvName, MTMVRefreshInfo refreshInfo, MTMVAlterOpType opType) {
6366
this.mvName = Objects.requireNonNull(mvName, "require mvName object");
@@ -148,6 +151,14 @@ public void setIvmInfo(IvmInfo ivmInfo) {
148151
this.ivmInfo = ivmInfo == null ? null : new IvmInfo(ivmInfo);
149152
}
150153

154+
public Map<String, MTMVPartitionState> getPartitionStates() {
155+
return partitionStates;
156+
}
157+
158+
public void setPartitionStates(Map<String, MTMVPartitionState> partitionStates) {
159+
this.partitionStates = MTMVPartitionState.copyOf(partitionStates);
160+
}
161+
151162
@Override
152163
public String toString() {
153164
return "AlterMTMV{"

‎fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,22 @@
3232
import org.apache.doris.nereids.exceptions.AnalysisException;
3333
import org.apache.doris.persist.AlterMTMV;
3434
import org.apache.doris.persist.ReplaceTableOperationLog;
35+
import org.apache.doris.persist.gson.GsonUtils;
3536
import org.apache.doris.utframe.TestWithFeService;
3637

38+
import com.google.gson.JsonObject;
39+
import com.google.gson.JsonParser;
3740
import org.junit.jupiter.api.Assertions;
3841
import org.junit.jupiter.api.Test;
3942

43+
import java.io.ByteArrayInputStream;
44+
import java.io.ByteArrayOutputStream;
45+
import java.io.DataInputStream;
46+
import java.io.DataOutputStream;
47+
import java.io.IOException;
48+
import java.nio.charset.StandardCharsets;
49+
import java.util.LinkedHashMap;
50+
import java.util.Map;
4051
import java.util.Set;
4152

4253

@@ -403,6 +414,84 @@ public void testAlterIvmInfoPersistence() throws Exception {
403414
Assertions.assertEquals(schemaChangeVersion, mtmv.getSchemaChangeVersion());
404415
}
405416

417+
@Test
418+
public void testReplayAlterPartitionStates() throws Exception {
419+
Config.enable_table_stream = true;
420+
createDatabaseAndUse("alter_partition_states_test");
421+
createTable("CREATE TABLE alter_partition_states_test.states_base (k1 int, v1 int)\n"
422+
+ "DUPLICATE KEY(k1)\n"
423+
+ "DISTRIBUTED BY HASH(k1) BUCKETS 1\n"
424+
+ "PROPERTIES ('replication_num' = '1', 'binlog.enable' = 'true', 'binlog.format' = 'ROW')");
425+
createMvByNereids("CREATE MATERIALIZED VIEW states_mv\n"
426+
+ " BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL\n"
427+
+ " DISTRIBUTED BY RANDOM BUCKETS 2\n"
428+
+ " PROPERTIES ('replication_num' = '1')\n"
429+
+ " AS SELECT k1, v1 FROM states_base");
430+
431+
MTMV mtmv = (MTMV) Env.getCurrentInternalCatalog()
432+
.getDb("alter_partition_states_test").get()
433+
.getTableOrMetaException("states_mv");
434+
String partitionName = mtmv.getPartitionNames().iterator().next();
435+
TableNameInfo tableName = new TableNameInfo(mtmv.getQualifiedDbName(), mtmv.getName());
436+
437+
// A payload carrying state applies it. The live map keeps moving after the payload was taken,
438+
// and for a restart only the bytes in the journal matter, so both are driven here.
439+
MTMVPartitionState state = new MTMVPartitionState(0, 1);
440+
Map<String, MTMVPartitionState> states = new LinkedHashMap<>();
441+
states.put(partitionName, state);
442+
AlterMTMV withState = new AlterMTMV(tableName, MTMVAlterOpType.ALTER_PARTITION_STATES);
443+
withState.setPartitionStates(states);
444+
state.setLatestEpoch(7);
445+
// The MV starts without any state, so only the replayed payload can put it there.
446+
mtmv.alterPartitionStates(Map.of());
447+
448+
replayFromJournal(withState);
449+
450+
Map<String, MTMVPartitionState> applied = mtmv.getPartitionStates();
451+
Assertions.assertEquals(Set.of(partitionName), applied.keySet());
452+
Assertions.assertEquals(0, applied.get(partitionName).getRefreshEpoch());
453+
Assertions.assertEquals(1, applied.get(partitionName).getLatestEpoch());
454+
455+
// An explicit empty map empties the states.
456+
AlterMTMV empty = new AlterMTMV(tableName, MTMVAlterOpType.ALTER_PARTITION_STATES);
457+
empty.setPartitionStates(Map.of());
458+
Assertions.assertTrue(new String(journalBytes(empty), StandardCharsets.UTF_8).contains("\"pst\""),
459+
"the state member should be written under its serialized name");
460+
461+
replayFromJournal(empty);
462+
463+
Assertions.assertTrue(mtmv.getPartitionStates().isEmpty());
464+
465+
// A payload written before the member existed leaves the states alone instead of clearing them.
466+
mtmv.alterPartitionStates(Map.of(partitionName, new MTMVPartitionState(4, 6)));
467+
JsonObject legacy = JsonParser.parseString(GsonUtils.GSON.toJson(withState)).getAsJsonObject();
468+
Assertions.assertNotNull(legacy.remove("pst"));
469+
470+
replayFromJournal(GsonUtils.GSON.fromJson(legacy.toString(), AlterMTMV.class));
471+
472+
MTMVPartitionState kept = mtmv.getPartitionStates().get(partitionName);
473+
Assertions.assertEquals(4, kept.getRefreshEpoch());
474+
Assertions.assertEquals(6, kept.getLatestEpoch());
475+
}
476+
477+
/** The bytes the edit log writes for an alter record. */
478+
private static byte[] journalBytes(AlterMTMV alter) throws IOException {
479+
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
480+
try (DataOutputStream out = new DataOutputStream(bytes)) {
481+
alter.write(out);
482+
}
483+
return bytes.toByteArray();
484+
}
485+
486+
/** Replays an alter record the way a restart does: from what the journal wrote, not from memory. */
487+
private static void replayFromJournal(AlterMTMV alter) throws Exception {
488+
AlterMTMV replayed;
489+
try (DataInputStream in = new DataInputStream(new ByteArrayInputStream(journalBytes(alter)))) {
490+
replayed = AlterMTMV.read(in);
491+
}
492+
Env.getCurrentEnv().getAlterInstance().processAlterMTMV(replayed, true);
493+
}
494+
406495
@Test
407496
public void testCreateIncrementalMtmvAutoCreatesStream() throws Exception {
408497
createDatabaseAndUse("stream_test");

0 commit comments

Comments
 (0)