Skip to content

Commit d6c1a2b

Browse files
authored
[fix](ivm) Choose IVM baseline rebuild partitions from the MV partition mapping (#68180)
An IVM materialized view has to fully rebuild every MV partition that may hold rows of a base-table partition that was dropped / truncated / replaced / recovered. Those operations change the table through metadata and emit no row binlog, so the incremental path can never remove the stale rows: if such an MV partition is not marked, the orphan rows stay in the MV forever, with no error anywhere. Which MV partitions to mark was decided by asking the refresh snapshot which of them had *seen* the changed base partition (`MTMVRefreshSnapshot#getMvPartitionNames`). A snapshot is captured before the base table is read, so it is only a lower bound: a base partition that appears in the base table after the capture is not in it, and each partition's snapshot is written back by whichever task finishes last. When at least one other MV partition did match, the lookup returned that subset and the partition that had actually read the dropped one was silently left out: ``` partition p1: capture snapshot p1 reads the base table p1 still running base table: X is added p2 refreshes (its snapshot has X) X dropped marker: p1's snapshot has no X, p2's has -> only p2 is marked result: p1 keeps the rows of X forever ``` This asks the MV partition mapping instead: which MV partitions read the changed base partition is exactly the question, it is metadata rather than a lagging record, and it is the same mapping the refresh already uses for partition sync. | case | before | after | | --- | --- | --- | | MV partitions that read the changed base partition | marked | marked (unchanged) | | changed base partition read by no MV partition | whole MV rebuilt | nothing marked | | SELF_MANAGE MV, base table is not a PCT table, `RECOVER PARTITION`, a PCT table of a multi-PCT MV, mapping computation fails | whole MV rebuilt | whole MV rebuilt (unchanged) | The second row is intentional: the old lookup could not tell "no MV partition reads this partition" apart from "the snapshot does not know it", so it rebuilt the whole MV -- a table whose dynamic partitions are dropped ahead of the MV's partition sync forced a full rebuild every time. The mapping tells them apart, and the cases that genuinely cannot be answered from it are explicit checks rather than a guess from an empty result: - `RECOVER PARTITION` marks before the partition is added back to the table, so the partition is still in the recycle bin at that moment and no metadata-derived mapping can describe it. - The mapping is seeded from the MV's PCT tables and never gains a table later, so a joined partition table the MV's partition column does not reach is not described by it at all. - A mapping computation failure must not fail the base table DDL: it warns and rebuilds the whole MV. - The mapping reads the partition items of the MV and of every PCT table, so it takes their read locks, while the caller already holds the changed table's write lock: two concurrent partition DDLs on two PCT tables of the same MV would each hold the write lock the other one needs. Those reads are therefore taken with a bounded `tryReadLock` (`Table.TRY_LOCK_TIMEOUT_MS`, the same way the stream cleanup treats a busy table) and never block: while the other tables are free -- the normal case -- the selection still narrows to the MV partitions that read the changed base partition, and while one of them is being written the whole MV is rebuilt. Acquiring in id order cannot help here, because the first lock of the pair is already held before this point. The mapping is computed *before* the MV lock is taken, not inside it: computing it takes the partition items of the MV and of its PCT tables, and the MV lock is a leaf lock -- nothing is acquired under it today, and the lock analysis of the baseline barrier depends on that. The selection does not need to be atomic with the barrier it produces: the barrier is recorded under the lock, and the names it carries are intersected with the live partition names when they are consumed (`MTMVTask`). `MTMVRefreshSnapshot#getMvPartitionNames`, the old selection, is removed with the unit test that covered it; it has no other caller.
1 parent 4a4b857 commit d6c1a2b

12 files changed

Lines changed: 1025 additions & 95 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java‎

Lines changed: 240 additions & 13 deletions
Large diffs are not rendered by default.

‎fe/fe-core/src/main/java/org/apache/doris/common/util/MetaLockUtils.java‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,26 @@ public static void readLockTables(List<? extends TableIf> tableList) {
5252
}
5353
}
5454

55+
/**
56+
* Read-lock every table, giving up instead of waiting when one of them stays busy for {@code timeout}.
57+
* The locks taken so far are released before returning false, so a caller that cannot proceed holds
58+
* nothing. Use this where blocking is not an option: a caller that already holds another table's write
59+
* lock can deadlock against a thread that holds this one and wants that one.
60+
*
61+
* @return true when every table is read-locked, false when the locks were released again
62+
*/
63+
public static boolean tryReadLockTables(List<? extends TableIf> tableList, long timeout, TimeUnit unit) {
64+
for (int i = 0; i < tableList.size(); i++) {
65+
if (!tableList.get(i).tryReadLock(timeout, unit)) {
66+
for (int j = i - 1; j >= 0; j--) {
67+
tableList.get(j).readUnlock();
68+
}
69+
return false;
70+
}
71+
}
72+
return true;
73+
}
74+
5575
public static void readUnlockTables(List<? extends TableIf> tableList) {
5676
for (int i = tableList.size() - 1; i >= 0; i--) {
5777
tableList.get(i).readUnlock();

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,15 @@ public class MTMVPartitionUtil {
9191
new MTMVRelatedPartitionDescTransferGenerator()
9292
);
9393

94+
/**
95+
* Whether a partition_sync_limit is in effect, i.e. whether the windowed mapping leaves base partitions
96+
* out. Only that mapping is bounded by the window, so only there does its completeness depend on it.
97+
*/
98+
public static boolean isPartitionSyncLimitActive(Map<String, String> mvProperties) {
99+
return MTMVRelatedPartitionDescSyncLimitGenerator
100+
.generateMTMVPartitionSyncConfigByProperties(mvProperties).getSyncLimit() > 0;
101+
}
102+
94103
/**
95104
* Determine whether the partition is sync with retated partition and other baseTables
96105
*

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,12 @@
4545
import java.util.Set;
4646

4747
public class MTMVPropertyUtil {
48+
/** The properties a partition_sync_limit window is built from; see MTMV#alterMvProperties. */
49+
private static final List<String> PARTITION_SYNC_WINDOW_KEYS = Lists.newArrayList(
50+
PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT,
51+
PropertyAnalyzer.PROPERTIES_PARTITION_TIME_UNIT,
52+
PropertyAnalyzer.PROPERTIES_PARTITION_DATE_FORMAT);
53+
4854
public static final Set<String> MV_PROPERTY_KEYS = Sets.newHashSet(
4955
PropertyAnalyzer.PROPERTIES_GRACE_PERIOD,
5056
PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES,
@@ -231,6 +237,56 @@ public static boolean hasPartitionSyncLimit(Map<String, String> mvProperties) {
231237
return !StringUtils.isEmpty(value) && Integer.parseInt(value) > 0;
232238
}
233239

240+
/** Whether the given (altered) properties touch the partition_sync_limit window. */
241+
public static boolean containsPartitionSyncWindow(Map<String, String> properties) {
242+
for (String property : PARTITION_SYNC_WINDOW_KEYS) {
243+
if (properties.containsKey(property)) {
244+
return true;
245+
}
246+
}
247+
return false;
248+
}
249+
250+
/**
251+
* Whether the change from {@code oldWindowProperties} to {@code newWindowProperties} can bring base
252+
* partitions back into the window the MV maintains: a limit that is removed, or a wider one. Those
253+
* partitions' deltas were skipped while they were outside, so nothing incremental can repair them.
254+
*
255+
* <p>A window that starts applying is not such a change -- it only takes partitions out of what the MV
256+
* maintains, and partition sync drops those before the refresh plans -- and neither is a narrower one.
257+
* Windows in different units or date formats are not comparable without a clock, so a change to either
258+
* is treated as one that may widen.
259+
*/
260+
public static boolean partitionSyncWindowWidens(Map<String, String> oldWindowProperties,
261+
Map<String, String> newWindowProperties) {
262+
MTMVPartitionSyncConfig oldWindow = MTMVRelatedPartitionDescSyncLimitGenerator
263+
.generateMTMVPartitionSyncConfigByProperties(oldWindowProperties);
264+
if (oldWindow.getSyncLimit() <= 0) {
265+
return false;
266+
}
267+
MTMVPartitionSyncConfig newWindow = MTMVRelatedPartitionDescSyncLimitGenerator
268+
.generateMTMVPartitionSyncConfigByProperties(newWindowProperties);
269+
if (newWindow.getSyncLimit() <= 0) {
270+
return true;
271+
}
272+
return !oldWindow.getTimeUnit().equals(newWindow.getTimeUnit())
273+
|| !oldWindow.getDateFormat().equals(newWindow.getDateFormat())
274+
|| newWindow.getSyncLimit() > oldWindow.getSyncLimit();
275+
}
276+
277+
/**
278+
* The window the given properties describe, for comparing it across an ALTER. Only the properties the
279+
* window is built from are read, and the values are compared as they are stored: setting the same
280+
* window again changes nothing about which rows the MV owes and must not force a rebuild.
281+
*/
282+
public static Map<String, String> partitionSyncWindowOf(Map<String, String> properties) {
283+
Map<String, String> res = Maps.newHashMap();
284+
for (String property : PARTITION_SYNC_WINDOW_KEYS) {
285+
res.put(property, properties.get(property));
286+
}
287+
return res;
288+
}
289+
234290
/**
235291
* Look up the window limit configured for a base table, mirroring the
236292
* excluded_trigger_tables name-matching semantics (empty db/ctl wildcard).

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshSnapshot.java‎

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,8 @@
2424
import com.google.gson.annotations.SerializedName;
2525
import org.apache.commons.collections4.MapUtils;
2626

27-
import java.util.HashSet;
2827
import java.util.Iterator;
2928
import java.util.Map;
30-
import java.util.Optional;
3129
import java.util.Set;
3230
import java.util.concurrent.ConcurrentMap;
3331

@@ -61,28 +59,6 @@ public Set<String> getPctSnapshots(String mtmvPartitionName, BaseTableInfo pctTa
6159
return partitionSnapshot.getPctSnapshot(pctTableInfo).keySet();
6260
}
6361

64-
public Optional<Set<String>> getMvPartitionNames(BaseTableInfo pctTableInfo,
65-
Map<String, Long> pctPartitions) {
66-
Set<String> matchedPctPartitions = new HashSet<>();
67-
Set<String> mvPartitionNames = new HashSet<>();
68-
for (Map.Entry<String, MTMVRefreshPartitionSnapshot> entry : partitionSnapshots.entrySet()) {
69-
Map<String, MTMVSnapshotIf> pctSnapshots = entry.getValue().getPcts().get(pctTableInfo);
70-
if (pctSnapshots == null) {
71-
continue;
72-
}
73-
for (Map.Entry<String, Long> pctPartition : pctPartitions.entrySet()) {
74-
MTMVSnapshotIf snapshot = pctSnapshots.get(pctPartition.getKey());
75-
if (snapshot instanceof MTMVVersionSnapshot
76-
&& ((MTMVVersionSnapshot) snapshot).getId() == pctPartition.getValue()) {
77-
matchedPctPartitions.add(pctPartition.getKey());
78-
mvPartitionNames.add(entry.getKey());
79-
}
80-
}
81-
}
82-
return matchedPctPartitions.size() == pctPartitions.size()
83-
? Optional.of(mvPartitionNames) : Optional.empty();
84-
}
85-
8662
public boolean equalsWithBaseTable(String mtmvPartitionName, BaseTableInfo tableInfo,
8763
MTMVSnapshotIf baseTableCurrentSnapshot) {
8864
MTMVRefreshPartitionSnapshot partitionSnapshot = partitionSnapshots.get(mtmvPartitionName);

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescSyncLimitGenerator.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ public void apply(MTMVPartitionInfo mvPartitionInfo, Map<String, String> mvPrope
7777
* @param mvProperties
7878
* @return
7979
*/
80-
public MTMVPartitionSyncConfig generateMTMVPartitionSyncConfigByProperties(
80+
public static MTMVPartitionSyncConfig generateMTMVPartitionSyncConfigByProperties(
8181
Map<String, String> mvProperties) {
8282
int syncLimit = StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT)) ? -1
8383
: Integer.parseInt(mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT));

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -115,13 +115,22 @@ private void markIvmBaselineRebuild(BaseTableInfo baseTableInfo, boolean allPart
115115
if (MTMVPartitionUtil.isTableExcluded(mtmv.getExcludedTriggerTables(), baseTableName)) {
116116
continue;
117117
}
118+
boolean invalidated;
118119
if (allPartitionsChanged) {
119120
mtmv.invalidateIvmBaseline();
121+
invalidated = true;
120122
} else {
121-
mtmv.invalidateIvmBaseline(baseTableInfo, changedPartitions);
123+
invalidated = mtmv.invalidateIvmBaseline(baseTableInfo, changedPartitions);
124+
}
125+
// A partition change that no MV partition reads leaves nothing to rebuild, and saying that it
126+
// invalidated the baseline would claim a persisted barrier that does not exist.
127+
if (invalidated) {
128+
LOG.info("Invalidated IVM baseline, baseTable={}, mtmv={}, reason={}",
129+
baseTableInfo, mtmvInfo, reason);
130+
} else {
131+
LOG.info("No MV partition reads the changed base partitions, nothing to invalidate. "
132+
+ "baseTable={}, mtmv={}, reason={}", baseTableInfo, mtmvInfo, reason);
122133
}
123-
LOG.info("Invalidated IVM baseline, baseTable={}, mtmv={}, reason={}",
124-
baseTableInfo, mtmvInfo, reason);
125134
}
126135
}
127136

‎fe/fe-core/src/test/java/org/apache/doris/common/util/MetaLockUtilsTest.java‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
import org.junit.jupiter.api.Test;
2929

3030
import java.util.List;
31+
import java.util.concurrent.CountDownLatch;
3132
import java.util.concurrent.TimeUnit;
3233

3334
public class MetaLockUtilsTest {
@@ -60,6 +61,47 @@ public void testReadLockTables() {
6061
tableList.get(1).writeUnlock();
6162
}
6263

64+
/**
65+
* The read batch gives back what it took when a later table stays busy, so the caller holds nothing
66+
* while it decides what to do. Probed the same way as the tests above: a read lock still held would
67+
* block the write lock of the table that was acquired first.
68+
*/
69+
@Test
70+
public void testTryReadLockTablesReleasesWhatItTookWhenALaterTableIsBusy() throws Exception {
71+
List<Table> tables = ImmutableList.of(TableTest.newOlapTable(0, "readable", 0),
72+
TableTest.newOlapTable(1, "busy", 0));
73+
Table busy = tables.get(1);
74+
// Another thread has to hold the write lock: the same thread could take a read lock underneath
75+
// its own write lock (that direction is allowed), which is not the case under test. Both
76+
// acquisitions are bounded and the holder is asserted to end, so a failure cannot strand it.
77+
CountDownLatch locked = new CountDownLatch(1);
78+
CountDownLatch released = new CountDownLatch(1);
79+
Thread holder = new Thread(() -> {
80+
Assertions.assertTrue(busy.tryWriteLock(30, TimeUnit.SECONDS));
81+
locked.countDown();
82+
try {
83+
released.await(30, TimeUnit.SECONDS);
84+
} catch (InterruptedException e) {
85+
Thread.currentThread().interrupt();
86+
} finally {
87+
busy.writeUnlock();
88+
}
89+
});
90+
holder.start();
91+
try {
92+
Assertions.assertTrue(locked.await(30, TimeUnit.SECONDS));
93+
Assertions.assertFalse(MetaLockUtils.tryReadLockTables(tables, 100, TimeUnit.MILLISECONDS),
94+
"a busy table must fail the batch");
95+
Assertions.assertTrue(tables.get(0).tryWriteLock(100, TimeUnit.MILLISECONDS),
96+
"the read lock taken for the first table must have been released");
97+
tables.get(0).writeUnlock();
98+
} finally {
99+
released.countDown();
100+
holder.join(TimeUnit.SECONDS.toMillis(30));
101+
}
102+
Assertions.assertFalse(holder.isAlive(), "the holder should have released the table");
103+
}
104+
63105
@Test
64106
public void testWriteLockTables() throws MetaNotFoundException {
65107
MetaLockUtils.writeLockTables(tableList);

‎fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTest.java‎

Lines changed: 0 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,6 @@
6565
import java.util.List;
6666
import java.util.Map;
6767
import java.util.Map.Entry;
68-
import java.util.Optional;
6968
import java.util.Set;
7069
import java.util.concurrent.locks.ReentrantReadWriteLock;
7170

@@ -142,26 +141,6 @@ public void testCalculateDoublyPartitionMappings() throws AnalysisException {
142141
Assertions.assertEquals(baseToMv.get("baseP1_2"), "mvp1");
143142
}
144143

145-
@Test
146-
public void testChangedBasePartitionsRequireCompleteSnapshotMapping() {
147-
BaseTableInfo baseTableInfo = Mockito.mock(BaseTableInfo.class);
148-
MTMVRefreshPartitionSnapshot firstSnapshot = new MTMVRefreshPartitionSnapshot();
149-
firstSnapshot.getPctSnapshot(baseTableInfo).put("base_p1", new MTMVVersionSnapshot(1L, 11L));
150-
MTMVRefreshPartitionSnapshot secondSnapshot = new MTMVRefreshPartitionSnapshot();
151-
secondSnapshot.getPctSnapshot(baseTableInfo).put("base_p2", new MTMVVersionSnapshot(1L, 12L));
152-
MTMVRefreshSnapshot refreshSnapshot = new MTMVRefreshSnapshot();
153-
refreshSnapshot.updateSnapshots(
154-
Map.of("mv_p1", firstSnapshot, "mv_p2", secondSnapshot), Set.of("mv_p1", "mv_p2"));
155-
156-
Optional<Set<String>> mappedPartitions = refreshSnapshot.getMvPartitionNames(
157-
baseTableInfo, Map.of("base_p1", 11L, "base_p2", 12L));
158-
159-
Assertions.assertTrue(mappedPartitions.isPresent());
160-
Assertions.assertEquals(Set.of("mv_p1", "mv_p2"), mappedPartitions.get());
161-
Assertions.assertFalse(refreshSnapshot.getMvPartitionNames(
162-
baseTableInfo, Map.of("base_p1", 11L, "base_p3", 13L)).isPresent());
163-
}
164-
165144
private Map<PartitionKeyDesc, Set<String>> mockRelatedPartitionDescs() throws AnalysisException {
166145
Map<PartitionKeyDesc, Set<String>> res = Maps.newHashMap();
167146
Column k1 = new Column("k1", ScalarType.createType(PrimitiveType.TINYINT), true, null, "", "key1");

0 commit comments

Comments
 (0)