Skip to content

Commit 7bd89a0

Browse files
authored
[fix](ivm) Stop the incremental delta from reading partitions the MV dropped (apache#67814)
`partition_sync_limit` keeps only a recent slice of each base table's partitions, so an MV can be missing a partition its base table still has. The incremental delta read that base partition anyway, through the delta scan and through the join-opposite snapshot. A change to a non-partitioned dimension therefore produced delta rows for dates the MV has no partition for, and the insert failed with `no partition for this tuple`. Re-syncing cannot bring an expired partition back, so the retry loop in `executeIvmAttempt` could never recover: the task failed after exhausting its attempts, and because the write is atomic, the partitions the MV does keep were left unrepaired as well. A strict `REFRESH ... INCREMENTAL`, and a scheduled refresh of an MV declared without `FALLBACK`, fail outright. ### What changed The incremental delta now reads only the base partitions the MV's partition definition keeps. - `MTMVPartitionUtil.generateRelatedBasePartitionIds` returns that partition set per base table partitioned by the MV's partition column, and an empty value when `partition_sync_limit` is not set, which is the only property that can leave the MV without a base partition. - `IvmIncrRefreshManager` passes it to `IvmRewriteContext.incremental`, and `IvmDeltaRewriter` applies it as an upper bound on the partitions each base table may be read from, intersected with `ivm_partition_window_limit` when that is set too. - The bound is the partition set the MV is aligned to rather than the partitions it already has, so a base partition whose MV partition has not been added yet stays readable: the refresh still reports the missing partition and recovers it by syncing, which is what keeps a newly added base partition working. - A base table absent from the set keeps its full read, since the MV partition column does not come from it and limiting it would change join results without narrowing the target MV partitions. A set covering every partition of its table leaves the plan untouched, so an MV that mirrors all of its base partitions is unaffected. - Only olap tables are reachable: the delta reads them through their stream and restricts a scan by partition id, which a connector table has no equivalent of. The COMPLETE paths are unchanged. The PR also removes the `nonConcurrent` group from seven `mtmv_p0/ivm` suites that use neither debug points nor global variables or config, so they run in the parallel pool again. Trace issue: apache#65418
1 parent 4b5ab04 commit 7bd89a0

20 files changed

Lines changed: 670 additions & 20 deletions

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,63 @@ public static Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> genera
237237
return result.getRes();
238238
}
239239

240+
/**
241+
* Base partitions the MV's partition definition keeps, per base table partitioned by the MV's
242+
* partition column. This is the partition set the MV is aligned to, so it still contains a base
243+
* partition whose MV partition the next {@link #alignMvPartition} has yet to add, and no longer
244+
* contains one the partition properties filter out, such as an expired partition_sync_limit.
245+
*
246+
* <p>Empty when no restriction applies at all: without {@code partition_sync_limit} the MV
247+
* mirrors every base partition, so there is nothing to restrict. When present, the map holds an
248+
* entry for every base table the MV partition column comes from, each with the partitions that
249+
* table may be read from — an empty set meaning that table may not be read at all. Keeping the
250+
* empty set explicit is what separates "this table is restricted to nothing" from "this table
251+
* is not one of the MV partition column's sources".
252+
*
253+
* <p>Read without pinned snapshots, like {@link #alignMvPartition}. A base partition that
254+
* partition sync adds or drops at the same moment can therefore sit outside the returned set
255+
* for one refresh; the same refresh either recovers the partition by syncing again or leaves
256+
* its binlog pending for the next one.
257+
*
258+
* @return baseTableInfo ==> base partition ids, empty when no restriction applies
259+
*/
260+
public static Optional<Map<BaseTableInfo, Set<Long>>> generateRelatedBasePartitionIds(MTMV mtmv)
261+
throws AnalysisException {
262+
MTMVPartitionInfo mvPartitionInfo = mtmv.getMvPartitionInfo();
263+
if (mvPartitionInfo == null
264+
|| mvPartitionInfo.getPartitionType() == MTMVPartitionType.SELF_MANAGE
265+
|| !MTMVPropertyUtil.hasPartitionSyncLimit(mtmv.getMvProperties())) {
266+
return Optional.empty();
267+
}
268+
Map<BaseTableInfo, Set<Long>> res = Maps.newHashMap();
269+
// Only olap tables are restricted: the delta rewrite reads them through their stream and
270+
// selects partitions by id, which an external table has no equivalent of. A connector
271+
// table therefore stays out of the scope and keeps its full read, so supporting one as an
272+
// IVM base table means giving the scope a partition identity it can express, not just
273+
// widening these types.
274+
for (MTMVRelatedTableIf pctTable : mvPartitionInfo.getPctTables()) {
275+
if (pctTable instanceof OlapTable) {
276+
res.put(new BaseTableInfo((OlapTable) pctTable), Sets.newHashSet());
277+
}
278+
}
279+
Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> relatedDescs = generateRelatedPartitionDescs(
280+
mvPartitionInfo, mtmv.getMvProperties(), mtmv.getPartitionColumns(), Maps.newHashMap());
281+
for (Map<MTMVRelatedTableIf, Set<String>> relatedPartitions : relatedDescs.values()) {
282+
for (Entry<MTMVRelatedTableIf, Set<String>> entry : relatedPartitions.entrySet()) {
283+
if (!(entry.getKey() instanceof OlapTable)) {
284+
continue;
285+
}
286+
OlapTable baseTable = (OlapTable) entry.getKey();
287+
Set<Long> partitionIds = res.computeIfAbsent(
288+
new BaseTableInfo(baseTable), key -> Sets.newHashSet());
289+
for (String partitionName : entry.getValue()) {
290+
partitionIds.add(baseTable.getPartitionOrAnalysisException(partitionName).getId());
291+
}
292+
}
293+
}
294+
return Optional.of(res);
295+
}
296+
240297
/**
241298
* check if table is sync with all baseTables
242299
*

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,19 @@ public static Map<TableNameInfo, Integer> getIvmPartitionWindowLimit(Map<String,
218218
PropertyAnalyzer.PROPERTIES_IVM_PARTITION_WINDOW_LIMIT));
219219
}
220220

221+
/**
222+
* True when the MV keeps only a recent slice of each base table's partitions. This is the only
223+
* property that can leave the MV without a partition the base table still has, so it is also
224+
* the only case where the incremental delta has to be told which base partitions it may read.
225+
*/
226+
public static boolean hasPartitionSyncLimit(Map<String, String> mvProperties) {
227+
if (mvProperties == null) {
228+
return false;
229+
}
230+
String value = mvProperties.get(PropertyAnalyzer.PROPERTIES_PARTITION_SYNC_LIMIT);
231+
return !StringUtils.isEmpty(value) && Integer.parseInt(value) > 0;
232+
}
233+
221234
/**
222235
* Look up the window limit configured for a base table, mirroring the
223236
* excluded_trigger_tables name-matching semantics (empty db/ctl wildcard).

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDeltaRewriter.java‎

Lines changed: 62 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import org.apache.doris.common.Config;
2727
import org.apache.doris.common.Pair;
2828
import org.apache.doris.info.TableNameInfoUtils;
29+
import org.apache.doris.mtmv.BaseTableInfo;
2930
import org.apache.doris.mtmv.MTMVPartitionUtil;
3031
import org.apache.doris.mtmv.MTMVPropertyUtil;
3132
import org.apache.doris.nereids.trees.expressions.Slot;
@@ -37,6 +38,7 @@
3738
import org.apache.doris.nereids.types.DataType;
3839
import org.apache.doris.qe.ConnectContext;
3940

41+
import java.util.ArrayList;
4042
import java.util.HashMap;
4143
import java.util.HashSet;
4244
import java.util.List;
@@ -66,7 +68,8 @@ public Plan generateIncrRefreshPlan(Plan sinkChild, IvmRewriteResult rewriteResu
6668
Pair<Plan, List<LogicalProject<?>>> prefixChain = helper.detachAdaptProjectChain(sinkChild);
6769
Plan rootPlan = prefixChain.first;
6870
long refreshVersion = refreshContext.getMtmv().getNextRefreshVersion();
69-
IvmDeltaRewriteState rewriteState = createDeltaRewriteState(rootPlan, refreshContext, refreshVersion);
71+
IvmDeltaRewriteState rewriteState = createDeltaRewriteState(rootPlan, refreshContext, refreshVersion,
72+
rewriteContext.getIncrementalScopePartitionIds());
7073
Optional<IvmDeltaRewriteResult> deltaResult = rewriteDelta(rootPlan, refreshContext, rewriteState);
7174
if (!deltaResult.isPresent()) {
7275
return new LogicalEmptyRelation(
@@ -153,7 +156,8 @@ private static Pair<Plan, Map<Slot, Slot>> rewriteSnapshot(Plan plan, IvmDeltaRe
153156
return IvmDeltaRewriteHelper.INSTANCE.freshPlan(rewritten);
154157
}
155158

156-
private IvmDeltaRewriteState createDeltaRewriteState(Plan plan, IvmIncrRefreshContext ctx, long refreshVersion) {
159+
private IvmDeltaRewriteState createDeltaRewriteState(Plan plan, IvmIncrRefreshContext ctx, long refreshVersion,
160+
Map<BaseTableInfo, Set<Long>> scopePartitionIds) {
157161
Map<OlapTable, OlapTableStream> streams = new HashMap<>();
158162
// Window limits apply to every base table in the plan, including excluded
159163
// trigger tables (their snapshot side is windowed too, so the property saves
@@ -181,11 +185,67 @@ private IvmDeltaRewriteState createDeltaRewriteState(Plan plan, IvmIncrRefreshCo
181185
new TableNameInfo(table.getFullQualifiers()), windowLimits));
182186
}
183187
}
188+
applyScopePartitionIds(windowPartitionIdsByTable, planTables, scopePartitionIds);
184189
return new IvmDeltaRewriteState(streams, ctx.isIncludeExhaustedStreams(), refreshVersion,
185190
DataType.fromCatalogType(ctx.getMtmv().getColumn(Column.SEQUENCE_COL).getType()),
186191
windowPartitionIdsByTable);
187192
}
188193

194+
/**
195+
* Limits every base table whose partition column feeds the MV's partition column to the base
196+
* partitions the MV's partition definition keeps. The delta and the join-opposite snapshot
197+
* would otherwise also read base partitions the MV does not keep, expired by
198+
* partition_sync_limit, and then try to write their rows into MV partitions that do not exist,
199+
* which fails the whole insert with "no partition for this tuple" and takes the partitions
200+
* that do exist down with it.
201+
*
202+
* <p>The limit is the partition set the MV is aligned to rather than the partitions it already
203+
* has, so a base partition whose MV partition has not been added yet stays readable: the
204+
* refresh still reports the missing partition and recovers it by syncing.
205+
*
206+
* <p>Tables outside {@code scopePartitionIds} keep their full read: the MV partition column
207+
* does not come from them, so limiting them would change join results without narrowing the
208+
* set of MV partitions the delta can target. A scope that covers every partition of its table
209+
* is left alone as well, so an MV that mirrors all of its base partitions keeps the plan it
210+
* had before this restriction existed. An entry with no partitions is the opposite case and is
211+
* applied as it stands: the scan then reads nothing, which is how the delta and the snapshot
212+
* side both stay out of a table the MV has no partition for.
213+
*
214+
* <p>The scope is keyed by base table identity, but only olap tables are reachable here: the
215+
* delta reads them through {@code OlapTableStream} and restricts a scan by partition id.
216+
*/
217+
private static void applyScopePartitionIds(Map<OlapTable, List<Long>> windowPartitionIdsByTable,
218+
Set<OlapTable> planTables, Map<BaseTableInfo, Set<Long>> scopePartitionIds) {
219+
if (scopePartitionIds.isEmpty()) {
220+
return;
221+
}
222+
for (OlapTable table : planTables) {
223+
Set<Long> tableScope = scopePartitionIds.get(new BaseTableInfo(table));
224+
if (tableScope == null || tableScope.containsAll(table.getPartitionIds())) {
225+
continue;
226+
}
227+
List<Long> windowPartitionIds = windowPartitionIdsByTable.get(table);
228+
List<Long> readablePartitionIds;
229+
if (windowPartitionIds == null) {
230+
readablePartitionIds = new ArrayList<>(tableScope);
231+
} else {
232+
// Both limits apply, so only their intersection is readable. One keeps the
233+
// partitions above a time cutoff and the other the last N by partition value, so
234+
// each is a suffix of the same value order and the two cannot be disjoint.
235+
readablePartitionIds = new ArrayList<>();
236+
for (Long partitionId : windowPartitionIds) {
237+
if (tableScope.contains(partitionId)) {
238+
readablePartitionIds.add(partitionId);
239+
}
240+
}
241+
}
242+
// Sorted like every other partition selection handed to a scan, so the plan shape does
243+
// not depend on the order the two limits were combined in.
244+
readablePartitionIds.sort(Long::compareTo);
245+
windowPartitionIdsByTable.put(table, readablePartitionIds);
246+
}
247+
}
248+
189249
boolean isExcludedTriggerTable(LogicalOlapScan scan, Set<TableNameInfo> excludedTriggerTables) {
190250
if (excludedTriggerTables == null || excludedTriggerTables.isEmpty()) {
191251
return false;

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManager.java‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
import org.apache.doris.common.util.DebugUtil;
2323
import org.apache.doris.common.util.Util;
2424
import org.apache.doris.datasource.InternalCatalog;
25+
import org.apache.doris.mtmv.BaseTableInfo;
26+
import org.apache.doris.mtmv.MTMVPartitionUtil;
2527
import org.apache.doris.mtmv.MTMVPlanUtil;
2628
import org.apache.doris.nereids.StatementContext;
2729
import org.apache.doris.nereids.analyzer.UnboundTableSink;
@@ -39,9 +41,12 @@
3941
import org.apache.logging.log4j.LogManager;
4042
import org.apache.logging.log4j.Logger;
4143

44+
import java.util.Collections;
4245
import java.util.List;
46+
import java.util.Map;
4347
import java.util.Objects;
4448
import java.util.Optional;
49+
import java.util.Set;
4550

4651
/**
4752
* Minimal orchestration entry point for incremental refresh.
@@ -92,7 +97,16 @@ void executeInternalRefresh(IvmIncrRefreshContext context) throws Exception {
9297
MTMV mtmv = context.getMtmv();
9398
StatementContext statementContext = new StatementContext(
9499
context.getConnectContext(), new OriginStatement(mtmv.getQuerySql(), 0));
95-
statementContext.setIvmRewriteContext(Optional.of(IvmRewriteContext.incremental(mtmv)));
100+
// The delta may only read the base partitions the MV's partition definition keeps. A base
101+
// partition outside that set, expired by partition_sync_limit, would otherwise still be
102+
// read through the delta and the join-opposite snapshot, and its rows would have no MV
103+
// partition to land in. A base partition inside the set stays readable even when its MV
104+
// partition is not there yet, so that the refresh still reports the missing partition and
105+
// recovers it by syncing.
106+
Map<BaseTableInfo, Set<Long>> scopePartitionIds =
107+
MTMVPartitionUtil.generateRelatedBasePartitionIds(mtmv).orElse(Collections.emptyMap());
108+
statementContext.setIvmRewriteContext(
109+
Optional.of(IvmRewriteContext.incremental(mtmv, scopePartitionIds)));
96110
// Excluded trigger tables do not produce delta and must not be validated for
97111
// binlog / key-type support during the incremental analyze.
98112
statementContext.setExcludedTriggerTables(mtmv.getExcludedTriggerTables());

0 commit comments

Comments
 (0)