Skip to content

Commit e65b4e9

Browse files
seawindeCopilot
authored andcommitted
Ivm pick create alter validation (apache#6)
* [improvement](fe) Add IVM create and alter validation for MTMV (apache#5) Issue Number: close #xxx Problem Summary: 1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call 2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW) 3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables) 4. Block ALTER MTMV refresh method to/from INCREMENTAL 5. Validate base table models when ALTER MTMV excluded_trigger_tables 6. Extract MTMVPropertyUtil.parseTableNameInfos utility 7. Add comprehensive unit tests for all validation paths * [fix](fe) Fix IVM ExprId collision by reusing parser StatementContext Issue Number: close #xxx Problem Summary: Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal() and restored the original (parser-created) StatementContext in the finally block. This caused ExprId collisions during IVM INCREMENTAL refresh because the parser StatementContext has a much smaller ExprId counter than the analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads exprIdStart from the ConnectContext's StatementContext after analysis. The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext() (the parser-created StatementContext) directly for analysis instead of creating a new one. This way all ExprId allocations accumulate in the same StatementContext that doRefreshInternal() later reads. None - Test: Regression test (test_ivm_agg_2) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Fix buildRowId to compute proper row-id for excluded trigger tables Issue Number: close #xxx Problem Summary: buildRowId() short-circuited to UuidNumeric() for excluded trigger tables, losing the deterministic row-id (hash of unique keys) for MOW and non-MOW UNIQUE_KEYS tables. The fix removes the early return and only uses the isExcludedTriggerTable flag to suppress AnalysisException for unsupported table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute buildRowIdHash(keySlots) regardless of exclusion status. None - Test: Unit Test (IvmNormalizeMtmvTest) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [test](fe) Update IvmNormalizeMtmvTest for excluded MOW table row-id fix Issue Number: close #xxx Problem Summary: Update testExcludedMowTableUsesTransientRowId to expect deterministic hash-based row-id (Cast expression) instead of UuidNumeric for excluded MOW tables, matching the fix in buildRowId(). None - Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * [fix](fe) Refine ALTER MTMV refresh method compatibility rules Issue Number: close #xxx Problem Summary: The ALTER MTMV refresh method validation was too restrictive (blocking all changes to/from INCREMENTAL) or too permissive in some cases. The new rules: - COMPLETE <-> INCREMENTAL: forbidden (must recreate MV) - COMPLETE/INCREMENTAL -> AUTO: allowed - AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV) - Same method (no-op): allowed None - Test: Regression test - Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed) - Does this need documentation: No Co-authored-by: Copilot <[email protected]> * opt code * [test](fe) Isolate CreateMTMVCommandTest statement context Issue Number: None Related PR: None Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext. None - Test: FE unit test - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Fix TableNameInfo imports after rebasing IVM branch Issue Number: None Related PR: None Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing org.apache.doris.info.TableNameInfo while the current base branch provides the type in org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased branch compiles and the targeted IVM FE tests run successfully again. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: No - Does this need documentation: No * [fix](fe) Use deterministic row id for excluded AGG_KEYS tables Issue Number: None Related PR: None Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes non-key value columns. None - Test: Unit Test - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest - org.apache.doris.mtmv.ivm.IvmUtilTest - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest - org.apache.doris.catalog.ShowCreateMTMVTest - Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys) - Does this need documentation: No * fix comment * fix comment * opt code --------- Co-authored-by: Copilot <[email protected]>
1 parent 68ea091 commit e65b4e9

16 files changed

Lines changed: 868 additions & 51 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java‎

Lines changed: 5 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType;
3737
import org.apache.doris.mtmv.MTMVPartitionUtil;
3838
import org.apache.doris.mtmv.MTMVPlanUtil;
39+
import org.apache.doris.mtmv.MTMVPropertyUtil;
3940
import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVRefreshState;
4041
import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVState;
4142
import org.apache.doris.mtmv.MTMVRefreshInfo;
@@ -52,7 +53,6 @@
5253

5354
import com.google.common.collect.Lists;
5455
import com.google.common.collect.Maps;
55-
import com.google.common.collect.Sets;
5656
import com.google.gson.annotations.SerializedName;
5757
import org.apache.commons.collections4.MapUtils;
5858
import org.apache.commons.lang3.StringUtils;
@@ -377,31 +377,15 @@ public Set<TableNameInfo> getExcludedTriggerTables() {
377377
}
378378

379379
private Set<TableNameInfo> parseExcludedTriggerTables() {
380-
Set<TableNameInfo> res = Sets.newHashSet();
381-
if (StringUtils.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES))) {
382-
return res;
383-
}
384-
String[] split = mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES).split(",");
385-
for (String alias : split) {
386-
res.add(new TableNameInfo(alias));
387-
}
388-
return res;
380+
return MTMVPropertyUtil.parseTableNameInfos(
381+
mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES));
389382
}
390383

391384
public Set<TableNameInfo> getQueryRewriteConsistencyRelaxedTables() {
392-
Set<TableNameInfo> res = Sets.newHashSet();
393385
readMvLock();
394386
try {
395-
String stillRewrittenTables
396-
= mvProperties.get(PropertyAnalyzer.ASYNC_MV_QUERY_REWRITE_CONSISTENCY_RELAXED_TABLES);
397-
if (StringUtils.isEmpty(stillRewrittenTables)) {
398-
return res;
399-
}
400-
String[] split = stillRewrittenTables.split(",");
401-
for (String alias : split) {
402-
res.add(new TableNameInfo(alias));
403-
}
404-
return res;
387+
return MTMVPropertyUtil.parseTableNameInfos(
388+
mvProperties.get(PropertyAnalyzer.ASYNC_MV_QUERY_REWRITE_CONSISTENCY_RELAXED_TABLES));
405389
} finally {
406390
readMvUnlock();
407391
}

‎fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -362,6 +362,7 @@ private void exec(Set<String> refreshPartitionNames,
362362
// StatementContext is null and a NullPointerException is thrown.
363363
mtmvCtx.setStatementContext(statementContext);
364364
statementContext.setConnectContext(mtmvCtx);
365+
statementContext.setExcludedTriggerTables(mtmv.getExcludedTriggerTables());
365366
for (Entry<MvccTableInfo, MvccSnapshot> entry : snapshots.entrySet()) {
366367
statementContext.setSnapshot(entry.getKey(), entry.getValue());
367368
}

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
import org.apache.doris.catalog.TableIf;
3535
import org.apache.doris.catalog.TableIf.TableType;
3636
import org.apache.doris.catalog.Type;
37+
import org.apache.doris.catalog.info.TableNameInfo;
3738
import org.apache.doris.common.ErrorCode;
3839
import org.apache.doris.common.ErrorReport;
3940
import org.apache.doris.common.FeConstants;
@@ -93,6 +94,7 @@
9394
import com.google.common.collect.Maps;
9495
import com.google.common.collect.Sets;
9596
import org.apache.commons.collections4.CollectionUtils;
97+
import org.apache.commons.lang3.StringUtils;
9698
import org.apache.logging.log4j.LogManager;
9799
import org.apache.logging.log4j.Logger;
98100

@@ -511,7 +513,28 @@ public static MTMVAnalyzeQueryInfo analyzeQuery(ConnectContext ctx, Map<String,
511513
MTMVPartitionDefinition mvPartitionDefinition, DistributionDescriptor distribution,
512514
List<SimpleColumnDefinition> simpleColumnDefinitions, Map<String, String> properties, List<String> keys,
513515
LogicalPlan logicalQuery, boolean isIvm) throws UserException {
516+
if (!isIvm) {
517+
return analyzeQueryInternal(ctx, mvProperties, mvPartitionDefinition, distribution,
518+
simpleColumnDefinitions, properties, keys, logicalQuery, false, Sets.newHashSet());
519+
}
520+
MTMVAnalyzeQueryInfo queryInfo = analyzeQueryInternal(ctx, mvProperties, mvPartitionDefinition, distribution,
521+
simpleColumnDefinitions, properties, keys, logicalQuery, true,
522+
getExcludedTriggerTables(mvProperties));
523+
return queryInfo;
524+
}
525+
526+
private static MTMVAnalyzeQueryInfo analyzeQueryInternal(ConnectContext ctx, Map<String, String> mvProperties,
527+
MTMVPartitionDefinition mvPartitionDefinition, DistributionDescriptor distribution,
528+
List<SimpleColumnDefinition> simpleColumnDefinitions, Map<String, String> properties, List<String> keys,
529+
LogicalPlan logicalQuery, boolean isIvm, Set<TableNameInfo> excludedTriggerTables)
530+
throws UserException {
531+
// Reuse the StatementContext already on the ConnectContext (set by NereidsParser during
532+
// SQL parsing or by the user session). Do NOT create a new StatementContext here — the
533+
// IVM refresh pipeline reads ExprId tracking from ctx.getStatementContext() after this
534+
// method returns (IvmRefreshManager.doRefreshInternal reads exprIdStart). Creating a
535+
// separate StatementContext and restoring the original would lose ExprId allocations.
514536
try (StatementContext statementContext = ctx.getStatementContext()) {
537+
statementContext.setExcludedTriggerTables(excludedTriggerTables);
515538
NereidsPlanner planner = new NereidsPlanner(statementContext);
516539
// this is for expression column name infer when not use alias
517540
LogicalSink<Plan> logicalSink = new UnboundResultSink<>(logicalQuery);
@@ -584,6 +607,39 @@ public static MTMVAnalyzeQueryInfo analyzeQuery(ConnectContext ctx, Map<String,
584607
}
585608
}
586609

610+
private static Set<TableNameInfo> getExcludedTriggerTables(Map<String, String> mvProperties) {
611+
if (mvProperties == null) {
612+
return Sets.newHashSet();
613+
}
614+
return MTMVPropertyUtil.parseTableNameInfos(
615+
mvProperties.get(PropertyAnalyzer.PROPERTIES_EXCLUDED_TRIGGER_TABLES));
616+
}
617+
618+
public static void validateAlterExcludedTriggerTables(MTMV mtmv, Map<String, String> mvProperties) {
619+
Set<TableNameInfo> oldExcludedTriggerTables = mtmv.getExcludedTriggerTables();
620+
Set<TableNameInfo> newExcludedTriggerTables = getExcludedTriggerTables(mvProperties);
621+
for (TableNameInfo oldExcludedTriggerTable : oldExcludedTriggerTables) {
622+
boolean isCovered = newExcludedTriggerTables.stream()
623+
.anyMatch(newExcludedTriggerTable -> isExcludedTriggerTableScopeCovered(
624+
oldExcludedTriggerTable, newExcludedTriggerTable));
625+
if (!isCovered) {
626+
throw new AnalysisException(
627+
"Cannot ALTER excluded_trigger_tables to narrow existing entry '"
628+
+ oldExcludedTriggerTable
629+
+ "'. Existing excluded trigger tables can only be expanded.");
630+
}
631+
}
632+
}
633+
634+
private static boolean isExcludedTriggerTableScopeCovered(TableNameInfo oldExcludedTriggerTable,
635+
TableNameInfo newExcludedTriggerTable) {
636+
return oldExcludedTriggerTable.getTbl().equals(newExcludedTriggerTable.getTbl())
637+
&& (StringUtils.isEmpty(newExcludedTriggerTable.getDb())
638+
|| newExcludedTriggerTable.getDb().equals(oldExcludedTriggerTable.getDb()))
639+
&& (StringUtils.isEmpty(newExcludedTriggerTable.getCtl())
640+
|| newExcludedTriggerTable.getCtl().equals(oldExcludedTriggerTable.getCtl()));
641+
}
642+
587643
/**
588644
* validate column name
589645
*/

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPropertyUtil.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.doris.mtmv;
1919

2020
import org.apache.doris.catalog.Env;
21+
import org.apache.doris.catalog.info.TableNameInfo;
2122
import org.apache.doris.common.util.PropertyAnalyzer;
2223
import org.apache.doris.mysql.privilege.PrivPredicate;
2324
import org.apache.doris.nereids.exceptions.AnalysisException;
@@ -125,6 +126,20 @@ private static void analyzeExcludedTriggerTables(String value) {
125126
// do nothing
126127
}
127128

129+
public static Set<TableNameInfo> parseTableNameInfos(String value) {
130+
Set<TableNameInfo> tableNameInfos = Sets.newHashSet();
131+
if (StringUtils.isEmpty(value)) {
132+
return tableNameInfos;
133+
}
134+
for (String tableName : value.split(",")) {
135+
String trimmed = tableName.trim();
136+
if (!trimmed.isEmpty()) {
137+
tableNameInfos.add(new TableNameInfo(trimmed));
138+
}
139+
}
140+
return tableNameInfos;
141+
}
142+
128143
private static void analyzeDataChangeStillRewrittenTables(String value) {
129144
// do nothing
130145
}

‎fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.doris.catalog.Partition;
2828
import org.apache.doris.catalog.TableIf;
2929
import org.apache.doris.catalog.View;
30+
import org.apache.doris.catalog.info.TableNameInfo;
3031
import org.apache.doris.common.Id;
3132
import org.apache.doris.common.IdGenerator;
3233
import org.apache.doris.common.Pair;
@@ -351,6 +352,7 @@ public enum TableFrom {
351352

352353
private final Set<CTEId> mustInlineCTE = new HashSet<>();
353354
private final Set<String> usedAIResourceNames = new LinkedHashSet<>();
355+
private final Set<TableNameInfo> excludedTriggerTables = new HashSet<>();
354356

355357
private final Map<String, Integer> lowerCaseTableNamesCache = Maps.newHashMap();
356358
private final Map<String, Integer> lowerCaseDatabaseNamesCache = Maps.newHashMap();
@@ -446,6 +448,17 @@ public Map<List<String>, TableIf> getOneLevelTables() {
446448
return oneLevelTables;
447449
}
448450

451+
public Set<TableNameInfo> getExcludedTriggerTables() {
452+
return excludedTriggerTables;
453+
}
454+
455+
public void setExcludedTriggerTables(Set<TableNameInfo> excludedTriggerTables) {
456+
this.excludedTriggerTables.clear();
457+
if (excludedTriggerTables != null) {
458+
this.excludedTriggerTables.addAll(excludedTriggerTables);
459+
}
460+
}
461+
449462
public Set<MTMV> getCandidateMTMVs() {
450463
return candidateMTMVs;
451464
}

‎fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,8 @@
4242
import com.google.common.collect.ImmutableSet;
4343
import com.google.common.collect.Lists;
4444
import com.google.common.collect.Maps;
45-
import org.antlr.v4.runtime.BailErrorStrategy;
4645
import org.antlr.v4.runtime.CharStreams;
4746
import org.antlr.v4.runtime.CommonTokenStream;
48-
import org.antlr.v4.runtime.DefaultErrorStrategy;
4947
import org.antlr.v4.runtime.ParserRuleContext;
5048
import org.antlr.v4.runtime.Recognizer;
5149
import org.antlr.v4.runtime.Token;
@@ -419,15 +417,13 @@ public static ParserRuleContext toAst(
419417
try {
420418
// first, try parsing with potentially faster SLL mode
421419
parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
422-
parser.setErrorHandler(new BailErrorStrategy());
423420
tree = parseFunction.apply(parser);
424421
} catch (ParseCancellationException ex) {
425422
// if we fail, parse with LL mode
426423
tokenStream.seek(0); // rewind input stream
427424
parser.reset();
428425

429426
parser.getInterpreter().setPredictionMode(PredictionMode.LL);
430-
parser.setErrorHandler(new DefaultErrorStrategy());
431427
tree = parseFunction.apply(parser);
432428
}
433429
return tree;

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/IvmNormalizeMtmv.java‎

Lines changed: 58 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,16 @@
2020
import org.apache.doris.catalog.Column;
2121
import org.apache.doris.catalog.KeysType;
2222
import org.apache.doris.catalog.OlapTable;
23+
import org.apache.doris.catalog.info.TableNameInfo;
2324
import org.apache.doris.common.Pair;
25+
import org.apache.doris.info.TableNameInfoUtils;
26+
import org.apache.doris.mtmv.MTMVPartitionUtil;
2427
import org.apache.doris.mtmv.ivm.IvmAggMeta;
2528
import org.apache.doris.mtmv.ivm.IvmAggMeta.AggTarget;
2629
import org.apache.doris.mtmv.ivm.IvmAggMeta.AggType;
2730
import org.apache.doris.mtmv.ivm.IvmNormalizeResult;
2831
import org.apache.doris.mtmv.ivm.IvmUtil;
32+
import org.apache.doris.nereids.StatementContext;
2933
import org.apache.doris.nereids.exceptions.AnalysisException;
3034
import org.apache.doris.nereids.jobs.JobContext;
3135
import org.apache.doris.nereids.trees.expressions.Alias;
@@ -108,6 +112,7 @@
108112
* <h3>Scan-level row-id injection</h3>
109113
* <ul>
110114
* <li>MOW (UNIQUE_KEYS + merge-on-write): hash(uk columns) → deterministic
115+
* <li>Excluded AGG_KEYS table: hash(agg key columns) → deterministic
111116
* <li>DUP_KEYS: uuid_numeric() → non-deterministic
112117
* <li>Other key types: not supported, throws.
113118
* </ul>
@@ -122,6 +127,7 @@ public class IvmNormalizeMtmv extends DefaultPlanRewriter<Boolean> implements Cu
122127
ImmutableSet.of(Count.class, Sum.class, Avg.class, Min.class, Max.class);
123128

124129
private final IvmNormalizeResult normalizeResult = new IvmNormalizeResult();
130+
private StatementContext statementContext;
125131

126132
@Override
127133
public Plan rewriteRoot(Plan plan, JobContext jobContext) {
@@ -133,6 +139,7 @@ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
133139
if (jobContext.getCascadesContext().getIvmNormalizeResult().isPresent()) {
134140
return plan;
135141
}
142+
statementContext = jobContext.getCascadesContext().getStatementContext();
136143
jobContext.getCascadesContext().setIvmNormalizeResult(normalizeResult);
137144
Plan result = plan.accept(this, true);
138145
normalizeResult.setNormalizedPlan(result);
@@ -468,31 +475,65 @@ private NamedExpression rewriteIvmHiddenOutput(NamedExpression output, Map<Strin
468475

469476
/**
470477
* Builds the row-id expression and returns whether it is deterministic as a pair.
471-
* - MOW: (buildRowIdHash(uk...), true) — stable across refreshes
472-
* - DUP_KEYS: (UuidNumeric(), false) — random per insert
473-
* - Other key types: throws AnalysisException
478+
* - UNIQUE_KEYS (MOW or excluded): (buildRowIdHash(uk...), true) — stable across refreshes
479+
* - DUP_KEYS: (UuidNumeric(), false) — random per insert
480+
* - Excluded AGG_KEYS: (buildRowIdHash(agg key...), true) — stable across refreshes
481+
* - Other key types: throws AnalysisException (unless excluded trigger table)
474482
*/
475483
private Pair<Expression, Boolean> buildRowId(OlapTable table, LogicalOlapScan scan) {
476484
KeysType keysType = table.getKeysType();
477-
if (keysType == KeysType.UNIQUE_KEYS && table.getEnableUniqueKeyMergeOnWrite()) {
478-
List<String> keyColNames = table.getBaseSchemaKeyColumns().stream()
479-
.map(Column::getName)
480-
.collect(Collectors.toList());
481-
List<Expression> keySlots = scan.getOutput().stream()
482-
.filter(s -> keyColNames.contains(s.getName()))
483-
.collect(Collectors.toList());
484-
if (keySlots.isEmpty()) {
485-
throw new AnalysisException("IVM: no unique key columns found for MOW table: "
486-
+ table.getName());
485+
boolean isExcludedTriggerTable = isExcludedTriggerTable(table);
486+
if (keysType == KeysType.UNIQUE_KEYS) {
487+
if (!table.getEnableUniqueKeyMergeOnWrite() && !isExcludedTriggerTable) {
488+
throw new AnalysisException(
489+
"INCREMENTAL materialized view requires UNIQUE_KEYS base tables "
490+
+ "to enable Merge-On-Write. Table '"
491+
+ table.getName() + "' has MOW disabled."
492+
+ " If this table does not participate in incremental refresh, "
493+
+ "add it to 'excluded_trigger_tables'.");
487494
}
488-
return Pair.of(IvmUtil.buildRowIdHash(keySlots), true);
495+
return buildDeterministicRowIdFromBaseKeys(table, scan);
489496
}
490497
if (keysType == KeysType.DUP_KEYS) {
491498
return Pair.of(new UuidNumeric(), false);
492499
}
493-
throw new AnalysisException("IVM does not support table key type: " + keysType
494-
+ " for table: " + table.getName()
495-
+ ". Only MOW (UNIQUE_KEYS with merge-on-write) and DUP_KEYS are supported.");
500+
if (keysType == KeysType.AGG_KEYS && isExcludedTriggerTable) {
501+
return buildDeterministicRowIdFromBaseKeys(table, scan);
502+
}
503+
if (isExcludedTriggerTable) {
504+
return Pair.of(new UuidNumeric(), false);
505+
}
506+
throw new AnalysisException(
507+
"INCREMENTAL materialized view requires base tables to be "
508+
+ "UNIQUE_KEYS with Merge-On-Write or DUP_KEYS. Table '"
509+
+ table.getName() + "' is " + keysType
510+
+ ". If this table does not participate in incremental refresh, "
511+
+ "add it to 'excluded_trigger_tables'.");
512+
}
513+
514+
private Pair<Expression, Boolean> buildDeterministicRowIdFromBaseKeys(OlapTable table, LogicalOlapScan scan) {
515+
Set<String> keyColNames = table.getBaseSchemaKeyColumns().stream()
516+
.map(Column::getName)
517+
.collect(Collectors.toSet());
518+
List<Expression> keySlots = scan.getOutput().stream()
519+
.filter(slot -> keyColNames.contains(slot.getName()))
520+
.collect(Collectors.toList());
521+
if (keySlots.isEmpty()) {
522+
throw new AnalysisException("IVM: no key columns found for "
523+
+ table.getKeysType() + " table: " + table.getName());
524+
}
525+
return Pair.of(IvmUtil.buildRowIdHash(keySlots), true);
526+
}
527+
528+
private boolean isExcludedTriggerTable(OlapTable table) {
529+
if (statementContext == null || statementContext.getExcludedTriggerTables().isEmpty()) {
530+
return false;
531+
}
532+
TableNameInfo tableNameInfo = TableNameInfoUtils.fromTableOrNull(table);
533+
if (tableNameInfo == null) {
534+
return false;
535+
}
536+
return MTMVPartitionUtil.isTableExcluded(statementContext.getExcludedTriggerTables(), tableNameInfo);
496537
}
497538

498539
/**

0 commit comments

Comments
 (0)