Skip to content

Commit 3050a9a

Browse files
authored
[fix](ivm) Invalidate the baseline when a column used by the MV is dropped (#67837)
A light schema change on an IVM base table (`DROP COLUMN`, usually followed by an `ADD COLUMN` with the same name) only changes metadata and emits no row binlog. When the dropped column is one the MV uses, the next `REFRESH ... INCREMENTAL` consumes an empty delta and reports **SUCCESS**, while the MV silently keeps the rows that were computed under the old column. Dropping the column did reach the MTMV hook, but that only moved the MV status to `SCHEMA_CHANGE`. That state merely re-analyses the MV query on the next refresh - and by then the column is usually back under the same name, so the analysis succeeds and the incremental refresh proceeds. The IVM baseline barrier (`requireCompleteBaselineRebuild`) was never raised on this path: it is only set for `REPLACE TABLE`, `REPLACE PARTITION` and partition changes. ### What this PR does Re-analyse the MV query on the `alterTable` path, right after the alter was applied, and invalidate the IVM baseline when the query no longer binds. This needs no lineage machinery: dropping or renaming a column the MV uses makes the query unanalysable, so the failure itself is the dependency signal - and dropping a column the MV does not reference leaves the incremental path untouched. Scope: on an IVM base table only `DROP COLUMN` and `RENAME TABLE` can change a referenced column at all (row binlog tables reject `MODIFY COLUMN`, `RENAME COLUMN` and `REORDER COLUMNS`), so those two operations are the whole surface. A dropped base table needs no handling - the IVM stream records the base table id and stops being usable, which already fails the refresh. The analysis runs in a context of its own rather than the session that issued the alter, because the underlying `analyzeQueryWithSql` reuses and closes the statement context it is handed.
1 parent b10f3e0 commit 3050a9a

5 files changed

Lines changed: 253 additions & 4 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java‎

Lines changed: 57 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -335,7 +335,9 @@ public void refreshComplete(MTMV mtmv, MTMVRelation relation, MTMVTask task) {
335335
*/
336336
@Override
337337
public void dropTable(Table table) {
338-
processBaseTableChange(new BaseTableInfo(table), "The base table has been deleted:");
338+
// A dropped base table is already caught by the IVM stream guard (the stream records the
339+
// base table id, so it stops being usable once the table is gone), no need to re-analyze.
340+
processBaseTableChange(new BaseTableInfo(table), "The base table has been deleted:", false);
339341
}
340342

341343
/**
@@ -347,9 +349,56 @@ public void dropTable(Table table) {
347349
public void alterTable(BaseTableInfo oldTableInfo, Optional<BaseTableInfo> newTableInfo, boolean isReplace) {
348350
// when replace, need deal two table
349351
if (isReplace) {
350-
processBaseTableChange(newTableInfo.get(), "The base table has been updated:");
352+
// REPLACE TABLE already invalidates the IVM baseline explicitly, see Alter#processReplaceTable
353+
processBaseTableChange(newTableInfo.get(), "The base table has been updated:", false);
354+
}
355+
// A RENAME leaves every column alone, and the failure it does cause -- the MV query still
356+
// spells the old name -- is already reported by the refresh itself (MTMVTask#run resolves
357+
// the base tables from the query before it ever looks at the baseline). Invalidating here
358+
// would only leave a stale flag behind: rename the table back and the query is analyzable
359+
// again, yet every strict INCREMENTAL refresh would stay rejected until a COMPLETE one ran.
360+
boolean renamed = !isReplace && newTableInfo.isPresent()
361+
&& !Objects.equals(oldTableInfo.getTableName(), newTableInfo.get().getTableName());
362+
processBaseTableChange(oldTableInfo, "The base table has been updated:", !renamed);
363+
}
364+
365+
/**
366+
* An IVM baseline is only valid while the MV query can still be analyzed against the current
367+
* base table schema. Re-analyzing the MV query here (right after the alter was applied) is what
368+
* detects a changed column identity: dropping or renaming a column the MV uses makes the query
369+
* unanalyzable, and a column re-added with the same name is a different column, so pre-existing
370+
* rows read its default value instead.
371+
*
372+
* <p>Such a change is metadata-only for light schema changes and emits no binlog, so an
373+
* incremental refresh would consume an empty delta and report SUCCESS while silently keeping the
374+
* rows computed under the old column epoch. Invalidating the baseline makes a strict INCREMENTAL
375+
* refresh fail and tell the user to run a COMPLETE refresh instead.
376+
*
377+
* <p>Only IVM is covered: a plain MTMV keeps its previous behaviour (status only).
378+
*/
379+
private void invalidateIvmBaselineIfQueryUnusable(BaseTableInfo baseTableInfo, Table mtmvTable) {
380+
if (!(mtmvTable instanceof MTMV) || !((MTMV) mtmvTable).isIvm()) {
381+
return;
382+
}
383+
MTMV mtmv = (MTMV) mtmvTable;
384+
// Analyse in a context owned by this check, never the session that issued the alter: the check
385+
// must not disturb the running statement, and it has to work on threads that have no session.
386+
// Setting a thread local is how a context is made current, so restore the previous one.
387+
ConnectContext previousCtx = ConnectContext.get();
388+
try {
389+
MTMVPlanUtil.ensureMTMVQueryUsable(mtmv,
390+
MTMVPlanUtil.createMTMVContext(mtmv, MTMVPlanUtil.DISABLE_RULES_WHEN_RUN_MTMV_TASK));
391+
} catch (Exception e) {
392+
LOG.info("Invalidate IVM baseline, the MV query is no longer usable. baseTable={}, mtmv={}, "
393+
+ "reason={}", baseTableInfo, mtmv.getName(), e.getMessage());
394+
mtmv.invalidateIvmBaseline();
395+
} finally {
396+
if (previousCtx != null) {
397+
previousCtx.setThreadLocalInfo();
398+
} else {
399+
ConnectContext.remove();
400+
}
351401
}
352-
processBaseTableChange(oldTableInfo, "The base table has been updated:");
353402
}
354403

355404
@Override
@@ -411,7 +460,8 @@ private void processBaseViewChange(BaseTableInfo baseViewInfo, String msgPrefix)
411460
}
412461
}
413462

414-
private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefix) {
463+
private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefix,
464+
boolean checkIvmQueryUsable) {
415465
Set<BaseTableInfo> mtmvsByBaseTable = getMtmvsByBaseTableOneLevelAndFromView(baseTableInfo);
416466
if (CollectionUtils.isEmpty(mtmvsByBaseTable)) {
417467
return;
@@ -424,6 +474,9 @@ private void processBaseTableChange(BaseTableInfo baseTableInfo, String msgPrefi
424474
LOG.warn(e);
425475
continue;
426476
}
477+
if (checkIvmQueryUsable) {
478+
invalidateIvmBaselineIfQueryUnusable(baseTableInfo, mtmv);
479+
}
427480
TableNameInfo tableNameInfo = new TableNameInfo(mtmv.getQualifiedDbName(),
428481
mtmv.getName());
429482
MTMVStatus status = new MTMVStatus(MTMVState.SCHEMA_CHANGE,

‎fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -422,6 +422,14 @@ public void testWithRowBinlogOpNotSupported() throws Exception {
422422
createTable(create);
423423
expectException("ALTER TABLE test." + tableName + " MODIFY COLUMN v1 BIGINT", "Table With binlog<row>");
424424

425+
// 1b) RENAME COLUMN / REORDER COLUMNS are not allowed on row binlog tables either.
426+
// This matters for IVM: dropping a column an MV references invalidates the IVM baseline, so
427+
// every other way of changing a referenced column has to be rejected here. If one of them
428+
// were ever allowed, it would become a new way to leave an MV stale without being noticed.
429+
expectException("ALTER TABLE test." + tableName + " RENAME COLUMN v1 TO v1_renamed",
430+
"Table With binlog<row>");
431+
expectException("ALTER TABLE test." + tableName + " ORDER BY (k1, v1)", "Table With binlog<row>");
432+
425433
// 2) VARIANT not supported
426434
String createVariant = "CREATE TABLE test.binlog_variant (k1 INT NOT NULL, v1 VARIANT) "
427435
+ "UNIQUE KEY(k1) DISTRIBUTED BY HASH(k1) BUCKETS 1 "

‎fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,24 @@ public void testDropPartitionMarksBaselineRebuild() throws Exception {
104104
Assertions.assertTrue(getMtmv(db).getIvmInfo().isBaselineRebuildRequired());
105105
}
106106

107+
@Test
108+
public void testDropColumnMarksBaselineRebuildOnlyWhenReferenced() throws Exception {
109+
String db = "ivm_broken_drop_column";
110+
createPartitionedIvmTableAndMv(db);
111+
MTMV mtmv = getMtmv(db);
112+
113+
// ivm_mv selects dt, k1, v1. Dropping a column it does not use must leave the baseline alone.
114+
executeSql("ALTER TABLE ivm_base ADD COLUMN spare int");
115+
executeSql("ALTER TABLE ivm_base DROP COLUMN spare");
116+
Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired());
117+
118+
// Dropping a column the MV uses makes the MV query unanalyzable: the change is metadata-only
119+
// and emits no binlog, so an incremental refresh would silently keep the rows of the old
120+
// column. The baseline has to be invalidated instead.
121+
executeSql("ALTER TABLE ivm_base DROP COLUMN v1");
122+
Assertions.assertTrue(mtmv.getIvmInfo().isBaselineRebuildRequired());
123+
}
124+
107125
@Test
108126
public void testPublishedPctPartitionUsesPartitionsBaselineRebuild() throws Exception {
109127
String db = "ivm_partitions_baseline_rebuild";
@@ -217,6 +235,24 @@ public void testRenameTableDoesNotMarkBaselineRebuild() throws Exception {
217235
Assertions.assertFalse(getMtmv(db).getIvmInfo().isBaselineRebuildRequired());
218236
}
219237

238+
@Test
239+
public void testRenameTableBackKeepsIncrementalRefreshStartable() throws Exception {
240+
String db = "ivm_broken_rename_table_back";
241+
createPartitionedIvmTableAndMv(db);
242+
243+
executeSql("ALTER TABLE ivm_base RENAME ivm_base_renamed");
244+
executeSql("ALTER TABLE ivm_base_renamed RENAME ivm_base");
245+
246+
// A rename changes no column, so it must not invalidate the baseline in either direction:
247+
// once the table is renamed back, the MV query is analyzable again and a strict INCREMENTAL
248+
// refresh has to be able to start. A "baseline rebuild required" flag left behind by the
249+
// rename would reject every one of them until a COMPLETE refresh had been run, even though
250+
// nothing the MV depends on ever changed.
251+
MTMV mtmv = getMtmv(db);
252+
Assertions.assertFalse(mtmv.getIvmInfo().isBaselineRebuildRequired());
253+
Assertions.assertDoesNotThrow(() -> mtmv.validateIvmRefreshStart(mtmv.getSchemaChangeVersion()));
254+
}
255+
220256
@Test
221257
public void testReplaceTableMarksBaselineRebuild() throws Exception {
222258
String db = "ivm_broken_replace_table";
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !mv_rows_baseline --
3+
10 2 300
4+
20 1 300
5+
6+
-- !mv_rows_after_aba --
7+
0 3 600
8+
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
import org.awaitility.Awaitility
19+
import static java.util.concurrent.TimeUnit.SECONDS
20+
21+
// Dropping a column that an IVM references, then re-adding a column with the same name
22+
// (schema ABA), used to let a strict INCREMENTAL refresh report SUCCESS while silently
23+
// keeping the rows computed under the old column epoch.
24+
//
25+
// The base table change is metadata-only (light schema change) and emits no binlog, so the
26+
// delta is empty and the refresh has nothing to apply -- the MV baseline is simply stale.
27+
//
28+
// Expected: dropping a referenced column invalidates the IVM baseline, so a strict
29+
// INCREMENTAL refresh is rejected and the user is told to run a COMPLETE refresh.
30+
// Dropping an unreferenced column must still leave the incremental path untouched.
31+
suite("test_ivm_drop_referenced_column_baseline_rebuild") {
32+
def tableName = "ivm_drop_ref_col_t"
33+
def mvName = "ivm_drop_ref_col_mv"
34+
35+
sql """DROP MATERIALIZED VIEW IF EXISTS ${mvName}"""
36+
sql """DROP TABLE IF EXISTS ${tableName}"""
37+
38+
sql """
39+
CREATE TABLE ${tableName} (
40+
id BIGINT NOT NULL,
41+
grp INT NULL,
42+
amount BIGINT NULL,
43+
spare INT NULL
44+
)
45+
UNIQUE KEY(id)
46+
DISTRIBUTED BY HASH(id) BUCKETS 1
47+
PROPERTIES (
48+
"replication_num" = "1",
49+
"enable_unique_key_merge_on_write" = "true",
50+
"binlog.enable" = "true",
51+
"binlog.format" = "ROW",
52+
"binlog.need_historical_value" = "true"
53+
)
54+
"""
55+
sql """INSERT INTO ${tableName} VALUES (1, 10, 100, 7), (2, 10, 200, 8), (3, 20, 300, 9)"""
56+
57+
sql """
58+
CREATE MATERIALIZED VIEW ${mvName}
59+
BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
60+
KEY(grp)
61+
DISTRIBUTED BY HASH(grp) BUCKETS 1
62+
PROPERTIES ("replication_num" = "1")
63+
AS SELECT grp, COUNT(*) AS cnt, SUM(amount) AS total
64+
FROM ${tableName} GROUP BY grp
65+
"""
66+
67+
def ddlJobCount = { String table ->
68+
return sql("""SHOW ALTER TABLE COLUMN WHERE TableName = '${table}'""").size()
69+
}
70+
71+
// `SHOW ALTER TABLE COLUMN` keeps finished jobs, so wait for a *new* job that is FINISHED.
72+
def waitDdlFinished = { String table, int previousJobCount ->
73+
Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({
74+
def jobs = sql """SHOW ALTER TABLE COLUMN WHERE TableName = '${table}'"""
75+
return jobs.size() > previousJobCount
76+
&& jobs.every({ row -> row[9].toString() == 'FINISHED' })
77+
})
78+
}
79+
80+
def lastTaskId = null
81+
// tasks('type'='mv') can briefly miss the just-finished task, so require a *new* TaskId.
82+
def waitTerminalTask = { String mv ->
83+
def taskResult
84+
Awaitility.await().atMost(300, SECONDS).pollInterval(2, SECONDS).until({
85+
taskResult = sql_return_maparray("""
86+
SELECT TaskId, Status, RefreshMode, IvmFallbackReason, ErrorMsg
87+
FROM tasks('type'='mv')
88+
WHERE MvDatabaseName = '${context.dbName}' AND MvName = '${mv}'
89+
ORDER BY CreateTime DESC, TaskId DESC LIMIT 1
90+
""")
91+
return !taskResult.isEmpty()
92+
&& taskResult[0].TaskId.toString() != lastTaskId
93+
&& taskResult[0].Status.toString() != 'PENDING'
94+
&& taskResult[0].Status.toString() != 'RUNNING'
95+
})
96+
lastTaskId = taskResult[0].TaskId.toString()
97+
return taskResult[0]
98+
}
99+
100+
// ---------------------------------------------------------------- 1. baseline
101+
sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE"""
102+
def task = waitTerminalTask(mvName)
103+
assertEquals("SUCCESS", task.Status.toString(), "baseline COMPLETE refresh: " + task.ErrorMsg)
104+
order_qt_mv_rows_baseline "SELECT grp, cnt, total FROM ${mvName}"
105+
106+
// ------------------------------------- 2. unreferenced column: no baseline invalidation
107+
def before = ddlJobCount(tableName)
108+
sql """ALTER TABLE ${tableName} DROP COLUMN spare"""
109+
waitDdlFinished(tableName, before)
110+
111+
sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL"""
112+
task = waitTerminalTask(mvName)
113+
assertEquals("SUCCESS", task.Status.toString(),
114+
"dropping an unreferenced column must not invalidate the IVM baseline: " + task.ErrorMsg)
115+
116+
// ---------------------------------------- 3. referenced column: strict INCREMENTAL rejected
117+
before = ddlJobCount(tableName)
118+
sql """ALTER TABLE ${tableName} DROP COLUMN grp"""
119+
waitDdlFinished(tableName, before)
120+
121+
sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL"""
122+
task = waitTerminalTask(mvName)
123+
assertEquals("FAILED", task.Status.toString(),
124+
"dropping a referenced column must reject a strict INCREMENTAL refresh")
125+
126+
// -------------------------------- 4. same-name re-add (schema ABA) is still rejected
127+
before = ddlJobCount(tableName)
128+
sql """ALTER TABLE ${tableName} ADD COLUMN grp INT NULL DEFAULT '0'"""
129+
waitDdlFinished(tableName, before)
130+
131+
sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL"""
132+
task = waitTerminalTask(mvName)
133+
assertEquals("FAILED", task.Status.toString(),
134+
"schema ABA must not be silently accepted by a strict INCREMENTAL refresh")
135+
assertTrue(task.ErrorMsg.toString().contains("baseline rebuild is pending"),
136+
"expected a pending baseline rebuild hint, got: " + task.ErrorMsg)
137+
138+
// ------------------------------------ 5. COMPLETE rebuild reflects current base semantics
139+
// Every pre-existing row now reads the re-added column's default value.
140+
sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE"""
141+
task = waitTerminalTask(mvName)
142+
assertEquals("SUCCESS", task.Status.toString(), "COMPLETE rebuild after ABA: " + task.ErrorMsg)
143+
order_qt_mv_rows_after_aba "SELECT grp, cnt, total FROM ${mvName}"
144+
}

0 commit comments

Comments
 (0)