Skip to content

Commit cc473a3

Browse files
morningmanclaude
andcommitted
[fix](ivm) Fall back to COMPLETE when an IVM stream offset has expired
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: An IVM materialized view reads each unique-key (merge-on-write) base table through an internal table stream created with MIN_DELTA. Planning a MIN_DELTA read refuses to start from an offset that the base table's `changelog.ttl_seconds` has expired: `ChangelogUtils.effectiveStartTso` throws "Changelog offset has expired according to changelog.ttl_seconds". The incremental refresh turns a failure into a fallback only when `IvmFailureClassifier` recognizes its message, and this message was not one of them. `IvmIncrRefreshManager.doRefresh` rethrew it, so the task failed even when the refresh allowed a fallback (`REFRESH ... AUTO`, `INCREMENTAL FALLBACK`, the scheduled refresh of a `REFRESH INCREMENTAL FALLBACK` MV). The offset does not move on a failed refresh, so while the TTL stays as it is every later refresh fails the same way, until someone runs a manual COMPLETE refresh. `IvmFailureReason.CHANGELOG_BROKEN` had no producer in production code, although `MTMVTask.handleIvmFallbackResult` already sends it straight to COMPLETE. The delta is not the only read of that offset. When some MV partitions need a rebuild (for example after a TRUNCATE of a base partition), the incremental attempt first rebuilds them with a partition refresh, and that refresh reads the tables the MV does not partition by through their streams at the offsets (`StreamReadMode.SNAPSHOT`). With such an offset expired, the rebuild failed with the same message before the delta ran, and that failure went straight out of the attempt as well. How to reproduce: create a base table with the changelog enabled and a large TTL, build an IVM MV on it, then set `changelog.ttl_seconds` to 1, wait two seconds, insert a row and run `REFRESH MATERIALIZED VIEW mv AUTO`. | Refresh after the offset expired | Before | After | | --- | --- | --- | | `REFRESH ... INCREMENTAL` (strict) | FAILED, `IvmFallbackReason` empty | FAILED, `IvmFallbackReason = CHANGELOG_BROKEN`, the error says `reason=CHANGELOG_BROKEN` and keeps the offset and the cutoff TSO | | `REFRESH ... AUTO` | FAILED, the MV keeps the old rows | SUCCESS through COMPLETE, `IvmFallbackReason = CHANGELOG_BROKEN` | | the next `REFRESH ... AUTO` | fails again while the TTL is unchanged | SUCCESS, incremental | | `REFRESH ... AUTO` that first rebuilds a dirty MV partition while a joined table's offset has expired | FAILED | SUCCESS through COMPLETE, `IvmFallbackReason = CHANGELOG_BROKEN` (a strict request fails with that reason) | The fix classifies that message as `CHANGELOG_BROKEN`, matching the constant `ChangelogUtils` throws with rather than a copy of it, and the rebuild before the delta answers that failure through `handleIvmFallbackResult` the way the delta does; any other rebuild failure still fails the attempt. COMPLETE is the only attempt that resets the stream baselines, which is why the reason goes there directly instead of trying a partition refresh first: a partition refresh still reads the streams at their offsets. The comment and the log of that branch said "previous run incomplete", which is not what the reason means; they now describe the expired changelog, and the log carries the detail with both TSOs. ### Release note Fix an IVM materialized view refresh failing on every run after a base table's `changelog.ttl_seconds` expired the stream offset. A refresh that allows fallback now rebuilds the MV with COMPLETE; a strict INCREMENTAL refresh fails with reason `CHANGELOG_BROKEN`. ### Check List (For Author) - Test: Regression test / Unit Test - Regression test: `mtmv_p0/ivm/test_ivm_changelog_offset_expired` (new, nonConcurrent), covering the delta and the rebuild before it. Run on a local shared-nothing cluster. The delta case was first run on the base commit, where strict and AUTO both FAILED; the rebuild case was reproduced by hand on a build that had only the classifier change (AUTO FAILED on the rebuild). With the whole fix the suite passes, twice in a row against the generated `.out`. Not run on a cloud cluster. - Unit Test: `IvmIncrRefreshManagerTest` (new case for the expired offset), `MTMVTaskTest` (two new cases for the rebuild, strict and fallback), `IvmFailureReasonTest`, `ChangelogUtilsTest`: 94 tests pass. - Behavior changed: Yes. A refresh that allows fallback recovers with COMPLETE instead of failing; a strict INCREMENTAL refresh reports `CHANGELOG_BROKEN`. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 62e3d81 commit cc473a3

7 files changed

Lines changed: 411 additions & 5 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@
6666
import org.apache.doris.mtmv.MTMVRelatedTableIf;
6767
import org.apache.doris.mtmv.MTMVRelation;
6868
import org.apache.doris.mtmv.MTMVUtil;
69+
import org.apache.doris.mtmv.ivm.IvmFailureClassifier;
6970
import org.apache.doris.mtmv.ivm.IvmFailureReason;
7071
import org.apache.doris.mtmv.ivm.IvmIncrRefreshContext;
7172
import org.apache.doris.mtmv.ivm.IvmIncrRefreshManager;
@@ -828,6 +829,17 @@ private AttemptResultType executeIvmAttempt(MTMVRefreshContext refreshContext,
828829
this.refreshMode = generateRefreshMode(toRebuild);
829830
try {
830831
executePartitionBasedRefresh(refreshContext, RefreshMode.PARTITIONS, ctx, toRebuild);
832+
} catch (JobException e) {
833+
// The rebuild reads the tables the MV does not partition by through their streams at the
834+
// offsets, so an offset changelog.ttl_seconds has expired stops it the way it stops the delta
835+
// below -- and it is the same failure of this attempt, answered the same way.
836+
String detail = Util.getRootCauseMessage(e);
837+
if (IvmFailureClassifier.classifyExecutionFailure(detail)
838+
.filter(reason -> reason == IvmFailureReason.CHANGELOG_BROKEN).isEmpty()) {
839+
throw e;
840+
}
841+
return handleIvmFallbackResult(
842+
IvmIncrRefreshResult.fallback(IvmFailureReason.CHANGELOG_BROKEN, detail), request);
831843
} finally {
832844
recordRebuiltPartitions(request);
833845
}
@@ -1184,11 +1196,12 @@ private AttemptResultType handleIvmFallbackResult(IvmIncrRefreshResult ivmResult
11841196
+ ", detail=" + ivmResult.getDetailMessage());
11851197
}
11861198
if (ivmResult.getFailureReason() == IvmFailureReason.CHANGELOG_BROKEN) {
1187-
// The previous task already entered the IVM execution phase. If
1188-
// fallback is allowed, jump directly to COMPLETE recovery instead of
1189-
// trying PARTITIONS first.
1190-
LOG.warn("IVM previous run incomplete for mv={}, taskId={}. Continuing with COMPLETE recovery.",
1191-
mtmv.getName(), getTaskId());
1199+
// changelog.ttl_seconds has expired the offset of a stream this attempt reads, so no incremental
1200+
// attempt can catch the MV up. A partition refresh still reads the streams at their offsets, so
1201+
// go directly to COMPLETE: it is the only attempt that resets the stream baselines.
1202+
LOG.warn("IVM stream changelog expired for mv={}, detail={}, taskId={}. "
1203+
+ "Continuing with COMPLETE refresh.",
1204+
mtmv.getName(), ivmResult.getDetailMessage(), getTaskId());
11921205
return AttemptResultType.FALLBACK_TO_COMPLETE;
11931206
}
11941207
if (ivmResult.getFailureReason().requiresCompleteRefresh()) {

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmFailureClassifier.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717

1818
package org.apache.doris.mtmv.ivm;
1919

20+
import org.apache.doris.changelog.ChangelogUtils;
21+
2022
import java.util.Objects;
2123
import java.util.Optional;
2224

@@ -48,6 +50,11 @@ public static Optional<IvmFailureReason> classifyExecutionFailure(String detail)
4850
if (detail.contains(MV_PARTITION_NOT_FOUND_MSG)) {
4951
return Optional.of(IvmFailureReason.MV_PARTITION_NOT_FOUND);
5052
}
53+
// Planning a MIN_DELTA read refuses an offset that changelog.ttl_seconds has expired: the changelog
54+
// the stream has to read from is gone, so no delta can catch the MV up.
55+
if (detail.contains(ChangelogUtils.CHANGELOG_OFFSET_EXPIRED)) {
56+
return Optional.of(IvmFailureReason.CHANGELOG_BROKEN);
57+
}
5158
return Optional.empty();
5259
}
5360
}

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmFailureReason.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
/** Reasons IVM analysis or execution can fail. */
2121
public enum IvmFailureReason {
2222
// ATTN(changelog-rename): formerly "BINLOG_BROKEN"
23+
/** The changelog a stream has to read from is outside the retention: changelog.ttl_seconds expired its offset. */
2324
CHANGELOG_BROKEN,
2425
// ATTN(changelog-rename): formerly "BINLOG_NOT_ENABLED"
2526
CHANGELOG_NOT_ENABLED,

‎fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.doris.catalog.Partition;
2828
import org.apache.doris.catalog.info.TableNameInfo;
2929
import org.apache.doris.catalog.stream.OlapTableStream;
30+
import org.apache.doris.changelog.ChangelogUtils;
3031
import org.apache.doris.common.AnalysisException;
3132
import org.apache.doris.common.Config;
3233
import org.apache.doris.common.DdlException;
@@ -1775,6 +1776,91 @@ public void testExecuteIvmAttemptFallsBackToCompleteForBrokenBaseline() throws E
17751776
}
17761777
}
17771778

1779+
/**
1780+
* The rebuild an incremental attempt runs first, for the MV partitions that need one, reads the tables the MV
1781+
* does not partition by through their streams at the offsets. An offset changelog.ttl_seconds has expired stops
1782+
* it the way it stops the delta, so the answer is the same: a request that may fall back goes to COMPLETE, the
1783+
* only attempt that resets the stream baselines.
1784+
*/
1785+
@Test
1786+
public void testExpiredOffsetInTheRebuildBeforeTheDeltaFallsBackToComplete() throws Exception {
1787+
MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of(MTMVTaskTriggerMode.MANUAL, null,
1788+
RefreshMode.INCREMENTAL, true, null));
1789+
1790+
Object result = runIvmAttemptWhoseRebuildReadsAnExpiredOffset(task);
1791+
1792+
Assertions.assertEquals("FALLBACK_TO_COMPLETE", result.toString());
1793+
Assertions.assertEquals(IvmFailureReason.CHANGELOG_BROKEN.name(),
1794+
Deencapsulation.getField(task, "ivmFallbackReason"));
1795+
}
1796+
1797+
/** A request that may not fall back fails instead, with the reason it would have fallen back for. */
1798+
@Test
1799+
public void testExpiredOffsetInTheRebuildBeforeTheDeltaFailsAStrictRequest() {
1800+
MTMVTask task = new MTMVTask(mtmv, relation, MTMVTaskContext.of(MTMVTaskTriggerMode.MANUAL, null,
1801+
RefreshMode.INCREMENTAL, false, null));
1802+
1803+
JobException failure = Assertions.assertThrows(JobException.class,
1804+
() -> runIvmAttemptWhoseRebuildReadsAnExpiredOffset(task));
1805+
1806+
Assertions.assertTrue(failure.getMessage().contains("reason=" + IvmFailureReason.CHANGELOG_BROKEN));
1807+
Assertions.assertTrue(failure.getMessage().contains(ChangelogUtils.CHANGELOG_OFFSET_EXPIRED));
1808+
Assertions.assertEquals(IvmFailureReason.CHANGELOG_BROKEN.name(),
1809+
Deencapsulation.getField(task, "ivmFallbackReason"));
1810+
}
1811+
1812+
/**
1813+
* Runs the incremental attempt with p1 needing a rebuild and p2 caught up, the rebuild's read refused the way
1814+
* planning refuses a MIN_DELTA read from an expired offset. The delta must not be reached.
1815+
*/
1816+
private Object runIvmAttemptWhoseRebuildReadsAnExpiredOffset(MTMVTask task) throws Exception {
1817+
Mockito.when(mtmv.isIvm()).thenReturn(true);
1818+
Mockito.when(mtmv.getName()).thenReturn("test_mv");
1819+
Mockito.when(mtmv.hasRefreshSnapshot()).thenReturn(true);
1820+
Mockito.when(mtmv.getRefreshPartitionNum()).thenReturn(1);
1821+
Mockito.when(mtmvPartitionInfo.getPctInfos()).thenReturn(Collections.emptyList());
1822+
Mockito.when(mtmv.getPartitionNames()).thenReturn(Sets.newHashSet(poneName, ptwoName));
1823+
Mockito.when(mtmv.getPartitionStates()).thenReturn(Maps.newHashMap(Map.of(
1824+
poneName, new MTMVPartitionState(1, 2),
1825+
ptwoName, new MTMVPartitionState(2, 2))));
1826+
MTMVRefreshContext refreshContext = Mockito.mock(MTMVRefreshContext.class);
1827+
Mockito.when(refreshContext.getByPartitionName(Mockito.anyString())).thenReturn(Collections.emptyMap());
1828+
mtmvPartitionUtilStatic.when(() -> MTMVPartitionUtil.generatePartitionSnapshots(
1829+
Mockito.same(refreshContext), Mockito.anySet(), Mockito.anySet()))
1830+
.thenReturn(Collections.emptyMap());
1831+
RuntimeException expired = Assertions.assertThrows(RuntimeException.class,
1832+
() -> ChangelogUtils.effectiveStartTso(99L, 100L, true));
1833+
1834+
ConnectContext mtmvCtx = new ConnectContext();
1835+
mtmvCtx.setQueryId(new TUniqueId(1L, 2L));
1836+
mtmvCtx.setThreadLocalInfo();
1837+
UpdateMvByPartitionCommand command = Mockito.mock(UpdateMvByPartitionCommand.class);
1838+
try (MockedStatic<MTMVPlanUtil> mtmvPlanUtilStatic = Mockito.mockStatic(MTMVPlanUtil.class);
1839+
MockedStatic<UpdateMvByPartitionCommand> updateMvStatic
1840+
= Mockito.mockStatic(UpdateMvByPartitionCommand.class);
1841+
MockedConstruction<IvmIncrRefreshManager> delta
1842+
= Mockito.mockConstruction(IvmIncrRefreshManager.class)) {
1843+
mtmvPlanUtilStatic.when(() -> MTMVPlanUtil.createMTMVContext(Mockito.eq(mtmv), Mockito.anyList()))
1844+
.thenReturn(mtmvCtx);
1845+
updateMvStatic.when(() -> UpdateMvByPartitionCommand.from(
1846+
Mockito.eq(mtmv), Mockito.anySet(), Mockito.anyMap(), Mockito.any(StatementContext.class)))
1847+
.thenReturn(command);
1848+
mtmvPlanUtilStatic.when(() -> MTMVPlanUtil.executeCommand(
1849+
Mockito.eq(mtmvCtx), Mockito.eq(command), Mockito.any(StatementContext.class),
1850+
Mockito.anyString(), Mockito.any(Consumer.class))).thenThrow(expired);
1851+
1852+
Object request = Deencapsulation.invoke(task, "resolveRefreshRequest");
1853+
try {
1854+
return Deencapsulation.invoke(task, "executeIvmAttempt", refreshContext, request, mtmvCtx,
1855+
Lists.newArrayList());
1856+
} finally {
1857+
Assertions.assertTrue(delta.constructed().isEmpty());
1858+
}
1859+
} finally {
1860+
ConnectContext.remove();
1861+
}
1862+
}
1863+
17781864

17791865

17801866

‎fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManagerTest.java‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@
2020
import org.apache.doris.catalog.Column;
2121
import org.apache.doris.catalog.MTMV;
2222
import org.apache.doris.catalog.info.TableNameInfo;
23+
import org.apache.doris.changelog.ChangelogUtils;
2324
import org.apache.doris.mtmv.MTMVPlanUtil;
2425
import org.apache.doris.nereids.StatementContext;
2526
import org.apache.doris.nereids.analyzer.UnboundTableSink;
27+
import org.apache.doris.nereids.exceptions.AnalysisException;
2628
import org.apache.doris.nereids.trees.plans.commands.Command;
2729
import org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand;
2830
import org.apache.doris.qe.ConnectContext;
@@ -172,6 +174,25 @@ public void testManagerReturnsFallbackWithKnownExecutionFailureReason() throws E
172174
IvmFailureClassifier.NON_DETERMINISTIC_ROW_ID_MSG_PREFIX);
173175
}
174176

177+
@Test
178+
public void testManagerReturnsChangelogBrokenFallbackForExpiredStreamOffset() throws Exception {
179+
// What planning a MIN_DELTA read throws once changelog.ttl_seconds has expired the stream offset.
180+
AnalysisException expired = Assertions.assertThrows(AnalysisException.class,
181+
() -> ChangelogUtils.effectiveStartTso(99L, 100L, true));
182+
MTMV mtmv = mockMtmv();
183+
Command cmd = Mockito.mock(Command.class);
184+
TestIvmIncrRefreshManager manager = new TestIvmIncrRefreshManager(newContext(mtmv), makeCommands(cmd));
185+
manager.failureMessage = expired.getMessage();
186+
187+
IvmIncrRefreshResult result = manager.doRefresh(manager.context);
188+
189+
Assertions.assertFalse(result.isSuccess());
190+
Assertions.assertEquals(IvmFailureReason.CHANGELOG_BROKEN, result.getFailureReason());
191+
// A strict refresh reports this detail, so it has to keep the offset and the cutoff.
192+
Assertions.assertTrue(result.getDetailMessage().contains(expired.getMessage()));
193+
Assertions.assertTrue(manager.executeCalled);
194+
}
195+
175196
@Test
176197
public void testManagerReturnsChangelogNotEnabledFallbackOnIvmException() throws Exception {
177198
MTMV mtmv = mockMtmv();
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
-- This file is automatically generated. You should know what you did if you want to edit this
2+
-- !baseline_task --
3+
SUCCESS COMPLETE NONE false false
4+
5+
-- !baseline_mv --
6+
1 10
7+
2 20
8+
9+
-- !strict_task --
10+
FAILED NONE CHANGELOG_BROKEN true true
11+
12+
-- !mv_after_strict --
13+
1 10
14+
2 20
15+
16+
-- !fallback_task --
17+
SUCCESS COMPLETE CHANGELOG_BROKEN false false
18+
19+
-- !mv_after_fallback --
20+
1 10
21+
2 20
22+
3 30
23+
24+
-- !task_after_fallback --
25+
SUCCESS NONE NONE false false
26+
27+
-- !mv_after_a_further_refresh --
28+
1 10
29+
2 20
30+
3 30
31+
4 40
32+
33+
-- !partitioned_baseline_mv --
34+
1 2026-01-10 100 10
35+
2 2026-02-10 200 10
36+
37+
-- !rebuild_strict_task --
38+
FAILED PARTIAL CHANGELOG_BROKEN true true
39+
40+
-- !partitioned_mv_after_strict --
41+
1 2026-01-10 100 10
42+
2 2026-02-10 200 10
43+
44+
-- !rebuild_fallback_task --
45+
SUCCESS COMPLETE CHANGELOG_BROKEN false false
46+
47+
-- !partitioned_mv_after_fallback --
48+
1 2026-01-10 100 20
49+
2 2026-02-10 200 20
50+
51+
-- !partitioned_task_after_fallback --
52+
SUCCESS NONE NONE false false
53+
54+
-- !partitioned_mv_after_a_further_refresh --
55+
1 2026-01-10 100 30
56+
2 2026-02-10 200 20
57+

0 commit comments

Comments
 (0)