Skip to content

Commit 9dc9dc7

Browse files
authored
[feature](ivm) Report how a refresh task refreshed an IVM MV, and count the refresh tasks on the Master (#69856)
### What problem does this PR solve? Issue Number: #69907 Related PR: None Problem Summary: **In short.** A refresh of an IVM materialized view can keep the MV up incrementally or quietly recompute it, and nothing tells the two apart: `tasks("type"="mv")` does not say which kinds of work a task committed, and `/metrics` has no IVM series at all. This PR makes each refresh task of an IVM MV report how it refreshed the MV (`IvmRefreshMethod` in `tasks()`) and why it failed or fell back, keeps a refresh summary on the MV for the IVM status table that follows, and counts the tasks on the Master in five new counters. **Background.** An IVM MV is refreshed by a task that runs one or more attempts: the incremental attempt (the delta of its base tables' streams, after rebuilding the MV partitions an invalidation marked), a partition rebuild, and a whole-MV rebuild. A request that may fall back (AUTO, INCREMENTAL FALLBACK, PARTITIONS FALLBACK) moves on to the next attempt when one gives up, for example to COMPLETE after the delta meets a MIN/MAX boundary. `tasks()` already reports `RefreshMode` (which partitions the task covered), `IvmFallbackReason` (why the incremental attempt gave up, or why a strict request failed) and `IvmRebuiltPartitions`. **The problem, and what it cost.** - An operator cannot tell a refresh that kept the MV up incrementally from one that recomputed it. `RefreshMode` says `COMPLETE` for any whole-MV scope, and `IvmFallbackReason` is set even when the attempt that replaced the delta wrote nothing, or when a strict request failed. - No alert can watch IVM refreshes. The asynchronous MV counters (`async_materialized_view_task_success_num`, `_failed_num`, `_skip_num`) cover every MV and say nothing about the method, the fallbacks, or why tasks fail. An MV that falls back to COMPLETE on every run, the expensive failure mode of IVM, goes unnoticed until the load of the cluster shows it. - A task that the full MV task queue turns away never runs, and nothing counts it. - The task history keeps only `max_persistence_task_count` tasks, so how many times in a row an MV has failed, or when it last fell back, cannot be read back from it. **How this PR fixes it.** - `MTMVTask` notes each batch that commits data (the delta, a partition rebuild, a whole-MV rebuild) and decides `IvmRefreshMethod` once, as the task publishes its result: `NONE`, `INCREMENTAL`, `PARTITION_REBUILD`, `COMPLETE`, or `MIXED` when more than one kind committed, typically the rebuild of invalidated partitions before the delta. `tasks("type"="mv")` gets the `IvmRefreshMethod` column, NULL while a task runs and for a task of an MV that is not an IVM MV. - A task falls back when the incremental attempt gives up and an attempt replacing it commits data. Not fallbacks: building the first baseline of an MV no refresh has succeeded on, the rebuild of invalidated partitions before the delta, an explicit COMPLETE or PARTITIONS request, and a fallback that commits nothing. - A failed task reports why: the `IvmFailureReason` a request that may not fall back failed with, or else the stage it failed in, `QUERY_UNUSABLE`, `PARTITION_SYNC_FAILED`, `EXECUTION_FAILED` or `OTHER`, told apart by where the failure was thrown rather than by its message. - The method, the failure reason and whether the task fell back are journaled with the task, as new Gson fields that an older FE ignores. The MV keeps an `IvmRefreshSummary`, updated from each task result in `MTMV.addTaskResult`, which the Master and the replay of the same journal record run alike, from what the record carries: consecutive failures, the latest success and failure (with the reason and the error message, cut to 512 characters), the latest method and run time, and the fallback count with the latest reason. A canceled task leaves it as it is. Nothing reads it in this PR; `information_schema.ivm_mv_status` and the per-MV series of a follow-up PR do. - Five counters, counted on the Master as each task of an IVM MV ends. Every label combination is registered up front, so each of the 61 series exists from 0: | Series | Type | Labels | Value | | --- | --- | --- | --- | | `doris_fe_ivm_refresh_tasks_total` | counter | `result`=success/no_change/failed/canceled, `method`=none/incremental/partition_rebuild/complete/mixed | refresh tasks that ended; `no_change` is a SUCCESS that committed nothing | | `doris_fe_ivm_refresh_duration_seconds_total` | counter | `method` | the run time, StartTime to FinishTime, of the tasks that succeeded or failed; a canceled task has no finish time | | `doris_fe_ivm_fallbacks_total` | counter | `reason`=the `IvmFailureReason` values, `incomplete_refresh_snapshot` | tasks that fell back | | `doris_fe_ivm_refresh_failures_total` | counter | `reason`=the `IvmFailureReason` values, the four stages | tasks that failed | | `doris_fe_ivm_schedule_rejected_total` | counter | | tasks the full MV task queue turned away | Like the other FE event counters they live in the registry of every FE and only the Master counts in them, so a query sums them over the FEs; a restart or a new Master starts them from 0. - `AbstractTask.onRejected`, a new hook that `TaskProcessor.addTask` calls when the executor queue rejects a task. Both the instant dispatch (`JobScheduler`) and the timer dispatch (`DispatchTaskHandler`) add tasks through it, and `MTMVTask` counts the rejection for an IVM MV. A timer-dispatched task that is turned away is canceled as before, so it also counts as `canceled`. **Results.** Before this PR, `tasks()` has no method column and `/metrics` has none of these series. After it, on a local cluster, one IVM MV refreshed through each case (regression `test_ivm_refresh_method_metrics`): | Refresh | `tasks()`: Status, IvmRefreshMethod, IvmFallbackReason | Counters that moved, besides the run time | | --- | --- | --- | | first AUTO refresh of a new MV | SUCCESS, COMPLETE, NULL | `tasks{success,complete}` | | INCREMENTAL after an insert | SUCCESS, INCREMENTAL, NULL | `tasks{success,incremental}` | | INCREMENTAL with nothing new | SUCCESS, NONE, NULL | `tasks{no_change,none}` | | INCREMENTAL after deleting the row that holds a group's minimum | FAILED, NONE, MIN_MAX_BOUNDARY_HIT | `tasks{failed,none}`, `failures{min_max_boundary_hit}` | | INCREMENTAL FALLBACK in the same state | SUCCESS, COMPLETE, MIN_MAX_BOUNDARY_HIT | `tasks{success,complete}`, `fallbacks{min_max_boundary_hit}` | | COMPLETE | SUCCESS, COMPLETE, NULL | `tasks{success,complete}` | | INCREMENTAL after truncating a base partition and inserting into another | SUCCESS, MIXED, NULL | `tasks{success,mixed}` | The first refresh of a new MV finds every partition still to be filled and goes straight to COMPLETE; it counts no fallback. The MV holds what its base table does after each case. **Classes, and how they call each other.** - `IvmRefreshMethod`, `IvmRefreshFailureStage` (`org.apache.doris.mtmv.ivm`): the methods, and the stages a failure is reported by. - `MTMVTask`: notes committed batches (`recordCommitted`), decides whether giving up the incremental attempt is a fallback (`givingUpTheIncrementalAttemptIsAFallback`), moves the failure stage along, settles the method and the failure reason in `after()`, and counts the task (`recordIvmRefreshMetrics`) and a rejection (`onRejected`). - `IvmRefreshSummary` (`org.apache.doris.mtmv.ivm`): persisted on `MTMV` and updated by `update(MTMVTask)` from `MTMV.addTaskResult`; `MTMV.hasSucceededIvmRefresh` tells the first baseline apart. - `MetricRepo`: registers the five counters. `DoubleCounterMetric` (new) keeps a total in seconds without rounding each addition. - `AbstractTask.onRejected` (new, empty by default), called by `TaskProcessor.addTask`. ``` MTMVTask.run() worker thread ├─ executeIvmAttempt │ ├─ rebuild invalidated partitions ──► recordCommitted(PARTITION_REBUILD) │ └─ delta (IvmIncrRefreshManager) ───► recordCommitted(INCREMENTAL) │ gave up ─► fallingBack = givingUpTheIncrementalAttemptIsAFallback() ├─ PARTITIONS / COMPLETE attempts ───────► recordCommitted(...) ivmFallback once fallingBack └─ failureStage: QUERY_UNUSABLE ─► PARTITION_SYNC_FAILED ─► EXECUTION_FAILED, or an IvmFailureReason onSuccess / onFail / cancel ─► after() ├─ ivmRefreshMethod, ivmFailureReason settled ├─ recordIvmRefreshMetrics() ─► MetricRepo.COUNTER_IVM_* Master only └─ Env.addMTMVTaskResult ─► MTMV.addTaskResult ─► IvmRefreshSummary.update(task) (journaled as ADD_TASK; the replay calls addTaskResult the same way) TaskProcessor.addTask ── queue full ─► task.onRejected() ─► MTMVTask: COUNTER_IVM_SCHEDULE_REJECTED ``` Nothing else changes: no thrift or proto change, no new config, and the existing columns and series stay as they are.
1 parent fcf143c commit 9dc9dc7

17 files changed

Lines changed: 1581 additions & 2 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
import org.apache.doris.mtmv.MTMVStatus;
5757
import org.apache.doris.mtmv.MTMVUtil;
5858
import org.apache.doris.mtmv.ivm.IvmInfo;
59+
import org.apache.doris.mtmv.ivm.IvmRefreshSummary;
5960
import org.apache.doris.mtmv.ivm.IvmUtil;
6061
import org.apache.doris.nereids.StatementContext;
6162
import org.apache.doris.nereids.rules.analysis.SessionVarGuardRewriter;
@@ -126,6 +127,10 @@ public class MTMV extends OlapTable {
126127
*/
127128
@SerializedName("pst")
128129
private Map<String, MTMVPartitionState> partitionStates = Maps.newLinkedHashMap();
130+
// What the refresh tasks of an IVM MV have come to; updated with each task result, see addTaskResult. An MV
131+
// that is not an IVM MV leaves it empty.
132+
@SerializedName("irs")
133+
private IvmRefreshSummary ivmRefreshSummary = new IvmRefreshSummary();
129134
// Increased every time rewrite cache is invalidated to prevent publishing stale in-flight cache builds.
130135
private transient long rewriteCacheGeneration;
131136
private long schemaChangeVersion;
@@ -320,6 +325,19 @@ public boolean isIvm() {
320325
return getIvmInfo().isEnableIvm();
321326
}
322327

328+
/**
329+
* Whether a refresh task of this IVM MV has ever succeeded. Until one has, the MV has no baseline for the
330+
* incremental refresh to keep up, and the refresh that builds it is not the incremental refresh falling back.
331+
*/
332+
public boolean hasSucceededIvmRefresh() {
333+
readMvLock();
334+
try {
335+
return ivmRefreshSummary.hasSucceeded();
336+
} finally {
337+
readMvUnlock();
338+
}
339+
}
340+
323341
/**
324342
* Whether this MV is an IVM MV built as UNIQUE KEY + merge-on-write, which is one of the two table
325343
* models: it is maintained through the hidden row-id, and the layout its refresh reads is pinned
@@ -447,6 +465,11 @@ public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) {
447465
this.status.setRefreshState(MTMVRefreshState.FAIL);
448466
}
449467
this.jobInfo.addHistoryTask(task);
468+
if (ivmInfo.isEnableIvm()) {
469+
// From the task as the journal carries it, here and in the replay alike, so that the summary does
470+
// not depend on how many tasks the history keeps and a follower that takes over has the same one.
471+
ivmRefreshSummary.update(task);
472+
}
450473
compatiblePctSnapshot(partitionSnapshots);
451474
// What this task wrote is described by the epochs just recorded, so a partition the result
452475
// left dirty is left out: its snapshot would otherwise come back after an invalidation

‎fe/fe-core/src/main/java/org/apache/doris/job/executor/TaskProcessor.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ public boolean addTask(AbstractTask task) {
5353
return true;
5454
} catch (RejectedExecutionException e) {
5555
log.warn("Failed to add task to executor, task id: {}", task.getTaskId(), e);
56+
task.onRejected();
5657
return false;
5758
}
5859
}

‎fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java‎

Lines changed: 160 additions & 2 deletions
Large diffs are not rendered by default.

‎fe/fe-core/src/main/java/org/apache/doris/job/task/AbstractTask.java‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,13 @@ public boolean cancel(boolean needWaitCancelComplete) throws JobException {
163163
}
164164
}
165165

166+
/**
167+
* Called when the executor turns this task away because its queue is full, so the task never runs. The caller
168+
* goes on to fail or cancel it as it would any other task.
169+
*/
170+
public void onRejected() {
171+
}
172+
166173
/**
167174
* Abstract method for implementing the task-specific cancellation logic.
168175
* Subclasses must override this method to provide their own implementation of how a task should be canceled.
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.metric;
19+
20+
import java.util.concurrent.atomic.DoubleAdder;
21+
22+
/**
23+
* A counter of a quantity with a fractional part, such as a total in seconds, which a LongCounterMetric would have to
24+
* round each addition of.
25+
*/
26+
public class DoubleCounterMetric extends CounterMetric<Double> {
27+
28+
public DoubleCounterMetric(String name, MetricUnit unit, String description) {
29+
super(name, unit, description);
30+
}
31+
32+
private final DoubleAdder value = new DoubleAdder();
33+
34+
@Override
35+
public void increase(Double delta) {
36+
value.add(delta);
37+
}
38+
39+
@Override
40+
public Double getValue() {
41+
return value.sum();
42+
}
43+
}

‎fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
import org.apache.doris.job.common.TaskStatus;
3838
import org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob;
3939
import org.apache.doris.job.extensions.insert.streaming.StreamingJobStatistic;
40+
import org.apache.doris.job.extensions.mtmv.MTMVTask;
4041
import org.apache.doris.load.EtlJobType;
4142
import org.apache.doris.load.loadv2.JobState;
4243
import org.apache.doris.load.loadv2.LoadManager;
@@ -50,6 +51,7 @@
5051
import org.apache.doris.mtmv.MTMVRefreshEnum.RefreshMethod;
5152
import org.apache.doris.mtmv.MTMVRefreshEnum.RefreshTrigger;
5253
import org.apache.doris.mtmv.MTMVUtil;
54+
import org.apache.doris.mtmv.ivm.IvmRefreshMethod;
5355
import org.apache.doris.mysql.privilege.Auth;
5456
import org.apache.doris.nereids.NereidsPlanner;
5557
import org.apache.doris.persist.EditLog;
@@ -272,6 +274,20 @@ public final class MetricRepo {
272274
public static GaugeMetric<Integer> GAUGE_ASYNC_MATERIALIZED_VIEW_TASK_RUNNING_NUM;
273275
public static GaugeMetric<Integer> GAUGE_ASYNC_MATERIALIZED_VIEW_TASK_PENDING_NUM;
274276

277+
// The refresh tasks of IVM materialized views, counted on the Master as each one ends.
278+
// The result label of COUNTER_IVM_REFRESH_TASKS: no_change is a SUCCESS that committed nothing.
279+
public static final String IVM_REFRESH_RESULT_SUCCESS = "success";
280+
public static final String IVM_REFRESH_RESULT_NO_CHANGE = "no_change";
281+
public static final String IVM_REFRESH_RESULT_FAILED = "failed";
282+
public static final String IVM_REFRESH_RESULT_CANCELED = "canceled";
283+
// by result, then by IvmRefreshMethod
284+
public static Map<String, Map<IvmRefreshMethod, LongCounterMetric>> COUNTER_IVM_REFRESH_TASKS;
285+
public static Map<IvmRefreshMethod, DoubleCounterMetric> COUNTER_IVM_REFRESH_DURATION_SECONDS;
286+
// by MTMVTask.IVM_FALLBACK_REASONS and MTMVTask.IVM_FAILURE_REASONS
287+
public static Map<String, LongCounterMetric> COUNTER_IVM_FALLBACKS;
288+
public static Map<String, LongCounterMetric> COUNTER_IVM_REFRESH_FAILURES;
289+
public static LongCounterMetric COUNTER_IVM_SCHEDULE_REJECTED;
290+
275291
// Statistics
276292
public static GaugeMetric<Double> GAUGE_STATISTICS_UNHEALTHY_TABLE_RATE;
277293
public static GaugeMetric<Double> GAUGE_STATISTICS_UNHEALTHY_COLUMN_RATE;
@@ -1146,6 +1162,44 @@ public Integer getValue() {
11461162
}
11471163
};
11481164
DORIS_METRIC_REGISTER.addMetrics(GAUGE_ASYNC_MATERIALIZED_VIEW_TASK_PENDING_NUM);
1165+
1166+
// IVM refresh tasks, counted by MTMVTask as each task of an IVM MV ends. Every combination of labels is
1167+
// registered here, so that each series exists from 0 rather than appearing with its first event.
1168+
COUNTER_IVM_REFRESH_TASKS = Maps.newHashMap();
1169+
for (String result : new String[] {IVM_REFRESH_RESULT_SUCCESS, IVM_REFRESH_RESULT_NO_CHANGE,
1170+
IVM_REFRESH_RESULT_FAILED, IVM_REFRESH_RESULT_CANCELED}) {
1171+
Map<IvmRefreshMethod, LongCounterMetric> byMethod = Maps.newEnumMap(IvmRefreshMethod.class);
1172+
for (IvmRefreshMethod method : IvmRefreshMethod.values()) {
1173+
LongCounterMetric counter = new LongCounterMetric("ivm_refresh_tasks_total", MetricUnit.NOUNIT,
1174+
"number of the refresh tasks of IVM materialized views that ended, by result and by the"
1175+
+ " kinds of work that committed data");
1176+
counter.addLabel(new MetricLabel("result", result))
1177+
.addLabel(new MetricLabel("method", method.name().toLowerCase()));
1178+
DORIS_METRIC_REGISTER.addMetrics(counter);
1179+
byMethod.put(method, counter);
1180+
}
1181+
COUNTER_IVM_REFRESH_TASKS.put(result, byMethod);
1182+
}
1183+
COUNTER_IVM_REFRESH_DURATION_SECONDS = Maps.newEnumMap(IvmRefreshMethod.class);
1184+
for (IvmRefreshMethod method : IvmRefreshMethod.values()) {
1185+
DoubleCounterMetric counter = new DoubleCounterMetric("ivm_refresh_duration_seconds_total",
1186+
MetricUnit.SECONDS, "how long the refresh tasks of IVM materialized views that succeeded or"
1187+
+ " failed ran, from start to finish, in seconds");
1188+
counter.addLabel(new MetricLabel("method", method.name().toLowerCase()));
1189+
DORIS_METRIC_REGISTER.addMetrics(counter);
1190+
COUNTER_IVM_REFRESH_DURATION_SECONDS.put(method, counter);
1191+
}
1192+
COUNTER_IVM_FALLBACKS = addIvmRefreshReasonCounters("ivm_fallbacks_total",
1193+
"number of the refresh tasks of IVM materialized views that fell back from the incremental refresh"
1194+
+ " and committed data in the refresh that replaced it, by the reason",
1195+
MTMVTask.IVM_FALLBACK_REASONS);
1196+
COUNTER_IVM_REFRESH_FAILURES = addIvmRefreshReasonCounters("ivm_refresh_failures_total",
1197+
"number of the refresh tasks of IVM materialized views that failed, by the reason",
1198+
MTMVTask.IVM_FAILURE_REASONS);
1199+
COUNTER_IVM_SCHEDULE_REJECTED = new LongCounterMetric("ivm_schedule_rejected_total", MetricUnit.NOUNIT,
1200+
"number of the refresh tasks of IVM materialized views that the full task queue turned away");
1201+
DORIS_METRIC_REGISTER.addMetrics(COUNTER_IVM_SCHEDULE_REJECTED);
1202+
11491203
GAUGE_MAX_TABLE_SIZE_BYTES = new GaugeMetricImpl<>("max_table_size_bytes", MetricUnit.BYTES, "", 0L);
11501204
DORIS_METRIC_REGISTER.addMetrics(GAUGE_MAX_TABLE_SIZE_BYTES);
11511205

@@ -1761,6 +1815,19 @@ public static synchronized String getMetric(MetricVisitor visitor) {
17611815
return visitor.finish();
17621816
}
17631817

1818+
// One counter per reason, keyed by the reason as the task records it and labeled with it in lower case.
1819+
private static Map<String, LongCounterMetric> addIvmRefreshReasonCounters(String name, String description,
1820+
List<String> reasons) {
1821+
Map<String, LongCounterMetric> counters = Maps.newHashMap();
1822+
for (String reason : reasons) {
1823+
LongCounterMetric counter = new LongCounterMetric(name, MetricUnit.NOUNIT, description);
1824+
counter.addLabel(new MetricLabel("reason", reason.toLowerCase()));
1825+
DORIS_METRIC_REGISTER.addMetrics(counter);
1826+
counters.put(reason, counter);
1827+
}
1828+
return counters;
1829+
}
1830+
17641831
public static <M extends Metric<?>> AutoMappedMetric<M> addLabeledMetrics(String label, Supplier<M> metric) {
17651832
return new AutoMappedMetric<>(value -> {
17661833
M m = metric.get();
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.mtmv.ivm;
19+
20+
/**
21+
* The stage of a refresh task of an IVM MV that a failure happened in, which is the reason a failed task reports
22+
* when no {@link IvmFailureReason} names it.
23+
*
24+
* <p>An {@link IvmFailureReason} names the failures of the incremental refresh itself, which a request that may
25+
* not fall back fails with. Every other failure is told apart by where it was thrown rather than by its message,
26+
* so that the reasons stay a fixed set.
27+
*/
28+
public enum IvmRefreshFailureStage {
29+
/** The MV's query could not be analyzed against its base tables, or can no longer be used. */
30+
QUERY_UNUSABLE,
31+
/** The MV's partitions could not be synchronized with the partitions of its base tables. */
32+
PARTITION_SYNC_FAILED,
33+
/** A refresh attempt failed while it ran. */
34+
EXECUTION_FAILED,
35+
/** Any other stage: resolving the request, deciding the attempts, or after every attempt gave up. */
36+
OTHER
37+
}
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.mtmv.ivm;
19+
20+
import java.util.Set;
21+
22+
/**
23+
* How a refresh task of an IVM MV actually refreshed it, judged by the kinds of work that committed data.
24+
*
25+
* <p>The request does not say it: an AUTO or INCREMENTAL FALLBACK request may end in the incremental refresh, a
26+
* partition rebuild, a whole-MV rebuild, or several of them, and which of them wrote data is what tells a refresh
27+
* that kept the MV up incrementally from one that recomputed it. The refresh mode a task reports is a different
28+
* question -- which partitions it covers -- so the two are reported side by side.
29+
*/
30+
public enum IvmRefreshMethod {
31+
/** Nothing was committed: every partition was up to date, or the task ended before writing anything. */
32+
NONE,
33+
/** Only the incremental refresh committed data. */
34+
INCREMENTAL,
35+
/**
36+
* Only partition rebuilds committed data. The rebuild an incremental attempt runs first, for the partitions the
37+
* criterion says cannot be caught up, is one of them.
38+
*/
39+
PARTITION_REBUILD,
40+
/** Only a whole-MV rebuild committed data. */
41+
COMPLETE,
42+
/** More than one kind committed data, typically the rebuild of invalidated partitions and then the delta. */
43+
MIXED;
44+
45+
/** The method of a task whose committed work was of these kinds, which are any of the three single kinds. */
46+
public static IvmRefreshMethod of(Set<IvmRefreshMethod> committedKinds) {
47+
if (committedKinds.isEmpty()) {
48+
return NONE;
49+
}
50+
if (committedKinds.size() > 1) {
51+
return MIXED;
52+
}
53+
return committedKinds.iterator().next();
54+
}
55+
}

0 commit comments

Comments
 (0)