What problem does this PR solve?

Issue Number: #69907

Related PR: None

Problem Summary:

In short. A refresh of an IVM materialized view can keep the MV up incrementally or quietly recompute it, and nothing tells the two apart: tasks("type"="mv") does not say which kinds of work a task committed, and /metrics has no IVM series at all. This PR makes each refresh task of an IVM MV report how it refreshed the MV (IvmRefreshMethod in tasks()) and why it failed or fell back, keeps a refresh summary on the MV for the IVM status table that follows, and counts the tasks on the Master in five new counters.

Background. An IVM MV is refreshed by a task that runs one or more attempts: the incremental attempt (the delta of its base tables' streams, after rebuilding the MV partitions an invalidation marked), a partition rebuild, and a whole-MV rebuild. A request that may fall back (AUTO, INCREMENTAL FALLBACK, PARTITIONS FALLBACK) moves on to the next attempt when one gives up, for example to COMPLETE after the delta meets a MIN/MAX boundary. tasks() already reports RefreshMode (which partitions the task covered), IvmFallbackReason (why the incremental attempt gave up, or why a strict request failed) and IvmRebuiltPartitions.

The problem, and what it cost.

• An operator cannot tell a refresh that kept the MV up incrementally from one that recomputed it. RefreshMode says COMPLETE for any whole-MV scope, and IvmFallbackReason is set even when the attempt that replaced the delta wrote nothing, or when a strict request failed.
• No alert can watch IVM refreshes. The asynchronous MV counters (async_materialized_view_task_success_num, _failed_num, _skip_num) cover every MV and say nothing about the method, the fallbacks, or why tasks fail. An MV that falls back to COMPLETE on every run, the expensive failure mode of IVM, goes unnoticed until the load of the cluster shows it.
• A task that the full MV task queue turns away never runs, and nothing counts it.
• The task history keeps only max_persistence_task_count tasks, so how many times in a row an MV has failed, or when it last fell back, cannot be read back from it.

How this PR fixes it.

• MTMVTask notes each batch that commits data (the delta, a partition rebuild, a whole-MV rebuild) and decides IvmRefreshMethod once, as the task publishes its result: NONE, INCREMENTAL, PARTITION_REBUILD, COMPLETE, or MIXED when more than one kind committed, typically the rebuild of invalidated partitions before the delta. tasks("type"="mv") gets the IvmRefreshMethod column, NULL while a task runs and for a task of an MV that is not an IVM MV.
• A task falls back when the incremental attempt gives up and an attempt replacing it commits data. Not fallbacks: building the first baseline of an MV no refresh has succeeded on, the rebuild of invalidated partitions before the delta, an explicit COMPLETE or PARTITIONS request, and a fallback that commits nothing.
• A failed task reports why: the IvmFailureReason a request that may not fall back failed with, or else the stage it failed in, QUERY_UNUSABLE, PARTITION_SYNC_FAILED, EXECUTION_FAILED or OTHER, told apart by where the failure was thrown rather than by its message.
• The method, the failure reason and whether the task fell back are journaled with the task, as new Gson fields that an older FE ignores. The MV keeps an IvmRefreshSummary, updated from each task result in MTMV.addTaskResult, which the Master and the replay of the same journal record run alike, from what the record carries: consecutive failures, the latest success and failure (with the reason and the error message, cut to 512 characters), the latest method and run time, and the fallback count with the latest reason. A canceled task leaves it as it is. Nothing reads it in this PR; information_schema.ivm_mv_status and the per-MV series of a follow-up PR do.
• Five counters, counted on the Master as each task of an IVM MV ends. Every label combination is registered up front, so each of the 61 series exists from 0:

Like the other FE event counters they live in the registry of every FE and only the Master counts in them, so a query sums them over the FEs; a restart or a new Master starts them from 0.

• AbstractTask.onRejected, a new hook that TaskProcessor.addTask calls when the executor queue rejects a task. Both the instant dispatch (JobScheduler) and the timer dispatch (DispatchTaskHandler) add tasks through it, and MTMVTask counts the rejection for an IVM MV. A timer-dispatched task that is turned away is canceled as before, so it also counts as canceled.

Results. Before this PR, tasks() has no method column and /metrics has none of these series. After it, on a local cluster, one IVM MV refreshed through each case (regression test_ivm_refresh_method_metrics):

The first refresh of a new MV finds every partition still to be filled and goes straight to COMPLETE; it counts no fallback. The MV holds what its base table does after each case.

Classes, and how they call each other.

• IvmRefreshMethod, IvmRefreshFailureStage (org.apache.doris.mtmv.ivm): the methods, and the stages a failure is reported by.
• MTMVTask: notes committed batches (recordCommitted), decides whether giving up the incremental attempt is a fallback (givingUpTheIncrementalAttemptIsAFallback), moves the failure stage along, settles the method and the failure reason in after(), and counts the task (recordIvmRefreshMetrics) and a rejection (onRejected).
• IvmRefreshSummary (org.apache.doris.mtmv.ivm): persisted on MTMV and updated by update(MTMVTask) from MTMV.addTaskResult; MTMV.hasSucceededIvmRefresh tells the first baseline apart.
• MetricRepo: registers the five counters. DoubleCounterMetric (new) keeps a total in seconds without rounding each addition.
• AbstractTask.onRejected (new, empty by default), called by TaskProcessor.addTask.

MTMVTask.run()                                            worker thread
  ├─ executeIvmAttempt
  │    ├─ rebuild invalidated partitions ──► recordCommitted(PARTITION_REBUILD)
  │    └─ delta (IvmIncrRefreshManager) ───► recordCommitted(INCREMENTAL)
  │    gave up ─► fallingBack = givingUpTheIncrementalAttemptIsAFallback()
  ├─ PARTITIONS / COMPLETE attempts ───────► recordCommitted(...)   ivmFallback once fallingBack
  └─ failureStage: QUERY_UNUSABLE ─► PARTITION_SYNC_FAILED ─► EXECUTION_FAILED, or an IvmFailureReason

onSuccess / onFail / cancel ─► after()
  ├─ ivmRefreshMethod, ivmFailureReason settled
  ├─ recordIvmRefreshMetrics() ─► MetricRepo.COUNTER_IVM_*            Master only
  └─ Env.addMTMVTaskResult ─► MTMV.addTaskResult ─► IvmRefreshSummary.update(task)
                                (journaled as ADD_TASK; the replay calls addTaskResult the same way)

TaskProcessor.addTask ── queue full ─► task.onRejected() ─► MTMVTask: COUNTER_IVM_SCHEDULE_REJECTED

Nothing else changes: no thrift or proto change, no new config, and the existing columns and series stay as they are.

Release note

tasks("type"="mv") has a new column IvmRefreshMethod: how a refresh task of an IVM materialized view refreshed it (NONE, INCREMENTAL, PARTITION_REBUILD, COMPLETE or MIXED). New FE metrics doris_fe_ivm_refresh_tasks_total{result,method}, doris_fe_ivm_refresh_duration_seconds_total{method}, doris_fe_ivm_fallbacks_total{reason}, doris_fe_ivm_refresh_failures_total{reason} and doris_fe_ivm_schedule_rejected_total.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

  Unit test: MTMVTaskTest checks the method each kind of committed batch gives, that the rebuild of invalidated partitions before the delta is MIXED and no fallback, that only what commits after the incremental attempt gave up makes a fallback, that building the first baseline is not one, the stage a failure reports (query, partition sync, execution) and the IVM reason of a strict request, the IvmRefreshMethod column, and that the report is journaled while what it is decided from is not. IvmRefreshSummaryTest checks consecutive failures across success, no change, failure and cancel, the latest success and failure, the error message cut, and the fallback count. MTMVTest checks that the Master and the replay of the same journal records end with the same summary with a task history of one, and that an MV that is not an IVM MV keeps none. IvmRefreshMetricsTest checks that every series is there from 0 and how each result is counted; TaskProcessorTest that only the task the full queue rejects is told so. The mtmv, IVM, metric and job executor tests with CreateMTMVCommandTest, ShowCreateMTMVTest and DropMaterializedViewTest pass: 60 classes, 884 cases.

  Regression test: mtmv_p0/ivm/test_ivm_refresh_method_metrics (nonConcurrent: it asserts which counters of the Master moved), twice on a local shared-nothing cluster; not run in cloud mode.

  Manual: promtool check metrics on the /metrics of the Master reports no parsing error and no lint finding on the new series.

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes. The doris-website change comes with the documentation of the changelog and IVM monitoring.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label