What problem does this PR solve?

Issue Number: #69907

Related PR: None

Problem Summary:

In short. The BEs clean up the changelog of a table by TTL, but /metrics cannot tell whether that cleanup runs, fails, or reclaims anything. This PR counts each cleanup on the BE as it finishes, in three new series: how many succeeded or failed, how many bytes of expired changelog they dropped, and when one last succeeded.

Background. A table with changelog.enable keeps its changelog in a hidden changelog index, and changelog.ttl_seconds bounds how long the changelog is kept. The BEs remove what is older than that in a cumulative compaction of a tablet of that index (a changelog tablet): rowsets whose commit TSO is at or before the TTL cutoff are left out of the merge, so the output replaces them without their rows. Two submitters start these compactions, both as CHANGELOG_COMPACTION: the changelog TTL scanner, for tablets it finds holding expired rowsets, and the regular compaction producer; either compaction picks expired rowsets first.

The problem, and what it cost.

• No alert can watch the cleanup. The bvars changelog_ttl_discovered, changelog_ttl_submitted and changelog_ttl_failed are only on the brpc /vars page, and they count what the TTL scanner found and submitted, not what the compactions did: a cleanup that fails after it was submitted counts nowhere, and nothing counts the bytes removed.
• The only changelog series in /metrics, compaction_task_state_total{type="changelog"}, is the number of changelog compactions running and pending. A cleanup that keeps failing lets the changelog grow without bound, and nothing on the BE says why.

How this PR fixes it.

• Three series, counted as a compaction that drops expired changelog ends, and registered at startup so they exist from 0:

• A compaction is a cleanup when one of its inputs is expired changelog: at or below the visible version its TTL snapshot was taken at, with a commit TSO at or before the cutoff. That is exactly what filter_changelog_ttl_rowsets leaves out of the merge, and the two now ask one predicate, Compaction::is_expired_changelog_rowset, which pick_expired_changelog_rowset uses too. It is decided from the inputs rather than from what the merge filtered, so that a cleanup that fails before its merge, a cloud compaction that cannot refresh its tablet meta for example, is counted as failed.
• Counted where the execution of the compaction ends: CumulativeCompaction::execute_compact in shared-nothing mode, CloudCumulativeCompaction::record_compaction_success and record_compaction_failure in cloud mode. A compaction that does not get to run has not started a cleanup and is not counted: one that finds the tablet under another cumulative compaction, whether in its preparation or as its execution takes the cumulative compaction lock again, and one whose job the meta service refuses.
• In cloud mode the BE that runs the compaction counts it, so a query sums over the BEs; the shared storage is not measured. The dropped rowsets become stale rowsets, whose files the stale rowset sweep or the recycler deletes later; the bytes are counted when the rowsets leave the tablet.
• The bvars stay as they are.

Results. Before this PR, the /metrics of a BE has none of these series. After it, on a local shared-nothing cluster, a table with changelog.ttl_seconds = 1 and two rows inserted (regression test_changelog_ttl_cleanup_metrics):

The 1112 bytes are what the BE logs for that changelog compaction: input_rowsets_total_size=1.09 KB, output_rowset_total_size=0, filtered_row_num=2.

Classes, and how they call each other.

• DorisMetrics: the three series.
• Compaction: is_expired_changelog_rowset (new), shared by pick_expired_changelog_rowset and filter_changelog_ttl_rowsets; record_changelog_ttl_cleanup (new) counts a cleanup from the inputs.
• CumulativeCompaction, CloudCumulativeCompaction: call it as the execution ends.

CumulativeCompaction::execute_compact                      shared-nothing
  └─ CompactionMixin::execute_compact ─► merge: filter_changelog_ttl_rowsets ─┐
  └─ record_changelog_ttl_cleanup(ok)                                          │ is_expired_changelog_rowset
CloudCumulativeCompaction::execute_compact                 cloud               │
  └─ CloudCompactionMixin::execute_compact ─► merge: same filter ─────────────┤
  └─ record_compaction_success ─► record_changelog_ttl_cleanup(true)           │
     record_compaction_failure ─► record_changelog_ttl_cleanup(false)          │
                                    └─ DorisMetrics::changelog_ttl_* ◄─────────┘ (from the inputs)

Nothing else changes: no config, no thrift or proto change, and what the compactions pick and drop stays the same.

Release note

New BE metrics doris_be_changelog_ttl_cleanup_total{result}, doris_be_changelog_ttl_reclaimed_bytes_total and doris_be_changelog_ttl_last_success_timestamp_seconds.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

  Unit test: CumulativeCompactionTest.TestRecordChangelogTtlCleanup checks what counts as a cleanup (not a compaction whose inputs hold no expired changelog, nor one that took no TTL snapshot; a rowset above the visible version of the snapshot, or an empty one, is not expired changelog) and that the reclaimed bytes are those of the rowsets filter_changelog_ttl_rowsets drops; CompactionMetricsTest.TestChangelogTtlCleanupCountedAsCompactionFinishes runs CumulativeCompaction::execute_compact up to its merge and checks a success and a failure; CloudCompactionTest.changelog_ttl_cleanup_counted_as_compaction_finishes does the same through record_compaction_success and record_compaction_failure. With the success hooks removed, the two that go through them fail. Together with the other compaction, changelog TTL and DorisMetrics suites, 114 cases pass and one is skipped outside Release.

  Regression test: changelog_p0/test_changelog_ttl_cleanup_metrics (nonConcurrent: it waits for the TTL to expire), on a local shared-nothing cluster; not run in cloud mode.

  Manual test: promtool check metrics on the /metrics of the BE reports no parsing error; the only finding on the new series is "no help text", which every BE series gets (the BE writes no HELP lines).

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes. The doris-website change comes with the documentation of the changelog and IVM monitoring.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label