Skip to content

Commit 2b8908b

Browse files
authored
[feature](be) Count the changelog TTL cleanups in the BE metrics (#69868)
### What problem does this PR solve? Issue Number: #69907 Related PR: None Problem Summary: **In short.** The BEs clean up the changelog of a table by TTL, but `/metrics` cannot tell whether that cleanup runs, fails, or reclaims anything. This PR counts each cleanup on the BE as it finishes, in three new series: how many succeeded or failed, how many bytes of expired changelog they dropped, and when one last succeeded. **Background.** A table with `changelog.enable` keeps its changelog in a hidden changelog index, and `changelog.ttl_seconds` bounds how long the changelog is kept. The BEs remove what is older than that in a cumulative compaction of a tablet of that index (a changelog tablet): rowsets whose commit TSO is at or before the TTL cutoff are left out of the merge, so the output replaces them without their rows. Two submitters start these compactions, both as `CHANGELOG_COMPACTION`: the changelog TTL scanner, for tablets it finds holding expired rowsets, and the regular compaction producer; either compaction picks expired rowsets first. **The problem, and what it cost.** - No alert can watch the cleanup. The bvars `changelog_ttl_discovered`, `changelog_ttl_submitted` and `changelog_ttl_failed` are only on the brpc `/vars` page, and they count what the TTL scanner found and submitted, not what the compactions did: a cleanup that fails after it was submitted counts nowhere, and nothing counts the bytes removed. - The only changelog series in `/metrics`, `compaction_task_state_total{type="changelog"}`, is the number of changelog compactions running and pending. A cleanup that keeps failing lets the changelog grow without bound, and nothing on the BE says why. **How this PR fixes it.** - Three series, counted as a compaction that drops expired changelog ends, and registered at startup so they exist from 0: | Series | Type | Labels | Value | | --- | --- | --- | --- | | `doris_be_changelog_ttl_cleanup_total` | counter | `result`=success/failed | the compactions that drop expired changelog rowsets, counted as each ends | | `doris_be_changelog_ttl_reclaimed_bytes_total` | counter | | the total disk size of the expired rowsets the successful ones dropped | | `doris_be_changelog_ttl_last_success_timestamp_seconds` | gauge | | when one last succeeded, in seconds since the epoch | - A compaction is a cleanup when one of its inputs is expired changelog: at or below the visible version its TTL snapshot was taken at, with a commit TSO at or before the cutoff. That is exactly what `filter_changelog_ttl_rowsets` leaves out of the merge, and the two now ask one predicate, `Compaction::is_expired_changelog_rowset`, which `pick_expired_changelog_rowset` uses too. It is decided from the inputs rather than from what the merge filtered, so that a cleanup that fails before its merge, a cloud compaction that cannot refresh its tablet meta for example, is counted as failed. - Counted where the execution of the compaction ends: `CumulativeCompaction::execute_compact` in shared-nothing mode, `CloudCumulativeCompaction::record_compaction_success` and `record_compaction_failure` in cloud mode. A compaction that does not get to run has not started a cleanup and is not counted: one that finds the tablet under another cumulative compaction, whether in its preparation or as its execution takes the cumulative compaction lock again, and one whose job the meta service refuses. - In cloud mode the BE that runs the compaction counts it, so a query sums over the BEs; the shared storage is not measured. The dropped rowsets become stale rowsets, whose files the stale rowset sweep or the recycler deletes later; the bytes are counted when the rowsets leave the tablet. - The bvars stay as they are. **Results.** Before this PR, the `/metrics` of a BE has none of these series. After it, on a local shared-nothing cluster, a table with `changelog.ttl_seconds` = 1 and two rows inserted (regression `test_changelog_ttl_cleanup_metrics`): | | `cleanup_total{result="success"}` | `cleanup_total{result="failed"}` | `reclaimed_bytes_total` | `last_success_timestamp_seconds` | | --- | --- | --- | --- | --- | | BE started | 0 | 0 | 0 | 0 | | once the changelog rows have been cleaned up | 1 | 0 | 1112 | the time of the cleanup | The 1112 bytes are what the BE logs for that changelog compaction: `input_rowsets_total_size=1.09 KB`, `output_rowset_total_size=0`, `filtered_row_num=2`. **Classes, and how they call each other.** - `DorisMetrics`: the three series. - `Compaction`: `is_expired_changelog_rowset` (new), shared by `pick_expired_changelog_rowset` and `filter_changelog_ttl_rowsets`; `record_changelog_ttl_cleanup` (new) counts a cleanup from the inputs. - `CumulativeCompaction`, `CloudCumulativeCompaction`: call it as the execution ends. ``` CumulativeCompaction::execute_compact shared-nothing └─ CompactionMixin::execute_compact ─► merge: filter_changelog_ttl_rowsets ─┐ └─ record_changelog_ttl_cleanup(ok) │ is_expired_changelog_rowset CloudCumulativeCompaction::execute_compact cloud │ └─ CloudCompactionMixin::execute_compact ─► merge: same filter ─────────────┤ └─ record_compaction_success ─► record_changelog_ttl_cleanup(true) │ record_compaction_failure ─► record_changelog_ttl_cleanup(false) │ └─ DorisMetrics::changelog_ttl_* ◄─────────┘ (from the inputs) ``` Nothing else changes: no config, no thrift or proto change, and what the compactions pick and drop stays the same.
1 parent 9dc9dc7 commit 2b8908b

10 files changed

Lines changed: 377 additions & 4 deletions

File tree

‎be/src/cloud/cloud_cumulative_compaction.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -317,6 +317,7 @@ void CloudCumulativeCompaction::record_compaction_success(
317317
DorisMetrics::instance()->cumulative_compaction_bytes_total->increment(
318318
_input_rowsets_total_size);
319319
cumu_output_size << _output_rowset->total_disk_size();
320+
record_changelog_ttl_cleanup(true);
320321

321322
cloud_tablet()->set_last_cumu_compaction_status(Status::OK().to_string());
322323
cloud_tablet()->set_last_cumu_compaction_success_time(UnixMillis());
@@ -325,6 +326,7 @@ void CloudCumulativeCompaction::record_compaction_success(
325326
Status CloudCumulativeCompaction::record_compaction_failure(Status status) {
326327
LOG(WARNING) << "fail to do " << compaction_name() << ". res=" << status
327328
<< ", tablet=" << _tablet->tablet_id() << ", output_version=" << _output_version;
329+
record_changelog_ttl_cleanup(false);
328330
cloud_tablet()->set_last_cumu_compaction_status(status.to_string());
329331
cloud_tablet()->set_last_cumu_compaction_failure_time(UnixMillis());
330332
return status;

‎be/src/common/metrics/doris_metrics.cpp‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,13 @@ DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(cumulative_compaction_task_pending_total, M
127127
"", compaction_task_state_total,
128128
Labels({{"type", "cumulative"}}));
129129

130+
DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(changelog_ttl_cleanup_success_total, MetricUnit::OPERATIONS,
131+
"", changelog_ttl_cleanup_total,
132+
Labels({{"result", "success"}}));
133+
DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(changelog_ttl_cleanup_failed_total, MetricUnit::OPERATIONS, "",
134+
changelog_ttl_cleanup_total, Labels({{"result", "failed"}}));
135+
DEFINE_COUNTER_METRIC_PROTOTYPE_2ARG(changelog_ttl_reclaimed_bytes_total, MetricUnit::BYTES);
136+
130137
DEFINE_COUNTER_METRIC_PROTOTYPE_5ARG(segment_read_total, MetricUnit::OPERATIONS,
131138
"(segment_v2) total number of segments read", segment_read,
132139
Labels({{"type", "segment_read_total"}}));
@@ -211,6 +218,9 @@ DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(all_segments_num, MetricUnit::NOUNIT);
211218
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(compaction_used_permits, MetricUnit::NOUNIT);
212219
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(compaction_waitting_permits, MetricUnit::NOUNIT);
213220

221+
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(changelog_ttl_last_success_timestamp_seconds,
222+
MetricUnit::SECONDS);
223+
214224
DEFINE_HISTOGRAM_METRIC_PROTOTYPE_2ARG(tablet_version_num_distribution, MetricUnit::NOUNIT);
215225

216226
DEFINE_GAUGE_CORE_METRIC_PROTOTYPE_2ARG(query_scan_bytes_per_second, MetricUnit::BYTES);
@@ -368,6 +378,10 @@ DorisMetrics::DorisMetrics() : _metric_registry(_s_registry_name) {
368378
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, cumulative_compaction_task_running_total);
369379
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, cumulative_compaction_task_pending_total);
370380

381+
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, changelog_ttl_cleanup_success_total);
382+
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, changelog_ttl_cleanup_failed_total);
383+
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, changelog_ttl_reclaimed_bytes_total);
384+
371385
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, segment_read_total);
372386
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, condition_cache_search_count);
373387
INT_COUNTER_METRIC_REGISTER(_server_metric_entity, condition_cache_hit_count);
@@ -410,6 +424,8 @@ DorisMetrics::DorisMetrics() : _metric_registry(_s_registry_name) {
410424
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, compaction_used_permits);
411425
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, compaction_waitting_permits);
412426

427+
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, changelog_ttl_last_success_timestamp_seconds);
428+
413429
HISTOGRAM_METRIC_REGISTER(_server_metric_entity, tablet_version_num_distribution);
414430

415431
INT_GAUGE_METRIC_REGISTER(_server_metric_entity, query_scan_bytes_per_second);

‎be/src/common/metrics/doris_metrics.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,12 @@ class DorisMetrics {
124124
IntCounter* cumulative_compaction_task_running_total = nullptr;
125125
IntCounter* cumulative_compaction_task_pending_total = nullptr;
126126

127+
// The compactions that drop expired changelog rowsets, counted as each of them finishes.
128+
IntCounter* changelog_ttl_cleanup_success_total = nullptr;
129+
IntCounter* changelog_ttl_cleanup_failed_total = nullptr;
130+
// The size of the expired changelog rowsets the successful ones dropped.
131+
IntCounter* changelog_ttl_reclaimed_bytes_total = nullptr;
132+
127133
IntCounter* publish_task_request_total = nullptr;
128134
IntCounter* publish_task_failed_total = nullptr;
129135

@@ -182,6 +188,9 @@ class DorisMetrics {
182188
// permits required by the compaction task which is waiting for permits
183189
IntGauge* compaction_waitting_permits = nullptr;
184190

191+
// when a compaction that drops expired changelog rowsets last succeeded, in seconds since the epoch
192+
IntGauge* changelog_ttl_last_success_timestamp_seconds = nullptr;
193+
185194
HistogramMetric* tablet_version_num_distribution = nullptr;
186195

187196
// The following metrics will be calculated

‎be/src/storage/compaction/compaction.cpp‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -334,13 +334,17 @@ void Compaction::snapshot_changelog_ttl() {
334334
_changelog_ttl_seconds = config.ttl_seconds();
335335
}
336336

337+
bool Compaction::is_expired_changelog_rowset(const RowsetSharedPtr& rowset) const {
338+
return rowset->end_version() <= _changelog_ttl_visible_version &&
339+
changelog_rowset_expired(*rowset->rowset_meta(), _changelog_ttl_cutoff_tso);
340+
}
341+
337342
bool Compaction::pick_expired_changelog_rowset(const std::vector<RowsetSharedPtr>& candidates) {
338343
if (_changelog_ttl_cutoff_tso < 0) {
339344
return false;
340345
}
341346
auto expired = std::ranges::find_if(candidates, [this](const RowsetSharedPtr& rowset) {
342-
return rowset->end_version() <= _changelog_ttl_visible_version &&
343-
changelog_rowset_expired(*rowset->rowset_meta(), _changelog_ttl_cutoff_tso);
347+
return is_expired_changelog_rowset(rowset);
344348
});
345349
if (expired == candidates.end()) {
346350
return false;
@@ -388,15 +392,41 @@ void Compaction::filter_changelog_ttl_rowsets() {
388392
if (rowset->num_rows() == 0) {
389393
continue;
390394
}
391-
if (rowset->end_version() > _changelog_ttl_visible_version ||
392-
!changelog_rowset_expired(*rowset->rowset_meta(), _changelog_ttl_cutoff_tso)) {
395+
if (!is_expired_changelog_rowset(rowset)) {
393396
_data_input_rowsets.push_back(rowset);
394397
continue;
395398
}
396399
_changelog_ttl_filtered_rows += rowset->num_rows();
397400
}
398401
}
399402

403+
void Compaction::record_changelog_ttl_cleanup(bool succeeded) const {
404+
if (_changelog_ttl_cutoff_tso < 0) {
405+
return;
406+
}
407+
// Told from the inputs rather than from what the merge filtered, which is nothing yet for a cleanup that fails
408+
// before its merge: a cloud compaction that cannot refresh its tablet meta, for example.
409+
bool drops_expired_changelog = false;
410+
int64_t expired_bytes = 0;
411+
for (const auto& rowset : _input_rowsets) {
412+
if (is_expired_changelog_rowset(rowset)) {
413+
drops_expired_changelog = true;
414+
expired_bytes += rowset->total_disk_size();
415+
}
416+
}
417+
if (!drops_expired_changelog) {
418+
return;
419+
}
420+
if (!succeeded) {
421+
DorisMetrics::instance()->changelog_ttl_cleanup_failed_total->increment(1);
422+
return;
423+
}
424+
DorisMetrics::instance()->changelog_ttl_cleanup_success_total->increment(1);
425+
DorisMetrics::instance()->changelog_ttl_reclaimed_bytes_total->increment(expired_bytes);
426+
DorisMetrics::instance()->changelog_ttl_last_success_timestamp_seconds->set_value(
427+
UnixSeconds());
428+
}
429+
400430
Status Compaction::merge_input_rowsets() {
401431
MergeInputRowsetsContext context;
402432
RETURN_IF_ERROR(prepare_merge_input_rowsets_execution(&context));

‎be/src/storage/compaction/compaction.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,10 @@ class Compaction {
126126
void snapshot_row_ttl();
127127
bool pick_expired_changelog_rowset(const std::vector<RowsetSharedPtr>& candidates);
128128
void filter_changelog_ttl_rowsets();
129+
// Whether the rowset is expired changelog, which this compaction drops instead of merging it.
130+
bool is_expired_changelog_rowset(const RowsetSharedPtr& rowset) const;
131+
// Counts this compaction in the changelog TTL cleanup metrics as it finishes, if it drops expired changelog.
132+
void record_changelog_ttl_cleanup(bool succeeded) const;
129133

130134
virtual Status prepare_merge_input_rowsets(MergeInputRowsetsResult* /*result*/) {
131135
return Status::OK();

‎be/src/storage/compaction/cumulative_compaction.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ Status CumulativeCompaction::execute_compact() {
121121
SCOPED_ATTACH_TASK(_mem_tracker);
122122

123123
st = CompactionMixin::execute_compact();
124+
record_changelog_ttl_cleanup(st.ok());
124125
RETURN_IF_ERROR(st);
125126

126127
TEST_SYNC_POINT_RETURN_WITH_VALUE(

‎be/test/cloud/cloud_compaction_test.cpp‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -822,6 +822,56 @@ TEST_F(CloudCompactionTest, changelog_ttl_snapshot_selects_expired_rowset) {
822822
EXPECT_EQ(compaction._input_rowsets.front(), expired);
823823
}
824824

825+
TEST_F(CloudCompactionTest, changelog_ttl_cleanup_counted_as_compaction_finishes) {
826+
auto tablet_meta = create_cloud_compaction_test_tablet_meta(10020);
827+
tablet_meta->set_tablet_role(TabletRolePB::TABLET_ROLE_CHANGELOG);
828+
BinlogConfig binlog_config(true, 10, 1024, 10, BinlogFormatPB::CHANGELOG, false);
829+
tablet_meta->set_binlog_config(binlog_config);
830+
const int64_t reference_tso = 100000L << kTsoLogicalBits;
831+
_cluster_info->advance_changelog_ttl_reference_tso(reference_tso);
832+
const int64_t cutoff = changelog_ttl_cutoff_tso(reference_tso, 10);
833+
auto tablet = std::make_shared<CloudTablet>(_engine, tablet_meta);
834+
TestableCloudCumulativeCompaction compaction(_engine, tablet);
835+
compaction.snapshot_changelog_ttl();
836+
837+
auto expired = create_rowset(Version(2, 2), 1, false, 1024);
838+
expired->rowset_meta()->set_num_rows(1);
839+
expired->rowset_meta()->set_commit_tso(cutoff);
840+
auto retained = create_rowset(Version(3, 3), 1, false, 2048);
841+
retained->rowset_meta()->set_num_rows(1);
842+
retained->rowset_meta()->set_commit_tso(cutoff + 1);
843+
compaction.set_input_rowsets({expired, retained});
844+
compaction.set_output_rowset(create_rowset(Version(2, 3), 1, false, 2048));
845+
846+
auto* metrics = DorisMetrics::instance();
847+
// Recording the success also counts the compaction in the cumulative compaction totals, which DorisMetricsTest
848+
// reads as absolute values in this same process.
849+
const int64_t cumulative_deltas = metrics->cumulative_compaction_deltas_total->value();
850+
const int64_t cumulative_bytes = metrics->cumulative_compaction_bytes_total->value();
851+
Defer restore_cumulative_totals {[&]() {
852+
metrics->cumulative_compaction_deltas_total->set_value(cumulative_deltas);
853+
metrics->cumulative_compaction_bytes_total->set_value(cumulative_bytes);
854+
}};
855+
const int64_t successes = metrics->changelog_ttl_cleanup_success_total->value();
856+
const int64_t failures = metrics->changelog_ttl_cleanup_failed_total->value();
857+
const int64_t reclaimed_bytes = metrics->changelog_ttl_reclaimed_bytes_total->value();
858+
const int64_t started_s = UnixSeconds();
859+
compaction.record_compaction_success(std::chrono::steady_clock::now());
860+
EXPECT_EQ(metrics->changelog_ttl_cleanup_success_total->value(), successes + 1);
861+
EXPECT_EQ(metrics->changelog_ttl_cleanup_failed_total->value(), failures);
862+
// the expired rowset is dropped, the retained one is merged
863+
EXPECT_EQ(metrics->changelog_ttl_reclaimed_bytes_total->value(), reclaimed_bytes + 1024);
864+
const int64_t last_success_s = metrics->changelog_ttl_last_success_timestamp_seconds->value();
865+
EXPECT_GE(last_success_s, started_s);
866+
EXPECT_LE(last_success_s, UnixSeconds());
867+
868+
EXPECT_FALSE(compaction.record_compaction_failure(Status::InternalError("injected")).ok());
869+
EXPECT_EQ(metrics->changelog_ttl_cleanup_success_total->value(), successes + 1);
870+
EXPECT_EQ(metrics->changelog_ttl_cleanup_failed_total->value(), failures + 1);
871+
EXPECT_EQ(metrics->changelog_ttl_reclaimed_bytes_total->value(), reclaimed_bytes + 1024);
872+
EXPECT_EQ(metrics->changelog_ttl_last_success_timestamp_seconds->value(), last_success_s);
873+
}
874+
825875
TEST_F(CloudCompactionTest, changelog_ttl_prevents_cumulative_point_from_skipping_expired) {
826876
auto old_parallel_cumu_compaction = config::enable_parallel_cumu_compaction;
827877
Defer restore_config(

‎be/test/storage/compaction/compaction_metrics_test.cpp‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@
3131
#include "common/status.h"
3232
#include "cpp/sync_point.h"
3333
#include "io/fs/local_file_system.h"
34+
#include "runtime/cluster_info.h"
35+
#include "runtime/exec_env.h"
36+
#include "storage/changelog.h"
3437
#include "storage/compaction/compaction.h"
3538
#include "storage/compaction/cumulative_compaction.h"
3639
#include "storage/compaction/cumulative_compaction_policy.h"
@@ -39,6 +42,7 @@
3942
#include "storage/storage_engine.h"
4043
#include "storage/tablet/tablet_manager.h"
4144
#include "util/threadpool.h"
45+
#include "util/time.h"
4246

4347
namespace doris {
4448
using namespace config;
@@ -246,4 +250,82 @@ TEST_F(CompactionMetricsTest, TestCompactionReadWriteThroughput) {
246250
EXPECT_EQ(DorisMetrics::instance()->local_compaction_read_rows_total->value(), 50 * 28);
247251
}
248252

253+
TEST_F(CompactionMetricsTest, TestChangelogTtlCleanupCountedAsCompactionFinishes) {
254+
ClusterInfo cluster;
255+
auto* previous_cluster = ExecEnv::GetInstance()->cluster_info();
256+
ExecEnv::GetInstance()->set_cluster_info(&cluster);
257+
Defer restore_cluster {[&]() { ExecEnv::GetInstance()->set_cluster_info(previous_cluster); }};
258+
const int64_t reference_tso = 2000L << kTsoLogicalBits;
259+
const int64_t cutoff = changelog_ttl_cutoff_tso(reference_tso, 1);
260+
cluster.advance_changelog_ttl_reference_tso(reference_tso);
261+
262+
TabletMetaSharedPtr tablet_meta;
263+
tablet_meta.reset(new TabletMeta(1, 2, 15673, 15674, 4, 5, TTabletSchema(), 6, {{7, 8}},
264+
UniqueId(9, 10), TTabletType::TABLET_TYPE_DISK,
265+
TCompressionType::LZ4F));
266+
tablet_meta->set_tablet_role(TabletRolePB::TABLET_ROLE_CHANGELOG);
267+
BinlogConfig binlog_config(true, 1, 1024, 10, BinlogFormatPB::CHANGELOG, false);
268+
tablet_meta->set_binlog_config(binlog_config);
269+
TabletSharedPtr tablet(new Tablet(*(_storage_engine.get()), tablet_meta, _data_dir.get(),
270+
CUMULATIVE_SIZE_BASED_POLICY));
271+
ASSERT_TRUE(tablet->init().ok());
272+
auto visible = std::make_shared<VersionWithTime>();
273+
visible->version.store(3);
274+
tablet->set_visible_version(visible);
275+
auto expired = create_rowset({2, 2}, 1, false, 1024);
276+
expired->rowset_meta()->set_commit_tso(cutoff);
277+
auto retained = create_rowset({3, 3}, 1, false, 2048);
278+
retained->rowset_meta()->set_commit_tso(cutoff + 1);
279+
280+
// The compaction runs up to its merge, and the execution ends there with the status injected here: the merge and
281+
// what follows it need the segment files these rowsets do not have.
282+
Status execution_status;
283+
auto* sp = SyncPoint::get_instance();
284+
sp->enable_processing();
285+
auto return_ok = [](auto&& values) {
286+
try_any_cast<std::pair<Status, bool>*>(values.back())->second = true;
287+
};
288+
sp->set_call_back("compaction::CompactionMixin::build_basic_info", return_ok);
289+
sp->set_call_back("compaction::CompactionMixin::execute_compact_impl", [&](auto&& values) {
290+
auto* pairs = try_any_cast<std::pair<Status, bool>*>(values.back());
291+
pairs->first = execution_status;
292+
pairs->second = true;
293+
});
294+
sp->set_call_back("compaction::CompactionMixin::execute_compact", return_ok);
295+
sp->set_call_back("cumulative_compaction::CumulativeCompaction::execute_compact", return_ok);
296+
Defer defer {[&]() {
297+
sp->clear_call_back("compaction::CompactionMixin::build_basic_info");
298+
sp->clear_call_back("compaction::CompactionMixin::execute_compact_impl");
299+
sp->clear_call_back("compaction::CompactionMixin::execute_compact");
300+
sp->clear_call_back("cumulative_compaction::CumulativeCompaction::execute_compact");
301+
}};
302+
auto execute = [&](Status status) {
303+
execution_status = std::move(status);
304+
CumulativeCompaction compaction(*_storage_engine, tablet);
305+
compaction.snapshot_changelog_ttl();
306+
compaction._input_rowsets = {expired, retained};
307+
return compaction.execute_compact();
308+
};
309+
310+
auto* metrics = DorisMetrics::instance();
311+
const int64_t successes = metrics->changelog_ttl_cleanup_success_total->value();
312+
const int64_t failures = metrics->changelog_ttl_cleanup_failed_total->value();
313+
const int64_t reclaimed_bytes = metrics->changelog_ttl_reclaimed_bytes_total->value();
314+
const int64_t started_s = UnixSeconds();
315+
ASSERT_TRUE(execute(Status::OK()).ok());
316+
EXPECT_EQ(metrics->changelog_ttl_cleanup_success_total->value(), successes + 1);
317+
EXPECT_EQ(metrics->changelog_ttl_cleanup_failed_total->value(), failures);
318+
// the expired rowset is dropped, the retained one is merged
319+
EXPECT_EQ(metrics->changelog_ttl_reclaimed_bytes_total->value(), reclaimed_bytes + 1024);
320+
const int64_t last_success_s = metrics->changelog_ttl_last_success_timestamp_seconds->value();
321+
EXPECT_GE(last_success_s, started_s);
322+
EXPECT_LE(last_success_s, UnixSeconds());
323+
324+
EXPECT_FALSE(execute(Status::InternalError("injected execution failure")).ok());
325+
EXPECT_EQ(metrics->changelog_ttl_cleanup_success_total->value(), successes + 1);
326+
EXPECT_EQ(metrics->changelog_ttl_cleanup_failed_total->value(), failures + 1);
327+
EXPECT_EQ(metrics->changelog_ttl_reclaimed_bytes_total->value(), reclaimed_bytes + 1024);
328+
EXPECT_EQ(metrics->changelog_ttl_last_success_timestamp_seconds->value(), last_success_s);
329+
}
330+
249331
} // namespace doris

0 commit comments

Comments
 (0)