Skip to content

Commit eaa4ef9

Browse files
authored
[Opt](cloud) Add rate limit for BE to MS rpc (#60344)
This PR implements a two-layer MS (Meta Service) RPC rate limiting system for Doris cloud mode BE: 1. **Host-level rate limiting** — Token-bucket based QPS limiter for all 21 MS RPC types, preventing a single BE from overwhelming the MS with burst traffic. 2. **Table-level adaptive backpressure** — When MS returns `MS_BUSY` error code, BE dynamically identifies and throttles the top-k highest-QPS tables using a state machine, and automatically relaxes limits after the pressure subsides. --- ## Part 1: BE Host-Level Rate Limiting ### Problem In cloud mode, all BE nodes send RPCs (get_tablet, prepare_rowset, commit_rowset, etc.) to a shared Meta Service. A single BE experiencing load spikes (e.g., large batch imports, compaction storms) can send excessive RPC traffic that overwhelms MS, degrading service for all BEs. ### Solution Introduce `HostLevelMSRpcRateLimiters` — a per-BE, per-RPC-type rate limiter using token bucket algorithm. **Architecture:** - All 21 MS RPC types are enumerated in `MetaServiceRPC` enum (defined via X-macro for maintainability) - Each RPC type has an independent `TokenBucketRateLimiterHolder` with its own QPS limit - QPS limits are configured per CPU core: `actual_qps = config_value × num_cores` - Thread-safe design using `atomic_shared_ptr<RpcRateLimiter>` array for lock-free concurrent access during `limit()` calls - Each rate limiter includes a `bvar::LatencyRecorder` to monitor sleep durations caused by rate limiting **Configuration:** | Config | Default | Description | |--------|---------|-------------| | `enable_ms_rpc_host_level_rate_limit` | `true` | Global enable/disable switch | | `ms_rpc_qps_default` | `100` | Default per-core QPS for all RPCs | | `ms_rpc_qps_<rpc_name>` | `-1` | Per-RPC override (`-1` = use default, `0` = disabled) | All QPS configs are mutable (`DEFINE_mInt32`), allowing runtime adjustment without restart. `reset_all()` re-reads configs and recreates rate limiters. **Integration:** Rate limiting is applied inside the `retry_rpc()` template function in `cloud_meta_mgr.cpp`, which wraps all MS RPC calls. The `RpcRateLimitCtx` struct carries the rate limiter reference. Rate limiting executes before each RPC attempt (including retries), with the call to `apply_rate_limit()` performing a `bthread_usleep` if the token bucket requires waiting. **New files:** - `be/src/cloud/cloud_ms_rpc_rate_limiters.h` / `.cpp` - `be/test/cloud/cloud_ms_rpc_rate_limiters_test.cpp` --- ## Part 2: BE Table-Level Adaptive Backpressure ### Problem Host-level rate limiting applies uniformly across all tables. When MS reports overload (`MAX_QPS_LIMIT`), it's often caused by a small number of high-traffic tables (e.g., tables with many concurrent stream load jobs). A uniform rate limit would unnecessarily penalize all tables, while the hot tables continue to dominate the RPC traffic. ### Solution Implement table-level adaptive throttling for load-related RPCs. When MS returns `MAX_QPS_LIMIT`, BE identifies the top-k highest-QPS tables and progressively reduces their QPS limits, while leaving other tables unaffected. **Scope:** Only 5 load-related RPC types participate in table-level throttling: - `PREPARE_ROWSET` - `COMMIT_ROWSET` - `UPDATE_TMP_ROWSET` - `UPDATE_PACKED_FILE_INFO` - `UPDATE_DELETE_BITMAP` **Architecture (4 components with clear separation of concerns):** ``` MS_BUSY signal (MAX_QPS_LIMIT) │ ▼ ┌─────────────────────────┐ ┌──────────────────────────┐ │ RpcThrottleCoordinator │──────▶│ RpcThrottleStateMachine │ │ (timing control) │ │ (pure state logic) │ │ - upgrade cooldown │ │ - upgrade history stack │ │ - downgrade trigger │ │ - limit calculation │ └─────────────────────────┘ └──────────┬───────────────┘ │ Actions ▼ ┌─────────────────────────┐ ┌──────────────────────────┐ │ TableRpcQpsRegistry │ │ TableRpcThrottler │ │ (QPS statistics) │ │ (limit enforcement) │ │ - per-table bvar │ │ - StrictQpsLimiter │ │ - top-k query │ │ - per (rpc, table) │ └─────────────────────────┘ └──────────────────────────┘ ``` **Component details:** 1. **`TableRpcQpsRegistry`** — Tracks per-(rpc_type, table_id) QPS using `bvar::PerSecond<bvar::Adder>`. Supports efficient top-k query via min-heap. Configurable time window via `ms_rpc_table_qps_window_sec` (immutable, default 10s). 2. **`RpcThrottleStateMachine`** — Pure state machine with no time awareness or side effects. Maintains upgrade history as a stack for clean rollback. - `on_upgrade(snapshot)`: For each top-k table in the QPS snapshot, calculates `new_limit = current_qps × ratio` (first time) or `current_limit × ratio` (already limited), with a floor of `ms_rpc_table_qps_limit_floor`. Returns `SET_LIMIT` actions. - `on_downgrade()`: Pops the most recent upgrade from history. If the table had a prior limit, restores it (`SET_LIMIT`). If no prior limit, removes it (`REMOVE_LIMIT`). 3. **`RpcThrottleCoordinator`** — Timing control layer using tick counts (1 tick = 1 ms). - `report_ms_busy()`: Returns true if enough ticks have passed since last upgrade (cooldown). - `tick(n)`: Advances time by n ticks. Returns true if downgrade should trigger (no MS_BUSY for `downgrade_after_ticks`). 4. **`TableRpcThrottler`** — Enforces QPS limits using `StrictQpsLimiter` (strict fixed-interval, no burst allowed). Each (rpc_type, table_id) pair has its own limiter. Returns the time point when the request may execute; the caller sleeps until then. 5. **`MSBackpressureHandler`** — Orchestrator that wires all components together: - `on_ms_busy()`: Called when `retry_rpc` receives `MAX_QPS_LIMIT`. Consults coordinator for cooldown, builds QPS snapshot from registry, feeds to state machine, applies resulting actions to throttler. - `before_rpc()` / `after_rpc()`: Called around each load-related RPC for throttle enforcement and QPS recording. - Background tick thread: Runs every 1 second, advances coordinator by 1000 ticks. Triggers downgrade when enough time has passed without MS_BUSY. **Upgrade/Downgrade lifecycle example:** ``` Time 0s: MS returns MAX_QPS_LIMIT → Upgrade level 1: top-2 tables (A: 100 qps, B: 80 qps) → A limited to 50 qps, B limited to 40 qps Time 2s: MS returns MAX_QPS_LIMIT again (cooldown 5s not passed) → Skipped Time 6s: MS returns MAX_QPS_LIMIT (cooldown passed) → Upgrade level 2: top-2 tables now (A: 50 qps, C: 60 qps) → A limited to 25 qps, C limited to 30 qps Time 11s: No MS_BUSY for 5s → Downgrade: undo level 2 → A restored to 50 qps, C limit removed Time 16s: No MS_BUSY for 5s → Downgrade: undo level 1 → A limit removed, B limit removed ``` **Configuration:** | Config | Default | Mutable | Description | |--------|---------|---------|-------------| | `enable_ms_backpressure_handling` | `false` | Yes | Global enable/disable switch | | `ms_rpc_table_qps_window_sec` | `3` | No | bvar time window for QPS calculation | | `ms_backpressure_upgrade_interval_ms` | `3000` | Yes | Minimum cooldown between upgrades | | `ms_backpressure_upgrade_top_k` | `2` | Yes | Number of top tables to throttle per upgrade | | `ms_backpressure_throttle_ratio` | `0.75` | Yes | QPS decay ratio on upgrade | | `ms_rpc_table_qps_limit_floor` | `1.0` | Yes | Minimum QPS limit (won't throttle below this) | | `ms_backpressure_downgrade_interval_ms` | `3000` | Yes | Time without MS_BUSY before downgrade | **Observability (bvar metrics):** - `ms_rpc_backpressure_upgrade_count` / `_60s` — Upgrade event counts - `ms_rpc_backpressure_downgrade_count` / `_60s` — Downgrade event counts - `ms_rpc_backpressure_ms_busy_count` / `_60s` — MS_BUSY signal counts - `ms_rpc_backpressure_throttle_wait_<rpc_name>` — Per-RPC-type throttle wait latency - `ms_rpc_backpressure_throttled_tables_<rpc_name>` — Number of throttled tables per RPC type **New files:** - `be/src/cloud/cloud_throttle_state_machine.h` / `.cpp` - `be/src/cloud/cloud_ms_backpressure_handler.h` / `.cpp` - `be/test/cloud/cloud_throttle_state_machine_test.cpp` - `be/test/cloud/cloud_ms_backpressure_handler_test.cpp` **Also renamed (not part of the feature, cleanup):** - `common/cpp/s3_rate_limiter.h/.cpp` → `common/cpp/token_bucket_rate_limiter.h/.cpp` (more general naming since it's now used beyond S3) ## Part 3: System Table for Table-Level Throttler Observability ### Problem The table-level backpressure system operates transparently inside BE. When issues arise, users and DBAs have no way to inspect which tables are being throttled, what their QPS limits are, or what their current QPS is — beyond checking raw bvar metrics. ### Solution Add a new system table `information_schema.backend_ms_rpc_table_throttlers` that exposes the real-time state of the `TableRpcThrottler` on each BE. This table is a **Backend-Partitioned Schema Table**, meaning each BE reports its own throttling data, and queries are distributed to all alive BEs and aggregated. **Schema:** | Column | Type | Description | |--------|------|-------------| | `BE_ID` | BIGINT | Backend ID | | `TABLE_ID` | BIGINT | Table ID being throttled | | `RPC_TYPE` | VARCHAR(64) | RPC type name (e.g., `PREPARE_ROWSET`, `COMMIT_ROWSET`) | | `QPS_LIMIT` | DOUBLE | Current QPS limit enforced on this (table, rpc) pair | | `CURRENT_QPS` | DOUBLE | Current observed QPS for this (table, rpc) pair | **Usage examples:** ```sql -- View all currently throttled tables across all BEs SELECT * FROM information_schema.backend_ms_rpc_table_throttlers; -- View throttled tables on a specific BE SELECT * FROM information_schema.backend_ms_rpc_table_throttlers WHERE BE_ID = 10001; -- Find the most severely throttled tables SELECT * FROM information_schema.backend_ms_rpc_table_throttlers ORDER BY QPS_LIMIT ASC; ``` ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> - Added host-level token-bucket rate limiting for all MS RPCs (enabled by default via `enable_ms_rpc_host_level_rate_limit`) - Added table-level adaptive backpressure handling triggered by MS `MAX_QPS_LIMIT` response (disabled by default via `enable_ms_backpressure_handling`)
1 parent 2dc0d0a commit eaa4ef9

51 files changed

Lines changed: 4417 additions & 107 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/cloud/cloud_cumulative_compaction.cpp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -434,7 +434,8 @@ Status CloudCumulativeCompaction::modify_rowsets() {
434434
DBUG_BLOCK);
435435
auto status = _engine.meta_mgr().cloud_update_delete_bitmap_without_lock(
436436
*cloud_tablet(), pre_rowsets_delete_bitmap.get(), pre_rowset_to_versions,
437-
_output_rowset->start_version(), _output_rowset->end_version());
437+
cloud_tablet()->table_id(), _output_rowset->start_version(),
438+
_output_rowset->end_version());
438439
if (!status.ok()) {
439440
LOG(WARNING) << "failed to agg pre rowsets delete bitmap to ms. tablet_id="
440441
<< _tablet->tablet_id() << ", pre rowset num=" << pre_rowsets.size()

‎be/src/cloud/cloud_delete_task.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,13 +96,14 @@ Status CloudDeleteTask::execute(CloudStorageEngine& engine, const TPushReq& requ
9696
RETURN_IF_ERROR(rowset_writer->build(rowset));
9797
rowset->rowset_meta()->set_delete_predicate(std::move(del_pred));
9898

99-
auto st = engine.meta_mgr().prepare_rowset(*rowset_writer->rowset_meta(), "");
99+
auto st =
100+
engine.meta_mgr().prepare_rowset(*rowset_writer->rowset_meta(), "", tablet->table_id());
100101
if (!st.ok()) {
101102
LOG(WARNING) << "failed to prepare rowset, status=" << st.to_string();
102103
return st;
103104
}
104105

105-
st = engine.meta_mgr().commit_rowset(*rowset->rowset_meta(), "");
106+
st = engine.meta_mgr().commit_rowset(*rowset->rowset_meta(), "", tablet->table_id());
106107
if (!st.ok()) {
107108
LOG(WARNING) << "failed to commit rowset, status=" << st.to_string();
108109
return st;

‎be/src/cloud/cloud_delta_writer.cpp‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,8 @@ Status CloudDeltaWriter::commit_rowset() {
120120
}
121121

122122
// Handle normal rowset with data
123-
return _engine.meta_mgr().commit_rowset(*rowset_meta(), "");
123+
return _engine.meta_mgr().commit_rowset(*rowset_meta(), "",
124+
_rowset_builder->tablet()->table_id());
124125
}
125126

126127
Status CloudDeltaWriter::_commit_empty_rowset() {
@@ -138,7 +139,8 @@ Status CloudDeltaWriter::_commit_empty_rowset() {
138139
return Status::OK();
139140
}
140141
// write a empty rowset kv to keep version continuous
141-
return _engine.meta_mgr().commit_rowset(*rowset_meta(), "");
142+
return _engine.meta_mgr().commit_rowset(*rowset_meta(), "",
143+
_rowset_builder->tablet()->table_id());
142144
}
143145

144146
Status CloudDeltaWriter::set_txn_related_info() {

‎be/src/cloud/cloud_full_compaction.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -379,7 +379,7 @@ Status CloudFullCompaction::_cloud_full_compaction_update_delete_bitmap(int64_t
379379
RETURN_IF_ERROR(_engine.meta_mgr().update_delete_bitmap(
380380
*cloud_tablet(), -1, initiator, delete_bitmap.get(), delete_bitmap.get(),
381381
_output_rowset->rowset_id().to_string(), storage_resource,
382-
config::delete_bitmap_store_write_version));
382+
config::delete_bitmap_store_write_version, cloud_tablet()->table_id()));
383383
LOG_INFO("update delete bitmap in CloudFullCompaction, tablet_id={}, range=[{}-{}]",
384384
_tablet->tablet_id(), _input_rowsets.front()->start_version(),
385385
_input_rowsets.back()->end_version())

‎be/src/cloud/cloud_meta_mgr.cpp‎

Lines changed: 266 additions & 40 deletions
Large diffs are not rendered by default.

‎be/src/cloud/cloud_meta_mgr.h‎

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,8 @@ class StartTabletJobResponse;
4848
class TabletJobInfoPB;
4949
class TabletStatsPB;
5050
class TabletIndexPB;
51+
class HostLevelMSRpcRateLimiters;
52+
class MSBackpressureHandler;
5153

5254
using StorageVaultInfos = std::vector<
5355
std::tuple<std::string, std::variant<S3Conf, HdfsVaultInfo>, StorageVaultPB_PathFormat>>;
@@ -77,17 +79,18 @@ class CloudMetaMgr {
7779
CloudTablet* tablet, std::unique_lock<bthread::Mutex>& lock /* _sync_meta_lock */,
7880
const SyncOptions& options = {}, SyncRowsetStats* sync_stats = nullptr);
7981

80-
Status prepare_rowset(const RowsetMeta& rs_meta, const std::string& job_id,
82+
Status prepare_rowset(const RowsetMeta& rs_meta, const std::string& job_id, int64_t table_id,
8183
std::shared_ptr<RowsetMeta>* existed_rs_meta = nullptr);
8284

83-
Status commit_rowset(RowsetMeta& rs_meta, const std::string& job_id,
85+
Status commit_rowset(RowsetMeta& rs_meta, const std::string& job_id, int64_t table_id,
8486
std::shared_ptr<RowsetMeta>* existed_rs_meta = nullptr);
8587
void cache_committed_rowset(RowsetMetaSharedPtr rs_meta, int64_t expiration_time);
8688

87-
Status update_tmp_rowset(const RowsetMeta& rs_meta);
89+
Status update_tmp_rowset(const RowsetMeta& rs_meta, int64_t table_id);
8890

8991
Status update_packed_file_info(const std::string& packed_file_path,
90-
const cloud::PackedFileInfoPB& packed_file_info);
92+
const cloud::PackedFileInfoPB& packed_file_info,
93+
int64_t table_id);
9194

9295
Status commit_txn(const StreamLoadContext& ctx, bool is_2pc);
9396

@@ -141,12 +144,12 @@ class CloudMetaMgr {
141144
DeleteBitmap* delete_bitmap, DeleteBitmap* delete_bitmap_v2,
142145
std::string rowset_id,
143146
std::optional<StorageResource> storage_resource,
144-
int64_t store_version, int64_t txn_id = -1,
147+
int64_t store_version, int64_t table_id, int64_t txn_id = -1,
145148
bool is_explicit_txn = false, int64_t next_visible_version = -1);
146149

147150
Status cloud_update_delete_bitmap_without_lock(
148151
const CloudTablet& tablet, DeleteBitmap* delete_bitmap,
149-
std::map<std::string, int64_t>& rowset_to_versions,
152+
std::map<std::string, int64_t>& rowset_to_versions, int64_t table_id,
150153
int64_t pre_rowset_agg_start_version = 0, int64_t pre_rowset_agg_end_version = 0);
151154

152155
Status get_delete_bitmap_update_lock(const CloudTablet& tablet, int64_t lock_id,
@@ -175,6 +178,14 @@ class CloudMetaMgr {
175178
Status get_cluster_status(std::unordered_map<std::string, std::pair<int32_t, int64_t>>* result,
176179
std::string* my_cluster_id = nullptr);
177180

181+
void set_host_level_ms_rpc_rate_limiters(HostLevelMSRpcRateLimiters* limiters) {
182+
host_level_ms_rpc_rate_limiters_ = limiters;
183+
}
184+
185+
void set_ms_backpressure_handler(MSBackpressureHandler* handler) {
186+
ms_backpressure_handler_ = handler;
187+
}
188+
178189
private:
179190
bool sync_tablet_delete_bitmap_by_cache(CloudTablet* tablet, std::ranges::range auto&& rs_metas,
180191
DeleteBitmap* delete_bitmap);
@@ -202,6 +213,9 @@ class CloudMetaMgr {
202213
void check_table_size_correctness(RowsetMeta& rs_meta);
203214
int64_t get_segment_file_size(RowsetMeta& rs_meta);
204215
int64_t get_inverted_index_file_size(RowsetMeta& rs_meta);
216+
217+
HostLevelMSRpcRateLimiters* host_level_ms_rpc_rate_limiters_ {nullptr};
218+
MSBackpressureHandler* ms_backpressure_handler_ {nullptr};
205219
};
206220

207221
} // namespace cloud

0 commit comments

Comments
 (0)