Repository navigation
Commit eaa4ef9
authored
[Opt](cloud) Add rate limit for BE to MS rpc (#60344)
This PR implements a two-layer MS (Meta Service) RPC rate limiting
system for Doris cloud mode BE:
1. **Host-level rate limiting** — Token-bucket based QPS limiter for all
21 MS RPC types, preventing a single BE from overwhelming the MS with
burst traffic.
2. **Table-level adaptive backpressure** — When MS returns `MS_BUSY`
error code, BE dynamically identifies and throttles the top-k
highest-QPS tables using a state machine, and automatically relaxes
limits after the pressure subsides.
---
## Part 1: BE Host-Level Rate Limiting
### Problem
In cloud mode, all BE nodes send RPCs (get_tablet, prepare_rowset,
commit_rowset, etc.) to a shared Meta Service. A single BE experiencing
load spikes (e.g., large batch imports, compaction storms) can send
excessive RPC traffic that overwhelms MS, degrading service for all BEs.
### Solution
Introduce `HostLevelMSRpcRateLimiters` — a per-BE, per-RPC-type rate
limiter using token bucket algorithm.
**Architecture:**
- All 21 MS RPC types are enumerated in `MetaServiceRPC` enum (defined
via X-macro for maintainability)
- Each RPC type has an independent `TokenBucketRateLimiterHolder` with
its own QPS limit
- QPS limits are configured per CPU core: `actual_qps = config_value ×
num_cores`
- Thread-safe design using `atomic_shared_ptr<RpcRateLimiter>` array for
lock-free concurrent access during `limit()` calls
- Each rate limiter includes a `bvar::LatencyRecorder` to monitor sleep
durations caused by rate limiting
**Configuration:**
| Config | Default | Description |
|--------|---------|-------------|
| `enable_ms_rpc_host_level_rate_limit` | `true` | Global enable/disable
switch |
| `ms_rpc_qps_default` | `100` | Default per-core QPS for all RPCs |
| `ms_rpc_qps_<rpc_name>` | `-1` | Per-RPC override (`-1` = use default,
`0` = disabled) |
All QPS configs are mutable (`DEFINE_mInt32`), allowing runtime
adjustment without restart. `reset_all()` re-reads configs and recreates
rate limiters.
**Integration:**
Rate limiting is applied inside the `retry_rpc()` template function in
`cloud_meta_mgr.cpp`, which wraps all MS RPC calls. The
`RpcRateLimitCtx` struct carries the rate limiter reference. Rate
limiting executes before each RPC attempt (including retries), with the
call to `apply_rate_limit()` performing a `bthread_usleep` if the token
bucket requires waiting.
**New files:**
- `be/src/cloud/cloud_ms_rpc_rate_limiters.h` / `.cpp`
- `be/test/cloud/cloud_ms_rpc_rate_limiters_test.cpp`
---
## Part 2: BE Table-Level Adaptive Backpressure
### Problem
Host-level rate limiting applies uniformly across all tables. When MS
reports overload (`MAX_QPS_LIMIT`), it's often caused by a small number
of high-traffic tables (e.g., tables with many concurrent stream load
jobs). A uniform rate limit would unnecessarily penalize all tables,
while the hot tables continue to dominate the RPC traffic.
### Solution
Implement table-level adaptive throttling for load-related RPCs. When MS
returns `MAX_QPS_LIMIT`, BE identifies the top-k highest-QPS tables and
progressively reduces their QPS limits, while leaving other tables
unaffected.
**Scope:** Only 5 load-related RPC types participate in table-level
throttling:
- `PREPARE_ROWSET`
- `COMMIT_ROWSET`
- `UPDATE_TMP_ROWSET`
- `UPDATE_PACKED_FILE_INFO`
- `UPDATE_DELETE_BITMAP`
**Architecture (4 components with clear separation of concerns):**
```
MS_BUSY signal (MAX_QPS_LIMIT)
│
▼
┌─────────────────────────┐ ┌──────────────────────────┐
│ RpcThrottleCoordinator │──────▶│ RpcThrottleStateMachine │
│ (timing control) │ │ (pure state logic) │
│ - upgrade cooldown │ │ - upgrade history stack │
│ - downgrade trigger │ │ - limit calculation │
└─────────────────────────┘ └──────────┬───────────────┘
│ Actions
▼
┌─────────────────────────┐ ┌──────────────────────────┐
│ TableRpcQpsRegistry │ │ TableRpcThrottler │
│ (QPS statistics) │ │ (limit enforcement) │
│ - per-table bvar │ │ - StrictQpsLimiter │
│ - top-k query │ │ - per (rpc, table) │
└─────────────────────────┘ └──────────────────────────┘
```
**Component details:**
1. **`TableRpcQpsRegistry`** — Tracks per-(rpc_type, table_id) QPS using
`bvar::PerSecond<bvar::Adder>`. Supports efficient top-k query via
min-heap. Configurable time window via `ms_rpc_table_qps_window_sec`
(immutable, default 10s).
2. **`RpcThrottleStateMachine`** — Pure state machine with no time
awareness or side effects. Maintains upgrade history as a stack for
clean rollback.
- `on_upgrade(snapshot)`: For each top-k table in the QPS snapshot,
calculates `new_limit = current_qps × ratio` (first time) or
`current_limit × ratio` (already limited), with a floor of
`ms_rpc_table_qps_limit_floor`. Returns `SET_LIMIT` actions.
- `on_downgrade()`: Pops the most recent upgrade from history. If the
table had a prior limit, restores it (`SET_LIMIT`). If no prior limit,
removes it (`REMOVE_LIMIT`).
3. **`RpcThrottleCoordinator`** — Timing control layer using tick counts
(1 tick = 1 ms).
- `report_ms_busy()`: Returns true if enough ticks have passed since
last upgrade (cooldown).
- `tick(n)`: Advances time by n ticks. Returns true if downgrade should
trigger (no MS_BUSY for `downgrade_after_ticks`).
4. **`TableRpcThrottler`** — Enforces QPS limits using
`StrictQpsLimiter` (strict fixed-interval, no burst allowed). Each
(rpc_type, table_id) pair has its own limiter. Returns the time point
when the request may execute; the caller sleeps until then.
5. **`MSBackpressureHandler`** — Orchestrator that wires all components
together:
- `on_ms_busy()`: Called when `retry_rpc` receives `MAX_QPS_LIMIT`.
Consults coordinator for cooldown, builds QPS snapshot from registry,
feeds to state machine, applies resulting actions to throttler.
- `before_rpc()` / `after_rpc()`: Called around each load-related RPC
for throttle enforcement and QPS recording.
- Background tick thread: Runs every 1 second, advances coordinator by
1000 ticks. Triggers downgrade when enough time has passed without
MS_BUSY.
**Upgrade/Downgrade lifecycle example:**
```
Time 0s: MS returns MAX_QPS_LIMIT
→ Upgrade level 1: top-2 tables (A: 100 qps, B: 80 qps)
→ A limited to 50 qps, B limited to 40 qps
Time 2s: MS returns MAX_QPS_LIMIT again (cooldown 5s not passed)
→ Skipped
Time 6s: MS returns MAX_QPS_LIMIT (cooldown passed)
→ Upgrade level 2: top-2 tables now (A: 50 qps, C: 60 qps)
→ A limited to 25 qps, C limited to 30 qps
Time 11s: No MS_BUSY for 5s
→ Downgrade: undo level 2
→ A restored to 50 qps, C limit removed
Time 16s: No MS_BUSY for 5s
→ Downgrade: undo level 1
→ A limit removed, B limit removed
```
**Configuration:**
| Config | Default | Mutable | Description |
|--------|---------|---------|-------------|
| `enable_ms_backpressure_handling` | `false` | Yes | Global
enable/disable switch |
| `ms_rpc_table_qps_window_sec` | `3` | No | bvar time window for QPS
calculation |
| `ms_backpressure_upgrade_interval_ms` | `3000` | Yes | Minimum
cooldown between upgrades |
| `ms_backpressure_upgrade_top_k` | `2` | Yes | Number of top tables to
throttle per upgrade |
| `ms_backpressure_throttle_ratio` | `0.75` | Yes | QPS decay ratio on
upgrade |
| `ms_rpc_table_qps_limit_floor` | `1.0` | Yes | Minimum QPS limit
(won't throttle below this) |
| `ms_backpressure_downgrade_interval_ms` | `3000` | Yes | Time without
MS_BUSY before downgrade |
**Observability (bvar metrics):**
- `ms_rpc_backpressure_upgrade_count` / `_60s` — Upgrade event counts
- `ms_rpc_backpressure_downgrade_count` / `_60s` — Downgrade event
counts
- `ms_rpc_backpressure_ms_busy_count` / `_60s` — MS_BUSY signal counts
- `ms_rpc_backpressure_throttle_wait_<rpc_name>` — Per-RPC-type throttle
wait latency
- `ms_rpc_backpressure_throttled_tables_<rpc_name>` — Number of
throttled tables per RPC type
**New files:**
- `be/src/cloud/cloud_throttle_state_machine.h` / `.cpp`
- `be/src/cloud/cloud_ms_backpressure_handler.h` / `.cpp`
- `be/test/cloud/cloud_throttle_state_machine_test.cpp`
- `be/test/cloud/cloud_ms_backpressure_handler_test.cpp`
**Also renamed (not part of the feature, cleanup):**
- `common/cpp/s3_rate_limiter.h/.cpp` →
`common/cpp/token_bucket_rate_limiter.h/.cpp` (more general naming since
it's now used beyond S3)
## Part 3: System Table for Table-Level Throttler Observability
### Problem
The table-level backpressure system operates transparently inside BE.
When issues arise, users and DBAs have no way to inspect which tables
are being throttled, what their QPS limits are, or what their current
QPS is — beyond checking raw bvar metrics.
### Solution
Add a new system table
`information_schema.backend_ms_rpc_table_throttlers` that exposes the
real-time state of the `TableRpcThrottler` on each BE. This table is a
**Backend-Partitioned Schema Table**, meaning each BE reports its own
throttling data, and queries are
distributed to all alive BEs and aggregated.
**Schema:**
| Column | Type | Description |
|--------|------|-------------|
| `BE_ID` | BIGINT | Backend ID |
| `TABLE_ID` | BIGINT | Table ID being throttled |
| `RPC_TYPE` | VARCHAR(64) | RPC type name (e.g., `PREPARE_ROWSET`,
`COMMIT_ROWSET`) |
| `QPS_LIMIT` | DOUBLE | Current QPS limit enforced on this (table, rpc)
pair |
| `CURRENT_QPS` | DOUBLE | Current observed QPS for this (table, rpc)
pair |
**Usage examples:**
```sql
-- View all currently throttled tables across all BEs
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers;
-- View throttled tables on a specific BE
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers WHERE BE_ID = 10001;
-- Find the most severely throttled tables
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers ORDER BY QPS_LIMIT ASC;
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Added host-level token-bucket rate limiting for all MS RPCs (enabled
by default via `enable_ms_rpc_host_level_rate_limit`)
- Added table-level adaptive backpressure handling triggered by MS
`MAX_QPS_LIMIT` response (disabled by default via
`enable_ms_backpressure_handling`)1 parent 2dc0d0a commit eaa4ef9
51 files changed
Lines changed: 4417 additions & 107 deletions
File tree
- be
- src
- cloud
- information_schema
- io
- cache
- fs
- runtime
- storage/compaction
- util
- test/cloud
- cloud
- src
- common
- meta-service
- recycler
- test
- common/cpp
- fe/fe-core/src/main/java/org/apache/doris
- analysis
- catalog
- planner
- gensrc/thrift
- regression-test/suites/fault_injection_p0/cloud
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
434 | 434 | | |
435 | 435 | | |
436 | 436 | | |
437 | | - | |
| 437 | + | |
| 438 | + | |
438 | 439 | | |
439 | 440 | | |
440 | 441 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
96 | 96 | | |
97 | 97 | | |
98 | 98 | | |
99 | | - | |
| 99 | + | |
| 100 | + | |
100 | 101 | | |
101 | 102 | | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
105 | | - | |
| 106 | + | |
106 | 107 | | |
107 | 108 | | |
108 | 109 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
| 123 | + | |
| 124 | + | |
124 | 125 | | |
125 | 126 | | |
126 | 127 | | |
| |||
138 | 139 | | |
139 | 140 | | |
140 | 141 | | |
141 | | - | |
| 142 | + | |
| 143 | + | |
142 | 144 | | |
143 | 145 | | |
144 | 146 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
379 | 379 | | |
380 | 380 | | |
381 | 381 | | |
382 | | - | |
| 382 | + | |
383 | 383 | | |
384 | 384 | | |
385 | 385 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
| |||
77 | 79 | | |
78 | 80 | | |
79 | 81 | | |
80 | | - | |
| 82 | + | |
81 | 83 | | |
82 | 84 | | |
83 | | - | |
| 85 | + | |
84 | 86 | | |
85 | 87 | | |
86 | 88 | | |
87 | | - | |
| 89 | + | |
88 | 90 | | |
89 | 91 | | |
90 | | - | |
| 92 | + | |
| 93 | + | |
91 | 94 | | |
92 | 95 | | |
93 | 96 | | |
| |||
141 | 144 | | |
142 | 145 | | |
143 | 146 | | |
144 | | - | |
| 147 | + | |
145 | 148 | | |
146 | 149 | | |
147 | 150 | | |
148 | 151 | | |
149 | | - | |
| 152 | + | |
150 | 153 | | |
151 | 154 | | |
152 | 155 | | |
| |||
175 | 178 | | |
176 | 179 | | |
177 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
178 | 189 | | |
179 | 190 | | |
180 | 191 | | |
| |||
202 | 213 | | |
203 | 214 | | |
204 | 215 | | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
205 | 219 | | |
206 | 220 | | |
207 | 221 | | |
| |||
0 commit comments