Repository navigation
Commit 92f75f3
authored
[feature](memory) Global mem control on scan nodes (apache#61271)
📋 Overview
This commit introduces a global memory control system for scan nodes in
Apache Doris, enabling adaptive scanner concurrency management based on
available memory. The system prevents OOM (Out Of Memory) errors by
dynamically
adjusting scanner parallelism according to query-level memory
constraints.
---
🏗️ Architecture: Three-Layer Memory Control System
Layer 1: Query Level - ScannerMemShareArbitrator
Purpose: Fair memory distribution across all scan operators in a query
QueryContext (owns)
└── ScannerMemShareArbitrator
├── query_mem_limit: Total query memory (e.g., 100GB)
├── scan_mem_limit: Memory allocated for all scans (query_mem_limit ×
max_scan_mem_ratio)
└── total_scanner_mem_bytes: Atomic counter tracking total memory usage
Key Responsibilities:
- Distributes scan_mem_limit proportionally across multiple scan nodes
- Tracks global scanner memory usage atomically
- Implements proportional sharing algorithm
Layer 2: Scan Node Level - ScannerMemLimiter
Purpose: Control scanner concurrency for a specific scan node across all
instances
ScanOperatorX (owns)
└── ScannerMemLimiter
├── parallelism: Number of parallel instances
├── scan_mem_limit: Memory allocated by arbitrator (updated dynamically)
├── estimated_block_mem_bytes: Average memory per block (running
average)
└── running_scanner_count: Active scanners across all instances
Key Responsibilities:
- Calculates available scanner count based on memory
- Maintains running average of block memory usage
- Distributes scanners fairly across parallel instances
Layer 3: Instance Level - ScannerContext
Purpose: Manages scanners for a single scan operator instance
ScannerContext (per instance)
├── References: _mem_share_arb, _scanner_mem_limiter
├── _ins_idx: Instance index (0, 1, 2, ...)
├── _enable_adaptive_scanners: Feature flag
└── _adaptive_processor: Adaptive scheduling logic
---
🔄 How It Works: Memory Flow in a Query
Phase 1: Initialization (Query Start)
1. QueryContext created
└── Creates ScannerMemShareArbitrator
├── query_mem_limit = 100GB (from session variable)
├── max_scan_mem_ratio = 0.3 (30% of query memory for scans)
└── scan_mem_limit = 30GB
2. Each ScanOperatorX (Scan Node) initialized
└── Registers with arbitrator
└── arbitrator->register_scan_node()
└── total_scanner_mem_bytes += 64MB (DEFAULT_SCANNER_MEM_BYTES)
└── Creates ScannerMemLimiter
├── parallelism = 4 (query has 4 parallel instances)
├── serial_scan = false
└── query_scan_mem_limit = 30GB
3. Each ScannerContext (Instance) initialized
└── ScannerContext::init()
├── update_open_scanner_context_count(1)
├── reestimated_block_mem_bytes(64MB) // Initial estimate
└── _adjust_scan_mem_limit(64MB, 64MB) // Register with arbitrator
---
Phase 2: Runtime Adaptation (During Execution)
Memory Re-estimation Loop
// When a block is scanned
ScannerContext::reestimated_block_mem_bytes(actual_block_size)
└─> ScannerMemLimiter::reestimated_block_mem_bytes(actual_block_size)
└─> Running average calculation:
new_estimate = (old_estimate × count + actual_size) / (count + 1)
// Example: 64MB → 100MB → avg = 82MB
Memory Limit Adjustment
ScannerContext::_adjust_scan_mem_limit(old_mem, new_mem)
└─> ScannerMemShareArbitrator::update_scanner_mem_bytes(old_mem,
new_mem)
├── Update total: total_scanner_mem_bytes += (new_mem - old_mem)
├── Calculate ratio: ratio = new_mem / total
└── Return proportional share: scan_mem_limit × ratio
└─> ScannerMemLimiter::update_scan_mem_limit(new_limit)
└── scan_mem_limit = new_limit (atomic update)
Scanner Count Calculation
ScannerContext::_available_pickup_scanner_count()
└─> ScannerMemLimiter::available_scanner_count(ins_idx)
├── max_count = scan_mem_limit / estimated_block_mem_bytes
│ // Example: 10GB / 100MB = 100 scanners total
│
├── per_count = max_count / parallelism
│ // Example: 100 / 4 = 25 scanners per instance
│
└── Distribution logic:
if (serial_scan)
instance[0] gets all (max_count)
else
instances distribute evenly with remainder to first instances
// Instance 0,1,2,3 each gets 25 scanners
---
📊 Real-World Example
Scenario: Query with 2 scan nodes, 4 parallel instances
Query Memory: 100GB
max_scan_mem_ratio: 0.3
Scan Memory Budget: 30GB
Initial State
ScannerMemShareArbitrator:
scan_mem_limit = 30GB
total_scanner_mem_bytes = 128MB (2 nodes × 64MB)
ScanNode1_Limiter:
estimated_block_mem_bytes = 64MB
ScanNode2_Limiter:
estimated_block_mem_bytes = 64MB
After ScanNode1 scans some data
ScanNode1_Context[0]::reestimated_block_mem_bytes(200MB)
└─> ScanNode1_Limiter::estimated_block_mem_bytes = 200MB (updated)
└─> _adjust_scan_mem_limit(64MB, 200MB)
└─> Arbitrator updates:
total = 128MB - 64MB + 200MB = 264MB
ratio = 200MB / 264MB = 75.8%
ScanNode1_limit = 30GB × 75.8% = 22.7GB
ScanNode1_Limiter::available_scanner_count(0)
max_count = 22.7GB / 200MB = 116 scanners
per_count = 116 / 4 = 29 scanners per instance
ScanNode2 gets remaining memory
ScanNode2_Context[0]::_adjust_scan_mem_limit(64MB, 64MB)
└─> Arbitrator updates:
ratio = 64MB / 264MB = 24.2%
ScanNode2_limit = 30GB × 24.2% = 7.3GB
ScanNode2_Limiter::available_scanner_count(0)
max_count = 7.3GB / 64MB = 117 scanners
per_count = 117 / 4 = 29 scanners per instance
Result: Memory dynamically allocated based on actual usage!
---
🎯 Key Benefits
1. Prevents OOM
- Limits total scan memory to query_mem_limit × max_scan_mem_ratio
- Dynamically adjusts scanner concurrency to stay within limits
2. Fair Resource Sharing
- Multiple scan nodes share memory proportionally
- Nodes using more memory get proportionally more budget
3. Adaptive to Workload
- Small blocks → More scanners allowed
- Large blocks → Fewer scanners to prevent OOM
4. Instance-Level Fairness
- Scanners distributed evenly across parallel instances
- Serial mode gives all scanners to first instance
---
🔧 Configuration
Session Variables (FE)
-- Enable adaptive scanning (default: varies)
SET enable_adaptive_scan = true;
-- Max ratio of query memory for scans (default: 0.3 = 30%)
SET max_scan_mem_ratio = 0.3;
-- Total query memory limit
SET exec_mem_limit = 107374182400; -- 100GB
BE Configuration
// Minimum interval between scanner count adjustments
config::doris_scanner_dynamic_interval_ms = 100; // 100ms
---
📈 Algorithm Deep Dive
Proportional Sharing Algorithm
int64_t ScannerMemShareArbitrator::update_scanner_mem_bytes(old_value,
new_value) {
// Atomic update of total
int64_t diff = new_value - old_value;
int64_t total = total_scanner_mem_bytes.fetch_add(diff) + diff;
// Special cases
if (new_value == 0) return 0; // Context closing
if (total <= 0) return scan_mem_limit; // Only context active
// Proportional allocation
double ratio = new_value / max(total, new_value);
return scan_mem_limit × ratio;
}
Example Calculation:
- ScanNode1 uses 200MB, ScanNode2 uses 64MB → total = 264MB
- ScanNode1 ratio = 200/264 = 75.8% → gets 22.7GB
- ScanNode2 ratio = 64/264 = 24.2% → gets 7.3GB
- Total: 22.7GB + 7.3GB = 30GB ✓
Scanner Distribution Algorithm
int ScannerMemLimiter::available_scanner_count(int ins_idx) const {
int64_t max_count = scan_mem_limit / estimated_block_mem_bytes;
int64_t per_count = max_count / parallelism;
if (serial_scan) {
// First instance gets all scanners + remainder
per_count += (max_count - per_count × parallelism);
} else if (ins_idx < max_count - per_count × parallelism) {
// Distribute remainder to first instances
per_count += 1;
}
return per_count;
}
Example (100 scanners, 4 instances):
- Base: 100 / 4 = 25 per instance
- Remainder: 100 % 4 = 0
- Distribution: [25, 25, 25, 25]
Example (103 scanners, 4 instances):
- Base: 103 / 4 = 25 per instance
- Remainder: 103 % 4 = 3
- Distribution: [26, 26, 26, 25] (first 3 get +1)
---
🔍 Lifecycle Management
Context Opening
ScannerContext::init() {
if (enable_adaptive_scanners) {
int64_t c = limiter->update_open_scanner_context_count(1);
limiter->reestimated_block_mem_bytes(64MB);
if (c == 0) { // First context for this scan node
_adjust_scan_mem_limit(64MB, 64MB); // Register with arbitrator
}
}
}
Context Closing
ScannerContext::~ScannerContext() {
if (enable_adaptive_scanners) {
if (limiter->update_open_scanner_context_count(-1) == 1) {
// Last context closing
_adjust_scan_mem_limit(current_mem, 0); // Release to arbitrator
}
}
}
---
🚀 Future Enhancements (TODOs in Code)
The commit includes commented-out code for future performance-based
adaptation:
1. IO Latency Monitoring: Adjust scanners based on disk/network speed
2. Source Speed Tracking: Monitor block production rate
3. Adaptive Speedup: Incrementally test if adding scanners improves
throughput
4. Slow IO Detection: Reduce scanners when IO becomes bottleneck
---
This comprehensive memory control system ensures Doris can handle
large-scale scans efficiently while preventing memory exhaustion, making
the system more robust and predictable under heavy workloads! 🎉1 parent 3139ee1 commit 92f75f3
23 files changed
Lines changed: 1195 additions & 325 deletions
File tree
- be
- src
- common
- exec
- common
- operator
- pipeline
- scan
- runtime
- test/exec/scan
- fe/fe-core/src/main/java/org/apache/doris/qe
- gensrc/thrift
- regression-test
- data
- mv_p0/test_casewhen
- unique_with_mow_p0/partial_update
- publish
- suites
- mv_p0/test_casewhen
- unique_with_mow_p0/partial_update
- publish
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
345 | 345 | | |
346 | 346 | | |
347 | 347 | | |
| 348 | + | |
348 | 349 | | |
349 | 350 | | |
350 | 351 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
402 | 402 | | |
403 | 403 | | |
404 | 404 | | |
405 | | - | |
| 405 | + | |
406 | 406 | | |
| 407 | + | |
| 408 | + | |
407 | 409 | | |
408 | 410 | | |
409 | 411 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
152 | 153 | | |
153 | 154 | | |
154 | 155 | | |
| |||
1008 | 1009 | | |
1009 | 1010 | | |
1010 | 1011 | | |
1011 | | - | |
1012 | | - | |
1013 | | - | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
1014 | 1017 | | |
1015 | | - | |
1016 | | - | |
| 1018 | + | |
| 1019 | + | |
1017 | 1020 | | |
1018 | | - | |
| 1021 | + | |
1019 | 1022 | | |
1020 | 1023 | | |
1021 | 1024 | | |
| |||
1182 | 1185 | | |
1183 | 1186 | | |
1184 | 1187 | | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
1185 | 1200 | | |
1186 | 1201 | | |
1187 | 1202 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
336 | 336 | | |
337 | 337 | | |
338 | 338 | | |
| 339 | + | |
339 | 340 | | |
340 | 341 | | |
341 | 342 | | |
| |||
437 | 438 | | |
438 | 439 | | |
439 | 440 | | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
440 | 444 | | |
441 | 445 | | |
442 | 446 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
123 | 122 | | |
124 | 123 | | |
125 | 124 | | |
126 | | - | |
127 | 125 | | |
128 | 126 | | |
129 | 127 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
266 | 266 | | |
267 | 267 | | |
268 | 268 | | |
269 | | - | |
| 269 | + | |
270 | 270 | | |
271 | 271 | | |
272 | 272 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
127 | 127 | | |
128 | 128 | | |
129 | 129 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
| 130 | + | |
140 | 131 | | |
141 | 132 | | |
142 | | - | |
| 133 | + | |
143 | 134 | | |
144 | 135 | | |
145 | 136 | | |
146 | 137 | | |
147 | | - | |
| 138 | + | |
148 | 139 | | |
149 | 140 | | |
150 | 141 | | |
151 | 142 | | |
152 | | - | |
| 143 | + | |
| 144 | + | |
153 | 145 | | |
154 | | - | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
155 | 159 | | |
156 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
157 | 165 | | |
158 | 166 | | |
159 | 167 | | |
| |||
0 commit comments