Skip to content

Commit 92f75f3

Browse files
authored
[feature](memory) Global mem control on scan nodes (apache#61271)
📋 Overview This commit introduces a global memory control system for scan nodes in Apache Doris, enabling adaptive scanner concurrency management based on available memory. The system prevents OOM (Out Of Memory) errors by dynamically adjusting scanner parallelism according to query-level memory constraints. --- 🏗️ Architecture: Three-Layer Memory Control System Layer 1: Query Level - ScannerMemShareArbitrator Purpose: Fair memory distribution across all scan operators in a query QueryContext (owns) └── ScannerMemShareArbitrator ├── query_mem_limit: Total query memory (e.g., 100GB) ├── scan_mem_limit: Memory allocated for all scans (query_mem_limit × max_scan_mem_ratio) └── total_scanner_mem_bytes: Atomic counter tracking total memory usage Key Responsibilities: - Distributes scan_mem_limit proportionally across multiple scan nodes - Tracks global scanner memory usage atomically - Implements proportional sharing algorithm Layer 2: Scan Node Level - ScannerMemLimiter Purpose: Control scanner concurrency for a specific scan node across all instances ScanOperatorX (owns) └── ScannerMemLimiter ├── parallelism: Number of parallel instances ├── scan_mem_limit: Memory allocated by arbitrator (updated dynamically) ├── estimated_block_mem_bytes: Average memory per block (running average) └── running_scanner_count: Active scanners across all instances Key Responsibilities: - Calculates available scanner count based on memory - Maintains running average of block memory usage - Distributes scanners fairly across parallel instances Layer 3: Instance Level - ScannerContext Purpose: Manages scanners for a single scan operator instance ScannerContext (per instance) ├── References: _mem_share_arb, _scanner_mem_limiter ├── _ins_idx: Instance index (0, 1, 2, ...) ├── _enable_adaptive_scanners: Feature flag └── _adaptive_processor: Adaptive scheduling logic --- 🔄 How It Works: Memory Flow in a Query Phase 1: Initialization (Query Start) 1. QueryContext created └── Creates ScannerMemShareArbitrator ├── query_mem_limit = 100GB (from session variable) ├── max_scan_mem_ratio = 0.3 (30% of query memory for scans) └── scan_mem_limit = 30GB 2. Each ScanOperatorX (Scan Node) initialized └── Registers with arbitrator └── arbitrator->register_scan_node() └── total_scanner_mem_bytes += 64MB (DEFAULT_SCANNER_MEM_BYTES) └── Creates ScannerMemLimiter ├── parallelism = 4 (query has 4 parallel instances) ├── serial_scan = false └── query_scan_mem_limit = 30GB 3. Each ScannerContext (Instance) initialized └── ScannerContext::init() ├── update_open_scanner_context_count(1) ├── reestimated_block_mem_bytes(64MB) // Initial estimate └── _adjust_scan_mem_limit(64MB, 64MB) // Register with arbitrator --- Phase 2: Runtime Adaptation (During Execution) Memory Re-estimation Loop // When a block is scanned ScannerContext::reestimated_block_mem_bytes(actual_block_size) └─> ScannerMemLimiter::reestimated_block_mem_bytes(actual_block_size) └─> Running average calculation: new_estimate = (old_estimate × count + actual_size) / (count + 1) // Example: 64MB → 100MB → avg = 82MB Memory Limit Adjustment ScannerContext::_adjust_scan_mem_limit(old_mem, new_mem) └─> ScannerMemShareArbitrator::update_scanner_mem_bytes(old_mem, new_mem) ├── Update total: total_scanner_mem_bytes += (new_mem - old_mem) ├── Calculate ratio: ratio = new_mem / total └── Return proportional share: scan_mem_limit × ratio └─> ScannerMemLimiter::update_scan_mem_limit(new_limit) └── scan_mem_limit = new_limit (atomic update) Scanner Count Calculation ScannerContext::_available_pickup_scanner_count() └─> ScannerMemLimiter::available_scanner_count(ins_idx) ├── max_count = scan_mem_limit / estimated_block_mem_bytes │ // Example: 10GB / 100MB = 100 scanners total │ ├── per_count = max_count / parallelism │ // Example: 100 / 4 = 25 scanners per instance │ └── Distribution logic: if (serial_scan) instance[0] gets all (max_count) else instances distribute evenly with remainder to first instances // Instance 0,1,2,3 each gets 25 scanners --- 📊 Real-World Example Scenario: Query with 2 scan nodes, 4 parallel instances Query Memory: 100GB max_scan_mem_ratio: 0.3 Scan Memory Budget: 30GB Initial State ScannerMemShareArbitrator: scan_mem_limit = 30GB total_scanner_mem_bytes = 128MB (2 nodes × 64MB) ScanNode1_Limiter: estimated_block_mem_bytes = 64MB ScanNode2_Limiter: estimated_block_mem_bytes = 64MB After ScanNode1 scans some data ScanNode1_Context[0]::reestimated_block_mem_bytes(200MB) └─> ScanNode1_Limiter::estimated_block_mem_bytes = 200MB (updated) └─> _adjust_scan_mem_limit(64MB, 200MB) └─> Arbitrator updates: total = 128MB - 64MB + 200MB = 264MB ratio = 200MB / 264MB = 75.8% ScanNode1_limit = 30GB × 75.8% = 22.7GB ScanNode1_Limiter::available_scanner_count(0) max_count = 22.7GB / 200MB = 116 scanners per_count = 116 / 4 = 29 scanners per instance ScanNode2 gets remaining memory ScanNode2_Context[0]::_adjust_scan_mem_limit(64MB, 64MB) └─> Arbitrator updates: ratio = 64MB / 264MB = 24.2% ScanNode2_limit = 30GB × 24.2% = 7.3GB ScanNode2_Limiter::available_scanner_count(0) max_count = 7.3GB / 64MB = 117 scanners per_count = 117 / 4 = 29 scanners per instance Result: Memory dynamically allocated based on actual usage! --- 🎯 Key Benefits 1. Prevents OOM - Limits total scan memory to query_mem_limit × max_scan_mem_ratio - Dynamically adjusts scanner concurrency to stay within limits 2. Fair Resource Sharing - Multiple scan nodes share memory proportionally - Nodes using more memory get proportionally more budget 3. Adaptive to Workload - Small blocks → More scanners allowed - Large blocks → Fewer scanners to prevent OOM 4. Instance-Level Fairness - Scanners distributed evenly across parallel instances - Serial mode gives all scanners to first instance --- 🔧 Configuration Session Variables (FE) -- Enable adaptive scanning (default: varies) SET enable_adaptive_scan = true; -- Max ratio of query memory for scans (default: 0.3 = 30%) SET max_scan_mem_ratio = 0.3; -- Total query memory limit SET exec_mem_limit = 107374182400; -- 100GB BE Configuration // Minimum interval between scanner count adjustments config::doris_scanner_dynamic_interval_ms = 100; // 100ms --- 📈 Algorithm Deep Dive Proportional Sharing Algorithm int64_t ScannerMemShareArbitrator::update_scanner_mem_bytes(old_value, new_value) { // Atomic update of total int64_t diff = new_value - old_value; int64_t total = total_scanner_mem_bytes.fetch_add(diff) + diff; // Special cases if (new_value == 0) return 0; // Context closing if (total <= 0) return scan_mem_limit; // Only context active // Proportional allocation double ratio = new_value / max(total, new_value); return scan_mem_limit × ratio; } Example Calculation: - ScanNode1 uses 200MB, ScanNode2 uses 64MB → total = 264MB - ScanNode1 ratio = 200/264 = 75.8% → gets 22.7GB - ScanNode2 ratio = 64/264 = 24.2% → gets 7.3GB - Total: 22.7GB + 7.3GB = 30GB ✓ Scanner Distribution Algorithm int ScannerMemLimiter::available_scanner_count(int ins_idx) const { int64_t max_count = scan_mem_limit / estimated_block_mem_bytes; int64_t per_count = max_count / parallelism; if (serial_scan) { // First instance gets all scanners + remainder per_count += (max_count - per_count × parallelism); } else if (ins_idx < max_count - per_count × parallelism) { // Distribute remainder to first instances per_count += 1; } return per_count; } Example (100 scanners, 4 instances): - Base: 100 / 4 = 25 per instance - Remainder: 100 % 4 = 0 - Distribution: [25, 25, 25, 25] Example (103 scanners, 4 instances): - Base: 103 / 4 = 25 per instance - Remainder: 103 % 4 = 3 - Distribution: [26, 26, 26, 25] (first 3 get +1) --- 🔍 Lifecycle Management Context Opening ScannerContext::init() { if (enable_adaptive_scanners) { int64_t c = limiter->update_open_scanner_context_count(1); limiter->reestimated_block_mem_bytes(64MB); if (c == 0) { // First context for this scan node _adjust_scan_mem_limit(64MB, 64MB); // Register with arbitrator } } } Context Closing ScannerContext::~ScannerContext() { if (enable_adaptive_scanners) { if (limiter->update_open_scanner_context_count(-1) == 1) { // Last context closing _adjust_scan_mem_limit(current_mem, 0); // Release to arbitrator } } } --- 🚀 Future Enhancements (TODOs in Code) The commit includes commented-out code for future performance-based adaptation: 1. IO Latency Monitoring: Adjust scanners based on disk/network speed 2. Source Speed Tracking: Monitor block production rate 3. Adaptive Speedup: Incrementally test if adding scanners improves throughput 4. Slow IO Detection: Reduce scanners when IO becomes bottleneck --- This comprehensive memory control system ensures Doris can handle large-scale scans efficiently while preventing memory exhaustion, making the system more robust and predictable under heavy workloads! 🎉
1 parent 3139ee1 commit 92f75f3

23 files changed

Lines changed: 1195 additions & 325 deletions

File tree

‎be/src/common/config.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,7 @@ DEFINE_mInt32(doris_scanner_row_num, "16384");
345345
DEFINE_mInt32(doris_scanner_row_bytes, "10485760");
346346
// single read execute fragment max run time millseconds
347347
DEFINE_mInt32(doris_scanner_max_run_time_ms, "1000");
348+
DEFINE_mInt32(doris_scanner_dynamic_interval_ms, "100");
348349
// (Advanced) Maximum size of per-query receive-side buffer
349350
DEFINE_mInt32(exchg_node_buffer_size_bytes, "20485760");
350351
DEFINE_mInt32(exchg_buffer_queue_capacity_factor, "64");

‎be/src/common/config.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -402,8 +402,10 @@ DECLARE_mInt32(doris_scan_range_max_mb);
402402
DECLARE_mInt32(doris_scanner_row_num);
403403
// single read execute fragment row bytes
404404
DECLARE_mInt32(doris_scanner_row_bytes);
405-
// single read execute fragment max run time millseconds
405+
// Deprecated. single read execute fragment max run time millseconds
406406
DECLARE_mInt32(doris_scanner_max_run_time_ms);
407+
// Minimum interval in milliseconds between adaptive scanner concurrency adjustments
408+
DECLARE_mInt32(doris_scanner_dynamic_interval_ms);
407409
// (Advanced) Maximum size of per-query receive-side buffer
408410
DECLARE_mInt32(exchg_node_buffer_size_bytes);
409411
DECLARE_mInt32(exchg_buffer_queue_capacity_factor);

‎be/src/exec/common/memory.cpp‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
// This file is copied from
18+
// https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/formatIPv6.cpp
19+
// and modified by Doris
20+
21+
#include "exec/common/memory.h"
22+
23+
namespace doris {
24+
25+
MemShareArbitrator::MemShareArbitrator(const TUniqueId& qid, int64_t query_mem_limit,
26+
double max_scan_ratio)
27+
: query_id(qid),
28+
query_mem_limit(query_mem_limit),
29+
mem_limit(std::max<int64_t>(
30+
1, static_cast<int64_t>(static_cast<double>(query_mem_limit) * max_scan_ratio))) {
31+
}
32+
33+
void MemShareArbitrator::register_scan_node() {
34+
total_mem_bytes.fetch_add(DEFAULT_SCANNER_MEM_BYTES);
35+
}
36+
37+
int64_t MemShareArbitrator::update_mem_bytes(int64_t old_value, int64_t new_value) {
38+
int64_t diff = new_value - old_value;
39+
int64_t total = total_mem_bytes.fetch_add(diff) + diff;
40+
if (new_value == 0) return 0;
41+
if (total <= 0) return mem_limit;
42+
// Proportional sharing: allocate based on this context's share of total usage
43+
double ratio = static_cast<double>(new_value) / static_cast<double>(std::max(total, new_value));
44+
return static_cast<int64_t>(static_cast<double>(mem_limit) * ratio);
45+
}
46+
47+
// ==================== MemLimiter ====================
48+
int MemLimiter::available_scanner_count(int ins_idx) const {
49+
int64_t mem_limit_value = mem_limit.load();
50+
int64_t running_tasks_count_value = running_tasks_count.load();
51+
int64_t estimated_block_mem_bytes_value = get_estimated_block_mem_bytes();
52+
DCHECK_GT(estimated_block_mem_bytes_value, 0);
53+
54+
int64_t max_count = std::max(1L, mem_limit_value / estimated_block_mem_bytes_value);
55+
int64_t avail_count = max_count;
56+
int64_t per_count = avail_count / parallelism;
57+
if (serial_operator) {
58+
per_count += (avail_count - per_count * parallelism);
59+
} else if (ins_idx < avail_count - per_count * parallelism) {
60+
per_count += 1;
61+
}
62+
63+
VLOG_DEBUG << "available_scanner_count. max_count=" << max_count << "("
64+
<< running_tasks_count_value << "/" << estimated_block_mem_bytes_value
65+
<< "), operator_mem_limit = " << operator_mem_limit
66+
<< ", running_tasks_count = " << running_tasks_count_value
67+
<< ", parallelism = " << parallelism << ", avail_count = " << avail_count
68+
<< ", ins_id = " << ins_idx << ", per_count = " << per_count
69+
<< " debug_string: " << debug_string();
70+
71+
return cast_set<int>(per_count);
72+
}
73+
74+
void MemLimiter::reestimated_block_mem_bytes(int64_t value) {
75+
if (value == 0) return;
76+
value = std::min(value, operator_mem_limit);
77+
78+
std::lock_guard<std::mutex> L(lock);
79+
auto old_value = estimated_block_mem_bytes.load();
80+
int64_t total =
81+
get_estimated_block_mem_bytes() * estimated_block_mem_bytes_update_count + value;
82+
estimated_block_mem_bytes_update_count += 1;
83+
estimated_block_mem_bytes = total / estimated_block_mem_bytes_update_count;
84+
VLOG_DEBUG << fmt::format(
85+
"reestimated_block_mem_bytes. MemLimiter = {}, estimated_block_mem_bytes = {}, "
86+
"old_value = {}, value: {}",
87+
debug_string(), estimated_block_mem_bytes, old_value, value);
88+
}
89+
90+
} // namespace doris

‎be/src/exec/common/memory.h‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <fmt/format.h>
21+
#include <gen_cpp/Types_types.h>
22+
23+
#include <atomic>
24+
#include <string>
25+
26+
#include "common/cast_set.h"
27+
#include "common/factory_creator.h"
28+
#include "common/logging.h"
29+
#include "util/uid_util.h"
30+
31+
namespace doris {
32+
static constexpr int64_t DEFAULT_SCANNER_MEM_BYTES = 64 * 1024 * 1024; // 64MB default
33+
34+
// Query-level memory arbitrator that distributes memory fairly across all scan contexts
35+
struct MemShareArbitrator {
36+
ENABLE_FACTORY_CREATOR(MemShareArbitrator)
37+
TUniqueId query_id;
38+
int64_t query_mem_limit = 0;
39+
int64_t mem_limit = 0;
40+
std::atomic<int64_t> total_mem_bytes = 0;
41+
42+
MemShareArbitrator(const TUniqueId& qid, int64_t query_mem_limit, double max_scan_ratio);
43+
44+
// Update memory allocation when scanner memory usage changes
45+
// Returns new scan memory limit for this context
46+
int64_t update_mem_bytes(int64_t old_value, int64_t new_value);
47+
void register_scan_node();
48+
std::string debug_string() const {
49+
return fmt::format("query_id: {}, query_mem_limit: {}, mem_limit: {}", print_id(query_id),
50+
query_mem_limit, mem_limit);
51+
}
52+
};
53+
54+
// Scan-context-level memory limiter that controls scanner concurrency based on memory
55+
struct MemLimiter {
56+
private:
57+
TUniqueId query_id;
58+
mutable std::mutex lock;
59+
// Parallelism of the scan operator
60+
const int64_t parallelism = 0;
61+
const bool serial_operator = false;
62+
const int64_t operator_mem_limit;
63+
std::atomic<int64_t> running_tasks_count = 0;
64+
65+
std::atomic<int64_t> estimated_block_mem_bytes = 0;
66+
int64_t estimated_block_mem_bytes_update_count = 0;
67+
int64_t arb_mem_bytes = 0;
68+
std::atomic<int64_t> open_tasks_count = 0;
69+
70+
// Memory limit for this scan node (shared by all instances), updated by memory share arbitrator
71+
std::atomic<int64_t> mem_limit = 0;
72+
73+
public:
74+
ENABLE_FACTORY_CREATOR(MemLimiter)
75+
MemLimiter(const TUniqueId& qid, int64_t parallelism, bool serial_operator_, int64_t mem_limit)
76+
: query_id(qid),
77+
parallelism(parallelism),
78+
serial_operator(serial_operator_),
79+
operator_mem_limit(mem_limit) {}
80+
~MemLimiter() { DCHECK_LE(open_tasks_count, 0); }
81+
82+
// Calculate available scanner count based on memory limit
83+
int available_scanner_count(int ins_idx) const;
84+
85+
int64_t update_running_tasks_count(int delta) { return running_tasks_count += delta; }
86+
87+
// Re-estimated the average memory usage of a block, and update the estimated_block_mem_bytes accordingly.
88+
void reestimated_block_mem_bytes(int64_t value);
89+
void update_mem_limit(int64_t value) { mem_limit = value; }
90+
// Update the memory usage of this context to memory share arbitrator.
91+
// NOTE: This could be accessed by parallel tasks without synchronization, but it's fine
92+
// since the memory share arbitrator will do proportional sharing based on the ratio of this
93+
// context's memory usage to total memory usage, so even if there are some fluctuations in the
94+
// memory usage, the overall proportional sharing will still work.
95+
void update_arb_mem_bytes(int64_t value) {
96+
value = std::min(value, operator_mem_limit);
97+
arb_mem_bytes = value;
98+
}
99+
int64_t get_arb_scanner_mem_bytes() const { return arb_mem_bytes; }
100+
101+
int64_t get_estimated_block_mem_bytes() const { return estimated_block_mem_bytes; }
102+
103+
int64_t update_open_tasks_count(int delta) { return open_tasks_count.fetch_add(delta); }
104+
std::string debug_string() const {
105+
return fmt::format(
106+
"query_id: {}, parallelism: {}, serial_operator: {}, operator_mem_limit: {}, "
107+
"running_tasks_count: {}, estimated_block_mem_bytes: {}, "
108+
"estimated_block_mem_bytes_update_count: {}, arb_mem_bytes: {}, "
109+
"open_tasks_count: {}, mem_limit: {}",
110+
print_id(query_id), parallelism, serial_operator, operator_mem_limit,
111+
running_tasks_count.load(), estimated_block_mem_bytes.load(),
112+
estimated_block_mem_bytes_update_count, arb_mem_bytes, open_tasks_count, mem_limit);
113+
}
114+
};
115+
116+
} // namespace doris

‎be/src/exec/operator/scan_operator.cpp‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@ Status ScanLocalState<Derived>::init(RuntimeState* state, LocalStateInfo& info)
149149
set_scan_ranges(state, info.scan_ranges);
150150

151151
_wait_for_rf_timer = ADD_TIMER(common_profile(), "WaitForRuntimeFilter");
152+
_instance_idx = info.task_idx;
152153
return Status::OK();
153154
}
154155

@@ -1008,14 +1009,16 @@ template <typename Derived>
10081009
Status ScanLocalState<Derived>::_start_scanners(
10091010
const std::list<std::shared_ptr<ScannerDelegate>>& scanners) {
10101011
auto& p = _parent->cast<typename Derived::Parent>();
1011-
_scanner_ctx.store(ScannerContext::create_shared(state(), this, p._output_tuple_desc,
1012-
p.output_row_descriptor(), scanners, p.limit(),
1013-
_scan_dependency, &p._shared_scan_limit
1012+
_scanner_ctx.store(ScannerContext::create_shared(
1013+
state(), this, p._output_tuple_desc, p.output_row_descriptor(), scanners, p.limit(),
1014+
_scan_dependency, &p._shared_scan_limit, p._mem_arb, p._mem_limiter, _instance_idx,
1015+
_state->get_query_ctx()->get_query_options().__isset.enable_adaptive_scan &&
1016+
_state->get_query_ctx()->get_query_options().enable_adaptive_scan
10141017
#ifdef BE_TEST
1015-
,
1016-
max_scanners_concurrency(state())
1018+
,
1019+
max_scanners_concurrency(state())
10171020
#endif
1018-
));
1021+
));
10191022
return Status::OK();
10201023
}
10211024

@@ -1182,6 +1185,18 @@ Status ScanOperatorX<LocalStateType>::init(const TPlanNode& tnode, RuntimeState*
11821185
if (query_options.__isset.max_pushdown_conditions_per_column) {
11831186
_max_pushdown_conditions_per_column = query_options.max_pushdown_conditions_per_column;
11841187
}
1188+
#ifdef BE_TEST
1189+
_mem_arb = nullptr;
1190+
#else
1191+
_mem_arb = state->get_query_ctx()->mem_arb();
1192+
#endif
1193+
if (_mem_arb) {
1194+
_mem_arb->register_scan_node();
1195+
_mem_limiter =
1196+
MemLimiter::create_shared(state->query_id(), state->query_parallel_instance_num(),
1197+
OperatorX<LocalStateType>::is_serial_operator(),
1198+
state->get_query_ctx()->get_query_options().mem_limit);
1199+
}
11851200
// tnode.olap_scan_node.push_down_agg_type_opt field is deprecated
11861201
// Introduced a new field : tnode.push_down_agg_type_opt
11871202
//

‎be/src/exec/operator/scan_operator.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,7 @@ class ScanLocalState : public ScanLocalStateBase {
336336
// ScanLocalState owns the ownership of scanner, scanner context only has its weakptr
337337
std::list<std::shared_ptr<ScannerDelegate>> _scanners;
338338
Arena _arena;
339+
int _instance_idx = 0;
339340
};
340341

341342
template <typename LocalStateType>
@@ -437,6 +438,9 @@ class ScanOperatorX : public OperatorX<LocalStateType> {
437438
const int _parallel_tasks = 0;
438439

439440
std::vector<int> _topn_filter_source_node_ids;
441+
442+
std::shared_ptr<MemShareArbitrator> _mem_arb = nullptr;
443+
std::shared_ptr<MemLimiter> _mem_limiter = nullptr;
440444
};
441445

442446
#include "common/compile_check_end.h"

‎be/src/exec/pipeline/pipeline_fragment_context.cpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,11 +119,9 @@
119119
#include "load/stream_load/new_load_stream_mgr.h"
120120
#include "runtime/exec_env.h"
121121
#include "runtime/fragment_mgr.h"
122-
#include "runtime/result_block_buffer.h"
123122
#include "runtime/result_buffer_mgr.h"
124123
#include "runtime/runtime_state.h"
125124
#include "runtime/thread_context.h"
126-
#include "service/backend_options.h"
127125
#include "util/countdown_latch.h"
128126
#include "util/debug_util.h"
129127
#include "util/uid_util.h"

‎be/src/exec/scan/scanner.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -266,7 +266,7 @@ void Scanner::_collect_profile_before_close() {
266266
_state->update_num_rows_load_unselected(_counter.num_rows_unselected);
267267
}
268268

269-
void Scanner::update_scan_cpu_timer() {
269+
void Scanner::_update_scan_cpu_timer() {
270270
int64_t cpu_time = _cpu_watch.elapsed_time();
271271
_scan_cpu_timer += cpu_time;
272272
if (_state && _state->get_query_ctx()) {

‎be/src/exec/scan/scanner.h‎

Lines changed: 23 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -127,33 +127,41 @@ class Scanner {
127127

128128
Status _do_projections(Block* origin_block, Block* output_block);
129129

130-
public:
131-
int64_t get_time_cost_ns() const { return _per_scanner_timer; }
132-
133-
int64_t projection_time() const { return _projection_timer; }
134-
int64_t get_rows_read() const { return _num_rows_read; }
135-
136-
bool has_prepared() const { return _has_prepared; }
137-
138-
Status try_append_late_arrival_runtime_filter();
139-
130+
private:
140131
// Call start_wait_worker_timer() when submit the scanner to the thread pool.
141132
// And call update_wait_worker_timer() when it is actually being executed.
142-
void start_wait_worker_timer() {
133+
void _start_wait_worker_timer() {
143134
_watch.reset();
144135
_watch.start();
145136
}
146137

147-
void start_scan_cpu_timer() {
138+
void _start_scan_cpu_timer() {
148139
_cpu_watch.reset();
149140
_cpu_watch.start();
150141
}
151142

152-
void update_wait_worker_timer() { _scanner_wait_worker_timer += _watch.elapsed_time(); }
143+
void _update_wait_worker_timer() { _scanner_wait_worker_timer += _watch.elapsed_time(); }
144+
void _update_scan_cpu_timer();
153145

154-
int64_t get_scanner_wait_worker_timer() const { return _scanner_wait_worker_timer; }
146+
public:
147+
void resume() {
148+
_update_wait_worker_timer();
149+
_start_scan_cpu_timer();
150+
}
151+
void pause() {
152+
_update_scan_cpu_timer();
153+
_start_wait_worker_timer();
154+
}
155+
int64_t get_time_cost_ns() const { return _per_scanner_timer; }
156+
157+
int64_t projection_time() const { return _projection_timer; }
158+
int64_t get_rows_read() const { return _num_rows_read; }
155159

156-
void update_scan_cpu_timer();
160+
bool has_prepared() const { return _has_prepared; }
161+
162+
Status try_append_late_arrival_runtime_filter();
163+
164+
int64_t get_scanner_wait_worker_timer() const { return _scanner_wait_worker_timer; }
157165

158166
// Some counters need to be updated realtime, for example, workload group policy need
159167
// scan bytes to cancel the query exceed limit.

0 commit comments

Comments
 (0)