Skip to content

Commit e0bf487

Browse files
authored
[improvement](be) Use thread pool scan scheduler by default (apache#68610)
### What problem does this PR solve? Issue Number: None Related PR: apache#51690, apache#58144, apache#58273, apache#65814, apache#67070 Problem Summary: Workload groups pick the scan scheduler for internal and external table scans from `enable_task_executor_in_internal_table` and `enable_task_executor_in_external_table`. Both configs currently default to `true` on master, so every workload group creates `TaskExecutorSimplifiedScanScheduler` (the time-sharing task executor) for local (`ls_*`) and remote (`rs_*`) scans. The time-sharing executor is still maturing (for example the self-deadlock fixed in apache#58273; the internal-table default was already reverted once in apache#58144), so the default should be the `ThreadPoolSimplifiedScanScheduler` for both internal and external tables. This change sets both configs' default values to `false`. The task executor can still be enabled explicitly through `be.conf`. Making the thread pool path the default exposed four gaps. 1. Pending scanners after an exhausted shared LIMIT. For a `SELECT ... LIMIT n` over many tablets, the thread pool admission (`ScannerContext::can_admit_scan_task`, rewritten in apache#67070) keeps admitting the remaining pending scanners after peer scanners have exhausted the shared scan LIMIT. Admission is kept that way on purpose: holding pending scanners back based on the in-flight or completed task counters, as apache#65814 does on the TaskExecutor path, ties the schedule to bookkeeping that is easy to get wrong and can leave a context with nothing to wake the operator. Instead, `Scanner::get_block_after_projects()` now checks the shared LIMIT before it reads: once the counter is exhausted, the scanner returns an empty block, so an admitted scanner never reads its range for rows the operator will cut anyway. It does not report EOS for that: EOS means the underlying read is exhausted, while the LIMIT is applied by the scan operator, which stops the scanners after `reached_limit()`; `ScannerContext::get_block_from_queue()` also finishes the context once the counter is exhausted with nothing in flight. The two pre-existing checks in `Scanner::get_block()` that set `eof` from the shared counter are removed for the same reason, so the shared LIMIT is handled in one place. 2. Rows held in scanner padding under a shared LIMIT. `Scanner::get_block()` charges the shared LIMIT counter as soon as rows pass the filters, but with projections `get_block_after_projects()` may keep a small block in `_padding_block` and emit a later block first. Once peer scanners exhaust the counter, the context reaches EOS without running that scanner again, and the charged rows it holds are lost; for example, a parallel projected `LIMIT 4096` could return 4086 rows. The counter is now decremented only by the rows a call actually returns, so held padding is never charged and the counter equals "LIMIT minus returned rows" between calls. The padding batch target is capped by the remaining LIMIT when a call starts, so a scanner does not read on just to fill half a batch when fewer rows are needed, and the read loop stops as soon as peers exhaust the LIMIT, returning the rows it already holds. Projected scans with selective filters keep coalescing small blocks. The scan does not cap its output at the LIMIT (the operator cuts extra rows), so no further accounting is needed. 3. Exceptions in the row-ID fetch RPC callback. `multiget_data_v2` submits its callback to the remote scan scheduler; on a `ThreadPool` worker there is no exception boundary, and `RowIdStorageReader::read_by_rowids()` constructs `SlotDescriptor(pslot)` before its local `try`, so an unknown `PTypeNode.type` would throw `doris::Exception` and terminate the BE. The callback now calls `RowIdStorageReader::read_by_rowids_for_rpc()`, which converts a thrown exception into an error status on the response before the closure completes the RPC. `ExecEnv::set_id_manager()` is added under `BE_TEST` for the unit test. 4. The shared LIMIT pointer of a scanner initialized while the counter is negative. `ScanLocalState::shared_scan_limit_ptr()` returned null whenever the live counter was negative, and `Scanner::init()` caches that pointer once. Concurrent scanners can return more than the LIMIT before the operator makes its final cut, leaving the counter negative, so a scanner whose local state opened at that moment never checked or charged the shared LIMIT and could read a long selective tail after peers had satisfied the query. The pointer is now returned whenever the operator has a SQL LIMIT, which is how `ScannerContext` already decides it. ### Release note The BE configs `enable_task_executor_in_internal_table` and `enable_task_executor_in_external_table` now default to `false`; internal and external table scans use the thread pool scan scheduler by default. ### Check List (For Author) - Test: - Unit Test: added `WorkloadGroupManagerTest.DefaultScanSchedulersUseThreadPool`, `ScannerContextTest.thread_pool_admission_ignores_shared_limit`, `ScannerProjectionTest.shared_limit_exhausted_before_call_returns_empty_block_without_eos`, `ScannerProjectionTest.shared_limit_counts_only_emitted_rows`, `ScannerProjectionTest.shared_limit_drops_padding_after_peer_satisfies_limit`, `ScannerProjectionTest.shared_limit_flushes_padding_before_reading_more`, `ScannerProjectionTest.shared_limit_flushes_merged_padding_at_limit`, `ScannerProjectionTest.shared_limit_counts_returned_rows_without_projection`, `ScannerProjectionTest.shared_limit_counts_returned_rows_without_padding`, `ScannerProjectionTest.shared_limit_keeps_padding_small_blocks`, `ScannerProjectionTest.shared_limit_exhausted_by_own_rows_stops_reading_without_eos`, `ScannerProjectionTest.shared_limit_exhausted_by_peer_mid_read_returns_held_rows_without_eos`, `ScannerProjectionTest.shared_limit_observed_by_scanner_initialized_under_negative_counter`, `ScannerProjectionTest.no_shared_limit_without_sql_limit` and `RowIdStorageReaderTest.RpcHelperReportsThrownExceptionOnThreadPoolPath` (runs the RPC helper on a real `ThreadPoolSimplifiedScanScheduler`; the test process aborts without the fix); ran `ScannerProjectionTest.*`, `ScannerContextTest.*`, `ScannerLateArrivalRfTest.*`, `WorkloadGroupManagerTest.*`, `RowIdStorageReaderTest.*`, `SubmitExternalScanTasksTest.*` - Regression test: ran on a local cluster with the new default: query_p0/limit (19 suites) and correctness_p0/test_shared_scan_limit_pending_tasks - Behavior changed: Yes (default scan scheduler for internal and external table scans changes from the task executor to the thread pool; it can be switched back via the two BE configs) - Does this need documentation: No
1 parent 5a561bd commit e0bf487

14 files changed

Lines changed: 856 additions & 56 deletions

‎be/src/common/config.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -339,9 +339,9 @@ DEFINE_Int32(task_executor_max_concurrency_per_task, "-1");
339339
DEFINE_Int32(task_executor_initial_max_concurrency_per_task, "-1");
340340

341341
// Enable task executor in internal table scan.
342-
DEFINE_Bool(enable_task_executor_in_internal_table, "true");
342+
DEFINE_Bool(enable_task_executor_in_internal_table, "false");
343343
// Enable task executor in external table scan.
344-
DEFINE_Bool(enable_task_executor_in_external_table, "true");
344+
DEFINE_Bool(enable_task_executor_in_external_table, "false");
345345

346346
// number of scanner thread pool size for olap table
347347
// and the min thread num of remote scanner thread pool

‎be/src/exec/operator/mock_scan_operator.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,12 @@ class MockScanOperatorX final : public ScanOperatorX<MockScanLocalState> {
8484
_output_blocks.push_back(std::move(block));
8585
}
8686

87+
// Mirrors the constructor of ScanOperatorX: the shared counter starts at the SQL LIMIT.
88+
void set_limit_for_test(int64_t limit) {
89+
_limit = limit;
90+
_shared_scan_limit.store(limit, std::memory_order_relaxed);
91+
}
92+
8793
Status get_block_impl(RuntimeState* state, Block* block, bool* eos) override {
8894
if (_output_blocks.empty()) {
8995
*eos = true;

‎be/src/exec/operator/scan_operator.cpp‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1072,10 +1072,12 @@ int64_t ScanLocalState<Derived>::limit_per_scanner() {
10721072

10731073
template <typename Derived>
10741074
std::atomic<int64_t>* ScanLocalState<Derived>::shared_scan_limit_ptr() {
1075-
auto* p = &_parent->cast<typename Derived::Parent>()._shared_scan_limit;
1076-
// -1 means "no SQL LIMIT" — return nullptr so callers naturally skip
1077-
// all limit logic.
1078-
return p->load(std::memory_order_relaxed) < 0 ? nullptr : p;
1075+
auto& parent = _parent->cast<typename Derived::Parent>();
1076+
// Decide from the SQL LIMIT, not from the counter. -1 means "no SQL LIMIT" — return nullptr
1077+
// so callers naturally skip all limit logic. Concurrent scanners can return more than the
1078+
// LIMIT before the operator makes its final cut, leaving this counter negative. A scanner
1079+
// initialized then must still observe that the shared LIMIT is exhausted.
1080+
return parent.limit() < 0 ? nullptr : &parent._shared_scan_limit;
10791081
}
10801082

10811083
template <typename Derived>

‎be/src/exec/rowid_fetcher.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,13 @@ static void scatter_scan_blocks_to_result_block(
196196
}
197197
}
198198

199+
void RowIdStorageReader::read_by_rowids_for_rpc(const PMultiGetRequestV2& request,
200+
PMultiGetResponseV2* response) {
201+
Status st = Status::OK();
202+
ASSIGN_STATUS_IF_CATCH_EXCEPTION(st = read_by_rowids(request, response), st);
203+
st.to_protobuf(response->mutable_status());
204+
}
205+
199206
Status RowIdStorageReader::read_by_rowids(const PMultiGetRequestV2& request,
200207
PMultiGetResponseV2* response) {
201208
if (request.request_block_descs_size()) {

‎be/src/exec/rowid_fetcher.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,14 @@ class RowIdStorageReader {
8282

8383
static Status read_by_rowids(const PMultiGetRequestV2& request, PMultiGetResponseV2* response);
8484

85+
// Runs read_by_rowids() for the multiget_data_v2 RPC and serializes the outcome into the
86+
// response status. The remote scan scheduler runs the RPC callback on a plain ThreadPool
87+
// worker, which has no exception boundary of its own, so a doris::Exception thrown while
88+
// reading (for example by a slot of the request whose PTypeNode this BE does not know) is
89+
// converted into an error status here instead of terminating the process.
90+
static void read_by_rowids_for_rpc(const PMultiGetRequestV2& request,
91+
PMultiGetResponseV2* response);
92+
8593
private:
8694
static bool should_use_file_scanner_v2(const TQueryOptions& query_options,
8795
const TFileScanRangeParams& scan_params,

‎be/src/exec/scan/scanner.cpp‎

Lines changed: 60 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919

2020
#include <glog/logging.h>
2121

22+
#include <algorithm>
23+
2224
#include "common/config.h"
2325
#include "common/status.h"
2426
#include "core/block/column_with_type_and_name.h"
@@ -87,8 +89,18 @@ Status Scanner::init(RuntimeState* state, const VExprContextSPtrs& conjuncts) {
8789
return Status::OK();
8890
}
8991

92+
// Keep the single-use padding loop inline with its projection and LIMIT accounting.
93+
// NOLINTNEXTLINE(readability-function-cognitive-complexity)
9094
Status Scanner::get_block_after_projects(RuntimeState* state, Block* block, bool* eos) {
9195
SCOPED_CONCURRENCY_COUNT(ConcurrencyStatsManager::instance().vscanner_get_block);
96+
// Peer scanners may have exhausted the shared LIMIT while this scanner waited to be
97+
// scheduled. Admission does not look at the LIMIT, so the scanner returns an empty block
98+
// instead of reading on. It does not report EOS: the operator already holds enough rows and
99+
// stops the scanners after cutting the LIMIT itself, and ScannerContext parks the scanner
100+
// rather than resubmitting it while the outstanding work drains.
101+
if (_shared_scan_limit_exhausted()) {
102+
return Status::OK();
103+
}
92104
const auto& row_descriptor = _local_state->_parent->operator_row_desc_before_projection();
93105
if (_has_projection) {
94106
_origin_block.clear_column_data(row_descriptor.num_slots());
@@ -99,58 +111,68 @@ Status Scanner::get_block_after_projects(RuntimeState* state, Block* block, bool
99111
// before the next split is read. Padding those blocks first would make correctness
100112
// depend on whether two file tails happen to share one output batch.
101113
RETURN_IF_ERROR(get_block(state, &_origin_block, eos));
102-
return _do_projections(&_origin_block, block);
103-
}
104-
const auto min_batch_size = std::max(state->batch_size() / 2, 1);
105-
const auto block_max_bytes = state->preferred_block_size_bytes();
106-
while (_padding_block.rows() < min_batch_size && _padding_block.bytes() < block_max_bytes &&
107-
!*eos) {
108-
RETURN_IF_ERROR(get_block(state, &_origin_block, eos));
109-
if (*eos) {
110-
// For the final block, merge any padding directly and return eos in this call.
111-
// The merged tail can be larger than the target batch, but each source block is
112-
// already bounded by the lower scanner.
113-
RETURN_IF_ERROR(_merge_padding_block());
114-
_origin_block.clear_column_data(row_descriptor.num_slots());
115-
break;
116-
}
117-
if (_origin_block.rows() >= min_batch_size) {
118-
break;
114+
} else {
115+
auto min_batch_size = std::max(state->batch_size() / 2, 1);
116+
if (_shared_scan_limit) {
117+
// Pending padding is not charged until it is returned. Bound this call's batch
118+
// target by the shared remaining rows to avoid reading on just to fill a batch.
119+
min_batch_size = cast_set<int>(std::clamp<int64_t>(
120+
_shared_scan_limit->load(std::memory_order_acquire), 0, min_batch_size));
119121
}
122+
const auto block_max_bytes = state->preferred_block_size_bytes();
123+
// Stop reading as soon as peers exhaust the shared LIMIT; the rows already held are
124+
// still returned below.
125+
while (_padding_block.rows() < min_batch_size &&
126+
_padding_block.bytes() < block_max_bytes && !*eos &&
127+
!_shared_scan_limit_exhausted()) {
128+
RETURN_IF_ERROR(get_block(state, &_origin_block, eos));
129+
if (*eos) {
130+
// For the final block, merge any padding directly and return eos in this call.
131+
// The merged tail can be larger than the target batch, but each source block
132+
// is already bounded by the lower scanner.
133+
RETURN_IF_ERROR(_merge_padding_block());
134+
_origin_block.clear_column_data(row_descriptor.num_slots());
135+
break;
136+
}
137+
if (_origin_block.rows() >= min_batch_size) {
138+
break;
139+
}
120140

121-
if (_origin_block.rows() + _padding_block.rows() <= state->batch_size() &&
122-
_origin_block.bytes() + _padding_block.bytes() <= block_max_bytes) {
123-
RETURN_IF_ERROR(_merge_padding_block());
124-
_origin_block.clear_column_data(row_descriptor.num_slots());
125-
} else {
126-
if (_origin_block.rows() < _padding_block.rows()) {
127-
_padding_block.swap(_origin_block);
141+
if (_origin_block.rows() + _padding_block.rows() <= state->batch_size() &&
142+
_origin_block.bytes() + _padding_block.bytes() <= block_max_bytes) {
143+
RETURN_IF_ERROR(_merge_padding_block());
144+
_origin_block.clear_column_data(row_descriptor.num_slots());
145+
} else {
146+
if (_origin_block.rows() < _padding_block.rows()) {
147+
_padding_block.swap(_origin_block);
148+
}
149+
break;
128150
}
129-
break;
130151
}
131-
}
132152

133-
if (_origin_block.empty() && !_padding_block.empty()) {
134-
_padding_block.swap(_origin_block);
153+
if (_origin_block.empty() && !_padding_block.empty()) {
154+
_padding_block.swap(_origin_block);
155+
}
135156
}
136-
return _do_projections(&_origin_block, block);
157+
RETURN_IF_ERROR(_do_projections(&_origin_block, block));
137158
} else {
138-
return get_block(state, block, eos);
159+
RETURN_IF_ERROR(get_block(state, block, eos));
160+
}
161+
162+
// Only rows actually produced by this call consume the shared LIMIT. In particular, padding
163+
// kept for a later call must not make the context finish before that call can return it.
164+
// Exhausting the counter is not EOS either: the next call returns an empty block, and the
165+
// scan operator stops this scanner once it has cut the LIMIT.
166+
if (_shared_scan_limit != nullptr && block->rows() > 0) {
167+
_shared_scan_limit->fetch_sub(cast_set<int64_t>(block->rows()), std::memory_order_acq_rel);
139168
}
169+
return Status::OK();
140170
}
141171

142172
Status Scanner::get_block(RuntimeState* state, Block* block, bool* eof) {
143173
// only empty block should be here
144174
DCHECK(block->rows() == 0);
145175

146-
// Stop early if other scanners have already collected enough rows
147-
// for the SQL LIMIT. Skipped when _shared_scan_limit is null (topn
148-
// path or no LIMIT).
149-
if (_shared_scan_limit && _shared_scan_limit->load(std::memory_order_acquire) <= 0) {
150-
*eof = true;
151-
return Status::OK();
152-
}
153-
154176
// scanner running time
155177
SCOPED_RAW_TIMER(&_per_scanner_timer);
156178
int64_t rows_read_threshold = _num_rows_read + config::doris_scanner_row_num;
@@ -183,13 +205,6 @@ Status Scanner::get_block(RuntimeState* state, Block* block, bool* eof) {
183205
}
184206
// record rows return (after filter) for _limit check
185207
_num_rows_return += block->rows();
186-
// Publish progress to the shared counter so peer scanners can
187-
// observe it. The counter may go negative when several scanners
188-
// subtract concurrently; that is harmless because the operator's
189-
// reached_limit() makes the final cut.
190-
if (_shared_scan_limit && block->rows() > 0) {
191-
_shared_scan_limit->fetch_sub(block->rows(), std::memory_order_acq_rel);
192-
}
193208
} while (!_should_stop && !state->is_cancelled() && block->rows() == 0 && !(*eof) &&
194209
_num_rows_read < rows_read_threshold);
195210
}
@@ -202,7 +217,6 @@ Status Scanner::get_block(RuntimeState* state, Block* block, bool* eof) {
202217
// set eof to true if per scanner limit is reached
203218
// currently for query: ORDER BY key LIMIT n
204219
*eof = *eof || (_limit > 0 && _num_rows_return >= _limit);
205-
*eof = *eof || (_shared_scan_limit && _shared_scan_limit->load(std::memory_order_acquire) <= 0);
206220

207221
return Status::OK();
208222
}

‎be/src/exec/scan/scanner.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,15 @@ class Scanner {
140140
return Status::OK();
141141
}
142142

143+
// True once scanners of this operator have produced the whole shared LIMIT. The scanner then
144+
// stops reading but does not report EOS for it: EOS means the underlying read is exhausted,
145+
// while the LIMIT is applied by the scan operator, which stops the scanners once it holds
146+
// enough rows.
147+
bool _shared_scan_limit_exhausted() const {
148+
return _shared_scan_limit != nullptr &&
149+
_shared_scan_limit->load(std::memory_order_acquire) <= 0;
150+
}
151+
143152
// Update the counters before closing this scanner
144153
virtual void _collect_profile_before_close();
145154

‎be/src/exec/scan/scanner_context.cpp‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -418,6 +418,12 @@ Status ScannerContext::get_block_from_queue(RuntimeState* state, Block* block, b
418418
(_is_shared_scan_limit_exhausted() && _in_flight_tasks_num == 0))) {
419419
_set_scanner_done();
420420
_is_finished = true;
421+
} else if (scan_task != nullptr && !scan_task->is_eos() && _is_shared_scan_limit_exhausted()) {
422+
// Scanners of this operator have already produced the whole shared LIMIT, so this scanner
423+
// would only return empty blocks until the outstanding work drains. Park it instead of
424+
// resubmitting it. The finish check above did not fire, so a completed or in-flight task
425+
// remains to wake the operator, and the context finishes once nothing is in flight.
426+
push_pending_scan_task(std::move(scan_task), l);
421427
} else if (scan_task != nullptr) {
422428
// Check the terminal state before scheduling more work. In particular, after the shared
423429
// LIMIT is exhausted, submitting another Context runnable can turn a completed query into
@@ -867,7 +873,8 @@ std::shared_ptr<ScanTask> ScannerContext::_pull_next_scan_task(
867873
if (!_pending_tasks.empty()) {
868874
// Do not submit more pending scanners after the shared LIMIT is exhausted while
869875
// completed or in-flight tasks can still make progress. If neither exists, allow pending
870-
// scanners to be submitted so they can report EOS and wake the pipeline task.
876+
// scanners to be submitted so they can complete without reading and wake the pipeline
877+
// task, which finishes the context once nothing is in flight.
871878
if (_is_shared_scan_limit_exhausted() &&
872879
(_in_flight_tasks_num != 0 || !_completed_tasks.empty())) {
873880
return nullptr;

‎be/src/runtime/exec_env.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,7 @@ class ExecEnv {
348348
_inverted_index_query_cache = inverted_index_query_cache;
349349
}
350350
void set_process_profile(ProcessProfile* pp) { this->_process_profile = pp; }
351+
void set_id_manager(IdManager* id_manager) { this->_id_manager = id_manager; }
351352
void set_delete_bitmap_agg_cache(DeleteBitmapAggCache* c) { _delete_bitmap_agg_cache = c; }
352353
void set_storage_page_cache(StoragePageCache* c) { this->_storage_page_cache = c; }
353354
void set_ann_index_ivf_list_cache(AnnIndexIVFListCache* c) {

‎be/src/service/internal_service.cpp‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2063,9 +2063,7 @@ void PInternalService::multiget_data_v2(google::protobuf::RpcController* control
20632063
MonotonicStopWatch watch;
20642064
watch.start();
20652065
brpc::ClosureGuard closure_guard(done);
2066-
response->mutable_status()->set_status_code(0);
2067-
Status st = RowIdStorageReader::read_by_rowids(*request, response);
2068-
st.to_protobuf(response->mutable_status());
2066+
RowIdStorageReader::read_by_rowids_for_rpc(*request, response);
20692067
LOG(INFO) << "multiget_data finished, cost(us):"
20702068
<< watch.elapsed_time() / 1000;
20712069
return true;

0 commit comments

Comments
 (0)