Skip to content

Commit 1b3cae9

Browse files
committed
[improvement](be) Refactor thread-pool scan scheduling (branch-4.2)
### What problem does this PR solve? Issue Number: None Related PR: #67070 Problem Summary: This is a rewrite of #67070 for branch-4.2. It is not a cherry-pick: #67070 builds on scheduler refactors that branch-4.2 does not have, and those are intentionally not picked here: - #61617 (ScanTask state machine, `_pending_tasks` / `_completed_tasks`) was reverted on this release line by #62191 and stays reverted. - #61271 (adaptive scanners and the scan memory limiter), #62222 (limit push down to the segment iterator) and #65814 (shared scan limit) are large behavior changes of their own. The admission logic is therefore written against the branch-4.2 model (`_pending_scanners`, `_tasks_queue`, `_num_scheduled_scanners`, `cached_blocks`). The parts of #67070 that depend on the shared scan limit or on adaptive scanners / the memory limiter have no counterpart here and are dropped. The problem is the same on branch-4.2: the ThreadPool scan scheduler submits one runnable per scanner and serializes scheduling of every Context through a scheduler-wide lock. With many scanners this inflates queue occupancy and couples ThreadPool admission to the TaskExecutor scheduling logic. With this change the ThreadPool scheduler: - queues at most one runnable per `ScannerContext`; the runnable admits one pending scanner under the Context transfer lock, submits its successor and then scans without the lock; - keeps submission failures local to the Context (`TOO_MANY_TASKS` or the shutdown error make only that Context terminal); - checks the terminal Context state in `get_block_from_queue()` before rescheduling a drained scanner; - fails the Context at its next scheduling attempt when the pool has been stopped while its runnable was still queued, instead of treating the runnable that shutdown dropped as still pending. Admission keeps the limits branch-4.2 applies through `_get_margin()` and `_pull_next_scan_task()`: a Context never exceeds `_max_scan_concurrency` (cached results count as occupied slots), low memory mode caps running scanners, and once the pool has no slack (`active + queued >= min_active_scan_threads`) a Context is held at `_min_scan_concurrency`, or at `_max_scan_concurrency` while the operator is starving. One scanner is always admitted when nothing is progressing so the operator can be woken. A worker admitting the task it runs itself does not count its own thread against the pool budget, matching `_get_margin()`, which runs on the operator thread. The Context runnable waits in the pool queue on behalf of the scanner it admits, so that wait is credited to the scanner's wait-worker time (`Scanner::add_wait_worker_time()`), keeping `ScannerWorkerWaitTime` / `PerScannerWaitTime` meaningful on the ThreadPool path. TaskExecutor remains the default scan scheduler and keeps its behavior. Generic ThreadPool behavior is unchanged. `debug_string()` now reports `_is_context_queued` and `_num_finished_scanners`, and the ThreadPool path logs admission, refusal and runnable submission at `VLOG_DEBUG`. ### Release note None ### Check List (For Author) - Test: - Unit Test: `./run-be-ut.sh --run --filter="ScannerContextTest.*:ThreadPoolTest.*:*ScanScheduler*:*Scanner*"` (132 tests passed); the new ThreadPool tests were also repeated 50 times with `GTEST_REPEAT=50` without failure. The new tests cover admission state and limits (including the admitting worker and the starving target), the admit -> execute -> publish -> consume -> re-admit chain on a real `ThreadPoolSimplifiedScanScheduler`, runnable de-duplication, submission failure, a stopped scheduler with and without a queued runnable, admission and successor-submission failure publishing, wait-time crediting and terminal EOS. - Behavior changed: Yes. The opt-in ThreadPool scan scheduler queues and admits work per `ScannerContext`. - Does this need documentation: No
1 parent 6ee1e25 commit 1b3cae9

7 files changed

Lines changed: 1085 additions & 23 deletions

File tree

‎be/src/exec/scan/scanner.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,10 @@ class Scanner {
169169

170170
void update_wait_worker_timer() { _scanner_wait_worker_timer += _watch.elapsed_time(); }
171171

172+
// Credit wait time spent before start_wait_worker_timer() could be called, e.g. while the
173+
// ThreadPool Context runnable that admits this scanner was queued for a worker.
174+
void add_wait_worker_time(int64_t wait_ns) { _scanner_wait_worker_timer += wait_ns; }
175+
172176
int64_t get_scanner_wait_worker_timer() const { return _scanner_wait_worker_timer; }
173177

174178
void update_scan_cpu_timer();

‎be/src/exec/scan/scanner_context.cpp‎

Lines changed: 113 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,8 @@ Status ScannerContext::get_block_from_queue(RuntimeState* state, Block* block, b
278278
}
279279

280280
std::shared_ptr<ScanTask> scan_task = nullptr;
281+
// Whether all cached blocks of scan_task were consumed, so the scanner needs scheduling again.
282+
bool scan_task_drained = false;
281283

282284
if (!_tasks_queue.empty() && !done()) {
283285
// https://en.cppreference.com/w/cpp/container/list/front
@@ -323,19 +325,21 @@ Status ScannerContext::get_block_from_queue(RuntimeState* state, Block* block, b
323325
_scan_starving = _tasks_queue.empty() &&
324326
_num_finished_scanners < cast_set<int32_t>(_all_scanners.size()) &&
325327
(_num_scheduled_scanners > 0 || !_pending_scanners.empty());
326-
RETURN_IF_ERROR(
327-
_scanner_scheduler->schedule_scan_task(shared_from_this(), nullptr, l));
328328
} else {
329329
_scan_starving = _tasks_queue.empty();
330-
RETURN_IF_ERROR(
331-
_scanner_scheduler->schedule_scan_task(shared_from_this(), scan_task, l));
332330
}
331+
scan_task_drained = true;
333332
}
334333
}
335334

336335
if (_num_finished_scanners == _all_scanners.size() && _tasks_queue.empty()) {
337336
_set_scanner_done();
338337
_is_finished = true;
338+
} else if (scan_task_drained) {
339+
// Check the terminal state before scheduling more work, so a completed Context never
340+
// submits another runnable that could only fail on a full scanner pool.
341+
RETURN_IF_ERROR(_scanner_scheduler->schedule_scan_task(
342+
shared_from_this(), scan_task->is_eos() ? nullptr : scan_task, l));
339343
}
340344

341345
*eos = done();
@@ -448,17 +452,119 @@ std::string ScannerContext::debug_string() {
448452
return fmt::format(
449453
"id: {}, total scanners: {}, pending tasks: {},"
450454
" _should_stop: {}, _is_finished: {}, free blocks: {},"
451-
" limit: {}, _num_running_scanners: {}, _max_thread_num: {},"
455+
" limit: {}, _num_running_scanners: {}, _is_context_queued: {},"
456+
" _num_finished_scanners: {}, _max_thread_num: {},"
452457
" _max_bytes_in_queue: {}, query_id: {}",
453458
ctx_id, _all_scanners.size(), _tasks_queue.size(), _should_stop, _is_finished,
454-
_free_blocks.size_approx(), limit, _num_scheduled_scanners, _max_scan_concurrency,
455-
_max_bytes_in_queue, print_id(_query_id));
459+
_free_blocks.size_approx(), limit, _num_scheduled_scanners, _is_context_queued,
460+
_num_finished_scanners, _max_scan_concurrency, _max_bytes_in_queue,
461+
print_id(_query_id));
456462
}
457463

458464
void ScannerContext::_set_scanner_done() {
459465
_dependency->set_always_ready();
460466
}
461467

468+
bool ScannerContext::is_context_queued(const std::unique_lock<std::mutex>& transfer_lock) const {
469+
DORIS_CHECK(transfer_lock.owns_lock());
470+
return _is_context_queued;
471+
}
472+
473+
void ScannerContext::set_context_queued(bool queued,
474+
const std::unique_lock<std::mutex>& transfer_lock) {
475+
DORIS_CHECK(transfer_lock.owns_lock());
476+
DORIS_CHECK(_is_context_queued != queued);
477+
_is_context_queued = queued;
478+
}
479+
480+
void ScannerContext::set_context_failure(const Status& failure,
481+
const std::unique_lock<std::mutex>& transfer_lock) {
482+
DORIS_CHECK(transfer_lock.owns_lock());
483+
DORIS_CHECK(!failure.ok());
484+
_process_status = failure;
485+
_is_finished = true;
486+
_set_scanner_done();
487+
}
488+
489+
void ScannerContext::push_pending_scan_task(std::shared_ptr<ScanTask> scan_task,
490+
const std::unique_lock<std::mutex>& transfer_lock) {
491+
DORIS_CHECK(transfer_lock.owns_lock());
492+
DORIS_CHECK(scan_task != nullptr);
493+
DORIS_CHECK(scan_task->cached_blocks.empty());
494+
DORIS_CHECK(!scan_task->is_eos());
495+
_pending_scanners.push(std::move(scan_task));
496+
}
497+
498+
bool ScannerContext::can_admit_scan_task(const std::unique_lock<std::mutex>& transfer_lock,
499+
bool admitting_on_worker) const {
500+
DORIS_CHECK(transfer_lock.owns_lock());
501+
if (done() || _pending_scanners.empty()) {
502+
return false;
503+
}
504+
505+
// Blocks waiting in _tasks_queue still occupy a concurrency slot until the operator consumes
506+
// them. Counting both prevents a fast producer from exceeding the per-Context scanner limit.
507+
const int32_t current_concurrency =
508+
cast_set<int32_t>(_tasks_queue.size()) + _num_scheduled_scanners;
509+
// Keep one task progressing whatever the limits are. Otherwise no worker can publish a result
510+
// and wake the operator to make another scheduling decision.
511+
if (current_concurrency == 0) {
512+
return true;
513+
}
514+
// The per-Context ceiling applied by _pull_next_scan_task().
515+
if (current_concurrency >= _max_scan_concurrency) {
516+
return false;
517+
}
518+
// In low memory mode _get_margin() limits the number of running scanners.
519+
if (low_memory_mode() && _num_scheduled_scanners >= low_memory_mode_scanners()) {
520+
return false;
521+
}
522+
// Mirror the scheduler-wide budget of _get_margin(): while the pool has slack a Context may
523+
// ramp to its maximum. Once it has none, a Context is held at its target concurrency, which is
524+
// its minimum unless the operator is starving. Both counters are read here under
525+
// _transfer_lock exactly as the TaskExecutor path reads them. A worker admitting the task it
526+
// runs itself is already counted as active; like the task _get_margin() is about to submit, it
527+
// must not count against the budget, otherwise the pool would stop one slot short of it.
528+
const int32_t busy_scan_slots = _scanner_scheduler->get_active_threads() +
529+
_scanner_scheduler->get_queue_size() -
530+
(admitting_on_worker ? 1 : 0);
531+
if (busy_scan_slots < _min_scan_concurrency_of_scan_scheduler) {
532+
return true;
533+
}
534+
const int32_t target_scan_concurrency =
535+
_scan_starving && _tasks_queue.empty() ? _max_scan_concurrency : _min_scan_concurrency;
536+
return current_concurrency < target_scan_concurrency;
537+
}
538+
539+
std::shared_ptr<ScanTask> ScannerContext::try_get_next_scan_task(
540+
const std::unique_lock<std::mutex>& transfer_lock, int64_t context_queue_wait_ns) {
541+
if (!can_admit_scan_task(transfer_lock, true)) {
542+
VLOG_DEBUG << fmt::format(
543+
"[{}|{}] refuse admission, pending: {}, task queue: {}, scheduled: {}, done: {}",
544+
print_id(_query_id), ctx_id, _pending_scanners.size(), _tasks_queue.size(),
545+
_num_scheduled_scanners, done());
546+
return nullptr;
547+
}
548+
549+
// Pop and count as scheduled while holding the same lock used by completion and consumption.
550+
// Thus concurrent Context workers cannot admit the same task or both pass the limit check.
551+
auto scan_task = _pending_scanners.top();
552+
_pending_scanners.pop();
553+
// ThreadPool admission bypasses ScannerScheduler::submit(); restart the per-scanner wait
554+
// timer here so it measures admission-to-execution instead of everything since the previous
555+
// attempt paused, which would include time the cached blocks waited for the operator. The
556+
// Context runnable waited for a worker on behalf of this scanner, so credit that wait too.
557+
if (auto scanner_delegate = scan_task->scanner.lock()) {
558+
scanner_delegate->_scanner->start_wait_worker_timer();
559+
scanner_delegate->_scanner->add_wait_worker_time(context_queue_wait_ns);
560+
}
561+
++_num_scheduled_scanners;
562+
VLOG_DEBUG << fmt::format("[{}|{}] admit scanner, pending: {}, task queue: {}, scheduled: {}",
563+
print_id(_query_id), ctx_id, _pending_scanners.size(),
564+
_tasks_queue.size(), _num_scheduled_scanners);
565+
return scan_task;
566+
}
567+
462568
void ScannerContext::update_peak_running_scanner(int num) {
463569
_local_state->_peak_running_scanner->add(num);
464570
}

‎be/src/exec/scan/scanner_context.h‎

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,43 @@ class ScannerContext : public std::enable_shared_from_this<ScannerContext>,
193193
std::unique_lock<std::mutex>& transfer_lock,
194194
std::unique_lock<std::shared_mutex>& scheduler_lock);
195195

196+
// Context scheduling and operator consumption share this lock so queue-state changes and task
197+
// admission form one atomic decision. For example, two worker callbacks cannot both admit the
198+
// last available concurrency slot.
199+
std::mutex& transfer_lock() { return _transfer_lock; }
200+
201+
// One Context submission represents many pending scanners in the ThreadPool scheduler.
202+
// Keeping this separate from scanner execution prevents duplicate runnables from accumulating.
203+
bool is_context_queued(const std::unique_lock<std::mutex>& transfer_lock) const;
204+
// Transition the Context runnable's queue state. The caller must hold _transfer_lock.
205+
void set_context_queued(bool queued, const std::unique_lock<std::mutex>& transfer_lock);
206+
207+
// Publish a scheduler failure and make the Context terminal. The caller must hold
208+
// _transfer_lock so a retained ThreadPool callback cannot admit another scanner concurrently.
209+
void set_context_failure(const Status& failure,
210+
const std::unique_lock<std::mutex>& transfer_lock);
211+
212+
// Return a scanner to the admission queue after its blocks are consumed. It may not own cached
213+
// blocks and may not be EOS: EOS scanners are terminal and must not run again.
214+
void push_pending_scan_task(std::shared_ptr<ScanTask> scan_task,
215+
const std::unique_lock<std::mutex>& transfer_lock);
216+
217+
// Return whether a Context worker can currently admit one pending scanner. This check has no
218+
// side effects, so the scheduler can avoid submitting a runnable that would immediately exit.
219+
// It always admits one scanner when nothing is progressing so the operator can be woken, and
220+
// otherwise applies the same limits as _get_margin() and _pull_next_scan_task() on the
221+
// TaskExecutor path. `admitting_on_worker` is true when the caller is the pool worker that will
222+
// run the admitted task itself. The caller must hold _transfer_lock.
223+
bool can_admit_scan_task(const std::unique_lock<std::mutex>& transfer_lock,
224+
bool admitting_on_worker) const;
225+
226+
// Atomically check whether this context can start another scan task, move one task from
227+
// pending to scheduled, and return it. It is called by the pool worker that will run the task;
228+
// `context_queue_wait_ns` is how long that worker's Context runnable waited in the pool queue.
229+
// The caller must hold _transfer_lock.
230+
std::shared_ptr<ScanTask> try_get_next_scan_task(
231+
const std::unique_lock<std::mutex>& transfer_lock, int64_t context_queue_wait_ns);
232+
196233
protected:
197234
/// Four criteria to determine whether to increase the parallelism of the scanners
198235
/// 1. It ran for at least `SCALE_UP_DURATION` ms after last scale up
@@ -225,7 +262,14 @@ class ScannerContext : public std::enable_shared_from_this<ScannerContext>,
225262
int64_t _max_bytes_in_queue = 0;
226263
// Using stack so that we can resubmit scanner in a LIFO order, maybe more cache friendly
227264
std::stack<std::shared_ptr<ScanTask>> _pending_scanners;
228-
// Scanner that is submitted to the scheduler.
265+
// True from the start of one Context submission until its runnable starts. The marker may
266+
// remain true when no runnable was retained: a failed submission makes the Context terminal,
267+
// and a submission that threw inside _run_context() publishes the error through the task that
268+
// was already admitted. In both cases the operator observes _process_status, so no further
269+
// submission is attempted. It does not describe scanners executing on workers. Only used by
270+
// the ThreadPool scheduler. Protected by _transfer_lock.
271+
bool _is_context_queued = false;
272+
// Scanner that is submitted to the scheduler, or directly admitted for thread-pool execution.
229273
std::atomic_int _num_scheduled_scanners = 0;
230274
// Scanner that is eos or error.
231275
int32_t _num_finished_scanners = 0;

‎be/src/exec/scan/scanner_scheduler.cpp‎

Lines changed: 17 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -71,17 +71,7 @@ Status ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx,
7171
TabletStorageType type = scanner_delegate->_scanner->get_storage_type();
7272
auto sumbit_task = [&]() {
7373
auto work_func = [scanner_ref = scan_task, ctx]() {
74-
auto status = [&] {
75-
RETURN_IF_CATCH_EXCEPTION(_scanner_scan(ctx, scanner_ref));
76-
return Status::OK();
77-
}();
78-
79-
if (!status.ok()) {
80-
scanner_ref->set_status(status);
81-
ctx->push_back_scan_task(scanner_ref);
82-
return true;
83-
}
84-
return scanner_ref->is_eos();
74+
return execute_scan_task(ctx, scanner_ref);
8575
};
8676
SimplifiedScanTask simple_scan_task = {work_func, ctx, scan_task};
8777
return this->submit_scan_task(simple_scan_task);
@@ -100,6 +90,22 @@ Status ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx,
10090
return Status::OK();
10191
}
10292

93+
bool ScannerScheduler::execute_scan_task(const std::shared_ptr<ScannerContext>& ctx,
94+
const std::shared_ptr<ScanTask>& scan_task) {
95+
// Both schedulers admit tasks differently, but exceptions must always become a completed task
96+
// so the operator observes the error and releases the task's scheduled concurrency slot.
97+
auto status = [&] {
98+
RETURN_IF_CATCH_EXCEPTION(_scanner_scan(ctx, scan_task));
99+
return Status::OK();
100+
}();
101+
if (!status.ok()) {
102+
scan_task->set_status(status);
103+
ctx->push_back_scan_task(scan_task);
104+
return true;
105+
}
106+
return scan_task->is_eos();
107+
}
108+
103109
void handle_reserve_memory_failure(RuntimeState* state, std::shared_ptr<ScannerContext> ctx,
104110
const Status& st, size_t reserve_size) {
105111
ctx->clear_free_blocks();

‎be/src/exec/scan/scanner_scheduler.h‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -138,7 +138,11 @@ class ScannerScheduler {
138138
protected:
139139
int _min_active_scan_threads;
140140

141-
private:
141+
// Execute one admitted task for both scheduler implementations. The return value is consumed
142+
// by TaskExecutor to distinguish terminal EOS/error tasks from scanners that remain runnable.
143+
static bool execute_scan_task(const std::shared_ptr<ScannerContext>& ctx,
144+
const std::shared_ptr<ScanTask>& scan_task);
145+
142146
static void _scanner_scan(std::shared_ptr<ScannerContext> ctx,
143147
std::shared_ptr<ScanTask> scan_task);
144148

@@ -240,12 +244,14 @@ class ThreadPoolSimplifiedScanScheduler MOCK_REMOVE(final) : public ScannerSched
240244
std::unique_lock<std::mutex>& transfer_lock) override;
241245

242246
private:
247+
// `submit_time_ns` is the MonotonicNanos() at which the runnable was submitted.
248+
void _run_context(std::shared_ptr<ScannerContext> scanner_ctx, int64_t submit_time_ns);
249+
243250
std::unique_ptr<ThreadPool> _scan_thread_pool;
244251
std::atomic<bool> _is_stop;
245252
std::weak_ptr<CgroupCpuCtl> _cgroup_cpu_ctl;
246253
std::string _sched_name;
247254
std::string _workload_group;
248-
std::shared_mutex _lock;
249255
};
250256

251257
class TaskExecutorSimplifiedScanScheduler final : public ScannerScheduler {

0 commit comments

Comments
 (0)