Skip to content

Commit 55f5116

Browse files
morningmanclaude
andcommitted
[improvement](be) Park a JNI scanner that waits for heap instead of blocking a scan worker
### What problem does this PR solve? Issue Number: None Related PR: #68712, #68710 Problem Summary: With enable_jni_heap_admission set, a JNI reader whose range declares more heap than the budget has left waited in JniScanHeapGate::acquire() on the worker that ran its scan turn. The readers already admitted keep their shares until their Java scanners close, which takes them more turns: a turn reads one block and gives its worker back, and the next turn is scheduled only after the operator consumes that block, and only if the scanner context admits it. Once a workload group's remote scan scheduler has min_active_file_scan_threads (8 x cores by default) tasks active or queued - and a thread waiting at the gate counts as active - each context keeps only its minimum of one task in flight. A holder whose context had a reader waiting then stayed in _pending_tasks, nothing reconsidered it until that reader completed a turn, which needed the holder's share, and every waiter opened above the budget once jni_scanner_heap_max_wait_ms (60 s) ran out. On a 16-core BE nine concurrent SELECT * over a fluss primary-key table of 16 declaring buckets get there; with fewer workers than waiters, the pool's thread limit does the same. The wait no longer holds a worker: - JniScanHeapGate::request(bytes, stop_waiting) returns an Admission at once: admitted when the reader fits, stopped when its scan already stopped, otherwise waiting with a SharedListenableFuture<Void>. The gate has a thread of its own. It completes the futures - never inline on a thread that gave a share back, which may hold its scanner context's lock (stop_scanners() closing scanners) - and every 100 ms it notices stopped scans (the IOContext's should_stop, set by try_stop()), readers past the wait limit, and a changed budget. The admission rules do not change. - A JNI reader that has to wait opens nothing, defers its Java scanner to its first read, and reports what it waits for through waiting_for() (V2 JniTableReader, forwarded by the hybrid readers through TableReader::waiting_for(); V1 JniReader, which takes the scanner's IOContext from ReaderInitContext). When its turn comes it opens the scanner, or ends its split if its scan stopped first. - FileScannerV2 and FileScanner end the turn without a block (Scanner::take_waiting_for()). ScannerScheduler::_scanner_scan parks the task: ScanTask::State::PARKED, outside _in_flight_tasks_num, so it holds neither a worker nor a slot, and the context schedules again because that slot may now admit a pending holder. When the future is done, ScannerContext::_resume_parked_task() schedules the task as if the operator had consumed its block, on either scheduler. The wait limit stays, now documented per split: in a join whose two sides both declare, the probe side's scanners keep their shares while the probe waits for the build, whose later splits queue behind them, and only the limit ends that. ### Release note None ### Check List (For Author) - Test: Unit Test - JniScanHeapGateTest (11, rewritten for the asynchronous gate): among them readers past the wait limit admitted together above the budget, and futures completed on the gate's thread - ScannerContextTest (4 new): scanners that each declare the whole budget, on a one-worker ThreadPool and TaskExecutor scheduler and on a busy pool that holds the context to one task in flight, all finish with nobody admitted by the wait limit; a context that stops while they wait ends its parked tasks. The debug-string case covers the parked count - JniTableReaderTest (2 new): a split that does not fit waits without opening its Java scanner, and closed unopened gives its share back; one whose scan stops while it waits opens nothing - 269 BE tests in 31 suites pass (scan scheduling, file scanners V1/V2, the JNI readers, the paimon/fluss/hudi readers, the task executor); the new cases also 20 times, shuffled. clang-tidy, clang-format 16 and the build-hygiene gate pass - Behavior changed: Yes. With enable_jni_heap_admission set, a JNI reader that waits for heap parks its scanner instead of blocking a scan thread - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 568c7e2 commit 55f5116

28 files changed

Lines changed: 1113 additions & 307 deletions

‎be/src/common/config.cpp‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1888,7 +1888,9 @@ DEFINE_mInt64(hdfs_jni_write_max_retry_time, "3");
18881888
// declare together. Only statements with enable_jni_heap_admission declare it
18891889
// (util/jni_scan_heap_gate.h)
18901890
DEFINE_mDouble(jni_scanner_heap_budget_ratio, "0.5");
1891-
// The longest such a JNI scanner waits for its share before it opens anyway, in milliseconds
1891+
// The longest the Java scanner of one split that declared its heap waits for its share before it
1892+
// opens anyway, in milliseconds. A scan of many splits can wait this long once per split. The wait
1893+
// holds no scan thread: the scanner is parked until the share comes
18921894
DEFINE_mInt64(jni_scanner_heap_max_wait_ms, "60000");
18931895

18941896
// The min thread num for NonBlockCloseThreadPool

‎be/src/common/config.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1902,7 +1902,9 @@ DECLARE_mInt64(hdfs_jni_write_max_retry_time);
19021902
// declare together. Only statements with enable_jni_heap_admission declare it
19031903
// (util/jni_scan_heap_gate.h)
19041904
DECLARE_mDouble(jni_scanner_heap_budget_ratio);
1905-
// The longest such a JNI scanner waits for its share before it opens anyway, in milliseconds
1905+
// The longest the Java scanner of one split that declared its heap waits for its share before it
1906+
// opens anyway, in milliseconds. A scan of many splits can wait this long once per split. The wait
1907+
// holds no scan thread: the scanner is parked until the share comes
19061908
DECLARE_mInt64(jni_scanner_heap_max_wait_ms);
19071909

19081910
// The min thread num for NonBlockCloseThreadPool

‎be/src/exec/scan/file_scanner.cpp‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -569,6 +569,19 @@ Status FileScanner::_get_block_wrapped(RuntimeState* state, Block* block, bool*
569569
return Status::OK();
570570
}
571571

572+
// A JNI reader whose range declared its heap opens its Java scanner only once the JNI heap
573+
// gate admits it. Until then this turn ends without a block, and the scheduler runs the
574+
// scanner again once the gate is done with it.
575+
if (_current_range.__isset.jni_heap_bytes && _current_range.jni_heap_bytes > 0) {
576+
auto* jni_reader = dynamic_cast<JniReader*>(_cur_reader.get());
577+
// Only paimon declares in V1, and only for a split it reads through JNI.
578+
DORIS_CHECK(jni_reader != nullptr);
579+
if (auto waiting_for = jni_reader->waiting_for()) {
580+
_waiting_for = std::move(waiting_for);
581+
return Status::OK();
582+
}
583+
}
584+
572585
// Init src block for load job based on the data file schema (e.g. parquet)
573586
// For query job, simply set _src_block_ptr to block.
574587
size_t read_rows = 0;
@@ -1004,6 +1017,7 @@ void FileScanner::_fill_base_init_context(ReaderInitContext* ctx) {
10041017
ctx->range = &_current_range;
10051018
ctx->table_info_node = TableSchemaChangeHelper::ConstNode::get_instance();
10061019
ctx->push_down_agg_type = _get_push_down_agg_type();
1020+
ctx->io_ctx = _io_ctx;
10071021
}
10081022

10091023
Status FileScanner::_get_next_reader() {

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -483,6 +483,15 @@ Status FileScannerV2::_get_block_impl(RuntimeState* state, Block* block, bool* e
483483
return Status::OK();
484484
}
485485
}
486+
// A JNI reader whose split declared its heap opens its Java scanner only once the JNI heap
487+
// gate admits it. Until then this turn ends without a block, and the scheduler runs the
488+
// scanner again once the gate is done with it.
489+
if (_current_range.__isset.jni_heap_bytes && _current_range.jni_heap_bytes > 0) {
490+
if (auto waiting_for = _table_reader->waiting_for()) {
491+
_waiting_for = std::move(waiting_for);
492+
return Status::OK();
493+
}
494+
}
486495

487496
{
488497
if (_table_reader_rf_num != _applied_rf_num) {

‎be/src/exec/scan/scanner.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ Status Scanner::get_block_after_projects(RuntimeState* state, Block* block, bool
104104
const auto min_batch_size = std::max(state->batch_size() / 2, 1);
105105
const auto block_max_bytes = state->preferred_block_size_bytes();
106106
while (_padding_block.rows() < min_batch_size && _padding_block.bytes() < block_max_bytes &&
107-
!*eos) {
107+
!*eos && !_waiting_for.has_value()) {
108108
RETURN_IF_ERROR(get_block(state, &_origin_block, eos));
109109
if (*eos) {
110110
// For the final block, merge any padding directly and return eos in this call.
@@ -192,7 +192,7 @@ Status Scanner::get_block(RuntimeState* state, Block* block, bool* eof) {
192192
_shared_scan_limit->fetch_sub(block->rows(), std::memory_order_acq_rel);
193193
}
194194
} while (!_should_stop && !state->is_cancelled() && block->rows() == 0 && !(*eof) &&
195-
_num_rows_read < rows_read_threshold);
195+
_num_rows_read < rows_read_threshold && !_waiting_for.has_value());
196196
}
197197

198198
if (state->is_cancelled()) {

‎be/src/exec/scan/scanner.h‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,12 @@
2323
#include <atomic>
2424
#include <functional>
2525
#include <optional>
26+
#include <utility>
2627
#include <vector>
2728

2829
#include "common/status.h"
2930
#include "core/block/block.h"
31+
#include "exec/scan/task_executor/listenable_future.h"
3032
#include "runtime/exec_env.h"
3133
#include "runtime/runtime_state.h"
3234
#include "storage/tablet/tablet.h"
@@ -93,6 +95,14 @@ class Scanner {
9395
// Try to stop scanner, and all running readers.
9496
virtual void try_stop() { _should_stop = true; };
9597

98+
// What the turn that just ended without a block waits for, if anything: get_block() returned no
99+
// rows and no eof because the scanner cannot read on until this future is done - for now, the
100+
// JNI heap gate admitting the Java scanner of the split it is on. The scheduler parks the scanner
101+
// instead of keeping a worker meanwhile, and runs it again once the future is done.
102+
std::optional<SharedListenableFuture<Void>> take_waiting_for() {
103+
return std::exchange(_waiting_for, std::nullopt);
104+
}
105+
96106
virtual std::string get_name() { return ""; }
97107

98108
// return the readable name of current scan range.
@@ -311,6 +321,10 @@ class Scanner {
311321

312322
bool _should_stop = false;
313323

324+
// Set by _get_block_impl() when it returns without a block because the scanner has to wait; see
325+
// take_waiting_for().
326+
std::optional<SharedListenableFuture<Void>> _waiting_for;
327+
314328
// Cached pointer to ScanOperator's remaining-limit counter. Null when
315329
// this scanner is on the topn path or the query has no LIMIT.
316330
std::atomic<int64_t>* _shared_scan_limit = nullptr;

‎be/src/exec/scan/scanner_context.cpp‎

Lines changed: 60 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include <ctime>
2727
#include <memory>
2828
#include <mutex>
29+
#include <optional>
2930
#include <ostream>
3031
#include <shared_mutex>
3132
#include <tuple>
@@ -362,6 +363,61 @@ void ScannerContext::push_completed_scan_task(std::shared_ptr<ScanTask> scan_tas
362363
_dependency->set_ready();
363364
}
364365

366+
void ScannerContext::park_scan_task(std::shared_ptr<ScanTask> scan_task,
367+
SharedListenableFuture<Void> waiting_for) {
368+
{
369+
std::unique_lock<std::mutex> l(_transfer_lock);
370+
scan_task->set_state(ScanTask::State::PARKED);
371+
_in_flight_tasks_num--;
372+
++_parked_tasks_num;
373+
// The slot it held may now admit a pending scanner - one whose block the operator consumed
374+
// while this context had no slot to spare, holding what this task waits for, among them.
375+
// Nothing else would look: a parked task never completes a scan attempt to make the operator
376+
// schedule again.
377+
if (!done()) {
378+
Status status = _scanner_scheduler->schedule_scan_task(shared_from_this(), nullptr, l);
379+
if (!status.ok()) {
380+
set_context_failure(status, l);
381+
}
382+
}
383+
}
384+
std::weak_ptr<ScannerContext> weak_ctx = shared_from_this();
385+
// Runs on the thread that completes the future, or right here if it is done already.
386+
waiting_for.add_callback([weak_ctx, scan_task](const Void&, const Status&) {
387+
if (auto ctx = weak_ctx.lock()) {
388+
ctx->_resume_parked_task(scan_task);
389+
}
390+
});
391+
}
392+
393+
void ScannerContext::_resume_parked_task(const std::shared_ptr<ScanTask>& scan_task) {
394+
auto task_execution_lock = task_exec_ctx();
395+
if (task_execution_lock == nullptr) {
396+
// The query has finished; nothing will read this task again.
397+
return;
398+
}
399+
#ifndef BE_TEST
400+
// Scheduling allocates for the query, not for the gate's thread that completed the future. A
401+
// future done before park_scan_task() added this callback runs it on the worker that parked
402+
// the task, which is attached to this query already and must stay so.
403+
std::optional<AttachTask> attach;
404+
if (!thread_context()->is_attach_task()) {
405+
attach.emplace(_state);
406+
}
407+
#endif
408+
std::unique_lock<std::mutex> l(_transfer_lock);
409+
--_parked_tasks_num;
410+
if (done()) {
411+
return;
412+
}
413+
// Like a task whose block the operator just consumed: it may run again.
414+
scan_task->set_state(ScanTask::State::PENDING);
415+
Status status = _scanner_scheduler->schedule_scan_task(shared_from_this(), scan_task, l);
416+
if (!status.ok()) {
417+
set_context_failure(status, l);
418+
}
419+
}
420+
365421
Status ScannerContext::get_block_from_queue(RuntimeState* state, Block* block, bool* eos, int id) {
366422
if (state->is_cancelled()) {
367423
_set_scanner_done();
@@ -545,14 +601,15 @@ std::string ScannerContext::debug_string() {
545601
return fmt::format(
546602
"_query_id: {}, id: {}, total scanners: {}, pending tasks: {}, completed tasks: {},"
547603
" _should_stop: {}, _is_finished: {}, free blocks: {},"
548-
" limit: {}, remaining_limit: {}, _in_flight_tasks_num: {}, _is_context_queued: {}, "
549-
"_num_finished_scanners: {}, _max_scan_concurrency: {}, expected_scanners: {},"
604+
" limit: {}, remaining_limit: {}, _in_flight_tasks_num: {}, _parked_tasks_num: {}, "
605+
"_is_context_queued: {}, _num_finished_scanners: {}, _max_scan_concurrency: {}, "
606+
"expected_scanners: {},"
550607
" _max_bytes_in_queue: {}, _ins_idx: {}, _enable_adaptive_scanners: {}, "
551608
"_mem_share_arb: {}, _scanner_mem_limiter: {}",
552609
print_id(_query_id), ctx_id, _all_scanners.size(), _pending_tasks.size(),
553610
_completed_tasks.size(), _should_stop, _is_finished, _free_blocks.size_approx(), limit,
554611
_shared_scan_limit->load(std::memory_order_relaxed), _in_flight_tasks_num,
555-
_is_context_queued, _num_finished_scanners, _max_scan_concurrency,
612+
_parked_tasks_num, _is_context_queued, _num_finished_scanners, _max_scan_concurrency,
556613
_enable_adaptive_scanners ? _adaptive_processor->expected_scanners : -1,
557614
_max_bytes_in_queue, _ins_idx, _enable_adaptive_scanners,
558615
_enable_adaptive_scanners ? _mem_share_arb->debug_string() : "NULL",

‎be/src/exec/scan/scanner_context.h‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ class ScanTask {
9696
IN_FLIGHT, // scheduled and running
9797
COMPLETED, // finished with result or error, waiting to be collected by scan node
9898
EOS, // finished and no more data, waiting to be collected by scan node
99+
PARKED, // waiting, off the workers, for what its scanner cannot read on without
99100
};
100101
ScanTask(std::weak_ptr<ScannerDelegate> delegate_scanner);
101102

@@ -131,8 +132,9 @@ class ScanTask {
131132
case State::PENDING:
132133
// A task returns to PENDING after the operator consumes its non-EOS cached block.
133134
// For example, one scanner may produce several blocks, so COMPLETED is not terminal.
135+
// A parked task returns to it once what it waited for is done.
134136
DCHECK(_state == State::PENDING || _state == State::IN_FLIGHT ||
135-
_state == State::COMPLETED)
137+
_state == State::COMPLETED || _state == State::PARKED)
136138
<< (int)_state;
137139
DCHECK(cached_block == nullptr);
138140
break;
@@ -150,6 +152,10 @@ class ScanTask {
150152
DCHECK(_state == State::IN_FLIGHT || status.is<ErrorCode::END_OF_FILE>())
151153
<< (int)_state;
152154
break;
155+
case State::PARKED:
156+
DCHECK(_state == State::IN_FLIGHT) << (int)_state;
157+
DCHECK(cached_block == nullptr);
158+
break;
153159
default:
154160
break;
155161
}
@@ -215,6 +221,15 @@ class ScannerContext : public std::enable_shared_from_this<ScannerContext>,
215221
// block and returns a non-EOS task to PENDING for its next scan attempt.
216222
void push_completed_scan_task(std::shared_ptr<ScanTask> scan_task);
217223

224+
// Park a task whose scan attempt ended without a block because its scanner cannot read on until
225+
// `waiting_for` is done (Scanner::take_waiting_for()). A parked task holds no worker and no
226+
// concurrency slot - another scanner of this context, perhaps one holding what it waits for,
227+
// is admitted in its place - and returns to scheduling, as if the operator had just consumed
228+
// its block, once the future is done. The caller must not touch the task afterwards: by then
229+
// another worker may already be running it.
230+
void park_scan_task(std::shared_ptr<ScanTask> scan_task,
231+
SharedListenableFuture<Void> waiting_for);
232+
218233
// Return true if this ScannerContext need no more process
219234
bool done() const { return _is_finished || _should_stop; }
220235

@@ -299,6 +314,8 @@ class ScannerContext : public std::enable_shared_from_this<ScannerContext>,
299314
/// 4. At most scale up `MAX_SCALE_UP_RATIO` times to `_max_thread_num`
300315
void _set_scanner_done();
301316
bool _is_shared_scan_limit_exhausted() const;
317+
// The callback of a parked task's future; runs on whichever thread completed it.
318+
void _resume_parked_task(const std::shared_ptr<ScanTask>& scan_task);
302319

303320
RuntimeState* _state = nullptr;
304321
ScanLocalStateBase* _local_state = nullptr;
@@ -364,6 +381,9 @@ class ScannerContext : public std::enable_shared_from_this<ScannerContext>,
364381
// but must be read under _transfer_lock whenever combined with _completed_tasks.size()
365382
// to form a consistent concurrency snapshot.
366383
std::atomic_int _in_flight_tasks_num = 0;
384+
// Tasks parked by park_scan_task() until what their scanners wait for is done. They are not in
385+
// flight and occupy no concurrency slot. Protected by _transfer_lock.
386+
int32_t _parked_tasks_num = 0;
367387
// Scanner that is eos or error.
368388
int32_t _num_finished_scanners = 0;
369389
// weak pointer for _scanners, used in stop function

‎be/src/exec/scan/scanner_scheduler.cpp‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include <functional>
2323
#include <list>
2424
#include <memory>
25+
#include <optional>
2526
#include <ostream>
2627
#include <string>
2728
#include <utility>
@@ -182,6 +183,8 @@ void ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
182183

183184
Status status = Status::OK();
184185
bool eos = false;
186+
// Set when the scanner ended the attempt without a block because it cannot read on yet.
187+
std::optional<SharedListenableFuture<Void>> waiting_for;
185188
auto append_late_arrival_runtime_filter = [&] {
186189
Status rf_status = scanner->try_append_late_arrival_runtime_filter();
187190
if (!rf_status.ok()) {
@@ -270,10 +273,20 @@ void ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
270273
// We got a new created block or a reused block.
271274
status = scanner->get_block_after_projects(state, free_block.get(), &eos);
272275
first_read = false;
276+
waiting_for = scanner->take_waiting_for();
273277
if (!status.ok()) {
278+
waiting_for.reset();
274279
LOG(WARNING) << "Scan thread read Scanner failed: " << status.to_string();
275280
break;
276281
}
282+
if (waiting_for.has_value() && !eos && free_block->rows() == 0) {
283+
// Nothing to hand over: the block goes back, and the task is parked below.
284+
ctx->return_free_block(std::move(free_block));
285+
break;
286+
}
287+
// Rows padded before the scanner had to wait still go out; it says so again
288+
// on its next attempt.
289+
waiting_for.reset();
277290
// Check column type only after block is read successfully.
278291
// Or it may cause a crash when the block is not normal.
279292
_make_sure_virtual_col_is_materialized(scanner, free_block.get());
@@ -312,12 +325,25 @@ void ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
312325
if (UNLIKELY(!status.ok())) {
313326
scan_task->set_status(status);
314327
eos = true;
328+
// Also an exception after the scanner said it had to wait: a failed attempt ends it.
329+
waiting_for.reset();
315330
}
316331

317332
// Always update scanner profile to properly account for CPU time on the same
318333
// thread that started the CPU timer (CLOCK_THREAD_CPUTIME_ID is per-thread).
319334
update_scanner_profile();
320335

336+
if (waiting_for.has_value()) {
337+
// A failed attempt ends the scanner, and so does an EOS one: neither waits for anything.
338+
DORIS_CHECK(status.ok() && !eos);
339+
VLOG_DEBUG << fmt::format("Scanner context {} parks a task until its scanner can read on",
340+
ctx->ctx_id);
341+
// The last thing this worker does with the task, as push_completed_scan_task() is below:
342+
// once the future is done, another worker may run it.
343+
ctx->park_scan_task(scan_task, std::move(*waiting_for));
344+
return;
345+
}
346+
321347
if (eos) {
322348
scanner->mark_to_need_to_close();
323349
scan_task->set_state(ScanTask::State::EOS);

‎be/src/format/generic_reader.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,10 @@ namespace doris {
4747
class ColumnPredicate;
4848
} // namespace doris
4949

50+
namespace doris::io {
51+
struct IOContext;
52+
} // namespace doris::io
53+
5054
namespace doris {
5155

5256
class Block;
@@ -69,6 +73,8 @@ struct ReaderInitContext {
6973
const TFileScanRangeParams* params = nullptr;
7074
const TFileRangeDesc* range = nullptr;
7175
TPushAggOp::type push_down_agg_type = TPushAggOp::type::NONE;
76+
// The scanner's, marked when its scan stops (FileScanner::try_stop()).
77+
std::shared_ptr<io::IOContext> io_ctx;
7278

7379
// ---- Output slots (populated by on_before_init_reader, consumed by _do_init_reader) ----
7480
// column_names: the list of file columns to read. Populated by on_before_init_reader

0 commit comments

Comments
 (0)