Skip to content

Commit 1d89558

Browse files
morningmanclaude
andcommitted
[fix](be) Open no Java scanner for a JNI reader whose scan stopped while it waited for heap
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: When the scan of a reader waiting at the JNI heap gate stopped - its query cancelled, or (V2) its scanner told to stop, for instance by a satisfied limit - JniScanHeapGate::acquire() admitted it whatever the budget said: it took a real permit, its declaration went into the account, and both callers then created and opened the Java scanner. A cancelled query therefore had every one of its waiting readers admitted at once, on their next 100 ms poll, all above the budget, each opening a scanner for a query with nothing left to read. On the V1 path (JniReader, what paimon uses when enable_file_scanner_v2 is false) it was worse: FileScanner checks cancellation only at the top of its loop, so a reader admitted this way read its first block in the same pass - and the first block of a merge read is where the row groups of all its files are loaded, the very allocation the gate keeps under the budget. On V2 get_block() checks should_stop before fetching a batch, but the gate also lets a reader go on a cancelled query before should_stop is set, and every scanner opened still connects to its source (for fluss a connection and a table RPC). acquire() now returns whether it admitted the reader. A reader whose scan has stopped leaves the line without a permit, taking nothing from the account, and the waiters behind it are woken since another may now be first and fit. JniReader::open() then returns the query's cancel reason; JniTableReader::_open_jni_scanner() ends the split as get_block() ends one whose scan stopped (eof, then close), so get_block() reports the end and the scanner the stop. Both now wait before attaching to the JVM, which a reader whose scan stops never does; on V2 that also lets a unit test drive the real open path without a JVM. ### Release note None ### Check List (For Author) - Test: Unit Test - JniScanHeapGateTest: the stop case now expects no permit and an unchanged account; new cases for a reader whose scan has stopped when it arrives (nothing, even though it fits) and for a stopped reader first in line letting the next one in while the holder still holds. The gate and its test were compiled with the BE UT flags (ASAN) and linked into a standalone gtest binary, the gate's other dependencies stubbed (its two configs, the BE_TEST maximum heap, doris_check_fail): all 9 cases pass, 10 runs out of 10 - JniTableReaderTest: new case that drives the real _open_jni_scanner() for a split declaring its heap whose scan has stopped - no permit, no Java scanner, the split ends with eos. Not run locally: after a branch switch the UT target would rebuild nearly all of BE. It, the gate, both readers and both tests compile (-fsyntax-only) with the BE UT flags, and the sources with the Release flags - Behavior changed: No (only readers of a statement that sets enable_jni_heap_admission wait at the gate; such a reader whose scan stops while it waits no longer opens) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 77aa9be commit 1d89558

7 files changed

Lines changed: 172 additions & 51 deletions

File tree

‎be/src/format/jni/jni_reader.cpp‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -149,20 +149,26 @@ Status JniReader::open(RuntimeState* state, RuntimeProfile* profile) {
149149
batch_size = _state->batch_size();
150150
}
151151
_batch_size = batch_size;
152-
RETURN_IF_ERROR(Jni::Env::Get(&env));
153-
// Only a reader whose range declared the heap it will hold waits for it; see the constructor.
152+
// Only a reader whose range declared the heap it will hold waits for it; see the constructor. It
153+
// waits before attaching to the JVM, which a reader whose query is cancelled meanwhile never does.
154154
if (_jni_heap_bytes > 0) {
155155
int64_t heap_wait_ns = 0;
156-
JniScanHeapGate::instance()->acquire(
157-
_jni_heap_bytes,
158-
// A cancelled query opens without waiting for heap and stops at its next block.
159-
[this]() { return _state != nullptr && _state->is_cancelled(); }, &_heap_permit,
160-
&heap_wait_ns);
156+
const bool admitted = JniScanHeapGate::instance()->acquire(
157+
_jni_heap_bytes, [this]() { return _state != nullptr && _state->is_cancelled(); },
158+
&_heap_permit, &heap_wait_ns);
161159
if (_profile != nullptr) {
162160
COUNTER_UPDATE(_jvm_heap_wait_time, heap_wait_ns);
163161
COUNTER_UPDATE(_jvm_heap_declared_bytes, _jni_heap_bytes);
164162
}
163+
if (!admitted) {
164+
// The query was cancelled while this reader waited for heap: it opens no Java scanner.
165+
// Opening it would not be harmless - FileScanner reads the first block in the same
166+
// pass, and a merge read loads the row groups of all its files for that block.
167+
DORIS_CHECK(_state != nullptr && _state->is_cancelled());
168+
return _state->cancel_reason();
169+
}
165170
}
171+
RETURN_IF_ERROR(Jni::Env::Get(&env));
166172
// The permit covers the Java scanner: close() releases it, and here it goes if the scanner
167173
// did not open.
168174
Defer release_unless_opened {[this]() {

‎be/src/format_v2/jni/jni_table_reader.cpp‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -461,9 +461,15 @@ Status JniTableReader::_open_jni_scanner() {
461461
}
462462
_apply_common_scanner_params();
463463

464+
// Before attaching to the JVM, which a split whose scan stops while it waits never does.
465+
if (!_admit_by_declared_heap()) {
466+
// The scan stopped while this split waited for its share of the JVM heap. It opens no Java
467+
// scanner and reads nothing: it ends the way get_block() ends a split whose scan stopped.
468+
_eof = true;
469+
return _close_jni_scanner();
470+
}
464471
JNIEnv* env = nullptr;
465472
RETURN_IF_ERROR(Jni::Env::Get(&env));
466-
_admit_by_declared_heap();
467473
// The permit covers the Java scanner: _reset_split_state() releases it along with the scanner,
468474
// and here it goes if no scanner was created.
469475
Defer release_without_scanner {[this]() {
@@ -488,18 +494,17 @@ Status JniTableReader::_open_jni_scanner() {
488494
return Status::OK();
489495
}
490496

491-
void JniTableReader::_admit_by_declared_heap() {
497+
bool JniTableReader::_admit_by_declared_heap() {
492498
// Only the connector of a statement that set enable_jni_heap_admission declares the heap a
493499
// reader will hold, and only for the readers that hold much; every other reader opens at once.
494500
if (!_current_range.__isset.jni_heap_bytes || _current_range.jni_heap_bytes <= 0) {
495-
return;
501+
return true;
496502
}
497503
int64_t wait_ns = 0;
498-
JniScanHeapGate::instance()->acquire(
504+
const bool admitted = JniScanHeapGate::instance()->acquire(
499505
_current_range.jni_heap_bytes,
500506
[this]() {
501-
// Stopped or cancelled: open without waiting for heap, and see the stop at the
502-
// first get_block(), the way a reader that was already open does.
507+
// Stopped or cancelled: this split will not be read.
503508
return (_io_ctx != nullptr && _io_ctx->should_stop) ||
504509
(_runtime_state != nullptr && _runtime_state->is_cancelled());
505510
},
@@ -508,6 +513,7 @@ void JniTableReader::_admit_by_declared_heap() {
508513
COUNTER_UPDATE(_jvm_heap_wait_time, wait_ns);
509514
COUNTER_UPDATE(_jvm_heap_declared_bytes, _current_range.jni_heap_bytes);
510515
}
516+
return admitted;
511517
}
512518

513519
void JniTableReader::_apply_common_scanner_params() {

‎be/src/format_v2/jni/jni_table_reader.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,8 @@ class JniTableReader : public TableReader {
101101
void _prepare_jni_scanner_schema();
102102
void _apply_common_scanner_params();
103103
// Waits at the JVM heap gate for the heap this split's range declares, if it declares any.
104-
void _admit_by_declared_heap();
104+
// False when the scan stopped meanwhile: then the split opens no Java scanner.
105+
[[nodiscard]] bool _admit_by_declared_heap();
105106
Status _create_jni_scanner(JNIEnv* env, int batch_size);
106107
// get_next
107108
Status _fill_jni_block(JniDataBridge::TableMetaAddress& table_meta, size_t num_rows);

‎be/src/util/jni_scan_heap_gate.cpp‎

Lines changed: 23 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ JniScanHeapGate* JniScanHeapGate::instance() {
6565
return gate;
6666
}
6767

68-
void JniScanHeapGate::acquire(int64_t bytes, const std::function<bool()>& stop_waiting,
68+
bool JniScanHeapGate::acquire(int64_t bytes, const std::function<bool()>& stop_waiting,
6969
Permit* permit, int64_t* wait_ns) {
7070
DORIS_CHECK(bytes > 0);
7171
DORIS_CHECK(permit != nullptr);
@@ -81,26 +81,34 @@ void JniScanHeapGate::acquire(int64_t bytes, const std::function<bool()>& stop_w
8181
const bool stop = stop_waiting();
8282
lock.lock();
8383
const int64_t waited = MonotonicNanos() - start;
84-
bool admit = stop || _fits(ticket, bytes);
85-
if (!admit && waited >= config::jni_scanner_heap_max_wait_ms * 1000 * 1000) {
86-
LOG_EVERY_T(WARNING, 10) << "A JNI scanner opens after waiting " << waited / 1000 / 1000
87-
<< " ms for its share of the JVM heap, longer than "
88-
"jni_scanner_heap_max_wait_ms: it declared "
89-
<< bytes / MB << " MB, while " << _holders << " scanners hold "
90-
<< _admitted_bytes / MB << " MB of a " << _budget() / MB
91-
<< " MB budget and " << _waiting.size() - 1 << " others wait";
92-
admit = true;
84+
bool admit = false;
85+
if (!stop) {
86+
admit = _fits(ticket, bytes);
87+
if (!admit && waited >= config::jni_scanner_heap_max_wait_ms * 1000 * 1000) {
88+
LOG_EVERY_T(WARNING, 10)
89+
<< "A JNI scanner opens after waiting " << waited / 1000 / 1000
90+
<< " ms for its share of the JVM heap, longer than "
91+
"jni_scanner_heap_max_wait_ms: it declared "
92+
<< bytes / MB << " MB, while " << _holders << " scanners hold "
93+
<< _admitted_bytes / MB << " MB of a " << _budget() / MB
94+
<< " MB budget and " << _waiting.size() - 1 << " others wait";
95+
admit = true;
96+
}
9397
}
94-
if (admit) {
98+
if (stop || admit) {
9599
_waiting.erase(std::find(_waiting.begin(), _waiting.end(), ticket));
100+
*wait_ns = waited;
101+
// Whoever is first in line now may fit.
102+
_cv.notify_all();
103+
if (stop) {
104+
// The reader's scan has stopped: it will open nothing, so it takes no share.
105+
return false;
106+
}
96107
_admitted_bytes += bytes;
97108
++_holders;
98109
permit->_gate = this;
99110
permit->_bytes = bytes;
100-
*wait_ns = waited;
101-
// Whoever is first in line now may fit.
102-
_cv.notify_all();
103-
return;
111+
return true;
104112
}
105113
_cv.wait_for(lock, POLL_INTERVAL);
106114
}

‎be/src/util/jni_scan_heap_gate.h‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -46,10 +46,12 @@ namespace doris {
4646
// JVM hold, and garbage nobody has collected yet, are not the gate's business.
4747
//
4848
// Readers that do not fit wait in the order they came, so a large one is not passed over for ever by
49-
// small ones arriving behind it. Three things end a wait whatever the account says: no reader holding
50-
// a share (nobody would give one back for this reader, so one that declares more than the whole budget
51-
// runs alone), the caller asking to stop waiting (a cancelled query), and the wait outlasting
52-
// jni_scanner_heap_max_wait_ms.
49+
// small ones arriving behind it. Two things admit a waiting reader whatever the account says: no
50+
// reader holding a share (nobody would give one back for this reader, so one that declares more than
51+
// the whole budget runs alone), and the wait outlasting jni_scanner_heap_max_wait_ms. A reader whose
52+
// scan stops while it waits - a cancelled query, a satisfied limit - leaves the line without a share
53+
// and opens nothing: admitting it would let every waiting reader of a cancelled query open at once,
54+
// above the budget.
5355
class JniScanHeapGate {
5456
public:
5557
// Held by a reader from before its Java scanner opens until that scanner is closed.
@@ -78,12 +80,12 @@ class JniScanHeapGate {
7880
// of the -Xmx the JVM was started with.
7981
static JniScanHeapGate* instance();
8082

81-
// Waits until `bytes` fit (see the class comment) and hands the reader `permit` for them.
82-
// `stop_waiting` is asked whenever the gate looks again; once it answers true the reader is
83-
// admitted without further waiting, and is expected to notice the stop itself. `wait_ns` gets the
84-
// time spent here.
85-
void acquire(int64_t bytes, const std::function<bool()>& stop_waiting, Permit* permit,
86-
int64_t* wait_ns);
83+
// Waits until `bytes` fit (see the class comment), hands the reader `permit` for them and returns
84+
// true. `stop_waiting` is asked whenever the gate looks, the first time before any wait; once it
85+
// answers true the reader leaves the line without a permit and this returns false: its scan has
86+
// stopped, and it must not open its Java scanner. `wait_ns` gets the time spent here.
87+
[[nodiscard]] bool acquire(int64_t bytes, const std::function<bool()>& stop_waiting,
88+
Permit* permit, int64_t* wait_ns);
8789

8890
int64_t admitted_bytes() const;
8991
int64_t holders() const;

‎be/test/format_v2/jni/jni_table_reader_test.cpp‎

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
#include "format_v2/jni/hudi_jni_reader.h"
4545
#include "io/io_common.h"
4646
#include "runtime/runtime_state.h"
47+
#include "util/jni_scan_heap_gate.h"
4748

4849
namespace doris::format {
4950
namespace {
@@ -112,7 +113,20 @@ class FakeJniTableReader final : public JniTableReader {
112113
}
113114
};
114115

115-
Status init_reader(FakeJniTableReader* reader, const std::shared_ptr<io::IOContext>& io_ctx,
116+
// Opens its splits with JniTableReader's own _open_jni_scanner(). Creating a Java scanner fails in
117+
// BE UT - no Java plugin named "test" exists - so a split gets through prepare_split() only if it
118+
// never creates one.
119+
class RealOpenJniTableReader final : public JniTableReader {
120+
protected:
121+
Jni::PluginRef plugin_ref() const override { return {"test", "fake"}; }
122+
123+
Status build_scanner_params(std::map<std::string, std::string>* params) const override {
124+
params->clear();
125+
return Status::OK();
126+
}
127+
};
128+
129+
Status init_reader(JniTableReader* reader, const std::shared_ptr<io::IOContext>& io_ctx,
116130
RuntimeProfile* scanner_profile = nullptr) {
117131
return reader->init({
118132
.projected_columns = {},
@@ -332,6 +346,41 @@ TEST(JniTableReaderTest, CancellationStopsBeforeFetchingAnotherJavaBatch) {
332346
EXPECT_EQ(reader.close_calls, 1);
333347
}
334348

349+
TEST(JniTableReaderTest, ASplitWhoseScanStoppedOpensNoJavaScannerForTheHeapItDeclared) {
350+
auto io_ctx = std::make_shared<io::IOContext>();
351+
RealOpenJniTableReader reader;
352+
ASSERT_TRUE(init_reader(&reader, io_ctx).ok());
353+
// The statement asked for admission and the range declares its heap, but the scan has stopped -
354+
// a satisfied limit, or a cancelled query - before the split opens.
355+
io_ctx->should_stop = true;
356+
TFileRangeDesc range;
357+
range.__set_jni_heap_bytes(100L * 1024 * 1024);
358+
const int64_t holders = JniScanHeapGate::instance()->holders();
359+
360+
ASSERT_TRUE(reader.prepare_split({
361+
.partition_values = {},
362+
.conjuncts = std::nullopt,
363+
.partition_prune_conjuncts = {},
364+
.all_runtime_filters_applied = true,
365+
.condition_cache_digest = std::nullopt,
366+
.cache = nullptr,
367+
.current_range = range,
368+
.current_split_format = FileFormat::JNI,
369+
.global_rowid_context = std::nullopt,
370+
})
371+
.ok());
372+
// No share of the heap, no Java scanner: the split ends without reading.
373+
EXPECT_EQ(JniScanHeapGate::instance()->holders(), holders);
374+
EXPECT_FALSE(reader.TEST_scanner_opened());
375+
EXPECT_TRUE(reader.TEST_eof());
376+
377+
Block block;
378+
bool eos = false;
379+
ASSERT_TRUE(reader.get_block(&block, &eos).ok());
380+
EXPECT_TRUE(eos);
381+
EXPECT_TRUE(reader.close().ok());
382+
}
383+
335384
TEST(JniTableReaderTest, EndOfSplitRemainsIdempotentAfterScannerClose) {
336385
FakeJniTableReader reader;
337386
ASSERT_TRUE(init_reader(&reader, nullptr).ok());

‎be/test/util/jni_scan_heap_gate_test.cpp‎

Lines changed: 60 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,11 @@ class JniScanHeapGateTest : public testing::Test {
4141

4242
void TearDown() override { config::jni_scanner_heap_max_wait_ms = _saved_max_wait_ms; }
4343

44+
// For a reader whose scan never stops, so it always ends up admitted.
4445
void acquire(int64_t mb, JniScanHeapGate::Permit* permit, int64_t* wait_ns = nullptr) {
4546
int64_t ignored = 0;
46-
_gate.acquire(
47-
mb * MB, []() { return false; }, permit, wait_ns ? wait_ns : &ignored);
47+
EXPECT_TRUE(_gate.acquire(
48+
mb * MB, []() { return false; }, permit, wait_ns ? wait_ns : &ignored));
4849
}
4950

5051
// Acquires `permit` on a thread of its own and reports when it got it.
@@ -168,7 +169,7 @@ TEST_F(JniScanHeapGateTest, ABiggerBudgetAppliesToTheReadersAlreadyWaiting) {
168169
EXPECT_EQ(_gate.admitted_bytes(), 600 * MB);
169170
}
170171

171-
TEST_F(JniScanHeapGateTest, StopsWaitingWhenAskedTo) {
172+
TEST_F(JniScanHeapGateTest, AReaderWhoseScanStopsLeavesWithoutAShare) {
172173
JniScanHeapGate::Permit holder;
173174
acquire(500, &holder);
174175

@@ -179,16 +180,64 @@ TEST_F(JniScanHeapGateTest, StopsWaitingWhenAskedTo) {
179180
});
180181
JniScanHeapGate::Permit stopped;
181182
int64_t wait_ns = 0;
182-
_gate.acquire(
183-
100 * MB, [&cancelled]() { return cancelled.load(); }, &stopped, &wait_ns);
183+
// Its query is cancelled while it waits. Admitting it would open a Java scanner above the budget
184+
// for a query with nothing left to read - all of that query's waiting readers at once.
185+
EXPECT_FALSE(_gate.acquire(
186+
100 * MB, [&cancelled]() { return cancelled.load(); }, &stopped, &wait_ns));
184187
canceller.join();
185-
// Admitted beyond the budget, so that the reader can see the stop itself; the permit is real and
186-
// gives its share back like any other.
187-
EXPECT_TRUE(stopped.held());
188-
EXPECT_EQ(_gate.admitted_bytes(), 600 * MB);
189-
EXPECT_GE(wait_ns, 200LL * 1000 * 1000);
190-
stopped.release();
188+
EXPECT_FALSE(stopped.held());
189+
EXPECT_EQ(_gate.holders(), 1);
191190
EXPECT_EQ(_gate.admitted_bytes(), 500 * MB);
191+
EXPECT_EQ(_gate.waiters(), 0);
192+
EXPECT_GE(wait_ns, 200LL * 1000 * 1000);
193+
}
194+
195+
TEST_F(JniScanHeapGateTest, AReaderWhoseScanHasStoppedTakesNothingEvenWhenItFits) {
196+
JniScanHeapGate::Permit permit;
197+
int64_t wait_ns = -1;
198+
EXPECT_FALSE(_gate.acquire(
199+
100 * MB, []() { return true; }, &permit, &wait_ns));
200+
EXPECT_FALSE(permit.held());
201+
EXPECT_EQ(_gate.holders(), 0);
202+
EXPECT_EQ(_gate.admitted_bytes(), 0);
203+
EXPECT_EQ(_gate.waiters(), 0);
204+
EXPECT_LT(wait_ns, 100LL * 1000 * 1000);
205+
}
206+
207+
TEST_F(JniScanHeapGateTest, AStoppedReaderFirstInLineLetsTheNextOneIn) {
208+
JniScanHeapGate::Permit holder;
209+
acquire(300, &holder);
210+
211+
// 300 + 400 > 500: the large reader waits first in line, and a small one behind it although
212+
// 300 + 100 would fit.
213+
std::atomic<bool> large_stopped {false};
214+
std::atomic<bool> large_returned {false};
215+
JniScanHeapGate::Permit large;
216+
std::thread large_waiter([this, &large_stopped, &large_returned, &large]() {
217+
int64_t ignored = 0;
218+
EXPECT_FALSE(_gate.acquire(
219+
400 * MB, [&large_stopped]() { return large_stopped.load(); }, &large, &ignored));
220+
large_returned.store(true);
221+
});
222+
ASSERT_TRUE(waiters_become(1));
223+
JniScanHeapGate::Permit small;
224+
std::atomic<bool> small_admitted {false};
225+
auto small_waiter = acquire_async(100, &small, &small_admitted);
226+
ASSERT_TRUE(waiters_become(2));
227+
std::this_thread::sleep_for(std::chrono::milliseconds(300));
228+
EXPECT_FALSE(small_admitted.load());
229+
230+
// The large reader's scan stops: it leaves the line, and the small one, first now, fits while
231+
// the holder still holds its share.
232+
large_stopped.store(true);
233+
EXPECT_TRUE(becomes_true(large_returned));
234+
EXPECT_TRUE(becomes_true(small_admitted));
235+
large_waiter.join();
236+
small_waiter.join();
237+
EXPECT_FALSE(large.held());
238+
EXPECT_EQ(_gate.holders(), 2);
239+
EXPECT_EQ(_gate.admitted_bytes(), 400 * MB);
240+
EXPECT_EQ(_gate.waiters(), 0);
192241
}
193242

194243
TEST_F(JniScanHeapGateTest, GivesUpWaitingAfterTheLongestWait) {

0 commit comments

Comments
 (0)