You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1d89558
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](be) Open no Java scanner for a JNI reader whose scan stopped while it waited for heap
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
When the scan of a reader waiting at the JNI heap gate stopped - its query cancelled, or (V2) its
scanner told to stop, for instance by a satisfied limit - JniScanHeapGate::acquire() admitted it
whatever the budget said: it took a real permit, its declaration went into the account, and both
callers then created and opened the Java scanner. A cancelled query therefore had every one of its
waiting readers admitted at once, on their next 100 ms poll, all above the budget, each opening a
scanner for a query with nothing left to read.
On the V1 path (JniReader, what paimon uses when enable_file_scanner_v2 is false) it was worse:
FileScanner checks cancellation only at the top of its loop, so a reader admitted this way read its
first block in the same pass - and the first block of a merge read is where the row groups of all its
files are loaded, the very allocation the gate keeps under the budget. On V2 get_block() checks
should_stop before fetching a batch, but the gate also lets a reader go on a cancelled query before
should_stop is set, and every scanner opened still connects to its source (for fluss a connection and
a table RPC).
acquire() now returns whether it admitted the reader. A reader whose scan has stopped leaves the line
without a permit, taking nothing from the account, and the waiters behind it are woken since another
may now be first and fit. JniReader::open() then returns the query's cancel reason;
JniTableReader::_open_jni_scanner() ends the split as get_block() ends one whose scan stopped (eof,
then close), so get_block() reports the end and the scanner the stop. Both now wait before attaching
to the JVM, which a reader whose scan stops never does; on V2 that also lets a unit test drive the
real open path without a JVM.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- JniScanHeapGateTest: the stop case now expects no permit and an unchanged account; new cases for
a reader whose scan has stopped when it arrives (nothing, even though it fits) and for a stopped
reader first in line letting the next one in while the holder still holds. The gate and its test
were compiled with the BE UT flags (ASAN) and linked into a standalone gtest binary, the gate's
other dependencies stubbed (its two configs, the BE_TEST maximum heap, doris_check_fail): all 9
cases pass, 10 runs out of 10
- JniTableReaderTest: new case that drives the real _open_jni_scanner() for a split declaring its
heap whose scan has stopped - no permit, no Java scanner, the split ends with eos. Not run
locally: after a branch switch the UT target would rebuild nearly all of BE. It, the gate, both
readers and both tests compile (-fsyntax-only) with the BE UT flags, and the sources with the
Release flags
- Behavior changed: No (only readers of a statement that sets enable_jni_heap_admission wait at the
gate; such a reader whose scan stops while it waits no longer opens)
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
0 commit comments