Skip to content

Commit 7ac0eb2

Browse files
morningmanclaude
andauthored
[improvement](fluss) Pool fluss connections and share a scan node's reader cache across instances (apache#68711)
### What problem does this PR solve? Issue Number: None Related PR: apache#66399 (fluss catalog) Problem Summary: **In short.** Two fixed costs made fluss scans slow regardless of how much data they read. Every scan range spent about two seconds closing a fluss connection of its own, invisible in the profile. A union read of a primary-key table also read each bucket's log tail once per pipeline instance instead of once per backend. Together they made a union read with any log tail slower than not using union read at all: `COUNT(*)` took 8.6 s against 4.9 s with union read disabled on a log table, and 21.9 s against 11.7 s on a primary-key table. This PR pools fluss connections across scan ranges and shares a file scan node's reader cache across its instances. A union-read `COUNT(*)` with a tail is now 2-15x faster than union read disabled, and a `COUNT(*)` on a three-row table that took 2 s takes 24 ms. **Background** - BE reads a fluss table through JNI: `FlussJniScanner` (plugin `fe/be-java-extensions/fluss-scanner`) reads one scan range, which is one bucket of a log table, one bucket of a primary-key table (`PK_FULL`, a kv snapshot plus the change log after it), or the log tail of a union read (`LOG` / `PK_TAIL`). - A union read splits a table into the part already tiered into the lake (paimon splits, read natively or through paimon JNI) and the log tail not tiered yet (fluss ranges). For a primary-key table every lake split must drop the rows whose keys the tail updated, so `FlussUnionLakeReader` first reads the keys of that bucket's tail (a JNI read) and caches them in the scan node's `ShardedKVCache`, where the other lake splits of the bucket find them. The same cache holds iceberg delete files and deletion vectors. - A scan node runs as several pipeline instances per backend (up to `parallel_pipeline_task_num`), and splits are dealt to instances with no regard for the bucket or delete file they share. **The problem, and what it cost** 1. **Two seconds per range in close.** Every range opened a fluss `Connection` in `openInternal` and closed it in `closeInternal`, on the scan thread. Closing shuts the connection's netty client down with netty's default graceful shutdown, which returns only after two quiet seconds (`NettyClient#close`); fluss has no shorter close. The profile does not show it: BE publishes the JNI scanner's counters before it calls the Java `close()`. Measured on master (Release BE, local fluss cluster, 16-bucket tables of 30M rows): | `COUNT(*)` on master | Time | Where it went | |---|---|---| | 3-row log table | 2069 ms | 2044 ms closing its one connection | | 30M-row log table, 16 ranges | 4480 ms | about 32 s of closes, summed over the ranges | | log lake table with a 1% log tail, union read | 8320 ms | 32 ms reading the tail; nearly all the rest closing connections | | the same table with no tail, union read | 130 ms | | | the same table, union read disabled | 4610 ms | | A small tail made union read slower than not using it. 2. **The tail keys read once per instance.** The cache lived in `FileScanLocalState`, one per pipeline instance. On a 16-bucket primary-key table whose lake has 80 splits spread over 9 instances, `COUNT(*)` read the tails 69 times where 16 would do (cache hits 11 of 80), each read paying the two seconds above, and kept 6.6M keys in the JVM heap for 1.6M distinct ones: 21.9 s, against about 11.7 s with union read disabled. Iceberg delete files and deletion vectors were likewise parsed once per instance. 3. **A connection per range costs more than its close.** Every range also paid a connect, and every connection started fluss's default of four netty network threads, each with its own selector (on macOS a kqueue and a pipe pair). Eight concurrent `SELECT *` over a 16-bucket table held over a hundred connections and up to 2400 file descriptors in BE; past 1024 descriptors, libcurl on macOS (which waits with `select()`) fails BE's S3 reads with "A libcurl function was given a bad argument". 4. **A bug in the same lifecycle (also on master).** A `PK_FULL` range starts copying its bucket's kv snapshot on the connection's download threads when it opens. If the range is closed before the snapshot arrives (the query was cancelled or failed elsewhere), closing the connection runs `shutdownNow()` on its download pool, which drops the queued files, and the copy waits for them without a timeout. The copy thread, the thread waiting to close the snapshot reader, and a half-copied snapshot directory under `java.io.tmpdir/fluss` then stay for the life of BE. **How this PR fixes it** Four commits, and a fifth from review that keeps an `OutOfMemoryError` in the cleanup of the first two from losing a connection (described with them): 1. **Pool connections; close the rest off the scan thread.** Ranges borrow connections from the new `FlussConnectionPool`: - one range at a time per connection, so a primary-key range keeps three download threads to itself, as now; a connection just outlives its range and serves the next one; - keyed by the whole client configuration, so catalogs with different servers, credentials or options never share one; - a range read to its end gives the connection back, and the most recently returned connection is lent next. A range that failed or was closed early (a LIMIT, a cancel) has its connection closed instead, since it may be broken by what failed it; - a reaper closes connections nobody borrowed for 60 s. Its sweep catches `Throwable`, because a periodic task that throws once is never run again, and takes expired connections out one at a time, handing each to the closer at once, so an error mid-sweep costs at most the connection in hand. Discarded and idle connections are closed by `FlussConnectionCloser` on daemon threads, at most 256 at a time; past that the caller closes its own and the log says so once a minute. A connection that cannot be handed to a closer thread at all (an `OutOfMemoryError` building the task or the thread) is closed by the caller as well. A failure to close is logged and does not fail a query whose rows were already returned. The profile gains `FlussJniConnectionsOpened`. 2. **Keep a `PK_FULL` range's connection until its snapshot reader lets go.** `SafeKvSnapshotAndLogBatchScanner#released()` completes once the fluss snapshot reader is closed. `closeInternal` gives back or discards the connection after that: at once in every case except a range closed before its snapshot arrived, whose connection is handed over by the waiter thread after the late publication. The waiter keeps waiting through an `OutOfMemoryError` on its own thread, and a failure to build it falls back to waiting on the closing thread, as a failure to start it already did. 3. **One network thread per connection.** A pooled connection serves one range at a time, one stream of requests, so BE opens connections with `netty.client.num-network-threads = 1` unless the catalog sets `fluss.netty.client.num-network-threads`, in which case its value wins and those connections are pooled apart. FE's connection, one per catalog, keeps the fluss default. 4. **One reader cache per scan node per backend (BE).** `ShardedKVCache` moves from `FileScanLocalState` to `FileScanOperatorX` and is created in `prepare()`, so every instance of the node reads through one cache. It is safe to share: every cached value is plain data (roaring bitmaps, delete-row vectors, key blocks with an immutable hash index) referring to no instance's profile, IO context or runtime state; the cache locks per shard and was already shared by the scanners of an instance; nothing is erased and the operator outlives every local state and scanner. The shard count stays what the per-instance caches had between them: the fragment's instance count times the per-instance scanner limit, capped by the remote scan thread pool. What it buys: - Union read of fluss tables is faster than union read disabled at every tail size. - Small tables and back-to-back queries no longer cost two seconds; repeated queries open no connection at all. - Iceberg delete files, deletion vectors and fluss tail keys are parsed once per scan node per backend. - Fewer netty threads and file descriptors in BE under concurrent fluss scans. - A cancelled primary-key read no longer leaks threads and partly copied snapshot files. **Results** Release BE, 2 GB JVM heap, local fluss 1.0.0 cluster, 30M-row tables of 16 buckets; master against this PR on the same data: | Query | master | this PR | |---|---|---| | 3-row log table, `COUNT(*)` | 2069 ms | 24 ms | | 30M-row log table, `COUNT(*)` | 4480 ms | 400 ms | | log table with a 15M-row tail, union read, `COUNT(*)` | 8612 ms | 405 ms | | the same with union read disabled | 4864 ms | 568 ms | | primary-key table (compacted lake, 100K updated keys per bucket in the tail), union read, `COUNT(*)` | 21929 ms | 1038 ms | | same, tail-key cache hits / time in tail reads summed over scanners | 11 of 80 / 139.8 s | 61 of 80 / 0.9 s | | same table with an uncompacted lake (32 JNI merge splits), union read, `COUNT(*)` | 13.8 s | 1.6 s | Union read against union read disabled after this PR, by tail size (`COUNT(*)`, ms): | Log table tail | 0.1% | 1% | 10% | 50% | |---|---|---|---|---| | union read | 175 | 177 | 213 | 375 | | union read disabled | 633 | 621 | 568 | 839 | | Primary-key table tail (keys per bucket) | 1K | 10K | 100K | |---|---|---|---| | union read, compacted lake | 490 | 641 | 1079 | | union read, uncompacted lake | 1229 | 1357 | 1660 | | union read disabled | about 7600-8000 | | | What the pool adds over closing in the background alone, which stops at its 256 closes in flight: | Workload | background close only | pooled (this PR) | |---|---|---| | 64-bucket log table, `COUNT(*)` 20 times back to back | median 133 ms, every fourth query 2.2 s | median 103 ms, slowest 128 ms | | connections a repeated query opens (`FlussJniConnectionsOpened`) | one per range | 0 | | partitioned primary-key lake table (30 partitions x 4 buckets, 1% tail), union `COUNT(*)`, 240 JNI reads per query | 2090 ms | 1667 ms | | the same, four at once | 11.3 s each, 3326 BE threads | 4.9 s each, 2067 BE threads | | 16-bucket log table, `SELECT *` with 16 scanners | 10.8 s | 10.5 s | | 16-bucket primary-key table, `SELECT *` with 16 scanners | 6.67 s | 6.46 s | One network thread per connection (16-bucket log and primary-key tables of 10M rows, only the plugin swapped, four A/B rounds): | Measure | 4 network threads (fluss default) | 1 network thread (this PR) | |---|---|---| | after one `SELECT *` (16 connections left in the pool): netty client threads / file descriptors | 32 / 625 | 16 / 479 | | 4 and 8 concurrent `SELECT *`: peak file descriptors in BE | 1238-2415 | 811-1167 | | 4 and 8 concurrent `SELECT *`: peak threads in BE | 2186-2361 | 2085-2245 | | single and concurrent queries: median latency | | within 5% either way | Snapshot copy (commit 2), after six out-of-memory `SELECT *` queries in a row over a 16-bucket primary-key table: | Left behind | before the fix | with the fix | |---|---|---| | copy threads parked in `FileDownloadUtils.transferAllDataToDirectory` | 2 | 0 | | `fluss-snapshot-publication-close` threads blocked behind them | 2 | 0 | | half-copied snapshot directories under `java.io.tmpdir/fluss` | 2 (17 MB, 4.3 MB) | 0 | Union read, union read disabled and the expected values agree on `COUNT(*)`, `SUM(id)` and `SUM(price)` in every state measured. **Classes, and how they call each other** - `FlussJniScanner` (existing): `openInternal` borrows a `Lease` from the pool and sets one network thread unless the catalog set a number; `closeInternal` closes scanner and table, then gives the connection back (read to the end) or discards it (failed or closed early), for a `PK_FULL` range only after `SafeKvSnapshotAndLogBatchScanner#released()`; reports `FlussJniConnectionsOpened`. - `FlussConnectionPool` (new): `borrow` / `giveBack` / `discard` / `closeIdleLongerThan`, idle connections per client configuration, most recent first; the `fluss-connection-reaper` thread sweeps every 10 s. - `FlussConnectionCloser` (new): closes connections on up to 256 daemon threads, on the caller's thread past that or when one cannot be handed over. - `SafeKvSnapshotAndLogBatchScanner` / `PublicationGuardedBatchScanner` (existing): `released()` completes when the fluss snapshot reader has been closed, by whichever path closed it. - `FileScanOperatorX` (BE, existing): owns `_kv_cache` now, created in `prepare()`; `FileScanLocalState` hands it to every `FileScanner` / `FileScannerV2` it creates. - `FlussUnionLakeReader`, the iceberg delete-file readers, the deletion-vector readers (BE, untouched): use the cache through the scanner, as before. ``` BE, one file scan node instance 1 .. N (pipeline instances of the node on this BE) FileScanOperatorX::_kv_cache <---------- every scanner of every instance (was: one cache per instance) '- FlussUnionLakeReader: keys of bucket b's log tail, read once per BE '- iceberg delete files, deletion vectors scanner --JNI--> FlussJniScanner (one per range) openInternal: FlussConnectionPool.borrow(client config) -> an idle connection, or a new one (netty.client.num-network-threads = 1 unless the catalog sets it) ... read ... closeInternal: close scanner, table read to its end ? pool.giveBack(lease) : pool.discard(lease) PK_FULL closed before its snapshot arrived: after SafeKvSnapshotAndLogBatchScanner.released() FlussConnectionPool --discard, or idle for 60 s (reaper, every 10 s)--> FlussConnectionCloser <= 256 daemon threads, each waiting out netty's 2 s graceful shutdown ``` Not touched: FE planning, which ranges and splits a union read produces, and how a range reads its records. **Merge order.** No dependency on other PRs. Best merged before apache#68712: once a scan runs up to 16 scanners per instance by default again, every scan opens up to 16 fluss connections at once, which this PR reuses and slims. ### Release note Fluss catalog scans reuse fluss client connections across scan ranges and no longer spend about two seconds per scan range closing one, which speeds up aggregations, union reads with a log tail, back-to-back queries and queries on small or heavily partitioned fluss tables. Fluss union reads of primary-key tables, iceberg delete files and deletion vectors are read once per scan node per backend instead of once per pipeline instance. BE opens fluss connections with one network thread unless the catalog sets `fluss.netty.client.num-network-threads`. A cancelled read of a fluss primary-key table no longer leaves threads and partly copied snapshot files behind. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> **Unit tests.** `FlussConnectionPoolTest` (new, 9): a range borrows what the range before it gave back; ranges reading at once each get their own connection and none is lent twice (8 threads x 2000 borrows); other settings never share; discarded and idle connections are closed; a sweep that throws `OutOfMemoryError` leaves the next sweeps coming, and one that fails while handing connections to the closer keeps the ones it had not reached. `FlussConnectionCloserTest` (new, 2): the caller does not wait for the close; past 256 the caller closes its own. `FlussJniScannerLogTest` (3 new): closing a range does not wait for its connection; a range closed early or failing does not give its connection back; one network thread unless the catalog sets the number. `FlussJniScannerPkTest` (1 new): the first snapshot file is swapped for a FIFO that holds the copy, the range is closed mid-copy and the snapshot directory must go (still there after 60 s without the fix, gone in 1.3 s with it). `SafeKvSnapshotAndLogBatchScannerTest` (3 new): the connection is released only once the snapshot reader is closed, at once if the snapshot had arrived, and the publication waiter outlasts an `OutOfMemoryError` on its own thread. BE `FileScanOperatorFlussTest.EveryInstanceOfTheNodeReadsThroughTheNodesCache` (new). On this branch: fluss-scanner 79 tests pass; the two tests added in review fail without their fix. BE: the file scan, file scanner, scanner scheduling, fluss union and iceberg reader suites (168 tests) pass. **Regression.** `external_table_p0/fluss`, 17 suites, pass against a local fluss 1.0.0 docker environment. **Manual.** The benchmarks above on a Release BE. - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> BE keeps idle fluss connections for up to 60 s and opens them with one network thread unless the catalog sets `fluss.netty.client.num-network-threads`; a failure to close a fluss connection is logged instead of failing the query; the scan node's reader cache is shared by its instances on a backend; new profile counter `FlussJniConnectionsOpened`. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 8a1ec48 commit 7ac0eb2

13 files changed

Lines changed: 1462 additions & 37 deletions

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 23 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include <algorithm>
2323
#include <memory>
2424

25+
#include "common/cast_set.h"
2526
#include "core/assert_cast.h"
2627
#include "core/data_type/data_type_array.h"
2728
#include "core/data_type/data_type_map.h"
@@ -60,6 +61,12 @@ bool contains_variant_type(const DataTypePtr& input) {
6061
}
6162
}
6263

64+
// The most file scanners one instance of a file scan node runs: the max_file_scanners_concurrency
65+
// session variable, 16 when it is not set.
66+
int file_scanners_per_instance(RuntimeState* state) {
67+
return state->max_file_scanners_concurrency() > 0 ? state->max_file_scanners_concurrency() : 16;
68+
}
69+
6370
} // namespace
6471

6572
PushDownType FileScanLocalState::_should_push_down_binary_predicate(
@@ -107,8 +114,7 @@ int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
107114
*
108115
* If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
109116
*/
110-
return (state->max_file_scanners_concurrency() > 0 ? state->max_file_scanners_concurrency()
111-
: 16) *
117+
return file_scanners_per_instance(state) *
112118
(state->query_parallel_instance_num() / _parent->parallelism(state));
113119
}
114120

@@ -198,12 +204,7 @@ Status FileScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
198204
}
199205

200206
auto& p = _parent->cast<FileScanOperatorX>();
201-
// There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance.
202-
uint32_t shard_num =
203-
std::min(ScannerScheduler::default_remote_scan_thread_num() / p.parallelism(state()),
204-
_max_scanners);
205-
shard_num = std::max(shard_num, 1U);
206-
_kv_cache = std::make_unique<ShardedKVCache>(shard_num);
207+
DORIS_CHECK(p._kv_cache != nullptr);
207208
const TFileScanRangeParams* scan_params = nullptr;
208209
if (state()->get_query_ctx() != nullptr &&
209210
state()->get_query_ctx()->file_scan_range_params_map.count(parent_id()) > 0) {
@@ -235,11 +236,11 @@ Status FileScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
235236
ScannerSPtr scanner;
236237
if (use_file_scanner_v2) {
237238
scanner = FileScannerV2::create_shared(state(), this, p._limit, _split_source,
238-
_scanner_profile.get(), _kv_cache.get(),
239+
_scanner_profile.get(), p._kv_cache.get(),
239240
&p._colname_to_slot_id);
240241
} else {
241242
scanner = FileScanner::create_shared(state(), this, p._limit, _split_source,
242-
_scanner_profile.get(), _kv_cache.get(),
243+
_scanner_profile.get(), p._kv_cache.get(),
243244
&p._colname_to_slot_id);
244245
}
245246
RETURN_IF_ERROR(scanner->init(state(), _conjuncts));
@@ -327,6 +328,18 @@ Status FileScanLocalState::_process_conjuncts(RuntimeState* state) {
327328

328329
Status FileScanOperatorX::prepare(RuntimeState* state) {
329330
RETURN_IF_ERROR(ScanOperatorX<FileScanLocalState>::prepare(state));
331+
// Sharded for the scanners that can reach the cache at once, which are now every instance's:
332+
// as many as the per-instance caches it replaces had between them. That is the fragment's
333+
// instances, not this operator's parallelism: a serial operator has one instance, and it runs
334+
// the scanners of all of them (max_scanners_concurrency). Counted in 64 bits: the scanners per
335+
// instance are a session variable with no upper bound, and in an int the product overflows
336+
// before std::min can cap it, leaving the whole node one shard to load under.
337+
const int64_t shard_num =
338+
std::min<int64_t>(ScannerScheduler::default_remote_scan_thread_num(),
339+
static_cast<int64_t>(state->query_parallel_instance_num()) *
340+
file_scanners_per_instance(state));
341+
_kv_cache =
342+
std::make_unique<ShardedKVCache>(cast_set<uint32_t>(std::max<int64_t>(shard_num, 1)));
330343
if (state->get_query_ctx() != nullptr &&
331344
state->get_query_ctx()->file_scan_range_params_map.contains(node_id())) {
332345
TFileScanRangeParams& params =

‎be/src/exec/operator/file_scan_operator.h‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,12 +86,6 @@ class FileScanLocalState final : public ScanLocalState<FileScanLocalState> {
8686
const std::set<std::string> fn_name) const override;
8787
std::shared_ptr<SplitSourceConnector> _split_source = nullptr;
8888
int _max_scanners;
89-
// A in memory cache to save some common components
90-
// of the this scan node. eg:
91-
// 1. iceberg delete file
92-
// 2. parquet file meta
93-
// KVCache<std::string> _kv_cache;
94-
std::unique_ptr<ShardedKVCache> _kv_cache;
9589
TupleId _output_tuple_id = -1;
9690
};
9791

@@ -119,6 +113,11 @@ class FileScanOperatorX final : public ScanOperatorX<FileScanLocalState> {
119113

120114
const std::string _table_name;
121115
bool _batch_split_mode = false;
116+
// What this scan node's readers parse once and reuse across splits: iceberg delete files,
117+
// deletion vectors, the keys of a fluss union read's log tails. One cache for every instance of
118+
// the node on this backend, so a split reuses what a split in another instance already read;
119+
// splits are dealt to instances with no regard for which delete file or bucket they share.
120+
std::unique_ptr<ShardedKVCache> _kv_cache;
122121
};
123122

124123
/// Instantiated once in scan_operator.cpp; suppresses per-TU implicit instantiation.

‎be/test/exec/operator/file_scan_operator_fluss_test.cpp‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,52 @@
1717

1818
#include <gtest/gtest.h>
1919

20+
#include <cstdint>
21+
#include <limits>
22+
#include <list>
23+
#include <map>
24+
#include <memory>
2025
#include <string>
26+
#include <utility>
27+
#include <vector>
2128

29+
#include "common/object_pool.h"
2230
#include "exec/operator/file_scan_operator.h"
31+
#include "exec/pipeline/dependency.h"
2332
#include "exec/scan/file_scanner_v2.h"
33+
#include "exec/scan/scanner.h"
34+
#include "exec/scan/scanner_scheduler.h"
2435
#include "gen_cpp/PlanNodes_types.h"
36+
#include "runtime/descriptors.h"
37+
#include "testutil/mock/mock_runtime_state.h"
2538

2639
namespace doris::pipeline {
2740
namespace {
2841

42+
// One tuple with no slots: all a file scan node needs to be initialized and prepared.
43+
Status create_one_tuple_descriptors(ObjectPool* pool, DescriptorTbl** descriptors) {
44+
TDescriptorTable thrift_descriptors;
45+
TTupleDescriptor tuple_descriptor;
46+
tuple_descriptor.id = 0;
47+
tuple_descriptor.byteSize = 0;
48+
tuple_descriptor.numNullBytes = 0;
49+
thrift_descriptors.tupleDescriptors.push_back(tuple_descriptor);
50+
return DescriptorTbl::create(pool, thrift_descriptors, descriptors);
51+
}
52+
53+
// A file scan node over that tuple.
54+
TPlanNode file_scan_plan_node() {
55+
TPlanNode plan_node;
56+
plan_node.node_id = 0;
57+
plan_node.node_type = TPlanNodeType::FILE_SCAN_NODE;
58+
plan_node.num_children = 0;
59+
plan_node.limit = -1;
60+
plan_node.row_tuples.push_back(0);
61+
plan_node.file_scan_node.tuple_id = 0;
62+
plan_node.__isset.file_scan_node = true;
63+
return plan_node;
64+
}
65+
2966
TFileScanRangeParams scan_level_format(const std::string& table_format) {
3067
TFileScanRangeParams params;
3168
if (!table_format.empty()) {
@@ -115,4 +152,75 @@ TEST(FileScanOperatorFlussTest, ScannerV2SupportsEveryRangeKindOfAForcedNode) {
115152
EXPECT_TRUE(FileScannerV2::is_supported(params, lake_native));
116153
}
117154

155+
// Every instance of a file scan node on a backend reads through the node's one cache. A primary-key
156+
// union read keeps the keys of each bucket's log tail there, as iceberg keeps its parsed delete
157+
// files, and splits are dealt to instances with no regard for the bucket or delete file they share.
158+
// With a cache per instance, each instance holding a split of a bucket read that bucket's tail again
159+
// over JNI: 69 reads where 16 would do on a 16-bucket table spread over 9 instances.
160+
TEST(FileScanOperatorFlussTest, EveryInstanceOfTheNodeReadsThroughTheNodesCache) {
161+
ObjectPool pool;
162+
DescriptorTbl* descriptors = nullptr;
163+
ASSERT_TRUE(create_one_tuple_descriptors(&pool, &descriptors).ok());
164+
MockRuntimeState state;
165+
state.set_desc_tbl(descriptors);
166+
167+
const TPlanNode plan_node = file_scan_plan_node();
168+
FileScanOperatorX node(&pool, plan_node, 0, *descriptors, /*parallel_tasks=*/1);
169+
ASSERT_TRUE(node.init(plan_node, &state).ok());
170+
ASSERT_TRUE(node.prepare(&state).ok());
171+
ASSERT_NE(node._kv_cache, nullptr);
172+
173+
TFileRangeDesc log_range = jni_range("fluss");
174+
log_range.table_format_params.__set_fluss_params({{"fluss.range_type", "LOG"}});
175+
TFileScanRange file_scan_range;
176+
file_scan_range.__set_params(scan_level_format("fluss"));
177+
// A query reads no source tuple; tuple 0 there would make this scan a load.
178+
file_scan_range.params.__set_src_tuple_id(1);
179+
file_scan_range.ranges.push_back(log_range);
180+
TScanRangeParams scan_range;
181+
scan_range.scan_range.ext_scan_range.__set_file_scan_range(file_scan_range);
182+
scan_range.scan_range.__isset.ext_scan_range = true;
183+
const std::vector<TScanRangeParams> scan_ranges {scan_range};
184+
185+
RuntimeProfile profile("EveryInstanceOfTheNodeReadsThroughTheNodesCache");
186+
const std::map<int, std::pair<std::shared_ptr<BasicSharedState>,
187+
std::vector<std::shared_ptr<Dependency>>>>
188+
shared_state_map;
189+
for (int instance = 0; instance < 2; ++instance) {
190+
auto local_state = FileScanLocalState::create_unique(&state, &node);
191+
LocalStateInfo info {&profile, scan_ranges, nullptr, shared_state_map, instance};
192+
ASSERT_TRUE(local_state->init(&state, info).ok());
193+
std::list<ScannerSPtr> scanners;
194+
ASSERT_TRUE(local_state->_init_scanners(&scanners).ok());
195+
ASSERT_FALSE(scanners.empty());
196+
for (const auto& scanner : scanners) {
197+
const auto* file_scanner = dynamic_cast<FileScannerV2*>(scanner.get());
198+
ASSERT_NE(file_scanner, nullptr);
199+
EXPECT_EQ(file_scanner->_kv_cache, node._kv_cache.get()) << "instance " << instance;
200+
}
201+
}
202+
}
203+
204+
// The node's cache is sharded for the scanners of all its instances: the instance count times
205+
// max_file_scanners_concurrency, a session variable with no upper bound. A user may set it to
206+
// INT32_MAX to mean "no limit", and with two instances that product overflows an int, which would
207+
// leave the node a single shard, behind whose lock every scanner of every instance loads its delete
208+
// files one at a time. The count has to come out capped by the remote scan threads instead.
209+
TEST(FileScanOperatorFlussTest, CacheShardsOfAnUnlimitedScannerSettingAreCappedNotOverflowed) {
210+
ObjectPool pool;
211+
DescriptorTbl* descriptors = nullptr;
212+
ASSERT_TRUE(create_one_tuple_descriptors(&pool, &descriptors).ok());
213+
MockRuntimeState state;
214+
state.set_desc_tbl(descriptors);
215+
state._query_options.__set_parallel_instance(2);
216+
state._query_options.__set_max_file_scanners_concurrency(std::numeric_limits<int32_t>::max());
217+
218+
const TPlanNode plan_node = file_scan_plan_node();
219+
FileScanOperatorX node(&pool, plan_node, 0, *descriptors, /*parallel_tasks=*/2);
220+
ASSERT_TRUE(node.init(plan_node, &state).ok());
221+
ASSERT_TRUE(node.prepare(&state).ok());
222+
EXPECT_EQ(node._kv_cache->_num_shards,
223+
static_cast<uint32_t>(ScannerScheduler::default_remote_scan_thread_num()));
224+
}
225+
118226
} // namespace doris::pipeline
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.fluss;
19+
20+
import org.apache.fluss.client.Connection;
21+
import org.slf4j.Logger;
22+
import org.slf4j.LoggerFactory;
23+
24+
import java.util.concurrent.ExecutorService;
25+
import java.util.concurrent.SynchronousQueue;
26+
import java.util.concurrent.ThreadPoolExecutor;
27+
import java.util.concurrent.TimeUnit;
28+
import java.util.concurrent.atomic.AtomicInteger;
29+
import java.util.concurrent.atomic.AtomicLong;
30+
31+
/**
32+
* Closes fluss connections without making the caller wait for them.
33+
*
34+
* <p>Closing a {@link Connection} shuts its netty client down with netty's default graceful period,
35+
* which returns only after two quiet seconds, and fluss offers no shorter one. Scan ranges no longer
36+
* close connections - they give them back to {@link FlussConnectionPool} - but the pool closes the
37+
* ones nobody has borrowed for a while, and one sweep can expire every connection a burst of ranges
38+
* opened. Closed one after another they would hold that sweep up for two seconds each, so the pool
39+
* hands them here and moves on.
40+
*
41+
* <p><b>Bounded.</b> A connection that is closing still holds its client threads (three netty threads
42+
* each) until the quiet period passes, so at most {@link #MAX_CLOSING} close here at a time; past that
43+
* the caller closes its own, which only slows the sweep that brought it.
44+
*/
45+
final class FlussConnectionCloser {
46+
47+
private static final Logger LOG = LoggerFactory.getLogger(FlussConnectionCloser.class);
48+
49+
/** Connections closing in the background at once; each is one thread here and three of its own. */
50+
static final int MAX_CLOSING = 256;
51+
52+
/** Scanners closing inline again is worth a line in the log, not one per connection. */
53+
private static final long SATURATION_LOG_INTERVAL_NANOS = TimeUnit.MINUTES.toNanos(1);
54+
55+
private static final AtomicInteger THREAD_COUNTER = new AtomicInteger();
56+
57+
private static final AtomicLong LAST_SATURATION_LOG_NANOS =
58+
new AtomicLong(System.nanoTime() - SATURATION_LOG_INTERVAL_NANOS);
59+
60+
/**
61+
* One thread per closing connection and no queue: a queued connection would hold its threads
62+
* while waiting for a turn, which is the pile-up the bound exists to prevent. A connection that
63+
* finds every thread busy is closed by the thread that brought it.
64+
*/
65+
private static final ExecutorService CLOSER = new ThreadPoolExecutor(
66+
0, MAX_CLOSING, 60L, TimeUnit.SECONDS, new SynchronousQueue<>(),
67+
runnable -> {
68+
Thread thread = new Thread(runnable,
69+
"fluss-connection-closer-" + THREAD_COUNTER.incrementAndGet());
70+
// Never what keeps BE's JVM alive, and never worth waiting for at shutdown.
71+
thread.setDaemon(true);
72+
// Shutting the client down can still load fluss classes; like JniScanner does around
73+
// open and close, run it under the loader of the plugin that can see them.
74+
thread.setContextClassLoader(FlussConnectionCloser.class.getClassLoader());
75+
return thread;
76+
},
77+
(close, executor) -> closeOnCallingThread(close));
78+
79+
private FlussConnectionCloser() {
80+
}
81+
82+
/**
83+
* Every closer thread is busy, so the thread that brought the connection closes it, two seconds
84+
* a connection, and says so in the log: nothing else would show it.
85+
*/
86+
private static void closeOnCallingThread(Runnable close) {
87+
long now = System.nanoTime();
88+
long last = LAST_SATURATION_LOG_NANOS.get();
89+
if (now - last >= SATURATION_LOG_INTERVAL_NANOS && LAST_SATURATION_LOG_NANOS.compareAndSet(last, now)) {
90+
LOG.info("{} fluss connections are closing in the background; further connections are closed "
91+
+ "by the thread that brings them, waiting about two seconds for each, until some of "
92+
+ "those are done", MAX_CLOSING);
93+
}
94+
close.run();
95+
}
96+
97+
/**
98+
* Closes {@code connection} without making the caller wait for it, unless {@link #MAX_CLOSING}
99+
* are closing already. A failure to close is logged and nothing else: the scans the connection
100+
* served have already returned their rows, and nothing may fail over the cleanup of one of their
101+
* connections.
102+
*
103+
* <p>Nothing else holds a connection by the time it gets here - the pool and the range have let go of
104+
* it - so one that cannot be handed to a closer thread is closed by the caller. Handing it over
105+
* allocates the task and perhaps a thread, which fails with an {@code OutOfMemoryError} while BE's JVM
106+
* heap is full or no native thread is to be had; dropped there, the connection would keep its threads
107+
* and sockets for the life of the process.
108+
*/
109+
static void close(Connection connection) {
110+
try {
111+
CLOSER.execute(() -> closeQuietly(connection));
112+
} catch (RuntimeException | Error handoffFailure) {
113+
closeQuietly(connection);
114+
}
115+
}
116+
117+
private static void closeQuietly(Connection connection) {
118+
try {
119+
connection.close();
120+
} catch (Exception e) {
121+
LOG.warn("Failed to close a fluss connection", e);
122+
}
123+
}
124+
}

0 commit comments

Comments
 (0)