Skip to content

Commit 222bca5

Browse files
morningmanclaude
andcommitted
[improvement](be) Share a file scan node's reader cache across its instances
### What problem does this PR solve? Issue Number: None Related PR: #66399 Problem Summary: Readers of a file scan node keep what they parse once per node in a `ShardedKVCache`: iceberg position and equality delete files, deletion vectors, and, for a fluss union read of a primary-key table, the keys of each bucket's log tail. Each `FileScanLocalState` built a cache of its own, so the cache was per pipeline instance. Splits are dealt to the instances with no regard for the delete file or bucket they share, so every instance that got a split of the same bucket or delete file parsed it again and kept its own copy. For a fluss union read each such parse is a JNI read of the bucket's log tail. On a 16-bucket primary-key table whose lake has 80 splits, spread over 9 instances, `COUNT(*)` read the tails 69 times where 16 would do (`FlussUnionTailCacheHitCount` = 11 of 80 splits) and kept 6.6M keys in the JVM heap for 1.6M distinct ones. The fix moves the cache to `FileScanOperatorX`, created in `prepare()`, so every instance of the node on a backend reads through one cache. This is safe to share: - every cached value is plain data (roaring bitmaps, delete-row vectors, key blocks with an immutable hash index) and none refers to an instance's profile, IO context or runtime state; the readers that fill them use such objects only inside the creating function; - the cache locks per shard and was already shared by the scanners of one instance; - nothing erases entries, the operator outlives every local state and scanner, and the fragment releases operators under the query memory tracker. The shard count is what the per-instance caches had between them: the fragment's instance count times the per-instance scanner limit, capped by the remote scan thread pool. It is the fragment's instances and not the operator's parallelism because a serial operator has one instance that runs the scanners of all of them (`max_scanners_concurrency`). Measured on a Release BE against a local fluss cluster: a 16-bucket primary-key table of 30M rows, all of them in the lake, and 1.6M keys (100K per bucket) updated since, in the log tail. Before is master; after has this commit and the background connection close of the first commit of this series, which takes a fixed two seconds off each of these tail reads: - Compacted lake (80 native splits), `COUNT(*)` under union read: 21.9 s -> 1.0 s. Splits that found the tail keys already cached 11 -> 61 of 80; keys held in the JVM heap 6.6M -> 1.6M; time in tail reads summed over the scanners 139.8 s -> 0.9 s. - Uncompacted lake (32 JNI merge splits), `COUNT(*)`: 13.8 s -> 1.6 s; cached 0 -> 16 of 32; keys held 3.2M -> 1.6M. - Union read, union read disabled and the expected values agree on `COUNT(*)`, `SUM(id)` and `SUM(price)` for both tables, before and after. ### Release note Scans of iceberg tables with delete files, of tables with deletion vectors, and fluss union reads of primary-key tables no longer parse the same delete file or log tail once per pipeline instance. ### Check List (For Author) - Test: Unit Test, Manual test - BE UT `FileScanOperatorFlussTest.EveryInstanceOfTheNodeReadsThroughTheNodesCache` (new) - 168 related BE unit tests pass (`FileScanOperator*`, `FileScannerV2Test.*`, `VfileScannerExceptionTest.*`, `ScannerContextTest.*`, `*FlussUnion*`, `*IcebergReader*`) - Manual: the benchmark above - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 38b689a commit 222bca5

3 files changed

Lines changed: 96 additions & 16 deletions

File tree

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include <algorithm>
2323
#include <memory>
2424

25+
#include "common/cast_set.h"
2526
#include "core/assert_cast.h"
2627
#include "core/data_type/data_type_array.h"
2728
#include "core/data_type/data_type_map.h"
@@ -60,6 +61,12 @@ bool contains_variant_type(const DataTypePtr& input) {
6061
}
6162
}
6263

64+
// The most file scanners one instance of a file scan node runs: the max_file_scanners_concurrency
65+
// session variable, 16 when it is not set.
66+
int file_scanners_per_instance(RuntimeState* state) {
67+
return state->max_file_scanners_concurrency() > 0 ? state->max_file_scanners_concurrency() : 16;
68+
}
69+
6370
} // namespace
6471

6572
PushDownType FileScanLocalState::_should_push_down_binary_predicate(
@@ -107,8 +114,7 @@ int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
107114
*
108115
* If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
109116
*/
110-
return (state->max_file_scanners_concurrency() > 0 ? state->max_file_scanners_concurrency()
111-
: 16) *
117+
return file_scanners_per_instance(state) *
112118
(state->query_parallel_instance_num() / _parent->parallelism(state));
113119
}
114120

@@ -198,12 +204,7 @@ Status FileScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
198204
}
199205

200206
auto& p = _parent->cast<FileScanOperatorX>();
201-
// There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance.
202-
uint32_t shard_num =
203-
std::min(ScannerScheduler::default_remote_scan_thread_num() / p.parallelism(state()),
204-
_max_scanners);
205-
shard_num = std::max(shard_num, 1U);
206-
_kv_cache = std::make_unique<ShardedKVCache>(shard_num);
207+
DORIS_CHECK(p._kv_cache != nullptr);
207208
const TFileScanRangeParams* scan_params = nullptr;
208209
if (state()->get_query_ctx() != nullptr &&
209210
state()->get_query_ctx()->file_scan_range_params_map.count(parent_id()) > 0) {
@@ -235,11 +236,11 @@ Status FileScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
235236
ScannerSPtr scanner;
236237
if (use_file_scanner_v2) {
237238
scanner = FileScannerV2::create_shared(state(), this, p._limit, _split_source,
238-
_scanner_profile.get(), _kv_cache.get(),
239+
_scanner_profile.get(), p._kv_cache.get(),
239240
&p._colname_to_slot_id);
240241
} else {
241242
scanner = FileScanner::create_shared(state(), this, p._limit, _split_source,
242-
_scanner_profile.get(), _kv_cache.get(),
243+
_scanner_profile.get(), p._kv_cache.get(),
243244
&p._colname_to_slot_id);
244245
}
245246
RETURN_IF_ERROR(scanner->init(state(), _conjuncts));
@@ -327,6 +328,14 @@ Status FileScanLocalState::_process_conjuncts(RuntimeState* state) {
327328

328329
Status FileScanOperatorX::prepare(RuntimeState* state) {
329330
RETURN_IF_ERROR(ScanOperatorX<FileScanLocalState>::prepare(state));
331+
// Sharded for the scanners that can reach the cache at once, which are now every instance's:
332+
// as many as the per-instance caches it replaces had between them. That is the fragment's
333+
// instances, not this operator's parallelism: a serial operator has one instance, and it runs
334+
// the scanners of all of them (max_scanners_concurrency).
335+
const int shard_num =
336+
std::min(ScannerScheduler::default_remote_scan_thread_num(),
337+
state->query_parallel_instance_num() * file_scanners_per_instance(state));
338+
_kv_cache = std::make_unique<ShardedKVCache>(cast_set<uint32_t>(std::max(shard_num, 1)));
330339
if (state->get_query_ctx() != nullptr &&
331340
state->get_query_ctx()->file_scan_range_params_map.contains(node_id())) {
332341
TFileScanRangeParams& params =

‎be/src/exec/operator/file_scan_operator.h‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,12 +86,6 @@ class FileScanLocalState final : public ScanLocalState<FileScanLocalState> {
8686
const std::set<std::string> fn_name) const override;
8787
std::shared_ptr<SplitSourceConnector> _split_source = nullptr;
8888
int _max_scanners;
89-
// A in memory cache to save some common components
90-
// of the this scan node. eg:
91-
// 1. iceberg delete file
92-
// 2. parquet file meta
93-
// KVCache<std::string> _kv_cache;
94-
std::unique_ptr<ShardedKVCache> _kv_cache;
9589
TupleId _output_tuple_id = -1;
9690
};
9791

@@ -119,6 +113,11 @@ class FileScanOperatorX final : public ScanOperatorX<FileScanLocalState> {
119113

120114
const std::string _table_name;
121115
bool _batch_split_mode = false;
116+
// What this scan node's readers parse once and reuse across splits: iceberg delete files,
117+
// deletion vectors, the keys of a fluss union read's log tails. One cache for every instance of
118+
// the node on this backend, so a split reuses what a split in another instance already read;
119+
// splits are dealt to instances with no regard for which delete file or bucket they share.
120+
std::unique_ptr<ShardedKVCache> _kv_cache;
122121
};
123122

124123
/// Instantiated once in scan_operator.cpp; suppresses per-TU implicit instantiation.

‎be/test/exec/operator/file_scan_operator_fluss_test.cpp‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,21 @@
1717

1818
#include <gtest/gtest.h>
1919

20+
#include <list>
21+
#include <map>
22+
#include <memory>
2023
#include <string>
24+
#include <utility>
25+
#include <vector>
2126

27+
#include "common/object_pool.h"
2228
#include "exec/operator/file_scan_operator.h"
29+
#include "exec/pipeline/dependency.h"
2330
#include "exec/scan/file_scanner_v2.h"
31+
#include "exec/scan/scanner.h"
2432
#include "gen_cpp/PlanNodes_types.h"
33+
#include "runtime/descriptors.h"
34+
#include "testutil/mock/mock_runtime_state.h"
2535

2636
namespace doris::pipeline {
2737
namespace {
@@ -115,4 +125,66 @@ TEST(FileScanOperatorFlussTest, ScannerV2SupportsEveryRangeKindOfAForcedNode) {
115125
EXPECT_TRUE(FileScannerV2::is_supported(params, lake_native));
116126
}
117127

128+
// Every instance of a file scan node on a backend reads through the node's one cache. A primary-key
129+
// union read keeps the keys of each bucket's log tail there, as iceberg keeps its parsed delete
130+
// files, and splits are dealt to instances with no regard for the bucket or delete file they share.
131+
// With a cache per instance, each instance holding a split of a bucket read that bucket's tail again
132+
// over JNI: 69 reads where 16 would do on a 16-bucket table spread over 9 instances.
133+
TEST(FileScanOperatorFlussTest, EveryInstanceOfTheNodeReadsThroughTheNodesCache) {
134+
ObjectPool pool;
135+
TDescriptorTable thrift_descriptors;
136+
TTupleDescriptor tuple_descriptor;
137+
tuple_descriptor.id = 0;
138+
tuple_descriptor.byteSize = 0;
139+
tuple_descriptor.numNullBytes = 0;
140+
thrift_descriptors.tupleDescriptors.push_back(tuple_descriptor);
141+
DescriptorTbl* descriptors = nullptr;
142+
ASSERT_TRUE(DescriptorTbl::create(&pool, thrift_descriptors, &descriptors).ok());
143+
MockRuntimeState state;
144+
state.set_desc_tbl(descriptors);
145+
146+
TPlanNode plan_node;
147+
plan_node.node_id = 0;
148+
plan_node.node_type = TPlanNodeType::FILE_SCAN_NODE;
149+
plan_node.num_children = 0;
150+
plan_node.limit = -1;
151+
plan_node.row_tuples.push_back(0);
152+
plan_node.file_scan_node.tuple_id = 0;
153+
plan_node.__isset.file_scan_node = true;
154+
FileScanOperatorX node(&pool, plan_node, 0, *descriptors, /*parallel_tasks=*/1);
155+
ASSERT_TRUE(node.init(plan_node, &state).ok());
156+
ASSERT_TRUE(node.prepare(&state).ok());
157+
ASSERT_NE(node._kv_cache, nullptr);
158+
159+
TFileRangeDesc log_range = jni_range("fluss");
160+
log_range.table_format_params.__set_fluss_params({{"fluss.range_type", "LOG"}});
161+
TFileScanRange file_scan_range;
162+
file_scan_range.__set_params(scan_level_format("fluss"));
163+
// A query reads no source tuple; tuple 0 there would make this scan a load.
164+
file_scan_range.params.__set_src_tuple_id(1);
165+
file_scan_range.ranges.push_back(log_range);
166+
TScanRangeParams scan_range;
167+
scan_range.scan_range.ext_scan_range.__set_file_scan_range(file_scan_range);
168+
scan_range.scan_range.__isset.ext_scan_range = true;
169+
const std::vector<TScanRangeParams> scan_ranges {scan_range};
170+
171+
RuntimeProfile profile("EveryInstanceOfTheNodeReadsThroughTheNodesCache");
172+
const std::map<int, std::pair<std::shared_ptr<BasicSharedState>,
173+
std::vector<std::shared_ptr<Dependency>>>>
174+
shared_state_map;
175+
for (int instance = 0; instance < 2; ++instance) {
176+
auto local_state = FileScanLocalState::create_unique(&state, &node);
177+
LocalStateInfo info {&profile, scan_ranges, nullptr, shared_state_map, instance};
178+
ASSERT_TRUE(local_state->init(&state, info).ok());
179+
std::list<ScannerSPtr> scanners;
180+
ASSERT_TRUE(local_state->_init_scanners(&scanners).ok());
181+
ASSERT_FALSE(scanners.empty());
182+
for (const auto& scanner : scanners) {
183+
const auto* file_scanner = dynamic_cast<FileScannerV2*>(scanner.get());
184+
ASSERT_NE(file_scanner, nullptr);
185+
EXPECT_EQ(file_scanner->_kv_cache, node._kv_cache.get()) << "instance " << instance;
186+
}
187+
}
188+
}
189+
118190
} // namespace doris::pipeline

0 commit comments

Comments
 (0)