Skip to content

Commit 73c481f

Browse files
authored
[feature](cloud) Support file cache write index only (apache#64995)
### Purpose This PR adds a new BE config, `enable_file_cache_write_index_file_only`, to make cloud rowset writes and compaction outputs write only index-related content into file cache. This avoids actively putting segment data into file cache during write, reducing cache pollution from large data pages. The main target scenario is a limited local cache where query performance should prioritize indexes and segment metadata. With this mode enabled, the write path protects inverted index files and segment index/footer ranges, while normal data pages are still loaded and cached by the query read path when needed. ### High-Level Changes - Add BE config `enable_file_cache_write_index_file_only`, default `false`. - Make rowset/segment file-cache write decisions file-type aware: - `SEGMENT_FILE` does not create a full-file cache builder and does not allow adaptive write in index-only mode. - `INVERTED_INDEX_FILE` is still written into file cache through the direct write path. - segment index/footer ranges are preloaded into file cache by `SegmentIndexFileCacheLoader` after segment close. - Update packed file handling to honor the original file type when deciding whether to write file cache. - Add profile statistics for segment footer/index reads and separate them from data-page and inverted-index reads. - Replace correlated boolean parameters in `CachedRemoteFileReader::_update_stats` with a read-type enum: - data page - inverted index - segment footer/index - Add BE UTs and cloud regressions for both normal file and packed file paths. ### Behavior When `enable_file_cache_write_index_file_only=true`: - Segment data is not actively written into file cache during write, flush, or compaction output. - Segment index/footer ranges are synchronously preloaded into file cache after segment close. - Independent inverted index files are written into file cache through the direct write path. - Query-side data-page reads keep the existing behavior. The query read path may still fill cache according to current rules. - Segment open, footer/index access, and inverted index access can hit the index-related content preloaded by the write path. - Query profile exposes separate cache/remote read statistics for data pages, inverted indexes, and segment footer/index reads. ### Interaction With Existing Configs The write-side file-cache behavior is decided in this order: 1. `enable_file_cache=false` is the highest-priority global switch. - All file-cache writes are disabled. - This includes independent inverted index direct cache and segment index/footer preload. 2. `enable_file_cache=true` and `enable_file_cache_write_index_file_only=true` enables global forced index-only mode. - Segment data does not create a full-file cache builder. - Segment data cannot be written by adaptive write. - Segment index/footer ranges are written into file cache by `SegmentIndexFileCacheLoader` after segment close. - Independent inverted index files are written through the `INVERTED_INDEX_FILE` direct write path. - This config has higher priority than `enable_file_cache_write_base_compaction_index_only` and `enable_file_cache_write_cumu_compaction_index_only`. - These index writes do not depend on `enable_file_cache_adaptive_write`, compaction keep configs, hit-ratio thresholds, or request/session `write_file_cache`. 3. `enable_file_cache=true`, `enable_file_cache_write_index_file_only=false`, and a compaction-level index-only config is enabled applies index-only behavior only to the matching compaction output. - `enable_file_cache_write_base_compaction_index_only=true` only forces base compaction output to use index-only file-cache writes. - `enable_file_cache_write_cumu_compaction_index_only=true` only forces cumulative compaction output to use index-only file-cache writes. - These two configs do not affect load, normal flush, schema change output, or query read-path cache fill. - Compaction types not matched by these configs continue to use existing `write_file_cache`, adaptive write, compaction keep, and hit-ratio decisions. 4. `enable_file_cache=true` and all three index-only configs are `false` preserves existing behavior. - `write_file_cache`, `enable_file_cache_adaptive_write`, compaction keep configs, schema-change hit ratios, and existing rowset/file-writer decisions continue to control whether cache is written. #### With `enable_file_cache_adaptive_write` - When index-only is disabled, `enable_file_cache_adaptive_write=true` keeps the existing adaptive write behavior. Segment data may be written into file cache if the cache-space check passes. - When index-only is enabled, adaptive write no longer participates in segment data decisions. - Even if `enable_file_cache_adaptive_write=true`, segment data is still not written into file cache, while segment index/footer and independent inverted index files are still written. #### With Compaction Output Keep Configs Related configs include: - `enable_file_cache_keep_base_compaction_output` - `file_cache_keep_base_compaction_output_min_hit_ratio` - compaction output hit-ratio checks When index-only is disabled, existing compaction output cache behavior is preserved. When index-only is enabled, compaction output is forced into index-only behavior: - Compaction output segment data is not actively written into file cache. - Segment index/footer ranges are synchronously written into file cache. - Independent inverted index files are direct-cached. - Compaction keep configs and hit-ratio thresholds no longer decide whether index content is written into file cache. #### With Base/Cumulative Compaction Index-Only Configs Related configs include: - `enable_file_cache_write_base_compaction_index_only` - `enable_file_cache_write_cumu_compaction_index_only` These two configs are narrower compaction-output policies and only take effect when `enable_file_cache_write_index_file_only=false`. - When `enable_file_cache_write_base_compaction_index_only=true`, base compaction output uses index-only behavior: output segment data is not actively cached, while output segment index/footer ranges and independent inverted index files are written into file cache. - When `enable_file_cache_write_cumu_compaction_index_only=true`, cumulative compaction output uses index-only behavior: output segment data is not actively cached, while output segment index/footer ranges and independent inverted index files are written into file cache. - If only one of the two configs is enabled, the other compaction type keeps the existing cache-write decision flow. - If `enable_file_cache_write_index_file_only=true`, the global index-only config takes precedence and the base/cumulative compaction-specific configs no longer change the final behavior. - `enable_file_cache=false` is still the highest-priority switch and disables all of these file-cache writes. #### With Schema Change Output Thresholds - When index-only is disabled, `file_cache_keep_schema_change_output_min_hit_ratio` keeps its existing semantics. - When index-only is enabled, schema change output also follows index-only behavior: - segment data is not actively written into file cache; - index-related content is written into file cache; - schema-change hit-ratio thresholds no longer decide whether index content is cached. #### With `disable_file_cache` / Request `write_file_cache` - `enable_file_cache=false` disables everything and has the highest priority. - When index-only is enabled, the BE config has higher priority than request/session `write_file_cache=false`. - Even if session `disable_file_cache=true` makes request-side `write_file_cache=false`, as long as BE `enable_file_cache=true` and index-only is enabled: - index-related content is still written into file cache; - segment data is still not written into file cache. - When index-only is disabled, `disable_file_cache=true` preserves existing behavior and load output does not actively write file cache. #### With Packed File - When index-only is disabled, packed small files keep the existing whole-file cache behavior. - When index-only is enabled: - packed small segment files must not be whole-file cached, preventing segment data from entering file cache; - segment index/footer ranges are written into file cache by `SegmentIndexFileCacheLoader`; - packed independent index small files are independent index files and can be whole-file cached. #### With `enable_flush_file_cache_async` - Segment index/footer ranges are written through synchronous dry-run reads and do not depend on write-path `enable_flush_file_cache_async`. - Independent inverted index files still use the direct write path, so they continue to follow existing async flush behavior. #### With `index_percent` in `file_cache_path` `SegmentIndexFileCacheLoader` marks index-related reads as index data so that they go to the `FileCacheType::INDEX` queue when possible. After enabling index-only mode, `index_percent` should be reviewed based on index size. The default `index_percent=5` may be insufficient for large indexes. ### Behavior Matrix | Scenario | index-only=false | index-only=true | | --- | --- | --- | | load segment data | follows existing cache policy | not actively cached | | load segment index/footer | depends on existing whole-file cache or read path | synchronously range-read into file cache after segment close | | load independent inverted index file | follows existing cache policy | direct-cached by write path | | cumulative compaction segment data | may be cached by existing behavior | not actively cached | | cumulative compaction segment index/footer | may be cached by existing behavior | synchronously range-read into file cache after segment close | | base compaction force-keep output | whole output rowset may be cached | only output segment index/footer is preloaded | | adaptive write has enough space | segment data may be cached | segment data is still not cached | | packed small index file | may be cached | can be whole-file cached | | packed small segment file | may be cached | not whole-file cached; internal index/footer is preloaded | | query read fill cache | unchanged | unchanged | | event-driven warm-up | unchanged | unchanged | ### Release note Add BE config `enable_file_cache_write_index_file_only` to support index-only file-cache writes for cloud rowset writes and compaction outputs.
1 parent 4f8c144 commit 73c481f

31 files changed

Lines changed: 2333 additions & 70 deletions

‎be/src/cloud/config.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@ DEFINE_mBool(enable_warmup_immediately_on_new_rowset, "false");
131131

132132
// Packed file manager config
133133
DEFINE_mBool(enable_packed_file, "true");
134+
DEFINE_mBool(enable_file_cache_write_index_file_only, "false");
134135
DEFINE_mInt64(packed_file_size_threshold_bytes, "5242880"); // 5MB
135136
DEFINE_mInt64(packed_file_time_threshold_ms, "100"); // 100ms
136137
DEFINE_mInt64(packed_file_try_lock_timeout_ms, "5"); // 5ms

‎be/src/cloud/config.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,7 @@ DECLARE_mBool(enable_warmup_immediately_on_new_rowset);
175175

176176
// Packed file manager config
177177
DECLARE_mBool(enable_packed_file);
178+
DECLARE_mBool(enable_file_cache_write_index_file_only);
178179
DECLARE_mInt64(packed_file_size_threshold_bytes);
179180
DECLARE_mInt64(packed_file_time_threshold_ms);
180181
DECLARE_mInt64(packed_file_try_lock_timeout_ms);

‎be/src/io/cache/block_file_cache_profile.cpp‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,16 @@ FileCacheStatistics diff_file_cache_statistics(const FileCacheStatistics& curren
9898
SUBTRACT_FIELD(inverted_index_remote_io_timer);
9999
SUBTRACT_FIELD(inverted_index_peer_io_timer);
100100
SUBTRACT_FIELD(inverted_index_io_timer);
101+
102+
SUBTRACT_FIELD(segment_footer_index_num_local_io_total);
103+
SUBTRACT_FIELD(segment_footer_index_num_remote_io_total);
104+
SUBTRACT_FIELD(segment_footer_index_num_peer_io_total);
105+
SUBTRACT_FIELD(segment_footer_index_bytes_read_from_local);
106+
SUBTRACT_FIELD(segment_footer_index_bytes_read_from_remote);
107+
SUBTRACT_FIELD(segment_footer_index_bytes_read_from_peer);
108+
SUBTRACT_FIELD(segment_footer_index_local_io_timer);
109+
SUBTRACT_FIELD(segment_footer_index_remote_io_timer);
110+
SUBTRACT_FIELD(segment_footer_index_peer_io_timer);
101111
#undef SUBTRACT_FIELD
102112
return diff;
103113
}
@@ -156,6 +166,25 @@ FileCacheProfileReporter::FileCacheProfileReporter(RuntimeProfile* profile) {
156166
ADD_CHILD_TIMER_WITH_LEVEL(profile, "InvertedIndexPeerIOUseTimer", cache_profile, 1);
157167
inverted_index_io_timer =
158168
ADD_CHILD_TIMER_WITH_LEVEL(profile, "InvertedIndexIOTimer", cache_profile, 1);
169+
170+
segment_footer_index_num_local_io_total = ADD_CHILD_COUNTER_WITH_LEVEL(
171+
profile, "SegmentFooterIndexNumLocalIOTotal", TUnit::UNIT, cache_profile, 1);
172+
segment_footer_index_num_remote_io_total = ADD_CHILD_COUNTER_WITH_LEVEL(
173+
profile, "SegmentFooterIndexNumRemoteIOTotal", TUnit::UNIT, cache_profile, 1);
174+
segment_footer_index_num_peer_io_total = ADD_CHILD_COUNTER_WITH_LEVEL(
175+
profile, "SegmentFooterIndexNumPeerIOTotal", TUnit::UNIT, cache_profile, 1);
176+
segment_footer_index_bytes_scanned_from_cache = ADD_CHILD_COUNTER_WITH_LEVEL(
177+
profile, "SegmentFooterIndexBytesScannedFromCache", TUnit::BYTES, cache_profile, 1);
178+
segment_footer_index_bytes_scanned_from_remote = ADD_CHILD_COUNTER_WITH_LEVEL(
179+
profile, "SegmentFooterIndexBytesScannedFromRemote", TUnit::BYTES, cache_profile, 1);
180+
segment_footer_index_bytes_scanned_from_peer = ADD_CHILD_COUNTER_WITH_LEVEL(
181+
profile, "SegmentFooterIndexBytesScannedFromPeer", TUnit::BYTES, cache_profile, 1);
182+
segment_footer_index_local_io_timer = ADD_CHILD_TIMER_WITH_LEVEL(
183+
profile, "SegmentFooterIndexLocalIOUseTimer", cache_profile, 1);
184+
segment_footer_index_remote_io_timer = ADD_CHILD_TIMER_WITH_LEVEL(
185+
profile, "SegmentFooterIndexRemoteIOUseTimer", cache_profile, 1);
186+
segment_footer_index_peer_io_timer = ADD_CHILD_TIMER_WITH_LEVEL(
187+
profile, "SegmentFooterIndexPeerIOUseTimer", cache_profile, 1);
159188
}
160189

161190
void FileCacheProfileReporter::update(const FileCacheStatistics* statistics) const {
@@ -193,6 +222,25 @@ void FileCacheProfileReporter::update(const FileCacheStatistics* statistics) con
193222
COUNTER_UPDATE(inverted_index_remote_io_timer, statistics->inverted_index_remote_io_timer);
194223
COUNTER_UPDATE(inverted_index_peer_io_timer, statistics->inverted_index_peer_io_timer);
195224
COUNTER_UPDATE(inverted_index_io_timer, statistics->inverted_index_io_timer);
225+
226+
COUNTER_UPDATE(segment_footer_index_num_local_io_total,
227+
statistics->segment_footer_index_num_local_io_total);
228+
COUNTER_UPDATE(segment_footer_index_num_remote_io_total,
229+
statistics->segment_footer_index_num_remote_io_total);
230+
COUNTER_UPDATE(segment_footer_index_num_peer_io_total,
231+
statistics->segment_footer_index_num_peer_io_total);
232+
COUNTER_UPDATE(segment_footer_index_bytes_scanned_from_cache,
233+
statistics->segment_footer_index_bytes_read_from_local);
234+
COUNTER_UPDATE(segment_footer_index_bytes_scanned_from_remote,
235+
statistics->segment_footer_index_bytes_read_from_remote);
236+
COUNTER_UPDATE(segment_footer_index_bytes_scanned_from_peer,
237+
statistics->segment_footer_index_bytes_read_from_peer);
238+
COUNTER_UPDATE(segment_footer_index_local_io_timer,
239+
statistics->segment_footer_index_local_io_timer);
240+
COUNTER_UPDATE(segment_footer_index_remote_io_timer,
241+
statistics->segment_footer_index_remote_io_timer);
242+
COUNTER_UPDATE(segment_footer_index_peer_io_timer,
243+
statistics->segment_footer_index_peer_io_timer);
196244
}
197245

198246
} // namespace doris::io

‎be/src/io/cache/block_file_cache_profile.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,16 @@ struct FileCacheProfileReporter {
9898
RuntimeProfile::Counter* inverted_index_peer_io_timer = nullptr;
9999
RuntimeProfile::Counter* inverted_index_io_timer = nullptr;
100100

101+
RuntimeProfile::Counter* segment_footer_index_num_local_io_total = nullptr;
102+
RuntimeProfile::Counter* segment_footer_index_num_remote_io_total = nullptr;
103+
RuntimeProfile::Counter* segment_footer_index_num_peer_io_total = nullptr;
104+
RuntimeProfile::Counter* segment_footer_index_bytes_scanned_from_cache = nullptr;
105+
RuntimeProfile::Counter* segment_footer_index_bytes_scanned_from_remote = nullptr;
106+
RuntimeProfile::Counter* segment_footer_index_bytes_scanned_from_peer = nullptr;
107+
RuntimeProfile::Counter* segment_footer_index_local_io_timer = nullptr;
108+
RuntimeProfile::Counter* segment_footer_index_remote_io_timer = nullptr;
109+
RuntimeProfile::Counter* segment_footer_index_peer_io_timer = nullptr;
110+
101111
FileCacheProfileReporter(RuntimeProfile* profile);
102112
void update(const FileCacheStatistics* statistics) const;
103113
};

‎be/src/io/cache/cached_remote_file_reader.cpp‎

Lines changed: 53 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -321,13 +321,23 @@ Status CachedRemoteFileReader::read_at_impl(size_t offset, Slice result, size_t*
321321
}
322322
if (!io_ctx->is_warmup) {
323323
// update stats increment in this reading procedure for file cache metrics
324+
const auto file_cache_read_type =
325+
io_ctx->is_inverted_index
326+
? FileCacheReadType::INVERTED_INDEX
327+
: (io_ctx->is_index_data ? FileCacheReadType::SEGMENT_FOOTER_INDEX
328+
: FileCacheReadType::DATA);
324329
FileCacheStatistics fcache_stats_increment;
325-
_update_stats(stats, &fcache_stats_increment, io_ctx->is_inverted_index);
330+
_update_stats(stats, &fcache_stats_increment, file_cache_read_type);
326331
io::FileCacheMetrics::instance().update(&fcache_stats_increment);
327332
}
328333
if (io_ctx->file_cache_stats) {
329334
// update stats in io_ctx, for query profile
330-
_update_stats(stats, io_ctx->file_cache_stats, io_ctx->is_inverted_index);
335+
const auto file_cache_read_type =
336+
io_ctx->is_inverted_index
337+
? FileCacheReadType::INVERTED_INDEX
338+
: (io_ctx->is_index_data ? FileCacheReadType::SEGMENT_FOOTER_INDEX
339+
: FileCacheReadType::DATA);
340+
_update_stats(stats, io_ctx->file_cache_stats, file_cache_read_type);
331341
}
332342
};
333343
std::unique_ptr<int, decltype(defer_func)> defer((int*)0x01, std::move(defer_func));
@@ -609,7 +619,7 @@ Status CachedRemoteFileReader::read_at_impl(size_t offset, Slice result, size_t*
609619

610620
void CachedRemoteFileReader::_update_stats(const ReadStatistics& read_stats,
611621
FileCacheStatistics* statis,
612-
bool is_inverted_index) const {
622+
FileCacheReadType read_type) const {
613623
if (statis == nullptr) {
614624
return;
615625
}
@@ -639,22 +649,52 @@ void CachedRemoteFileReader::_update_stats(const ReadStatistics& read_stats,
639649
statis->get_timer += read_stats.get_timer;
640650
statis->set_timer += read_stats.set_timer;
641651

642-
if (is_inverted_index) {
652+
auto update_index_stats = [&](int64_t& num_local_io_total, int64_t& num_remote_io_total,
653+
int64_t& num_peer_io_total, int64_t& bytes_read_from_local,
654+
int64_t& bytes_read_from_remote, int64_t& bytes_read_from_peer,
655+
int64_t& local_io_timer, int64_t& remote_io_timer,
656+
int64_t& peer_io_timer) {
643657
if (read_stats.bytes_read_from_local > 0) {
644-
statis->inverted_index_num_local_io_total++;
645-
statis->inverted_index_bytes_read_from_local += read_stats.bytes_read_from_local;
658+
num_local_io_total++;
659+
bytes_read_from_local += read_stats.bytes_read_from_local;
646660
}
647661
if (read_stats.bytes_read_from_remote > 0) {
648-
statis->inverted_index_num_remote_io_total++;
649-
statis->inverted_index_bytes_read_from_remote += read_stats.bytes_read_from_remote;
650-
statis->inverted_index_remote_io_timer += read_stats.remote_read_timer;
662+
num_remote_io_total++;
663+
bytes_read_from_remote += read_stats.bytes_read_from_remote;
664+
remote_io_timer += read_stats.remote_read_timer;
651665
}
652666
if (read_stats.bytes_read_from_peer > 0) {
653-
statis->inverted_index_num_peer_io_total++;
654-
statis->inverted_index_bytes_read_from_peer += read_stats.bytes_read_from_peer;
655-
statis->inverted_index_peer_io_timer += read_stats.peer_read_timer;
667+
num_peer_io_total++;
668+
bytes_read_from_peer += read_stats.bytes_read_from_peer;
669+
peer_io_timer += read_stats.peer_read_timer;
656670
}
657-
statis->inverted_index_local_io_timer += read_stats.local_read_timer;
671+
local_io_timer += read_stats.local_read_timer;
672+
};
673+
674+
switch (read_type) {
675+
case FileCacheReadType::DATA:
676+
break;
677+
case FileCacheReadType::INVERTED_INDEX:
678+
update_index_stats(
679+
statis->inverted_index_num_local_io_total,
680+
statis->inverted_index_num_remote_io_total,
681+
statis->inverted_index_num_peer_io_total,
682+
statis->inverted_index_bytes_read_from_local,
683+
statis->inverted_index_bytes_read_from_remote,
684+
statis->inverted_index_bytes_read_from_peer, statis->inverted_index_local_io_timer,
685+
statis->inverted_index_remote_io_timer, statis->inverted_index_peer_io_timer);
686+
break;
687+
case FileCacheReadType::SEGMENT_FOOTER_INDEX:
688+
update_index_stats(statis->segment_footer_index_num_local_io_total,
689+
statis->segment_footer_index_num_remote_io_total,
690+
statis->segment_footer_index_num_peer_io_total,
691+
statis->segment_footer_index_bytes_read_from_local,
692+
statis->segment_footer_index_bytes_read_from_remote,
693+
statis->segment_footer_index_bytes_read_from_peer,
694+
statis->segment_footer_index_local_io_timer,
695+
statis->segment_footer_index_remote_io_timer,
696+
statis->segment_footer_index_peer_io_timer);
697+
break;
658698
}
659699

660700
g_skip_cache_sum << read_stats.skip_cache;

‎be/src/io/cache/cached_remote_file_reader.h‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,12 @@ class CachedRemoteFileReader final : public FileReader,
7575
const IOContext* io_ctx) override;
7676

7777
private:
78+
enum class FileCacheReadType {
79+
DATA,
80+
INVERTED_INDEX,
81+
SEGMENT_FOOTER_INDEX,
82+
};
83+
7884
void _insert_file_reader(FileBlockSPtr file_block);
7985

8086
// Execute remote read (S3 or peer).
@@ -83,7 +89,7 @@ class CachedRemoteFileReader final : public FileReader,
8389
ReadStatistics& stats, const IOContext* io_ctx);
8490

8591
void _update_stats(const ReadStatistics& stats, FileCacheStatistics* state,
86-
bool is_inverted_index) const;
92+
FileCacheReadType read_type) const;
8793

8894
bool _is_doris_table = false;
8995
int64_t _tablet_id = -1;

‎be/src/io/fs/file_reader.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
#include <stddef.h>
2222

2323
#include <memory>
24+
#include <string>
2425

2526
#include "common/status.h"
2627
#include "io/fs/path.h"

‎be/src/io/fs/file_writer.h‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ struct FileWriterOptions {
4444
// this shortens the inconsistent time window.
4545
bool used_by_s3_committer = false;
4646
bool write_file_cache = false;
47+
bool allow_adaptive_file_cache_write = true;
4748
bool is_cold_data = false;
4849
bool sync_file_data = true; // Whether flush data into storage system
4950
uint64_t file_cache_expiration_time = 0; // Relative time
@@ -109,13 +110,15 @@ class FileWriter {
109110
io::UInt128Wrapper path_hash = BlockFileCache::hash(path.filename().native());
110111
BlockFileCache* file_cache_ptr = FileCacheFactory::instance()->get_by_path(path_hash);
111112

112-
bool has_enough_file_cache_space = config::enable_file_cache_adaptive_write &&
113+
bool has_enough_file_cache_space = opts->allow_adaptive_file_cache_write &&
114+
config::enable_file_cache_adaptive_write &&
113115
(opts->approximate_bytes_to_write > 0) &&
114116
(file_cache_ptr->approximate_available_cache_size() >
115117
opts->approximate_bytes_to_write);
116118

117119
VLOG_DEBUG << "path:" << path.filename().native()
118120
<< ", write_file_cache:" << opts->write_file_cache
121+
<< ", allow_adaptive_file_cache_write:" << opts->allow_adaptive_file_cache_write
119122
<< ", has_enough_file_cache_space:" << has_enough_file_cache_space
120123
<< ", approximate_bytes_to_write:" << opts->approximate_bytes_to_write
121124
<< ", file_cache_available_size:"

‎be/src/io/fs/packed_file_system.cpp‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include <string_view>
2121
#include <utility>
2222

23+
#include "cloud/config.h"
2324
#include "common/status.h"
2425
#include "io/fs/file_reader.h"
2526
#include "io/fs/packed_file_reader.h"
@@ -96,8 +97,13 @@ Status PackedFileSystem::create_file_impl(const Path& file, FileWriterPtr* write
9697
return Status::OK();
9798
}
9899

100+
auto append_info = _append_info;
101+
if (config::enable_file_cache_write_index_file_only && opts != nullptr) {
102+
append_info.write_file_cache = opts->write_file_cache;
103+
}
104+
99105
// Wrap with PackedFileWriter
100-
*writer = std::make_unique<PackedFileWriter>(std::move(inner_writer), file, _append_info);
106+
*writer = std::make_unique<PackedFileWriter>(std::move(inner_writer), file, append_info);
101107
return Status::OK();
102108
}
103109

@@ -106,11 +112,19 @@ Status PackedFileSystem::open_file_impl(const Path& file, FileReaderSPtr* reader
106112
// Check if this file is in a packed file
107113
std::string file_path = file.native();
108114
auto it = _index_map.find(file_path);
109-
bool is_packed_file = (it != _index_map.end());
115+
PackedSliceLocation manager_location;
116+
const PackedSliceLocation* packed_location = nullptr;
117+
if (it != _index_map.end()) {
118+
packed_location = &it->second;
119+
} else if (PackedFileManager::instance()
120+
->get_packed_slice_location(file_path, &manager_location)
121+
.ok()) {
122+
packed_location = &manager_location;
123+
}
110124

111-
if (is_packed_file) {
125+
if (packed_location != nullptr) {
112126
// File is in packed file, open packed file and wrap with PackedFileReader
113-
const auto& index = it->second;
127+
const auto& index = *packed_location;
114128
FileReaderSPtr inner_reader;
115129

116130
// Create options for opening the packed file

‎be/src/io/fs/s3_file_system.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
#include "common/config.h"
3636
#include "common/logging.h"
3737
#include "common/status.h"
38+
#include "cpp/sync_point.h"
3839
#include "io/fs/err_utils.h"
3940
#include "io/fs/file_system.h"
4041
#include "io/fs/file_writer.h"
@@ -195,6 +196,7 @@ Status S3FileSystem::create_file_impl(const Path& file, FileWriterPtr* writer,
195196

196197
Status S3FileSystem::open_file_internal(const Path& file, FileReaderSPtr* reader,
197198
const FileReaderOptions& opts) {
199+
TEST_SYNC_POINT_CALLBACK("S3FileSystem::open_file_internal", &file, &opts);
198200
auto key = DORIS_TRY(get_key(file));
199201
*reader = DORIS_TRY(S3FileReader::create(_client, _bucket, key, opts.file_size, nullptr));
200202
return Status::OK();

0 commit comments

Comments
 (0)