Skip to content

Commit 8d88f1c

Browse files
authored
[perf](load) introduce sequential prefetch for FileScannerV2 CSV reads (#69716)
### What problem does this PR solve? Issue Number: None Related PR: #69333 Problem Summary: After CSV ingestion switched to FileScannerV2, `DelimitedTextReader` opened remote files directly through `FileFactory`, bypassing the sequential prefetch used by the original CSV reader. S3 reads therefore block the scanner instead of overlapping remote I/O with CSV parsing. Restore `DelegateReader::create_file_reader(SEQUENTIAL)` in the shared CSV/Text reader, passing the shared IOContext and adjusted split range. Unknown-size first splits use an end sentinel bounded by the physical file size. Tracing remains outside the prefetch layer, and shared stream-load pipes retain their existing ownership and cancellation behavior. To safely integrate asynchronous reads with V2 scanner statistics: - Each prefetch buffer owns its cache statistics and an IOContext snapshot. The consumer merges completed statistics exactly once before consuming, reusing or closing the buffer. Workers never update the scanner's cache-statistics object. - Close drains active reads even after a foreground timeout, keeping borrowed scanner state alive until I/O completes. Queued tasks observe CLOSED and skip reading. Cleanup can consequently take longer than `buffered_reader_read_timeout_ms` while an underlying request is outstanding. - The ring modulus uses allocated capacity: a 6 MiB setting allocates one 4 MiB buffer and wraps at 4 MiB. Settings below 4 MiB use one buffer; the 16 MiB default is unchanged. - CSV/Text filesystem readers close before collecting prefetch profile counters. Repeated close does not publish counters twice. ### Performance For the TPC-H dataset load benchmark, elapsed time decreased from **2617.33 s** to **1696.06 s**, saving **921.27 s**. This is a **35.2% reduction in load time**, or approximately **1.54x speedup**, for the measured workload. | Dataset | Before | After | Load time reduction | Speedup | | --- | ---: | ---: | ---: | ---: | | TPC-H | 2617.33 s | 1696.06 s | 35.2% | 1.54x | ### Release note Restore asynchronous S3 prefetch for FileScannerV2 CSV and text reads, with safe cache-statistics aggregation and cleanup, consistent ring capacity and populated prefetch profiles. ### Check List (For Author) - Test: - Added S3 mock coverage for whole-file, mid-row split and unknown-size reads; checks row values, outer tracing, background S3 reads, prefetch profile publication and repeated close. - Added buffered-reader tests for four concurrent workers with isolated cache statistics, partial consumption, slot reuse, unconsumed-buffer collection, successful reads finishing after foreground timeout, explicit close/destruction, and exact bytes across 0/1/6/10/16 MiB ring settings from zero/nonzero offsets. Added a shared stream-pipe ownership test. - Passed Clang 16 syntax checking of the complete `buffered_reader_test.cpp` translation unit against repository and installed third-party headers with `-Werror -Wunused-member-function`. - Passed repository clang-format 16 formatting/check scripts, `build-support/check-build-hygiene.sh`, and `git diff --check`. - Full BE build and runtime tests are **not verified locally**: `./run-be-ut.sh --run --filter='BufferedReaderTest.*:CsvV2ReaderTest.*' -j 8` stops at JDK initialization (`Failed setting boot class path`). Separate production/CSV syntax attempts encounter installed C++ library and stale generated-header incompatibilities. No compilation database is available for clang-tidy. Runtime and sanitizer validation remain pending CI. - The S3 mock test disables file cache; the concurrent statistics test exercises the actual prefetch implementation with a mock reader updating `FileCacheStatistics`. No cached-S3 end-to-end runtime result is claimed. - TPC-H load benchmark: elapsed time decreased from 2617.33 s to 1696.06 s (35.2% less load time; approximately 1.54x speedup). - Behavior changed: - [x] Yes. Eligible S3 CSV/Text readers use sequential prefetch. Shared prefetch readers isolate cache counters, drain outstanding reads during cleanup and normalize ring capacity. CSV/Text publishes prefetch profiles on close. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 76011df commit 8d88f1c

5 files changed

Lines changed: 373 additions & 41 deletions

File tree

‎be/src/format_v2/delimited_text/delimited_text_reader.cpp‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
#include "format_v2/column_mapper.h"
3838
#include "format_v2/materialized_reader_util.h"
3939
#include "io/file_factory.h"
40+
#include "io/fs/buffered_reader.h"
4041
#include "io/fs/tracing_file_reader.h"
4142
#include "runtime/descriptors.h"
4243
#include "runtime/file_scan_profile.h"
@@ -486,6 +487,13 @@ Status DelimitedTextReader::close() {
486487
_line_reader.reset();
487488
}
488489
_decompressor.reset();
490+
// Stream pipes are shared with the load context; preserve their existing ownership and
491+
// cancellation behavior. Only filesystem readers enter the sequential prefetch path.
492+
if (_file_reader != nullptr && _scan_params->file_type != TFileType::FILE_STREAM) {
493+
// Drain prefetch workers before publishing their counters or releasing IOContext refs.
494+
RETURN_IF_ERROR(_file_reader->close());
495+
_file_reader->collect_profile_before_close();
496+
}
489497
_file_reader.reset();
490498
_tracing_file_reader.reset();
491499
_requested_columns.clear();
@@ -605,8 +613,14 @@ Status DelimitedTextReader::_open_file() {
605613
} else {
606614
auto reader_options = FileFactory::get_reader_options(_runtime_state->query_options(),
607615
*_file_description);
608-
auto file_reader = DORIS_TRY(FileFactory::create_file_reader(
609-
*_system_properties, *_file_description, reader_options, _profile));
616+
// Include the prefix used to align non-first splits. Unknown-size first splits read to
617+
// EOF; PrefetchBufferedReader bounds this range by the opened file's physical size.
618+
const auto prefetch_end = _size < 0 ? std::numeric_limits<size_t>::max()
619+
: cast_set<size_t>(_start_offset + _size);
620+
auto file_reader = DORIS_TRY(io::DelegateReader::create_file_reader(
621+
_profile, *_system_properties, *_file_description, reader_options,
622+
io::DelegateReader::AccessMode::SEQUENTIAL, _io_ctx,
623+
io::PrefetchRange(_start_offset, prefetch_end)));
610624
_file_reader = _io_ctx && _io_ctx->file_reader_stats
611625
? std::make_shared<io::TracingFileReader>(std::move(file_reader),
612626
_io_ctx->file_reader_stats)

‎be/src/io/fs/buffered_reader.cpp‎

Lines changed: 30 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -413,6 +413,13 @@ void PrefetchBuffer::reset_offset(size_t offset) {
413413
_prefetched.notify_all();
414414
return;
415415
}
416+
_merge_file_cache_stats();
417+
// Snapshot on the consumer thread, not concurrently with scanner counter updates.
418+
// Cached readers get per-buffer counters; uncached readers keep a null stats pointer.
419+
_prefetch_io_ctx = *_io_ctx;
420+
if (_io_ctx->file_cache_stats != nullptr) {
421+
_prefetch_io_ctx.file_cache_stats = &_file_cache_stats;
422+
}
416423
_buffer_status = BufferStatus::RESET;
417424
_offset = offset;
418425
_prefetched.notify_all();
@@ -470,7 +477,7 @@ void PrefetchBuffer::prefetch_buffer() {
470477

471478
{
472479
SCOPED_RAW_TIMER(&_statis.read_time);
473-
s = _reader->read_at(_offset, Slice {_buf.data(), buf_size}, &_len, _io_ctx);
480+
s = _reader->read_at(_offset, Slice {_buf.data(), buf_size}, &_len, &_prefetch_io_ctx);
474481
}
475482
if (UNLIKELY(s.ok() && buf_size != _len)) {
476483
// This indicates that the data size returned by S3 object storage is smaller than what we requested,
@@ -497,6 +504,7 @@ void PrefetchBuffer::prefetch_buffer() {
497504
_offset, s.to_string());
498505
_prefetch_status = std::move(s);
499506
}
507+
_has_unmerged_cache_stats = _prefetch_io_ctx.file_cache_stats != nullptr;
500508
_buffer_status = BufferStatus::PREFETCHED;
501509
_prefetched.notify_all();
502510
// eof would come up with len == 0, it would be handled by read_buffer
@@ -582,6 +590,7 @@ Status PrefetchBuffer::read_buffer(size_t off, const char* out, size_t buf_len,
582590
if (UNLIKELY(BufferStatus::CLOSED == _buffer_status)) {
583591
return Status::OK();
584592
}
593+
_merge_file_cache_stats();
585594
}
586595
RETURN_IF_ERROR(_prefetch_status);
587596
// there is only parquet would do not sequence read
@@ -612,15 +621,23 @@ Status PrefetchBuffer::read_buffer(size_t off, const char* out, size_t buf_len,
612621
return Status::OK();
613622
}
614623

624+
void PrefetchBuffer::_merge_file_cache_stats() {
625+
// Called only by the consumer with _lock held and no pending read. A buffer may be
626+
// consumed in several slices; publish each completed request exactly once.
627+
if (_has_unmerged_cache_stats) {
628+
_io_ctx->file_cache_stats->merge_from(_file_cache_stats);
629+
_file_cache_stats = {};
630+
_has_unmerged_cache_stats = false;
631+
}
632+
}
633+
615634
void PrefetchBuffer::close() {
616635
std::unique_lock lck {_lock};
617-
// in case _reader still tries to write to the buf after we close the buffer
618-
if (!_prefetched.wait_for(lck,
619-
std::chrono::milliseconds(config::buffered_reader_read_timeout_ms),
620-
[this]() { return _buffer_status != BufferStatus::PENDING; })) {
621-
_prefetch_status = Status::TimedOut("time out when close prefetch buffer");
622-
return;
623-
}
636+
// A foreground read timeout does not cancel the underlying I/O. Wait for it to finish
637+
// before releasing scanner-owned IOContext pointees, including cache statistics and
638+
// the cache-write limiter. Queued tasks observe CLOSED and do not start another read.
639+
_prefetched.wait(lck, [this]() { return _buffer_status != BufferStatus::PENDING; });
640+
_merge_file_cache_stats();
624641
_buffer_status = BufferStatus::CLOSED;
625642
_prefetched.notify_all();
626643
// Explicitly release the backing buffer here, in the calling (query) thread which has a
@@ -647,14 +664,14 @@ PrefetchBufferedReader::PrefetchBufferedReader(RuntimeProfile* profile, io::File
647664
}
648665
_io_ctx = _io_ctx_holder.get();
649666
if (buffer_size == -1L) {
650-
buffer_size = config::remote_storage_read_buffer_mb * 1024 * 1024;
667+
buffer_size = int64_t {config::remote_storage_read_buffer_mb} * 1024 * 1024;
651668
}
652669
_size = _reader->size();
653-
_whole_pre_buffer_size = buffer_size;
670+
// Use the allocated capacity as the ring modulus. For example, 6 MiB allocates
671+
// one 4 MiB buffer, so offset 4 MiB must wrap to slot 0, not index a missing slot 1.
672+
const int64_t buffer_num = std::max<int64_t>(1, buffer_size / s_max_pre_buffer_size);
673+
_whole_pre_buffer_size = buffer_num * s_max_pre_buffer_size;
654674
_file_range.end_offset = std::min(_file_range.end_offset, _size);
655-
int buffer_num = buffer_size > s_max_pre_buffer_size
656-
? cast_set<int>(buffer_size) / cast_set<int>(s_max_pre_buffer_size)
657-
: 1;
658675
std::function<void(PrefetchBuffer&)> sync_buffer = nullptr;
659676
if (profile != nullptr) {
660677
const char* prefetch_buffered_reader = "PrefetchBufferedReader";

‎be/src/io/fs/buffered_reader.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -469,6 +469,11 @@ struct PrefetchBuffer : std::enable_shared_from_this<PrefetchBuffer>, public Pro
469469
io::FileReaderSPtr _reader;
470470
std::shared_ptr<const IOContext> _io_ctx_holder;
471471
const IOContext* _io_ctx = nullptr;
472+
// Only the worker writes these statistics. The consumer merges them after observing
473+
// completion under _lock, before reusing or closing the buffer.
474+
IOContext _prefetch_io_ctx;
475+
FileCacheStatistics _file_cache_stats;
476+
bool _has_unmerged_cache_stats = false;
472477
PODArray<char> _buf;
473478
BufferStatus _buffer_status {BufferStatus::RESET};
474479
std::mutex _lock;
@@ -515,6 +520,9 @@ struct PrefetchBuffer : std::enable_shared_from_this<PrefetchBuffer>, public Pro
515520
void _collect_profile_at_runtime() override {}
516521

517522
void _collect_profile_before_close() override;
523+
524+
private:
525+
void _merge_file_cache_stats();
518526
};
519527

520528
constexpr int64_t s_max_pre_buffer_size = 4 * 1024 * 1024; // 4MB

‎be/test/format_v2/delimited_text/csv_reader_test.cpp‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,12 @@
2020
#include <gtest/gtest.h>
2121

2222
#include <algorithm>
23+
#include <atomic>
24+
#include <cstring>
2325
#include <filesystem>
2426
#include <fstream>
2527
#include <memory>
28+
#include <thread>
2629

2730
#include "common/config.h"
2831
#include "common/consts.h"
@@ -44,17 +47,95 @@
4447
#include "exprs/vexpr_context.h"
4548
#include "format/csv/csv_reader.h"
4649
#include "format_v2/column_mapper.h"
50+
#include "io/fs/stream_load_pipe.h"
4751
#include "io/io_common.h"
52+
#include "runtime/exec_env.h"
4853
#include "runtime/runtime_profile.h"
4954
#include "testutil/desc_tbl_builder.h"
5055
#include "testutil/mock/mock_runtime_state.h"
56+
#include "testutil/mock/obj_storage_client_test_stub.h"
5157
#include "testutil/scoped_temp_dir.h"
5258
#include "util/debug_points.h"
5359
#include "util/defer_op.h"
60+
#include "util/s3_util.h"
61+
#include "util/threadpool.h"
5462

5563
namespace doris::format::csv {
5664
namespace {
5765

66+
class CsvPrefetchS3Client : public io::ObjStorageClientTestStub {
67+
public:
68+
const std::string data = "id,name,score\n1,alice,10\n2,bob,20\n3,carol,30\n";
69+
const std::thread::id scanner_thread = std::this_thread::get_id();
70+
std::atomic<size_t> foreground_reads {0};
71+
std::atomic<size_t> reads {0};
72+
73+
io::ObjStorageHeadResult head_object(const io::ObjStoragePath&) override {
74+
return {.resp = io::ObjStorageResponse::OK(),
75+
.file_size = static_cast<long long>(data.size())};
76+
}
77+
78+
io::ObjStorageResponse get_object(const io::ObjStoragePath&, void* buffer, size_t offset,
79+
size_t bytes_read, size_t* size_return) override {
80+
++reads;
81+
if (std::this_thread::get_id() == scanner_thread) {
82+
++foreground_reads;
83+
}
84+
EXPECT_LE(offset + bytes_read, data.size());
85+
*size_return = std::min(bytes_read, data.size() - offset);
86+
std::memcpy(buffer, data.data() + offset, *size_return);
87+
return io::ObjStorageResponse::OK();
88+
}
89+
90+
// Only HEAD and GET belong to this read test. Explicit overrides avoid generating unused
91+
// gMock expectation helpers for this anonymous-namespace client.
92+
io::ObjStorageUploadResult create_multipart_upload(const io::ObjStoragePath&) override {
93+
return {.resp = unexpected_operation()};
94+
}
95+
96+
io::ObjStorageResponse put_object(const io::ObjStoragePath&, std::string_view) override {
97+
return unexpected_operation();
98+
}
99+
100+
io::ObjStorageUploadResult upload_part(const io::ObjStoragePath&, const std::string&,
101+
std::string_view, int) override {
102+
return {.resp = unexpected_operation()};
103+
}
104+
105+
io::ObjStorageResponse complete_multipart_upload(
106+
const io::ObjStoragePath&, const std::string&,
107+
const std::vector<io::ObjStorageCompletedPart>&) override {
108+
return unexpected_operation();
109+
}
110+
111+
io::ObjStorageResponse delete_objects(const io::ObjStoragePath&,
112+
std::vector<std::string>) override {
113+
return unexpected_operation();
114+
}
115+
116+
io::ObjStorageResponse delete_object(const io::ObjStoragePath&) override {
117+
return unexpected_operation();
118+
}
119+
120+
std::string generate_presigned_url(const io::ObjStoragePath&, int64_t) override {
121+
ADD_FAILURE() << "Unexpected signed URL request in CSV prefetch test";
122+
return {};
123+
}
124+
125+
io::ObjStorageListPageResult list_objects_page(const io::ObjStoragePath&,
126+
std::string_view) override {
127+
return {.resp = unexpected_operation()};
128+
}
129+
130+
private:
131+
static io::ObjStorageResponse unexpected_operation() {
132+
ADD_FAILURE() << "Unexpected object storage operation in CSV prefetch test";
133+
return {.status = {TStatusCode::NOT_IMPLEMENTED_ERROR,
134+
"CSV prefetch test only supports HEAD and GET"},
135+
.http_code = 0};
136+
}
137+
};
138+
58139
TFileScanRangeParams csv_scan_params() {
59140
TFileScanRangeParams params;
60141
params.__set_format_type(TFileFormatType::FORMAT_CSV_PLAIN);
@@ -512,6 +593,78 @@ class CsvV2ReaderTest : public testing::Test {
512593
TFileScanRangeParams _params;
513594
};
514595

596+
TEST_F(CsvV2ReaderTest, S3PrefetchPreservesSplitRowsAndReadStatistics) {
597+
auto* env = ExecEnv::GetInstance();
598+
auto saved_pool = std::move(env->_buffered_reader_prefetch_thread_pool);
599+
const auto saved_in_memory_size = config::in_memory_file_size;
600+
Defer restore([&] {
601+
env->_buffered_reader_prefetch_thread_pool.reset();
602+
env->_buffered_reader_prefetch_thread_pool = std::move(saved_pool);
603+
config::in_memory_file_size = saved_in_memory_size;
604+
S3ClientFactory::instance().clear_client_creator_for_test();
605+
});
606+
ASSERT_TRUE(ThreadPoolBuilder("CsvPrefetchTest")
607+
.set_min_threads(1)
608+
.set_max_threads(1)
609+
.build(&env->_buffered_reader_prefetch_thread_pool)
610+
.ok());
611+
// Keep the small fixture on the same prefetch path as a large S3 CSV file.
612+
config::in_memory_file_size = 0;
613+
auto client = std::make_shared<CsvPrefetchS3Client>();
614+
S3ClientFactory::instance().set_client_creator_for_test(
615+
[client](const S3ClientConf&) { return client; });
616+
_params.__set_file_type(TFileType::FILE_S3);
617+
_state._query_options.__set_enable_file_cache(false);
618+
619+
// Cover a complete file, a split starting inside Alice's row, and an unknown-size first split.
620+
for (const int64_t split_start : {0, 17, -1}) {
621+
SCOPED_TRACE(split_start);
622+
auto properties = std::make_shared<io::FileSystemProperties>();
623+
properties->system_type = TFileType::FILE_S3;
624+
properties->properties = {{"AWS_ENDPOINT", "http://csv-prefetch.invalid"},
625+
{"AWS_REGION", "test-region"}};
626+
auto desc = std::make_unique<io::FileDescription>();
627+
desc->path = "s3://csv-prefetch/reader.csv";
628+
desc->range_start_offset = std::max<int64_t>(0, split_start);
629+
desc->file_size = split_start < 0 ? -1 : static_cast<int64_t>(client->data.size());
630+
desc->range_size = desc->file_size - desc->range_start_offset;
631+
io::FileReaderStats stats;
632+
auto io_ctx = std::make_shared<io::IOContext>();
633+
io_ctx->file_reader_stats = &stats;
634+
auto reader =
635+
std::make_unique<CsvReader>(properties, desc, io_ctx, &_profile, &_params, _slots);
636+
ASSERT_TRUE(reader->init(&_state).ok());
637+
std::vector<ColumnDefinition> schema;
638+
ASSERT_TRUE(reader->get_schema(&schema).ok());
639+
auto request = std::make_shared<FileScanRequest>();
640+
for (int32_t id = 0; id < 3; ++id) {
641+
request->non_predicate_columns.push_back(
642+
LocalColumnIndex::top_level(LocalColumnId(id)));
643+
request->local_positions.emplace(LocalColumnId(id), LocalIndex(id));
644+
}
645+
ASSERT_TRUE(reader->open(request).ok());
646+
auto block = make_block(schema, {0, 1, 2});
647+
size_t rows = 0;
648+
bool eof = false;
649+
ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok());
650+
ASSERT_EQ(rows, split_start > 0 ? 2 : 3);
651+
EXPECT_EQ(nullable_int_at(*block.get_by_position(0).column, 0), split_start > 0 ? 2 : 1);
652+
EXPECT_EQ(nullable_string_at(*block.get_by_position(1).column, rows - 1), "carol");
653+
EXPECT_GT(stats.read_calls, 0);
654+
EXPECT_GT(stats.read_bytes, 0);
655+
const auto previous_prefetch_io = counter_value(&_profile, "PreRequestIO");
656+
ASSERT_TRUE(reader->close().ok());
657+
EXPECT_GT(counter_value(&_profile, "PreRequestIO"), previous_prefetch_io);
658+
EXPECT_GT(counter_value(&_profile, "PreRequestBytes"), 0);
659+
EXPECT_GT(counter_value(&_profile, "ReadTime"), 0);
660+
const auto collected_prefetch_io = counter_value(&_profile, "PreRequestIO");
661+
ASSERT_TRUE(reader->close().ok());
662+
EXPECT_EQ(counter_value(&_profile, "PreRequestIO"), collected_prefetch_io);
663+
}
664+
EXPECT_GT(client->reads.load(), 0);
665+
EXPECT_EQ(client->foreground_reads.load(), 0);
666+
}
667+
515668
// Scenario: CSV v2 exposes FE-provided file slots as nullable file-local schema using column_idxs
516669
// as CSV field ordinals.
517670
TEST_F(CsvV2ReaderTest, SchemaUsesSlotTypesAndColumnIdxs) {
@@ -927,6 +1080,17 @@ TEST_F(CsvV2ReaderTest, StreamInputRequiresLoadIdBeforeOpeningPipe) {
9271080
<< status;
9281081
}
9291082

1083+
TEST_F(CsvV2ReaderTest, ClosePreservesSharedStreamPipe) {
1084+
_params.__set_file_type(TFileType::FILE_STREAM);
1085+
auto reader = create_unknown_size_reader(_file_path, &_params, _slots, &_state, &_profile);
1086+
auto pipe = std::make_shared<io::StreamLoadPipe>();
1087+
reader->_file_reader = pipe;
1088+
ASSERT_TRUE(reader->close().ok());
1089+
EXPECT_FALSE(pipe->closed());
1090+
reader.reset();
1091+
EXPECT_FALSE(pipe->closed());
1092+
}
1093+
9301094
// Scenario: CSV has no footer row count, so v2 COUNT pushdown scans the split and returns the
9311095
// counted row count through FileAggregateResult.
9321096
TEST_F(CsvV2ReaderTest, CountAggregateScansRows) {

0 commit comments

Comments
 (0)