Skip to content

Commit 374d87d

Browse files
authored
[perf](csv reader) Reduce V2 CSV parsing overhead (#69875)
### What problem does this PR solve? Issue Number: None Related PR: #69716 Problem Summary: After restoring sequential prefetch, large V2 CSV loads still spend substantial CPU time in row parsing and string materialization. The reader compares a general separator at every byte, updates atomic profile counters per cell, starts three stage timers per row, and repeats checked string-column casts per cell. It also ignores the row cap supplied by TableReader. This change combines four optimizations on master: - Compare the separator byte directly for plain single-character separators; retain enclosed, multi-character, Hive/OpenCSV and PROTO parsing paths. - Accumulate exact work counters locally and publish at get_block/count/schema return, including errors. Sample stage timings every 1024 physical read_line calls, starting with the first, and expose unscaled SampledReadLineTime, SampledSplitLineTime and SampledDeserializeTime with sample counts. DelimitedTextReader remains full elapsed time; split-prefix alignment has its own full timer. - Validate nullable string column types and reserve offsets/null maps once per mutable batch, then insert strings directly with the existing NULL, quoting and escape semantics. - Honor FileReader::set_batch_size, retaining the existing byte budget and runtime default when no explicit cap is supplied. The caller still controls the maximum batch size; this does not force 16352-row blocks. Shared counters and batch sizing also apply to the Hive text reader. ### Performance benchmark On TPC-DS SF100, elapsed time decreased from **1,944 seconds to 1,375 seconds**: | Benchmark | Before | After | Time reduction | Speedup | | --- | ---: | ---: | ---: | ---: | | TPC-DS SF100 | 1,944 s | 1,375 s | 29.3% (569 s) | 1.41× | ### Release note Reduce V2 CSV parsing overhead and honor caller-provided batch sizes. Delimited-text per-stage profile timers are now explicitly named Sampled* and contain unscaled sampled durations; use the full DelimitedTextReader timer for elapsed-time comparisons. ### Check List (For Author) - Test - [x] Unit Test: added cases for dynamic/zero batch caps, byte limits, filtered blocks, cross-batch sampling and exact counters, single/multi-character separators, NULL and escape behavior, missing fields, reordered projection and copy-on-write. Extended error, rejected-row and COUNT counter coverage. - Execution: run-be-ut.sh was attempted with CsvV2ReaderTest.*, TextV2ReaderTest.* and LegacyAndV2/PlainCsvSplitTest.*. It exits before C++ compilation because the local JDK 17 fails to initialize (Failed setting boot class path), including outside the sandbox. Full build/UT remain pending. Performance benchmark results are reported above. - Passed: repository clang-format 16 formatting/check scripts, build-support/check-build-hygiene.sh and git diff --check. - Behavior changed: Yes. Existing batch-size contract is honored, and detailed stage timers are explicitly sampled. CSV value semantics are unchanged. - Does this need documentation: No. The profile includes a timing explanation; no new configuration or SQL syntax. For further benchmark comparisons, use identical hardware, input, compression, cache conditions and load batch/byte-budget settings for the parent and this commit. Compare total load time, Scanner CPU time, DelimitedTextReader time and RowsRead/BlockNumber. Do not directly compare Sampled* timers with the previous full stage timers.
1 parent 45d1598 commit 374d87d

6 files changed

Lines changed: 328 additions & 34 deletions

File tree

‎be/src/format_v2/delimited_text/csv_reader.cpp‎

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323

2424
#include "core/assert_cast.h"
2525
#include "core/column/column_nullable.h"
26+
#include "core/column/column_string.h"
2627
#include "core/data_type/data_type_string.h"
2728
#include "core/data_type_serde/data_type_string_serde.h"
2829
#include "format/file_reader/new_plain_binary_line_reader.h"
@@ -221,6 +222,14 @@ void CsvReader::_split_line(const Slice& line) {
221222
append_value(value_start, separator_position - value_start);
222223
value_start = separator_position + _value_separator.size();
223224
}
225+
} else if (_value_separator.size() == 1) {
226+
const char separator = _value_separator[0];
227+
for (size_t i = 0; i < line.size; ++i) {
228+
if (line.data[i] == separator) {
229+
append_value(value_start, i - value_start);
230+
value_start = i + 1;
231+
}
232+
}
224233
} else {
225234
for (size_t i = 0; i < line.size;) {
226235
if (starts_with_at(line, i, _value_separator)) {
@@ -236,25 +245,39 @@ void CsvReader::_split_line(const Slice& line) {
236245
append_value(value_start, line.size - value_start);
237246
}
238247

248+
void CsvReader::_prepare_columns(std::vector<MutableColumnPtr>& columns, size_t batch_size) {
249+
for (const auto& column : _requested_columns) {
250+
if (!column.nullable_string_fast_path) {
251+
continue;
252+
}
253+
auto& nullable = assert_cast<ColumnNullable&>(*columns[column.block_position.value()]);
254+
auto& strings = assert_cast<ColumnString&>(nullable.get_nested_column());
255+
strings.get_offsets().reserve(strings.size() + batch_size);
256+
nullable.get_null_map_data().reserve(nullable.size() + batch_size);
257+
}
258+
}
259+
239260
Status CsvReader::_deserialize_one_cell(const RequestedColumn& column, IColumn* output,
240261
Slice value) {
241262
DORIS_CHECK(output != nullptr);
242263
if (_hive_csv_parser && column.file_column_id.value() >= _split_values.size()) {
243264
return _append_null(output);
244265
}
245266
if (column.nullable_string_fast_path) {
246-
auto& null_column = assert_cast<ColumnNullable&>(*output);
247-
// String is the hottest CSV type. Avoid the generic nullable serde wrapper here:
248-
// deserialize directly into the nested string column and append the null map bit ourselves.
267+
// _prepare_columns validates the concrete types once per batch, after copy-on-write.
268+
auto& null_column = assert_cast<ColumnNullable&, TypeCheckOnRelease::DISABLE>(*output);
269+
auto& string_column = assert_cast<ColumnString&, TypeCheckOnRelease::DISABLE>(
270+
null_column.get_nested_column());
249271
if (_hive_csv_parser) {
250272
// Splitting and decoding are one operation for OpenCSV; a second CSV serde pass
251273
// would unescape literal data and confuse empty strings with missing fields.
252-
null_column.get_nested_column().insert_data(value.data, value.size);
274+
string_column.insert_data(value.data, value.size);
253275
null_column.get_null_map_data().push_back(0);
254276
return Status::OK();
255277
}
256278
if (_empty_field_as_null && value.size == 0) {
257-
null_column.insert_data(nullptr, 0);
279+
string_column.insert_default();
280+
null_column.get_null_map_data().push_back(1);
258281
return Status::OK();
259282
}
260283
// CSV keeps empty-field handling separate from null_format matching. An empty
@@ -263,16 +286,16 @@ Status CsvReader::_deserialize_one_cell(const RequestedColumn& column, IColumn*
263286
const bool quoted = _options.converted_from_string && value.trim_double_quotes();
264287
if (!quoted && _options.null_len > 0 && value.size == _options.null_len &&
265288
std::memcmp(value.data, _options.null_format, value.size) == 0) {
266-
null_column.insert_data(nullptr, 0);
289+
string_column.insert_default();
290+
null_column.get_null_map_data().push_back(1);
267291
return Status::OK();
268292
}
269-
static DataTypeStringSerDe string_serde(TYPE_STRING);
270-
auto status = string_serde.deserialize_one_cell_from_csv(null_column.get_nested_column(),
271-
value, _options);
272-
if (!status.ok()) {
273-
null_column.insert_data(nullptr, 0);
274-
return Status::OK();
293+
// Keep the same unescaping as StringSerDe, which always succeeds for CSV strings.
294+
if (_options.escape_char != 0) {
295+
escape_string_for_csv(value.data, &value.size, _options.escape_char,
296+
_options.quote_char);
275297
}
298+
string_column.insert_data(value.data, value.size);
276299
null_column.get_null_map_data().push_back(0);
277300
return Status::OK();
278301
}

‎be/src/format_v2/delimited_text/csv_reader.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ class CsvReader final : public ::doris::format::DelimitedTextReader {
6060
Status _create_line_reader() override;
6161
Status _validate_line(const Slice& line) override;
6262
void _split_line(const Slice& line) override;
63+
void _prepare_columns(std::vector<MutableColumnPtr>& columns, size_t batch_size) override;
6364
Status _deserialize_one_cell(const RequestedColumn& column, IColumn* output,
6465
Slice value) override;
6566
Slice _normalize_value(Slice value) const override;

‎be/src/format_v2/delimited_text/delimited_text_reader.cpp‎

Lines changed: 69 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
#include "runtime/file_scan_profile.h"
4444
#include "runtime/runtime_state.h"
4545
#include "util/decompressor.h"
46+
#include "util/defer_op.h"
4647
#include "util/string_util.h"
4748
#include "util/utf8_check.h"
4849

@@ -192,11 +193,21 @@ void DelimitedTextReader::_init_profile() {
192193
_text_profile.create_line_reader_time =
193194
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "CreateLineReaderTime", DELIMITED_TEXT_PROFILE, 1);
194195
_text_profile.read_line_time =
195-
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "ReadLineTime", DELIMITED_TEXT_PROFILE, 1);
196+
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "SampledReadLineTime", DELIMITED_TEXT_PROFILE, 1);
196197
_text_profile.split_line_time =
197-
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "SplitLineTime", DELIMITED_TEXT_PROFILE, 1);
198-
_text_profile.deserialize_time =
199-
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "DeserializeTime", DELIMITED_TEXT_PROFILE, 1);
198+
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "SampledSplitLineTime", DELIMITED_TEXT_PROFILE, 1);
199+
_text_profile.deserialize_time = ADD_CHILD_TIMER_WITH_LEVEL(_profile, "SampledDeserializeTime",
200+
DELIMITED_TEXT_PROFILE, 1);
201+
_text_profile.split_prefix_time =
202+
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "SplitPrefixTime", DELIMITED_TEXT_PROFILE, 1);
203+
_text_profile.sampled_read_line_calls = ADD_CHILD_COUNTER_WITH_LEVEL(
204+
_profile, "SampledReadLineCalls", TUnit::UNIT, DELIMITED_TEXT_PROFILE, 1);
205+
_text_profile.sampled_parsed_rows = ADD_CHILD_COUNTER_WITH_LEVEL(
206+
_profile, "SampledParsedRows", TUnit::UNIT, DELIMITED_TEXT_PROFILE, 1);
207+
_profile->add_info_string(
208+
"DelimitedTextTiming",
209+
"Stage timers are unscaled samples of every 1024th read_line call "
210+
"(starting with the first); DelimitedTextReader is full elapsed time");
200211
_text_profile.conjunct_filter_time =
201212
ADD_CHILD_TIMER_WITH_LEVEL(_profile, "ConjunctFilterTime", DELIMITED_TEXT_PROFILE, 1);
202213
_text_profile.delete_conjunct_filter_time = ADD_CHILD_TIMER_WITH_LEVEL(
@@ -285,6 +296,7 @@ Status DelimitedTextReader::init(RuntimeState* state) {
285296

286297
Status DelimitedTextReader::infer_schema(RuntimeState* state,
287298
std::vector<ColumnDefinition>* schema) {
299+
Defer profile_flush {[this] { _flush_profile(); }};
288300
_schema_only = true;
289301
RETURN_IF_ERROR(init(state));
290302
if (_file_description->range_start_offset != 0) {
@@ -360,16 +372,23 @@ Status DelimitedTextReader::open(std::shared_ptr<FileScanRequest> request) {
360372
return Status::OK();
361373
}
362374

375+
void DelimitedTextReader::set_batch_size(size_t batch_size) {
376+
_batch_size = std::max(size_t {1}, batch_size);
377+
}
378+
363379
Status DelimitedTextReader::get_block(Block* file_block, size_t* rows, bool* eof) {
364380
SCOPED_TIMER(_text_profile.total_time);
381+
Defer profile_flush {[this] { _flush_profile(); }};
365382
DORIS_CHECK(file_block != nullptr);
366383
DORIS_CHECK(rows != nullptr);
367384
DORIS_CHECK(eof != nullptr);
368385
if (_line_reader == nullptr) {
369386
return Status::InternalError("{} v2 reader is not open", _reader_name);
370387
}
371388

372-
const auto batch_size = _runtime_state != nullptr ? _runtime_state->batch_size() : 4096;
389+
const size_t batch_size =
390+
_batch_size != 0 ? _batch_size
391+
: (_runtime_state != nullptr ? _runtime_state->batch_size() : 4096);
373392
const auto max_block_bytes = _runtime_state != nullptr
374393
? _runtime_state->preferred_block_size_bytes()
375394
: std::numeric_limits<size_t>::max();
@@ -379,6 +398,7 @@ Status DelimitedTextReader::get_block(Block* file_block, size_t* rows, bool* eof
379398
{
380399
auto columns_guard = file_block->mutate_columns_scoped();
381400
auto& columns = columns_guard.mutable_columns();
401+
_prepare_columns(columns, batch_size);
382402
// Delimited text readers are column-pruned but not lazy materialized: all file-local
383403
// columns requested by TableReader are decoded before file-local conjuncts are evaluated.
384404
while (*rows < batch_size && !_line_reader_eof &&
@@ -437,6 +457,7 @@ Status DelimitedTextReader::get_block(Block* file_block, size_t* rows, bool* eof
437457
Status DelimitedTextReader::get_aggregate_result(const FileAggregateRequest& request,
438458
FileAggregateResult* result) {
439459
SCOPED_TIMER(_text_profile.total_time);
460+
Defer profile_flush {[this] { _flush_profile(); }};
440461
DORIS_CHECK(result != nullptr);
441462
if (request.agg_type != TPushAggOp::type::COUNT) {
442463
return Status::NotSupported("{} v2 reader only supports COUNT aggregate pushdown",
@@ -461,7 +482,7 @@ Status DelimitedTextReader::get_aggregate_result(const FileAggregateRequest& req
461482
break;
462483
}
463484
if (line.size == 0) {
464-
update_counter(_text_profile.empty_lines_read, 1);
485+
++_pending_profile.empty_lines_read;
465486
if (_empty_line_as_record() ||
466487
(_runtime_state != nullptr && _runtime_state->is_read_csv_empty_line_as_null())) {
467488
++count;
@@ -638,7 +659,7 @@ Status DelimitedTextReader::_read_next_line(Slice* line, bool* eof) {
638659
DORIS_CHECK(line != nullptr);
639660
DORIS_CHECK(eof != nullptr);
640661
if (_align_split_prefix) {
641-
SCOPED_TIMER(_text_profile.read_line_time);
662+
SCOPED_TIMER(_text_profile.split_prefix_time);
642663
DCHECK_EQ(_skip_lines, 1);
643664
size_t skipped_lines = 0;
644665
auto* text_reader = assert_cast<NewPlainTextLineReader*>(_line_reader.get());
@@ -648,13 +669,19 @@ Status DelimitedTextReader::_read_next_line(Slice* line, bool* eof) {
648669
_align_split_prefix = false;
649670
_skip_lines = 0;
650671
_bom_removed = true;
651-
update_counter(_text_profile.skipped_lines, skipped_lines);
672+
_pending_profile.skipped_lines += skipped_lines;
652673
}
653674
while (true) {
654675
const uint8_t* ptr = nullptr;
655676
size_t size = 0;
656677
{
657-
SCOPED_TIMER(_text_profile.read_line_time);
678+
_sample_current_line =
679+
_profile != nullptr && (_read_line_calls++ % PROFILE_SAMPLE_INTERVAL == 0);
680+
std::optional<ScopedRawTimer<MonotonicStopWatch, int64_t>> timer;
681+
if (_sample_current_line) {
682+
++_pending_profile.sampled_read_line_calls;
683+
timer.emplace(&_pending_profile.read_line_time);
684+
}
658685
RETURN_IF_ERROR(_line_reader->read_line(&ptr, &size, &_line_reader_eof, _io_ctx.get()));
659686
}
660687
if (_line_reader_eof && size == 0) {
@@ -670,12 +697,12 @@ Status DelimitedTextReader::_read_next_line(Slice* line, bool* eof) {
670697
if (_skip_lines > 0) {
671698
--_skip_lines;
672699
_bom_removed = true;
673-
update_counter(_text_profile.skipped_lines, 1);
700+
++_pending_profile.skipped_lines;
674701
continue;
675702
}
676703
*line = Slice(ptr, size);
677704
*eof = false;
678-
update_counter(_text_profile.raw_lines_read, 1);
705+
++_pending_profile.raw_lines_read;
679706
return Status::OK();
680707
}
681708
}
@@ -685,12 +712,12 @@ Status DelimitedTextReader::_fill_columns_from_line(const Slice& line,
685712
size_t* rows) {
686713
DORIS_CHECK(columns != nullptr);
687714
if (line.size == 0) {
688-
update_counter(_text_profile.empty_lines_read, 1);
715+
++_pending_profile.empty_lines_read;
689716
if (!_empty_line_as_record()) {
690717
if (_runtime_state != nullptr && _runtime_state->is_read_csv_empty_line_as_null()) {
691718
for (const auto& column : _requested_columns) {
692719
RETURN_IF_ERROR(_append_null((*columns)[column.block_position.value()].get()));
693-
update_counter(_text_profile.cells_deserialized, 1);
720+
++_pending_profile.cells_deserialized;
694721
}
695722
++(*rows);
696723
}
@@ -700,10 +727,17 @@ Status DelimitedTextReader::_fill_columns_from_line(const Slice& line,
700727
RETURN_IF_ERROR(_validate_line(line));
701728

702729
{
703-
SCOPED_TIMER(_text_profile.split_line_time);
730+
std::optional<ScopedRawTimer<MonotonicStopWatch, int64_t>> timer;
731+
if (_sample_current_line) {
732+
++_pending_profile.sampled_parsed_rows;
733+
timer.emplace(&_pending_profile.split_line_time);
734+
}
704735
_split_line(line);
705736
}
706-
SCOPED_TIMER(_text_profile.deserialize_time);
737+
std::optional<ScopedRawTimer<MonotonicStopWatch, int64_t>> timer;
738+
if (_sample_current_line) {
739+
timer.emplace(&_pending_profile.deserialize_time);
740+
}
707741
if (_request->reject_invalid_rows) {
708742
const auto expected = cast_set<size_t>(_scan_params->num_of_columns_from_file);
709743
const bool ignore_extra = _scan_params->file_attributes.ignore_csv_redundant_col;
@@ -722,12 +756,31 @@ Status DelimitedTextReader::_fill_columns_from_line(const Slice& line,
722756
? _split_values[field_index]
723757
: Slice(_options.null_format, _options.null_len);
724758
RETURN_IF_ERROR(_deserialize_one_cell(column, output, _normalize_value(value)));
725-
update_counter(_text_profile.cells_deserialized, 1);
759+
++_pending_profile.cells_deserialized;
726760
}
727761
++(*rows);
728762
return Status::OK();
729763
}
730764

765+
void DelimitedTextReader::_flush_profile() {
766+
update_counter(_text_profile.raw_lines_read, _pending_profile.raw_lines_read);
767+
update_counter(_text_profile.empty_lines_read, _pending_profile.empty_lines_read);
768+
update_counter(_text_profile.skipped_lines, _pending_profile.skipped_lines);
769+
update_counter(_text_profile.cells_deserialized, _pending_profile.cells_deserialized);
770+
update_counter(_text_profile.read_line_time, _pending_profile.read_line_time);
771+
update_counter(_text_profile.split_line_time, _pending_profile.split_line_time);
772+
update_counter(_text_profile.deserialize_time, _pending_profile.deserialize_time);
773+
update_counter(_text_profile.sampled_read_line_calls, _pending_profile.sampled_read_line_calls);
774+
update_counter(_text_profile.sampled_parsed_rows, _pending_profile.sampled_parsed_rows);
775+
_pending_profile = {};
776+
}
777+
778+
void DelimitedTextReader::_prepare_columns(std::vector<MutableColumnPtr>& columns,
779+
size_t batch_size) {
780+
(void)columns;
781+
(void)batch_size;
782+
}
783+
731784
Status DelimitedTextReader::_validate_line(const Slice& line) {
732785
(void)line;
733786
return Status::OK();

‎be/src/format_v2/delimited_text/delimited_text_reader.h‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ class DelimitedTextReader : public FileReader {
5555
TableColumnMapperOptions options) const override;
5656
Status open(std::shared_ptr<FileScanRequest> request) override;
5757
Status get_block(Block* file_block, size_t* rows, bool* eof) override;
58+
void set_batch_size(size_t batch_size) override;
5859
Status get_aggregate_result(const FileAggregateRequest& request,
5960
FileAggregateResult* result) override;
6061
Status close() override;
@@ -64,6 +65,9 @@ class DelimitedTextReader : public FileReader {
6465
RuntimeProfile::Counter* total_time = nullptr;
6566
RuntimeProfile::Counter* open_file_time = nullptr;
6667
RuntimeProfile::Counter* create_line_reader_time = nullptr;
68+
RuntimeProfile::Counter* split_prefix_time = nullptr;
69+
RuntimeProfile::Counter* sampled_read_line_calls = nullptr;
70+
RuntimeProfile::Counter* sampled_parsed_rows = nullptr;
6771
RuntimeProfile::Counter* read_line_time = nullptr;
6872
RuntimeProfile::Counter* split_line_time = nullptr;
6973
RuntimeProfile::Counter* deserialize_time = nullptr;
@@ -112,6 +116,8 @@ class DelimitedTextReader : public FileReader {
112116
// Split one logical line into `_split_values`. The common materialization path then resolves
113117
// requested field ordinals against `_split_values`.
114118
virtual void _split_line(const Slice& line) = 0;
119+
// Prepare mutable output columns once per batch, after Block has resolved copy-on-write.
120+
virtual void _prepare_columns(std::vector<MutableColumnPtr>& columns, size_t batch_size);
115121
// Deserialize a single normalized field into the requested output column using the
116122
// format-specific serde API.
117123
virtual Status _deserialize_one_cell(const RequestedColumn& column, IColumn* output,
@@ -178,6 +184,26 @@ class DelimitedTextReader : public FileReader {
178184
Status _fill_columns_from_line(const Slice& line, std::vector<MutableColumnPtr>* columns,
179185
size_t* rows);
180186

187+
void _flush_profile();
188+
189+
// Publish exact counts once per API call instead of updating atomic counters per cell.
190+
struct PendingProfile {
191+
int64_t raw_lines_read = 0;
192+
int64_t empty_lines_read = 0;
193+
int64_t skipped_lines = 0;
194+
int64_t cells_deserialized = 0;
195+
int64_t read_line_time = 0;
196+
int64_t split_line_time = 0;
197+
int64_t deserialize_time = 0;
198+
int64_t sampled_read_line_calls = 0;
199+
int64_t sampled_parsed_rows = 0;
200+
} _pending_profile;
201+
// Keep sampling periodic across batches, including skipped headers and EOF probes.
202+
static constexpr size_t PROFILE_SAMPLE_INTERVAL = 1024;
203+
uint64_t _read_line_calls = 0;
204+
bool _sample_current_line = false;
205+
// Zero means use the runtime default until TableReader supplies its predicted row cap.
206+
size_t _batch_size = 0;
181207
std::string _reader_name;
182208
};
183209

0 commit comments

Comments
 (0)