Repository navigation
Commit 374d87d
authored
[perf](csv reader) Reduce V2 CSV parsing overhead (#69875)
### What problem does this PR solve?
Issue Number: None
Related PR: #69716
Problem Summary:
After restoring sequential prefetch, large V2 CSV loads still spend
substantial CPU time in row parsing and string materialization. The
reader compares a general separator at every byte, updates atomic
profile counters per cell, starts three stage timers per row, and
repeats checked string-column casts per cell. It also ignores the row
cap supplied by TableReader.
This change combines four optimizations on master:
- Compare the separator byte directly for plain single-character
separators; retain enclosed, multi-character, Hive/OpenCSV and PROTO
parsing paths.
- Accumulate exact work counters locally and publish at
get_block/count/schema return, including errors. Sample stage timings
every 1024 physical read_line calls, starting with the first, and expose
unscaled SampledReadLineTime, SampledSplitLineTime and
SampledDeserializeTime with sample counts. DelimitedTextReader remains
full elapsed time; split-prefix alignment has its own full timer.
- Validate nullable string column types and reserve offsets/null maps
once per mutable batch, then insert strings directly with the existing
NULL, quoting and escape semantics.
- Honor FileReader::set_batch_size, retaining the existing byte budget
and runtime default when no explicit cap is supplied. The caller still
controls the maximum batch size; this does not force 16352-row blocks.
Shared counters and batch sizing also apply to the Hive text reader.
### Performance benchmark
On TPC-DS SF100, elapsed time decreased from **1,944 seconds to 1,375
seconds**:
| Benchmark | Before | After | Time reduction | Speedup |
| --- | ---: | ---: | ---: | ---: |
| TPC-DS SF100 | 1,944 s | 1,375 s | 29.3% (569 s) | 1.41× |
### Release note
Reduce V2 CSV parsing overhead and honor caller-provided batch sizes.
Delimited-text per-stage profile timers are now explicitly named
Sampled* and contain unscaled sampled durations; use the full
DelimitedTextReader timer for elapsed-time comparisons.
### Check List (For Author)
- Test
- [x] Unit Test: added cases for dynamic/zero batch caps, byte limits,
filtered blocks, cross-batch sampling and exact counters,
single/multi-character separators, NULL and escape behavior, missing
fields, reordered projection and copy-on-write. Extended error,
rejected-row and COUNT counter coverage.
- Execution: run-be-ut.sh was attempted with CsvV2ReaderTest.*,
TextV2ReaderTest.* and LegacyAndV2/PlainCsvSplitTest.*. It exits before
C++ compilation because the local JDK 17 fails to initialize (Failed
setting boot class path), including outside the sandbox. Full build/UT
remain pending. Performance benchmark results are reported above.
- Passed: repository clang-format 16 formatting/check scripts,
build-support/check-build-hygiene.sh and git diff --check.
- Behavior changed: Yes. Existing batch-size contract is honored, and
detailed stage timers are explicitly sampled. CSV value semantics are
unchanged.
- Does this need documentation: No. The profile includes a timing
explanation; no new configuration or SQL syntax.
For further benchmark comparisons, use identical hardware, input,
compression, cache conditions and load batch/byte-budget settings for
the parent and this commit. Compare total load time, Scanner CPU time,
DelimitedTextReader time and RowsRead/BlockNumber. Do not directly
compare Sampled* timers with the previous full stage timers.1 parent 45d1598 commit 374d87d
6 files changed
Lines changed: 328 additions & 34 deletions
File tree
- be
- src/format_v2/delimited_text
- test/format_v2/delimited_text
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
221 | 222 | | |
222 | 223 | | |
223 | 224 | | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
224 | 233 | | |
225 | 234 | | |
226 | 235 | | |
| |||
236 | 245 | | |
237 | 246 | | |
238 | 247 | | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
239 | 260 | | |
240 | 261 | | |
241 | 262 | | |
242 | 263 | | |
243 | 264 | | |
244 | 265 | | |
245 | 266 | | |
246 | | - | |
247 | | - | |
248 | | - | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
249 | 271 | | |
250 | 272 | | |
251 | 273 | | |
252 | | - | |
| 274 | + | |
253 | 275 | | |
254 | 276 | | |
255 | 277 | | |
256 | 278 | | |
257 | | - | |
| 279 | + | |
| 280 | + | |
258 | 281 | | |
259 | 282 | | |
260 | 283 | | |
| |||
263 | 286 | | |
264 | 287 | | |
265 | 288 | | |
266 | | - | |
| 289 | + | |
| 290 | + | |
267 | 291 | | |
268 | 292 | | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
275 | 297 | | |
| 298 | + | |
276 | 299 | | |
277 | 300 | | |
278 | 301 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| |||
192 | 193 | | |
193 | 194 | | |
194 | 195 | | |
195 | | - | |
| 196 | + | |
196 | 197 | | |
197 | | - | |
198 | | - | |
199 | | - | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
200 | 211 | | |
201 | 212 | | |
202 | 213 | | |
| |||
285 | 296 | | |
286 | 297 | | |
287 | 298 | | |
| 299 | + | |
288 | 300 | | |
289 | 301 | | |
290 | 302 | | |
| |||
360 | 372 | | |
361 | 373 | | |
362 | 374 | | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
363 | 379 | | |
364 | 380 | | |
| 381 | + | |
365 | 382 | | |
366 | 383 | | |
367 | 384 | | |
368 | 385 | | |
369 | 386 | | |
370 | 387 | | |
371 | 388 | | |
372 | | - | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
373 | 392 | | |
374 | 393 | | |
375 | 394 | | |
| |||
379 | 398 | | |
380 | 399 | | |
381 | 400 | | |
| 401 | + | |
382 | 402 | | |
383 | 403 | | |
384 | 404 | | |
| |||
437 | 457 | | |
438 | 458 | | |
439 | 459 | | |
| 460 | + | |
440 | 461 | | |
441 | 462 | | |
442 | 463 | | |
| |||
461 | 482 | | |
462 | 483 | | |
463 | 484 | | |
464 | | - | |
| 485 | + | |
465 | 486 | | |
466 | 487 | | |
467 | 488 | | |
| |||
638 | 659 | | |
639 | 660 | | |
640 | 661 | | |
641 | | - | |
| 662 | + | |
642 | 663 | | |
643 | 664 | | |
644 | 665 | | |
| |||
648 | 669 | | |
649 | 670 | | |
650 | 671 | | |
651 | | - | |
| 672 | + | |
652 | 673 | | |
653 | 674 | | |
654 | 675 | | |
655 | 676 | | |
656 | 677 | | |
657 | | - | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
658 | 685 | | |
659 | 686 | | |
660 | 687 | | |
| |||
670 | 697 | | |
671 | 698 | | |
672 | 699 | | |
673 | | - | |
| 700 | + | |
674 | 701 | | |
675 | 702 | | |
676 | 703 | | |
677 | 704 | | |
678 | | - | |
| 705 | + | |
679 | 706 | | |
680 | 707 | | |
681 | 708 | | |
| |||
685 | 712 | | |
686 | 713 | | |
687 | 714 | | |
688 | | - | |
| 715 | + | |
689 | 716 | | |
690 | 717 | | |
691 | 718 | | |
692 | 719 | | |
693 | | - | |
| 720 | + | |
694 | 721 | | |
695 | 722 | | |
696 | 723 | | |
| |||
700 | 727 | | |
701 | 728 | | |
702 | 729 | | |
703 | | - | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
704 | 735 | | |
705 | 736 | | |
706 | | - | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
707 | 741 | | |
708 | 742 | | |
709 | 743 | | |
| |||
722 | 756 | | |
723 | 757 | | |
724 | 758 | | |
725 | | - | |
| 759 | + | |
726 | 760 | | |
727 | 761 | | |
728 | 762 | | |
729 | 763 | | |
730 | 764 | | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
731 | 784 | | |
732 | 785 | | |
733 | 786 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
| |||
64 | 65 | | |
65 | 66 | | |
66 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
67 | 71 | | |
68 | 72 | | |
69 | 73 | | |
| |||
112 | 116 | | |
113 | 117 | | |
114 | 118 | | |
| 119 | + | |
| 120 | + | |
115 | 121 | | |
116 | 122 | | |
117 | 123 | | |
| |||
178 | 184 | | |
179 | 185 | | |
180 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
181 | 207 | | |
182 | 208 | | |
183 | 209 | | |
| |||
0 commit comments