Repository navigation
Commit ec36f8f
[fix](be) Correct the external row fetch InitReaderAvgTime and GetBlockAvgTime
### What problem does this PR solve?
Issue Number: None
Related PR: #67005
Problem Summary: The second phase of TopN lazy materialization on an
external table (RowIdStorageReader::read_batch_external_row) reads each
scan range of a request block with its own reader and reports two
profile values per request block: InitReaderAvgTime and
GetBlockAvgTime, the mean reader-init and block-read time over those
ranges. MaterializationOperator lists them per backend in the query
profile. Both are wrong on master:
- InitReaderAvgTime assigns each range's init time instead of adding it,
so it reports the last range's time divided by the number of ranges:
ranges that took 10, 20 and 60 ms give 20 ms instead of 30 ms.
- GetBlockAvgTime is filled with the init-time average, so the
block-read time never reaches the profile.
- The averages are accumulated in the caller's variables, which
read_by_rowids() zeroes once per RPC and passes to every request
block. When an RPC fetches two lazily materialized external tables,
the second block's sums start from the first block's averages.
branch-4.1/4.2 fixed the first two in #67005 (a Lance profile PR) by
summing the init times and reporting the get-block average, but kept the
shared accumulators, so a second request block reports inflated
averages there: 46ms and 466ms instead of 30ms and 300ms in the unit test
below once the first block left 50 and 500 ms.
The summary now lives in
RowIdStorageReader::add_external_fetch_avg_time_profile(). It sums both
times over the block's ranges from zero, overwrites the caller's
variables with the averages and reports each average under its own
key. The "Query stats" log line at the end of read_by_rowids() therefore
prints the averages of the RPC's last external request block, the same
block whose scan range count it already prints.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- BE UT RowIdStorageReaderTest.ExternalFetchProfileAveragesInitAndGetBlockTimes:
three ranges (10/100, 20/200, 60/600 ms) with the caller's
variables preset to 50/500 ms by a previous request block give
30ms and 300ms.
- Behavior changed: No
- Does this need documentation: No
Co-authored-by: zhangstar333 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 7c049b0 commit ec36f8f
3 files changed
Lines changed: 66 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
685 | 685 | | |
686 | 686 | | |
687 | 687 | | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
688 | 709 | | |
689 | 710 | | |
690 | 711 | | |
| |||
902 | 923 | | |
903 | 924 | | |
904 | 925 | | |
905 | | - | |
906 | | - | |
907 | 926 | | |
908 | 927 | | |
909 | 928 | | |
| |||
913 | 932 | | |
914 | 933 | | |
915 | 934 | | |
916 | | - | |
917 | | - | |
918 | | - | |
919 | | - | |
920 | | - | |
921 | | - | |
| 935 | + | |
| 936 | + | |
922 | 937 | | |
923 | 938 | | |
924 | 939 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
148 | 149 | | |
149 | 150 | | |
150 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
151 | 162 | | |
152 | 163 | | |
153 | 164 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| |||
301 | 302 | | |
302 | 303 | | |
303 | 304 | | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
304 | 336 | | |
305 | 337 | | |
306 | 338 | | |
| |||
0 commit comments