Repository navigation
[opt](scan) Use lazy-init for segment iterators and avoid caching all segments in the rowset reader - #35432
Conversation
|
run buildall |
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
clang-tidy review says "All clean, LGTM! 👍" |
| for (size_t i = 0; i < seg_ptrs.size(); i++) { | ||
| (*segment_num_rows)[i] = seg_ptrs[i]->num_rows(); | ||
| } | ||
| segment_num_rows->assign(_segments_rows.cbegin(), _segments_rows.cend()); |
There was a problem hiding this comment.
could you make sure that this method is called after get_segment_iterators??? I think it is very dangerous to make such assumption
TPC-H: Total hot run time: 41019 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 168376 ms |
ClickBench: Total hot run time: 30.37 s |
|
|
||
| Status Segment::new_iterator(SchemaSPtr schema, const StorageReadOptions& read_options, | ||
| std::unique_ptr<RowwiseIterator>* iter) { | ||
| auto st = _create_column_readers_once.call([&] { |
There was a problem hiding this comment.
Should also check in other methods that will use column readers.
|
clang-tidy review says "All clean, LGTM! 👍" |
| status = seg_ptr->new_iterator(_input_schema, local_options, &iter); | ||
| if (_segment_row_ranges.empty()) { | ||
| _read_options.row_ranges.clear(); | ||
| iter = std::make_unique<LazyInitSegmentIterator>(seg_ptr, _input_schema, |
There was a problem hiding this comment.
Add some comment here to explain why we need lazy init segment iterator.
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
… segments in the rowset reader
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40625 ms |
|
|
||
| const bool is_merge_iterator = _is_merge_iterator(); | ||
| const bool use_lazy_init_iterators = | ||
| !is_merge_iterator && _read_context->reader_type == ReaderType::READER_QUERY; |
There was a problem hiding this comment.
why need check reader type?
| } | ||
| segment_num_rows->assign(_segments_rows.cbegin(), _segments_rows.cend()); | ||
| return Status::OK(); | ||
| } |
There was a problem hiding this comment.
some method like new_column_iterator also need load column readers
TPC-DS: Total hot run time: 169491 ms |
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 40107 ms |
TPC-DS: Total hot run time: 169908 ms |
ClickBench: Total hot run time: 30.96 s |
|
TeamCity be ut coverage result: |
… segments in the rowset reader (#35432) ## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (#35432) ## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (#35484) ## Proposed changes pick #35432 ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (apache#35432) ## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (apache#35484) ## Proposed changes pick apache#35432 ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (apache#35484) ## Proposed changes pick apache#35432 ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (apache#35484) ## Proposed changes pick apache#35432 ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segments in the rowset reader (apache#35484) ## Proposed changes pick apache#35432 ## Further comments If this is a relatively large or complex change, kick off the discussion at [[email protected]](mailto:[email protected]) by explaining why you chose the solution you did and what alternatives you considered, etc...
… segment iterators ### What problem does this PR solve? Issue Number: N/A Related PR: apache#4958, apache#7785, apache#35432 Problem Summary: Between BlockReader and SegmentIterator a read went through seven objects of four kinds, and the merge of the rows was done twice: BetaRowsetReader merged the overlapping segments of each rowset with a VMergeIterator, then VCollectIterator merged the rowsets with a two-level heap (the rowset with the most rows against a sub-heap of the others, PR apache#4958, a row-storage era optimization). Every layer had its own way of saying "same key as the previous row" (the segment merge set it on the loser of a comparison and handed it up as a BlockWithSameBit; the rowset merge set it again; Level0Iterator carried it across Blocks), its own tie-break between versions of a key (data_id in the segment merge, the rowset version in the rowset merge), its own end-of-data signal (EndOfFile with rows still in the Block, OK with no rows, row_pos == -1, _rs_reader == nullptr, _cur_child == nullptr) and its own mode flags (Level0Iterator read by Block or by BlockView depending on the child, Level1Iterator merged or concatenated depending on a flag). The three bugs fixed by the overlapping-segments PR all sat on the hand-overs between these layers. The read path is now one iterator per layer, each with one contract: BlockReader captures the rowsets, builds the rows of the read, picks the TabletBlockIterator of the read mode └ TabletBlockIterator folds the rows into output Blocks: Direct, UniqueKey, SequenceMap, AggKey, ChangelogMinDelta, ChangelogDetail, TopN └ VMergeIterator one heap over the segment iterators of every rowset (each or VUnionIterator segment of an overlapping rowset, one VUnionIterator for a rowset whose segments do not overlap), in (rowset version, segment) order └ LazyInitSegmentIterator -> SegmentIterator - RowwiseIterator has two ways to read: next_batch(Block*) and next_row(IteratorRowRef*). next_batch returns OK only with rows and EndOfFile only with none; next_row hands out one row whose reference lives until the next call, with is_same set when the key repeats the previous row's. BlockWithSameBit, BlockView, is_merge_iterator(), data_id() and empty() are gone. - VMergeIterator is rewritten: a flat binary heap of child indexes, the child order (rowset version, then segment) as the only tie-break after the sequence column, and run detection: the top child advances without touching the heap while its next row still sorts before the best row of any other child. The same-key bit and skip_same are computed in one place, against the row handed out before. next_batch copies runs with insert_range_from. - VCollectIterator, Level0Iterator and Level1Iterator are deleted. The scanner-local TopN read moves into TopNBlockIterator, reading one key-ordered iterator per rowset; the best n rows seen so far live in its Candidates helper (offer a batch, trim to n, worst value for the TopN runtime predicate), the same logic as _topn_next had. - RowsetReader is a factory of segment iterators (get_segment_iterators, new_iterator); the per-rowset fields it wrote into the shared RowsetReaderContext (rowset_id, merged_rows) are gone. next_batch stays for the readers of one rowset (schema change). - SegmentIterator keeps the batch limit of the adaptive batch size in its own member instead of rewriting _opts.block_row_max, reads on over batches a predicate filtered completely so next_batch never returns OK without rows (checking for cancellation on the way), and asserts it is not read after EndOfFile. - TabletReader::next_block_with_aggregation sets eof only with an empty Block; the scanner's workaround for an eof with rows is removed. Every read returns the same rows in the same order. What changes around them: the byte budget of an output Block is checked after every copied run and every 64 rows instead of being predicted per row, so a Block may end a little later; eof is reported with an empty Block instead of together with the last rows; a LazyInitSegmentIterator opens its segment with the io context of its rowset reader. The profile counters keep their names. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test - BE UT: 531 tests of the storage read path (BlockReader*, VGenericIteratorsTest*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, TabletReaderTest*, SegmentIterator*, SegmentsKeyBoundsTruncation*, IndexBuilder*, ...). Rewritten: vgeneric_iterators_test (merge/union tests on the new interfaces, plus next_row, skip_same, row-limit and union next_row tests), block_reader_changelog_merge_test (a real VMergeIterator of two rowsets under the changelog iterators), block_reader_change_next_block_test (fake rows through next_row, eof only with an empty Block). Kept: block_reader_table_models_test (every table model and read mode on real segments), block_reader_overlapping_segments_test, agg_value_aggregator_test and the compaction, schema change, rowid conversion and segcompaction suites. The key bounds, expr zone map and index builder tests follow the renamed or removed members. - Regression on a one-BE cluster with the p0 pipeline settings: fault_injection_p0 (test_overlapping_segments_merge, test_topn_pruned_struct_overlapping_segments), unique_with_mor_p0, unique_seq_map_p0, topn_optimize, delete_p0, data_model_p0, changelog_p0 (25/25), schema_change_p0 (125/126), compaction (56/58), unique_with_mow_p0 (107/109); the 6 suites not passed all need an S3 bucket for a broker load or for cold storage, which this environment has no access to. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Fable 5.1 <[email protected]>
### What problem does this PR solve? Issue Number: N/A Related PR: apache#35432, apache#54936 Problem Summary: One load that flushes several memtables writes a rowset whose segments overlap on keys, so the same key can sit in two segments of one rowset. Reading such a rowset merges its segments in BetaRowsetReader (VMergeIterator) before VCollectIterator merges the rowsets, and three bugs on that path return wrong rows for merge-on-read tables, in queries and in horizontal compaction. 1. The older row of a key wins (merge-on-read UNIQUE_KEYS, REPLACE columns of AGG_KEYS). The segment merge orders the rows of one key and one sequence value by the data_id of their iterators, so that the segment flushed later supplies the newer row. Since apache#35432 the merge reads LazyInitSegmentIterators, which did not override data_id() and all answered 0, so the tie fell to the heap order and a read could keep the row of the older segment. Vertical compaction orders by iterator position and kept the newer row, so the result of a query also changed after compaction. LazyInitSegmentIterator now answers the id of its segment, the same value SegmentIterator::data_id() reports. 2. A key group cut by a Block boundary is returned twice (AGG_KEYS, and sequence-map tables). The VCollectIterator::Level0Iterator of the lowest-version rowset reads it Block by Block together with the same-key bits of its segment merge, and took the bit of the next row before refresh_current_row() had moved to the next Block, so the first row of every Block lost its bit and BlockReader started a new key group there. The bit is now taken after the refresh. 3. A sequence-map table loses column groups (apache#54936). Such a table needs every row of a key, because BlockReader takes each column group from the row with that group's largest sequence value; VCollectIterator therefore does not skip same-key rows of it, but TabletReader still asked the segment merge to keep only the newest row of a key (is_unique). A column group whose largest sequence value was in another segment lost its values. The segment merge now keeps every row of a sequence-map table. Reproduction (regression test test_overlapping_segments_merge): with the MemTable.need_flush debug point and batch_size = 20, load numbers 0..39 as k = number % 20 into an AGG_KEYS table, a merge-on-read UNIQUE_KEYS table and a sequence-map table, so that keys 0..19 sit in both of two overlapping segments and the second segment holds the newer rows; then read with batch_size = 3. - Before: the AGG_KEYS table returned 7 of the 20 keys twice and its REPLACE column took the older value, the UNIQUE_KEYS table returned the older value of every key, and the sequence-map table returned the older row's v2 group for every key. After a full compaction the AGG_KEYS and UNIQUE_KEYS tables (vertical compaction) were right, while the sequence-map table (horizontal compaction) kept the wrong values. - After: every key comes back once with the newer values, and every column group of the sequence-map table with its largest-sequence values, before and after the full compaction. ### Release note Fix wrong query results and compaction output of merge-on-read UNIQUE KEY tables (including tables with sequence mapping) and AGGREGATE KEY tables when one load wrote the same key into several segments of one rowset. ### Check List (For Author) - Test: Regression test / Unit Test - New BE UT block_reader_overlapping_segments_test.cpp: all 4 tests fail before the fix and pass after it. Also ran BlockReader*, VCollect*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, VerticalMergeIteratorTest*, OrderedDataCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, EstimateCollectedEnough* and CollectedEnough*: 254 tests pass. be/test/exprs/function/function_string_test.cpp was left out of the local UT build because it does not compile on master (FunctionContext::FRAGMENT_LOCAL, from apache#69223). - New regression test fault_injection_p0/test_overlapping_segments_merge: run on a build without the fix it returned the wrong rows above; with the fix it passes. - Behavior changed: Yes. Reads of such rowsets return the newer row of a key, fold every key group once, and keep every column group of a sequence-map table. Rowsets that an older version already compacted keep what they were written with. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
… segment iterators Issue Number: N/A Related PR: apache#4958, apache#7785, apache#35432 Problem Summary: Between BlockReader and SegmentIterator a read went through seven objects of four kinds, and the merge of the rows was done twice: BetaRowsetReader merged the overlapping segments of each rowset with a VMergeIterator, then VCollectIterator merged the rowsets with a two-level heap (the rowset with the most rows against a sub-heap of the others, PR apache#4958, a row-storage era optimization). Every layer had its own way of saying "same key as the previous row" (the segment merge set it on the loser of a comparison and handed it up as a BlockWithSameBit; the rowset merge set it again; Level0Iterator carried it across Blocks), its own tie-break between versions of a key (data_id in the segment merge, the rowset version in the rowset merge), its own end-of-data signal (EndOfFile with rows still in the Block, OK with no rows, row_pos == -1, _rs_reader == nullptr, _cur_child == nullptr) and its own mode flags (Level0Iterator read by Block or by BlockView depending on the child, Level1Iterator merged or concatenated depending on a flag). The three bugs fixed by the overlapping-segments PR all sat on the hand-overs between these layers. The read path is now one iterator per layer, each with one contract: BlockReader captures the rowsets, builds the rows of the read, picks the TabletBlockIterator of the read mode └ TabletBlockIterator folds the rows into output Blocks: Direct, UniqueKey, SequenceMap, AggKey, ChangelogMinDelta, ChangelogDetail, TopN └ VMergeIterator one heap over the segment iterators of every rowset (each or VUnionIterator segment of an overlapping rowset, one VUnionIterator for a rowset whose segments do not overlap), in (rowset version, segment) order └ LazyInitSegmentIterator -> SegmentIterator - RowwiseIterator has two ways to read: next_batch(Block*) and next_row(IteratorRowRef*). next_batch returns OK only with rows and EndOfFile only with none; next_row hands out one row whose reference lives until the next call, with is_same set when the key repeats the previous row's. BlockWithSameBit, BlockView, is_merge_iterator(), data_id() and empty() are gone. - VMergeIterator is rewritten: a flat binary heap of child indexes, the child order (rowset version, then segment) as the only tie-break after the sequence column, and run detection: the top child advances without touching the heap while its next row still sorts before the best row of any other child. The same-key bit and skip_same are computed in one place, against the row handed out before. next_batch copies runs with insert_range_from. - VCollectIterator, Level0Iterator and Level1Iterator are deleted. The scanner-local TopN read moves into TopNBlockIterator, reading one key-ordered iterator per rowset; the best n rows seen so far live in its Candidates helper (offer a batch, trim to n, worst value for the TopN runtime predicate), the same logic as _topn_next had. - RowsetReader is a factory of segment iterators (get_segment_iterators, new_iterator); the per-rowset fields it wrote into the shared RowsetReaderContext (rowset_id, merged_rows) are gone. next_batch stays for the readers of one rowset (schema change). - SegmentIterator keeps the batch limit of the adaptive batch size in its own member instead of rewriting _opts.block_row_max, reads on over batches a predicate filtered completely so next_batch never returns OK without rows (checking for cancellation on the way), and asserts it is not read after EndOfFile. - TabletReader::next_block_with_aggregation sets eof only with an empty Block; the scanner's workaround for an eof with rows is removed. Every read returns the same rows in the same order. What changes around them: the byte budget of an output Block is checked after every copied run and every 64 rows instead of being predicted per row, so a Block may end a little later; eof is reported with an empty Block instead of together with the last rows; a LazyInitSegmentIterator opens its segment with the io context of its rowset reader. The profile counters keep their names. None - Test: Unit Test / Regression test - BE UT: 531 tests of the storage read path (BlockReader*, VGenericIteratorsTest*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, TabletReaderTest*, SegmentIterator*, SegmentsKeyBoundsTruncation*, IndexBuilder*, ...). Rewritten: vgeneric_iterators_test (merge/union tests on the new interfaces, plus next_row, skip_same, row-limit and union next_row tests), block_reader_changelog_merge_test (a real VMergeIterator of two rowsets under the changelog iterators), block_reader_change_next_block_test (fake rows through next_row, eof only with an empty Block). Kept: block_reader_table_models_test (every table model and read mode on real segments), block_reader_overlapping_segments_test, agg_value_aggregator_test and the compaction, schema change, rowid conversion and segcompaction suites. The key bounds, expr zone map and index builder tests follow the renamed or removed members. - Regression on a one-BE cluster with the p0 pipeline settings: fault_injection_p0 (test_overlapping_segments_merge, test_topn_pruned_struct_overlapping_segments), unique_with_mor_p0, unique_seq_map_p0, topn_optimize, delete_p0, data_model_p0, changelog_p0 (25/25), schema_change_p0 (125/126), compaction (56/58), unique_with_mow_p0 (107/109); the 6 suites not passed all need an S3 bucket for a broker load or for cold storage, which this environment has no access to. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Fable 5.1 <[email protected]>
### What problem does this PR solve? Issue Number: N/A Related PR: apache#35432, apache#54936 Problem Summary: One load that flushes several memtables writes a rowset whose segments overlap on keys, so the same key can sit in two segments of one rowset. Reading such a rowset merges its segments in BetaRowsetReader (VMergeIterator) before VCollectIterator merges the rowsets, and three bugs on that path return wrong rows for merge-on-read tables, in queries and in horizontal compaction. 1. The older row of a key wins (merge-on-read UNIQUE_KEYS, REPLACE columns of AGG_KEYS). The segment merge orders the rows of one key and one sequence value by the data_id of their iterators, so that the segment flushed later supplies the newer row. Since apache#35432 the merge reads LazyInitSegmentIterators, which did not override data_id() and all answered 0, so the tie fell to the heap order and a read could keep the row of the older segment. Vertical compaction orders by iterator position and kept the newer row, so the result of a query also changed after compaction. LazyInitSegmentIterator now answers the id of its segment, the same value SegmentIterator::data_id() reports. 2. A key group cut by a Block boundary is returned twice (AGG_KEYS, and sequence-map tables). The VCollectIterator::Level0Iterator of the lowest-version rowset reads it Block by Block together with the same-key bits of its segment merge, and took the bit of the next row before refresh_current_row() had moved to the next Block, so the first row of every Block lost its bit and BlockReader started a new key group there. The bit is now taken after the refresh. 3. A sequence-map table loses column groups (apache#54936). Such a table needs every row of a key, because BlockReader takes each column group from the row with that group's largest sequence value; VCollectIterator therefore does not skip same-key rows of it, but TabletReader still asked the segment merge to keep only the newest row of a key (is_unique). A column group whose largest sequence value was in another segment lost its values. The segment merge now keeps every row of a sequence-map table. Reproduction (regression test test_overlapping_segments_merge): with the MemTable.need_flush debug point and batch_size = 20, load numbers 0..39 as k = number % 20 into an AGG_KEYS table, a merge-on-read UNIQUE_KEYS table and a sequence-map table, so that keys 0..19 sit in both of two overlapping segments and the second segment holds the newer rows; then read with batch_size = 3. - Before: the AGG_KEYS table returned 7 of the 20 keys twice and its REPLACE column took the older value, the UNIQUE_KEYS table returned the older value of every key, and the sequence-map table returned the older row's v2 group for every key. After a full compaction the AGG_KEYS and UNIQUE_KEYS tables (vertical compaction) were right, while the sequence-map table (horizontal compaction) kept the wrong values. - After: every key comes back once with the newer values, and every column group of the sequence-map table with its largest-sequence values, before and after the full compaction. ### Release note Fix wrong query results and compaction output of merge-on-read UNIQUE KEY tables (including tables with sequence mapping) and AGGREGATE KEY tables when one load wrote the same key into several segments of one rowset. ### Check List (For Author) - Test: Regression test / Unit Test - New BE UT block_reader_overlapping_segments_test.cpp: all 4 tests fail before the fix and pass after it. Also ran BlockReader*, VCollect*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, VerticalMergeIteratorTest*, OrderedDataCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, EstimateCollectedEnough* and CollectedEnough*: 254 tests pass. be/test/exprs/function/function_string_test.cpp was left out of the local UT build because it does not compile on master (FunctionContext::FRAGMENT_LOCAL, from apache#69223). - New regression test fault_injection_p0/test_overlapping_segments_merge: run on a build without the fix it returned the wrong rows above; with the fix it passes. - Behavior changed: Yes. Reads of such rowsets return the newer row of a key, fold every key group once, and keep every column group of a sequence-map table. Rowsets that an older version already compacted keep what they were written with. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
… segment iterators Issue Number: N/A Related PR: apache#4958, apache#7785, apache#35432 Problem Summary: Between BlockReader and SegmentIterator a read went through seven objects of four kinds, and the merge of the rows was done twice: BetaRowsetReader merged the overlapping segments of each rowset with a VMergeIterator, then VCollectIterator merged the rowsets with a two-level heap (the rowset with the most rows against a sub-heap of the others, PR apache#4958, a row-storage era optimization). Every layer had its own way of saying "same key as the previous row" (the segment merge set it on the loser of a comparison and handed it up as a BlockWithSameBit; the rowset merge set it again; Level0Iterator carried it across Blocks), its own tie-break between versions of a key (data_id in the segment merge, the rowset version in the rowset merge), its own end-of-data signal (EndOfFile with rows still in the Block, OK with no rows, row_pos == -1, _rs_reader == nullptr, _cur_child == nullptr) and its own mode flags (Level0Iterator read by Block or by BlockView depending on the child, Level1Iterator merged or concatenated depending on a flag). The three bugs fixed by the overlapping-segments PR all sat on the hand-overs between these layers. The read path is now one iterator per layer, each with one contract: BlockReader captures the rowsets, builds the rows of the read, picks the TabletBlockIterator of the read mode └ TabletBlockIterator folds the rows into output Blocks: Direct, UniqueKey, SequenceMap, AggKey, ChangelogMinDelta, ChangelogDetail, TopN └ VMergeIterator one heap over the segment iterators of every rowset (each or VUnionIterator segment of an overlapping rowset, one VUnionIterator for a rowset whose segments do not overlap), in (rowset version, segment) order └ LazyInitSegmentIterator -> SegmentIterator - RowwiseIterator has two ways to read: next_batch(Block*) and next_row(IteratorRowRef*). next_batch returns OK only with rows and EndOfFile only with none; next_row hands out one row whose reference lives until the next call, with is_same set when the key repeats the previous row's. BlockWithSameBit, BlockView, is_merge_iterator(), data_id() and empty() are gone. - VMergeIterator is rewritten: a flat binary heap of child indexes, the child order (rowset version, then segment) as the only tie-break after the sequence column, and run detection: the top child advances without touching the heap while its next row still sorts before the best row of any other child. The same-key bit and skip_same are computed in one place, against the row handed out before. next_batch copies runs with insert_range_from. - VCollectIterator, Level0Iterator and Level1Iterator are deleted. The scanner-local TopN read moves into TopNBlockIterator, reading one key-ordered iterator per rowset; the best n rows seen so far live in its Candidates helper (offer a batch, trim to n, worst value for the TopN runtime predicate), the same logic as _topn_next had. - RowsetReader is a factory of segment iterators (get_segment_iterators, new_iterator); the per-rowset fields it wrote into the shared RowsetReaderContext (rowset_id, merged_rows) are gone. next_batch stays for the readers of one rowset (schema change). - SegmentIterator keeps the batch limit of the adaptive batch size in its own member instead of rewriting _opts.block_row_max, reads on over batches a predicate filtered completely so next_batch never returns OK without rows (checking for cancellation on the way), and asserts it is not read after EndOfFile. - TabletReader::next_block_with_aggregation sets eof only with an empty Block; the scanner's workaround for an eof with rows is removed. Every read returns the same rows in the same order. What changes around them: the byte budget of an output Block is checked after every copied run and every 64 rows instead of being predicted per row, so a Block may end a little later; eof is reported with an empty Block instead of together with the last rows; a LazyInitSegmentIterator opens its segment with the io context of its rowset reader. The profile counters keep their names. None - Test: Unit Test / Regression test - BE UT: 531 tests of the storage read path (BlockReader*, VGenericIteratorsTest*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, TabletReaderTest*, SegmentIterator*, SegmentsKeyBoundsTruncation*, IndexBuilder*, ...). Rewritten: vgeneric_iterators_test (merge/union tests on the new interfaces, plus next_row, skip_same, row-limit and union next_row tests), block_reader_changelog_merge_test (a real VMergeIterator of two rowsets under the changelog iterators), block_reader_change_next_block_test (fake rows through next_row, eof only with an empty Block). Kept: block_reader_table_models_test (every table model and read mode on real segments), block_reader_overlapping_segments_test, agg_value_aggregator_test and the compaction, schema change, rowid conversion and segcompaction suites. The key bounds, expr zone map and index builder tests follow the renamed or removed members. - Regression on a one-BE cluster with the p0 pipeline settings: fault_injection_p0 (test_overlapping_segments_merge, test_topn_pruned_struct_overlapping_segments), unique_with_mor_p0, unique_seq_map_p0, topn_optimize, delete_p0, data_model_p0, changelog_p0 (25/25), schema_change_p0 (125/126), compaction (56/58), unique_with_mow_p0 (107/109); the 6 suites not passed all need an S3 bucket for a broker load or for cold storage, which this environment has no access to. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Fable 5.1 <[email protected]>
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...