Skip to content

Commit fc1b822

Browse files
authored
[opt](be) Cache segment row bitmap cardinality (#66742)
`SegmentIterator` recalculates the row bitmap cardinality for every output batch. For large segments, this repeatedly traverses all Roaring bitmap containers and adds significant scan overhead. This change caches the cardinality after all bitmap filters are applied during lazy initialization and reuses it for column reservation and batch reads. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
1 parent 978d5c8 commit fc1b822

2 files changed

Lines changed: 9 additions & 4 deletions

File tree

‎be/src/storage/segment/segment_iterator.cpp‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -584,6 +584,10 @@ Status SegmentIterator::_lazy_init(Block* block) {
584584

585585
RETURN_IF_ERROR(_apply_ann_topn_predicate());
586586

587+
// _row_bitmap is immutable after _range_iter is created. Cache its cardinality to avoid
588+
// traversing all roaring containers for every batch.
589+
_row_bitmap_cardinality = _row_bitmap.cardinality();
590+
587591
if (_opts.read_orderby_key_reverse) {
588592
_range_iter.reset(new BackwardBitmapRangeIterator(_row_bitmap));
589593
} else {
@@ -594,8 +598,7 @@ Status SegmentIterator::_lazy_init(Block* block) {
594598
// prediction) because the predictor may increase block_row_max on subsequent batches
595599
// up to this ceiling. Using the current (possibly reduced) _opts.block_row_max would
596600
// cause heap-buffer-overflow if a later prediction is larger.
597-
auto nrows_reserve_limit =
598-
std::min(_row_bitmap.cardinality(), uint64_t(_initial_block_row_max));
601+
auto nrows_reserve_limit = std::min(_row_bitmap_cardinality, uint64_t(_initial_block_row_max));
599602
_current_columns.resize(_schema->num_read_columns());
600603

601604
for (size_t i = 0; i < _schema->num_read_columns(); i++) {
@@ -619,7 +622,7 @@ Status SegmentIterator::_lazy_init(Block* block) {
619622
// shortcut only changes how fast those default rows are produced.
620623
_count_emit_shortcut = _should_engage_count_emit_shortcut(block);
621624
if (_count_emit_shortcut) {
622-
_count_emit_rows_remaining = _row_bitmap.cardinality();
625+
_count_emit_rows_remaining = _row_bitmap_cardinality;
623626
}
624627

625628
return Status::OK();
@@ -3041,7 +3044,7 @@ Status SegmentIterator::_next_batch_internal(Block* block) {
30413044

30423045
// If the row bitmap size is smaller than nrows_read_limit, there's no need to reserve that many column rows.
30433046
uint32_t nrows_read_limit =
3044-
std::min(cast_set<uint32_t>(_row_bitmap.cardinality()), _opts.block_row_max);
3047+
std::min(cast_set<uint32_t>(_row_bitmap_cardinality), _opts.block_row_max);
30453048
if (_can_opt_limit_reads()) {
30463049
// No SegmentIterator-side conjunct remains to be evaluated, so LIMIT is equivalent before
30473050
// and after filtering. Cap the first read directly; this is the no-conjunct fast path that

‎be/src/storage/segment/segment_iterator.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -368,6 +368,8 @@ class SegmentIterator : public RowwiseIterator {
368368
std::vector<std::unique_ptr<IndexIterator>> _index_iterators;
369369
// after init(), `_row_bitmap` contains all rowid to scan
370370
roaring::Roaring _row_bitmap;
371+
// Cardinality cached after all row bitmap filters are applied in _lazy_init().
372+
uint64_t _row_bitmap_cardinality = 0;
371373
// an iterator for `_row_bitmap` that can be used to extract row range to scan
372374
std::unique_ptr<BitmapRangeIterator> _range_iter;
373375
// the next rowid to read

0 commit comments

Comments
 (0)