Repository navigation
[opt](file cache) Add PageIO read-ahead, I/O coalescing, and cache block hole filling #67292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
bobhan1
merged 21 commits into
apache:master
from
bobhan1:feature/read-ahead-io-coalescing-phase2
Oct 30, 2026
Merged
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
7654cf7
[feature](be) Add linear file range coalescing
bobhan1 90cf5b6
[feature](be) Add cache-aware file range planning
bobhan1 85894d0
[feature](be) Add file range read scheduler
bobhan1 c64e719
[feature](be) Reuse remote-only reads for no-write ranges
bobhan1 563b5ba
[feature](be) Add query read-ahead runtime ownership
bobhan1 e2a4012
[improvement](be) Add opt-in file range IO tracing
bobhan1 8a08ae1
[fix](be) Scope filesystem error-code imports to their functions
bobhan1 4271da9
[feature](be) Add page-driven column read-ahead windows
bobhan1 e7dbfa1
[feature](be) Add segment range read-ahead coordination
bobhan1 00fa56b
[feature](be) Connect scalar column readers to range read-ahead
bobhan1 8190e0f
[feature](be) Initialize scan read-ahead once after pruning
bobhan1 d03f274
[feature](be) Add complete-block cache write handoff and admission
bobhan1 a547700
[feature](be) Split consumed ranges for cache writeback
bobhan1 7cfe76c
[feature](be) Add cache block hole-fill planning
bobhan1 a7e3899
[feature](be) Add background partial block writeback
bobhan1 999f5c6
[feature](be) Initialize background hole fill
bobhan1 6fd7bb6
[feature](be) Route consumed ranges to cache writeback
bobhan1 1ae1272
[feature](be) Connect read-ahead range writeback
bobhan1 fd6df2c
[feature](be) Extend range read-ahead to point queries
bobhan1 a881473
[fix](be) Validate S3 upload buffer alignment on config updates
bobhan1 b237ebc
[fix](be) Handle irregular cache blocks in writeback coverage checks
bobhan1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
[feature](be) Add segment range read-ahead coordination
### What problem does this PR solve? Issue Number: #67611 (tracking) Related PR: #67292, #67579, #67612 Problem Summary: Coordinate all resolvable column plans, probe Page Cache, plan and submit ranges, and register exact page slices. Reuse a physical page's existing buffer when independent column windows overlap it. Keep synchronous fallback and per-page ownership accounting; expose planner and consumption statistics. Register only ranges admitted by the scheduler, preserving the planner's original range indices and buffer offsets. Complete unadmitted pages immediately so PageIO uses the original reader, and publish consumed-range callbacks only for admitted buffers. The window, coalescing, block-fill and hole-fill policies are unchanged. The full read-ahead feature adds six translation-unit users of the RLE header: its segment/row-id helpers, query/segment tests, and cached-reader coverage. Preserve master 8d88f1c's updated baseline and add only those six users, from 828 to 834 (limit 918), while keeping the dependency rules and 10% slack unchanged. The complete rebased feature measures 916 users. This avoids mixing a global expression/reader header refactor into the feature. ### Release note Internal segment-level coordination for concurrent data-page reads. ### Check List (For Author) - Test: Unit Test — integrated ASAN/PCH validation covers 141 tests from 8 suites through ./run-be-ut.sh --run -j100. Includes partial budget admission, original page offsets, cancellation, allocation rollback, executor rejection, consumed-range callbacks and row-store switch fallback. Individual intermediate commits are not built separately; target-branch validation is recorded on the PR. Clang-format 16, header hygiene and diff checks pass. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in #67611 and the related PR descriptions.
- Loading branch information
commit e7dbfa1d4503acd16d6b2495bcfbf7a966e050fb
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,322 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #include "storage/segment/segment_read_ahead.h" | ||
|
|
||
| #include <butil/iobuf.h> | ||
|
|
||
| #include <algorithm> | ||
| #include <cstring> | ||
| #include <utility> | ||
|
|
||
| #include "common/logging.h" | ||
| #include "io/fs/read_ahead_metrics.h" | ||
| #include "storage/cache/page_cache.h" | ||
|
|
||
| namespace doris::segment_v2 { | ||
| namespace { | ||
|
|
||
| struct PendingPage { | ||
| ColumnReadAhead* column {nullptr}; | ||
| ColumnReadAheadPage page; | ||
| }; | ||
|
|
||
| } // namespace | ||
|
|
||
| SegmentReadAheadFileReader::SegmentReadAheadFileReader( | ||
| io::FileReaderSPtr inner, std::shared_ptr<io::ReadAheadStatistics> statistics) | ||
| : _inner(std::move(inner)), _statistics(std::move(statistics)) { | ||
| DORIS_CHECK(_inner != nullptr); | ||
| } | ||
|
|
||
| void SegmentReadAheadFileReader::_register_page(ColumnReadAhead* column, | ||
| const ColumnReadAheadPage& page, | ||
| std::shared_ptr<BufferedRange> range, | ||
| size_t buffer_offset) { | ||
| DORIS_CHECK(column != nullptr); | ||
| DORIS_CHECK(range != nullptr); | ||
| std::lock_guard lock(_mutex); | ||
| const PageKey key {.offset = page.range.offset, .size = page.range.size}; | ||
| auto [slot, inserted] = _pages.try_emplace(key); | ||
| if (inserted) { | ||
| slot->second.range = std::move(range); | ||
| slot->second.buffer_offset = buffer_offset; | ||
| } else { | ||
| DORIS_CHECK(slot->second.range == range); | ||
| DORIS_CHECK(slot->second.buffer_offset == buffer_offset); | ||
| } | ||
| const PageOwner owner {.column = column, .page_index = page.page_index}; | ||
| DORIS_CHECK(std::ranges::find(slot->second.owners, owner) == slot->second.owners.end()); | ||
| slot->second.owners.push_back(owner); | ||
| } | ||
|
|
||
| bool SegmentReadAheadFileReader::_reuse_page(ColumnReadAhead* column, | ||
| const ColumnReadAheadPage& page) { | ||
| std::lock_guard lock(_mutex); | ||
| auto slot = _pages.find({.offset = page.range.offset, .size = page.range.size}); | ||
| if (slot == _pages.end()) { | ||
| return false; | ||
| } | ||
| const PageOwner owner {.column = column, .page_index = page.page_index}; | ||
| DCHECK(std::ranges::find(slot->second.owners, owner) == slot->second.owners.end()); | ||
| slot->second.owners.push_back(owner); | ||
| return true; | ||
| } | ||
|
|
||
| void SegmentReadAheadFileReader::_release_page(ColumnReadAhead* column, | ||
| const ColumnReadAheadPage& page) { | ||
| DORIS_CHECK(column != nullptr); | ||
| std::lock_guard lock(_mutex); | ||
| const PageKey key {.offset = page.range.offset, .size = page.range.size}; | ||
| auto slot = _pages.find(key); | ||
| if (slot == _pages.end()) { | ||
| return; | ||
| } | ||
| const PageOwner owner {.column = column, .page_index = page.page_index}; | ||
| auto entry = std::ranges::find(slot->second.owners, owner); | ||
| if (entry == slot->second.owners.end()) { | ||
| return; | ||
| } | ||
| slot->second.owners.erase(entry); | ||
| if (slot->second.owners.empty()) { | ||
| _pages.erase(slot); | ||
| } | ||
| } | ||
|
|
||
| bool SegmentReadAheadFileReader::try_read_page(size_t offset, Slice output, Status* status) { | ||
| const PageKey key {.offset = offset, .size = output.size}; | ||
| std::shared_ptr<BufferedRange> range; | ||
| size_t buffer_offset = 0; | ||
| { | ||
| std::lock_guard lock(_mutex); | ||
| const auto slot = _pages.find(key); | ||
| if (slot == _pages.end()) { | ||
| return false; | ||
| } | ||
| range = slot->second.range; | ||
| buffer_offset = slot->second.buffer_offset; | ||
| } | ||
|
|
||
| bool ready = false; | ||
| { | ||
| SCOPED_TIMER(&_statistics->wait_time); | ||
| *status = range->read->wait(&ready); | ||
| } | ||
| if (!status->ok()) { | ||
| _finish_page(key, false); | ||
| return true; | ||
| } | ||
| DORIS_CHECK(output.size == key.size); | ||
| const Slice source = range->read->slice(buffer_offset, key.size); | ||
| std::memcpy(output.data, source.data, source.size); | ||
| auto& hits = ready ? _statistics->ready_page_hits : _statistics->wait_page_hits; | ||
| COUNTER_UPDATE(&hits, 1); | ||
| COUNTER_UPDATE(&_statistics->consumed_page_bytes, static_cast<int64_t>(source.size)); | ||
| io::read_ahead_bvars().consumed_page_bytes << static_cast<int64_t>(source.size); | ||
| _finish_page(key, true); | ||
| return true; | ||
| } | ||
|
|
||
| void SegmentReadAheadFileReader::_finish_page(const PageKey& key, bool consumed) { | ||
| std::vector<PageOwner> owners; | ||
| std::shared_ptr<BufferedRange> consumed_range; | ||
| { | ||
| std::lock_guard lock(_mutex); | ||
| auto slot = _pages.find(key); | ||
| if (slot == _pages.end()) { | ||
| return; | ||
| } | ||
| owners = std::move(slot->second.owners); | ||
| auto range = std::move(slot->second.range); | ||
| _pages.erase(slot); | ||
| if (consumed && !range->consumed) { | ||
| range->consumed = true; | ||
| COUNTER_UPDATE(&_statistics->consumed_range_bytes, | ||
| static_cast<int64_t>(range->read->range().size)); | ||
| consumed_range = std::move(range); | ||
| } | ||
| } | ||
|
|
||
| for (const auto& owner : owners) { | ||
| owner.column->complete(owner.page_index); | ||
| } | ||
| if (consumed_range != nullptr && consumed_range->consumer) { | ||
| auto consumer = std::move(consumed_range->consumer); | ||
| consumer(consumed_range->read->range(), consumed_range->read->data()); | ||
| } | ||
| } | ||
|
|
||
| Status SegmentReadAheadFileReader::read_at_impl(size_t offset, Slice result, size_t* bytes_read, | ||
| const io::IOContext* io_ctx) { | ||
| Status read_ahead_status; | ||
| if (try_read_page(offset, result, &read_ahead_status)) { | ||
| if (read_ahead_status.ok()) { | ||
| *bytes_read = result.size; | ||
| return Status::OK(); | ||
| } | ||
| } | ||
| return _inner->read_at(offset, result, bytes_read, io_ctx); | ||
| } | ||
|
|
||
| Status SegmentReadAheadFileReader::read_at_iobuf_impl(size_t offset, size_t bytes_req, | ||
| butil::IOBuf* out, size_t* bytes_read, | ||
| const io::IOContext* io_ctx) { | ||
| return _inner->read_at_iobuf(offset, bytes_req, out, bytes_read, io_ctx); | ||
| } | ||
|
|
||
| SegmentReadAhead::SegmentReadAhead(io::FileReaderSPtr source_reader, | ||
| io::FileRangeReadScheduler* scheduler, | ||
| std::shared_ptr<io::FileRangeReadContext> context, | ||
| io::FileRangeReadIOContext io_context, | ||
| SegmentReadAheadOptions options) | ||
| : _source_reader(std::move(source_reader)), | ||
| _scheduler(scheduler), | ||
| _context(std::move(context)), | ||
| _io_context(std::move(io_context)), | ||
| _options(std::move(options)), | ||
| _statistics(_io_context.statistics != nullptr | ||
| ? _io_context.statistics | ||
| : std::make_shared<io::ReadAheadStatistics>()), | ||
| _reader(std::shared_ptr<SegmentReadAheadFileReader>( | ||
| new SegmentReadAheadFileReader(_source_reader, _statistics))) { | ||
| DORIS_CHECK(_source_reader != nullptr); | ||
| DORIS_CHECK(_scheduler != nullptr); | ||
| DORIS_CHECK(_context != nullptr); | ||
| _io_context.statistics = _statistics; | ||
| } | ||
|
|
||
| SegmentReadAheadResult SegmentReadAhead::apply_plans(std::vector<ColumnReadAheadPlan> plans) { | ||
| COUNTER_UPDATE(&_statistics->plan_calls, 1); | ||
| SCOPED_TIMER(&_statistics->plan_time); | ||
| SegmentReadAheadResult result; | ||
| std::vector<PendingPage> misses; | ||
|
|
||
| for (auto& plan : plans) { | ||
| DORIS_CHECK(plan.column != nullptr); | ||
| for (const auto& page : plan.released_pages) { | ||
| _reader->_release_page(plan.column, page); | ||
| } | ||
| COUNTER_UPDATE(&_statistics->released_pages, | ||
| static_cast<int64_t>(plan.released_pages.size())); | ||
| COUNTER_UPDATE(&_statistics->candidate_pages, static_cast<int64_t>(plan.new_pages.size())); | ||
| for (const auto& page : plan.new_pages) { | ||
| ++result.new_pages; | ||
| COUNTER_UPDATE(&_statistics->candidate_bytes, static_cast<int64_t>(page.range.size)); | ||
| if (_options.page_cache_probe && _options.page_cache_probe(page.range)) { | ||
| plan.column->complete(page.page_index); | ||
| ++result.page_cache_hits; | ||
| COUNTER_UPDATE(&_statistics->page_cache_hits, 1); | ||
| COUNTER_UPDATE(&_statistics->page_cache_hit_bytes, | ||
| static_cast<int64_t>(page.range.size)); | ||
| } else if (!_reader->_reuse_page(plan.column, page)) { | ||
| misses.push_back({.column = plan.column, .page = page}); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (misses.empty()) { | ||
| return result; | ||
| } | ||
|
|
||
| std::ranges::sort(misses, [](const PendingPage& left, const PendingPage& right) { | ||
| return left.page.range.offset < right.page.range.offset || | ||
| (left.page.range.offset == right.page.range.offset && | ||
| left.page.range.size < right.page.range.size); | ||
| }); | ||
| std::vector<io::FileRange> input_ranges; | ||
| input_ranges.reserve(misses.size()); | ||
| for (const auto& miss : misses) { | ||
| if (input_ranges.empty() || input_ranges.back() != miss.page.range) { | ||
| input_ranges.push_back(miss.page.range); | ||
| } | ||
| } | ||
|
|
||
| io::FileRangePlan range_plan; | ||
| result.status = io::FileRangePlanner::plan(input_ranges, _source_reader->size(), | ||
| _options.range_plan, &range_plan, _statistics.get()); | ||
| if (!result.status.ok()) { | ||
| for (const auto& miss : misses) { | ||
| miss.column->complete(miss.page.page_index); | ||
| } | ||
| return result; | ||
| } | ||
|
|
||
| ReadAheadRangeConsumer consumer; | ||
| if (_options.range_consumer_factory) { | ||
| consumer = _options.range_consumer_factory(); | ||
| } | ||
| auto submit_result = | ||
| _scheduler->try_submit(range_plan.ranges, _source_reader, _io_context, _context); | ||
| result.reject_reason = submit_result.reject_reason; | ||
| result.status = submit_result.status; | ||
| if (!submit_result.accepted()) { | ||
| for (const auto& miss : misses) { | ||
| miss.column->complete(miss.page.page_index); | ||
| } | ||
| return result; | ||
| } | ||
|
|
||
| DORIS_CHECK(submit_result.reads.size() == range_plan.ranges.size()); | ||
| std::vector<std::shared_ptr<SegmentReadAheadFileReader::BufferedRange>> ranges( | ||
| range_plan.ranges.size()); | ||
| for (size_t index = 0; index < range_plan.ranges.size(); ++index) { | ||
| // Byte-budget admission may skip individual ranges while preserving their input slots. | ||
| if (submit_result.reads[index] == nullptr) { | ||
| continue; | ||
| } | ||
| ++result.submitted_ranges; | ||
| result.submitted_bytes += range_plan.ranges[index].size; | ||
| ranges[index] = std::make_shared<SegmentReadAheadFileReader::BufferedRange>( | ||
| SegmentReadAheadFileReader::BufferedRange { | ||
| .read = std::move(submit_result.reads[index]), .consumer = consumer}); | ||
| } | ||
|
|
||
| DORIS_CHECK(range_plan.input_locations.size() == input_ranges.size()); | ||
| size_t input_index = 0; | ||
| for (size_t index = 0; index < misses.size(); ++index) { | ||
| if (index > 0 && misses[index - 1].page.range != misses[index].page.range) { | ||
| ++input_index; | ||
| } | ||
| DORIS_CHECK(input_index < range_plan.input_locations.size()); | ||
| const auto& location = range_plan.input_locations[input_index]; | ||
| DORIS_CHECK(location.range_index < ranges.size()); | ||
| // No buffer was admitted for this page. Clear pending now; PageIO uses the original reader. | ||
| if (ranges[location.range_index] == nullptr) { | ||
| misses[index].column->complete(misses[index].page.page_index); | ||
| continue; | ||
| } | ||
| _reader->_register_page(misses[index].column, misses[index].page, | ||
| ranges[location.range_index], location.buffer_offset); | ||
| } | ||
| return result; | ||
| } | ||
|
|
||
| ReadAheadPageCacheProbe make_storage_page_cache_probe(const io::FileReaderSPtr& file_reader) { | ||
| DORIS_CHECK(file_reader != nullptr); | ||
| return [file_reader](const io::FileRange& page_range) { | ||
| auto* cache = StoragePageCache::instance(); | ||
| if (cache == nullptr) { | ||
| return false; | ||
| } | ||
| PageCacheHandle handle; | ||
| StoragePageCache::CacheKey key(file_reader->path().native(), file_reader->size(), | ||
| cast_set<int64_t>(page_range.offset)); | ||
| return cache->lookup(key, &handle, DATA_PAGE); | ||
| }; | ||
| } | ||
|
|
||
| } // namespace doris::segment_v2 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Count only actual read-ahead fallbacks.
read_ahead_statsis shared by the whole segment, but non-scalar/nested iterators are deliberately left on their originalFileReader. When such a DATA_PAGE reaches this branch, the dynamic cast at line 178 returns null, yet the code still incrementsfallback_pages,fallback_bytes, andfallback_timebefore the ordinary read.ReadAheadFallback*profile counters therefore report nested reads as failed or rejected read-ahead fallbacks. Gate this accounting on a successfulSegmentReadAheadFileReaderattempt and add nested-column coverage.