Repository navigation
[opt](file cache) Add PageIO read-ahead, I/O coalescing, and cache block hole filling #67292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
bobhan1
merged 21 commits into
apache:master
from
bobhan1:feature/read-ahead-io-coalescing-phase2
Oct 30, 2026
Merged
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
7654cf7
[feature](be) Add linear file range coalescing
bobhan1 90cf5b6
[feature](be) Add cache-aware file range planning
bobhan1 85894d0
[feature](be) Add file range read scheduler
bobhan1 c64e719
[feature](be) Reuse remote-only reads for no-write ranges
bobhan1 563b5ba
[feature](be) Add query read-ahead runtime ownership
bobhan1 e2a4012
[improvement](be) Add opt-in file range IO tracing
bobhan1 8a08ae1
[fix](be) Scope filesystem error-code imports to their functions
bobhan1 4271da9
[feature](be) Add page-driven column read-ahead windows
bobhan1 e7dbfa1
[feature](be) Add segment range read-ahead coordination
bobhan1 00fa56b
[feature](be) Connect scalar column readers to range read-ahead
bobhan1 8190e0f
[feature](be) Initialize scan read-ahead once after pruning
bobhan1 d03f274
[feature](be) Add complete-block cache write handoff and admission
bobhan1 a547700
[feature](be) Split consumed ranges for cache writeback
bobhan1 7cfe76c
[feature](be) Add cache block hole-fill planning
bobhan1 a7e3899
[feature](be) Add background partial block writeback
bobhan1 999f5c6
[feature](be) Initialize background hole fill
bobhan1 6fd7bb6
[feature](be) Route consumed ranges to cache writeback
bobhan1 1ae1272
[feature](be) Connect read-ahead range writeback
bobhan1 fd6df2c
[feature](be) Extend range read-ahead to point queries
bobhan1 a881473
[fix](be) Validate S3 upload buffer alignment on config updates
bobhan1 b237ebc
[fix](be) Handle irregular cache blocks in writeback coverage checks
bobhan1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
[feature](be) Connect read-ahead range writeback
### What problem does this PR solve? Issue Number: Part of #67611 Related PR: #67292, #67579 Problem Summary: Centralize query read-ahead construction so regular scans and exact-row callers share range planning, byte windows, cache probes, scheduler ownership and writeback setup. Merge per-read IO statistics into caller accounting and publish each consumed range once. Submit writeback only for consumed ranges that read remote bytes, preserving no-write policy and avoiding redundant hole filling for File Cache or inflight hits. Carry the read identity through the consumed-range callback so offline traces can attribute background work to its foreground source. Require both query read-ahead and asynchronous cache writes so disabling the writer preserves synchronous cache filling. Handle a switch changing during scan initialization through the original reader. Keep delayed hole-fill sources open with an aliasing shared_ptr that retains the cached wrapper; regression tests release both the foreground reader and writeback context before completing the queued fill. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation; rebase and synchronization validation is recorded in the related PRs and tracking issue. - Repository clang-format 16, header hygiene and git diff --check passed before folding. - No cluster, external-object-store benchmark or clang-tidy run during synchronization. - Behavior changed: Yes, includes final component behavior and applicable diagnostic instrumentation; default enable switches remain off. - Does this need documentation: Yes, tools/read_io_trace/README.md and related PR/tracking descriptions.
- Loading branch information
commit 1ae1272fe26a5a29c0b4729d81b646fb18516771
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Preserve the query cache-write budget for consumed ranges
For a native cloud query with peer reads enabled and a positive
file_cache_query_limit_bytes, the original reader selects synchronous writes and charges new blocks throughsplit_range_into_cells()->try_admit_cache_write(), even when async writes are enabled globally. Read-ahead instead fetches withNO_WRITE, then submits through this admission snapshot, which drops the limiter and its admission flag; neither complete-block nor hole-fill writeback charges it. If the limiter has not tripped before this context is created, a long scalar scan can populate File Cache beyond its query budget. Preserve that admission policy with ownership safe for deferred work.