[feature](be) Add file-range I/O coalescing and asynchronous read infrastructure by bobhan1 · Pull Request #67612 · apache/doris

bobhan1 · 2026-09-07T09:34:37Z

What problem does this PR solve?

Issue Number: Part of #67611

Related PR: #67292 (full implementation), #67579 (validation and follow-ups); rebased and synchronized on 2026-10-14

This is PR 1 of 4 in the PageIO read-ahead, I/O coalescing, and cache-block hole-filling series. It provides the shared range-planning and asynchronous-read infrastructure needed by the later PRs. The original implementation is split by component so each part can be reviewed and tested independently.

Storage-compute separated reads need to combine small, sometimes sparse data-page intervals into useful remote requests, execute independent reads concurrently, and retain the resulting buffers under query/BE memory limits. Cache-aware planning and asynchronous execution have different responsibilities: the planner decides the physical byte ranges; the scheduler executes those exact ranges without deciding page windows or cache-writeback policy.

Scope: range coalescing, optional boundary-block completion, asynchronous range reads, exact no-write cache reads, runtime ownership, thread-safe shared S3 reader statistics, the shared statistics contract with planner/scheduler collection, and opt-in S3/range IO diagnostics. Regular segment scans, point queries, and background hole filling are connected in later PRs. TopN second-stage read-ahead has been removed from the current series; its baseline behavior is preserved. Setting the new enable switch alone does not start PageIO read-ahead in this PR.

Component boundaries

flowchart LR
    Input["Ordered, disjoint input ranges"] --> Planner["FileRangePlanner"]
    Coalescer["FileRangeCoalescer"] --> Planner
    Planner --> Plan["Physical ranges and input-to-buffer locations"]
    Plan --> Scheduler["FileRangeReadScheduler"]
    Stats["Shared statistics: planner/scheduler Profile deltas and BE cumulative IO"] -.-> Planner
    Stats -.-> Scheduler
    Query["QueryContext: shared cancellation and query byte budget"] --> Scheduler
    Exec["ExecEnv: scheduler ownership and BE byte budget"] --> Scheduler
    Scheduler --> Pool["Existing segment_prefetch_thread_pool: one task per range"]
    Pool --> Reader["FileReader: exact read into an owned buffer"]
    Reader --> Handle["FileRangeRead: status, statistics, and buffer slices"]

Component	Responsibility	Deliberately outside its scope
`FileRangeCoalescer`	One pass over sorted, disjoint intervals; apply gap, merged-size, and read-amplification limits	Sorting, normalization, file reads
`FileRangePlanner`	Coalesce inputs, consider completing boundary cache blocks, and map each input to its final buffer	Buffers, submission, writeback
`FileRangeReadScheduler`	Query/BE byte admission, one task per range on the existing pool, completion, cancellation, and shutdown	Page windows, coalescing, cache-block completion
`CachedRemoteFileReader`	Serve exact `NO_WRITE` reads from fully covering in-flight buffers or downloaded cache, otherwise issue an exact remote read	Speculative range writeback
`ExecEnv` / `QueryContext`	BE scheduler lifetime and shared per-query context	Scanner integration, added in PR 3

Planning and read boundaries

Coalescing is linear in the number of inputs. A merge must satisfy all three limits below; a single indivisible input larger than max_range_bytes stays as one read.

Option	Meaning
`max_gap_bytes`	Maximum gap between the current merged range and the next input
`max_range_bytes`	Maximum byte span produced by a merge or optional block completion
`max_read_amplification_ratio`	Maximum merged byte span divided by the sum of original input bytes in that merged range

After coalescing, the planner considers the boundary cache blocks of each range. Blocks meeting block_fill_min_coverage, measured using original requested bytes, become candidates. Candidates are tried in increasing order of additional bytes; a completion is accepted only if the resulting physical range respects max_range_bytes. The final short block uses valid file bytes for its coverage denominator. The planner then returns the containing range index and buffer offset for every input.

The coalescing amplification limit governs the initial merge pass; optional block completion is separately governed by block coverage and final range size. These are reusable API options. Workload-specific read-ahead and hole-fill configuration is added with the corresponding consumers in later PRs.

Submission, buffer ownership, and failure behavior

flowchart TD
    Submit["try_submit: validate the entire batch"] --> Admit{"Query and BE bytes available?"}
    Admit -- No --> Reject["Reject batch; no read handles or retained reservation"]
    Admit -- Yes --> Allocate["Reserve bytes and allocate every range buffer"]
    Allocate -- Failure --> Reject
    Allocate -- Success --> Queue["Submit one thread-pool task per range"]
    Queue -- "All rejected" --> Reject
    Queue -- "At least one accepted" --> Handles["Return a handle for every range"]
    Handles --> Accepted["Accepted task: exact read and terminal status"]
    Handles --> Rejected["Unsubmitted range: FAILED handle; count SubmitFailedRanges"]
    Accepted --> Retain["Resident bytes remain charged while handles retain buffers"]
    Rejected --> Retain
    Retain --> Release["Last handle released: free buffer and release query/BE bytes"]

The existing thread pool owns queuing and controls execution concurrency. The scheduler has no private task queue, scheduler thread, or extra concurrency quota; it tracks accepted handles for cancellation and draining. Worker I/O contexts own the query ID and use task-local statistics rather than caller-owned statistics pointers. Terminal-state checks belong to FileRangeRead; rejected-result construction belongs to FileRangeReadSubmitResult.

Query cancellation skips queued reads and marks running reads cancelled after their source I/O returns. Shutdown stops admission, cancels accepted reads, and waits for their tasks before the shared executor is destroyed. The scheduler never shuts down the shared thread pool itself.

Observability and accounting

The planner and scheduler update shared counters with COUNTER_UPDATE and scoped timers. Read handles retain the statistics independently of the caller's Profile. IO completion publishes statistics before waking page consumers; BE cumulative totals are updated even after the last Profile report. A shared S3FileReader can serve concurrent range tasks, so its request, retry, byte, and timing counters are atomic; existing concurrent-read and retry tests cover this safety requirement.

Area	Counters and meaning
Planning	`ReadAheadInputPages/Bytes`, `ReadAheadCoalescedRanges/Bytes`, `ReadAheadBlockFillBytes`, and `ReadAheadPlannedRanges/Bytes` describe logical inputs and physical reads
Admission	`ReadAheadSubmittedRanges/Bytes` count accepted executor tasks; query/BE budget rejection has separate batch counters; `ReadAheadSubmitFailedRanges` counts executor rejection
Completion	`ReadAheadCompletedRanges`, `ReadAheadFailedRanges`, and `ReadAheadCancelledRanges` classify accepted tasks; failed handles for unsubmitted ranges stay outside these totals
IO	Source/successful-source/local/remote bytes, remote requests, queue wait and read time; successful-source bytes count full successful source reads, including buffers cancelled after IO returns
BE totals	Resident bytes, inflight ranges, rejection/failure counts, and cumulative source/successful-source/remote bytes and remote requests; suitable for before/after workload deltas

Once all accepted tasks finish, SubmittedRanges = CompletedRanges + FailedRanges + CancelledRanges. Executor rejection is recorded separately, including a rejected part of an otherwise accepted batch.

The common statistics contract also defines column initialization, page advancement, exact-row planning, window-maintenance, page-consumption and writeback counters for later PRs. Their producers, scanner Profile reporting, and hole-fill workers are not introduced by this PR. Existing File Cache counters and the new remote-byte counters may describe the same IO and should not be added together.

Exact cached reads

flowchart TD
    Request["Exact NO_WRITE range"] --> Inflight{"In-flight buffers cover the whole request?"}
    Inflight -- Yes --> Copy["Copy requested bytes; no remote read or cache write"]
    Inflight -- No --> Cache{"Downloaded cache blocks cover the whole request?"}
    Cache -- Yes --> Local["Read requested bytes from cache"]
    Cache -- No --> Remote["One exact remote read; no block-alignment expansion or cache population"]

The in-flight lookup supports requests spanning multiple cache blocks and checks complete coverage before copying. Partial coverage keeps the existing all-cache-or-remote behavior. No-write reads reuse _read_remote_only_on_cache_miss(); write-mode resolution is kept in the main reader implementation because it selects among read paths.

Runtime configuration

BE configuration	Default	Application
`enable_query_read_ahead`	`false`	Mutable switch introduced for later PageIO consumers; no scanner read-ahead is wired by this PR
`read_ahead_max_bytes_per_query`	256 MiB	Startup-configured limit on range buffers retained by one query
`read_ahead_max_bytes_per_be`	1 GiB	Startup-configured limit on range buffers retained across the BE

Cloud-mode ExecEnv creates the scheduler even while the switch is disabled, so later consumers can observe runtime enablement. QueryContext creates its shared range context lazily and propagates cancellation to it.

Read IO diagnostics

enable_read_io_trace=false dynamically enables actual S3 GET intervals/outcomes, range/task provenance, read-only cache/inflight coverage snapshots and typed timing events. Records are batched in memory and flushed by a dedicated writer to standalone JSONL files; they do not go to be.INFO and have no cumulative event cap. Normal and fast shutdown drain submitted records.

Configuration	Default	Meaning
`enable_read_io_trace`	false	Mutable diagnostic switch; leave disabled for performance measurements
`read_io_trace_dir`	empty	BE log directory's `read_io_trace/` subdirectory, unless explicitly set
`read_io_trace_flush_bytes`	1 MiB	Memory-batch flush threshold
`read_io_trace_flush_interval_ms`	1000	Periodic flush interval

The offline analyzer reports interval union/extra transfers, within-source and cross-source overlap, submission stage totals/percentiles, queue-scan costs and GET latency. doris_read_io_trace_record_time_ns exposes diagnostic overhead. Capture and interpretation guide. This PR provides the trace contract/writer/analyzer and S3, scheduler and cache-miss producers. Scanner consumption and hole-fill lifecycle/timing producers belong to later PRs.

Commit-by-commit review

These seven commits are the exact prefix of #67292: six infrastructure components and one unity-build fix. The original five components retain their boundaries; shared tracing, dedicated-file batching and analyzer tests are consolidated in component 6. Commit 7 scopes filesystem error-code imports for no-PCH builds. No scanner or hole-fill execution code is pulled into this PR.

#	Shared commit	Component
1	320ccaa05c7	Range coalescer
2	93d9cf2f258	Cache-aware planner and shared metrics
3	7e0f4d9083b	Asynchronous range scheduler
4	6ab3aa95b4b	Exact no-write cached reads
5	c451fbe0c72	Query runtime ownership
6	5ea5103b2de	Shared read IO trace and analyzer
7	6a63eecdc8b	Scope filesystem error-code imports for unity builds

Latest scope synchronization (2026-10-15)

All three PR branches are rebased onto the same upstream master 72e72588a42, fetched on 2026-10-14. This is the agreed cutoff for this round. The update preserves the agreed configuration defaults and range IO/writeback mechanisms; the 2026-10-15 runtime fix narrows new read-ahead to top-level scalar columns.

PR	Current head	Structure
#67579 validation	c18ca96044f	36 chronological commits
#67292 full implementation	42a0b9b0a8e	18 component commits + one unity-build fix
#67612 infrastructure	6a63eecdc8b	Exact seven-commit prefix of #67292

Conflict and compatibility resolution:

Retain both upstream's cache-key/FileFactory and column-test dependencies and the read-ahead/trace test helpers, including upstream's offset-iterator recording fields.
Construct StorageReadOptions with its required statistics sink and pass valid read options to the updated SegmentIterator constructor; adapt the existing fixtures in their owning commits.
Preserve upstream's request-scoped constant-column behavior: point-read prefetch and synchronous fallback both receive rowset version / commit_tso, and missing-column resolution reuses ReadSchema::tablet_column_from_slot.

The complete #67579 and #67292 trees are now identical. #67612 is the exact first seven commits of #67292: six infrastructure components plus the unity-build fix. The runtime fix is folded into the column-reader, regular-scan and point-query components; the common IO/writeback code is unchanged. TopN second-stage and Thrift/Protobuf files still match the shared master base.

The 2026-10-15 CI fix synchronizes #67579's 04951048d32 into #67612 and #67292 as 6a63eecdc8b. It only moves using namespace ErrorCode into the two localfs_error overloads; namespace lookup no longer leaks into sibling unity sources. The master base stays at 72e72588a42.

The 2026-10-15 runtime fix restricts new PageIO read-ahead to top-level scalar columns. Nested STRUCT/MAP lazy reads can leave a skipped child at a different ordinal from its active sibling; the added MAP prefetch assertion incorrectly required equality even with read-ahead disabled. ARRAY/MAP/STRUCT now use the upstream read implementations, original readers and legacy segment prefetchers, including their offsets, null maps and nested children. Point-query preparation also leaves complex/variant paths on the original reader. This was fixed first in #67579 and folded into #67292's column-reader, scan-initialization and point-query components. #67612's code and head are unchanged.

Defaults in the complete consumer branches remain unchanged from 2026-09-24; these nine settings are outside #67612's infrastructure scope:

Configuration	Default
`hole_fill_merge_delay_ms`	2000 ms
`hole_fill_max_pending_bytes_per_be`	4294967296 B (4 GiB)
`read_ahead_block_fill_min_coverage`	1.0
`read_ahead_max_read_amplification_ratio`	1.0
`hole_fill_max_read_amplification_ratio`	1.0
`read_ahead_lazy_window_bytes`	`1048576` B (1 MiB)
`read_ahead_eager_window_bytes`	4194304 B (4 MiB)
`hole_fill_workers_per_be`	32
`hole_fill_remote_read_threads_per_be`	64

Validation:

Before the final three-upstream-commit refresh (ed735be66ef base), independent ASAN/PCH runs passed: [fix](be) Fix Phase 2 no-PCH compilation (compile only) #67579 307 tests / 22 suites, [opt](file cache) Add PageIO read-ahead, I/O coalescing, and cache block hole filling #67292 202 tests / 8 suites, and [feature](be) Add file-range I/O coalescing and asynchronous read infrastructure #67612 81 tests / 9 suites; 15 trace-analyzer tests passed on each branch.
Replaying onto the final 72e72588a42 base introduces no further conflicts: range-diff confirms all feature patches are unchanged. On the 2026-10-14 [opt](file cache) Add PageIO read-ahead, I/O coalescing, and cache block hole filling #67292 head (344debd93a4, before the no-PCH unity-build fix), a fresh incremental ASAN/PCH build and 358 tests / 24 suites passed, together with 15 trace-analyzer tests.
The 2026-10-14 rebase used the repository scripts with -j100, without clean; formatting, header hygiene, diff checks, component-prefix checks and final tree comparisons passed. That rebase adapted existing fixtures without adding new cases.
On 2026-10-15, the no-PCH IO unity syntax-only compiler check reproduced the exact LIMIT_REACH ambiguity before the fix and passed after it, using ASAN/UBSAN, coverage and -Werror flags. The check was run for both target branches. Formatting, build hygiene, range-diff and tree/prefix checks pass. That syntax-only check was not a full BE rebuild or runtime test run; no cluster, external-object-store benchmark or clang-tidy run is claimed. Synchronization means pushed, not merged.
On 2026-10-15, after the scalar-only scope fix, 134 tests / 4 suites passed independently on [fix](be) Fix Phase 2 no-PCH compilation (compile only) #67579 and [opt](file cache) Add PageIO read-ahead, I/O coalescing, and cache block hole filling #67292, using run-be-ut.sh -j100 --run with ASAN/PCH and no clean. Coverage includes divergent nested STRUCT/MAP child ordinals, mixed scalar/complex scan planning and legacy prefetch routing, complex point-query preparation, page windows and synchronous fallback. Formatting, header hygiene, final-tree equality and the unchanged infrastructure prefix were verified. The P0 cluster regression has not been rerun locally; no cluster or clang-tidy was run.

2026-09-24 defaults synchronization and validation (historical)

The 2026-09-24 update changes defaults only; planning, queueing and concurrency mechanisms are unchanged. The shared master base remains 01efbca40a5; this update does not rebase onto a newer master.

Configuration	Default	This update
`hole_fill_merge_delay_ms`	2000 ms	Changed
`hole_fill_max_pending_bytes_per_be`	4294967296 B (4 GiB)	Changed
`read_ahead_block_fill_min_coverage`	1.0	Changed
`read_ahead_max_read_amplification_ratio`	1.0	Changed
`hole_fill_max_read_amplification_ratio`	1.0	Already matched; retained
`read_ahead_lazy_window_bytes`	`1048576` B (1 MiB)	Already matched; retained
`read_ahead_eager_window_bytes`	4194304 B (4 MiB)	Already matched; retained
`hole_fill_workers_per_be`	32	Already matched; retained
`hole_fill_remote_read_threads_per_be`	64	Already matched; retained

hole_fill_merge_delay_ms gives queued fragments 2 seconds to aggregate from first admission; 0 remains available to disable deliberate waiting. A 1.0 foreground amplification limit avoids merging across nonempty gaps, while full block-coverage eligibility prevents extra foreground completion bytes at the default. These settings do not cap total foreground/background duplicate traffic. Query read-ahead remains disabled by default.

PR	Current head	Structure and validation
#67579 validation	5f9e73ce8fe	35 chronological commits; 121 ASAN/PCH tests / 9 suites passed on 2026-09-24
#67292 full implementation	d92ce0d68a1	18 component commits; 25 ASAN/PCH tests / 3 suites passed on 2026-09-24
#67612 infrastructure	8c4d54869dc	Unchanged exact six-commit prefix of #67292; last independently verified on 2026-09-22: 75 ASAN/PCH tests / 8 suites, plus 15 analyzer tests

Source defaults commit 5f9e73ce8fe is folded into #67292's scan initialization (0b676dc846a), partial-block manager (0ca4025b827), hole-fill runtime (6c1d0f5bf69) and writeback test fixture (08a1fdd52d7). #67612 defines none of these nine consumer settings: its code/head are unchanged, and only its synchronization description is updated.

The two complete trees still differ only in #67579's validation-only err_utils.cpp no-PCH fix. Existing default-validation tests are aligned; the immediate-writeback fixture explicitly retains zero delay. No new tests or mechanism changes are added. The new ASAN/PCH runs use run-be-ut.sh -j100 without clean; formatting, header hygiene, diff checks and component/tree comparisons pass. No cluster, object-store workload, new analyzer run or clang-tidy run is claimed. Synchronization means pushed, not merged.

2026-09-22 synchronization and validation (historical)

All three branches are rebased onto the same upstream master 01efbca40a5. Recent interval diagnostics, batched trace files, submission/scan timing, deadline-ordered queue scans and dedicated hole-fill dispatch are synchronized by component.

PR	Head	Structure	Validation
#67579 validation	a03c3bc8cfd	34 chronological commits	ASAN/PCH: 55 tests / 7 suites passed
#67292 full implementation	df02672a373	18 component commits	Independent ASAN/PCH: 287 tests / 20 suites passed
#67612 infrastructure	8c4d54869dc	Exact six-commit prefix of #67292	Independent ASAN/PCH: 75 tests / 8 suites passed

Shared tracing is one additional infrastructure component. Hole-fill lifecycle timing, ordered scans and HoleFillDispatch are folded into the partial-block manager; consumed-range timing/provenance and callback wiring are folded into their owning components. #67612 intentionally excludes scanner/point-query consumers and background hole filling. The two complete source trees differ only in #67579's validation-only err_utils.cpp no-PCH fix. TopN second-stage integration remains outside the series.

Conflicts preserve upstream's column-reader tests and segment-loading refactor. Point-query reads use the new TabletColumn interface and preserve slot-derived defaults before a light-schema-change column reaches the tablet schema; an added BE test covers that case. Scanner tests also use a schema-bearing MockSegment, matching upstream's move of table properties out of StorageReadOptions; the obsolete options assignment is removed.

All runs use the repository scripts, ASAN/PCH and -j100, without a clean rebuild. The Python analyzer's 15 tests pass. Formatting, header hygiene, diff checks, component-prefix and tree-scope comparisons pass. No cluster, external-object-store benchmark or clang-tidy run; these results do not establish a workload speedup. Synchronization means pushed, not merged.

Historical scope and validation checkpoint

Historical scope synchronization (2026-09-15)

Branch head cdec0499937, based on master a6751eb3ac6b4667c3291544c5cfb1fae48e1556. Full implementation: af40b6c9022 (17 components); validation/follow-ups: 484f6cd106f (29 commits).

The 2026-09-15 update changes lazy-window and hole-fill defaults in the full/validation PRs only; those configurations are outside this infrastructure PR, so its code and five commits are unchanged.

The preceding 2026-09-14 follow-ups contribute the shared planning-timer contract/Profile tests and scheduler-owned result/state helpers. Compared with the previous infrastructure head 43bdd7d3c7c, only the six scheduler/statistics source and test files change. Existing runtime include ordering and scheduler-before-executor teardown are preserved. Column windows and their timer producers, scanner/point-query wiring, partial-block request/task ownership, and hole-fill execution remain outside this PR. TopN second-stage integration remains excluded from the series.

On 2026-09-14, this unchanged infrastructure head independently passed 72 ASAN/PCH tests / 8 suites; the then-current full/validation heads each passed 198 tests / 11 suites. The 2026-09-15 defaults-only validation is recorded in #67292 and #67579. No scanner or hole-fill code is borrowed from the full checkout during the infrastructure build.

Release note

Add internal file-range coalescing and asynchronous-read infrastructure with planning, accepted-task and cumulative IO metrics. Exact no-write cache reads can reuse fully covering in-flight write buffers. Scanner/PageIO read-ahead and background hole filling will be enabled by later PRs in #67611.

Check List (For Author)

Test
- Unit Test — Earlier window-synchronization validation passed an ASAN/PCH build and 69 tests from 7 suites at ea4107d6326 on 2026-09-11 using run-be-ut.sh -j100. This is a historical result; current synchronization results are recorded below.
- Formatting — repository clang-format 16 formatting and check scripts passed.
- BE build hygiene and git diff --check passed.
- Regression test — scanner integrations are outside this PR; integrated cloud smoke and regression belong to the later consumer PRs.
Behavior changed:
- Yes — exact no-write cache reads can reuse fully covered in-flight data; this PR does not activate scanner read-ahead.
Does this need documentation?
- No — internal infrastructure; the series and component boundaries are documented in [Tracking] PageIO read-ahead, I/O coalescing, and cache block hole filling #67611 and this description.

Validation coverage

Independent validation on 2026-10-14, before the final three-upstream-commit refresh (ed735be66ef base): 81 ASAN/PCH tests / 9 suites passed, plus 15 Python analyzer tests. The earlier command and results below remain historical. The current tests cover trace batching, timed flushes, shutdown, concurrent recording, real file-error recovery, interval provenance and cache/inflight coverage snapshots.

Historical 2026-09-14 head cdec0499937: Independent ASAN/PCH: 72 tests / 8 suites passed, using the command below. The new rejected-result test verifies reason/status preservation; shared planning-timer tests verify Profile deltas. The previous head 43bdd7d3c7c passed 71 tests / 8 suites and is historical.

Focused ASAN/PCH build and test command for the 2026-09-14 code synchronization (no clean rebuild; -j100):

./run-be-ut.sh --run --filter='S3FileReaderTest.*:FileRangeCoalescerTest.*:FileRangePlannerTest.*:FileRangeReadSchedulerTest.*:QueryContextReadAheadTest.*:ReadAheadMetricsTest.*:AsyncCachedRemoteFileReaderTest.*:BlockFileCacheTest.async_write*:BlockFileCacheTest.cache_write_mode_is_resolved_for_each_read_context:BlockFileCacheTest.direct_partial_hit_with_downloaded_remainder_should_not_read_remote_again:BlockFileCacheTest.cached_remote_file_reader_remote_only_on_miss' -j100

Test area	Result	Coverage
Coalescer / planner	7 + 15 passed	Gap, size and amplification boundaries; oversized input; optional block completion; EOF; input-to-buffer mapping; planner byte/range accounting
Scheduler	21 passed	Shared executor, parallel execution, whole-batch byte admission, partial executor rejection, accepted-task outcome accounting, buffer lifetime, cancellation races, shutdown, allocation/read failures, asynchronous IO after the final Profile report
Cached reader	22 passed	Exact cold misses, complete in-flight coverage across blocks, partial in-flight coverage, existing asynchronous cache-read paths, existing remote-only-on-miss policy
Shared S3 reader	2 passed	Concurrent requests on one reader preserve request/byte/timing totals; retry counters remain correct
Profile statistics	3 passed	Delta-only reporting, concurrent increments, and statistics lifetime beyond Profile destruction
Query context	2 passed	Shared context and cancellation before/after context creation

No workload performance claims or external-object-store regression results are included.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-09-07T09:34:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Read-ahead needs to coalesce sorted page ranges and opportunistically complete highly covered File Cache blocks without repeatedly rebuilding the whole plan. Add a planner that considers only the two boundary blocks of each base range, orders candidates by additional bytes, updates affected interval components, and produces compact input-to-buffer mappings. Its complexity is O(n log n), and an oversized atomic input remains standalone without preventing unrelated blocks from being completed. Include the shared named Profile/bvar statistics contract, planner byte/range accounting, and delta-reporting/lifetime tests. Consumers collect their own metrics in their component commits. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document) (cherry picked from commit 0295d45)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead needs to submit a complete batch of planned file ranges without blocking the scanner or coupling execution to page and cache-block policy. Add a generic asynchronous range reader with atomic query and BE resident-buffer byte admission, exact private buffers, worker-owned IO context, cancellation, shutdown, and terminal-state notification. Each admitted range is submitted directly as one task to the shared SegmentPrefetchThreadPool, which owns task queuing and controls physical read concurrency. The scheduler keeps no private task queue or range/concurrency quota. Byte reservations follow the last buffer reference so submitted and completed read-ahead data share the same memory bound. Account only accepted executor tasks as submitted; separate budget/executor rejection from read failure and queued cancellation. Record source/successful-source/local/remote bytes, timing, and BE cumulative IO totals independently of Profile lifetime, with accounting and cancellation-race tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document) (cherry picked from commit 3c593b0)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Read-ahead needs to coalesce sorted page ranges and opportunistically complete highly covered File Cache blocks without repeatedly rebuilding the whole plan. Add a planner that considers only the two boundary blocks of each base range, orders candidates by additional bytes, updates affected interval components, and produces compact input-to-buffer mappings. Its complexity is O(n log n), and an oversized atomic input remains standalone without preventing unrelated blocks from being completed. Include the shared named Profile/bvar statistics contract, planner byte/range accounting, and delta-reporting/lifetime tests. Consumers collect their own metrics in their component commits. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead needs to submit a complete batch of planned file ranges without blocking the scanner or coupling execution to page and cache-block policy. Add a generic asynchronous range reader with atomic query and BE resident-buffer byte admission, exact private buffers, worker-owned IO context, cancellation, shutdown, and terminal-state notification. Each admitted range is submitted directly as one task to the shared SegmentPrefetchThreadPool, which owns task queuing and controls physical read concurrency. The scheduler keeps no private task queue or range/concurrency quota. Byte reservations follow the last buffer reference so submitted and completed read-ahead data share the same memory bound. Account only accepted executor tasks as submitted; separate budget/executor rejection from read failure and queued cancellation. Record source/successful-source/local/remote bytes, timing, and BE cumulative IO totals independently of Profile lifetime, with accounting and cancellation-race tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Per-column byte windows expose page candidates independently, while remote IO must be planned and admitted as one segment-level batch. Add a coordinator that probes Page Cache, coalesces all cache misses into final file ranges, submits the batch atomically to the asynchronous range reader, serves exact page slices from completed buffers, falls back to the original reader on planning, admission, or IO failure, and forwards only ranges with consumed pages to the future writeback path. Track page candidates, Page Cache hits, consumed/released pages and range bytes. Share statistics with range handles and count synchronous data-page fallback at PageIO, including retries while excluding Page Cache hits and index/dictionary reads. Include Profile visibility and fallback tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Range read-ahead needs physical data-page ranges from every column involved in the current materialization phase. Add final ColumnIterator planning interfaces, extract physical page ranges from ordinal indexes, route eager and lazy roles through complex columns, plan nested children after their offsets become known, and consume submitted pages through SegmentReadAheadFileReader with fallback to the original reader after read or decode failures. Extend the column-reader tests to verify consumed-page accounting and synchronous fallback after decode failure. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: SegmentIterator decoded each scanner batch before the range reader could see all currently plannable columns, preserving serial remote IO dependencies across small batches. Initialize query-owned range read-ahead before column iterators, classify physical columns by their actual dependency stage, submit eager and lazy plans together before the first decode, and immediately plan nested dependent children after offsets become available. Planning, admission, and asynchronous read failures continue through the original read path, while the new query path takes precedence over the legacy segment prefetcher. Flush read-ahead counter deltas into the scanner Profile during reporting and before close. ### Release note Add disabled-by-default query data-page range read-ahead with byte-window and range-planning settings. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead can leave partially covered File Cache blocks that need bounded asynchronous completion before entering the cache writer. Add a BE-level manager that merges queued fragments, deduplicates active blocks, evicts the oldest queued block under pressure, fills block-local holes through dedicated remote-read workers, and hands completed buffers to the asynchronous cache writer without another copy. The worker set can grow or shrink at runtime; shrinking waits only for retiring workers to finish their active remote reads. Same-block submissions are detected before allocating another block buffer, and their fragment copies use a task-local lock so unrelated submissions do not hold the BE-wide queue lock behind a memcpy. Record bounded queue/worker and aggregate hole-fill IO/failure metrics. Combine inflight-buffer lookup and File Cache probing before admission and once at task start, outside the queue mutex. Reuse cached/downloading blocks and execute the planned GETs without per-GET probes; cover these boundaries with unit tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead produces consumed variable-length buffers that can span complete and partial File Cache blocks. Add RangeCacheWriteback to split each consumed range, submit complete blocks directly to the existing asynchronous write path, and enqueue partial fragments for background hole filling under the same invalidation epoch captured before the remote read starts. Record complete/partial/deduplicated/rejected block and byte counters. Reuse the partial-block manager's unified cache/inflight deduplication result, including mixed cached and uncached ranges. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead must publish only buffers that were actually consumed, preserve caller-visible IO statistics, and honor the original cache-write policy. Centralize query read-ahead construction so SegmentIterator and exact row-id readers share the same byte windows, range planner, page-cache probe, scheduler, and writeback setup. Each asynchronous submission owns its statistics and File Cache epoch; the first consumed page publishes the completed range once, while no-write and remote-only requests skip cache writeback. Share query statistics with range tasks and consumed-range callbacks. Populate File Cache only for consumed ranges that actually read remote bytes, avoiding hole filling for pure local/inflight hits, with local-hit and writeback-policy tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: TopN global delayed materialization reads selected rows through RowIdStorageReader after the initial scan. Reading each requested column independently serialized data-page misses and prevented the range planner from seeing the complete exact-row workload. Group internal-table row IDs by segment, initialize all requested column iterators first, submit their data pages as one read-ahead plan, and decode only after submission. Row-store fetches read the segment row IDs in one batch, while sorting, deduplication, output scattering, and fallback to the original reader preserve existing results. Export a read-ahead Profile snapshot once per RPC and merge each response once on the receiver, with a multi-relation aggregation test. ### Release note Use query range read-ahead for internal-table exact row-id fetches when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Cold data pages previously followed the synchronous reader one page or column at a time. Create one request-scoped range-read context, initialize the row-store or all missing-column iterators against the shared read-ahead reader, submit the required data pages before decoding, and retain the existing row-cache and result-padding behavior. Read-ahead initialization or IO failure falls back to the original reader. Append read-ahead Profile statistics to the existing point-query diagnostics; BE cumulative IO counters retain unsampled workload totals. ### Release note Use query range read-ahead for point-query data pages when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Read-ahead needs to coalesce sorted page ranges and opportunistically complete highly covered File Cache blocks without repeatedly rebuilding the whole plan. Add a planner that considers only the two boundary blocks of each base range, orders candidates by additional bytes, updates affected interval components, and produces compact input-to-buffer mappings. Its complexity is O(n log n), and an oversized atomic input remains standalone without preventing unrelated blocks from being completed. Include the shared named Profile/bvar statistics contract, planner byte/range accounting, and delta-reporting/lifetime tests. Consumers collect their own metrics in their component commits. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document) (cherry picked from commit c1a391b)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead needs to submit a complete batch of planned file ranges without blocking the scanner or coupling execution to page and cache-block policy. Add a generic asynchronous range reader with atomic query and BE resident-buffer byte admission, exact private buffers, worker-owned IO context, cancellation, shutdown, and terminal-state notification. Each admitted range is submitted directly as one task to the shared SegmentPrefetchThreadPool, which owns task queuing and controls physical read concurrency. The scheduler keeps no private task queue or range/concurrency quota. Byte reservations follow the last buffer reference so submitted and completed read-ahead data share the same memory bound. Account only accepted executor tasks as submitted; separate budget/executor rejection from read failure and queued cancellation. Record source/successful-source/local/remote bytes, timing, and BE cumulative IO totals independently of Profile lifetime, with accounting and cancellation-race tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document) (cherry picked from commit 809d9a1)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Read-ahead needs to coalesce sorted page ranges and opportunistically complete highly covered File Cache blocks without repeatedly rebuilding the whole plan. Add a planner that considers only the two boundary blocks of each base range, orders candidates by additional bytes, updates affected interval components, and produces compact input-to-buffer mappings. Its complexity is O(n log n), and an oversized atomic input remains standalone without preventing unrelated blocks from being completed. Include the shared named Profile/bvar statistics contract, planner byte/range accounting, and delta-reporting/lifetime tests. Consumers collect their own metrics in their component commits. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead needs to submit a complete batch of planned file ranges without blocking the scanner or coupling execution to page and cache-block policy. Add a generic asynchronous range reader with atomic query and BE resident-buffer byte admission, exact private buffers, worker-owned IO context, cancellation, shutdown, and terminal-state notification. Each admitted range is submitted directly as one task to the shared SegmentPrefetchThreadPool, which owns task queuing and controls physical read concurrency. The scheduler keeps no private task queue or range/concurrency quota. Byte reservations follow the last buffer reference so submitted and completed read-ahead data share the same memory bound. Account only accepted executor tasks as submitted; separate budget/executor rejection from read failure and queued cancellation. Record source/successful-source/local/remote bytes, timing, and BE cumulative IO totals independently of Profile lifetime, with accounting and cancellation-race tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Per-column byte windows expose page candidates independently, while remote IO must be planned and admitted as one segment-level batch. Add a coordinator that probes Page Cache, coalesces all cache misses into final file ranges, submits the batch atomically to the asynchronous range reader, serves exact page slices from completed buffers, falls back to the original reader on planning, admission, or IO failure, and forwards only ranges with consumed pages to the future writeback path. Track page candidates, Page Cache hits, consumed/released pages and range bytes. Share statistics with range handles and count synchronous data-page fallback at PageIO, including retries while excluding Page Cache hits and index/dictionary reads. Include Profile visibility and fallback tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Range read-ahead needs physical data-page ranges from every column involved in the current materialization phase. Add final ColumnIterator planning interfaces, extract physical page ranges from ordinal indexes, route eager and lazy roles through complex columns, plan nested children after their offsets become known, and consume submitted pages through SegmentReadAheadFileReader with fallback to the original reader after read or decode failures. Extend the column-reader tests to verify consumed-page accounting and synchronous fallback after decode failure. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: SegmentIterator decoded each scanner batch before the range reader could see all currently plannable columns, preserving serial remote IO dependencies across small batches. Initialize query-owned range read-ahead before column iterators, classify physical columns by their actual dependency stage, submit eager and lazy plans together before the first decode, and immediately plan nested dependent children after offsets become available. Planning, admission, and asynchronous read failures continue through the original read path, while the new query path takes precedence over the legacy segment prefetcher. Flush read-ahead counter deltas into the scanner Profile during reporting and before close. ### Release note Add disabled-by-default query data-page range read-ahead with page-position triggers. Configure compressed bytes per window through read_ahead_eager_window_bytes (4 MiB) and read_ahead_lazy_window_bytes (128 KiB), alongside the range-planning settings. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead can leave partially covered File Cache blocks that need bounded asynchronous completion before entering the cache writer. Add a BE-level manager that merges queued fragments, deduplicates active blocks, evicts the oldest queued block under pressure, fills block-local holes through dedicated remote-read workers, and hands completed buffers to the asynchronous cache writer without another copy. The worker set can grow or shrink at runtime; shrinking waits only for retiring workers to finish their active remote reads. Same-block submissions are detected before allocating another block buffer, and their fragment copies use a task-local lock so unrelated submissions do not hold the BE-wide queue lock behind a memcpy. Record bounded queue/worker and aggregate hole-fill IO/failure metrics. Combine inflight-buffer lookup and File Cache probing before admission and once at task start, outside the queue mutex. Reuse cached/downloading blocks and execute the planned GETs without per-GET probes; cover these boundaries with unit tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead produces consumed variable-length buffers that can span complete and partial File Cache blocks. Add RangeCacheWriteback to split each consumed range, submit complete blocks directly to the existing asynchronous write path, and enqueue partial fragments for background hole filling under the same invalidation epoch captured before the remote read starts. Record complete/partial/deduplicated/rejected block and byte counters. Reuse the partial-block manager's unified cache/inflight deduplication result, including mixed cached and uncached ranges. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead must publish only buffers that were actually consumed, preserve caller-visible IO statistics, and honor the original cache-write policy. Centralize query read-ahead construction so SegmentIterator and exact row-id readers share the same byte windows, range planner, page-cache probe, scheduler, and writeback setup. Each asynchronous submission owns its statistics and File Cache epoch; the first consumed page publishes the completed range once, while no-write and remote-only requests skip cache writeback. Share query statistics with range tasks and consumed-range callbacks. Populate File Cache only for consumed ranges that actually read remote bytes, avoiding hole filling for pure local/inflight hits, with local-hit and writeback-policy tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: TopN global delayed materialization reads selected rows through RowIdStorageReader after the initial scan. Reading each requested column independently serialized data-page misses and prevented the range planner from seeing the complete exact-row workload. Group internal-table row IDs by segment, initialize all requested column iterators first, submit their data pages as one read-ahead plan, and decode only after submission. Row-store fetches read the segment row IDs in one batch, while sorting, deduplication, output scattering, and fallback to the original reader preserve existing results. Export a read-ahead Profile snapshot once per RPC and merge each response once on the receiver, with a multi-relation aggregation test. ### Release note Use query range read-ahead for internal-table exact row-id fetches when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Cold data pages previously followed the synchronous reader one page or column at a time. Create one request-scoped range-read context, initialize the row-store or all missing-column iterators against the shared read-ahead reader, submit the required data pages before decoding, and retain the existing row-cache and result-padding behavior. Read-ahead initialization or IO failure falls back to the original reader. Append read-ahead Profile statistics to the existing point-query diagnostics; BE cumulative IO counters retain unsampled workload totals. ### Release note Use query range read-ahead for point-query data pages when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Expose complete-block submission, tracked owned-buffer handoff, and low-priority spare-capacity admission together as one cache-writer component. Reuse existing epoch checks and asynchronous disk writes. Background hole filling can pass a completed block without another data copy while normal foreground cache writes keep priority. ### Release note Internal complete-block and zero-copy handoff for best-effort cache population. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Provide bounded queued-fragment merging, aggregation delay, eviction, and cache/inflight reuse for partial cache blocks. Keep request invariants on PartialBlockWritebackRequest and buffer creation/remote reads on Task; the manager owns admission, queues, worker scheduling and lifetime. Read one hole directly on a block worker and multiple holes through a reusable token on the shared remote-read pool. Join all accepted GETs before writeback or failure cleanup. Preserve lock boundaries, epoch checks, finite timed-wait deadlines, and observability. ### Release note Internal background partial-block completion with bounded memory and parallel block-local remote reads. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead produces consumed variable-length buffers that can span complete and partial File Cache blocks. Add RangeCacheWriteback to split each consumed range, submit complete blocks directly to the existing asynchronous write path, and enqueue partial fragments for background hole filling under the same invalidation epoch captured before the remote read starts. Record complete/partial/deduplicated/rejected block and byte counters. Reuse the partial-block manager's unified cache/inflight deduplication result, including mixed cached and uncached ranges. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead must publish only buffers that were actually consumed, preserve caller-visible IO statistics, and honor the original cache-write policy. Centralize query read-ahead construction so SegmentIterator and exact row-id readers share the same byte windows, range planner, page-cache probe, scheduler, and writeback setup. Each asynchronous submission owns its statistics and File Cache epoch; the first consumed page publishes the completed range once, while no-write and remote-only requests skip cache writeback. Share query statistics with range tasks and consumed-range callbacks. Populate File Cache only for consumed ranges that actually read remote bytes, avoiding hole filling for pure local/inflight hits, with local-hit and writeback-policy tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Cold data pages previously followed the synchronous reader one page or column at a time. Create one request-scoped range-read context, initialize the row-store or all missing-column iterators against the shared read-ahead reader, submit the required data pages before decoding, and retain the existing row-cache and result-padding behavior. Read-ahead initialization or IO failure falls back to the original reader. Append read-ahead Profile statistics to the existing point-query diagnostics; BE cumulative IO counters retain unsampled workload totals. Include the reusable row-id page preparation helpers, SegmentReadAhead exact-row entry point, and its tests in this point-query component. TopN second-stage integration remains unchanged from master and is outside this PR series. ### Release note Use query range read-ahead for point-query data pages when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Build each scan column's eligible physical page indexes once and advance compressed-byte windows from the page index resolved by the column reader. A window's second selected page triggers the following window. Retain predictions for ascending batches inside reverse scans; group exact-row requests by distinct pages. ColumnReadAheadPlan owns reset and per-call timing aggregation, retaining vector capacity. Window accounting stays with ColumnReadAhead. ### Release note Internal page-driven column windows for query read-ahead. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Coordinate all resolvable column plans, probe Page Cache, plan and submit ranges, and register exact page slices. Reuse a physical page's existing buffer when independent column windows overlap it. Keep synchronous fallback and per-page ownership accounting; expose planner and consumption statistics. ### Release note Internal segment-level coordination for concurrent data-page reads. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Prepare eligible column pages once after pruning and advance scan windows at the data-page read entry before Page Cache lookup. Point reads and dependent nested columns retain exact-row preparation. Report initialization and page-advancement time, then let ColumnReadAheadPlan accumulate its maintenance sub-timers. Preserve existing page decoding, cache hits and synchronous fallback. ### Release note Connect physical-column data-page reads to byte-window read-ahead. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Initialize currently resolvable eager and lazy column windows together after segment pruning instead of rebuilding rowid plans every scanner batch. Later physical page reads advance each window. Retain reverse-scan behavior and configure compressed-byte windows, coalescing limits and optional foreground block completion. ### Release note Opt-in regular segment scans use page-driven read-ahead across scanner batches. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Expose complete-block submission, tracked owned-buffer handoff, and low-priority spare-capacity admission together as one cache-writer component. Reuse existing epoch checks and asynchronous disk writes. Background hole filling can pass a completed block without another data copy while normal foreground cache writes keep priority. ### Release note Internal complete-block and zero-copy handoff for best-effort cache population. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Provide bounded queued-fragment merging, aggregation delay, eviction, and cache/inflight reuse for partial cache blocks. Keep request invariants on PartialBlockWritebackRequest and buffer creation/remote reads on Task; the manager owns admission, queues, worker scheduling and lifetime. Read one hole directly on a block worker and multiple holes through a reusable token on the shared remote-read pool. Join all accepted GETs before writeback or failure cleanup. Preserve lock boundaries, epoch checks, finite timed-wait deadlines, and observability. ### Release note Internal background partial-block completion with bounded memory and parallel block-local remote reads. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead produces consumed variable-length buffers that can span complete and partial File Cache blocks. Add RangeCacheWriteback to split each consumed range, submit complete blocks directly to the existing asynchronous write path, and enqueue partial fragments for background hole filling under the same invalidation epoch captured before the remote read starts. Record complete/partial/deduplicated/rejected block and byte counters. Reuse the partial-block manager's unified cache/inflight deduplication result, including mixed cached and uncached ranges. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead must publish only buffers that were actually consumed, preserve caller-visible IO statistics, and honor the original cache-write policy. Centralize query read-ahead construction so SegmentIterator and exact row-id readers share the same byte windows, range planner, page-cache probe, scheduler, and writeback setup. Each asynchronous submission owns its statistics and File Cache epoch; the first consumed page publishes the completed range once, while no-write and remote-only requests skip cache writeback. Share query statistics with range tasks and consumed-range callbacks. Populate File Cache only for consumed ranges that actually read remote bytes, avoiding hole filling for pure local/inflight hits, with local-hit and writeback-policy tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Cold data pages previously followed the synchronous reader one page or column at a time. Create one request-scoped range-read context, initialize the row-store or all missing-column iterators against the shared read-ahead reader, submit the required data pages before decoding, and retain the existing row-cache and result-padding behavior. Read-ahead initialization or IO failure falls back to the original reader. Append read-ahead Profile statistics to the existing point-query diagnostics; BE cumulative IO counters retain unsampled workload totals. Include the reusable row-id page preparation helpers, SegmentReadAhead exact-row entry point, and its tests in this point-query component. TopN second-stage integration remains unchanged from master and is outside this PR series. ### Release note Use query range read-ahead for point-query data pages when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Initialize currently resolvable eager and lazy column windows together after segment pruning instead of rebuilding rowid plans every scanner batch. Later physical page reads advance each window. Retain reverse-scan behavior and configure compressed-byte windows, coalescing limits and optional foreground block completion. ### Release note Opt-in regular segment scans use page-driven read-ahead across scanner batches. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces opt-in scanner initialization with a 4 MiB eager window and a 1 MiB lazy window; keep page-driven advancement and IO/writeback mechanisms unchanged. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Expose complete-block submission, tracked owned-buffer handoff, and low-priority spare-capacity admission together as one cache-writer component. Reuse existing epoch checks and asynchronous disk writes. Background hole filling can pass a completed block without another data copy while normal foreground cache writes keep priority. ### Release note Internal complete-block and zero-copy handoff for best-effort cache population. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Provide bounded queued-fragment merging, aggregation delay, eviction, and cache/inflight reuse for partial cache blocks. Keep request invariants on PartialBlockWritebackRequest and buffer creation/remote reads on Task; the manager owns admission, queues, worker scheduling and lifetime. Read one hole directly on a block worker and multiple holes through a reusable token on the shared remote-read pool. Join all accepted GETs before writeback or failure cleanup. Preserve lock boundaries, epoch checks, finite timed-wait deadlines, and observability. ### Release note Internal background partial-block completion with bounded memory and parallel block-local remote reads. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in partial-block manager with zero deliberate aggregation delay by default; keep queued-fragment merging, worker scheduling and IO/writeback mechanisms unchanged. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead produces consumed variable-length buffers that can span complete and partial File Cache blocks. Add RangeCacheWriteback to split each consumed range, submit complete blocks directly to the existing asynchronous write path, and enqueue partial fragments for background hole filling under the same invalidation epoch captured before the remote read starts. Record complete/partial/deduplicated/rejected block and byte counters. Reuse the partial-block manager's unified cache/inflight deduplication result, including mixed cached and uncached ranges. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Query read-ahead must publish only buffers that were actually consumed, preserve caller-visible IO statistics, and honor the original cache-write policy. Centralize query read-ahead construction so SegmentIterator and exact row-id readers share the same byte windows, range planner, page-cache probe, scheduler, and writeback setup. Each asynchronous submission owns its statistics and File Cache epoch; the first consumed page publishes the completed range once, while no-write and remote-only requests skip cache writeback. Share query statistics with range tasks and consumed-range callbacks. Populate File Cache only for consumed ranges that actually read remote bytes, avoiding hole filling for pure local/inflight hits, with local-hit and writeback-policy tests. ### Release note None ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: Part of apache#67611 Related PR: apache#67292 Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Cold data pages previously followed the synchronous reader one page or column at a time. Create one request-scoped range-read context, initialize the row-store or all missing-column iterators against the shared read-ahead reader, submit the required data pages before decoding, and retain the existing row-cache and result-padding behavior. Read-ahead initialization or IO failure falls back to the original reader. Append read-ahead Profile statistics to the existing point-query diagnostics; BE cumulative IO counters retain unsampled workload totals. Include the reusable row-id page preparation helpers, SegmentReadAhead exact-row entry point, and its tests in this point-query component. TopN second-stage integration remains unchanged from master and is outside this PR series. ### Release note Use query range read-ahead for point-query data pages when enable_query_read_ahead is enabled. ### Check List (For Author) - Test: Unit Test - Component tests are included with the implementation. Focused ASAN/PCH results after this history fold are recorded in apache#67611 and apache#67612. - Repository clang-format 16, build hygiene and git diff --check. - Behavior changed: Yes (component functionality includes its observability and applicable cache-reuse checks; scheduling limits and query-result semantics are unchanged) - Does this need documentation: Yes (tracking issue apache#67611, split PR apache#67612, and the read-ahead observability document)

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67579, apache#67292, apache#67612 Problem Summary: Plan reset and timing aggregation, asynchronous read result construction and terminal-state checks, and partial-block request validation were implemented outside the objects whose state they describe. Move these operations to ColumnReadAheadPlan, FileRangeReadSubmitResult, FileRangeRead, and PartialBlockWritebackRequest. Keep partial-block buffer creation and remote reads on Task, while the manager retains queue admission, worker scheduling, and lifecycle accounting. Preserve vector capacity during plan reset, per-call statistics, lock boundaries, and the join of accepted remote reads before task cleanup. ### Release note None ### Check List (For Author) - Test: Unit Test — 198 tests from 11 suites passed through run-be-ut.sh with ASAN and -j100. Added plan-reset capacity/ownership, empty-plan timing aggregation, and rejected-result coverage. Existing read-ahead, scheduler, cache-writeback and hole-fill tests passed, including concurrent reads, failed submission/read cleanup, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. No cluster tests or clang-tidy were run. - Behavior changed: No — move object-local operations without changing read-ahead, IO, or writeback mechanisms. - Does this need documentation: No — internal responsibility cleanup; configuration and metric meanings are unchanged.

### What problem does this PR solve? Issue Number: None Related PR: apache#67292 Problem Summary: Read-ahead and background hole filling both need to combine sparse logical ranges into bounded physical reads. Add a pure one-pass coalescer whose contract requires validated options and sorted, disjoint, non-empty inputs. Each merge checks the maximum gap, resulting span, and cumulative read amplification. The range-size limit controls merging; an indivisible input larger than the limit remains one read. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh -j100 --run '--filter=FileRangeCoalescerTest.*' - Behavior changed: No; this commit adds an internal planning primitive. - Does this need documentation: No

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Add cache-aware range planning and the shared read-ahead statistics contract. Coalesce sorted requested ranges, optionally complete high-coverage boundary blocks, and map exact inputs back to final buffers. Keep planning timers for column initialization, exact-row batches, physical-page advancement, window discard and extension in the shared contract; collection remains with the later consumers. ### Release note Internal range planning and shared read-ahead Profile/bvar definitions. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: apache#67611 (tracking) Related PR: apache#67292, apache#67579, apache#67612 Problem Summary: Provide query/BE resident-byte admission, one exact range read per existing thread-pool task, cancellation, and owned buffer handles. Keep terminal-state checks on FileRangeRead and rejected-result construction on FileRangeReadSubmitResult. Preserve accepted-task accounting and atomic shared S3 reader statistics for concurrent requests. ### Release note Internal asynchronous file-range reads with query/BE byte budgets and IO accounting. ### Check List (For Author) - Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run. - Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics. - Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.

### What problem does this PR solve? Issue Number: None Related PR: apache#67292 Problem Summary: Asynchronous read-ahead ranges must preserve their coalesced byte boundaries and avoid populating File Cache before consumption is known. Route per-read NO_WRITE requests through CachedRemoteFileReader's remote-only-on-cache-miss path. Before probing downloaded cache blocks, reuse indexed in-flight async-write buffers when they completely cover the request, including requests spanning multiple cache blocks. If any in-flight block is absent, retain the existing all-cache-or-remote behavior: a fully downloaded range is served locally, while incomplete coverage issues one exact remote read for the entire planned range without creating cache cells, waiting for downloading blocks, reading from peers, or submitting cache writes. Keep cache-write mode resolution in the main reader dispatch because it selects among all read paths rather than belonging to the asynchronous cache-write implementation. ### Release note None ### Check List (For Author) - Test: Unit Test and BE build - ./run-be-ut.sh --run --filter='AsyncCachedRemoteFileReaderTest.exact_no_write*:BlockFileCacheTest.cached_remote_file_reader_remote_only_on_miss' -j100 (4 passed) - ./build.sh --be -j100 - Behavior changed: Yes; no-write asynchronous ranges reuse complete in-flight or downloaded-cache coverage and otherwise issue one exact remote read for the planned range. - Does this need documentation: Yes; covered by the Phase 2 design document.

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Concurrent range reads must be shared at BE scope while every scanner of one query shares cancellation and resident-buffer admission. Add a startup-owned FileRangeReadScheduler in Cloud mode, a lazily shared query context, cancellation propagation, and a mutable enable_query_read_ahead switch. Query and BE resident-buffer limits are configurable through read_ahead_max_bytes_per_query and read_ahead_max_bytes_per_be. The scheduler remains available while the switch is disabled so a runtime update can enable read-ahead for newly initialized reads without restarting the BE. ### Release note Support runtime updates to enable_query_read_ahead and configurable query/BE read-ahead memory limits. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter='FileRangeReadSchedulerTest.*:QueryContextReadAheadTest.*:SegmentReadAheadTest.*:ColumnReaderTest.*ReadAhead*' -j200 - ./build.sh --be -j200 - Behavior changed: Yes. Cloud BEs can enable or disable query read-ahead at runtime; new reads observe the updated switch. - Does this need documentation: Yes. The read-ahead design document lists these settings.

### What problem does this PR solve? Issue Number: None Related PR: apache#67611, apache#67612, apache#67292, apache#67579 Problem Summary: Aggregate range-read counters cannot identify repeated object intervals or separate diagnostic overhead from read latency. Synchronize the final shared diagnostics from the validation branch as one infrastructure component: stable IO identities and S3 outcomes, read-only cache/inflight coverage snapshots, task provenance, typed timing fields, a batched standalone JSONL writer and shutdown draining, and the offline overlap/timing analyzer. Keep scanner consumption and hole-fill lifecycle producers in their owning later components. Trace is disabled by default and does not change read planning, concurrency or writeback. ### Release note Add enable_read_io_trace=false and dedicated diagnostic files controlled by read_io_trace_dir, read_io_trace_flush_bytes=1 MiB and read_io_trace_flush_interval_ms=1000. Enabled diagnostics can affect timing and should be used for isolated captures. ### Check List (For Author) - Test: Unit Test - Independent incremental ASAN/PCH run-be-ut.sh -j100 on master d4f052b: 75 tests from 8 suites passed. - Python offline analyzer: 15 tests passed. - Repository clang-format 16, check-format, BE header hygiene and git diff --check passed. - The first build was interrupted after /tmp filled; resumed incrementally with a data-disk TMPDIR. No clean rebuild or ccache cleanup. - No cluster, external-object-store benchmark or clang-tidy run. - Behavior changed: Yes, opt-in diagnostic recording; read and persistence decisions are unchanged. - Does this need documentation: Yes, tools/read_io_trace/README.md and the related PR/tracking descriptions.

### What problem does this PR solve? Issue Number: None Related PR: apache#67292, apache#67612, apache#67579 Problem Summary: With PCH disabled, the IO unity translation unit includes err_utils.cpp before file_meta_cache.cpp. The namespace-scope ErrorCode using-directive leaks into the generated protobuf headers and makes LIMIT_REACH ambiguous. Scope the import to the two localfs_error overloads that use it, preserving the existing error-code mappings. This synchronizes the fix already present in apache#67579 into the infrastructure and full feature branches. ### Release note None ### Check List (For Author) - Test: Manual compiler check. Reproduced the original LIMIT_REACH error in IO unity_2 with PCH imports removed; the same ASAN/UBSAN, coverage and -Werror syntax-only check passes after the fix. Repository clang-format 16, format check, build-hygiene check and git diff --check pass. No full BE build or runtime tests were rerun for this scope-only change. - Behavior changed: No - Does this need documentation: No

bobhan1 · 2026-10-27T05:56:47Z

don't need

bobhan1 mentioned this pull request Sep 7, 2026

[Tracking] PageIO read-ahead, I/O coalescing, and cache block hole filling #67611

Closed

4 tasks

bobhan1 force-pushed the feature/file-range-io-infrastructure branch from 7dda07c to f159cd2 Compare September 10, 2026 08:44

bobhan1 mentioned this pull request Sep 10, 2026

[fix](be) Fix Phase 2 no-PCH compilation (compile only) #67579

Closed

5 tasks done

bobhan1 force-pushed the feature/file-range-io-infrastructure branch from f159cd2 to ea4107d Compare September 11, 2026 04:49

bobhan1 mentioned this pull request Sep 11, 2026

[opt](file cache) Add PageIO read-ahead, I/O coalescing, and cache block hole filling #67292

Merged

5 of 12 tasks

bobhan1 force-pushed the feature/file-range-io-infrastructure branch from 21a0a25 to cdec049 Compare September 14, 2026 07:15

bobhan1 added 7 commits October 14, 2026 15:59

Navigation Menu

Repository navigation

[feature](be) Add file-range I/O coalescing and asynchronous read infrastructure - #67612

[feature](be) Add file-range I/O coalescing and asynchronous read infrastructure#67612
bobhan1 wants to merge 7 commits into
apache:masterfrom
bobhan1:feature/file-range-io-infrastructure

bobhan1 commented Sep 7, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 7, 2026

Uh oh!

bobhan1 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bobhan1 commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Component boundaries

Planning and read boundaries

Submission, buffer ownership, and failure behavior

Observability and accounting

Exact cached reads

Runtime configuration

Read IO diagnostics

Commit-by-commit review

Latest scope synchronization (2026-10-15)

Historical scope synchronization (2026-09-15)

Release note

Check List (For Author)

Validation coverage

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 7, 2026

Uh oh!

bobhan1 commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bobhan1 commented Sep 7, 2026 •

edited

Loading