Skip to content

[fix](be) Fix Phase 2 no-PCH compilation (compile only) - #67579

Closed
bobhan1 wants to merge 42 commits into
apache:masterfrom
bobhan1:fix/phase2-no-pch-build
Closed

bobhan1 wants to merge 42 commits into
apache:masterfrom
bobhan1:fix/phase2-no-pch-build

Conversation

@bobhan1

@bobhan1 bobhan1 commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #67292 (full implementation)

Problem Summary:

This Draft PR was created for no-PCH compilation validation. It now carries the rebased Phase 2 component commits plus the compile fix, observability, hole-fill reuse, page-position windows, parallel hole reads, queued-fragment aggregation, planning timers, page-driven scan windows, object-owned operation cleanup, and configuration-default adjustments tracked below. Do not merge this validation PR; feature delivery is tracked in #67292.

With ENABLE_PCH=OFF and unity builds enabled, the IO unity translation unit includes err_utils.cpp before file_meta_cache.cpp. The namespace-scope using namespace ErrorCode in err_utils.cpp remains visible when the latter includes the generated protobuf headers, making PPlanFragmentCancelReason_MIN = LIMIT_REACH ambiguous between doris::LIMIT_REACH and doris::ErrorCode::LIMIT_REACH. PCH previously parsed these headers before the using-directive could affect them.

Move the using-directive into the two localfs_error overloads that need it. The additional fix changes one file with two insertions and two deletions and preserves the existing error-code mappings.

Current synchronization (2026-10-29)

Change #67579 commit #67292 commit
Apply the existing File Cache block-alignment constraint to runtime S3 upload-buffer configuration updates a23fc63d500 a881473fa15
Check writeback coverage against actual restored File Cache intervals b4b194ada0b b237ebcd675

The S3 validator protects new runtime configuration updates. The coverage fix also handles historical non-aligned blocks restored from disk: writeback deduplication walks actual contiguous DOWNLOADED/DOWNLOADING intervals, retaining the inflight shortcut and the existing grid-only probe(). The latest fix changes four cache/source-test files; windows, coalescing and hole-fill scheduling are preserved. Corresponding fixes have matching patch IDs. Both branches are rebased onto fixed upstream master fc1b82283cb, with 42 chronological commits in #67579 and 21 component/fix commits in #67292. Their final source trees are identical. The latest rebase conflict was in scan initialization: retain master's cached bitmap cardinality and prepare scan read-ahead on the normal batch path. The RLE reverse-reach baseline remains 834 with 916 measured users; no dependency refactoring was added.

Validation: Each branch independently passed 329 tests / 24 suites with run-be-ut.sh --run -j100 (ASAN/PCH) and a direct Release/PCH BE build with build.sh --be -j100, without a clean rebuild. Coverage includes range planning/scheduling, page-position windows, scalar and legacy complex-column reads, count-emission fast paths, point-query fallback, cache/inflight reuse, complete/partial writeback, hole filling, tracing and configuration validation. Full-checkout clang-format 16, build hygiene, gate self-tests, diff/range-diff and final tree-equality checks passed. No cluster or clang-tidy run.

Synchronization checkpoint (2026-10-27, historical)

Both complete implementations use upstream master 3dc73418a07. #67579 retains 40 chronological commits; #67292 retains 19 component commits. Their final source trees are identical.

Change #67579 commit #67292 component
Require asynchronous cache writes for query read-ahead and preserve synchronous cache filling when disabled 58848ac8f29 c21fd6839eb
Retain cached wrappers through queued and active hole filling ac26e560920 c21fd6839eb
Reuse master-resolved segments and preserve row-store switch fallback edcc3cff5b2, 1475936532c bb2d6e84feb
Admit ranges within the remaining query/BE budgets and synchronously read skipped pages c849b842cb1 420d849d8e5, 85cf049198f
Account only for six added RLE-header users: baseline 825 to 831; retain existing header dependencies 0dfaa897e91 85cf049198f
Remove the offline tools/read_io_trace directory; retain BE trace collection 687f3cdbed9 234925a1893

Validation: Each branch independently passed 236 tests / 13 suites with run-be-ut.sh --run -j100 (ASAN/PCH) and a direct Release/PCH BE build with build.sh --be -j100, without a clean rebuild. Build hygiene, gate self-tests, clang-format 16, diff/range-diff checks and final tree equality passed. No cluster or clang-tidy run.

#67611 and #67612 are closed and are no longer synchronization targets. The collapsed entries below retain historical checkpoints.

2026-09-24 defaults synchronization and validation (historical)

The 2026-09-24 update changes defaults only; planning, queueing and concurrency mechanisms are unchanged. The shared master base remains 01efbca40a5; this update does not rebase onto a newer master.

Configuration Default This update
hole_fill_merge_delay_ms 2000 ms Changed
hole_fill_max_pending_bytes_per_be 4294967296 B (4 GiB) Changed
read_ahead_block_fill_min_coverage 1.0 Changed
read_ahead_max_read_amplification_ratio 1.0 Changed
hole_fill_max_read_amplification_ratio 1.0 Already matched; retained
read_ahead_lazy_window_bytes 1048576 B (1 MiB) Already matched; retained
read_ahead_eager_window_bytes 4194304 B (4 MiB) Already matched; retained
hole_fill_workers_per_be 32 Already matched; retained
hole_fill_remote_read_threads_per_be 64 Already matched; retained

hole_fill_merge_delay_ms gives queued fragments 2 seconds to aggregate from first admission; 0 remains available to disable deliberate waiting. A 1.0 foreground amplification limit avoids merging across nonempty gaps, while full block-coverage eligibility prevents extra foreground completion bytes at the default. These settings do not cap total foreground/background duplicate traffic. Query read-ahead remains disabled by default.

PR Current head Structure and validation
#67579 validation 5f9e73ce8fe 35 chronological commits; 121 ASAN/PCH tests / 9 suites passed on 2026-09-24
#67292 full implementation d92ce0d68a1 18 component commits; 25 ASAN/PCH tests / 3 suites passed on 2026-09-24
#67612 infrastructure 8c4d54869dc Unchanged exact six-commit prefix of #67292; last independently verified on 2026-09-22: 75 ASAN/PCH tests / 8 suites, plus 15 analyzer tests

Source defaults commit 5f9e73ce8fe is folded into #67292's scan initialization (0b676dc846a), partial-block manager (0ca4025b827), hole-fill runtime (6c1d0f5bf69) and writeback test fixture (08a1fdd52d7). #67612 defines none of these nine consumer settings: its code/head are unchanged, and only its synchronization description is updated.

The two complete trees still differ only in #67579's validation-only err_utils.cpp no-PCH fix. Existing default-validation tests are aligned; the immediate-writeback fixture explicitly retains zero delay. No new tests or mechanism changes are added. The new ASAN/PCH runs use run-be-ut.sh -j100 without clean; formatting, header hygiene, diff checks and component/tree comparisons pass. No cluster, object-store workload, new analyzer run or clang-tidy run is claimed. Synchronization means pushed, not merged.

2026-09-22 synchronization and validation (historical)

All three branches are rebased onto the same upstream master 01efbca40a5. Recent interval diagnostics, batched trace files, submission/scan timing, deadline-ordered queue scans and dedicated hole-fill dispatch are synchronized by component.

PR Head Structure Validation
#67579 validation a03c3bc8cfd 34 chronological commits ASAN/PCH: 55 tests / 7 suites passed
#67292 full implementation df02672a373 18 component commits Independent ASAN/PCH: 287 tests / 20 suites passed
#67612 infrastructure 8c4d54869dc Exact six-commit prefix of #67292 Independent ASAN/PCH: 75 tests / 8 suites passed

Shared tracing is one additional infrastructure component. Hole-fill lifecycle timing, ordered scans and HoleFillDispatch are folded into the partial-block manager; consumed-range timing/provenance and callback wiring are folded into their owning components. #67612 intentionally excludes scanner/point-query consumers and background hole filling. The two complete source trees differ only in #67579's validation-only err_utils.cpp no-PCH fix. TopN second-stage integration remains outside the series.

Conflicts preserve upstream's column-reader tests and segment-loading refactor. Point-query reads use the new TabletColumn interface and preserve slot-derived defaults before a light-schema-change column reaches the tablet schema; an added BE test covers that case. Scanner tests also use a schema-bearing MockSegment, matching upstream's move of table properties out of StorageReadOptions; the obsolete options assignment is removed.

All runs use the repository scripts, ASAN/PCH and -j100, without a clean rebuild. The Python analyzer's 15 tests pass. Formatting, header hygiene, diff checks, component-prefix and tree-scope comparisons pass. No cluster, external-object-store benchmark or clang-tidy run; these results do not establish a workload speedup. Synchronization means pushed, not merged.

Recent follow-up synchronization ledger

Follow-up checkpoint before rebase Current #67579 commit #67292 synchronization #67612 synchronization
2026-10-15 — Keep complex columns on the original read path c18ca96044f aa6a74f1b18 column readers; 028d4b71f21 scalar-only scan routing; 42a0b9b0a8e point queries Outside infrastructure; code/head unchanged
2026-10-15 — Synchronize the no-PCH unity-build fix 04951048d32 already present 6a63eecdc8b shared build-fix commit 6a63eecdc8b same commit
e3382f4aff0 — Add opt-in read overlap diagnostics 12bc412e479 5ea5103b2de shared trace; 2003f22ceb3 hole lifecycle; 52c4acc24e9 / 2beda87cbdc consumption/writeback 5ea5103b2de shared S3/range/cache diagnostics only
4c87dbbbea2 — Buffer read IO traces in dedicated files d0bdaf78839 5ea5103b2de shared trace component 5ea5103b2de complete shared writer, batching, shutdown and analyzer
7845fde5f6d — Trace hole-fill submission and queue-scan latency 146f5e7b22e 5ea5103b2de shared timing fields/analyzer; 2003f22ceb3 / 52c4acc24e9 producers 5ea5103b2de shared timing/Profile contract and analyzer; no hole-fill producer
1c59525a90f — Avoid scanning delayed hole-fill tasks 197741c187f 2003f22ceb3 partial-block manager Outside infrastructure; tracked for PR 2
b3849c9d65d — Dispatch hole-fill tasks from a dedicated thread b0a3623e9ca 2003f22ceb3 partial-block manager Outside infrastructure; tracked for PR 2
2026-09-24 — Tune read-ahead and hole-fill defaults 050ce408611 028d4b71f21 scan policy; 2003f22ceb3 manager defaults/test; f1e2daab4f1 runtime settings; 52c4acc24e9 immediate-writeback fixture Consumer settings are outside infrastructure; code/head unchanged

All entries above are pushed to the applicable PR branches. Shared infrastructure remains reviewable independently; this validation PR keeps the chronological development commits and its separate no-PCH compile fix. The tracking issue is #67611.

Earlier rebase, benchmark, validation and synchronization checkpoints

Current rebase (2026-09-20)

Rebased only this PR onto upstream master 0451443dd09. Rebase checkpoint: e3382f4aff0; all 30 chronological component commits are retained.

  • Resolve the column-reader test include conflict by preserving both sets of dependencies.
  • Preserve upstream's segment-loading refactor and reuse its loading helper from the point-query read-ahead path.
  • Keep upstream's portable errno messages together with this branch's function-local ErrorCode imports.
  • Validation after rebase: incremental ASAN/PCH build with run-be-ut.sh -j100, 236 tests / 19 suites passed; 9 analyzer tests, clang-format 16, BE header hygiene and diff checks passed. No cluster or external-object-store workload was run.

Only #67579 is rebased and pushed. Other PR branches, their descriptions and tracking issue #67611 are unchanged. The synchronization tables below describe the earlier dated checkpoint.

Latest follow-up: dedicated hole-fill dispatch (2026-09-22)

Current branch head: b3849c9d65d.

The follow-up q4.1 benchmark at the preceding deadline-scan commit reduced cumulative foreground queue-mutex wait to 27.8 ms, but still recorded 1,053,030 scans for approximately 34,479 activated blocks with 32 workers; 1,018,551 scans encountered an unexpired head. This commit targets the remaining repeated timers, wakeups and trace records. Those figures describe the preceding implementation; a workload benchmark of this new dispatcher is still pending.

flowchart LR
    Submit[Foreground fragments] --> Waiting[Bounded mergeable queue]
    Waiting --> Dispatch[HoleFillDispatch<br/>deadline and capacity checks]
    Dispatch --> Ready[Bounded worker handoff]
    Ready --> Workers[Block workers<br/>complete and submit blocks]
    Workers --> Remote[Shared remote-read pool<br/>multi-range GETs]
    Workers --> Writer[Async cache writer]
    Workers -. slot released .-> Dispatch
Loading
  • One dedicated HoleFillDispatch thread owns merge-deadline and writer-capacity waits. Block workers consume ready tasks and retain their reusable remote-read tokens. The existing single-range inline path, parallel remote reads and configured concurrency remain unchanged.
  • Dispatched plus executing tasks are bounded by the configured block-worker count. Overflow stays in the mergeable queue, so dispatch cannot freeze the entire backlog before workers can process it. Queue nodes transfer via splice without allocating new handoff nodes. Active-task metrics include this bounded handoff, not just executing GETs.
  • Tail submissions and task completions notify the dispatcher only when they can advance its current wait. Preserve ordered deadlines, stale replacement, cross-writer capacity bypass, dynamic merge-delay changes and worker resizing. Shrinking allows already-dispatched work to drain before admitting more.
  • Shutdown joins the dispatcher, discards queued and undelivered tasks without issuing their GETs, then waits for executing workers before destroying the remote pool. No new configuration, default changes or foreground-buffer ownership changes.
  • Validation: final incremental ASAN/PCH run-be-ut.sh -j100, 110 tests / 9 suites passed; 15 Python analyzer tests passed. Coverage includes 32 workers with 1024 delayed tasks, full 32-way execution with overflow fragment merging, idle/active resizing, undelivered-task cancellation, deadline/epoch changes, writer capacity, eviction and remote reads. Repository clang-format 16, format checks, BE header hygiene and diff checks passed.

Architecture and trace interpretation. No cluster, external-object-store benchmark or clang-tidy run. Pushed only to #67579; #67292, #67612 and tracking issue #67611 remain unchanged.

Earlier follow-up: deadline-ordered hole-fill scans (2026-09-22)

Deadline-scan checkpoint: 1c59525a90f.

The archived q4.1 trace with a 2000 ms merge window measured 389.63 s of cumulative foreground queue-mutex wait and 78.40 s of locked worker scans. Of 219.53 million examined entries, 99.984% were still delayed. This follow-up removes that repeated delayed-task traversal:

  • Retain the existing list in admission-time order. A stale-epoch replacement moves to the tail in O(1) and starts a fresh merge window; ordinary fragment merges retain their original deadline. Updating the shared merge delay preserves the ordering.
  • Check the deadline before epoch/inflight probes and stop at the first future deadline. A fully delayed queue now requires one head inspection per scan, independent of its length. Trace counts describe visited entries, not the full backlog.
  • Continue scanning due tasks when one cache writer lacks capacity, allowing another writer to proceed. Worker/remote-pool concurrency, capacity retry, notifications, memory limits, merge-delay defaults and hole-read planning are unchanged.
  • Delayed stale/inflight task cleanup is deferred until its deadline; existing memory admission/eviction bounds the queue. Shutdown still clears queued tasks immediately, and pre-read validation remains intact.
  • Validation: incremental ASAN/PCH run-be-ut.sh -j100, 107 tests / 9 suites passed; 15 Python analyzer tests passed. Tests cover a 1024-task delayed queue with exactly one visited entry per scan, replacement ordering and renewed delay, dynamic delay changes, cross-writer capacity bypass, active merges, eviction and shutdown. Clang-format 16, format check, BE header hygiene and diff checks passed.

Updated trace interpretation. No post-change cluster benchmark or clang-tidy run; these tests establish the scan bound, not a query speedup. Pushed only to #67579; #67292, #67612 and tracking issue #67611 remain unchanged.

Earlier follow-up: hole-fill submission and queue-scan timing (2026-09-22)

Timing checkpoint: 7845fde5f6d. One additional diagnostic commit; no changes to read planning, merge delay, queue admission, concurrency or persistence.

The archived q4.1 comparison reduced foreground/hole-fill duplicate bytes with a 2000 ms merge window, but accumulated foreground writeback time increased from about 3.98 s to 271.85 s. Those measurements identify the submission path, not a proven lock-contention cause. This follow-up adds the missing measurements:

  • Profile splits ReadAheadWritebackTime into complete-block and partial-block submission timers, available even with tracing disabled.
  • Under the existing enable_read_io_trace switch, typed timing records separate cache probes, queue mutex wait/hold, block allocation, fragment mutex wait/hold, copies, and existing lifecycle recording. A range-completion event separates complete/partial dispatch.
  • One worker-scan record reports queue size, examined entries, merge-delay/capacity skips, discarded entries, and nested check times. Scan timestamps exclude condition-variable sleep; summaries are serialized after releasing the queue mutex. Process-level scans remain separate from per-query attribution.
  • The analyzer reports totals, p50/p95/p99/max, slow examples, queue-depth groups and successful GET latency by source. The single added BE counter, doris_read_io_trace_record_time_ns, exposes serialization/memory-append overhead. Nested timers and cumulative thread times are explicitly documented so they are not added together or mistaken for query wall time.
  • Validation: incremental ASAN/PCH run-be-ut.sh -j100, 87 tests / 9 suites passed; 15 Python analyzer tests passed. Clang-format 16, format check, BE header hygiene and diff checks passed. No cluster, external-object-store benchmark or clang-tidy run. These tests validate instrumentation, not a workload speedup.

Timing capture and interpretation. Pushed only to #67579; #67292, #67612 and tracking issue #67611 remain unchanged.

Earlier follow-up: interval overlap diagnostics and batched files (2026-09-20)

Diagnostic checkpoint: 4c87dbbbea2. The diagnostic event instrumentation and dedicated-file writer are two additional commits on the existing chronological branch.

  • Mutable enable_read_io_trace=false enables actual S3 GET intervals and outcomes, synchronous/read-ahead/hole-fill provenance, range/task identities, consumption/writeback, queued merges and active-ignored fragments.
  • Events now go to standalone read_io_trace.<process>.<index>.jsonl files, not be.INFO. Foreground threads serialize and append records to a memory batch; one lazy background writer swaps the batch under a short lock and performs file IO outside the lock.
  • Remove read_io_trace_max_events. There is no cumulative event cap, queue-capacity limit, or capacity-based backpressure/drop policy. The startup settings are read_io_trace_dir="" (defaults to the BE log directory's read_io_trace/), read_io_trace_flush_bytes=1048576 (1 MiB), and read_io_trace_flush_interval_ms=1000.
  • Disabling the switch stops new records and lets pending records flush. Normal shutdown and the fast _exit path drain submitted records. Pending-byte visibility and per-batch cumulative checkpoints let the analyzer check complete captures and reported writer failures; existing be.INFO captures remain readable.
  • Observe resident downloaded and inflight coverage before whole-range misses without changing the read decision. The analyzer computes successful bytes minus interval union, separates within-source and cross-source duplication, and links timing examples without double-counting third and later reads.
  • Capture one isolated query through background drain, then disable tracing and wait for doris_read_io_trace_pending_bytes=0 before copying all process files. Reconcile against the same-window s3_file_reader_bytes_read delta. Instrumentation can perturb timing; leave it disabled for performance measurements. Capture and analysis guide.
  • Validation of the batched writer: incremental ASAN/PCH run-be-ut.sh -j100, 81 tests / 8 suites passed; 12 analyzer tests, clang-format 16, BE header hygiene and diff checks passed. Coverage includes threshold/timer flushing, concurrent producers, shutdown and actual file-error recovery. No cluster, object-store workload or clang-tidy run. Reading, planning, merging and writeback mechanisms are unchanged.

Synchronization is intentionally deferred for these diagnostics: pushed only to #67579. #67292, #67612, their descriptions and tracking issue #67611 remain unchanged as requested. The following section records the earlier synchronized checkpoint.

Previous synchronization checkpoint (2026-09-15)

At that checkpoint, the shared master base was a6751eb3ac6b4667c3291544c5cfb1fae48e1556. At that checkpoint, heads were #67579 484f6cd106f, #67292 af40b6c9022, and #67612 cdec0499937. Synced means pushed to the target PR branch, not merged into master.

#67579 keeps its chronological 29 commits. #67292 is organized into 17 component commits; its first five are exactly #67612. Complete-block submission, owned-buffer handoff and low-priority admission are one cache-writer component. The scheduler teardown order enters with query runtime ownership. The full source tree still differs from this validation branch only in the no-PCH err_utils.cpp fix.

Follow-up commits on this branch

Follow-up in #67579 Status in #67292 Status in #67612
b0232d01c73 — No-PCH compile fix Validation-only; excluded. Excluded.
d7b3184ea0e — Read-ahead / hole-fill observability Folded into the owning components; TopN RPC/Profile collection remains removed. Shared statistics and planner/scheduler collection included; consumer collection stays outside.
e81d1fbe422 — Skip redundant hole-fill reads Folded into acdc509c795, consumed-range writeback and query wiring. Outside infrastructure.
47b907ff221 — Page-position window triggers Included in the current page-driven windows 8d4edb506f7 and integrations. Outside infrastructure.
4255f00afa7 — Parallel hole reads / single-range fast path Scheduler/S3 safety e86ee26fa90, block manager acdc509c795 and runtime 5098fccdbc6. Shared S3 statistics safety included; hole-fill execution stays outside.
a28aaea5826 — Queued-fragment aggregation Folded into block manager acdc509c795 and runtime 5098fccdbc6. Outside infrastructure.
4a4052daa49 — Column-planning timers Shared statistics aa6c8651faf, column windows 8d4edb506f7 and column reader 1b2f4146d0c. Shared timer contract and Profile tests included; column producers stay outside.
503e43afed9 — Page-driven scan advancement Folded into windows 8d4edb506f7, segment 0106540dedf, column reader 1b2f4146d0c, scanner b5267a0567a and shared statistics. Shared statistics only; scan mechanism outside.
d5c427fe337 — Object-owned operations Result/state logic e86ee26fa90, plan reset/timing 8d4edb506f7, column integration 1b2f4146d0c and block task/request logic acdc509c795. Scheduler result construction and terminal-state checks included.
484f6cd106f — Read-ahead / hole-fill defaults Folded into scanner b5267a0567a, block manager acdc509c795 and runtime 5098fccdbc6. Outside infrastructure; code unchanged.

Regular scans now build candidate physical pages once after pruning and advance compressed-byte windows at data-page reads, including Page Cache hits. Eager and lazy initial windows are submitted together. Exact-row point/nested reads group row IDs by distinct pages. Profile timers distinguish initialization, physical-page advancement, exact-row planning, and window maintenance. ColumnReadAheadPlan owns reset and timing aggregation; partial-block requests validate themselves and tasks own buffer creation/reads, while the manager keeps queue and lifetime accounting. The 2026-09-15 update changed four defaults only: read_ahead_lazy_window_bytes=1048576 (1 MiB), hole_fill_max_read_amplification_ratio=1.0, hole_fill_max_pending_bytes_per_be=1073741824 (1 GiB), and hole_fill_merge_delay_ms=0. Planning, concurrency, cache-block completion, and queue mechanisms are unchanged.

TopN/global delayed-materialization second-stage integration remains outside the current PRs. Its specific reader/operator/RPC/test files match the shared master base; point-query helpers remain in the final component.

Validation

Branch / PR Head Validation for the 2026-09-15 default-value update
Validation #67579 484f6cd106f 75 existing ASAN/PCH tests / 5 suites passed
Full implementation #67292 af40b6c9022 Independently built; 12 existing ASAN/PCH options and hole-planner tests / 2 suites passed
Infrastructure #67612 cdec0499937 Code unchanged; its 2026-09-14 result remains 72 tests / 8 suites

Both new runs use run-be-ut.sh -j100, ASAN and PCH, without a clean rebuild. No tests were added; the existing production-default assertions were updated. The source run covers column windows, scanner initialization, partial-block workers/options and hole planning. The three changed files are identical in the full and validation branches; their only overall tree difference remains the validation-only err_utils.cpp fix. Repository formatting, BE build hygiene and diff checks pass. No cluster, external-object-store workload or clang-tidy was run; these checks do not establish a workload speedup.

2026-09-14 validation before the default-value update
Branch / PR Then-current head Validation on 2026-09-14
Validation #67579 d5c427fe337 ASAN/PCH: 198 tests / 11 suites passed
Full implementation #67292 270a7dadbb5 Independent ASAN/PCH: 198 tests / 11 suites passed on the identical final source tree
Infrastructure #67612 cdec0499937 Independent ASAN/PCH: 72 tests / 8 suites passed

All runs use run-be-ut.sh -j100, ASAN and PCH, without a clean rebuild. Full-tree validation ran at 92dd0c2d85e; moving the existing scheduler teardown-order fix into its runtime component changed commit history only, and the final tree is byte-identical. The tests cover page-driven forward/reverse scans, Page Cache progression, exact-row preparation, plan reuse and statistics, independent-window buffer reuse, asynchronous reads and hole filling. Formatting, BE build hygiene and diff checks pass. No cluster, external-object-store workload or clang-tidy was run; these tests do not establish a workload speedup.

17 component commits: correspondence across the three PRs
Component Original commit(s) in #67579 Commit in #67292 #67612
Range coalescer 698f82ef834 e0205d32d15 e0205d32d15
Cache-aware planner and shared metrics 023c56eac37 aa6c8651faf aa6c8651faf
Asynchronous range scheduler 5a18d25af91 e86ee26fa90 e86ee26fa90
Exact no-write cached reads b48b029f358 6cea8dccf14 6cea8dccf14
Query runtime ownership 4ab7550815d cdec0499937 cdec0499937
Page-driven column windows 088e48731c6 8d4edb506f7 Later split PRs
Segment coordination / PageIO 35a96c79304 0106540dedf Later split PRs
Column-reader integration 6c418885c01 1b2f4146d0c Later split PRs
Regular scanner initialization b8bf74629f7 b5267a0567a Later split PRs
Complete-block handoff and admission 6366831e430, 73af53b7847, 93005a691d4 3a015cc8913 Later split PRs
Cache-block range splitting 394fa2f41f5 8dd2206087e Later split PRs
Hole-fill planner 5424828fda8 6a83e43e531 Later split PRs
Partial-block writeback manager c870edbd4f6 acdc509c795 Later split PRs
Hole-fill runtime wiring ba7b2c49afa 5098fccdbc6 Later split PRs
Consumed-range writeback 4baed43cf6f 5587ca87d29 Later split PRs
Query writeback wiring a39685eee60 f6c34677d88 Later split PRs
Point queries and exact-row helpers 4fb1bd0ca89 af40b6c9022 Later split PRs

Release note

None for the additional compile fix. The Phase 2 feature changes are described in #67292.

Check List (For Author)

  • Test:
    • The no-PCH reproduction and Release build below are earlier validation; current target-branch ASAN/PCH synchronization results are recorded above.
    • Manual test: reproduced the original LIMIT_REACH compilation failure with ENABLE_PCH=OFF and ENABLE_UNITY_BUILD=ON using ./build.sh --be -j100.
    • Post-fix BE build: ./build.sh --be -j100 passed through compilation, linking, and packaging (Successfully build Doris), using the existing Release configuration with ENABLE_PCH=OFF and ENABLE_UNITY_BUILD=ON.
    • build-support/clang-format.sh (clang-format 16), build-support/check-format.sh, build-support/check-build-hygiene.sh, and git diff --check passed.
    • clang-tidy was not continued, as requested.
  • Behavior changed:
    • No runtime behavior change in the additional compile fix.
  • Does this need documentation?
    • No additional documentation for this compile fix.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
bobhan1 force-pushed the fix/phase2-no-pch-build branch 4 times, most recently from e46032a to 5c594ab Compare September 10, 2026 08:20
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579

Problem Summary: Column read-ahead uses paired high and low byte watermarks to decide when to extend its window. Pending-byte accounting also changes after Page Cache hits and rejected submissions, even when the scan has not advanced. Replace the watermarks with one compressed-byte window size per column role and a trigger page position. Each window uses its second selected page as the trigger, or its only page for a one-page window. First access and a jump beyond planned coverage establish a window through the same path. Process current-batch pages in scan order so a batch crossing the trigger can append the following window in the same plan. Keep mandatory current-batch pages, cache probing, IO coalescing, asynchronous reads, and writeback on their existing paths.

### Release note

Replace the eager/lazy high and low watermark settings with read_ahead_eager_window_bytes (4 MiB) and read_ahead_lazy_window_bytes (128 KiB). Query page read-ahead advances by scan position and retains its existing query and BE buffer budgets.

### Check List (For Author)

- Test: Unit Test — 132 targeted ASAN tests, including 24 ColumnReadAhead cases and related scanner, rowid/TopN, range IO, and cache-reader tests; clang-format 16, build hygiene, and git diff --check passed. No cluster or regression workload was run.
- Behavior changed: Yes — page-position triggers replace pending-byte watermarks; first access and large jumps share the same window initialization.
- Does this need documentation: Yes — configuration and component interface comments are updated in this commit.
@bobhan1
bobhan1 force-pushed the fix/phase2-no-pch-build branch from a48e80c to 25eef4d Compare September 11, 2026 04:49
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579

Problem Summary: Column read-ahead uses paired high and low byte watermarks to decide when to extend its window. Pending-byte accounting also changes after Page Cache hits and rejected submissions, even when the scan has not advanced. Replace the watermarks with one compressed-byte window size per column role and a trigger page position. Each window uses its second selected page as the trigger, or its only page for a one-page window. First access and a jump beyond planned coverage establish a window through the same path. Process current-batch pages in scan order so a batch crossing the trigger can append the following window in the same plan. Keep mandatory current-batch pages, cache probing, IO coalescing, asynchronous reads, and writeback on their existing paths.

### Release note

Replace the eager/lazy high and low watermark settings with read_ahead_eager_window_bytes (4 MiB) and read_ahead_lazy_window_bytes (128 KiB). Query page read-ahead advances by scan position and retains its existing query and BE buffer budgets.

### Check List (For Author)

- Test: Unit Test — 132 targeted ASAN tests, including 24 ColumnReadAhead cases and related scanner, rowid/TopN, range IO, and cache-reader tests; clang-format 16, build hygiene, and git diff --check passed. No cluster or regression workload was run.
- Behavior changed: Yes — page-position triggers replace pending-byte watermarks; first access and large jumps share the same window initialization.
- Does this need documentation: Yes — configuration and component interface comments are updated in this commit.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: Each hole-fill worker issued the source reads for a cache block sequentially. Keep block workers responsible for planning and writeback, and execute their disjoint source ranges in a separate shared pthread pool. A per-block concurrent ThreadPoolToken joins accepted reads before either submitting the completed buffer or dropping the block. Preserve queued reads during shutdown and allow the read pool to resize without interrupting active GETs. Make the shared S3 reader statistics atomic for concurrent foreground and background reads.

### Release note

Partial File Cache blocks can fetch hole ranges concurrently. The new mutable BE setting hole_fill_read_threads_per_be defaults to 64 and controls the shared background source-read pool independently of the block-worker count.

### Check List (For Author)

- Test: Unit Test — 132 related BE unit tests passed through run-be-ut.sh with ASAN, PCH and -j100; 8 concurrency/statistics tests also passed 3 shuffled repetitions. clang-format v16, check-format, build hygiene and git diff --check passed.
- Behavior changed: Yes — source reads within one partial cache block can execute concurrently, and failed blocks are released after their accepted reads finish.
- Does this need documentation: Yes — the new BE setting and execution/lifecycle contracts are documented in code comments.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: Each hole-fill worker issued the source reads for a cache block sequentially. Keep block workers responsible for planning and writeback, and execute their disjoint remote ranges in a separate shared HoleFillRemoteReadPool. A per-block concurrent ThreadPoolToken joins accepted reads before either submitting the completed buffer or dropping the block. Preserve queued reads during shutdown and allow the remote-read pool to resize without interrupting active GETs. Make the shared S3 reader statistics atomic for concurrent foreground and background reads. Initialize both hole-fill and asynchronous cache-write worker pools with their configured worker counts as the minimum, and retain at least one remote-read pool thread.

### Release note

Partial File Cache blocks can fetch hole ranges concurrently. The new mutable BE setting hole_fill_remote_read_threads_per_be defaults to 64 and controls the shared background remote-read pool independently of the block-worker count.

### Check List (For Author)

- Test: Unit Test — 62 existing BE unit tests passed through run-be-ut.sh with ASAN, PCH and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest and PartialBlockWritebackOptionsTest. The naming and minimum-thread follow-up adds no new test cases. clang-format v16, check-format, build hygiene and git diff --check passed.
- Behavior changed: Yes — remote reads within one partial cache block can execute concurrently, and failed blocks are released after their accepted reads finish. Worker pools start with nonzero minimum thread counts.
- Does this need documentation: Yes — the new BE setting and execution/lifecycle contracts are documented in code comments.
@bobhan1
bobhan1 force-pushed the fix/phase2-no-pch-build branch from 3796127 to d8d795d Compare September 11, 2026 07:13
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: Each hole-fill worker issued the source reads for a cache block sequentially. Keep block workers responsible for planning and writeback, and execute their disjoint remote ranges in a separate shared HoleFillRemoteReadPool. Each block worker reuses one concurrent ThreadPoolToken across blocks, joining each block's accepted reads before either submitting the completed buffer or dropping the block. Destroy the token before signalling worker completion so the remote-read pool can be released safely. Preserve queued reads during shutdown and allow the remote-read pool to resize without interrupting active GETs. Make the shared S3 reader statistics atomic for concurrent foreground and background reads. Initialize both hole-fill and asynchronous cache-write worker pools with their configured worker counts as the minimum, and retain at least one remote-read pool thread.

### Release note

Partial File Cache blocks can fetch hole ranges concurrently. The new mutable BE setting hole_fill_remote_read_threads_per_be defaults to 64 and controls the shared background remote-read pool independently of the block-worker count.

### Check List (For Author)

- Test: Unit Test — 62 existing BE unit tests passed through run-be-ut.sh with ASAN, PCH and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest and PartialBlockWritebackOptionsTest. The naming, minimum-thread and token-reuse follow-ups add no new test cases. clang-format v16, check-format, build hygiene and git diff --check passed.
- Behavior changed: Yes — remote reads within one partial cache block can execute concurrently, and failed blocks are released after their accepted reads finish. Worker pools start with nonzero minimum thread counts.
- Does this need documentation: Yes — the new BE setting and execution/lifecycle contracts are documented in code comments.
@bobhan1
bobhan1 force-pushed the fix/phase2-no-pch-build branch from d8d795d to f63b6e1 Compare September 11, 2026 07:24
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: Each hole-fill worker issued the source reads for a cache block sequentially. Keep block workers responsible for planning and writeback. Read a single hole range directly on its block worker and execute multiple disjoint ranges in a separate shared HoleFillRemoteReadPool. Each block worker reuses one concurrent ThreadPoolToken for multi-range blocks, joining accepted reads before either submitting the completed buffer or dropping the block. Destroy the token before signalling worker completion so the remote-read pool can be released safely. Preserve queued reads during shutdown and allow the remote-read pool to resize without interrupting active GETs. Make the shared S3 reader statistics atomic for concurrent foreground and background reads. Initialize both hole-fill and asynchronous cache-write worker pools with their configured worker counts as the minimum, and retain at least one remote-read pool thread.

### Release note

Partial File Cache blocks can fetch hole ranges concurrently. The new mutable BE setting hole_fill_remote_read_threads_per_be defaults to 64 and controls the shared pool for multi-range hole reads. Single-range hole reads execute directly on block workers, outside this pool's concurrency limit.

### Check List (For Author)

- Test: Unit Test — 62 existing BE unit tests passed through run-be-ut.sh with ASAN, PCH and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest and PartialBlockWritebackOptionsTest. The follow-up refinements adapt existing cases without adding new tests. clang-format v16, check-format, build hygiene and git diff --check passed.
- Behavior changed: Yes — remote reads within one partial cache block can execute concurrently, and failed blocks are released after their accepted reads finish. Single-range hole reads execute directly on block workers. Worker pools start with nonzero minimum thread counts.
- Does this need documentation: Yes — the new BE setting and execution/lifecycle contracts are documented in code comments.
@bobhan1
bobhan1 force-pushed the fix/phase2-no-pch-build branch from f63b6e1 to 1cde489 Compare September 11, 2026 07:38
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: A partial block can become active immediately after its first fragment arrives, leaving little opportunity to absorb subsequent foreground fragments before planning remote hole reads. Keep tasks mergeable in the existing queue for a configurable minimum interval measured from successful admission. Later fragments preserve that timestamp. Workers skip tasks whose delay has not elapsed and wait on the existing condition variable until the earliest deadline or cache-writer capacity retry. Runtime updates apply to queued tasks and wake workers without affecting active reads. Preserve complete-block direct writeback, queued-task eviction, cache invalidation checks, shutdown, and existing worker and remote-read concurrency.

### Release note

Add mutable BE configuration hole_fill_merge_delay_ms, defaulting to 10 ms. When partial-block writeback is enabled, this setting gives queued blocks a short interval to merge foreground fragments before hole filling. Set it to 0 to disable the delay. Changing it affects existing queued tasks; merging another fragment does not restart the interval.

### Check List (For Author)

- Test: Unit Test — 77 existing tests passed through run-be-ut.sh with ASAN, PCH, and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest, PartialBlockWritebackOptionsTest, HoleFillPlannerTest, RangeCacheWritebackTest, and RangeWritebackDispatcherTest. Extended existing cases for minimum delay, stable admission deadlines, dynamic updates, active-task isolation, eviction, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: Yes — queued partial blocks wait for the configured aggregation interval before becoming active; complete-block writeback and active tasks keep their existing behavior.
- Does this need documentation: Yes — the new configuration and queue timing contract are documented in code comments.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: A partial block can become active immediately after its first fragment arrives, leaving little opportunity to absorb subsequent foreground fragments before planning remote hole reads. Keep tasks mergeable in the existing queue for a configurable minimum interval measured from successful admission. Later fragments preserve that timestamp. Workers skip tasks whose delay has not elapsed and wait on the existing condition variable until the earliest deadline or cache-writer capacity retry. Assert that a finite wakeup deadline has been selected before entering the timed wait. Runtime updates apply to queued tasks and wake workers without affecting active reads. Preserve complete-block direct writeback, queued-task eviction, cache invalidation checks, shutdown, and existing worker and remote-read concurrency.

### Release note

Add mutable BE configuration hole_fill_merge_delay_ms, defaulting to 10 ms. When partial-block writeback is enabled, this setting gives queued blocks a short interval to merge foreground fragments before hole filling. Set it to 0 to disable the delay. Changing it affects existing queued tasks; merging another fragment does not restart the interval.

### Check List (For Author)

- Test: Unit Test — 77 existing tests passed through run-be-ut.sh with ASAN, PCH, and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest, PartialBlockWritebackOptionsTest, HoleFillPlannerTest, RangeCacheWritebackTest, and RangeWritebackDispatcherTest. Extended existing cases for minimum delay, stable admission deadlines, dynamic updates, active-task isolation, eviction, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. The subsequent DCHECK/comment-only addition passed formatting and static path review; build and unit tests were not rerun for that addition. No cluster tests or clang-tidy were run.
- Behavior changed: Yes — queued partial blocks wait for the configured aggregation interval before becoming active; complete-block writeback and active tasks keep their existing behavior.
- Does this need documentation: Yes — the new configuration and queue timing contract are documented in code comments.
@bobhan1
bobhan1 force-pushed the fix/phase2-no-pch-build branch 2 times, most recently from 13266fa to a28aaea Compare September 11, 2026 09:01
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579

Problem Summary: Column read-ahead uses paired high and low byte watermarks to decide when to extend its window. Pending-byte accounting also changes after Page Cache hits and rejected submissions, even when the scan has not advanced. Replace the watermarks with one compressed-byte window size per column role and a trigger page position. Each window uses its second selected page as the trigger, or its only page for a one-page window. First access and a jump beyond planned coverage establish a window through the same path. Process current-batch pages in scan order so a batch crossing the trigger can append the following window in the same plan. Keep mandatory current-batch pages, cache probing, IO coalescing, asynchronous reads, and writeback on their existing paths.

### Release note

Replace the eager/lazy high and low watermark settings with read_ahead_eager_window_bytes (4 MiB) and read_ahead_lazy_window_bytes (128 KiB). Query page read-ahead advances by scan position and retains its existing query and BE buffer budgets.

### Check List (For Author)

- Test: Unit Test — 132 targeted ASAN tests, including 24 ColumnReadAhead cases and related scanner, rowid/TopN, range IO, and cache-reader tests; clang-format 16, build hygiene, and git diff --check passed. No cluster or regression workload was run.
- Behavior changed: Yes — page-position triggers replace pending-byte watermarks; first access and large jumps share the same window initialization.
- Does this need documentation: Yes — configuration and component interface comments are updated in this commit.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: Each hole-fill worker issued the source reads for a cache block sequentially. Keep block workers responsible for planning and writeback. Read a single hole range directly on its block worker and execute multiple disjoint ranges in a separate shared HoleFillRemoteReadPool. Each block worker reuses one concurrent ThreadPoolToken for multi-range blocks, joining accepted reads before either submitting the completed buffer or dropping the block. Destroy the token before signalling worker completion so the remote-read pool can be released safely. Preserve queued reads during shutdown and allow the remote-read pool to resize without interrupting active GETs. Make the shared S3 reader statistics atomic for concurrent foreground and background reads. Initialize both hole-fill and asynchronous cache-write worker pools with their configured worker counts as the minimum, and retain at least one remote-read pool thread.

### Release note

Partial File Cache blocks can fetch hole ranges concurrently. The new mutable BE setting hole_fill_remote_read_threads_per_be defaults to 64 and controls the shared pool for multi-range hole reads. Single-range hole reads execute directly on block workers, outside this pool's concurrency limit.

### Check List (For Author)

- Test: Unit Test — 62 existing BE unit tests passed through run-be-ut.sh with ASAN, PCH and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest and PartialBlockWritebackOptionsTest. The follow-up refinements adapt existing cases without adding new tests. clang-format v16, check-format, build hygiene and git diff --check passed.
- Behavior changed: Yes — remote reads within one partial cache block can execute concurrently, and failed blocks are released after their accepted reads finish. Single-range hole reads execute directly on block workers. Worker pools start with nonzero minimum thread counts.
- Does this need documentation: Yes — the new BE setting and execution/lifecycle contracts are documented in code comments.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: A partial block can become active immediately after its first fragment arrives, leaving little opportunity to absorb subsequent foreground fragments before planning remote hole reads. Keep tasks mergeable in the existing queue for a configurable minimum interval measured from successful admission. Later fragments preserve that timestamp. Workers skip tasks whose delay has not elapsed and wait on the existing condition variable until the earliest deadline or cache-writer capacity retry. Assert that a finite wakeup deadline has been selected before entering the timed wait. Runtime updates apply to queued tasks and wake workers without affecting active reads. Preserve complete-block direct writeback, queued-task eviction, cache invalidation checks, shutdown, and existing worker and remote-read concurrency.

### Release note

Add mutable BE configuration hole_fill_merge_delay_ms, defaulting to 10 ms. When partial-block writeback is enabled, this setting gives queued blocks a short interval to merge foreground fragments before hole filling. Set it to 0 to disable the delay. Changing it affects existing queued tasks; merging another fragment does not restart the interval.

### Check List (For Author)

- Test: Unit Test — 77 existing tests passed through run-be-ut.sh with ASAN, PCH, and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest, PartialBlockWritebackOptionsTest, HoleFillPlannerTest, RangeCacheWritebackTest, and RangeWritebackDispatcherTest. Extended existing cases for minimum delay, stable admission deadlines, dynamic updates, active-task isolation, eviction, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. The subsequent DCHECK/comment-only addition passed formatting and static path review; build and unit tests were not rerun for that addition. No cluster tests or clang-tidy were run.
- Behavior changed: Yes — queued partial blocks wait for the configured aggregation interval before becoming active; complete-block writeback and active tasks keep their existing behavior.
- Does this need documentation: Yes — the new configuration and queue timing contract are documented in code comments.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 11, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579

Problem Summary: ReadAheadPlanTime starts in SegmentReadAhead::apply_plans after physical-column preparation. It therefore omits initialization, request validation, per-row page lookup and window maintenance, including work that produces an empty plan. Add elapsed-time counters for physical-column preparation, its initialization and validation stage, window discard, current-batch planning and nested window extension. Collect the inner durations per column call and report them before empty plans are dropped. Keep the existing range-planning timer unchanged for comparison with earlier profiles. Timers are scoped to column calls and window operations rather than individual row lookups. No read-ahead, IO coalescing, cache writeback or hole-fill decisions change.

### Release note

Add ReadAheadColumnPlanTime and nested initialization, window-discard, current-batch and window-extension elapsed timers under PageReadAhead in query profiles. Parent timers include their children; the existing ReadAheadPlanTime retains its range-planning and submission scope. No configuration or bvar metrics change.

### Check List (For Author)

- Test: Unit Test — 50 tests from 5 suites passed through run-be-ut.sh with ASAN and -j100, covering ReadAheadMetricsTest, ColumnReadAheadTest, SegmentReadAheadTest, SegmentIteratorReadAheadTest and the relevant ColumnReaderTest cases. Extended existing tests for timer hierarchy, delta reporting, reused plan outputs and empty-plan accounting. clang-format v16, check-format, build hygiene and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: No — timing collection and Profile output only.
- Does this need documentation: No — timer boundaries and nesting are documented inline and represented in the Profile hierarchy.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 14, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579

Problem Summary: The SSB Flat SF1000 planning-delay0 comparison attributed 5,242.6 seconds of accumulated column-planning elapsed time to the current-rowid loop excluding window extension, or 91.6% of column planning. Each scanner batch repeatedly validates rowids, locates the same physical pages for every row, and walks the existing window for every column. Window extension itself accounted for only 6.7 seconds. These are accumulated elapsed counters, not query wall time or thread CPU time.

Build each scan column's candidate-page sequence once after segment pruning and submit the initial eager/lazy windows together. Advance subsequent windows directly from the ordinal page index at the data-page read entry, including Page Cache hits and sparse seeks. Retain high-page predictions while reverse scanners read an ascending batch. Keep point reads and dependent nested-column reads on the exact-rowid path, grouping rows by page. Reuse already buffered physical pages when column windows advance independently. Keep IO coalescing, cache-block completion, hole filling and all existing configuration defaults unchanged. Report page advancement separately through ReadAheadPageAdvanceTime; initialization includes candidate selection, while window-discard and window-extension counters remain sub-timers. This change has not yet been benchmarked on the SSB cluster.

### Release note

Reduce query read-ahead planning work by preparing candidate pages once and advancing windows on data-page reads. Add ReadAheadPageAdvanceTime to the query Profile; ReadAheadWindowExtendTime is now a shared sub-timer under ReadAheadColumnPlanTime rather than a child of current-batch planning.

### Check List (For Author)

- Test: Unit Test — 190 tests from 10 suites passed through run-be-ut.sh with ASAN and -j100, covering complete ColumnReaderTest, column/segment read-ahead, metrics, IO coalescing, range planning, asynchronous read scheduling, range writeback and hole filling. Added dense-scan, sparse/reverse progression, Page Cache, admission fallback and independent-window buffer-reuse coverage. clang-format v16, check-format, build hygiene and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: Yes — common scans plan once after pruning and advance on physical page reads instead of planning every rowid batch. Query results and synchronous fallback semantics are preserved.
- Does this need documentation: Yes — the scan planning flow and Profile timer scope need a follow-up synchronization in the design repository.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 14, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292, apache#67612

Problem Summary: Plan reset and timing aggregation, asynchronous read result construction and terminal-state checks, and partial-block request validation were implemented outside the objects whose state they describe. Move these operations to ColumnReadAheadPlan, FileRangeReadSubmitResult, FileRangeRead, and PartialBlockWritebackRequest. Keep partial-block buffer creation and remote reads on Task, while the manager retains queue admission, worker scheduling, and lifecycle accounting. Preserve vector capacity during plan reset, per-call statistics, lock boundaries, and the join of accepted remote reads before task cleanup.

### Release note

None

### Check List (For Author)

- Test: Unit Test — 198 tests from 11 suites passed through run-be-ut.sh with ASAN and -j100. Added plan-reset capacity/ownership, empty-plan timing aggregation, and rejected-result coverage. Existing read-ahead, scheduler, cache-writeback and hole-fill tests passed, including concurrent reads, failed submission/read cleanup, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: No — move object-local operations without changing read-ahead, IO, or writeback mechanisms.
- Does this need documentation: No — internal responsibility cleanup; configuration and metric meanings are unchanged.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 14, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579, apache#67612

Problem Summary: Build each scan column's eligible physical page indexes once and advance compressed-byte windows from the page index resolved by the column reader. A window's second selected page triggers the following window. Retain predictions for ascending batches inside reverse scans; group exact-row requests by distinct pages. ColumnReadAheadPlan owns reset and per-call timing aggregation, retaining vector capacity. Window accounting stays with ColumnReadAhead.

### Release note

Internal page-driven column windows for query read-ahead.

### Check List (For Author)

- Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run.
- Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics.
- Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Sep 14, 2026
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579, apache#67612

Problem Summary: Coordinate all resolvable column plans, probe Page Cache, plan and submit ranges, and register exact page slices. Reuse a physical page's existing buffer when independent column windows overlap it. Keep synchronous fallback and per-page ownership accounting; expose planner and consumption statistics.

### Release note

Internal segment-level coordination for concurrent data-page reads.

### Check List (For Author)

- Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run.
- Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics.
- Does this need documentation: Yes — component scope and current behavior are tracked in apache#67611 and the related PR descriptions.
Issue Number: None

Related PR: None

Problem Summary: Background hole filling produces an owned complete cache-block buffer. Add a handoff to the asynchronous cache writer that preserves that buffer without another payload copy, while retaining epoch fencing, inflight deduplication, and queue admission. Also expose a point-in-time spare-capacity check so low-priority hole-fill reads start only when the write queue can accept a block without evicting foreground work.

None

- Test: Unit Test
    - ./run-be-ut.sh -j100 --run '--filter=AsyncCacheWriteManagerTest.SubmitOwnedBlockPreservesBufferIdentity:AsyncCacheWriteManagerTest.ReportsOnlyUnusedPendingCapacity:AsyncCacheWriteManagerTest.SubmitBlockCopiesDeduplicatesAndRollsBackBackpressure:AsyncCacheWriteManagerTest.SubmitBlockRejectsStaleEpochAndAllocationFailure'
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: A background hole-fill task can observe spare write capacity before its remote read and find the queue full at final handoff. Add a spare-capacity-only admission mode so this low-priority result is rejected instead of evicting an already queued foreground cache write. Normal submissions retain the existing oldest-queued replacement behavior.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh -j100 --run '--filter=AsyncCacheWriteManagerTest.ReportsOnlyUnusedPendingCapacity:AsyncCacheWriteManagerTest.DropOldestReplacesOnlyOldestQueuedTaskAndKeepsInflightReaderAlive:AsyncCacheWriteManagerTest.SubmitOwnedBlockPreservesBufferIdentity'
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292

Problem Summary: Query read-ahead can leave partially covered File Cache blocks that need bounded asynchronous completion before entering the cache writer. Add a BE-level manager that merges queued fragments, deduplicates active blocks, evicts the oldest queued block under pressure, fills block-local holes through dedicated remote-read workers, and hands completed buffers to the asynchronous cache writer without another copy. The worker set can grow or shrink at runtime; shrinking waits only for retiring workers to finish their active remote reads. Same-block submissions are detected before allocating another block buffer, and their fragment copies use a task-local lock so unrelated submissions do not hold the BE-wide queue lock behind a memcpy.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run '--filter=PartialBlockWritebackManagerTest.*:PartialBlockWritebackOptionsTest.*' -j100
- Behavior changed: No; this commit adds an unwired internal component.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292

Problem Summary: Background partial-block writeback needs production defaults and a BE-owned lifetime bounded by the File Cache lifecycle. Add validated hole-fill configuration, create the manager after File Cache initialization, and stop it before File Cache teardown. Make hole_fill_workers_per_be mutable so a runtime configuration update resizes the dedicated remote-read workers without rebuilding the manager; allow 1 through 128 workers and keep the default at 32.

### Release note

Support runtime resizing of BE-wide hole-fill workers through hole_fill_workers_per_be.

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run '--filter=PartialBlockWritebackOptionsTest.*' -j100
- Behavior changed: Yes; runtime updates to hole_fill_workers_per_be resize the active hole-fill worker set.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292

Problem Summary: Query read-ahead produces consumed variable-length buffers that can span complete and partial File Cache blocks. Add RangeCacheWriteback to split each consumed range, submit complete blocks directly to the existing asynchronous write path, and enqueue partial fragments for background hole filling under the same invalidation epoch captured before the remote read starts.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh -j100 --run --filter=RangeCacheWritebackTest.*
- Behavior changed: No; this commit adds an internal writeback router that is not connected to SegmentIterator yet.
- Does this need documentation: No
Issue Number: None

Related PR: apache#67292

Problem Summary: Query read-ahead must publish only buffers that were actually consumed, preserve caller-visible IO statistics, and honor the original cache-write policy. Centralize query read-ahead construction so SegmentIterator and exact row-id readers share the same byte windows, range planner, page-cache probe, scheduler, and writeback setup. Each asynchronous submission owns its statistics and File Cache epoch; the first consumed page publishes the completed range once, while no-write and remote-only requests skip cache writeback.

Enable consumed query read-ahead ranges to populate File Cache when asynchronous cache writeback is enabled.

- Test: Unit Test
    - ./run-be-ut.sh --run --filter=AsyncCachedRemoteFileReaderTest.*:SegmentReadAheadTest.*:SegmentIteratorReadAheadTest.*:ColumnReaderTest.*ReadAhead* -j100
- Behavior changed: Yes; consumed successful query read-ahead ranges now contribute IO statistics and enter cache writeback when policy permits.
- Does this need documentation: Yes; covered by the read-ahead and hole-fill design document.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292

Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Cold data pages previously followed the synchronous reader one page or column at a time. Create one request-scoped range-read context, initialize the row-store or all missing-column iterators against the shared read-ahead reader, submit the required data pages before decoding, and retain the existing row-cache and result-padding behavior. Read-ahead initialization or IO failure falls back to the original reader. Include the reusable row-id page preparation helpers, SegmentReadAhead exact-row entry point, and its tests in this point-query component. TopN second-stage integration remains unchanged from master and is outside this PR series.

### Release note

Use query range read-ahead for point-query data pages when enable_query_read_ahead is enabled.

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run --filter=SegmentReadAheadTest.*:RowCacheTest.*:LookupConnectionCacheTest.* -j100
- Behavior changed: Yes; point-query data-page reads use the range read-ahead path when the feature is enabled.
- Does this need documentation: Yes; covered by the read-ahead and hole-fill design document.

# Conflicts:
#	be/src/service/point_query_executor.cpp
#	be/src/storage/tablet/base_tablet.h

# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 3dc7341
# Last commands done (19 commands done):
#    pick b9396d9 # [feature](be) Connect read-ahead range writeback
#    pick 7ccfb97 # [feature](be) Extend range read-ahead to point queries
# Next commands to do (21 remaining commands):
#    pick 9a6bcd5 # [fix](be) Scope filesystem error-code imports to their functions
#    pick 06c0770 # [improvement](be) Add page read-ahead and hole-fill observability
# You are currently rebasing branch 'fix/phase2-no-pch-build' on '3dc73418a07'.
#
# Changes to be committed:
#	modified:   be/src/service/point_query_executor.cpp
#	new file:   be/src/storage/segment/rowid_read_ahead.cpp
#	new file:   be/src/storage/segment/rowid_read_ahead.h
#	modified:   be/src/storage/segment/segment_read_ahead.cpp
#	modified:   be/src/storage/segment/segment_read_ahead.h
#	modified:   be/src/storage/tablet/base_tablet.cpp
#	modified:   be/src/storage/tablet/base_tablet.h
#	modified:   be/test/storage/segment/segment_read_ahead_test.cpp
#
# Untracked files:
#	.codex
#
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292

Problem Summary: With ENABLE_PCH=OFF and unity builds enabled, the IO unity translation unit includes err_utils.cpp before file_meta_cache.cpp. The namespace-scope ErrorCode using-directive then makes LIMIT_REACH ambiguous when the latter includes the generated protobuf enum. Move the using-directive into the two localfs_error overloads that need it, preserving the existing error-code mappings while preventing it from affecting sibling source files.

### Release note

None

### Check List (For Author)

- Test: Manual test. Reproduced the original ENABLE_PCH=OFF compile failure with ./build.sh --be -j100; build-support/clang-format.sh, build-support/check-format.sh, build-support/check-build-hygiene.sh, and git diff --check passed. Post-fix BE compilation is tracked in the compilation-only Draft PR.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: apache#67611 (tracking; keep open)

Related PR: apache#67292

Problem Summary: Page read-ahead, IO coalescing and partial-block writeback lack query-level attribution and BE-wide capacity and IO signals. Add named RuntimeProfile counters for planning, compressed candidate and cache-hit bytes, submission outcomes, source IO, page consumption and foreground writeback. Distinguish Query/BE budget rejection and executor rejection from failures of accepted reads, and account queued cancellations only for accepted tasks. Add successful-source and BE cumulative IO/consumption bytes so asynchronous tail work remains observable after the final Profile snapshot without delaying query completion. Record synchronous data-page fallback attempts, requested bytes and time at PageIO, including decode retries while excluding Page Cache hits and index/dictionary reads. Add aggregate hole-fill failure/read-time metrics and keep detailed failure causes in logs. Use COUNTER_UPDATE and scoped timers, with statistics owned independently of the reporting profile so outstanding reads can finish safely. Append point-query statistics to the existing sampled diagnostic log.

### Release note

Expose PageReadAhead query profile counters and BE-wide metrics for read-ahead IO, data-page fallback, buffer capacity and background cache-block hole filling.

### Check List (For Author)

- Test: Unit Test
    - 131 targeted ASAN tests from 18 suites passed via run-be-ut.sh -j100 with PCH enabled, as recorded in the original accounting commit.
    - clang-format 16, check-format, build hygiene and git diff --check passed before squashing.
    - The historical results above predate removal of TopN-specific collection; current branch validation is recorded in apache#67611.
- Behavior changed: Yes (observability only; read scheduling, cancellation, fallback and writeback policies are unchanged)
- Does this need documentation: Yes (doris-io/phase2_page_prefetch_and_io_coalescing/observability.md)
### What problem does this PR solve?

Issue Number: apache#67611 (tracking; keep open)

Related PR: apache#67292

Problem Summary: Consuming a read-ahead range served from File Cache could enqueue partial-block writeback and reread the rest of an already cached block from remote storage. Downloaded blocks were only detected later by the cache writer, after the redundant remote reads. Skip range writeback for local/inflight hits and combine inflight-buffer lookup with the existing cache probe to deduplicate blocks already cached or being filled. Perform the combined check before partial-block admission and once when a worker starts the task, outside the hole-fill queue mutex. Execute the planned hole reads without per-GET probes. Preserve existing queues, concurrency, active-task fragment handling, and final cache-writer deduplication and invalidation checks.

### Release note

Avoid redundant remote hole-fill reads for blocks already cached or being filled.

### Check List (For Author)

- Test: Unit Test
    - 105 targeted ASAN tests from 9 suites passed through run-be-ut.sh -j100 with PCH enabled on the same final source tree before squashing, including cached/downloading blocks, EOF, empty cells, queued and active completion races, inflight fast-path locking, mixed cached/uncached ranges, and local-hit read-ahead consumption.
    - clang-format 16, check-format, build hygiene, and git diff --check passed before squashing.
    - No cluster or regression workload was run. Clang-tidy was not run as requested.
- Behavior changed: Yes (skip redundant best-effort writeback and hole-fill IO; query results and scheduling limits are unchanged)
- Does this need documentation: No (cache-reuse fix with no configuration changes)
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579

Problem Summary: Column read-ahead uses paired high and low byte watermarks to decide when to extend its window. Pending-byte accounting also changes after Page Cache hits and rejected submissions, even when the scan has not advanced. Replace the watermarks with one compressed-byte window size per column role and a trigger page position. Each window uses its second selected page as the trigger, or its only page for a one-page window. First access and a jump beyond planned coverage establish a window through the same path. Process current-batch pages in scan order so a batch crossing the trigger can append the following window in the same plan. Keep mandatory current-batch pages, cache probing, IO coalescing, asynchronous reads, and writeback on their existing paths.

### Release note

Replace the eager/lazy high and low watermark settings with read_ahead_eager_window_bytes (4 MiB) and read_ahead_lazy_window_bytes (128 KiB). Query page read-ahead advances by scan position and retains its existing query and BE buffer budgets.

### Check List (For Author)

- Test: Unit Test — 132 targeted ASAN tests, including 24 ColumnReadAhead cases and related scanner, rowid/TopN, range IO, and cache-reader tests; clang-format 16, build hygiene, and git diff --check passed. No cluster or regression workload was run.
- Behavior changed: Yes — page-position triggers replace pending-byte watermarks; first access and large jumps share the same window initialization.
- Does this need documentation: Yes — configuration and component interface comments are updated in this commit.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: Each hole-fill worker issued the source reads for a cache block sequentially. Keep block workers responsible for planning and writeback. Read a single hole range directly on its block worker and execute multiple disjoint ranges in a separate shared HoleFillRemoteReadPool. Each block worker reuses one concurrent ThreadPoolToken for multi-range blocks, joining accepted reads before either submitting the completed buffer or dropping the block. Destroy the token before signalling worker completion so the remote-read pool can be released safely. Preserve queued reads during shutdown and allow the remote-read pool to resize without interrupting active GETs. Make the shared S3 reader statistics atomic for concurrent foreground and background reads. Initialize both hole-fill and asynchronous cache-write worker pools with their configured worker counts as the minimum, and retain at least one remote-read pool thread.

### Release note

Partial File Cache blocks can fetch hole ranges concurrently. The new mutable BE setting hole_fill_remote_read_threads_per_be defaults to 64 and controls the shared pool for multi-range hole reads. Single-range hole reads execute directly on block workers, outside this pool's concurrency limit.

### Check List (For Author)

- Test: Unit Test — 62 existing BE unit tests passed through run-be-ut.sh with ASAN, PCH and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest and PartialBlockWritebackOptionsTest. The follow-up refinements adapt existing cases without adding new tests. clang-format v16, check-format, build hygiene and git diff --check passed.
- Behavior changed: Yes — remote reads within one partial cache block can execute concurrently, and failed blocks are released after their accepted reads finish. Single-range hole reads execute directly on block workers. Worker pools start with nonzero minimum thread counts.
- Does this need documentation: Yes — the new BE setting and execution/lifecycle contracts are documented in code comments.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292

Problem Summary: A partial block can become active immediately after its first fragment arrives, leaving little opportunity to absorb subsequent foreground fragments before planning remote hole reads. Keep tasks mergeable in the existing queue for a configurable minimum interval measured from successful admission. Later fragments preserve that timestamp. Workers skip tasks whose delay has not elapsed and wait on the existing condition variable until the earliest deadline or cache-writer capacity retry. Assert that a finite wakeup deadline has been selected before entering the timed wait. Runtime updates apply to queued tasks and wake workers without affecting active reads. Preserve complete-block direct writeback, queued-task eviction, cache invalidation checks, shutdown, and existing worker and remote-read concurrency.

### Release note

Add mutable BE configuration hole_fill_merge_delay_ms, defaulting to 10 ms. When partial-block writeback is enabled, this setting gives queued blocks a short interval to merge foreground fragments before hole filling. Set it to 0 to disable the delay. Changing it affects existing queued tasks; merging another fragment does not restart the interval.

### Check List (For Author)

- Test: Unit Test — 77 existing tests passed through run-be-ut.sh with ASAN, PCH, and -j100, covering AsyncCacheWriteManagerTest, PartialBlockWritebackManagerTest, PartialBlockWritebackOptionsTest, HoleFillPlannerTest, RangeCacheWritebackTest, and RangeWritebackDispatcherTest. Extended existing cases for minimum delay, stable admission deadlines, dynamic updates, active-task isolation, eviction, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. The subsequent DCHECK/comment-only addition passed formatting and static path review; build and unit tests were not rerun for that addition. No cluster tests or clang-tidy were run.
- Behavior changed: Yes — queued partial blocks wait for the configured aggregation interval before becoming active; complete-block writeback and active tasks keep their existing behavior.
- Does this need documentation: Yes — the new configuration and queue timing contract are documented in code comments.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579

Problem Summary: ReadAheadPlanTime starts in SegmentReadAhead::apply_plans after physical-column preparation. It therefore omits initialization, request validation, per-row page lookup and window maintenance, including work that produces an empty plan. Add elapsed-time counters for physical-column preparation, its initialization and validation stage, window discard, current-batch planning and nested window extension. Collect the inner durations per column call and report them before empty plans are dropped. Keep the existing range-planning timer unchanged for comparison with earlier profiles. Timers are scoped to column calls and window operations rather than individual row lookups. No read-ahead, IO coalescing, cache writeback or hole-fill decisions change.

### Release note

Add ReadAheadColumnPlanTime and nested initialization, window-discard, current-batch and window-extension elapsed timers under PageReadAhead in query profiles. Parent timers include their children; the existing ReadAheadPlanTime retains its range-planning and submission scope. No configuration or bvar metrics change.

### Check List (For Author)

- Test: Unit Test — 50 tests from 5 suites passed through run-be-ut.sh with ASAN and -j100, covering ReadAheadMetricsTest, ColumnReadAheadTest, SegmentReadAheadTest, SegmentIteratorReadAheadTest and the relevant ColumnReaderTest cases. Extended existing tests for timer hierarchy, delta reporting, reused plan outputs and empty-plan accounting. clang-format v16, check-format, build hygiene and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: No — timing collection and Profile output only.
- Does this need documentation: No — timer boundaries and nesting are documented inline and represented in the Profile hierarchy.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579

Problem Summary: The SSB Flat SF1000 planning-delay0 comparison attributed 5,242.6 seconds of accumulated column-planning elapsed time to the current-rowid loop excluding window extension, or 91.6% of column planning. Each scanner batch repeatedly validates rowids, locates the same physical pages for every row, and walks the existing window for every column. Window extension itself accounted for only 6.7 seconds. These are accumulated elapsed counters, not query wall time or thread CPU time.

Build each scan column's candidate-page sequence once after segment pruning and submit the initial eager/lazy windows together. Advance subsequent windows directly from the ordinal page index at the data-page read entry, including Page Cache hits and sparse seeks. Retain high-page predictions while reverse scanners read an ascending batch. Keep point reads and dependent nested-column reads on the exact-rowid path, grouping rows by page. Reuse already buffered physical pages when column windows advance independently. Keep IO coalescing, cache-block completion, hole filling and all existing configuration defaults unchanged. Report page advancement separately through ReadAheadPageAdvanceTime; initialization includes candidate selection, while window-discard and window-extension counters remain sub-timers. This change has not yet been benchmarked on the SSB cluster.

### Release note

Reduce query read-ahead planning work by preparing candidate pages once and advancing windows on data-page reads. Add ReadAheadPageAdvanceTime to the query Profile; ReadAheadWindowExtendTime is now a shared sub-timer under ReadAheadColumnPlanTime rather than a child of current-batch planning.

### Check List (For Author)

- Test: Unit Test — 190 tests from 10 suites passed through run-be-ut.sh with ASAN and -j100, covering complete ColumnReaderTest, column/segment read-ahead, metrics, IO coalescing, range planning, asynchronous read scheduling, range writeback and hole filling. Added dense-scan, sparse/reverse progression, Page Cache, admission fallback and independent-window buffer-reuse coverage. clang-format v16, check-format, build hygiene and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: Yes — common scans plan once after pruning and advance on physical page reads instead of planning every rowid batch. Query results and synchronous fallback semantics are preserved.
- Does this need documentation: Yes — the scan planning flow and Profile timer scope need a follow-up synchronization in the design repository.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67579, apache#67292, apache#67612

Problem Summary: Plan reset and timing aggregation, asynchronous read result construction and terminal-state checks, and partial-block request validation were implemented outside the objects whose state they describe. Move these operations to ColumnReadAheadPlan, FileRangeReadSubmitResult, FileRangeRead, and PartialBlockWritebackRequest. Keep partial-block buffer creation and remote reads on Task, while the manager retains queue admission, worker scheduling, and lifecycle accounting. Preserve vector capacity during plan reset, per-call statistics, lock boundaries, and the join of accepted remote reads before task cleanup.

### Release note

None

### Check List (For Author)

- Test: Unit Test — 198 tests from 11 suites passed through run-be-ut.sh with ASAN and -j100. Added plan-reset capacity/ownership, empty-plan timing aggregation, and rejected-result coverage. Existing read-ahead, scheduler, cache-writeback and hole-fill tests passed, including concurrent reads, failed submission/read cleanup, and shutdown. clang-format v16, check-format, build hygiene, and git diff --check passed. No cluster tests or clang-tidy were run.
- Behavior changed: No — move object-local operations without changing read-ahead, IO, or writeback mechanisms.
- Does this need documentation: No — internal responsibility cleanup; configuration and metric meanings are unchanged.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking)

Related PR: apache#67292, apache#67579

Problem Summary: Adjust the opt-in read-ahead and hole-fill defaults together: use a 1 MiB lazy-column window, a 1.0 background hole-fill amplification limit, a 1 GiB BE-wide pending partial-block budget, and zero deliberate fragment-aggregation delay. Align PartialBlockWritebackOptions and the existing production-default test with the registered settings. Keep planning, queueing, concurrency, validators, and configuration mutability unchanged.

### Release note

For query read-ahead, which remains disabled by default, set read_ahead_lazy_window_bytes to 1048576 (1 MiB), hole_fill_max_read_amplification_ratio to 1.0, hole_fill_max_pending_bytes_per_be to 1073741824 (1 GiB), and hole_fill_merge_delay_ms to 0.

### Check List (For Author)

- Test: Unit Test
    - 75 existing tests from 5 focused ASAN/PCH read-ahead and hole-fill suites passed via run-be-ut.sh -j100; no new tests.
    - clang-format 16 formatting/check, BE build hygiene, and git diff --check.
- Behavior changed: Yes; adjust four defaults for opt-in read-ahead and hole filling without changing the mechanisms.
- Does this need documentation: Yes; synchronize configuration values and commit mappings in apache#67292, apache#67579, and apache#67611.
### What problem does this PR solve?

Issue Number: apache#67611 (tracking; this diagnostic does not complete the feature)

Related PR: apache#67579

Problem Summary: Aggregate S3 and hole-fill byte counters cannot establish whether foreground reads and background hole filling fetched the same object intervals. Add disabled-by-default, bounded JSON tracing at actual S3 GETs and the read-ahead consumption / partial-block lifecycle boundaries. Carry stable range/task identities and source labels through IOContext, and take read-only cache/inflight coverage snapshots before whole-range misses. Provide an offline interval-sweep analyzer that counts extra successful transfers once, distinguishes within-source and cross-source overlap, links representative GETs to late fragments, and checks capture continuity and S3 byte reconciliation. Keep IO planning, merging, admission, concurrency and writeback decisions unchanged. No cluster or benchmark is run by this change.

### Release note

Add mutable BE settings enable_read_io_trace=false and read_io_trace_max_events=1000000 for temporary S3/read-ahead/hole-fill diagnostics. The diagnostic logs can affect timing when enabled and are intended for isolated captures, not performance comparisons. See tools/read_io_trace/README.md for capture and analysis instructions.

### Check List (For Author)

- Test: Unit Test
    - 73 targeted ASAN/PCH BE tests from 6 suites passed through run-be-ut.sh -j100.
    - 9 offline analyzer tests passed, including interval-union arithmetic, triple reads, source/query isolation, temporal examples, missing logs and reconciliation failure.
    - Repository clang-format 16, formatting check, BE header hygiene and git diff --check passed.
    - No cluster, external object storage workload, performance benchmark or clang-tidy run.
- Behavior changed: Yes; opt-in diagnostic logging and coverage probes only. Read and writeback mechanisms are unchanged.
- Does this need documentation: Yes; tools/read_io_trace/README.md is included in this commit.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67579

Problem Summary: Read-overlap diagnostics currently put every event in be.INFO and stop recording after a process-wide event limit, mixing diagnostic records with operational logs and truncating complete workload captures. Append serialized records to an owned memory batch and let a lazy background writer flush it to dedicated JSONL files by byte threshold or interval. Remove the event limit without adding queue-capacity limits or backpressure. Drain submitted records during normal and fast BE shutdown. Add flush checkpoints and pending-byte visibility so the offline analyzer can check complete captures and reported file errors.

### Release note

Opt-in read IO diagnostics now use read_io_trace_dir (default: the BE log directory's read_io_trace subdirectory). read_io_trace_max_events is removed. New startup settings read_io_trace_flush_bytes and read_io_trace_flush_interval_ms default to 1 MiB and 1000 ms. enable_read_io_trace remains dynamically switchable and disabled by default.

### Check List (For Author)

- Test: Unit Test
    - Incremental ASAN/PCH run-be-ut.sh -j100: 81 tests from 8 suites passed, including dedicated-file batching, interval flushing, concurrent producers, shutdown and actual file-error recovery.
    - Offline analyzer: 12 Python unit tests passed.
    - clang-format 16, BE header hygiene and git diff --check passed. No cluster workload or clang-tidy run.
- Behavior changed: Yes; enabled diagnostic events go to batched standalone JSONL files instead of be.INFO, without a cumulative event limit. Read-ahead and hole-fill mechanisms are unchanged.
- Does this need documentation: Yes; tools/read_io_trace/README.md is updated in this commit.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67579

Problem Summary: A longer hole-fill fragment aggregation window reduced duplicate foreground/background reads in the q4.1 experiment, but foreground writeback submission time increased. The existing Profile and interval trace cannot distinguish shared queue mutex contention from cache probes, buffer allocation, fragment merging, copies or diagnostic recording. Add complete/partial dispatch Profile timers and opt-in typed timing records for foreground submissions and locked worker scans. Serialize scan summaries outside the queue mutex and keep condition-variable sleeps outside scan timings. Extend the offline analyzer with stage totals, percentiles, queue-depth groups and slow examples, preserving process scope for scans shared by queries. Read, merge, admission and persistence decisions remain unchanged.

### Release note

The existing enable_read_io_trace switch now also records writeback submission stages and hole-fill queue scans in the dedicated JSONL trace. Two lightweight Profile timers separate complete-block and partial-block submission even with tracing disabled. The doris_read_io_trace_record_time_ns counter measures diagnostic serialization and memory-append overhead. No new settings or default changes.

### Check List (For Author)

- Test: Unit Test
    - Incremental ASAN/PCH run-be-ut.sh -j100: 87 tests from 9 suites passed.
    - Python offline analyzer: 15 tests passed.
    - Repository clang-format 16, check-format, BE header hygiene and git diff --check passed.
    - No cluster, external-object-store benchmark or clang-tidy run.
- Behavior changed: Yes, additional diagnostic output and Profile counters; IO and queue mechanisms unchanged.
- Does this need documentation: Yes, capture boundaries, timing interpretation and comparison workflow are documented in tools/read_io_trace/README.md in this commit.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67579

Problem Summary: With a 2000 ms hole-fill merge window, the archived q4.1 trace measured 389.63 s of cumulative foreground queue-mutex wait and 78.40 s of locked worker scans; 99.984% of examined tasks had not reached their merge deadline. Workers repeatedly probed every delayed task under the same mutex needed by foreground fragment submission. Keep the existing list ordered by admission time, moving stale-epoch replacements to the tail with a fresh merge window. Check the deadline before epoch/inflight probes and stop at the first delayed entry. A fully delayed queue now requires one examined entry per scan, verified with 1024 queued tasks. Continue past capacity-blocked due tasks so another cache writer can make progress. Preserve runtime delay updates, worker concurrency, fragment merging and hole-read planning. Workload latency gains have not been measured after this change.

### Release note

Reduce foreground writeback contention when hole-fill fragment aggregation uses a delay, without adding configuration or changing its defaults.

### Check List (For Author)

- Test: Unit Test
    - Incremental ASAN/PCH run-be-ut.sh -j100: 107 tests from 9 suites passed.
    - Coverage includes a 1024-task delayed queue, stale-epoch replacement order and renewed delay, runtime delay changes, cross-writer capacity bypass, active merges, eviction and shutdown.
    - Python trace analyzer: 15 tests passed.
    - Repository clang-format 16, check-format, BE header hygiene and git diff --check passed.
    - No cluster, external-object-store benchmark or clang-tidy run.
- Behavior changed: Yes, delayed tasks are no longer repeatedly scanned or probed; stale replacements join the queue tail. Due-task checks, memory bounds, shutdown cleanup and concurrency limits remain in place.
- Does this need documentation: Yes, visited-entry semantics are documented in tools/read_io_trace/README.md in this commit.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67579

Problem Summary: The archived q4.1 deadline-scan run removed queue-lock contention but still recorded 1,053,030 scans for about 34,479 activated blocks with 32 workers; 1,018,551 scans encountered an unexpired head. Multiple workers independently waited for the same deadline. Introduce one HoleFillDispatch thread for deadline and capacity decisions and keep block workers dedicated to execution with reusable remote-read tokens. Bound dispatched plus executing tasks by worker count, transfer queue nodes with splice, and suppress notifications that cannot advance the current wait. Preserve queued fragment merging, dynamic resizing and cross-writer capacity bypass, and cancel undelivered tasks at shutdown. No post-change workload speedup is claimed.

Remove the offline tools/read_io_trace analyzer and its tests from this feature; retain the BE-side trace capture and diagnostic settings.

### Release note

Use a single hole-fill dispatcher to avoid redundant deadline wakeups while preserving configured read concurrency and queued fragment aggregation. No new configuration or default changes.

### Check List (For Author)

- Test: Unit Test
    - Pre-change incremental ASAN/PCH run-be-ut.sh -j100: 110 tests from 9 suites passed.
    - Coverage includes 32 workers with 1024 delayed tasks, full 32-way execution with overflow merging, idle and active resizing, handoff cancellation, epoch replacement, cache-writer capacity, eviction and remote reads.
    - Repository clang-format 16, check-format, BE header hygiene and git diff --check passed.
    - No cluster, external-object-store benchmark or clang-tidy run.
- Behavior changed: Yes, one dispatcher owns timing and hands off at most the configured worker slots; queued and undelivered tasks are cancelled at shutdown.
- Does this need documentation: Yes, dispatcher architecture and BE tracing are described in the active PRs.
### What problem does this PR solve?

Issue Number: Part of apache#67611

Related PR: apache#67292, apache#67579

Problem Summary: Partial-block writeback should collect foreground fragments
for 2000 ms by default and allow 4 GiB of pending buffers per BE. Default
foreground range planning should introduce no gap or partial-block completion
bytes, so set its amplification ratio and block-fill coverage to 1.0.

Keep the existing 1 MiB lazy window, 4 MiB eager window, 1.0 hole-fill
amplification ratio, 32 hole-fill workers and 64 remote read threads. Align
PartialBlockWritebackOptions with the configured merge delay and retain explicit
zero-delay scheduling in the existing immediate-writeback test fixture.
No scheduling or IO algorithm changes.

### Release note

When query read-ahead and partial-block writeback are enabled, the default
merge delay is 2000 ms and pending-buffer budget is 4 GiB per BE. Foreground
range merging and block completion default to a 1.0 amplification ratio and
1.0 minimum coverage respectively. Query read-ahead remains disabled by default.

### Check List (For Author)

- Test: ASAN BE unit tests via run-be-ut.sh -j100; 121 tests from 9 suites passed.
    - Formatting, build hygiene and git diff --check passed.
- Behavior changed: Yes, the four configurable defaults above.
- Does this need documentation: Yes, synchronize the related PR descriptions and tracking issue.
### What problem does this PR solve?

Issue Number: Part of apache#67611

Related PR: apache#67579, apache#67292

Problem Summary: The P0 regression aborts in MapFileColumnIterator::next_batch when a nested STRUCT/MAP lazy read leaves one child pruned. A skipped child retains its old ordinal while the active sibling seeks to the required items; the newly added prefetch assertion incorrectly requires their ordinals to match, even with query read-ahead disabled. Restrict the new read-ahead path to top-level scalar columns, remove complex-column preparation and dependent-item hooks, and restore the upstream ARRAY/MAP/STRUCT read implementations. Select the reader per column and retain the original segment prefetchers for complex columns. Point-query preparation likewise leaves complex iterators on Segment's existing path. Add coverage for divergent nested MAP ordinals, mixed scalar/complex scan routing, and complex point-query preparation.

### Release note

Keep ARRAY/MAP/STRUCT on the existing read path when query read-ahead is enabled and prevent an abort during nested lazy materialization.

### Check List (For Author)

- Test: Unit Test — 134 tests from 4 suites passed on the integrated apache#67579 tree with ./run-be-ut.sh -j100 --run (ASAN/PCH, no clean). Repository clang-format 16, header hygiene and git diff --check passed. No cluster or clang-tidy run.
- Behavior changed: Yes — only top-level scalar columns use the new page read-ahead; legacy complex-column prefetch and common IO/writeback mechanisms are preserved.
- Does this need documentation: Yes — scope and synchronization are recorded in apache#67611 and the related PR descriptions.
### What problem does this PR solve?

Issue Number: Part of apache#67611

Related PR: apache#67292

Problem Summary: Read-ahead fetches with NO_WRITE and depends on asynchronous writeback after consumption. Enabling it without the asynchronous cache writer silently prevents cold data pages from filling File Cache. Gate both scan and point-query read-ahead on the async-write switch so this configuration retains synchronous cache filling.

### Release note

Query read-ahead takes effect only when asynchronous File Cache writes are also enabled.

### Check List (For Author)

- Test: Unit Test added for the disabled-writer combination and synchronous cold-read cache reuse; run as part of the repair-series validation.
- Behavior changed: Yes, read-ahead remains inactive while asynchronous File Cache writes are disabled.
- Does this need documentation: Yes, dependency is recorded in the PR description.
### What problem does this PR solve?

Issue Number: Part of apache#67611

Related PR: apache#67292

Problem Summary: A delayed hole-fill task retained the inner remote reader, while destruction of its CachedRemoteFileReader wrapper closed that same inner reader. Later reads failed on HDFS and could abort DCHECK-enabled S3 builds. Use an aliasing shared_ptr whose pointer is the remote reader and whose owner is the cached wrapper, retaining that owner through each background task.

### Release note

Keep background cache hole-fill readers open until their tasks finish.

### Check List (For Author)

- Test: Unit Test added for submitting a delayed fragment, releasing the wrapper and writeback context, and completing cache fill; run as part of the repair-series validation.
- Behavior changed: Yes, queued hole fills retain their cached reader's lifetime.
- Does this need documentation: No, internal lifetime correction.
### What problem does this PR solve?

Issue Number: apache#67611

Related PR: apache#67292

Problem Summary: A point request can pass the read-ahead enable check and then observe either mutable feature switch disabled while loading its segment. The row-store helper asserted that the factory still returned a read-ahead object, aborting the BE. Reuse the already loaded segment and the original synchronous column iterator when the factory returns no object. A deterministic test closes each switch between segment loading and factory creation, reproduces SIGABRT before the fix, and verifies identical row-store bytes after the fix.

### Release note

Prevent BE aborts when query read-ahead or asynchronous File Cache writes are disabled during row-store point queries.

### Check List (For Author)

- Test: ASAN/PCH BE unit test PointQueryReadAheadTest.RowStoreFallsBackWhenReadAheadSwitchChanges passed; the same test reproduced SIGABRT before the fix. Build hygiene and clang-format 16 passed.
- Behavior changed: Yes; a runtime switch change falls back to the original reader.
- Does this need documentation: No; preserve the existing dynamic switch contract.

# Conflicts:
#	be/test/service/point_query_exector_test.cpp

# Please enter the commit message for your changes. Lines starting
# with '#' will be kept; you may remove them yourself if you want to.
# An empty message aborts the commit.
#
# interactive rebase in progress; onto 3dc7341
# Last commands done (39 commands done):
#    pick 47d9e5e # [fix](be) Keep cached readers alive during background hole filling
#    pick 60f0f2e # [fix](be) Fall back when row-store read-ahead is disabled
# Next command to do (1 remaining command):
#    pick 1d9b406 # [improvement](be) Admit read-ahead ranges within remaining byte budgets
# You are currently rebasing branch 'fix/phase2-no-pch-build' on '3dc73418a07'.
#
# Changes to be committed:
#	modified:   be/src/storage/tablet/base_tablet.cpp
#	modified:   be/test/service/point_query_exector_test.cpp
#
# Untracked files:
#	.codex
#
### What problem does this PR solve?

Issue Number: apache#67611

Related PR: apache#67292

Problem Summary: A wide scan submits the initial windows of all columns together. Whole-batch byte admission rejected every range when their aggregate size exceeded a query or BE budget, discarding useful concurrency even when many ranges could fit. Admit ranges in input order with one linear pass under both budget locks, skip ranges that do not fit, and allocate only admitted buffers. Return null handles at skipped input positions so the existing planner mapping remains intact. Segment coordination registers admitted page slices and completes skipped pages for synchronous fallback. Preserve per-buffer reservation lifetime, allocation rollback, executor failures, and cancellation. Count partially budget-rejected batches without double-counting an executor rejection; configuration defaults, windows, coalescing, and hole filling are unchanged.

### Release note

Retain partial read-ahead concurrency when a planned range batch exceeds its query or BE byte budget.

### Check List (For Author)

- Test: ASAN/PCH BE unit tests through ./run-be-ut.sh --run -j100: 141 tests from 8 suites passed. Coverage includes partial query/BE admission, oversized ranges followed by smaller ranges, original page offsets, consumed-range callbacks, cancellation during allocation, allocation rollback, and executor rejection accounting. Clang-format 16, header hygiene and git diff --check passed.
- Behavior changed: Yes; ranges that fit remain asynchronous while skipped ranges use the original synchronous path.
- Does this need documentation: Yes; update the component behavior in the existing PR and tracking descriptions.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292, apache#67579

Problem Summary: File Cache initialization rejects an S3 upload buffer size that is smaller than a cache block or not a multiple of the block size. The buffer size is mutable, but runtime updates did not register the same validation, so a later update could violate the layout assumed by cached reads and hole-fill deduplication. Register the existing alignment constraint as a config validator and cover accepted updates, rejected updates with value rollback, invalid block sizes, and File Cache disabled behavior. Cache probing, read-ahead, and hole-fill algorithms remain unchanged.

### Release note

Reject S3 upload buffer configuration updates that violate File Cache block alignment while File Cache is enabled.

### Check List (For Author)

- Test: Unit Test — 66 tests from 4 suites pass with ./run-be-ut.sh --run -j100 (ASAN/PCH), including configuration validation and partial-block/async cache writeback. Direct Release/PCH BE build passes with ./build.sh --be -j100. Clang-format 16, build hygiene, and diff checks pass. No cluster or clang-tidy run.
- Behavior changed: Yes — invalid runtime S3 upload buffer updates return INVALID_ARGUMENT and retain the previous value; defaults and IO mechanisms are unchanged.
- Does this need documentation: No — applies the existing File Cache startup constraint to runtime configuration updates.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#67292, apache#67579

Problem Summary: Older binaries could persist non-aligned File Cache blocks after a runtime S3 upload-buffer update. These intervals survive cache reopening even after the buffer setting is corrected. Partial-block writeback deduplication used a grid-only cache probe, so encountering a restored irregular interval could abort the BE. Check coverage using the actual cached intervals, accepting contiguous DOWNLOADED or DOWNLOADING blocks without allocating cache cells or changing LRU state. Keep the inflight shortcut and the existing grid probe unchanged.

### Release note

Avoid a BE abort during partial-block writeback when File Cache restores historical non-aligned blocks.

### Check List (For Author)

- Test: The restored-cache regression test reproduced SIGABRT before the fix. All 92 related tests in 5 suites passed with run-be-ut.sh --run -j100 (ASAN/PCH), including restored irregular cache gaps and mixed downloaded/downloading coverage. A direct Release/PCH BE build passed with build.sh --be -j100. Clang-format 16, build hygiene and git diff --check passed. No cluster or clang-tidy run.
- Behavior changed: Yes. Optional writeback deduplication supports actual cached block boundaries, including restored irregular blocks.
- Does this need documentation: No. The PR descriptions record this compatibility fix; public configuration semantics are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants