Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7654cf7
[feature](be) Add linear file range coalescing
bobhan1 Aug 28, 2026
90cf5b6
[feature](be) Add cache-aware file range planning
bobhan1 Aug 28, 2026
85894d0
[feature](be) Add file range read scheduler
bobhan1 Aug 28, 2026
c64e719
[feature](be) Reuse remote-only reads for no-write ranges
bobhan1 Aug 28, 2026
563b5ba
[feature](be) Add query read-ahead runtime ownership
bobhan1 Aug 28, 2026
e2a4012
[improvement](be) Add opt-in file range IO tracing
bobhan1 Sep 22, 2026
8a08ae1
[fix](be) Scope filesystem error-code imports to their functions
bobhan1 Oct 15, 2026
4271da9
[feature](be) Add page-driven column read-ahead windows
bobhan1 Aug 28, 2026
e7dbfa1
[feature](be) Add segment range read-ahead coordination
bobhan1 Oct 29, 2026
00fa56b
[feature](be) Connect scalar column readers to range read-ahead
bobhan1 Sep 2, 2026
8190e0f
[feature](be) Initialize scan read-ahead once after pruning
bobhan1 Oct 29, 2026
d03f274
[feature](be) Add complete-block cache write handoff and admission
bobhan1 Aug 28, 2026
a547700
[feature](be) Split consumed ranges for cache writeback
bobhan1 Aug 28, 2026
7cfe76c
[feature](be) Add cache block hole-fill planning
bobhan1 Aug 28, 2026
a7e3899
[feature](be) Add background partial block writeback
bobhan1 Aug 28, 2026
999f5c6
[feature](be) Initialize background hole fill
bobhan1 Aug 28, 2026
6fd7bb6
[feature](be) Route consumed ranges to cache writeback
bobhan1 Aug 28, 2026
1ae1272
[feature](be) Connect read-ahead range writeback
bobhan1 Aug 28, 2026
fd6df2c
[feature](be) Extend range read-ahead to point queries
bobhan1 Sep 2, 2026
a881473
[fix](be) Validate S3 upload buffer alignment on config updates
bobhan1 Oct 29, 2026
b237ebc
[fix](be) Handle irregular cache blocks in writeback coverage checks
bobhan1 Oct 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[feature](be) Add cache-aware file range planning
### What problem does this PR solve?

Issue Number: #67611 (tracking)

Related PR: #67292, #67579, #67612

Problem Summary: Add cache-aware range planning and the shared read-ahead statistics contract. Coalesce sorted requested ranges, optionally complete high-coverage boundary blocks, and map exact inputs back to final buffers. Keep planning timers for column initialization, exact-row batches, physical-page advancement, window discard and extension in the shared contract; collection remains with the later consumers.

### Release note

Internal range planning and shared read-ahead Profile/bvar definitions.

### Check List (For Author)

- Test: Unit Test — component coverage is included with this change. The integrated source snapshot d5c427f passed 198 ASAN tests from 11 suites via run-be-ut.sh -j100. This individual intermediate commit was not built separately; target-branch validation is recorded on the PR. Repository clang-format v16, build hygiene and diff checks passed on the integrated tree. No cluster or clang-tidy run.
- Behavior changed: Yes — introduces the opt-in component; this synchronization preserves existing defaults and IO/writeback semantics.
- Does this need documentation: Yes — component scope and current behavior are tracked in #67611 and the related PR descriptions.
  • Loading branch information
bobhan1 committed Oct 29, 2026
commit 90cf5b6ce615b8c4e48ac62625772082970fc0fe
258 changes: 258 additions & 0 deletions be/src/io/cache/file_range_planner.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,258 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#include "io/cache/file_range_planner.h"

#include <algorithm>
#include <cmath>
#include <map>
#include <utility>

#include "common/logging.h"
#include "io/fs/read_ahead_metrics.h"

namespace doris::io {

Status FileRangePlanOptions::validate() const {
RETURN_IF_ERROR(coalesce_options.validate());
if (cache_block_size == 0) {
return Status::InvalidArgument("file range plan cache block size must be positive");
}
if (!std::isfinite(block_fill_min_coverage) || block_fill_min_coverage <= 0.0 ||
block_fill_min_coverage > 1.0) {
return Status::InvalidArgument(
"file range plan block fill coverage must be finite and in (0, 1]");
}
return Status::OK();
}

namespace {

struct BlockCandidate {
FileRange range;
size_t missing_base_bytes {0};
};

Status validate_input_ranges(const std::vector<FileRange>& input_ranges, size_t file_size) {
size_t previous_end = 0;
for (const auto& range : input_ranges) {
if (range.size == 0) {
return Status::InvalidArgument("input file range at offset {} is empty", range.offset);
}
if (range.offset > file_size || range.size > file_size - range.offset) {
return Status::InvalidArgument(
"input file range at offset {} with size {} exceeds file size {}", range.offset,
range.size, file_size);
}
if (range.offset < previous_end) {
return Status::InvalidArgument(
"input file ranges must be sorted and disjoint: offset {} follows end {}",
range.offset, previous_end);
}
previous_end = range.end();
}
return Status::OK();
}

FileRange block_range(size_t offset, size_t file_size, size_t block_size) {
const size_t block_offset = offset / block_size * block_size;
return {.offset = block_offset, .size = std::min(block_size, file_size - block_offset)};
}

size_t intersection_bytes(const FileRange& left, const FileRange& right) {
const size_t begin = std::max(left.offset, right.offset);
const size_t end = std::min(left.end(), right.end());
return begin < end ? end - begin : 0;
}

// Count coverage in `target` with a monotonic cursor over sorted, disjoint ranges. The caller must
// visit targets in ascending order.
size_t covered_bytes(const std::vector<FileRange>& ranges, const FileRange& target,
size_t* cursor) {
while (*cursor < ranges.size() && ranges[*cursor].end() <= target.offset) {
++*cursor;
}

size_t result = 0;
for (size_t index = *cursor; index < ranges.size() && ranges[index].offset < target.end();
++index) {
result += intersection_bytes(ranges[index], target);
}
return result;
}

void append_boundary_block(std::vector<FileRange>* blocks, const FileRange& range, size_t file_size,
size_t block_size) {
const FileRange block = block_range(range.offset, file_size, block_size);
if (!blocks->empty() && blocks->back().offset == block.offset) {
DORIS_CHECK(blocks->back() == block);
return;
}
DORIS_CHECK(blocks->empty() || blocks->back().end() <= block.offset);
blocks->push_back(block);
}

std::vector<BlockCandidate> find_block_candidates(const std::vector<FileRange>& input_ranges,
const std::vector<FileRange>& base_ranges,
size_t file_size,
const FileRangePlanOptions& options) {
// Interior blocks are already covered by a base read, so only its two boundary blocks can add
// bytes. This bounds the candidate count to twice the number of base ranges.
std::vector<FileRange> boundary_blocks;
boundary_blocks.reserve(base_ranges.size() * 2);
for (const auto& base : base_ranges) {
append_boundary_block(&boundary_blocks, base, file_size, options.cache_block_size);
append_boundary_block(&boundary_blocks, {.offset = base.end() - 1, .size = 1}, file_size,
options.cache_block_size);
}

std::vector<BlockCandidate> candidates;
candidates.reserve(boundary_blocks.size());
size_t input_cursor = 0;
size_t base_cursor = 0;
for (const auto& block : boundary_blocks) {
const size_t input_bytes = covered_bytes(input_ranges, block, &input_cursor);
const double coverage = static_cast<double>(input_bytes) / static_cast<double>(block.size);
if (coverage < options.block_fill_min_coverage) {
continue;
}
const size_t base_bytes = covered_bytes(base_ranges, block, &base_cursor);
DORIS_CHECK(base_bytes <= block.size);
if (base_bytes < block.size) {
candidates.push_back({.range = block, .missing_base_bytes = block.size - base_bytes});
}
}
std::ranges::sort(candidates, [](const BlockCandidate& left, const BlockCandidate& right) {
return std::pair {left.missing_base_bytes, left.range.offset} <
std::pair {right.missing_base_bytes, right.range.offset};
});
return candidates;
}

// Treat each base range as one component, then try boundary-block fills from the fewest added bytes
// to the most. An accepted fill joins only the components it overlaps and still obeys the physical
// range-size limit.
std::vector<FileRange> choose_components(const std::vector<FileRange>& input_ranges,
const std::vector<FileRange>& base_ranges,
size_t file_size, const FileRangePlanOptions& options) {
std::map<size_t, FileRange> components;
for (const auto& base : base_ranges) {
components.emplace(base.offset, base);
}

// Try the cheapest fills first. The interval map updates only components touched by a
// candidate instead of rebuilding every component after each attempt.
const auto candidates = find_block_candidates(input_ranges, base_ranges, file_size, options);
for (const auto& candidate : candidates) {
auto first = components.lower_bound(candidate.range.offset);
if (first != components.begin()) {
auto previous = first;
--previous;
if (previous->second.end() > candidate.range.offset) {
first = previous;
}
}
DORIS_CHECK(first != components.end());
DORIS_CHECK(first->second.offset < candidate.range.end());

size_t merged_offset = std::min(candidate.range.offset, first->second.offset);
size_t merged_end = candidate.range.end();
auto last = first;
while (last != components.end() && last->second.offset < candidate.range.end()) {
DORIS_CHECK(last->second.end() > candidate.range.offset);
merged_end = std::max(merged_end, last->second.end());
++last;
}
if (merged_end - merged_offset > options.coalesce_options.max_range_bytes) {
continue;
}

components.erase(first, last);
components.emplace(merged_offset,
FileRange {.offset = merged_offset, .size = merged_end - merged_offset});
}

std::vector<FileRange> result;
result.reserve(components.size());
for (const auto& [offset, component] : components) {
DORIS_CHECK(offset == component.offset);
result.push_back(component);
}
return result;
}

// Map every original page range back to the containing physical read with one monotonic scan.
FileRangePlan build_final_plan(const std::vector<FileRange>& input_ranges,
std::vector<FileRange> components) {
FileRangePlan result {.ranges = std::move(components), .input_locations = {}};
result.input_locations.reserve(input_ranges.size());
size_t range_index = 0;
for (const auto& input : input_ranges) {
while (range_index < result.ranges.size() &&
result.ranges[range_index].end() <= input.offset) {
++range_index;
}
DORIS_CHECK(range_index < result.ranges.size());
const auto& range = result.ranges[range_index];
DORIS_CHECK(range.offset <= input.offset);
DORIS_CHECK(input.end() <= range.end());
result.input_locations.push_back(
{.range_index = range_index, .buffer_offset = input.offset - range.offset});
}

return result;
}

} // namespace

Status FileRangePlanner::plan(const std::vector<FileRange>& input_ranges, size_t file_size,
const FileRangePlanOptions& options, FileRangePlan* const output_plan,
ReadAheadStatistics* statistics) {
DORIS_CHECK(output_plan != nullptr);
RETURN_IF_ERROR(options.validate());
RETURN_IF_ERROR(validate_input_ranges(input_ranges, file_size));

const auto base_ranges = FileRangeCoalescer::coalesce(input_ranges, options.coalesce_options);
auto components = choose_components(input_ranges, base_ranges, file_size, options);
*output_plan = build_final_plan(input_ranges, std::move(components));
if (statistics != nullptr) {
size_t input_bytes = 0;
size_t coalesced_bytes = 0;
size_t planned_bytes = 0;
for (const auto& range : input_ranges) {
input_bytes += range.size;
}
for (const auto& range : base_ranges) {
coalesced_bytes += range.size;
}
for (const auto& range : output_plan->ranges) {
planned_bytes += range.size;
}
COUNTER_UPDATE(&statistics->input_pages, static_cast<int64_t>(input_ranges.size()));
COUNTER_UPDATE(&statistics->input_bytes, static_cast<int64_t>(input_bytes));
COUNTER_UPDATE(&statistics->coalesced_ranges, static_cast<int64_t>(base_ranges.size()));
COUNTER_UPDATE(&statistics->coalesced_bytes, static_cast<int64_t>(coalesced_bytes));
COUNTER_UPDATE(&statistics->block_fill_bytes,
static_cast<int64_t>(planned_bytes - coalesced_bytes));
COUNTER_UPDATE(&statistics->planned_ranges,
static_cast<int64_t>(output_plan->ranges.size()));
COUNTER_UPDATE(&statistics->planned_bytes, static_cast<int64_t>(planned_bytes));
}
return Status::OK();
}

} // namespace doris::io
72 changes: 72 additions & 0 deletions be/src/io/cache/file_range_planner.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#pragma once

#include <cstddef>
#include <vector>

#include "common/status.h"
#include "io/fs/file_range_coalescer.h"

namespace doris::io {

struct ReadAheadStatistics;

struct FileRangePlanOptions {
/// Limits for the initial range coalescing pass.
FileRangeCoalesceOptions coalesce_options;
/// File Cache block size used when considering boundary-block completion.
size_t cache_block_size {1};
/// Minimum input-byte coverage required before completing a boundary block.
double block_fill_min_coverage {1.0};

Status validate() const;
};

/// The location of one input range in its final read buffer.
struct FileRangeLocation {
/// Index into FileRangePlan::ranges.
size_t range_index {0};
/// Offset of the original input's first byte within that physical range buffer.
size_t buffer_offset {0};

bool operator==(const FileRangeLocation&) const = default;
};

struct FileRangePlan {
/// Physical ranges to read after coalescing and optional cache-block completion.
std::vector<FileRange> ranges;
/// One location per input in caller-supplied order; every input is contained by that range.
std::vector<FileRangeLocation> input_locations;

bool operator==(const FileRangePlan&) const = default;
};

/// Builds a read-only plan without allocating buffers or performing file IO.
class FileRangePlanner {
public:
/// `input_ranges` must be sorted by offset and mutually disjoint. The planner first coalesces
/// them, then completes high-coverage cache blocks when the resulting physical ranges remain
/// within `max_range_bytes`. One input larger than that limit remains a standalone read. The
/// physical EOF block uses its valid file bytes as the coverage denominator.
static Status plan(const std::vector<FileRange>& input_ranges, size_t file_size,
const FileRangePlanOptions& options, FileRangePlan* output_plan,
ReadAheadStatistics* statistics = nullptr);
};

} // namespace doris::io
Loading