Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7654cf7
[feature](be) Add linear file range coalescing
bobhan1 Aug 28, 2026
90cf5b6
[feature](be) Add cache-aware file range planning
bobhan1 Aug 28, 2026
85894d0
[feature](be) Add file range read scheduler
bobhan1 Aug 28, 2026
c64e719
[feature](be) Reuse remote-only reads for no-write ranges
bobhan1 Aug 28, 2026
563b5ba
[feature](be) Add query read-ahead runtime ownership
bobhan1 Aug 28, 2026
e2a4012
[improvement](be) Add opt-in file range IO tracing
bobhan1 Sep 22, 2026
8a08ae1
[fix](be) Scope filesystem error-code imports to their functions
bobhan1 Oct 15, 2026
4271da9
[feature](be) Add page-driven column read-ahead windows
bobhan1 Aug 28, 2026
e7dbfa1
[feature](be) Add segment range read-ahead coordination
bobhan1 Oct 29, 2026
00fa56b
[feature](be) Connect scalar column readers to range read-ahead
bobhan1 Sep 2, 2026
8190e0f
[feature](be) Initialize scan read-ahead once after pruning
bobhan1 Oct 29, 2026
d03f274
[feature](be) Add complete-block cache write handoff and admission
bobhan1 Aug 28, 2026
a547700
[feature](be) Split consumed ranges for cache writeback
bobhan1 Aug 28, 2026
7cfe76c
[feature](be) Add cache block hole-fill planning
bobhan1 Aug 28, 2026
a7e3899
[feature](be) Add background partial block writeback
bobhan1 Aug 28, 2026
999f5c6
[feature](be) Initialize background hole fill
bobhan1 Aug 28, 2026
6fd7bb6
[feature](be) Route consumed ranges to cache writeback
bobhan1 Aug 28, 2026
1ae1272
[feature](be) Connect read-ahead range writeback
bobhan1 Aug 28, 2026
fd6df2c
[feature](be) Extend range read-ahead to point queries
bobhan1 Sep 2, 2026
a881473
[fix](be) Validate S3 upload buffer alignment on config updates
bobhan1 Oct 29, 2026
b237ebc
[fix](be) Handle irregular cache blocks in writeback coverage checks
bobhan1 Oct 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[feature](be) Extend range read-ahead to point queries
### What problem does this PR solve?

Issue Number: None

Related PR: #67579, #67292

Problem Summary: Short-circuit point queries read row-store data or missing column-store values through exact row IDs. Create one request-scoped range-read context, initialize the scalar row-store or missing top-level scalar iterators against the shared read-ahead reader, and submit the required pages before decoding. Complex columns and extracted variant paths retain Segment's original iterator/reader resolution. Preserve row-cache behavior, result padding, request-scoped constants and synchronous fallback. Append read-ahead Profile statistics to point-query diagnostics; BE cumulative counters retain unsampled totals. Include exact-row helpers and tests in this component. TopN second-stage integration remains identical to the master base and is outside this PR series.

Handle a dynamic switch closing after a point request creates its context: if create_for_query returns no read-ahead object, read row-store data synchronously through the already loaded segment. The deterministic row-store test closes either enable switch after segment resolution, verifies that no segment-loader call is made, and compares the result with the original reader.

Rebase compatibility: reuse the segment returned by primary-key lookup in both the read-ahead and synchronous row-store paths, preserving master's segment and acquired-rowset lifetime.

### Release note

Use query range read-ahead for scalar point-query data pages when enable_query_read_ahead and enable_async_file_cache_write are enabled.

### Check List (For Author)

- Test: Unit Test — pre-rebase integrated ASAN/PCH validation covered 225 tests from 10 suites through ./run-be-ut.sh --run -j100. Includes partial budget admission, original page offsets, cancellation, allocation rollback, executor rejection, consumed-range callbacks and row-store switch fallback. Individual intermediate commits are not built separately; target-branch validation is recorded on the PR. Clang-format 16, header hygiene and diff checks pass. No cluster or clang-tidy run.
- Behavior changed: Yes — adds scalar point-query prefetch without changing complex-column decoding, scheduling limits or query results.
- Does this need documentation: Yes — scope and synchronization are recorded in the related active PR descriptions.
  • Loading branch information
bobhan1 committed Oct 29, 2026
commit fd6df2c335c41f62e2447765bc9bb3b488a3334f
147 changes: 112 additions & 35 deletions be/src/service/point_query_executor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
#include <stdlib.h>

#include <memory>
#include <sstream>
#include <unordered_map>
#include <vector>

Expand All @@ -43,6 +44,7 @@
#include "exprs/vexpr_fwd.h"
#include "exprs/vslot_ref.h"
#include "io/cache/remote_scan_cache_write_limiter.h"
#include "io/fs/read_ahead_metrics.h"
#include "io/io_common.h"
#include "runtime/descriptors.h"
#include "runtime/exec_env.h"
Expand All @@ -55,6 +57,8 @@
#include "storage/rowset/rowset_fwd.h"
#include "storage/schema.h"
#include "storage/segment/column_reader.h"
#include "storage/segment/rowid_read_ahead.h"
#include "storage/segment/segment_read_ahead.h"
#include "storage/tablet/tablet_schema.h"
#include "storage/utils.h"
#include "util/defer_op.h"
Expand Down Expand Up @@ -417,13 +421,29 @@ void PointQueryExecutor::print_profile() {
_profile_metrics.read_stats.file_cache_stats.remote_io_timer,
_profile_metrics.read_stats.file_cache_stats.write_cache_io_timer);

const auto read_ahead_profile = [&]() {
if (_profile_metrics.read_stats.read_ahead_stats == nullptr &&
_read_stats.read_ahead_stats == nullptr) {
return std::string {};
}
RuntimeProfile profile("PageReadAhead");
for (auto* stats : {&_profile_metrics.read_stats, &_read_stats}) {
if (stats->read_ahead_stats != nullptr) {
stats->read_ahead_stats->update_profile(&profile);
}
}
std::ostringstream output;
profile.pretty_print(&output);
return "\n" + output.str();
};

constexpr static int kSlowThreholdUs = 50 * 1000; // 50ms
if (total_us > kSlowThreholdUs) {
LOG(WARNING) << "slow query, " << stats_str;
LOG(WARNING) << "slow query, " << stats_str << read_ahead_profile();
} else if (VLOG_DEBUG_IS_ON) {
VLOG_DEBUG << stats_str;
VLOG_DEBUG << stats_str << read_ahead_profile();
} else {
LOG_EVERY_N(INFO, 1000) << stats_str;
LOG_EVERY_N(INFO, 1000) << stats_str << read_ahead_profile();
}
}

Expand Down Expand Up @@ -528,6 +548,12 @@ Status PointQueryExecutor::_lookup_row_key() {
Status PointQueryExecutor::_lookup_row_data() {
// 3. get values
SCOPED_TIMER(&_profile_metrics.lookup_data_ns);
std::shared_ptr<io::FileRangeReadContext> read_ahead_context;
if (segment_v2::SegmentReadAhead::enabled()) {
auto* scheduler = ExecEnv::GetInstance()->file_range_read_scheduler();
DORIS_CHECK(scheduler != nullptr);
read_ahead_context = scheduler->create_context();
}
{
auto result_columns_guard = _result_block->mutate_columns_scoped();
MutableColumns& result_columns = result_columns_guard.mutable_columns();
Expand All @@ -552,10 +578,18 @@ Status PointQueryExecutor::_lookup_row_data() {
io_ctx.reader_type = ReaderType::READER_QUERY;
io_ctx.file_cache_stats = &_profile_metrics.read_stats.file_cache_stats;
io_ctx.remote_scan_cache_write_limiter = _remote_scan_cache_write_limiter.get();
RETURN_IF_ERROR(_tablet->lookup_row_data(
_row_read_ctxs[i]._primary_key, _row_read_ctxs[i]._row_location.value(),
_row_read_ctxs[i]._segment, *(_row_read_ctxs[i]._rowset_ptr),
_profile_metrics.read_stats, value, use_row_cache, &io_ctx));
if (read_ahead_context != nullptr) {
RETURN_IF_ERROR(_tablet->lookup_row_data_with_read_ahead(
_row_read_ctxs[i]._primary_key, _row_read_ctxs[i]._row_location.value(),
_row_read_ctxs[i]._segment, *(_row_read_ctxs[i]._rowset_ptr),
_profile_metrics.read_stats, value, use_row_cache, &io_ctx,
read_ahead_context));
} else {
RETURN_IF_ERROR(_tablet->lookup_row_data(
_row_read_ctxs[i]._primary_key, _row_read_ctxs[i]._row_location.value(),
_row_read_ctxs[i]._segment, *(_row_read_ctxs[i]._rowset_ptr),
_profile_metrics.read_stats, value, use_row_cache, &io_ctx));
}
// serialize value to block, currently only jsonb row format
RETURN_IF_ERROR(JsonbSerializeUtil::jsonb_to_columns(
_reusable->get_data_type_serdes(), value.data(), value.size(),
Expand Down Expand Up @@ -598,37 +632,80 @@ Status PointQueryExecutor::_lookup_row_data() {
return seg->id() == row_loc.segment_id;
});
const auto& segment = *it;
const auto tablet_schema = _tablet->tablet_schema();
for (int cid : _reusable->missing_col_uids()) {
int pos = _reusable->get_col_uid_to_idx().at(cid);
StorageReadOptions read_ahead_options(_read_stats);
read_ahead_options.use_page_cache = !config::disable_storage_page_cache;
read_ahead_options.io_ctx = io_ctx;
// Pre-read iterators need the same rowset constants as the synchronous path:
// a version column in rowset [7-7] must produce 7, not its stored placeholder 0.
read_ahead_options.version = rowset->version();
read_ahead_options.commit_tso = rowset->commit_tso();
std::unique_ptr<segment_v2::SegmentReadAhead> read_ahead;
if (read_ahead_context != nullptr) {
const auto status = segment_v2::SegmentReadAhead::create_for_query(
segment->file_reader(), ExecEnv::GetInstance(), read_ahead_context,
read_ahead_options, &read_ahead);
if (!status.ok()) {
LOG_EVERY_N(WARNING, 100)
<< "failed to initialize point-read column read-ahead; use the "
"original read path: "
<< status;
}
}
if (read_ahead != nullptr) {
const auto missing_column_count = _reusable->missing_col_uids().size();
std::vector<StorageReadOptions> read_options(missing_column_count,
read_ahead_options);
std::vector<std::unique_ptr<ColumnIterator>> iterators(missing_column_count);
std::vector<segment_v2::RowIdColumnRead> column_reads;
column_reads.reserve(missing_column_count);
size_t read_index = 0;
for (int cid : _reusable->missing_col_uids()) {
int pos = _reusable->get_col_uid_to_idx().at(cid);
SlotDescriptor* slot = _reusable->tuple_desc()->slots()[pos];
column_reads.push_back({.slot = slot,
.result = &result_columns[pos],
.read_options = &read_options[read_index],
.iterator = &iterators[read_index]});
++read_index;
}
std::vector<segment_v2::rowid_t> row_ids {
static_cast<segment_v2::rowid_t>(row_loc.row_id)};
auto& column = result_columns[pos];
std::unique_ptr<ColumnIterator> iter;
SlotDescriptor* slot = _reusable->tuple_desc()->slots()[pos];
int32_t index = slot->col_unique_id() >= 0
? tablet_schema->field_index(slot->col_unique_id())
: tablet_schema->field_index(slot->col_name());
// A light schema change adds a column without writing anything, and
// BaseTablet::_max_version_schema only moves forward on write, so until the
// next load the tablet schema here does not have it. No segment holds it
// either, so describe it from the slot and let the segment answer with its
// default -- the same answer it gives once a load does bring the column in.
TabletColumn column_awaiting_a_load;
if (index < 0) {
column_awaiting_a_load = ReadSchema::tablet_column_from_slot(*slot);
RETURN_IF_ERROR(segment_v2::read_columns_by_rowids_with_read_ahead(
*segment, *_tablet->tablet_schema(), row_ids, column_reads,
*read_ahead));
} else {
const auto tablet_schema = _tablet->tablet_schema();
for (int cid : _reusable->missing_col_uids()) {
int pos = _reusable->get_col_uid_to_idx().at(cid);
std::vector<segment_v2::rowid_t> row_ids {
static_cast<segment_v2::rowid_t>(row_loc.row_id)};
auto& column = result_columns[pos];
std::unique_ptr<ColumnIterator> iter;
SlotDescriptor* slot = _reusable->tuple_desc()->slots()[pos];
int32_t index = slot->col_unique_id() >= 0
? tablet_schema->field_index(slot->col_unique_id())
: tablet_schema->field_index(slot->col_name());
// A light schema change adds a column without writing anything, and
// BaseTablet::_max_version_schema only moves forward on write, so until the
// next load the tablet schema here does not have it. No segment holds it
// either, so describe it from the slot and let the segment answer with its
// default -- the same answer it gives once a load does bring the column in.
TabletColumn column_awaiting_a_load;
if (index < 0) {
column_awaiting_a_load = ReadSchema::tablet_column_from_slot(*slot);
}
const TabletColumn& read_column =
index >= 0 ? tablet_schema->column(index) : column_awaiting_a_load;
StorageReadOptions storage_read_options(_read_stats);
storage_read_options.io_ctx = io_ctx;
// A point lookup bypasses TabletReader, so supply the rowset context here. For
// example, reading __DORIS_VERSION_COL__ from rowset [7-7] must return 7
// instead of its on-disk placeholder 0.
storage_read_options.version = rowset->version();
storage_read_options.commit_tso = rowset->commit_tso();
RETURN_IF_ERROR(segment->seek_and_read_by_rowid(
read_column, slot, row_ids, column, storage_read_options, iter));
}
const TabletColumn& read_column =
index >= 0 ? tablet_schema->column(index) : column_awaiting_a_load;
StorageReadOptions storage_read_options(_read_stats);
storage_read_options.io_ctx = io_ctx;
// A point lookup bypasses TabletReader, so supply the rowset context here. For
// example, reading __DORIS_VERSION_COL__ from rowset [7-7] must return 7
// instead of its on-disk placeholder 0.
storage_read_options.version = rowset->version();
storage_read_options.commit_tso = rowset->commit_tso();
RETURN_IF_ERROR(segment->seek_and_read_by_rowid(
read_column, slot, row_ids, column, storage_read_options, iter));
}
}
}
Expand Down
158 changes: 158 additions & 0 deletions be/src/storage/segment/rowid_read_ahead.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#include "storage/segment/rowid_read_ahead.h"

#include <algorithm>
#include <array>

#include "common/config.h"
#include "runtime/descriptors.h"
#include "storage/schema.h"
#include "storage/segment/column_reader.h"
#include "storage/segment/segment.h"
#include "storage/segment/segment_read_ahead.h"
#include "storage/tablet/tablet_schema.h"
#include "storage/types.h"

namespace doris::segment_v2 {
namespace {

Status prepare_column_iterator(Segment& segment, const TabletColumn& column,
StorageReadOptions& read_options,
std::unique_ptr<ColumnIterator>& iterator,
SegmentReadAhead& read_ahead) {
DORIS_CHECK(read_options.stats != nullptr);
if (iterator == nullptr) {
RETURN_IF_ERROR(segment.new_column_iterator(column, &iterator, &read_options));
auto io_ctx = read_options.io_ctx;
io_ctx.reader_type = ReaderType::READER_QUERY;
io_ctx.file_cache_stats = &read_options.stats->file_cache_stats;
ColumnIteratorOptions options {
.use_page_cache = !config::disable_storage_page_cache,
.file_reader = read_ahead.file_reader().get(),
.stats = read_options.stats,
.io_ctx = io_ctx,
};
RETURN_IF_ERROR(iterator->init(options));
}
return Status::OK();
}

Status prepare_slot_iterator(Segment& segment, const TabletSchema& schema, RowIdColumnRead& column,
SegmentReadAhead& read_ahead, ColumnIterator** prepared) {
DORIS_CHECK(column.slot != nullptr);
DORIS_CHECK(column.read_options != nullptr);
DORIS_CHECK(column.iterator != nullptr);
DORIS_CHECK(prepared != nullptr);
*prepared = nullptr;

// Extracted variant paths use Segment's existing path resolution and casting logic.
if (!column.slot->column_paths().empty()) {
return Status::OK();
}
const int index = column.slot->col_unique_id() >= 0
? schema.field_index(column.slot->col_unique_id())
: schema.field_index(column.slot->col_name());
if (index < 0) {
return Status::OK();
}
const auto& tablet_column = schema.column(index);
// Nested columns retain the original iterator and reader; their children can be read in
// different predicate/lazy phases, with independent positions.
if (!is_scalar_type(tablet_column.type())) {
return Status::OK();
}

RETURN_IF_ERROR(prepare_column_iterator(segment, tablet_column, *column.read_options,
*column.iterator, read_ahead));
*prepared = column.iterator->get();
return Status::OK();
}

} // namespace

Status prepare_columns_by_rowids_with_read_ahead(Segment& segment, const TabletSchema& schema,
const std::vector<uint32_t>& row_ids,
std::span<RowIdColumnRead> columns,
SegmentReadAhead& read_ahead) {
if (row_ids.empty()) {
return Status::OK();
}

std::vector<ColumnIterator*> prepared;
prepared.reserve(columns.size());
for (auto& column : columns) {
ColumnIterator* iterator = nullptr;
RETURN_IF_ERROR(prepare_slot_iterator(segment, schema, column, read_ahead, &iterator));
if (iterator != nullptr) {
prepared.push_back(iterator);
}
}
if (!prepared.empty()) {
static_cast<void>(read_ahead.prefetch_by_rowids(row_ids.data(), row_ids.size(), prepared));
}

return Status::OK();
}

Status read_columns_by_rowids_with_read_ahead(Segment& segment, const TabletSchema& schema,
const std::vector<uint32_t>& row_ids,
std::span<RowIdColumnRead> columns,
SegmentReadAhead& read_ahead) {
RETURN_IF_ERROR(prepare_columns_by_rowids_with_read_ahead(segment, schema, row_ids, columns,
read_ahead));

for (auto& column : columns) {
DORIS_CHECK(column.result != nullptr);
const int index = column.slot->col_unique_id() >= 0
? schema.field_index(column.slot->col_unique_id())
: schema.field_index(column.slot->col_name());
// A newly added column can be absent from the tablet schema until the next load.
// Preserve the point-query path's slot-derived default in that interval.
TabletColumn column_awaiting_a_load;
if (index < 0) {
column_awaiting_a_load = ReadSchema::tablet_column_from_slot(*column.slot);
}
const TabletColumn& read_column =
index >= 0 ? schema.column(index) : column_awaiting_a_load;
RETURN_IF_ERROR(segment.seek_and_read_by_rowid(read_column, column.slot, row_ids,
*column.result, *column.read_options,
*column.iterator));
}
return Status::OK();
}

Status read_column_by_rowids_with_read_ahead(Segment& segment, const TabletColumn& column,
const std::vector<uint32_t>& row_ids,
MutableColumnPtr& result,
StorageReadOptions& read_options,
std::unique_ptr<ColumnIterator>& iterator,
SegmentReadAhead& read_ahead) {
if (row_ids.empty()) {
return Status::OK();
}
DORIS_CHECK(std::is_sorted(row_ids.begin(), row_ids.end()));
DORIS_CHECK(std::adjacent_find(row_ids.begin(), row_ids.end()) == row_ids.end());
DORIS_CHECK(is_scalar_type(column.type()));
RETURN_IF_ERROR(prepare_column_iterator(segment, column, read_options, iterator, read_ahead));
std::array<ColumnIterator*, 1> columns {iterator.get()};
static_cast<void>(read_ahead.prefetch_by_rowids(row_ids.data(), row_ids.size(), columns));
return iterator->read_by_rowids(row_ids.data(), row_ids.size(), result);
}

} // namespace doris::segment_v2
Loading