Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7654cf7
[feature](be) Add linear file range coalescing
bobhan1 Aug 28, 2026
90cf5b6
[feature](be) Add cache-aware file range planning
bobhan1 Aug 28, 2026
85894d0
[feature](be) Add file range read scheduler
bobhan1 Aug 28, 2026
c64e719
[feature](be) Reuse remote-only reads for no-write ranges
bobhan1 Aug 28, 2026
563b5ba
[feature](be) Add query read-ahead runtime ownership
bobhan1 Aug 28, 2026
e2a4012
[improvement](be) Add opt-in file range IO tracing
bobhan1 Sep 22, 2026
8a08ae1
[fix](be) Scope filesystem error-code imports to their functions
bobhan1 Oct 15, 2026
4271da9
[feature](be) Add page-driven column read-ahead windows
bobhan1 Aug 28, 2026
e7dbfa1
[feature](be) Add segment range read-ahead coordination
bobhan1 Oct 29, 2026
00fa56b
[feature](be) Connect scalar column readers to range read-ahead
bobhan1 Sep 2, 2026
8190e0f
[feature](be) Initialize scan read-ahead once after pruning
bobhan1 Oct 29, 2026
d03f274
[feature](be) Add complete-block cache write handoff and admission
bobhan1 Aug 28, 2026
a547700
[feature](be) Split consumed ranges for cache writeback
bobhan1 Aug 28, 2026
7cfe76c
[feature](be) Add cache block hole-fill planning
bobhan1 Aug 28, 2026
a7e3899
[feature](be) Add background partial block writeback
bobhan1 Aug 28, 2026
999f5c6
[feature](be) Initialize background hole fill
bobhan1 Aug 28, 2026
6fd7bb6
[feature](be) Route consumed ranges to cache writeback
bobhan1 Aug 28, 2026
1ae1272
[feature](be) Connect read-ahead range writeback
bobhan1 Aug 28, 2026
fd6df2c
[feature](be) Extend range read-ahead to point queries
bobhan1 Sep 2, 2026
a881473
[fix](be) Validate S3 upload buffer alignment on config updates
bobhan1 Oct 29, 2026
b237ebc
[fix](be) Handle irregular cache blocks in writeback coverage checks
bobhan1 Oct 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[improvement](be) Add opt-in file range IO tracing
### What problem does this PR solve?

Issue Number: None

Related PR: #67292, #67579

Problem Summary: Aggregate range-read counters cannot identify repeated object intervals or separate diagnostic overhead from read latency. Synchronize the final shared diagnostics from the validation branch as one infrastructure component: stable IO identities and S3 outcomes, read-only cache/inflight coverage snapshots, task provenance, typed timing fields, a batched standalone JSONL writer and shutdown draining. Remove the offline tools/read_io_trace analyzer and its tests; retain the BE-side trace collection. Keep scanner consumption and hole-fill lifecycle producers in their owning later components. Trace is disabled by default and does not change read planning, concurrency or writeback.

### Release note

Add enable_read_io_trace=false and dedicated diagnostic files controlled by read_io_trace_dir, read_io_trace_flush_bytes=1 MiB and read_io_trace_flush_interval_ms=1000. Enabled diagnostics can affect timing and should be used for isolated captures.

### Check List (For Author)

- Test: Unit Test
    - Independent incremental ASAN/PCH run-be-ut.sh -j100 on master d4f052b: 75 tests from 8 suites passed.
    - Repository clang-format 16, check-format, BE header hygiene and git diff --check passed.
    - The first build was interrupted after /tmp filled; resumed incrementally with a data-disk TMPDIR. No clean rebuild or ccache cleanup.
    - No cluster, external-object-store benchmark or clang-tidy run.
- Behavior changed: Yes, opt-in diagnostic recording; read and persistence decisions are unchanged.
- Does this need documentation: Yes, BE diagnostic settings and the related active PR descriptions.
  • Loading branch information
bobhan1 committed Oct 29, 2026
commit e2a4012ad859589095587dbdace40abf9a5753a4
7 changes: 7 additions & 0 deletions be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1910,6 +1910,13 @@ DEFINE_mBool(enable_query_segment_file_cache_prefetch, "false");
DEFINE_mInt32(query_segment_file_cache_prefetch_block_size, "2");
// Enable exact range read-ahead for query data pages
DEFINE_mBool(enable_query_read_ahead, "false");
DEFINE_mBool(enable_read_io_trace, "false");
// Separate diagnostic files; empty uses the BE log directory's read_io_trace subdirectory.
DEFINE_String(read_io_trace_dir, "");
DEFINE_Int64(read_io_trace_flush_bytes, "1048576"); // 1 MiB
DEFINE_Validator(read_io_trace_flush_bytes, [](int64_t value) { return value > 0; });
DEFINE_Int32(read_io_trace_flush_interval_ms, "1000");
DEFINE_Validator(read_io_trace_flush_interval_ms, [](int32_t value) { return value > 0; });
// Query-level resident read-ahead buffer limit
DEFINE_Int64(read_ahead_max_bytes_per_query, "268435456"); // 256 MiB
DEFINE_Validator(read_ahead_max_bytes_per_query, [](int64_t value) { return value > 0; });
Expand Down
5 changes: 5 additions & 0 deletions be/src/common/config.h
Original file line number Diff line number Diff line change
Expand Up @@ -1894,6 +1894,11 @@ DECLARE_mBool(enable_query_segment_file_cache_prefetch);
DECLARE_mInt32(query_segment_file_cache_prefetch_block_size);
// Enable exact range read-ahead for query data pages
DECLARE_mBool(enable_query_read_ahead);
// Opt-in interval/lifecycle diagnostics, buffered and written to separate JSONL files.
DECLARE_mBool(enable_read_io_trace);
DECLARE_String(read_io_trace_dir);
DECLARE_Int64(read_io_trace_flush_bytes);
DECLARE_Int32(read_io_trace_flush_interval_ms);
// Query-level resident read-ahead buffer limit
DECLARE_Int64(read_ahead_max_bytes_per_query);
// BE-level resident read-ahead buffer limit
Expand Down
29 changes: 29 additions & 0 deletions be/src/io/cache/block_file_cache.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -889,6 +889,35 @@ Status BlockFileCache::get_downloaded_blocks_if_fully_covered(const UInt128Wrapp
return Status::OK();
}

std::vector<FileBlock::Range> BlockFileCache::downloaded_ranges(const UInt128Wrapper& hash,
size_t offset, size_t size) {
DORIS_CHECK(size > 0);
const size_t end = offset + size;
DORIS_CHECK(end > offset);
std::vector<FileBlock::Range> ranges;
std::lock_guard lock(_mutex);
const auto file = _files.find(hash);
if (file == _files.end()) {
return ranges;
}
const auto& blocks = file->second;
auto iterator = blocks.lower_bound(offset);
if (iterator != blocks.begin()) {
--iterator;
}
for (; iterator != blocks.end(); ++iterator) {
const auto& block = iterator->second.file_block;
const auto& range = block->range();
if (range.left >= end) {
break;
}
if (range.right >= offset && block->state() == FileBlock::State::DOWNLOADED) {
ranges.emplace_back(std::max(offset, range.left), std::min(end - 1, range.right));
}
}
return ranges;
}

FileBlocksProbeResult BlockFileCache::probe(const UInt128Wrapper& hash, size_t offset, size_t size,
const CacheContext& context) {
DORIS_CHECK(size > 0);
Expand Down
5 changes: 5 additions & 0 deletions be/src/io/cache/block_file_cache.h
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,11 @@ class BlockFileCache {
size_t size, const CacheContext& context,
FileBlocks* blocks, bool* fully_covered);

/// Diagnostic snapshot of resident DOWNLOADED intervals, clipped to the requested range.
/// Handles irregular block boundaries; never loads metadata, reserves space or touches LRU.
std::vector<FileBlock::Range> downloaded_ranges(const UInt128Wrapper& hash, size_t offset,
size_t size);

/**
* record blocks read directly by CachedRemoteFileReader
*/
Expand Down
48 changes: 48 additions & 0 deletions be/src/io/cache/cached_remote_file_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
#include "io/cache/remote_scan_cache_write_limiter.h"
#include "io/fs/file_reader.h"
#include "io/fs/local_file_system.h"
#include "io/fs/read_io_trace.h"
#include "io/io_common.h"
#include "runtime/exec_env.h"
#include "runtime/runtime_profile.h"
Expand Down Expand Up @@ -1196,6 +1197,50 @@ bool CachedRemoteFileReader::_try_read_from_inflight_buffers(size_t offset, Slic
return true;
}

void CachedRemoteFileReader::_trace_remote_miss(size_t offset, size_t bytes_req,
const IOContext* io_ctx) {
const size_t block_size = static_cast<size_t>(config::file_cache_each_block_size);
const size_t aligned_start = offset / block_size * block_size;
const size_t request_end = offset + bytes_req;
auto available = _cache->downloaded_ranges(_cache_hash, offset, bytes_req);
ReadIOTraceEvent event {.event = "remote_miss_coverage",
.context = io_ctx,
.file = path().native(),
.id = io_ctx->read_trace_id,
.offset = offset,
.size = bytes_req};
for (const auto& range : available) {
event.disk_bytes += range.size();
}
if (config::enable_async_file_cache_write_inflight_write_buffer_index) {
auto* inflight_index = _cache->inflight_write_buffer_index();
for (size_t block_offset = aligned_start; block_offset < request_end;
block_offset += block_size) {
const auto entry = inflight_index->lookup(_cache_hash, block_offset);
if (entry != nullptr) {
const size_t left = std::max(offset, entry->buffer_offset);
const size_t covered_end =
std::min(request_end, entry->buffer_offset + entry->buffer_size);
if (covered_end > left) {
event.inflight_bytes += covered_end - left;
available.emplace_back(left, covered_end - 1);
}
}
}
}
std::sort(available.begin(), available.end(),
[](const auto& left, const auto& right) { return left.left < right.left; });
size_t covered_end = offset;
for (const auto& range : available) {
const size_t end = range.right + 1;
if (end > covered_end) {
event.available_bytes += end - std::max(range.left, covered_end);
covered_end = end;
}
}
ReadIOTrace::record(event);
}

Status CachedRemoteFileReader::_read_remote_only_on_cache_miss(
size_t offset, Slice result, size_t bytes_req, bool is_dryrun, size_t* bytes_read,
ReadStatistics& stats, SourceReadBreakdown& source_read_breakdown,
Expand All @@ -1212,6 +1257,9 @@ Status CachedRemoteFileReader::_read_remote_only_on_cache_miss(
return Status::OK();
}

if (ReadIOTrace::enabled()) {
_trace_remote_miss(offset, bytes_req, io_ctx);
}
size_t remote_bytes_read = bytes_req;
SCOPED_RAW_TIMER(&stats.remote_read_timer);
RETURN_IF_ERROR(_remote_file_reader->read_at(offset, Slice(result.data, bytes_req),
Expand Down
4 changes: 4 additions & 0 deletions be/src/io/cache/cached_remote_file_reader.h
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,10 @@ class CachedRemoteFileReader final : public FileReader,
bool _try_read_from_inflight_buffers(size_t offset, Slice result, size_t bytes_req,
bool is_dryrun, ReadStatistics& stats);

/// Opt-in snapshot of cache coverage immediately before an exact remote read.
/// Probes never populate/touch cache blocks, and their result does not alter the IO decision.
void _trace_remote_miss(size_t offset, size_t size, const IOContext* io_ctx);

/// Read in-flight buffers or downloaded cache blocks only when one source fully covers the
/// request; otherwise read remote data directly without writing file cache.
/// @param[in] offset Original request offset.
Expand Down
17 changes: 17 additions & 0 deletions be/src/io/fs/file_range_read_scheduler.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
#include "core/allocator.h"
#include "cpp/sync_point.h"
#include "io/fs/read_ahead_metrics.h"
#include "io/fs/read_io_trace.h"
#include "runtime/memory/mem_tracker_limiter.h"
#include "runtime/thread_context.h"
#include "util/defer_op.h"
Expand Down Expand Up @@ -164,6 +165,7 @@ FileRangeRead::FileRangeRead(FileRange range, std::shared_ptr<MemTrackerLimiter>
FileRangeReadReservation reservation,
std::shared_ptr<ReadAheadStatistics> statistics)
: _range(range),
_trace_id(ReadIOTrace::next_id()),
_tracker(std::move(tracker)),
_reservation(std::move(reservation)),
_statistics(std::move(statistics)) {
Expand Down Expand Up @@ -613,6 +615,8 @@ void FileRangeReadScheduler::_run_task(ReadTask task) const {
}
task.io_context.io_context.file_cache_stats = &stats.file_cache;
task.io_context.io_context.file_reader_stats = &stats.file_reader;
task.io_context.io_context.read_trace_source = FileReadTraceSource::READ_AHEAD;
task.io_context.io_context.read_trace_id = task.read->trace_id();
Status status;
const int64_t read_start_ns = MonotonicNanos();
try {
Expand All @@ -631,6 +635,19 @@ void FileRangeReadScheduler::_run_task(ReadTask task) const {
stats.successful_source_bytes = stats.bytes_read;
}

if (ReadIOTrace::enabled()) {
ReadIOTrace::record({.event = "range_read_done",
.context = &task.io_context.io_context,
.file = task.reader->path().native(),
.id = task.read->trace_id(),
.offset = task.read->range().offset,
.size = task.read->range().size,
.start_ns = read_start_ns,
.bytes = stats.bytes_read,
.status = status.code(),
.remote_bytes = stats.file_cache.bytes_read_from_remote});
}

if (task.context->cancelled() || task.read->_is_cancel_requested() || !accepting()) {
task.read->_publish_cancelled(std::move(stats));
} else if (!status.ok()) {
Expand Down
2 changes: 2 additions & 0 deletions be/src/io/fs/file_range_read_scheduler.h
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ class FileRangeRead {
void request_cancel();
State state() const;
const FileRange& range() const { return _range; }
uint64_t trace_id() const { return _trace_id; }
/// Return the full buffer. The handle must be READY and owns the returned memory.
Slice data() const;
/// Return a checked subrange of a READY buffer.
Expand All @@ -197,6 +198,7 @@ class FileRangeRead {
void _publish_cancelled(FileRangeReadStats stats = {});
void _publish_from_running(State state, Status status, FileRangeReadStats stats);
const FileRange _range;
const uint64_t _trace_id;
char* _data {nullptr};
const std::shared_ptr<MemTrackerLimiter> _tracker;
FileRangeReadReservation _reservation;
Expand Down
3 changes: 3 additions & 0 deletions be/src/io/fs/read_ahead_metrics.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ void ReadAheadStatistics::update_profile(RuntimeProfile* profile) {
update("ReadAheadConsumedRangeBytes", "ReadAheadConsume", consumed_range_bytes);
update("ReadAheadReleasedPages", "ReadAheadConsume", released_pages);
update("ReadAheadWritebackTime", "ReadAheadWriteback", writeback_time);
update("ReadAheadCompleteBlockSubmitTime", "ReadAheadWritebackTime",
complete_block_submit_time);
update("ReadAheadPartialBlockSubmitTime", "ReadAheadWritebackTime", partial_block_submit_time);
update("ReadAheadCompleteBlocksSubmitted", "ReadAheadWriteback", complete_blocks_submitted);
update("ReadAheadCompleteBlockBytes", "ReadAheadWriteback", complete_block_bytes);
update("ReadAheadPartialBlocksQueued", "ReadAheadWriteback", partial_blocks_queued);
Expand Down
3 changes: 3 additions & 0 deletions be/src/io/fs/read_ahead_metrics.h
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,9 @@ struct ReadAheadStatistics {
RuntimeProfile::Counter consumed_range_bytes {TUnit::BYTES};
RuntimeProfile::Counter released_pages {TUnit::UNIT};
RuntimeProfile::Counter writeback_time {TUnit::TIME_NS};
/// Disjoint foreground dispatch sub-timers; neither includes background hole reads/disk writes.
RuntimeProfile::Counter complete_block_submit_time {TUnit::TIME_NS};
RuntimeProfile::Counter partial_block_submit_time {TUnit::TIME_NS};
RuntimeProfile::Counter complete_blocks_submitted {TUnit::UNIT};
RuntimeProfile::Counter complete_block_bytes {TUnit::BYTES};
RuntimeProfile::Counter partial_blocks_queued {TUnit::UNIT};
Expand Down
Loading