Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[improvement](be) Add opt-in file range IO tracing
### What problem does this PR solve?

Issue Number: None

Related PR: #67611, #67612, #67292, #67579

Problem Summary: Aggregate range-read counters cannot identify repeated object intervals or separate diagnostic overhead from read latency. Synchronize the final shared diagnostics from the validation branch as one infrastructure component: stable IO identities and S3 outcomes, read-only cache/inflight coverage snapshots, task provenance, typed timing fields, a batched standalone JSONL writer and shutdown draining, and the offline overlap/timing analyzer. Keep scanner consumption and hole-fill lifecycle producers in their owning later components. Trace is disabled by default and does not change read planning, concurrency or writeback.

### Release note

Add enable_read_io_trace=false and dedicated diagnostic files controlled by read_io_trace_dir, read_io_trace_flush_bytes=1 MiB and read_io_trace_flush_interval_ms=1000. Enabled diagnostics can affect timing and should be used for isolated captures.

### Check List (For Author)

- Test: Unit Test
    - Independent incremental ASAN/PCH run-be-ut.sh -j100 on master d4f052b: 75 tests from 8 suites passed.
    - Python offline analyzer: 15 tests passed.
    - Repository clang-format 16, check-format, BE header hygiene and git diff --check passed.
    - The first build was interrupted after /tmp filled; resumed incrementally with a data-disk TMPDIR. No clean rebuild or ccache cleanup.
    - No cluster, external-object-store benchmark or clang-tidy run.
- Behavior changed: Yes, opt-in diagnostic recording; read and persistence decisions are unchanged.
- Does this need documentation: Yes, tools/read_io_trace/README.md and the related PR/tracking descriptions.
  • Loading branch information
bobhan1 committed Oct 14, 2026
commit 5ea5103b2de13597f7b7233a0eb646a6ebfb06fb
7 changes: 7 additions & 0 deletions be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1885,6 +1885,13 @@ DEFINE_mBool(enable_query_segment_file_cache_prefetch, "false");
DEFINE_mInt32(query_segment_file_cache_prefetch_block_size, "2");
// Enable exact range read-ahead for query data pages
DEFINE_mBool(enable_query_read_ahead, "false");
DEFINE_mBool(enable_read_io_trace, "false");
// Separate diagnostic files; empty uses the BE log directory's read_io_trace subdirectory.
DEFINE_String(read_io_trace_dir, "");
DEFINE_Int64(read_io_trace_flush_bytes, "1048576"); // 1 MiB
DEFINE_Validator(read_io_trace_flush_bytes, [](int64_t value) { return value > 0; });
DEFINE_Int32(read_io_trace_flush_interval_ms, "1000");
DEFINE_Validator(read_io_trace_flush_interval_ms, [](int32_t value) { return value > 0; });
// Query-level resident read-ahead buffer limit
DEFINE_Int64(read_ahead_max_bytes_per_query, "268435456"); // 256 MiB
DEFINE_Validator(read_ahead_max_bytes_per_query, [](int64_t value) { return value > 0; });
Expand Down
5 changes: 5 additions & 0 deletions be/src/common/config.h
Original file line number Diff line number Diff line change
Expand Up @@ -1890,6 +1890,11 @@ DECLARE_mBool(enable_query_segment_file_cache_prefetch);
DECLARE_mInt32(query_segment_file_cache_prefetch_block_size);
// Enable exact range read-ahead for query data pages
DECLARE_mBool(enable_query_read_ahead);
// Opt-in interval/lifecycle diagnostics, buffered and written to separate JSONL files.
DECLARE_mBool(enable_read_io_trace);
DECLARE_String(read_io_trace_dir);
DECLARE_Int64(read_io_trace_flush_bytes);
DECLARE_Int32(read_io_trace_flush_interval_ms);
// Query-level resident read-ahead buffer limit
DECLARE_Int64(read_ahead_max_bytes_per_query);
// BE-level resident read-ahead buffer limit
Expand Down
29 changes: 29 additions & 0 deletions be/src/io/cache/block_file_cache.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -882,6 +882,35 @@ Status BlockFileCache::get_downloaded_blocks_if_fully_covered(const UInt128Wrapp
return Status::OK();
}

std::vector<FileBlock::Range> BlockFileCache::downloaded_ranges(const UInt128Wrapper& hash,
size_t offset, size_t size) {
DORIS_CHECK(size > 0);
const size_t end = offset + size;
DORIS_CHECK(end > offset);
std::vector<FileBlock::Range> ranges;
std::lock_guard lock(_mutex);
const auto file = _files.find(hash);
if (file == _files.end()) {
return ranges;
}
const auto& blocks = file->second;
auto iterator = blocks.lower_bound(offset);
if (iterator != blocks.begin()) {
--iterator;
}
for (; iterator != blocks.end(); ++iterator) {
const auto& block = iterator->second.file_block;
const auto& range = block->range();
if (range.left >= end) {
break;
}
if (range.right >= offset && block->state() == FileBlock::State::DOWNLOADED) {
ranges.emplace_back(std::max(offset, range.left), std::min(end - 1, range.right));
}
}
return ranges;
}

FileBlocksProbeResult BlockFileCache::probe(const UInt128Wrapper& hash, size_t offset, size_t size,
const CacheContext& context) {
DORIS_CHECK(size > 0);
Expand Down
5 changes: 5 additions & 0 deletions be/src/io/cache/block_file_cache.h
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,11 @@ class BlockFileCache {
size_t size, const CacheContext& context,
FileBlocks* blocks, bool* fully_covered);

/// Diagnostic snapshot of resident DOWNLOADED intervals, clipped to the requested range.
/// Handles irregular block boundaries; never loads metadata, reserves space or touches LRU.
std::vector<FileBlock::Range> downloaded_ranges(const UInt128Wrapper& hash, size_t offset,
size_t size);

/**
* record blocks read directly by CachedRemoteFileReader
*/
Expand Down
48 changes: 48 additions & 0 deletions be/src/io/cache/cached_remote_file_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
#include "io/cache/remote_scan_cache_write_limiter.h"
#include "io/fs/file_reader.h"
#include "io/fs/local_file_system.h"
#include "io/fs/read_io_trace.h"
#include "io/io_common.h"
#include "runtime/exec_env.h"
#include "runtime/runtime_profile.h"
Expand Down Expand Up @@ -1175,6 +1176,50 @@ bool CachedRemoteFileReader::_try_read_from_inflight_buffers(size_t offset, Slic
return true;
}

void CachedRemoteFileReader::_trace_remote_miss(size_t offset, size_t bytes_req,
const IOContext* io_ctx) {
const size_t block_size = static_cast<size_t>(config::file_cache_each_block_size);
const size_t aligned_start = offset / block_size * block_size;
const size_t request_end = offset + bytes_req;
auto available = _cache->downloaded_ranges(_cache_hash, offset, bytes_req);
ReadIOTraceEvent event {.event = "remote_miss_coverage",
.context = io_ctx,
.file = path().native(),
.id = io_ctx->read_trace_id,
.offset = offset,
.size = bytes_req};
for (const auto& range : available) {
event.disk_bytes += range.size();
}
if (config::enable_async_file_cache_write_inflight_write_buffer_index) {
auto* inflight_index = _cache->inflight_write_buffer_index();
for (size_t block_offset = aligned_start; block_offset < request_end;
block_offset += block_size) {
const auto entry = inflight_index->lookup(_cache_hash, block_offset);
if (entry != nullptr) {
const size_t left = std::max(offset, entry->buffer_offset);
const size_t covered_end =
std::min(request_end, entry->buffer_offset + entry->buffer_size);
if (covered_end > left) {
event.inflight_bytes += covered_end - left;
available.emplace_back(left, covered_end - 1);
}
}
}
}
std::sort(available.begin(), available.end(),
[](const auto& left, const auto& right) { return left.left < right.left; });
size_t covered_end = offset;
for (const auto& range : available) {
const size_t end = range.right + 1;
if (end > covered_end) {
event.available_bytes += end - std::max(range.left, covered_end);
covered_end = end;
}
}
ReadIOTrace::record(event);
}

Status CachedRemoteFileReader::_read_remote_only_on_cache_miss(
size_t offset, Slice result, size_t bytes_req, bool is_dryrun, size_t* bytes_read,
ReadStatistics& stats, SourceReadBreakdown& source_read_breakdown,
Expand All @@ -1191,6 +1236,9 @@ Status CachedRemoteFileReader::_read_remote_only_on_cache_miss(
return Status::OK();
}

if (ReadIOTrace::enabled()) {
_trace_remote_miss(offset, bytes_req, io_ctx);
}
size_t remote_bytes_read = bytes_req;
SCOPED_RAW_TIMER(&stats.remote_read_timer);
RETURN_IF_ERROR(_remote_file_reader->read_at(offset, Slice(result.data, bytes_req),
Expand Down
4 changes: 4 additions & 0 deletions be/src/io/cache/cached_remote_file_reader.h
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,10 @@ class CachedRemoteFileReader final : public FileReader,
bool _try_read_from_inflight_buffers(size_t offset, Slice result, size_t bytes_req,
bool is_dryrun, ReadStatistics& stats);

/// Opt-in snapshot of cache coverage immediately before an exact remote read.
/// Probes never populate/touch cache blocks, and their result does not alter the IO decision.
void _trace_remote_miss(size_t offset, size_t size, const IOContext* io_ctx);

/// Read in-flight buffers or downloaded cache blocks only when one source fully covers the
/// request; otherwise read remote data directly without writing file cache.
/// @param[in] offset Original request offset.
Expand Down
17 changes: 17 additions & 0 deletions be/src/io/fs/file_range_read_scheduler.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
#include "core/allocator.h"
#include "cpp/sync_point.h"
#include "io/fs/read_ahead_metrics.h"
#include "io/fs/read_io_trace.h"
#include "runtime/memory/mem_tracker_limiter.h"
#include "runtime/thread_context.h"
#include "util/defer_op.h"
Expand Down Expand Up @@ -159,6 +160,7 @@ FileRangeRead::FileRangeRead(FileRange range, std::shared_ptr<MemTrackerLimiter>
FileRangeReadReservation reservation,
std::shared_ptr<ReadAheadStatistics> statistics)
: _range(range),
_trace_id(ReadIOTrace::next_id()),
_tracker(std::move(tracker)),
_reservation(std::move(reservation)),
_statistics(std::move(statistics)) {
Expand Down Expand Up @@ -590,6 +592,8 @@ void FileRangeReadScheduler::_run_task(ReadTask task) const {
}
task.io_context.io_context.file_cache_stats = &stats.file_cache;
task.io_context.io_context.file_reader_stats = &stats.file_reader;
task.io_context.io_context.read_trace_source = FileReadTraceSource::READ_AHEAD;
task.io_context.io_context.read_trace_id = task.read->trace_id();
Status status;
const int64_t read_start_ns = MonotonicNanos();
try {
Expand All @@ -608,6 +612,19 @@ void FileRangeReadScheduler::_run_task(ReadTask task) const {
stats.successful_source_bytes = stats.bytes_read;
}

if (ReadIOTrace::enabled()) {
ReadIOTrace::record({.event = "range_read_done",
.context = &task.io_context.io_context,
.file = task.reader->path().native(),
.id = task.read->trace_id(),
.offset = task.read->range().offset,
.size = task.read->range().size,
.start_ns = read_start_ns,
.bytes = stats.bytes_read,
.status = status.code(),
.remote_bytes = stats.file_cache.bytes_read_from_remote});
}

if (task.context->cancelled() || task.read->_is_cancel_requested() || !accepting()) {
task.read->_publish_cancelled(std::move(stats));
} else if (!status.ok()) {
Expand Down
2 changes: 2 additions & 0 deletions be/src/io/fs/file_range_read_scheduler.h
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ class FileRangeRead {
void request_cancel();
State state() const;
const FileRange& range() const { return _range; }
uint64_t trace_id() const { return _trace_id; }
/// Return the full buffer. The handle must be READY and owns the returned memory.
Slice data() const;
/// Return a checked subrange of a READY buffer.
Expand All @@ -196,6 +197,7 @@ class FileRangeRead {
void _publish_cancelled(FileRangeReadStats stats = {});
void _publish_from_running(State state, Status status, FileRangeReadStats stats);
const FileRange _range;
const uint64_t _trace_id;
char* _data {nullptr};
const std::shared_ptr<MemTrackerLimiter> _tracker;
FileRangeReadReservation _reservation;
Expand Down
3 changes: 3 additions & 0 deletions be/src/io/fs/read_ahead_metrics.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ void ReadAheadStatistics::update_profile(RuntimeProfile* profile) {
update("ReadAheadConsumedRangeBytes", "ReadAheadConsume", consumed_range_bytes);
update("ReadAheadReleasedPages", "ReadAheadConsume", released_pages);
update("ReadAheadWritebackTime", "ReadAheadWriteback", writeback_time);
update("ReadAheadCompleteBlockSubmitTime", "ReadAheadWritebackTime",
complete_block_submit_time);
update("ReadAheadPartialBlockSubmitTime", "ReadAheadWritebackTime", partial_block_submit_time);
update("ReadAheadCompleteBlocksSubmitted", "ReadAheadWriteback", complete_blocks_submitted);
update("ReadAheadCompleteBlockBytes", "ReadAheadWriteback", complete_block_bytes);
update("ReadAheadPartialBlocksQueued", "ReadAheadWriteback", partial_blocks_queued);
Expand Down
3 changes: 3 additions & 0 deletions be/src/io/fs/read_ahead_metrics.h
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,9 @@ struct ReadAheadStatistics {
RuntimeProfile::Counter consumed_range_bytes {TUnit::BYTES};
RuntimeProfile::Counter released_pages {TUnit::UNIT};
RuntimeProfile::Counter writeback_time {TUnit::TIME_NS};
/// Disjoint foreground dispatch sub-timers; neither includes background hole reads/disk writes.
RuntimeProfile::Counter complete_block_submit_time {TUnit::TIME_NS};
RuntimeProfile::Counter partial_block_submit_time {TUnit::TIME_NS};
RuntimeProfile::Counter complete_blocks_submitted {TUnit::UNIT};
RuntimeProfile::Counter complete_block_bytes {TUnit::BYTES};
RuntimeProfile::Counter partial_blocks_queued {TUnit::UNIT};
Expand Down
Loading
Loading