Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7654cf7
[feature](be) Add linear file range coalescing
bobhan1 Aug 28, 2026
90cf5b6
[feature](be) Add cache-aware file range planning
bobhan1 Aug 28, 2026
85894d0
[feature](be) Add file range read scheduler
bobhan1 Aug 28, 2026
c64e719
[feature](be) Reuse remote-only reads for no-write ranges
bobhan1 Aug 28, 2026
563b5ba
[feature](be) Add query read-ahead runtime ownership
bobhan1 Aug 28, 2026
e2a4012
[improvement](be) Add opt-in file range IO tracing
bobhan1 Sep 22, 2026
8a08ae1
[fix](be) Scope filesystem error-code imports to their functions
bobhan1 Oct 15, 2026
4271da9
[feature](be) Add page-driven column read-ahead windows
bobhan1 Aug 28, 2026
e7dbfa1
[feature](be) Add segment range read-ahead coordination
bobhan1 Oct 29, 2026
00fa56b
[feature](be) Connect scalar column readers to range read-ahead
bobhan1 Sep 2, 2026
8190e0f
[feature](be) Initialize scan read-ahead once after pruning
bobhan1 Oct 29, 2026
d03f274
[feature](be) Add complete-block cache write handoff and admission
bobhan1 Aug 28, 2026
a547700
[feature](be) Split consumed ranges for cache writeback
bobhan1 Aug 28, 2026
7cfe76c
[feature](be) Add cache block hole-fill planning
bobhan1 Aug 28, 2026
a7e3899
[feature](be) Add background partial block writeback
bobhan1 Aug 28, 2026
999f5c6
[feature](be) Initialize background hole fill
bobhan1 Aug 28, 2026
6fd7bb6
[feature](be) Route consumed ranges to cache writeback
bobhan1 Aug 28, 2026
1ae1272
[feature](be) Connect read-ahead range writeback
bobhan1 Aug 28, 2026
fd6df2c
[feature](be) Extend range read-ahead to point queries
bobhan1 Sep 2, 2026
a881473
[fix](be) Validate S3 upload buffer alignment on config updates
bobhan1 Oct 29, 2026
b237ebc
[fix](be) Handle irregular cache blocks in writeback coverage checks
bobhan1 Oct 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[fix](be) Validate S3 upload buffer alignment on config updates
### What problem does this PR solve?

Issue Number: None

Related PR: #67292, #67579

Problem Summary: File Cache initialization rejects an S3 upload buffer size that is smaller than a cache block or not a multiple of the block size. The buffer size is mutable, but runtime updates did not register the same validation, so a later update could violate the layout assumed by cached reads and hole-fill deduplication. Register the existing alignment constraint as a config validator and cover accepted updates, rejected updates with value rollback, invalid block sizes, and File Cache disabled behavior. Cache probing, read-ahead, and hole-fill algorithms remain unchanged.

### Release note

Reject S3 upload buffer configuration updates that violate File Cache block alignment while File Cache is enabled.

### Check List (For Author)

- Test: Unit Test — 66 tests from 4 suites pass with ./run-be-ut.sh --run -j100 (ASAN/PCH), including configuration validation and partial-block/async cache writeback. Direct Release/PCH BE build passes with ./build.sh --be -j100. Clang-format 16, build hygiene, and diff checks pass. No cluster or clang-tidy run.
- Behavior changed: Yes — invalid runtime S3 upload buffer updates return INVALID_ARGUMENT and retain the previous value; defaults and IO mechanisms are unchanged.
- Does this need documentation: No — applies the existing File Cache startup constraint to runtime configuration updates.
  • Loading branch information
bobhan1 committed Oct 29, 2026
commit a881473fa156669740d93db678feec0484c3f7d0
7 changes: 7 additions & 0 deletions be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1515,6 +1515,13 @@ DEFINE_mInt32(tablet_path_check_batch_size, "1000");

// it must be larger than or equal to 5MB
DEFINE_mInt64(s3_write_buffer_size, "5242880");
// Startup checks this alignment too. Reject runtime updates such as 5.5 MiB with 1 MiB
// cache blocks, which would let upload writers create non-grid cache blocks.
DEFINE_Validator(s3_write_buffer_size, [](int64_t value) {
return !enable_file_cache ||
(file_cache_each_block_size > 0 && value >= file_cache_each_block_size &&
value % file_cache_each_block_size == 0);
});
// Log interval when doing s3 upload task
DEFINE_mInt32(s3_file_writer_log_interval_second, "60");
DEFINE_mInt64(file_cache_max_file_reader_cache_size, "1000000");
Expand Down
50 changes: 50 additions & 0 deletions be/test/common/config_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,15 @@
#include <gtest/gtest-message.h>
#include <gtest/gtest-test-part.h>

#include <cstdint>
#include <map>
#include <ostream>
#include <string>
#include <utility>

#include "common/status.h"
#include "gtest/gtest_pred_impl.h"
#include "util/defer_op.h"

namespace doris {
using namespace config;
Expand Down Expand Up @@ -149,4 +151,52 @@ TEST_F(ConfigTest, UpdateConfigs) {
EXPECT_EQ(cfg_std_string, "doris_config_test_string");
}

TEST(S3WriteBufferConfigTest, ValidateRuntimeUpdates) {
const bool saved_enable_file_cache = config::enable_file_cache;
const int64_t saved_block_size = config::file_cache_each_block_size;
const int64_t saved_buffer_size = config::s3_write_buffer_size;
auto* saved_config_map = config::full_conf_map;
Defer restore {[&]() {
config::enable_file_cache = saved_enable_file_cache;
config::file_cache_each_block_size = saved_block_size;
config::s3_write_buffer_size = saved_buffer_size;
config::full_conf_map = saved_config_map;
}};
// Exercise the production registry without changing the configuration dump used by other tests.
config::full_conf_map = nullptr;
constexpr int64_t kMiB = 1024 * 1024;
config::enable_file_cache = true;
config::file_cache_each_block_size = kMiB;

ASSERT_TRUE(config::set_config("s3_write_buffer_size", "8388608").ok());
EXPECT_EQ(config::s3_write_buffer_size, 8 * kMiB);

const int64_t invalid_sizes[] = {-1, 0, kMiB / 2, 5 * kMiB + kMiB / 2};
for (const auto size : invalid_sizes) {
const auto status = config::set_config("s3_write_buffer_size", std::to_string(size));
EXPECT_TRUE(status.is<ErrorCode::INVALID_ARGUMENT>()) << status;
EXPECT_EQ(config::s3_write_buffer_size, 8 * kMiB);
}

// Reject upload buffers smaller than one cache block.
config::file_cache_each_block_size = 8 * kMiB;
EXPECT_TRUE(config::set_config("s3_write_buffer_size", "4194304")
.is<ErrorCode::INVALID_ARGUMENT>());
EXPECT_EQ(config::s3_write_buffer_size, 8 * kMiB);
ASSERT_TRUE(config::set_config("s3_write_buffer_size", "16777216").ok());
EXPECT_EQ(config::s3_write_buffer_size, 16 * kMiB);

// Reject an invalid block size before evaluating the modulo operation.
config::file_cache_each_block_size = 0;
EXPECT_TRUE(config::set_config("s3_write_buffer_size", "8388608")
.is<ErrorCode::INVALID_ARGUMENT>());
EXPECT_EQ(config::s3_write_buffer_size, 16 * kMiB);

// Preserve the existing behavior when File Cache is disabled.
config::enable_file_cache = false;
config::file_cache_each_block_size = kMiB;
ASSERT_TRUE(config::set_config("s3_write_buffer_size", "5767168").ok());
EXPECT_EQ(config::s3_write_buffer_size, 5 * kMiB + kMiB / 2);
}

} // namespace doris