Skip to content

Commit 96f3b7e

Browse files
chenjunweichenjunwei
authored andcommitted
[fix](be) Align object storage encoding, logging and benchmark settings
### What problem does this PR solve? Related PR: apache#69605, apache#69082 Problem Summary: The integration branch differed from the reviewed child PR in three places. The file cache benchmark reader passed an unrecognized connection timeout key. The vault sync warning used the status of fetching the vault list after a create or refresh failed. Requests sent before the first reader or writer could use a different path encoding. Pass the recognized benchmark key, log the failing operation status and enable RFC 3986 encoding through the AWS SDK initialization options. Restore the SDK initialization regression test. ### Release note The BE uses strict RFC 3986 path encoding from AWS SDK startup and reports the failing operation in vault sync warnings. The file cache benchmark applies its configured S3 connection timeout to reads. ### Check List (For Author) - Test: Compared all three code paths and the encoding regression test with apache#69605 at b6ac43e. The six changed C++ files passed clang-format 16, the changed-line clang-tidy check and BE build hygiene. - Behavior changed: Yes, as described in the release note. - Does this need documentation: No
1 parent 7435a28 commit 96f3b7e

6 files changed

Lines changed: 14 additions & 7 deletions

File tree

‎be/src/cloud/cloud_storage_engine.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -500,7 +500,7 @@ void CloudStorageEngine::sync_storage_vault() {
500500
: std::visit(RefreshFSVaultVisitor {id, std::move(fs), path_format},
501501
vault_info);
502502
if (!status.ok()) [[unlikely]] {
503-
LOG(WARNING) << vault_process_error(id, vault_info, std::move(st));
503+
LOG(WARNING) << vault_process_error(id, vault_info, std::move(status));
504504
}
505505
}
506506

‎be/src/io/fs/obj_storage_file_reader.cpp‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717

1818
#include "io/fs/obj_storage_file_reader.h"
1919

20-
#include <aws/core/http/URI.h>
2120
#include <bvar/latency_recorder.h>
2221
#include <bvar/reducer.h>
2322
#include <fmt/format.h>
@@ -99,8 +98,6 @@ ObjStorageFileReader::ObjStorageFileReader(std::shared_ptr<const ObjClientHolder
9998
DorisMetrics::instance()->s3_file_reader_total->increment(1);
10099
obj_storage_file_reader_total << 1;
101100
obj_storage_file_being_read << 1;
102-
103-
Aws::Http::SetCompliantRfc3986Encoding(true);
104101
}
105102

106103
ObjStorageFileReader::~ObjStorageFileReader() {

‎be/src/io/fs/obj_storage_file_writer.cpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717

1818
#include "io/fs/obj_storage_file_writer.h"
1919

20-
#include <aws/core/http/URI.h>
2120
#include <bvar/recorder.h>
2221
#include <bvar/reducer.h>
2322
#include <bvar/window.h>
@@ -72,7 +71,6 @@ ObjStorageFileWriter::ObjStorageFileWriter(std::shared_ptr<ObjClientHolder> clie
7271
_obj_client(std::move(client)) {
7372
obj_storage_file_writer_total << 1;
7473
obj_storage_file_being_written << 1;
75-
Aws::Http::SetCompliantRfc3986Encoding(true);
7674

7775
init_cache_builder(opts, _obj_storage_path_opts.path);
7876
}

‎be/src/io/tools/file_cache_microbench.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1807,7 +1807,7 @@ class JobManager {
18071807
std::to_string(client_config.connection.max_connections);
18081808
props["AWS_REQUEST_TIMEOUT_MS"] =
18091809
std::to_string(client_config.connection.request_timeout_ms);
1810-
props["AWS_CONNECT_TIMEOUT_MS"] =
1810+
props["AWS_CONNECTION_TIMEOUT_MS"] =
18111811
std::to_string(client_config.connection.connect_timeout_ms);
18121812
props["use_path_style"] = s3_conf.use_virtual_addressing ? "false" : "true";
18131813

‎be/src/util/s3_util.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,11 @@ AwsSdkManager::AwsSdkManager() {
4242
_aws_options.loggingOptions.logger_create_fn = [logLevel] {
4343
return std::make_shared<DorisAWSLogger>(logLevel);
4444
};
45+
// Object keys must be encoded strictly by RFC 3986 in request URIs. The setting is global
46+
// to the SDK, and InitAPI sets it from these options, so it is not reset to the SDK default
47+
// false; every object storage client is built after this factory (ObjClientFactory
48+
// initializes it first).
49+
_aws_options.httpOptions.compliantRfc3986Encoding = true;
4550
Aws::InitAPI(_aws_options);
4651
_get_ca_cert_file_path();
4752

‎be/test/io/s3_client_factory_test.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,13 @@ class SyncPointProcessingGuard {
242242

243243
} // namespace
244244

245+
TEST_F(S3ClientFactoryTest, InitializesSdkWithRfc3986KeyEncoding) {
246+
// Object keys with reserved characters, e.g. `dt=2024-01-01` in a Hive partition path, are
247+
// percent-encoded only in this mode; the SDK default leaves `$&,:=@` as they are.
248+
AwsSdkManager::instance();
249+
EXPECT_TRUE(Aws::Http::s_compliantRfc3986Encoding);
250+
}
251+
245252
TEST_F(S3ClientFactoryTest, DistinguishesHashCollisions) {
246253
auto external_conf =
247254
make_hash_collision_conf("cloud-rate-limit-hash-collision.example.com", false);

0 commit comments

Comments
 (0)