Skip to content

Commit b6ac43e

Browse files
committed
[fix](be) Keep the review fixes for vault sync logging and SDK key encoding
### What problem does this PR solve? Related PR: apache#69605 Problem Summary: The previous commit "Preserve existing object storage behavior" returned three places to their state on master in order to keep this change a pure refactor. All three had been changed at a reviewer's request, so this commit restores them. - A failed vault sync logs the status of the failing create or refresh again. On master the warning received the status of get_storage_vault_info, which is OK at that point, so a create failure, including the new configuration errors of this change, was logged with err OK. - The AWS SDK enables RFC 3986 path encoding through SDKOptions before InitAPI again, the reader and writer constructors no longer set the process-wide flag, and the test that asserts the flag after initialization is back. - The file cache microbench reader passes AWS_CONNECTION_TIMEOUT_MS again. The key on master, AWS_CONNECT_TIMEOUT_MS, is one that nothing reads, so the read client ran without the 3000 ms connection timeout of the write client. The six files are identical to commit d3b60b3 again. ### Release note None ### Check List (For Author) - Test: Not compiled and not run in this commit; this machine has no BE build. The six files are byte-identical to d3b60b3, which was built and unit tested earlier. Build hygiene passed. - Behavior changed: Yes. The vault sync warning shows the failing status, and strict RFC 3986 key encoding is on from AWS SDK initialization instead of from the first reader or writer. - Does this need documentation: No
1 parent e0caadd commit b6ac43e

6 files changed

Lines changed: 14 additions & 7 deletions

File tree

‎be/src/cloud/cloud_storage_engine.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -500,7 +500,7 @@ void CloudStorageEngine::sync_storage_vault() {
500500
: std::visit(RefreshFSVaultVisitor {id, std::move(fs), path_format},
501501
vault_info);
502502
if (!status.ok()) [[unlikely]] {
503-
LOG(WARNING) << vault_process_error(id, vault_info, std::move(st));
503+
LOG(WARNING) << vault_process_error(id, vault_info, std::move(status));
504504
}
505505
}
506506

‎be/src/io/fs/s3_file_reader.cpp‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717

1818
#include "io/fs/s3_file_reader.h"
1919

20-
#include <aws/core/http/URI.h>
2120
#include <bvar/latency_recorder.h>
2221
#include <bvar/reducer.h>
2322
#include <fmt/format.h>
@@ -96,8 +95,6 @@ S3FileReader::S3FileReader(std::shared_ptr<const ObjClientHolder> client, std::s
9695
DorisMetrics::instance()->s3_file_reader_total->increment(1);
9796
s3_file_reader_total << 1;
9897
s3_file_being_read << 1;
99-
100-
Aws::Http::SetCompliantRfc3986Encoding(true);
10198
}
10299

103100
S3FileReader::~S3FileReader() {

‎be/src/io/fs/s3_file_writer.cpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717

1818
#include "io/fs/s3_file_writer.h"
1919

20-
#include <aws/core/http/URI.h>
2120
#include <bvar/recorder.h>
2221
#include <bvar/reducer.h>
2322
#include <bvar/window.h>
@@ -70,7 +69,6 @@ S3FileWriter::S3FileWriter(std::shared_ptr<ObjClientHolder> client, std::string
7069
_obj_client(std::move(client)) {
7170
s3_file_writer_total << 1;
7271
s3_file_being_written << 1;
73-
Aws::Http::SetCompliantRfc3986Encoding(true);
7472

7573
init_cache_builder(opts, _obj_storage_path_opts.path);
7674
}

‎be/src/io/tools/file_cache_microbench.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1807,7 +1807,7 @@ class JobManager {
18071807
std::to_string(client_config.connection.max_connections);
18081808
props["AWS_REQUEST_TIMEOUT_MS"] =
18091809
std::to_string(client_config.connection.request_timeout_ms);
1810-
props["AWS_CONNECT_TIMEOUT_MS"] =
1810+
props["AWS_CONNECTION_TIMEOUT_MS"] =
18111811
std::to_string(client_config.connection.connect_timeout_ms);
18121812
props["use_path_style"] = s3_conf.use_virtual_addressing ? "false" : "true";
18131813

‎be/src/util/s3_util.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,11 @@ S3ClientFactory::S3ClientFactory() {
4242
_aws_options.loggingOptions.logger_create_fn = [logLevel] {
4343
return std::make_shared<DorisAWSLogger>(logLevel);
4444
};
45+
// Object keys must be encoded strictly by RFC 3986 in request URIs. The setting is global
46+
// to the SDK, and InitAPI sets it from these options, so it is not reset to the SDK default
47+
// false; every object storage client is built after this factory (ObjClientFactory
48+
// initializes it first).
49+
_aws_options.httpOptions.compliantRfc3986Encoding = true;
4550
Aws::InitAPI(_aws_options);
4651
_get_ca_cert_file_path();
4752

‎be/test/io/s3_client_factory_test.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,13 @@ class SyncPointProcessingGuard {
242242

243243
} // namespace
244244

245+
TEST_F(S3ClientFactoryTest, InitializesSdkWithRfc3986KeyEncoding) {
246+
// Object keys with reserved characters, e.g. `dt=2024-01-01` in a Hive partition path, are
247+
// percent-encoded only in this mode; the SDK default leaves `$&,:=@` as they are.
248+
S3ClientFactory::instance();
249+
EXPECT_TRUE(Aws::Http::s_compliantRfc3986Encoding);
250+
}
251+
245252
TEST_F(S3ClientFactoryTest, DistinguishesHashCollisions) {
246253
auto external_conf =
247254
make_hash_collision_conf("cloud-rate-limit-hash-collision.example.com", false);

0 commit comments

Comments
 (0)