Skip to content

Commit 41567d7

Browse files
morningmanxylaaaaaclaude
committed
[fix](be) Keep ADLS names literal and canonicalize Azure hosts in validate_azure_uri
Review follow-up for #68117, three defects in the new native Azure URI contract: - S3URI::parse() stripped '?' and '#' from ABFS/WASB object paths. Iceberg's ADLSLocation ("^(abfss?|wasbs?)://([^/?#]+)(.*)?$") and Hadoop's Path both take everything after the authority as the object path, so an object named path/a?b#c.parquet was opened as path/a. Query and fragment are now only stripped for the S3 and HTTP(S) schemes. - The OneLake fence compared the raw authority, so an explicit port (onelake.dfs.fabric.microsoft.com:443) or a trailing DNS root dot made a genuine Fabric location pass the later checks and reach the native client. The fence now matches on the DNS host (case-folded, no port, no root dot). - Azure RA-GRS/RA-GZRS secondary endpoints append "-secondary" to the account label while keeping the primary account identity and keys (the SDK signs with the credential's account name and only swaps the host). The account check compared that label with the credential and rejected valid secondary reads. On official Azure hosts the suffix now folds to the primary account for the ABFS/WASB endpoint identity and for the credential account check; HTTP(S) transport matching stays exact. Tests: S3URITest.QueryAndFragmentAreLiteralOnlyInAdlsPaths (plus '?'/'#' names in IcebergAdlsPathsPreserveLiteralPercentSequences), S3ClientFactoryTest.OneLakeFenceIgnoresPortAndDnsRootDot, AllowsDocumentedAzureSecondaryEndpoints and SecondaryFoldingKeepsOtherAccountsApart. Issue: #68103 Related PR: #67470 Co-authored-by: xylaaaaa <[email protected]> Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 3eb12a8 commit 41567d7

5 files changed

Lines changed: 172 additions & 17 deletions

File tree

‎be/src/util/s3_uri.cpp‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,13 @@ bool S3URI::is_azure_endpoint(std::string_view authority) {
5757
});
5858
}
5959

60+
bool S3URI::_is_adls_scheme(std::string_view scheme) {
61+
return absl::EqualsIgnoreCase(scheme, _SCHEME_ABFS) ||
62+
absl::EqualsIgnoreCase(scheme, _SCHEME_ABFSS) ||
63+
absl::EqualsIgnoreCase(scheme, _SCHEME_WASB) ||
64+
absl::EqualsIgnoreCase(scheme, _SCHEME_WASBS);
65+
}
66+
6067
Status S3URI::_parsing_error(std::string_view message, bool azure_provider) const {
6168
if (azure_provider || _is_azure) {
6269
return Status::InvalidArgument("{}", message);
@@ -79,10 +86,7 @@ Status S3URI::_parse_authority(const std::string& scheme, const std::string& res
7986
_bucket = authority_split[0];
8087
// support s3://bucket1
8188
_key = authority_split.size() == 1 ? "/" : authority_split[1];
82-
} else if (absl::EqualsIgnoreCase(scheme, _SCHEME_ABFS) ||
83-
absl::EqualsIgnoreCase(scheme, _SCHEME_ABFSS) ||
84-
absl::EqualsIgnoreCase(scheme, _SCHEME_WASB) ||
85-
absl::EqualsIgnoreCase(scheme, _SCHEME_WASBS)) {
89+
} else if (_is_adls_scheme(scheme)) {
8690
// Azure Data Lake paths use container@account-host as the
8791
// authority. Keep the account host so the native Azure client
8892
// can derive its endpoint without consulting Hadoop settings.
@@ -161,10 +165,14 @@ Status S3URI::parse(bool azure_provider) {
161165
if (_key.empty()) {
162166
return _parsing_error("Invalid S3 key", azure_provider);
163167
}
164-
// Strip query and fragment if they exist
165-
std::vector<std::string> _query_split = absl::StrSplit(_key, _QUERY_DELIM);
166-
std::vector<std::string> _fragment_split = absl::StrSplit(_query_split[0], _FRAGMENT_DELIM);
167-
_key = _fragment_split[0];
168+
// Strip query and fragment if they exist. ABFS/WASB locations have neither:
169+
// ADLSLocation and Hadoop's Path take everything after the authority as the
170+
// object path, so '?' and '#' are literal object-name characters there.
171+
if (!_is_adls_scheme(_scheme)) {
172+
std::vector<std::string> _query_split = absl::StrSplit(_key, _QUERY_DELIM);
173+
std::vector<std::string> _fragment_split = absl::StrSplit(_query_split[0], _FRAGMENT_DELIM);
174+
_key = _fragment_split[0];
175+
}
168176
// ADLSFileIO's ADLSLocation passes ABFS/WASB names literally to the SDK:
169177
// an object called p=a%2Fb must not become p=a/b. Only HTTP(S) locations
170178
// represent URL-encoded names. The SDK encodes the resulting object name.

‎be/src/util/s3_uri.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ namespace doris {
4141
// Azure Data Lake locations intentionally retain their original URI form. The
4242
// object-storage factory uses the parsed endpoint/account to construct the
4343
// native Azure client; they are not routed through the Hadoop filesystem.
44+
// Like ADLSLocation and Hadoop's Path, an ABFS/WASB location has no query or
45+
// fragment: '?' and '#' after the authority are literal object-name characters.
4446
class S3URI {
4547
public:
4648
S3URI(const std::string& location) : _location(location) {}
@@ -65,6 +67,7 @@ class S3URI {
6567
Status _parse_authority(const std::string& scheme, const std::string& rest,
6668
bool azure_provider);
6769
Status _parsing_error(std::string_view message, bool azure_provider) const;
70+
static bool _is_adls_scheme(std::string_view scheme);
6871

6972
static const std::string _SCHEME_S3;
7073
static const std::string _SCHEME_ABFS;

‎be/src/util/s3_util.cpp‎

Lines changed: 47 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -323,6 +323,34 @@ std::string endpoint_authority(const std::string& endpoint) {
323323
return authority;
324324
}
325325

326+
// The host of an authority as DNS sees it: lower-cased, without an explicit
327+
// port and without a trailing root dot.
328+
std::string authority_host(std::string_view authority) {
329+
auto host = to_lower(std::string(authority.substr(0, authority.find(':'))));
330+
while (host.ends_with('.')) {
331+
host.pop_back();
332+
}
333+
return host;
334+
}
335+
336+
// Azure RA-GRS/RA-GZRS secondary endpoints append "-secondary" to the account
337+
// label of an official service host while keeping the primary account identity
338+
// and keys: the SDK signs secondary reads with the primary account name. Fold
339+
// that suffix so both hosts name the same logical account. Custom hosts are
340+
// returned unchanged; their labels carry no Azure meaning.
341+
std::string azure_primary_authority(std::string authority) {
342+
if (!S3URI::is_azure_endpoint(authority)) {
343+
return authority;
344+
}
345+
static constexpr std::string_view secondary = "-secondary";
346+
// An Azure service host always has a dotted suffix after the account label.
347+
const auto label = to_lower(authority.substr(0, authority.find('.')));
348+
if (label.size() > secondary.size() && label.ends_with(secondary)) {
349+
authority.erase(label.size() - secondary.size(), secondary.size());
350+
}
351+
return authority;
352+
}
353+
326354
#ifdef USE_AZURE
327355
// The endpoint handed to the SDK. Native endpoints were already normalized
328356
// while parsing the native protocol. Legacy SharedKey producers hand over an
@@ -530,7 +558,9 @@ Status S3ClientFactory::validate_azure_uri(const S3URI& uri, const S3ClientConf&
530558
}
531559
return Status::OK(); // Old SharedKey file-system paths use the S3 wire spelling.
532560
}
533-
const auto uri_host = to_lower(uri.get_endpoint());
561+
// Fence on the DNS host: an explicit port or a trailing root dot must not
562+
// let a genuine Fabric location slip past this check.
563+
const auto uri_host = authority_host(uri.get_endpoint());
534564
if (uri_host.ends_with(".dfs.fabric.microsoft.com") ||
535565
uri_host.ends_with(".blob.fabric.microsoft.com")) {
536566
return Status::NotSupported("OneLake data access requires its Hadoop storage binding");
@@ -542,20 +572,29 @@ Status S3ClientFactory::validate_azure_uri(const S3URI& uri, const S3ClientConf&
542572
// ABFS/WASB authorities identify the logical Azure account, while a configured custom
543573
// endpoint may be a proxy or emulator that intentionally has a different HTTP authority.
544574
// The native client uses conf.endpoint as the transport origin; retain account validation
545-
// below, but do not reject this valid proxy form. HTTP(S) locations carry their transport
546-
// origin directly and must still match exactly.
575+
// below, but do not reject this valid proxy form. A documented "-secondary" endpoint
576+
// names that same logical account. HTTP(S) locations carry their transport origin
577+
// directly and must still match exactly.
547578
const bool custom_transport = !S3URI::is_azure_endpoint(endpoint_authority(endpoint));
548-
if (endpoint_authority(endpoint) != endpoint_authority(uri_endpoint) &&
549-
!(custom_transport && !http_uri)) {
579+
const auto endpoint_identity = http_uri ? endpoint_authority(endpoint)
580+
: azure_primary_authority(endpoint_authority(endpoint));
581+
const auto uri_identity = http_uri ? endpoint_authority(uri_endpoint)
582+
: azure_primary_authority(endpoint_authority(uri_endpoint));
583+
if (endpoint_identity != uri_identity && !(custom_transport && !http_uri)) {
550584
return Status::InvalidArgument(
551585
"Azure URI account host conflicts with the storage endpoint");
552586
}
553587
if (http_uri && !endpoint.starts_with(uri.get_scheme() + "://")) {
554588
return Status::InvalidArgument("Azure URI scheme conflicts with the storage endpoint");
555589
}
556-
if (S3URI::is_azure_endpoint(uri.get_endpoint()) &&
557-
!iequal(conf.azure_credentials.account_name, uri.get_account())) {
558-
return Status::InvalidArgument("Azure URI account conflicts with the credential account");
590+
if (S3URI::is_azure_endpoint(uri.get_endpoint())) {
591+
// The credential always carries the primary account name, also for
592+
// reads against the account's "-secondary" host.
593+
const auto primary = azure_primary_authority(to_lower(uri.get_endpoint()));
594+
if (!iequal(conf.azure_credentials.account_name, primary.substr(0, primary.find('.')))) {
595+
return Status::InvalidArgument(
596+
"Azure URI account conflicts with the credential account");
597+
}
559598
}
560599
return Status::OK();
561600
}

‎be/test/io/s3_client_factory_test.cpp‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -902,6 +902,85 @@ TEST_F(S3ClientFactoryTest, RejectsNativeAzureLocationConflicts) {
902902
}
903903
}
904904

905+
TEST_F(S3ClientFactoryTest, OneLakeFenceIgnoresPortAndDnsRootDot) {
906+
// OneLake stays on its Hadoop binding. An explicit port, a trailing DNS root
907+
// dot or upper-case spelling must not route a Fabric location through the
908+
// native client, whichever transport endpoint the binding names.
909+
for (const auto* endpoint :
910+
{"https://onelake.dfs.fabric.microsoft.com", "https://proxy.example.test:8443/base"}) {
911+
auto properties = native_azure_shared_key_properties();
912+
properties["AZURE_ENDPOINT"] = endpoint;
913+
for (const auto* location :
914+
{"abfss://[email protected]/lakehouse/Files/file",
915+
"abfss://[email protected]:443/lakehouse/Files/file",
916+
"abfss://[email protected]./lakehouse/Files/file",
917+
"https://onelake.blob.fabric.microsoft.com:443/workspace/lakehouse/Files/file",
918+
"https://onelake.dfs.fabric.microsoft.com:443/workspace/lakehouse/Files/file"}) {
919+
S3URI uri(location);
920+
ASSERT_TRUE(uri.parse().ok()) << location;
921+
S3Conf conf;
922+
auto status = S3ClientFactory::convert_properties_to_s3_conf(properties, uri, &conf);
923+
EXPECT_TRUE(status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>())
924+
<< endpoint << " " << location << " " << status;
925+
}
926+
}
927+
}
928+
929+
TEST_F(S3ClientFactoryTest, AllowsDocumentedAzureSecondaryEndpoints) {
930+
// RA-GRS/RA-GZRS secondary reads keep the primary account identity and
931+
// keys; only the service host gains a "-secondary" label.
932+
for (const auto* endpoint : {"https://account-secondary.blob.core.windows.net",
933+
"https://account-secondary.dfs.core.windows.net:443",
934+
"https://account.blob.core.windows.net"}) {
935+
auto properties = native_azure_shared_key_properties();
936+
properties["AZURE_ENDPOINT"] = endpoint;
937+
for (const auto* location :
938+
{"abfss://[email protected]/path/file",
939+
"wasbs://[email protected]:443/path/file",
940+
"abfss://[email protected]/path/file"}) {
941+
S3URI uri(location);
942+
ASSERT_TRUE(uri.parse().ok()) << location;
943+
S3Conf conf;
944+
auto status = S3ClientFactory::convert_properties_to_s3_conf(properties, uri, &conf);
945+
EXPECT_TRUE(status.ok()) << endpoint << " " << location << " " << status;
946+
EXPECT_EQ(conf.client_conf.azure_credentials.account_name, "account");
947+
}
948+
}
949+
// HTTP(S) locations name their transport origin, so they still have to
950+
// match the configured endpoint exactly; the account check then applies
951+
// the same folding.
952+
auto properties = native_azure_shared_key_properties();
953+
properties["AZURE_ENDPOINT"] = "https://account-secondary.blob.core.windows.net";
954+
S3URI secondary_https("https://account-secondary.blob.core.windows.net/container/path/file");
955+
ASSERT_TRUE(secondary_https.parse().ok());
956+
S3Conf conf;
957+
auto status =
958+
S3ClientFactory::convert_properties_to_s3_conf(properties, secondary_https, &conf);
959+
EXPECT_TRUE(status.ok()) << status;
960+
S3URI primary_https("https://account.blob.core.windows.net/container/path/file");
961+
ASSERT_TRUE(primary_https.parse().ok());
962+
EXPECT_FALSE(
963+
S3ClientFactory::convert_properties_to_s3_conf(properties, primary_https, &conf).ok());
964+
}
965+
966+
TEST_F(S3ClientFactoryTest, SecondaryFoldingKeepsOtherAccountsApart) {
967+
// Folding "-secondary" must not merge distinct accounts: another account's
968+
// secondary host, an account literally named "secondary" and the secondary
969+
// host of the account named "account-secondary" all stay rejected.
970+
auto properties = native_azure_shared_key_properties();
971+
properties["AZURE_ENDPOINT"] = "https://account-secondary.blob.core.windows.net";
972+
for (const auto* location :
973+
{"abfss://[email protected]/path/file",
974+
"abfss://[email protected]/path/file",
975+
"abfss://[email protected]/path/file"}) {
976+
S3URI uri(location);
977+
ASSERT_TRUE(uri.parse().ok()) << location;
978+
S3Conf conf;
979+
EXPECT_FALSE(S3ClientFactory::convert_properties_to_s3_conf(properties, uri, &conf).ok())
980+
<< location;
981+
}
982+
}
983+
905984
TEST_F(S3ClientFactoryTest, AllowsAbfsUriWithCustomAzureTransportEndpoint) {
906985
auto properties = native_azure_shared_key_properties();
907986
properties["AZURE_ENDPOINT"] = "https://proxy.example.test:8443/base";

‎be/test/util/s3_uri_test.cpp‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,8 @@ TEST_F(S3URITest, IcebergAdlsPathsPreserveLiteralPercentSequences) {
164164
for (const auto* scheme : {"abfs", "abfss", "wasb", "wasbs"}) {
165165
for (const auto* key :
166166
{"data/p=a%2Fb/file.parquet", "path/a%20b+%2520/http://example//file",
167-
"path/100%/%2/%GG", "path/a b "}) {
167+
"path/100%/%2/%GG", "path/a b ", "path/a?b#c.parquet", "path/a#b?c.parquet",
168+
"?sv=2024-01-01&sig=fake", "#", "path/?/#/file"}) {
168169
const std::string service =
169170
std::string_view(scheme).starts_with("wasb") ? "blob" : "dfs";
170171
S3URI uri(std::string(scheme) + "://container@account." + service +
@@ -258,4 +259,29 @@ TEST_F(S3URITest, QueryAndFragment) {
258259
EXPECT_EQ("path/to/file", uri1.get_key());
259260
}
260261

262+
TEST_F(S3URITest, QueryAndFragmentAreLiteralOnlyInAdlsPaths) {
263+
// ADLSLocation and Hadoop's Path treat everything after an ABFS/WASB
264+
// authority as the object path. Every other scheme still carries a query
265+
// (an HTTP(S) Azure location may carry a SAS) and a fragment.
266+
for (const auto* location :
267+
{"abfss://[email protected]/path/a?b#c.parquet",
268+
"WASB://[email protected]/path/a?b#c.parquet"}) {
269+
S3URI uri(location);
270+
ASSERT_TRUE(uri.parse().ok()) << location;
271+
EXPECT_EQ(uri.get_key(), "path/a?b#c.parquet") << location;
272+
ASSERT_TRUE(uri.parse(true).ok()) << location;
273+
EXPECT_EQ(uri.get_key(), "path/a?b#c.parquet") << location;
274+
}
275+
for (const auto* location :
276+
{"https://account.blob.core.windows.net/container/path/a?sv=1&sig=fake#c",
277+
"https://custom.example.com/container/path/a?sv=1&sig=fake#c",
278+
"s3://container/path/a?sv=1&sig=fake#c"}) {
279+
S3URI uri(location);
280+
ASSERT_TRUE(uri.parse().ok()) << location;
281+
EXPECT_EQ(uri.get_key(), "path/a") << location;
282+
ASSERT_TRUE(uri.parse(true).ok()) << location;
283+
EXPECT_EQ(uri.get_key(), "path/a") << location;
284+
}
285+
}
286+
261287
} // end namespace doris

0 commit comments

Comments
 (0)