feat(services/azdls): Support parsing Azure Storage configs from connection strings by DerGut · Pull Request #6212 · apache/opendal

DerGut · 2025-05-21T21:00:56Z

Which issue does this PR close?

Closes #6211.

Rationale for this change

As mentioned in the issue description. Azure SDKs tend to allow users to pass a connection string. This PR parsing functionality to populate an AzdlsConfig struct with fields from a connection string.

Additionally, I'd like to add this for the Iceberg ADLS support, as mentioned in the issue.

What changes are included in this PR?

The primary change is the user-facing one to support parsing Azure connection strings into Azdls structs, see user-facing changes below.

There are internal refactorings that come along with it too:

services-azblob, services-azdls and services-azfile now share some abstractions in raw/azure.rs
- a shared raw::azure_account_name_from_endpoint()
- a shared raw::azure_config_from_connection_string()
services-azdls and services-azfile are now included in the tests feature flag set

Are there any user-facing changes?

Yes!

🟢 Additions

Azdls::from_connection_string(): users are now able to parse Azure connection strings to get an AzdlsBuilder
Both AzblobConfig and AzdlsConfig now implement From<reqsign::AzureStorageConfig>
Azblob::from_connection_string() now supports Azurite parameters UseDevelopmentStorage and DevelopmentStorageURI
Azblob::from_connection_string() now always extracts the account name (it previously only used it when shared key authentication was used)
Also added Azfile::from_connection_string() while I was at it

🟡 Changes

Azblob::from_connection_string() now errors when DefaultEndpointsProtocol is invalid (i.e. not http or https)
Azblob::from_connection_string() now errors when fields are invalid (i.e. don't have the format of key=value)
Azblob::from_connection_string() is more tolerant to multi-line connection strings (i.e. trims white space of individual key-value pairs too)

🔴 Removals

None

Signed-off-by: Jannik Steinmann <[email protected]>

jorgehermo9 · 2025-05-22T22:34:54Z

core/src/services/azdls/config.rs

+    /// config.client_secret = Some("myClientSecret".to_string());
+    /// config.tenant_id = Some("myTenantId".to_string());
+    /// ```
+    pub fn try_from_connection_string(conn_str: &str) -> Result<Self> {


I think this would be very useful for the next steps of #5482, this is how I would like to see custom Configurator::from_uri

opendal/core/src/types/builder.rs

Line 152 in 7309b4b

fn from_uri(uri: &Uri, options: impl IntoIterator<Item = (String, String)>) -> Result<Self> {

@Xuanwo

Thanks for the reference Jorge!

I'd like to rename the AzdlsConfig::try_from_connection_string() to AzdlsConfig::from_uri() to make it more generic, but I'm a bit cautious about treating the Azure connection string format as a URI.

Azure Storage connection strings are ;-separated key-value pairs. An example looks like this:

AccountName=example;DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net

When I parse such a string using http::uri::Uri, it interprets it as the host name:

let uri = conn_str.parse::<http::uri::Uri>().unwrap(); // Prints: host: AccountName=example;DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net println!("host: {}", uri.host().unwrap());

Technically, RFC#3986 (the RFC that defines the URI syntax) allows = and ; as part of the host name, but protocols like http further restrict allowed names via the DNS name syntax (e.g. RFC#1035). As I suspect the http crate to focus on URI parsing logic for HTTP protocol family, I'm not certain that this behavior won't change in the future.

In short: I'd rather refrain from treating Azure Storage connection strings as URIs (and parse them using that logic). So I think it makes sense to keep both APIs separate.

jorgehermo9 · 2025-05-22T22:43:48Z

core/src/services/azdls/config.rs

+fn parse_connection_string(conn_str: &str) -> Result<Vec<(&str, &str)>> {
+    conn_str
+        .split(';')
+        .map(|field| {


In #5482 we are going to introduce a query_pairs http util that does almost the same thing (but silently ignoring error instead of failing)

opendal/core/src/raw/http_util/uri.rs

Line 71 in 7309b4b

pub fn query_pairs(query: &str) -> Vec<(String, String)> {

Thanks, this is great! I'll change it (and add a separator: &str parameter to accommodate the ;) if your PR gets in first 👍

Xuanwo · 2025-05-23T12:13:50Z

Thank you, @DerGut, for initiating this, and thank you, @jorgehermo9, for joining the review. The approach to how OpenDAL implements from_uri is still unclear at this point. I suggest we exclude the design of from_uri from our current considerations.

Azblob has a from_connection_string API already. Perhaps we can extract this functionality to opendal::raw::azure_utils so that we can share the same implementation across all Azure-related services.

DerGut · 2025-05-23T12:32:56Z

Oh wow, I don't know why it didn't occur to me to check 🙈
I'll take a stab at it later today. Thanks for bringing it up!

Signed-off-by: Jannik Steinmann <[email protected]>

DerGut · 2025-05-26T15:37:28Z

I've refactored the connection string parsing logic into a shared raw/azure.rs. This is much better! I also noticed that we may even be able to push some of this down to reqsign in a later PR.

Signed-off-by: Jannik Steinmann <[email protected]>

DerGut · 2025-05-26T18:31:12Z

The test failures seem unrelated 🤔

Signed-off-by: DerGut <[email protected]>

Xuanwo

Most changes look good to me, perfect!

Xuanwo · 2025-05-29T04:53:41Z

core/src/raw/mod.rs


+#[cfg(any(feature = "services-azblob", feature = "services-azdls"))]
+mod azure;
+#[cfg(any(feature = "services-azblob", feature = "services-azdls"))]


Seems we need to add services-azfile here?

You are right, I missed this! Added it with 818a2ce

core/src/raw/azure.rs

Signed-off-by: DerGut <[email protected]>

Xuanwo

Thank you @DerGut a lot for building this!

DerGut added 3 commits May 21, 2025 22:21

Add AzdlConfig::try_from_connection_string

38f9e27

Signed-off-by: Jannik Steinmann <[email protected]>

Reuse DfsEndpoint in backend code

624a4db

Signed-off-by: Jannik Steinmann <[email protected]>

Ignore clippy warning

5fe8416

Signed-off-by: Jannik Steinmann <[email protected]>

DerGut requested a review from Xuanwo as a code owner May 21, 2025 21:00

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels May 21, 2025

DerGut changed the title ~~Support parsing AzdlConfig from Azure Storage connection strings~~ feat(services/azdls) Support parsing AzdlConfig from Azure Storage connection strings May 21, 2025

Fix CI warnings

f83267e

Signed-off-by: Jannik Steinmann <[email protected]>

DerGut mentioned this pull request May 22, 2025

feat(storage-azdls): Add Azure Datalake Storage support apache/iceberg-rust#1368

Merged

jorgehermo9 reviewed May 22, 2025

View reviewed changes

DerGut added 8 commits May 26, 2025 12:52

Create raw azure mod with account_name from endpoint util

91c46fc

Signed-off-by: Jannik Steinmann <[email protected]>

Move connection string parsing to azure mod

23c7bfb

Signed-off-by: Jannik Steinmann <[email protected]>

Include services-azdls in tests feature

0b7e03a

Signed-off-by: Jannik Steinmann <[email protected]>

Add support for development storage fields

aa1a572

Signed-off-by: Jannik Steinmann <[email protected]>

Fix doc string

865c390

Signed-off-by: Jannik Steinmann <[email protected]>

Include azdfs in AccessorInfo.name() docstring

e5873e0

Signed-off-by: Jannik Steinmann <[email protected]>

Fix CI warnings

f52d6ce

Signed-off-by: Jannik Steinmann <[email protected]>

Merge branch 'main' into main

13ea364

Add support for azfile

a313799

Signed-off-by: Jannik Steinmann <[email protected]>

DerGut changed the title ~~feat(services/azdls) Support parsing AzdlConfig from Azure Storage connection strings~~ feat(services/azdls) Support parsing Azure Storage configs from connection strings May 26, 2025

DerGut added 2 commits May 26, 2025 18:50

Fix file test

7df20d4

Signed-off-by: Jannik Steinmann <[email protected]>

Fix development storage test

56ffbf4

Signed-off-by: Jannik Steinmann <[email protected]>

DerGut and others added 3 commits May 29, 2025 02:03

Merge branch 'main' into main

50c2433

Pass endpoint in AzfileConfig::from

6a9f9bf

Signed-off-by: DerGut <[email protected]>

Reuse account name from endpoint

353060a

Signed-off-by: DerGut <[email protected]>

Xuanwo reviewed May 29, 2025

View reviewed changes

core/src/raw/azure.rs Outdated Show resolved Hide resolved

core/src/raw/azure.rs Show resolved Hide resolved

DerGut added 3 commits May 29, 2025 11:48

Include raw/azure when services-azfile feature is enabled

818a2ce

Signed-off-by: DerGut <[email protected]>

Simplify Option<String> comparison

5f27b61

Signed-off-by: DerGut <[email protected]>

Fix simplification

070ef84

Signed-off-by: DerGut <[email protected]>

Xuanwo changed the title ~~feat(services/azdls) Support parsing Azure Storage configs from connection strings~~ feat(services/azdls): Support parsing Azure Storage configs from connection strings May 29, 2025

Xuanwo approved these changes May 29, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label May 29, 2025

Xuanwo merged commit 60f4a0d into apache:main May 29, 2025
275 of 277 checks passed

DerGut mentioned this pull request Jun 9, 2025

feat(azure-storage): Add config parsing from connection string apache/opendal-reqsign#524

Open

vustef mentioned this pull request Dec 12, 2025

Allow passing endpoint through the "adls.connection-string" RelationalAI/iceberg-rust#23

Merged

Comments

Conversation

DerGut commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

🟢 Additions

🟡 Changes

🔴 Removals

Uh oh!

jorgehermo9 May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DerGut May 23, 2025

Choose a reason for hiding this comment

Uh oh!

jorgehermo9 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

DerGut May 23, 2025

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented May 23, 2025

Uh oh!

DerGut commented May 23, 2025

Uh oh!

DerGut commented May 26, 2025

Uh oh!

DerGut commented May 26, 2025

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Xuanwo May 29, 2025

Choose a reason for hiding this comment

Uh oh!

DerGut May 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DerGut commented May 21, 2025 •

edited

Loading

jorgehermo9 May 22, 2025 •

edited

Loading