feat(azure_blob sink): Update crates and migrate to the new SDK #24255

joshcoughlan · 2025-11-17T18:17:06Z

This PR migrates the Azure Blob sink to the latest official Azure SDK crates and aligns our internal auth/HTTP plumbing with the new pipeline model.

Highlights:

Replace legacy crates with official SDK:
- azure_storage_blob → 0.7.x
- azure_core → 0.30.x
- Remove azure_storage_blobs/azure_storage/azure_core_for_storage
Preserve existing auth modes:
- SAS via URL stays supported
- Access key via a custom Shared Key Policy integrated into the azure_core pipeline
Add a minimal connection string parser to support SAS and Shared Key without pulling in legacy crates
Keep the rest of Vector on reqwest 0.11.26 while the Azure SDK uses reqwest 0.12.24:
- Introduce an aliased reqwest_0_12_24 dependency and inject it as the Azure transport
- Avoid rippling reqwest upgrades through non-Azure code paths

Follow-ups planned:

Per-sink proxy configuration for the Azure HTTP client (azure_blob proxy support #21314)
Possibly Support Azure Active Directory tokens (Support Azure Active Directory tokens for the Azure Blob Store Sink #24167)

Vector configuration

Examples used during testing.

SAS-based connection string:

[sinks.azure_blob]
type = "azure_blob"
inputs = ["in"]
container_name = "vector-logs"
# Example SAS (redacted); must include the leading '?'
connection_string = "BlobEndpoint=https://<account>.blob.core.windows.net;SharedAccessSignature=?sv=...&ss=b&srt=...&sp=...&se=...&st=...&spr=https&sig=...;"
compression = "gzip"
encoding.codec = "json"

Shared Key (access key) connection string:

[sinks.azure_blob]
type = "azure_blob"
inputs = ["in"]
container_name = "vector-logs"
connection_string = "DefaultEndpointsProtocol=https;AccountName=<account>;AccountKey=<base64key>;EndpointSuffix=core.windows.net"
compression = "gzip"
encoding.codec = "json"
# Uses the custom SharedKeyAuthorizationPolicy added in this PR

How did you test this PR?

Local builds for all supported targets, including cross builds in a containerized environment
Tested in a live environment to confirm functionality

Change Type

Bug fix
[ X] New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
[X ] No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

This PR fixes an unreported bug introduced by Azure SDK version bump introduced regression in token refresh #23036 in which the documented SAS connection_string ("BlobEndpoint=https://mylogstorage.blob.core.windows.net/;SharedAccessSignature=generatedsastoken") doesn't work and fails with an error indicating it requires AccountName:
ERROR vector::topology::builder: Configuration error. error=Sink "testing-azure_blob": Account name missing in connection string

thomasqueirozb

Thanks for this contribution! I haven't had the time to look closely at everything yet, but I'm leaving some comments now because there is a lot to discuss.

As per your PR description it seems like we're now supporting new auth methods with this upgrade too, so it would be nice to include a changelog in this PR. Let me know if I misunderstood this.

I do have some concerns about using these new azure crates though. The azure_storage_blob crate has this warning:

Note: The azure_storage_blob crate is currently under active development and not all features may be implemented or work as intended. This crate is in beta and not suitable for Production environments. For any general feedback or usage issues, please open a GitHub issue

I see that we are using just some parts of that crate. Namely only BlobContainerClient/BlobContainerClientOptions, clients::BlobClient and models::BlockBlobClientUploadOptions.

Have you checked if they are just a port from what was being used in the old azure_storage_blobs crate or is it a full rewrite? If it is a port I'd have more confidence in using it. Either way I think our hands are tied here because MS doesn't seem to put enough effort into releasing good and stable Azure crates so anything as good as or better than the old azure_storage_blobs will do.

Cargo.toml

src/sinks/azure_common/shared_key_policy.rs

joshcoughlan · 2025-11-17T19:25:15Z

Have you checked if they are just a port from what was being used in the old azure_storage_blobs crate or is it a full rewrite? If it is a port I'd have more confidence in using it. Either way I think our hands are tied here because MS doesn't seem to put enough effort into releasing good and stable Azure crates so anything as good as or better than the old azure_storage_blobs will do.

I haven't, but I suspect it is a full rewrite since I think it is just adding support for auth from a url to the SDK. I wrote a fix for the SDK and submitted it to Microsoft, but was informed that they had fixed what I was trying to fix in a release of azure_storage_blob the day before.

Basically, clients now have a from_url method that allows you to pass a full URL with a SAS. This design closely matches our other SDKs and so it what we wanted to go with for Rust as well. Please give it a try and let us know if you have any feedback.
Azure/azure-sdk-for-rust#3329 (comment)

If I'm remembering properly, the only thing keeping SAS from working before is that the client constructor required the AD auth method. I think now it is making that credential optional, which then opened the door for auth from url.
Azure/azure-sdk-for-rust#3176

Anyway, I went about refactoring my original Vector PR to use the newly released azure_storage_blob SDK and broke it up into two PRs. I'm mainly doing all of this because we need per-sink proxy support and the legacy crates just didn't really support that so much. I have another PR that builds on this one to add proxy support: #24256

As per your PR description it seems like we're now supporting new auth methods with this upgrade too, so it would be nice to include a changelog in this PR. Let me know if I misunderstood this.

Same auth methods, SAS and Access key, are supported, but I think the implementation is different. I think this PR should help address #24167 though, so it can lead to additional auth methods.

thomasqueirozb · 2025-11-21T18:48:44Z

Same auth methods

Changelog not needed then

There are a few failing checks (and merge conflicts), I'll circle back to this once there are new commits on this branch.

fplantinga-guida · 2026-01-06T08:22:09Z

Any updates on this? @thomasqueirozb @joshcoughlan

joshcoughlan · 2026-01-07T16:53:45Z

Any updates on this? @thomasqueirozb @joshcoughlan

I'm working through this today getting the clippy whatnot passing and resolving merge conflicts. I didn't really realize there were other PRs waiting on this.

Also, full disclosure here, I'd be happy if more eyeballs were on this PR as a good amount of it is faith-based coding with AI guidance. I'm none too familiar with rust and having more knowledgeable humans involved wouldn't be a bad thing.

github-actions · 2026-01-07T20:28:43Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

Co-authored-by: Thomas <[email protected]>

joshcoughlan · 2026-01-08T02:11:12Z

I have read the CLA Document and I hereby sign the CLA

joshcoughlan · 2026-01-08T02:11:33Z

recheck

thomasqueirozb · 2026-01-09T17:43:45Z

Cargo.toml

 # Azure
-azure_core = { version = "0.25", default-features = false, features = ["reqwest", "hmac_openssl"], optional = true }
-azure_identity = { version = "0.25", default-features = false, features = ["reqwest"], optional = true }
+azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"] }


Suggested change

azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"] }

azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"], optional = true }

This should remain optional as we don't want this to be compiled in when Vector is built with minimal features

Also, looking at https://docs.rs/crate/azure_core/latest/features I'm worried that without reqwest_native_tls and maybe also without reqwest_deflate we can run into some issues...

FWIW reqwest now sets its default TLS feature to use rustls, instead of native-tls.

https://seanmonstar.com/blog/reqwest-v013-rustls-default/

thomasqueirozb · 2026-01-09T20:29:19Z

Cargo.toml

 sinks-aws_sns = ["aws-core", "dep:aws-sdk-sns"]
 sinks-axiom = ["sinks-http"]
-sinks-azure_blob = ["dep:azure_core", "dep:azure_identity", "dep:azure_storage", "dep:azure_storage_blobs"]
+sinks-azure_blob = ["dep:azure_storage_blob"]


Suggested change

sinks-azure_blob = ["dep:azure_storage_blob"]

sinks-azure_blob = ["dep:azure_core", "dep:azure_storage_blob"]

Refer to earlier comment

thomasqueirozb · 2026-01-09T20:30:48Z

src/sinks/azure_blob/integration_tests.rs

    assert!(blobs[0].clone().ends_with(".log"));
-    let (blob, blob_lines) = config.get_blob(blobs[0].clone()).await;
-    assert_eq!(blob.properties.content_type, String::from("text/plain"));
+    let (content_type, _content_encoding, blob_lines) = config.get_blob(blobs[0].clone()).await;


I'd assume that the encoding here is None right?

Suggested change

let (content_type, _content_encoding, blob_lines) = config.get_blob(blobs[0].clone()).await;

let (content_type, content_encoding, blob_lines) = config.get_blob(blobs[0].clone()).await;

assert_eq!(content_encoding, None);

We can copy what is done in the following test

thomasqueirozb · 2026-01-09T20:41:51Z

src/sinks/azure_common/config.rs

 #[derive(Debug)]
 pub struct AzureBlobResponse {
-    pub inner: PutBlockBlobResponse,
+    pub inner: (),


Suggested change

pub inner: (),

We should be able to remove this

thomasqueirozb · 2026-01-09T20:45:26Z

src/sinks/azure_common/config.rs

+    // Force Azure SDK to use reqwest_0_12_24 transport to avoid affecting global reqwest
+    options.client_options.transport = Some(azure_core::http::Transport::new(std::sync::Arc::new(
+        reqwest_0_12_24::ClientBuilder::new()
+            .build()
+            .map_err(|e| format!("Failed to build reqwest_0_12_24 client: {e}"))?,
+    )));


Why are we using another version of reqwest here? I'd assume that this would work just fine if we used reqwest::ClientBuilder instead - or is there a version issue?

thomasqueirozb · 2026-01-09T20:46:40Z

src/sinks/azure_common/shared_key_policy.rs

+            // Sort values (optional but deterministic)
+            vals.sort();


Is this required? (wouldn't surprise me if MS made it so...)

thomasqueirozb · 2026-01-09T20:52:30Z

src/sinks/azure_common/connection_string.rs

+This module intentionally avoids relying on the legacy Azure Storage SDK crates.
+It extracts only the fields we need and composes container/blob URLs suitable
+for the newer `azure_storage_blob` crate (>= 0.7).


Aren't there any helper functions present in the legacy crates that can help here? While I don't see anything necessarily wrong with this file it'd be best if we avoid adding custom parsers and delegate this to external library code if possible. If not possible it's fine too :)

thomasqueirozb · 2026-01-09T20:58:04Z

src/sinks/azure_common/service.rs

+            result.map(|_resp| AzureBlobResponse {
+                inner: (),


This doesn't look correct... OTOH I cannot find anywhere where inner is used (even in the current code base)...

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Nov 17, 2025

joshcoughlan marked this pull request as ready for review November 17, 2025 18:18

joshcoughlan requested a review from a team as a code owner November 17, 2025 18:18

joshcoughlan mentioned this pull request Nov 17, 2025

azure_blob proxy support #21314

Open

thomasqueirozb reviewed Nov 17, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

src/sinks/azure_common/shared_key_policy.rs Show resolved Hide resolved

pront added provider: azure Anything `azure` service provider related sink: azure_blob Anything `azure_blob` sink related labels Nov 17, 2025

thomasqueirozb added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Nov 18, 2025

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Nov 21, 2025

This was referenced Dec 10, 2025

feat(azure_blob sink): Add proxy support #24256

Open

feat(azure_logs_ingestion sink): Initial azure_logs_ingestion sink #22912

Open

github-actions bot removed the meta: awaiting author Pull requests that are awaiting their author. label Jan 7, 2026

joshcoughlan and others added 4 commits January 7, 2026 15:31

Update Azure crates and migrate to the new SDK

a86bbbb

add a reference comment for empty headers

62136c4

Update Cargo.toml

1484192

Co-authored-by: Thomas <[email protected]>

Fix failing tests

7785983

joshcoughlan force-pushed the update_azure_crates branch from 3cd1ced to 7785983 Compare January 7, 2026 22:57

joshcoughlan changed the title ~~feat(azure_blob): Update crates and migrate to the new SDK~~ feat(azure_blob sink): Update crates and migrate to the new SDK Jan 8, 2026

thomasqueirozb reviewed Jan 9, 2026

View reviewed changes

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Jan 9, 2026

	azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"] }
	azure_core = { version = "0.30", default-features = false, features = ["reqwest", "hmac_openssl"], optional = true }

	sinks-azure_blob = ["dep:azure_storage_blob"]
	sinks-azure_blob = ["dep:azure_core", "dep:azure_storage_blob"]

	let (content_type, _content_encoding, blob_lines) = config.get_blob(blobs[0].clone()).await;
	let (content_type, content_encoding, blob_lines) = config.get_blob(blobs[0].clone()).await;
	assert_eq!(content_encoding, None);

feat(azure_blob sink): Update crates and migrate to the new SDK #24255

Are you sure you want to change the base?

feat(azure_blob sink): Update crates and migrate to the new SDK #24255

Uh oh!

Conversation

joshcoughlan commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

thomasqueirozb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joshcoughlan commented Nov 17, 2025

Uh oh!

thomasqueirozb commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fplantinga-guida commented Jan 6, 2026

Uh oh!

joshcoughlan commented Jan 7, 2026

Uh oh!

github-actions bot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshcoughlan commented Jan 8, 2026

Uh oh!

joshcoughlan commented Jan 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshcoughlan commented Nov 17, 2025 •

edited

Loading

thomasqueirozb commented Nov 21, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2026 •

edited

Loading