.Net: Bug: Azure OpenAI client fails with 401 when throttling

When using AzureOpenAITextEmbeddingGenerationService SK client and sending too many requests, the Azure service throttling leads to a "401 Unauthorized" error instead of "429 Too Many Requests".

SK version: 1.20.0
.NET version: 8.0.401

I inspected the HTTP requests and noticed that the internal retry logic is sending a malformed request during retries. It appears to be adding the Authorization header twice (with the same token), e.g.

Request 1 .. 7: ok
Request 8: fails with HTTP Status Code 429
Request 8 is retried with two auth tokens: fails with HTTP Status Code 401

Code to repro:

public static class Program
{
    public static async Task Main()
    {
        var client = new AzureOpenAITextEmbeddingGenerationService(
            endpoint: "https://....openai.azure.com/",
            deploymentName: "text-embedding-ada-002",
            credential: new DefaultAzureCredential()
        );

        for (int i = 0; i < 200; i++)
        {
            Console.WriteLine($"## {i}");
            await client.GenerateEmbeddingsAsync([RndStr(), RndStr(), RndStr(), RndStr(), RndStr(), RndStr(), RndStr()]);
        }
    }

    private static string RndStr()
    {
        var random = new Random();
        return new(Enumerable.Repeat("ABCDEF GHIJKLMNOP QRSTUVW XYZabcdefgh ijklmnopqrst uvwxyz0123 456789", 8000)
            .Select(s => s[random.Next(s.Length)]).ToArray());
    }
}

Output:

## 0
## 1
## 2
## 3
## 4
## 5
## 6
## 7
## 8
Unhandled exception. Microsoft.SemanticKernel.HttpOperationException: Service request failed.
Status: 401 (Unauthorized)

 ---> System.ClientModel.ClientResultException: Service request failed.
Status: 401 (Unauthorized)

   at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at Azure.AI.OpenAI.Embeddings.AzureEmbeddingClient.GenerateEmbeddingsAsync(BinaryContent content, RequestOptions options)
   at OpenAI.Embeddings.EmbeddingClient.GenerateEmbeddingsAsync(IEnumerable`1 inputs, EmbeddingGenerationOptions options, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)
   --- End of inner exception stack trace ---
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.RunRequestAsync[T](Func`1 request)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetEmbeddingsAsync(String targetModel, IList`1 data, Kernel kernel, Nullable`1 dimensions, CancellationToken cancellationToken)
   at Program.Main() in Program.cs:line 21
   at Program.<Main>()

looks like a bug in the underlying Azure.AI.OpenAI 2.0.0-beta.5, I reported it here Azure/azure-sdk-for-net#46109 including a workaround

Roger Barreto (@rogerbarreto)

Roger Barreto (@rogerbarreto) I'm assigning this to you, can you monitor the issue Devis created an pick up the fix when it becomes available.

The original issue was closed, setting this as complete as the original issue is addressed and will be available automatically to Semantic Kernel package in a future release.

[BUG] Azure OpenAI client fails with 401 when throttling Azure/azure-sdk-for-net#46109

Roger Barreto (@rogerbarreto)

Roger Barreto (@rogerbarreto) the outstanding open issue here is the reliance on a pre-release package with a noticeable (quite annoying) bug that impacts all SK consumers.

Rather than waiting for the Azure SDK fix to propagate to SK and SK fix to propagate to consumers, I'd suggest that SK includes a workaround as soon as possible, e.g. by injecting a delegating handler:

internal sealed class AuthFixHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        if (request.Headers.TryGetValues("Authorization", out var headers) && headers.Count() > 1)
        {
            request.Headers.Authorization = new AuthenticationHeaderValue(
                request.Headers.Authorization!.Scheme,
                request.Headers.Authorization.Parameter);
        }

        return base.SendAsync(request, cancellationToken);
    }
}

.Net: Azure OpenAI Temporary Fix [401 Retry Bug] (#9465)

Upgrade SK (incl. Azure OpenAI throttling fix) and OllamaSharp (#874)

Fix exception occurring when Azure OpenAI service is throttling reque…

Navigation Menu

Repository navigation

.Net: Bug: Azure OpenAI client fails with 401 when throttling #8929

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

.Net: Bug: Azure OpenAI client fails with 401 when throttling #8929

Description

Activity

dluc commented on Sep 22, 2024

markwallace-microsoft commented on Sep 23, 2024

rogerbarreto commented on Oct 7, 2024

dluc commented on Oct 22, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions