Repository navigation
Closed
Labels
ClientThis issue points to a problem in the data-plane of the library.This issue points to a problem in the data-plane of the library.OpenAIService AttentionWorkflow: This issue is responsible by Azure service team.Workflow: This issue is responsible by Azure service team.customer-reportedIssues that are reported by GitHub users external to the Azure organization.Issues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as thatThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
Description
Library name and version
Azure.AI.OpenAI 2.0.0-beta.5
Describe the bug
When using AzureOpenAIClient and sending too many requests, the Azure service throttling leads to a "401 Unauthorized" error instead of "429 Too Many Requests".
Looking at the internal requests, looks like the code is retrying on 429 as expected, sending a malformed request containing the Authorization header twice (with the same token).
Expected behavior
The client should keep retrying on 429 and/or fail with a HTTP exception status code 429
Actual behavior
The client fails with a HTTP exception status code 401
Reproduction Steps
using Azure.AI.OpenAI;
using Azure.Identity;
using OpenAI.Embeddings;
public static class Program
{
public static async Task Main()
{
AzureOpenAIClient openAIClient = new(
endpoint: new Uri("https://....openai.azure.com/"),
credential: new DefaultAzureCredential());
var embeddingClient = openAIClient.GetEmbeddingClient("text-embedding-ada-002");
for (int i = 0; i < 200; i++)
{
Console.WriteLine($"## {i}");
await embeddingClient.GenerateEmbeddingsAsync([RndStr(), RndStr(), RndStr(), RndStr(), RndStr(), RndStr(), RndStr()]);
}
}
public static string RndStr()
{
var random = new Random();
return new(Enumerable.Repeat(" ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789 ", 8000)
.Select(s => s[random.Next(s.Length)]).ToArray());
}
}Output:
## 0
## 1
## 2
## 3
## 4
## 5
## 6
## 7
## 8
Unhandled exception. System.ClientModel.ClientResultException: Service request failed.
Status: 401 (Unauthorized)
at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
at Azure.AI.OpenAI.Embeddings.AzureEmbeddingClient.GenerateEmbeddingsAsync(BinaryContent content, RequestOptions options)
at OpenAI.Embeddings.EmbeddingClient.GenerateEmbeddingsAsync(IEnumerable`1 inputs, EmbeddingGenerationOptions options, CancellationToken cancellationToken)
at Program.Main() in Program.cs:line 38
at Program.<Main>()
Environment
.NET SDK:
Version: 8.0.401
Commit: 811edcc344
Workload version: 8.0.400-manifests.56cd0383
MSBuild version: 17.11.4+37eb419ad
Runtime Environment:
OS Name: Mac OS X
OS Version: 15.0
OS Platform: Darwin
RID: osx-arm64
Base Path: /usr/local/share/dotnet/sdk/8.0.401
Metadata
Metadata
Assignees
Labels
ClientThis issue points to a problem in the data-plane of the library.This issue points to a problem in the data-plane of the library.OpenAIService AttentionWorkflow: This issue is responsible by Azure service team.Workflow: This issue is responsible by Azure service team.customer-reportedIssues that are reported by GitHub users external to the Azure organization.Issues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as thatThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
Activity
github-actions commentedon Sep 21, 2024
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc Jose Alvarez (@jpalvarezl) Ralph (@ralph-msft) Travis Wilson (@trrwilson).
dluc commentedon Sep 22, 2024
FYI, I'm using this handler as a workaround:
alexmg commentedon Sep 26, 2024
A
PipelinePolicybased workaround for anyone that may be interested.This can be added to the policies on the
AzureOpenAIClientOptionswith thePipelinePosition.PerTryposition.mikeharder commentedon Oct 2, 2024
Suspected root cause on L56:
azure-sdk-for-net/sdk/openai/Azure.AI.OpenAI/src/Custom/AzureTokenAuthenticationPolicy.cs
Lines 46 to 56 in 3a55d32
I believe the code should use
Headers.SetValue()instead ofHeaders.Add()to avoid duplicate headers, since the sameHttpMessageis used on retries.Above code was recently modified in this PR: #44983
alexmg commentedon Oct 23, 2024
Is there an expected ETA for a release that contains this fix?
Being an upstream dependency of many other projects like Semantic Kernal and Kernel Memory this is causing issues for many downstream consumers.
annelo-msft commentedon Oct 23, 2024
Travis Wilson (@trrwilson), would you be able to share plans regarding the next release of the Azure.AI.OpenAI library?
Upgrade SK (incl. Azure OpenAI throttling fix) and OllamaSharp (#874)
Fix exception occurring when Azure OpenAI service is throttling reque…