Skip to content

Handle HTTPClient Timeout (TaskCancelledException) in Arcade Helix SDKs #2826

Open
Open
Handle HTTPClient Timeout (TaskCancelledException) in Arcade Helix SDKs#2826
Milestone
Arcade SDK
@MattGal

Description

@MattGal
MattGal
opened on May 18, 2019

Please resolve as dupe if this already exists. The problem may actually be in Microsoft.Rest.RetryDelegatingHandler (it's pretty common to catch all HttpRequestExceptions and ignore the TaskCanceleled ones)

Seen in the logs of this run:
https://dev.azure.com/dnceng/public/_build/results?buildId=192431&view=logs

2019-05-17T20:30:56.7945468Z   Stopping Azure Pipelines Test Run Linux arm64 Checked @ (Alpine.38.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.8-helix-arm64v8-46e69dd-20190313223330
2019-05-17T20:33:39.2480228Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : TaskCanceledException: The operation was canceled. [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2484166Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2487095Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2489735Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2492262Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2494814Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.Rest.RetryAfterDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2497670Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.Rest.RetryDelegatingHandler.<>c__DisplayClass11_0.<<SendAsync>b__1>d.MoveNext() [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2500622Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : --- End of stack trace from previous location where exception was thrown --- [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2503106Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2505546Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2508106Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.DotNet.Helix.Client.Aggregate.JobSummaryInternalAsync(IImmutableList`1 groupBy, Int32 maxResultSets, String filterBuild, String filterCreator, String filterName, String filterSource, String filterType, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs:line 581 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2511225Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.DotNet.Helix.Client.Aggregate.JobSummaryAsync(IImmutableList`1 groupBy, Int32 maxResultSets, String filterBuild, String filterCreator, String filterName, String filterSource, String filterType, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs:line 493 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2514621Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.MissionControlTestProcessingDoneAsync(String jobName) in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 68 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2516412Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.CheckHelixJobAsync(String jobName) in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 56 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2517757Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.ExecuteCore() in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 28 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2519034Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 43 [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2520604Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : IOException: Unable to read data from the transport connection: Operation canceled. [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2521656Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2545316Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2546747Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2548324Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2550044Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error :    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) [/__w/1/s/tests/helixpublishwitharcade.proj]
2019-05-17T20:33:39.2551213Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/2.0.0-beta.19257.7/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : SocketException: Operation canceled [/__w/1/s/tests/helixpublishwitharcade.proj]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions