Skip to content

[Debugger] Fix non-deterministic failure in UDS Integration Test#3168

Merged
shurivich merged 1 commit into
masterfrom
shurivich/try_fix_uds
Sep 6, 2022
Merged

[Debugger] Fix non-deterministic failure in UDS Integration Test#3168
shurivich merged 1 commit into
masterfrom
shurivich/try_fix_uds

Conversation

@shurivich

@shurivich shurivich commented Sep 5, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary of changes

Dispose the MockAgent
Set ShouldDeserializeTraces to the default (true)

Reason for change

Try fix non deterministic integration test failures from MethodProbeTest_UDS
Issue happens only on Windows with exception:

[ERR] An error occurred while sending data to the agent at C:/Users/AzDevOps/AppData/Local/Temp/s1hbabdt.ata to http://localhost/v0.4/traces System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at Datadog.Trace.HttpOverStreams.DatadogHttpClient.<>c__DisplayClass7_0.<<ReadResponseAsync>g__SkipUntil|1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Datadog.Trace.HttpOverStreams.DatadogHttpClient.ReadResponseAsync(Stream responseStream)
   at Datadog.Trace.HttpOverStreams.DatadogHttpClient.SendAsync(HttpRequest request, Stream requestStream, Stream responseStream)
   at Datadog.Trace.Agent.Transports.HttpStreamRequest.SendAsync(String verb, String contentType, IHttpContent content)
   at Datadog.Trace.Agent.Transports.HttpStreamRequest.PostAsync(ArraySegment`1 bytes, String contentType)
   at Datadog.Trace.Agent.Api.SendTracesAsyncImpl(IApiRequest request, Boolean finalTry, SendTracesState state)
   at Datadog.Trace.Agent.Api.SendWithRetry[T](Uri endpoint, SendCallback`1 callback, T state)

It could happen, because we don't dispose MockAgent

@shurivich
shurivich requested a review from a team as a code owner September 5, 2022 10:31
@github-actions github-actions Bot added the area:tests unit tests, integration tests label Sep 5, 2022
andrewlock
andrewlock approved these changes Sep 5, 2022
View reviewed changes

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🤞

@andrewlock

andrewlock commented Sep 5, 2022

Copy link
Copy Markdown
Member

For anyone else reviewing, this basically does two things:

  • Disposes the MockAgent
  • Sets ShouldDeserializeTraces to the default (true)

@OmerRaviv OmerRaviv changed the title Try Fix UDS [Debugger] Fix non-deterministic failure in UDS Integration Test Sep 5, 2022
OmerRaviv
OmerRaviv approved these changes Sep 5, 2022
View reviewed changes

@OmerRaviv OmerRaviv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewlock

andrewlock commented Sep 5, 2022

Copy link
Copy Markdown
Member

Benchmarks Report 🐌

Benchmarks for #3168 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 711μs 737ns 2.86μs 0.357 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 460μs 294ns 1.14μs 0 0 0 2.59 KB
#3168 WriteAndFlushEnrichedTraces net472 716μs 503ns 1.95μs 0.353 0 0 3.18 KB
#3168 WriteAndFlushEnrichedTraces netcoreapp3.1 462μs 549ns 2.13μs 0 0 0 2.59 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 183ns 0.258ns 1ns 0.0676 9.14E-05 0 425 B
master AllCycleSimpleBody netcoreapp3.1 236ns 0.322ns 1.25ns 0.00584 0 0 424 B
master AllCycleMoreComplexBody net472 183ns 0.11ns 0.424ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 234ns 0.188ns 0.704ns 0.00549 0 0 400 B
master BodyExtractorSimpleBody net472 251ns 0.327ns 1.22ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 222ns 0.399ns 1.44ns 0.00368 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.6μs 8.71ns 32.6ns 1.21 0.0217 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.4μs 24.2ns 90.5ns 0.0922 0 0 6.75 KB
#3168 AllCycleSimpleBody net472 183ns 0.212ns 0.819ns 0.0676 0 0 425 B
#3168 AllCycleSimpleBody netcoreapp3.1 238ns 0.28ns 1.05ns 0.00583 0 0 424 B
#3168 AllCycleMoreComplexBody net472 183ns 0.122ns 0.438ns 0.0637 0 0 401 B
#3168 AllCycleMoreComplexBody netcoreapp3.1 233ns 0.359ns 1.39ns 0.00545 0 0 400 B
#3168 BodyExtractorSimpleBody net472 275ns 0.538ns 2.01ns 0.0574 0 0 361 B
#3168 BodyExtractorSimpleBody netcoreapp3.1 226ns 0.194ns 0.728ns 0.00374 0 0 272 B
#3168 BodyExtractorMoreComplexBody net472 14.3μs 11.3ns 42.2ns 1.21 0.0215 0 7.62 KB
#3168 BodyExtractorMoreComplexBody netcoreapp3.1 12μs 19.1ns 74.1ns 0.0897 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 182μs 245ns 950ns 0.184 0 0 20.57 KB
#3168 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3168 SendRequest netcoreapp3.1 183μs 250ns 968ns 0.184 0 0 20.57 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.91μs 0.849ns 3.18ns 0.15 0.000949 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.52μs 0.739ns 2.86ns 0.0129 0 0 936 B
#3168 ExecuteNonQuery net472 1.91μs 0.778ns 3.01ns 0.15 0.000955 0 947 B
#3168 ExecuteNonQuery netcoreapp3.1 1.44μs 1.01ns 3.78ns 0.0129 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.47μs 0.662ns 2.48ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.55μs 0.282ns 1.02ns 0.0147 0 0 1.1 KB
master CallElasticsearchAsync net472 2.66μs 0.809ns 3.13ns 0.205 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.58μs 0.351ns 1.27ns 0.0166 0 0 1.22 KB
#3168 CallElasticsearch net472 2.54μs 0.828ns 2.99ns 0.183 0 0 1.16 KB
#3168 CallElasticsearch netcoreapp3.1 1.58μs 0.529ns 1.98ns 0.0149 0 0 1.1 KB
#3168 CallElasticsearchAsync net472 2.73μs 1.33ns 5.16ns 0.205 0 0 1.29 KB
#3168 CallElasticsearchAsync netcoreapp3.1 1.62μs 0.441ns 1.71ns 0.0162 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.64μs 6.43ns 24.1ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.76μs 3.9ns 15.1ns 0.0177 0 0 1.34 KB
#3168 ExecuteAsync net472 2.72μs 6.1ns 23.6ns 0.225 0 0 1.42 KB
#3168 ExecuteAsync netcoreapp3.1 1.73μs 4.36ns 16.9ns 0.018 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.78μs 12.5ns 48.5ns 0.439 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.56μs 7.26ns 27.1ns 0.0349 0 0 2.6 KB
#3168 SendAsync net472 5.64μs 16.8ns 65.2ns 0.44 0 0 2.77 KB
#3168 SendAsync netcoreapp3.1 3.67μs 8.81ns 34.1ns 0.0351 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.16μs 2.13ns 7.95ns 0.288 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.63μs 0.779ns 3.02ns 0.0251 0 0 1.85 KB
#3168 EnrichedLog net472 3.26μs 1.22ns 4.73ns 0.287 0 0 1.81 KB
#3168 EnrichedLog netcoreapp3.1 2.56μs 1.18ns 4.4ns 0.0244 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 153μs 94.1ns 365ns 0.691 0.23 0 4.65 KB
master EnrichedLog netcoreapp3.1 117μs 198ns 767ns 0.058 0 0 4.49 KB
#3168 EnrichedLog net472 149μs 180ns 696ns 0.673 0.224 0 4.65 KB
#3168 EnrichedLog netcoreapp3.1 117μs 63.2ns 245ns 0.0583 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.6μs 21.2ns 82ns 0.568 0.00276 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.37μs 9.07ns 32.7ns 0.0542 0 0 3.91 KB
#3168 EnrichedLog net472 5.49μs 16.3ns 60.9ns 0.569 0.00273 0 3.59 KB
#3168 EnrichedLog netcoreapp3.1 4.44μs 11.8ns 45.8ns 0.055 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.35μs 1.26ns 4.36ns 0.217 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.79μs 0.643ns 2.49ns 0.0179 0 0 1.32 KB
#3168 SendReceive net472 2.22μs 1.3ns 5.02ns 0.218 0 0 1.37 KB
#3168 SendReceive netcoreapp3.1 1.81μs 0.855ns 3.2ns 0.0173 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.98μs 2.01ns 7.77ns 0.353 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.31μs 2.74ns 10.6ns 0.0237 0 0 1.8 KB
#3168 EnrichedLog net472 4.99μs 2.73ns 10.6ns 0.354 0 0 2.23 KB
#3168 EnrichedLog netcoreapp3.1 4.37μs 2.06ns 7.7ns 0.024 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.13μs 0.34ns 1.27ns 0.128 0 0 810 B
master StartFinishSpan netcoreapp3.1 966ns 1.19ns 4.29ns 0.0101 0 0 760 B
master StartFinishScope net472 1.43μs 1.42ns 5.5ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.1μs 0.468ns 1.75ns 0.0121 0 0 880 B
#3168 StartFinishSpan net472 1.14μs 0.569ns 2.13ns 0.129 0 0 810 B
#3168 StartFinishSpan netcoreapp3.1 962ns 0.509ns 1.97ns 0.0101 0 0 760 B
#3168 StartFinishScope net472 1.36μs 0.548ns 2.12ns 0.141 0 0 891 B
#3168 StartFinishScope netcoreapp3.1 1.09μs 0.346ns 1.29ns 0.0119 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.6μs 0.604ns 2.34ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.18μs 0.703ns 2.63ns 0.0118 0 0 880 B
#3168 RunOnMethodBegin net472 1.55μs 0.525ns 2.03ns 0.141 0 0 891 B
#3168 RunOnMethodBegin netcoreapp3.1 1.25μs 0.56ns 2.1ns 0.0118 0 0 880 B

@shurivich shurivich self-assigned this Sep 6, 2022
@shurivich
shurivich merged commit 094dbd3 into master Sep 6, 2022
@shurivich
shurivich deleted the shurivich/try_fix_uds branch September 6, 2022 08:27
@github-actions github-actions Bot added this to the vNext milestone Sep 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robertpi robertpi robertpi approved these changes
@OmerRaviv OmerRaviv OmerRaviv approved these changes
@andrewlock andrewlock andrewlock approved these changes

Assignees

@shurivich shurivich

Labels

area:debugger area:tests unit tests, integration tests

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

4 participants

@shurivich @andrewlock @robertpi @OmerRaviv