Skip to content

Bump timeouts in DiscoveryServiceTests to reduce flake#3233

Merged
andrewlock merged 3 commits into
masterfrom
andrew/reduce-flake
Sep 22, 2022
Merged

Bump timeouts in DiscoveryServiceTests to reduce flake#3233
andrewlock merged 3 commits into
masterfrom
andrew/reduce-flake

Conversation

@andrewlock

@andrewlock andrewlock commented Sep 20, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

Summary of changes

  • Bump timeouts in DiscoveryServiceTests
  • Reduce minimum expected retries when faulty API.

Reason for change

Noticed some flake in unit tests, so go more conservative.

Implementation details

10s timeout -> 30s timeout
expect 5-6 checks -> expect 3-6 checks

Test coverage

N/A

Other details

N/A

@andrewlock
Bump timeouts in DiscoveryServiceTests
0360478 
Noticed some flake, so go more conservative
@andrewlock andrewlock added the area:tests unit tests, integration tests label Sep 20, 2022
@andrewlock
andrewlock requested a review from a team as a code owner September 20, 2022 10:59
@andrewlock andrewlock changed the title Bump timeouts in DiscoveryServiceTests Bump timeouts in DiscoveryServiceTests to reduce flake Sep 20, 2022
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
zacharycmontoya
zacharycmontoya reviewed Sep 20, 2022
View reviewed changes
Comment thread tracer/test/Datadog.Trace.Tests/Agent/DiscoveryServiceTests.cs Outdated
await ds.DisposeAsync();
// add some leeway in case of slowness
factory.RequestsSent.Count.Should().BeInRange(5, 6, "Should make at most 6 retries in 13s");
factory.RequestsSent.Count.Should().BeInRange(3, 6, "Should make at most 6 retries in 13s");

@zacharycmontoya zacharycmontoya Sep 20, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the comment should be, but I think it also needs to be updated

@andrewlock

andrewlock commented Sep 21, 2022

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #3233 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 713μs 386ns 1.44μs 0.355 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 461μs 156ns 582ns 0 0 0 2.58 KB
#3233 WriteAndFlushEnrichedTraces net472 714μs 618ns 2.39μs 0.357 0 0 3.18 KB
#3233 WriteAndFlushEnrichedTraces netcoreapp3.1 463μs 583ns 2.26μs 0 0 0 2.59 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 191ns 0.16ns 0.578ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 235ns 0.266ns 0.994ns 0.00587 0 0 424 B
master AllCycleMoreComplexBody net472 180ns 0.159ns 0.614ns 0.0638 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 236ns 0.217ns 0.841ns 0.00545 0 0 400 B
master BodyExtractorSimpleBody net472 259ns 0.321ns 1.2ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 219ns 0.17ns 0.658ns 0.00374 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.7μs 11.4ns 42.5ns 1.2 0.0219 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 10.8ns 40.4ns 0.0894 0 0 6.75 KB
#3233 AllCycleSimpleBody net472 191ns 0.166ns 0.644ns 0.0675 0 0 425 B
#3233 AllCycleSimpleBody netcoreapp3.1 241ns 0.226ns 0.815ns 0.00586 0 0 424 B
#3233 AllCycleMoreComplexBody net472 181ns 0.104ns 0.387ns 0.0637 0 0 401 B
#3233 AllCycleMoreComplexBody netcoreapp3.1 234ns 0.245ns 0.95ns 0.0055 0 0 400 B
#3233 BodyExtractorSimpleBody net472 272ns 0.193ns 0.697ns 0.0574 0 0 361 B
#3233 BodyExtractorSimpleBody netcoreapp3.1 213ns 0.309ns 1.2ns 0.00365 0 0 272 B
#3233 BodyExtractorMoreComplexBody net472 14.4μs 11.2ns 41.7ns 1.21 0.0216 0 7.62 KB
#3233 BodyExtractorMoreComplexBody netcoreapp3.1 12μs 17.3ns 67.1ns 0.0918 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 179μs 134ns 517ns 0.268 0 0 20.6 KB
#3233 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3233 SendRequest netcoreapp3.1 179μs 140ns 544ns 0.269 0 0 20.58 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.86μs 1.05ns 4.08ns 0.15 0.000939 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.39μs 0.796ns 2.98ns 0.0127 0 0 936 B
#3233 ExecuteNonQuery net472 1.87μs 1.76ns 6.8ns 0.15 0.000936 0 947 B
#3233 ExecuteNonQuery netcoreapp3.1 1.37μs 0.531ns 1.99ns 0.0123 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.59μs 2.43ns 9.43ns 0.184 0.00129 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.53μs 0.414ns 1.49ns 0.0145 0 0 1.1 KB
master CallElasticsearchAsync net472 2.66μs 2.09ns 8.11ns 0.205 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.735ns 2.65ns 0.0164 0 0 1.22 KB
#3233 CallElasticsearch net472 2.59μs 1.65ns 6.16ns 0.183 0 0 1.16 KB
#3233 CallElasticsearch netcoreapp3.1 1.59μs 0.887ns 3.32ns 0.0152 0 0 1.1 KB
#3233 CallElasticsearchAsync net472 2.78μs 2.54ns 9.82ns 0.205 0 0 1.29 KB
#3233 CallElasticsearchAsync netcoreapp3.1 1.58μs 0.8ns 2.99ns 0.0166 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.87μs 4.48ns 17.3ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.8μs 0.769ns 2.98ns 0.018 0 0 1.34 KB
#3233 ExecuteAsync net472 2.84μs 1.89ns 7.32ns 0.224 0 0 1.42 KB
#3233 ExecuteAsync netcoreapp3.1 1.71μs 0.542ns 1.88ns 0.018 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.9μs 2.99ns 11.2ns 0.44 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.77μs 1.82ns 7.05ns 0.0358 0 0 2.6 KB
#3233 SendAsync net472 5.81μs 3.36ns 13ns 0.439 0 0 2.77 KB
#3233 SendAsync netcoreapp3.1 3.55μs 9ns 34.9ns 0.0342 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.25μs 15.5ns 59.8ns 0.287 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.48μs 13.5ns 109ns 0.0242 0 0 1.85 KB
#3233 EnrichedLog net472 3.24μs 2.29ns 8.58ns 0.287 0 0 1.81 KB
#3233 EnrichedLog netcoreapp3.1 2.48μs 0.871ns 3.26ns 0.0249 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 152μs 199ns 770ns 0.682 0.227 0 4.65 KB
master EnrichedLog netcoreapp3.1 119μs 138ns 535ns 0 0 0 4.49 KB
#3233 EnrichedLog net472 153μs 247ns 957ns 0.687 0.229 0 4.66 KB
#3233 EnrichedLog netcoreapp3.1 117μs 227ns 880ns 0.0589 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.73μs 12.3ns 47.6ns 0.57 0.00292 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.35μs 1.73ns 6.48ns 0.0531 0 0 3.91 KB
#3233 EnrichedLog net472 5.86μs 7.22ns 27ns 0.568 0.00291 0 3.59 KB
#3233 EnrichedLog netcoreapp3.1 4.64μs 2.32ns 8.68ns 0.0531 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.23μs 2.36ns 9.15ns 0.217 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.82μs 0.922ns 3.57ns 0.0174 0 0 1.32 KB
#3233 SendReceive net472 2.2μs 0.9ns 3.37ns 0.218 0.0011 0 1.37 KB
#3233 SendReceive netcoreapp3.1 1.76μs 1.11ns 4.14ns 0.0176 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.13μs 1.7ns 6.37ns 0.353 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.33μs 1.36ns 5.08ns 0.0239 0 0 1.8 KB
#3233 EnrichedLog net472 5.01μs 2.33ns 8.73ns 0.354 0 0 2.23 KB
#3233 EnrichedLog netcoreapp3.1 4.48μs 1.06ns 3.81ns 0.0224 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.24μs 0.732ns 2.74ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 966ns 1.16ns 4.18ns 0.01 0 0 760 B
master StartFinishScope net472 1.41μs 1.06ns 4.12ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.05μs 0.383ns 1.38ns 0.0119 0 0 880 B
#3233 StartFinishSpan net472 1.22μs 0.955ns 3.7ns 0.128 0 0 810 B
#3233 StartFinishSpan netcoreapp3.1 913ns 0.304ns 1.14ns 0.0101 0 0 760 B
#3233 StartFinishScope net472 1.4μs 0.888ns 3.44ns 0.141 0 0 891 B
#3233 StartFinishScope netcoreapp3.1 1.1μs 0.356ns 1.33ns 0.012 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.51μs 1.08ns 4.17ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.16μs 0.387ns 1.45ns 0.0122 0 0 880 B
#3233 RunOnMethodBegin net472 1.5μs 0.974ns 3.77ns 0.141 0 0 891 B
#3233 RunOnMethodBegin netcoreapp3.1 1.15μs 0.41ns 1.48ns 0.0115 0 0 880 B

@andrewlock

andrewlock commented Sep 21, 2022

Copy link
Copy Markdown
Member Author

Code Coverage Report 📊

⛔ Merging #3233 into master will will decrease line coverage by 6%
⚠️ Merging #3233 into master will will decrease branch coverage by 4%
⛔ Merging #3233 into master will will increase complexity by 14

master #3233 Change
Lines 17963 / 24889 16543 / 24903
Lines % 72% 66% -6% ⛔
Branches 10472 / 15254 9863 / 15270
Branches % 69% 65% -4% ⚠️
Complexity 16477 16491 14 ⛔

View the full report for further details:

Datadog.Trace Breakdown ⛔

master #3233 Change
Lines % 72% 66% -6% ⛔
Branches % 69% 65% -4% ⚠️
Complexity 16477 16491 14 ⛔

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Agent.TraceSamplers.AnalyticsEventsSampler -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Agent.TraceSamplers.ErrorSampler -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.PInvoke.NativeMethodProbeDefinition -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.Sink.SnapshotSink -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.Snapshots.LazySnapshotSerializerFieldsAndPropsSelector -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.Snapshots.DebuggerSnapshotCreator -100% ⛔ -68% ⛔ 1 ⛔
BoundLineProbeLocation -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Agent.TraceSamplers.PrioritySampler -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.CapturedLines -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.Instrumentation.DebuggerReturn -100% ⛔ 0% ✔️ 0 ✔️
...And 65 more

View the full reports for further details:

@andrewlock
andrewlock merged commit 94b2d1d into master Sep 22, 2022
@andrewlock
andrewlock deleted the andrew/reduce-flake branch September 22, 2022 08:45
@github-actions github-actions Bot added this to the vNext milestone Sep 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zacharycmontoya zacharycmontoya zacharycmontoya approved these changes
@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:tests unit tests, integration tests

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewlock @zacharycmontoya @pierotibou