Skip to content

Add debug log in Api when sending to agent fails#3180

Merged
andrewlock merged 2 commits into
masterfrom
andrew/add-debug-log-to-api-send
Sep 7, 2022
Merged

Add debug log in Api when sending to agent fails#3180
andrewlock merged 2 commits into
masterfrom
andrew/add-debug-log-to-api-send

Conversation

@andrewlock

@andrewlock andrewlock commented Sep 7, 2022

Copy link
Copy Markdown
Member

Summary of changes

  • Add a debug log when we fail to send to the agent, but we will retry.
  • Enable debug logs for aspnetcore tests

Reason for change

We have some flakiness in the aspnetcore tests, which we think may be related to the MockTracerAgent, but can't figure out why. This is just to try to help with debugging

Implementation details

Add a debug log if not final try (previously we were only logging if it's final try).

Test coverage

N/A

Other details

Trying to understand failures like this one where we receive the same span twice.

@andrewlock andrewlock added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Sep 7, 2022
@andrewlock
andrewlock requested a review from a team as a code owner September 7, 2022 08:49
@andrewlock

andrewlock commented Sep 7, 2022

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #3180 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 736μs 205ns 767ns 0.368 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 462μs 178ns 642ns 0 0 0 2.58 KB
#3180 WriteAndFlushEnrichedTraces net472 715μs 354ns 1.33μs 0.355 0 0 3.18 KB
#3180 WriteAndFlushEnrichedTraces netcoreapp3.1 464μs 212ns 764ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 184ns 0.152ns 0.57ns 0.0676 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 240ns 0.257ns 0.961ns 0.00574 0 0 424 B
master AllCycleMoreComplexBody net472 184ns 0.175ns 0.655ns 0.0638 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 246ns 0.403ns 1.56ns 0.00539 0 0 400 B
master BodyExtractorSimpleBody net472 257ns 0.369ns 1.33ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 227ns 0.39ns 1.51ns 0.00372 0 0 272 B
master BodyExtractorMoreComplexBody net472 15μs 11.9ns 44.6ns 1.21 0.015 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.8μs 15.7ns 61ns 0.088 0 0 6.75 KB
#3180 AllCycleSimpleBody net472 204ns 0.422ns 1.63ns 0.0676 0 0 425 B
#3180 AllCycleSimpleBody netcoreapp3.1 235ns 0.3ns 1.16ns 0.0058 0 0 424 B
#3180 AllCycleMoreComplexBody net472 189ns 0.103ns 0.358ns 0.0637 0 0 401 B
#3180 AllCycleMoreComplexBody netcoreapp3.1 232ns 0.278ns 1.04ns 0.00536 0 0 400 B
#3180 BodyExtractorSimpleBody net472 258ns 0.217ns 0.84ns 0.0574 0 0 361 B
#3180 BodyExtractorSimpleBody netcoreapp3.1 231ns 0.446ns 1.73ns 0.00372 0 0 272 B
#3180 BodyExtractorMoreComplexBody net472 15μs 37.1ns 144ns 1.21 0.015 0 7.62 KB
#3180 BodyExtractorMoreComplexBody netcoreapp3.1 11.8μs 12.3ns 47.6ns 0.0888 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 181μs 117ns 451ns 0.27 0 0 20.57 KB
#3180 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3180 SendRequest netcoreapp3.1 179μs 154ns 598ns 0.268 0 0 20.57 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.87μs 0.659ns 2.55ns 0.15 0.000933 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.37μs 0.871ns 3.26ns 0.0123 0 0 936 B
#3180 ExecuteNonQuery net472 1.87μs 0.483ns 1.87ns 0.15 0.000934 0 947 B
#3180 ExecuteNonQuery netcoreapp3.1 1.48μs 0.338ns 1.27ns 0.0125 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.61μs 0.914ns 3.42ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.62μs 0.921ns 3.45ns 0.0146 0 0 1.1 KB
master CallElasticsearchAsync net472 2.65μs 9.11ns 37.5ns 0.204 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.577ns 2.23ns 0.0167 0 0 1.22 KB
#3180 CallElasticsearch net472 2.63μs 1.02ns 3.82ns 0.183 0 0 1.16 KB
#3180 CallElasticsearch netcoreapp3.1 1.57μs 0.684ns 2.56ns 0.0149 0 0 1.1 KB
#3180 CallElasticsearchAsync net472 2.68μs 0.846ns 3.28ns 0.204 0 0 1.29 KB
#3180 CallElasticsearchAsync netcoreapp3.1 1.67μs 1.09ns 4.21ns 0.0167 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.56μs 4.41ns 17.1ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.72μs 4.16ns 16.1ns 0.0187 0 0 1.34 KB
#3180 ExecuteAsync net472 2.54μs 7.16ns 27.7ns 0.225 0 0 1.42 KB
#3180 ExecuteAsync netcoreapp3.1 1.74μs 2.89ns 11.2ns 0.0182 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.66μs 11.6ns 45ns 0.437 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.57μs 8.98ns 33.6ns 0.0353 0 0 2.6 KB
#3180 SendAsync net472 5.69μs 9.83ns 36.8ns 0.439 0 0 2.77 KB
#3180 SendAsync netcoreapp3.1 3.55μs 6.65ns 25.7ns 0.0355 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.34μs 9.1ns 34ns 0.288 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.62μs 1.56ns 5.82ns 0.0248 0 0 1.85 KB
#3180 EnrichedLog net472 3.16μs 1.32ns 4.93ns 0.288 0 0 1.81 KB
#3180 EnrichedLog netcoreapp3.1 2.6μs 0.982ns 3.8ns 0.0248 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 150μs 88.6ns 332ns 0.67 0.223 0 4.65 KB
master EnrichedLog netcoreapp3.1 117μs 182ns 706ns 0.0581 0 0 4.49 KB
#3180 EnrichedLog net472 150μs 80ns 310ns 0.681 0.227 0 4.65 KB
#3180 EnrichedLog netcoreapp3.1 115μs 176ns 683ns 0.0572 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.54μs 12.7ns 49.2ns 0.568 0.00273 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.32μs 9.85ns 38.1ns 0.0525 0 0 3.91 KB
#3180 EnrichedLog net472 5.57μs 8.8ns 32.9ns 0.569 0.00277 0 3.59 KB
#3180 EnrichedLog netcoreapp3.1 4.34μs 11ns 42.6ns 0.0521 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.27μs 0.776ns 2.9ns 0.218 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.79μs 2.92ns 10.9ns 0.0178 0 0 1.32 KB
#3180 SendReceive net472 2.34μs 3.44ns 12.9ns 0.217 0 0 1.37 KB
#3180 SendReceive netcoreapp3.1 1.86μs 0.779ns 2.92ns 0.0177 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5μs 1.79ns 6.94ns 0.354 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.36μs 1.22ns 4.73ns 0.024 0 0 1.8 KB
#3180 EnrichedLog net472 5.12μs 1.65ns 6.19ns 0.352 0 0 2.23 KB
#3180 EnrichedLog netcoreapp3.1 4.41μs 1.97ns 7.62ns 0.0243 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.09μs 0.485ns 1.88ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 929ns 0.334ns 1.25ns 0.0102 0 0 760 B
master StartFinishScope net472 1.35μs 0.518ns 1.94ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.13μs 0.302ns 1.13ns 0.0119 0 0 880 B
#3180 StartFinishSpan net472 1.16μs 0.52ns 2.02ns 0.129 0 0 810 B
#3180 StartFinishSpan netcoreapp3.1 927ns 0.417ns 1.61ns 0.0102 0 0 760 B
#3180 StartFinishScope net472 1.45μs 0.624ns 2.42ns 0.141 0 0 891 B
#3180 StartFinishScope netcoreapp3.1 1.08μs 0.585ns 2.19ns 0.0118 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.53μs 0.587ns 2.27ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.21μs 0.32ns 1.2ns 0.012 0 0 880 B
#3180 RunOnMethodBegin net472 1.47μs 0.349ns 1.26ns 0.142 0 0 891 B
#3180 RunOnMethodBegin netcoreapp3.1 1.13μs 0.472ns 1.83ns 0.0123 0 0 880 B

@andrewlock

andrewlock commented Sep 7, 2022

Copy link
Copy Markdown
Member Author

Code Coverage Report 📊

✔️ Merging #3180 into master will not change line coverage
✔️ Merging #3180 into master will not change branch coverage
⛔ Merging #3180 into master will will increase complexity by 2

master #3180 Change
Lines 17684 / 24285 17789 / 24287
Lines % 73% 73% 0% ✔️
Branches 10429 / 14926 10477 / 14928
Branches % 70% 70% 0% ✔️
Complexity 16121 16123 2 ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3180 Change
Lines % 73% 73% 0% ✔️
Branches % 70% 70% 0% ✔️
Complexity 16121 16123 2 ⛔

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.CIVisibility 5% ✔️ 6% ✔️ 0 ✔️
Datadog.Trace.DiagnosticListeners.DiagnosticManager 12% ✔️ 6% ✔️ 0 ✔️

View the full reports for further details:

@andrewlock
andrewlock merged commit d101ea7 into master Sep 7, 2022
@andrewlock
andrewlock deleted the andrew/add-debug-log-to-api-send branch September 7, 2022 10:52
@github-actions github-actions Bot added this to the vNext milestone Sep 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@kevingosse kevingosse kevingosse approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations)

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

2 participants

@andrewlock @kevingosse