Use agent telemetry proxy instead of the standalone telemetry#6324
Conversation
For some reason, this causes "issues" with the tests that use the Curl handler, in that no telemetry is received
Datadog ReportBranch report: ❌ 1 Failed (1 Known Flaky), 451957 Passed, 2654 Skipped, 18h 50m 45.2s Total Time ❌ Failed Tests (1)
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6324) - mean (72ms) : 63, 80
. : milestone, 72,
master - mean (72ms) : 64, 80
. : milestone, 72,
section CallTarget+Inlining+NGEN
This PR (6324) - mean (1,112ms) : 1094, 1130
. : milestone, 1112,
master - mean (1,107ms) : 1085, 1128
. : milestone, 1107,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6324) - mean (109ms) : 106, 111
. : milestone, 109,
master - mean (108ms) : 106, 110
. : milestone, 108,
section CallTarget+Inlining+NGEN
This PR (6324) - mean (768ms) : 753, 783
. : milestone, 768,
master - mean (771ms) : 757, 785
. : milestone, 771,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6324) - mean (92ms) : 90, 95
. : milestone, 92,
master - mean (92ms) : 90, 93
. : milestone, 92,
section CallTarget+Inlining+NGEN
This PR (6324) - mean (728ms) : 714, 742
. : milestone, 728,
master - mean (726ms) : 707, 745
. : milestone, 726,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6324) - mean (192ms) : 185, 199
. : milestone, 192,
master - mean (191ms) : 186, 195
. : milestone, 191,
section CallTarget+Inlining+NGEN
This PR (6324) - mean (1,214ms) : 1194, 1234
. : milestone, 1214,
master - mean (1,213ms) : 1190, 1236
. : milestone, 1213,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6324) - mean (277ms) : 273, 281
. : milestone, 277,
master - mean (276ms) : 271, 281
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (6324) - mean (947ms) : 930, 965
. : milestone, 947,
master - mean (948ms) : 929, 967
. : milestone, 948,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6324) - mean (265ms) : 261, 270
. : milestone, 265,
master - mean (265ms) : 261, 270
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (6324) - mean (932ms) : 911, 952
. : milestone, 932,
master - mean (931ms) : 913, 950
. : milestone, 931,
|
Benchmarks Report for tracer 🐌Benchmarks for #6324 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Slower
|
| Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 | 1.193 | 492.96 | 588.14 | |
| Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 | 1.149 | 653.83 | 751.34 | |
| Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 | 1.126 | 566.57 | 637.94 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | StartFinishSpan |
net6.0 | 405ns | 0.665ns | 2.57ns | 0.00806 | 0 | 0 | 576 B |
| master | StartFinishSpan |
netcoreapp3.1 | 566ns | 0.73ns | 2.83ns | 0.00792 | 0 | 0 | 576 B |
| master | StartFinishSpan |
net472 | 672ns | 0.909ns | 3.52ns | 0.0916 | 0 | 0 | 578 B |
| master | StartFinishScope |
net6.0 | 493ns | 0.611ns | 2.37ns | 0.00982 | 0 | 0 | 696 B |
| master | StartFinishScope |
netcoreapp3.1 | 654ns | 1.08ns | 4.2ns | 0.00954 | 0 | 0 | 696 B |
| master | StartFinishScope |
net472 | 913ns | 0.51ns | 1.98ns | 0.104 | 0 | 0 | 658 B |
| #6324 | StartFinishSpan |
net6.0 | 397ns | 0.479ns | 1.86ns | 0.00814 | 0 | 0 | 576 B |
| #6324 | StartFinishSpan |
netcoreapp3.1 | 637ns | 1.24ns | 4.8ns | 0.0078 | 0 | 0 | 576 B |
| #6324 | StartFinishSpan |
net472 | 700ns | 0.466ns | 1.8ns | 0.0915 | 0 | 0 | 578 B |
| #6324 | StartFinishScope |
net6.0 | 588ns | 0.731ns | 2.83ns | 0.00982 | 0 | 0 | 696 B |
| #6324 | StartFinishScope |
netcoreapp3.1 | 747ns | 3.52ns | 13.6ns | 0.00929 | 0 | 0 | 696 B |
| #6324 | StartFinishScope |
net472 | 861ns | 0.702ns | 2.72ns | 0.105 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | RunOnMethodBegin |
net6.0 | 591ns | 0.63ns | 2.44ns | 0.0096 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
netcoreapp3.1 | 947ns | 1.12ns | 4.34ns | 0.00945 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
net472 | 1.16μs | 2.46ns | 9.54ns | 0.104 | 0 | 0 | 658 B |
| #6324 | RunOnMethodBegin |
net6.0 | 603ns | 0.266ns | 1.03ns | 0.0097 | 0 | 0 | 696 B |
| #6324 | RunOnMethodBegin |
netcoreapp3.1 | 927ns | 0.972ns | 3.76ns | 0.0094 | 0 | 0 | 696 B |
| #6324 | RunOnMethodBegin |
net472 | 1.17μs | 0.873ns | 3.38ns | 0.104 | 0 | 0 | 658 B |
## Summary of changes Fix the telemetry issues we saw after updating the VMs recently ## Reason for change We updated the VMs and suddenly .NET Core <=3.1 started failing on alpine when we used the "standalone telemetry agent", only when using the "curl" handler (not the socket handler). We eventually traced it down to the fact we were setting the url to `localhost`, not `127.0.0.1` as we do elsewhere. Presumably an update to the underlying libcurl caused it to choose the IPv6 address, which the listener is not bound to, causing telemetry to fail ## Implementation details - Reverted the change in #6324 - `localhost` -> `127.0.0.1` in the `TelemetryHelper` ## Test coverage The same ## Other details Reverting the change isn't necessary obviously, but it shows that we fixed it.
## Summary of changes Fix the telemetry issues we saw after updating the VMs recently ## Reason for change We updated the VMs and suddenly .NET Core <=3.1 started failing on alpine when we used the "standalone telemetry agent", only when using the "curl" handler (not the socket handler). We eventually traced it down to the fact we were setting the url to `localhost`, not `127.0.0.1` as we do elsewhere. Presumably an update to the underlying libcurl caused it to choose the IPv6 address, which the listener is not bound to, causing telemetry to fail ## Implementation details - Reverted the change in #6324 - `localhost` -> `127.0.0.1` in the `TelemetryHelper` ## Test coverage The same ## Other details Reverting the change isn't necessary obviously, but it shows that we fixed it.
Summary of changes
Use the agent proxy instead of the standalone telemetry in the HttpMessageHandlerTests
Reason for change
Currently (after updating the VM images) the non-socket handler tests fail due to a lack of telemetry. I really don't understand how or why - the logs show that we are successfully sending telemetry somewhere 😬 Will investigate that in parallel.
Note also, this only impacts alpine with .NET Core 2.1, 3.0, and 3.1. Debian or .NET 5+ are unaffected 🤷♂️
Implementation details
Switched to using the agent proxy for telemetry (and added some missing extension helpers to the helper)
Test coverage
The same really, there's no reason we need to use the external telemetry proxy for these tests.
Other details
Will follow up once we have more details. Hopefully this will finally unblock the builds