[CI Visibility] Ensure Datadog.Trace assembly is included in the Datadog.Trace.BenchmarkDotNet package by tonyredondo · Pull Request #5953 · DataDog/dd-trace-dotnet

tonyredondo · 2024-08-27T13:29:24Z

Summary of changes

This PR ensures that the Datadog.Trace assemblies are included in the Datadog.Trace.BenchmarkDotNet nuget package.

Reason for change

Datadog.Trace is required in the Datadog.Trace.BenchmarkDotnet integration, and since for v3 we are not publishing this assembly we must include it manually in the Datadog.Trace.BenchmarkDotnet package to avoid the following exception when using the BenchmarkDotNet integration

Implementation details

We just modified the csproj file to make sure we are including all assemblies from ProjectReference with the PrivateAssets=All flag

Test coverage

We need to think about how to test this, no tests are included in this PR.

Other details

…dog.Trace.BenchmarkDotNet package

andrewlock

MY EYES! 😝

andrewlock · 2024-08-27T13:41:27Z

I just realised that this explicitly reintroduces the possibility of version conflict which could be very problematic if customers are also automatically instrumenting, as we will likely crash them at some point 🙈

tonyredondo · 2024-08-27T13:46:08Z

I just realised that this explicitly reintroduces the possibility of version conflict which could be very problematic if customers are also automatically instrumenting, as we will likely crash them at some point 🙈

I think we can say that this integration is not compatible with automatic instrumentation. This is a nuget package that meant to be used only in the Benchmark project.

andrewlock · 2024-08-27T13:49:07Z

I just realised that this explicitly reintroduces the possibility of version conflict which could be very problematic if customers are also automatically instrumenting, as we will likely crash them at some point 🙈

I think we can say that this integration is not compatible with automatic instrumentation. This is a nuget package that meant to be used only in the Benchmark project.

CI Visibility is auto-instrumentation though 😉

datadog-ddstaging · 2024-08-27T13:50:39Z

Datadog Report

Branch report: tony/ensure-datadog-trace-in-nupkg
Commit report: 245d69b
Test service: dd-trace-dotnet

✅ 0 Failed, 301401 Passed, 1601 Skipped, 11h 39m 19.4s Total Time

andrewlock · 2024-08-27T13:54:09Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

tonyredondo · 2024-08-27T13:57:11Z

I just realised that this explicitly reintroduces the possibility of version conflict which could be very problematic if customers are also automatically instrumenting, as we will likely crash them at some point 🙈

I think we can say that this integration is not compatible with automatic instrumentation. This is a nuget package that meant to be used only in the Benchmark project.

CI Visibility is auto-instrumentation though 😉

Except the BenchmarkDotNet integration, because we don't want the CLR profiler to interfere with the measurements.

tonyredondo · 2024-08-27T14:45:38Z

I think to avoid any crash and bad usage from customers, we should avoid publishing the v3 of Datadog.Trace.BenchmarkDotNet until we have a good alternative without a crashing scenario.

andrewlock · 2024-08-27T18:20:36Z

Benchmarks Report for tracer 🐌

Benchmarks for #5953 compared to master:

1 benchmarks are faster, with geometric mean 1.156
2 benchmarks are slower, with geometric mean 1.129
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.68μs	42.5ns	272ns	0.0148	0.00738	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.91μs	52.5ns	273ns	0.0192	0.00962	0	5.62 KB
master	`StartStopWithChild`	net472	16μs	55.9ns	216ns	1.02	0.302	0.103	6.06 KB
#5953	`StartStopWithChild`	net6.0	7.58μs	33.2ns	120ns	0.0145	0.00723	0	5.42 KB
#5953	`StartStopWithChild`	netcoreapp3.1	9.94μs	56.3ns	398ns	0.0198	0.00989	0	5.62 KB
#5953	`StartStopWithChild`	net472	17.5μs	114ns	1.14μs	1.03	0.299	0.0998	6.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	465μs	200ns	722ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	634μs	278ns	1.08μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	836μs	677ns	2.62μs	0.417	3.3 KB
#5953	`WriteAndFlushEnrichedTraces`	net6.0	474μs	642ns	2.49μs	0	2.7 KB
#5953	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	651μs	330ns	1.23μs	0	2.7 KB
#5953	`WriteAndFlushEnrichedTraces`	net472	842μs	478ns	1.79μs	0.419	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	192μs	1.11μs	9.5μs	0.182	18.45 KB
master	`SendRequest`	netcoreapp3.1	208μs	1.15μs	7.46μs	0.198	20.61 KB
master	`SendRequest`	net472	0.001ns	0.000448ns	0.00168ns	0	0 b
#5953	`SendRequest`	net6.0	188μs	1.03μs	6.2μs	0.182	18.45 KB
#5953	`SendRequest`	netcoreapp3.1	207μs	990ns	3.96μs	0.203	20.61 KB
#5953	`SendRequest`	net472	0.00718ns	0.00235ns	0.00878ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5953

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.73 KB	41.95 KB	220 B	0.53%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	555μs	2.33μs	8.42μs	0.566	0	0	41.57 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	669μs	2.33μs	8.41μs	0.326	0	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	net472	852μs	2.35μs	9.1μs	8.39	2.52	0.419	53.3 KB
#5953	`WriteAndFlushEnrichedTraces`	net6.0	560μs	2.42μs	9.06μs	0.551	0	0	41.6 KB
#5953	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	683μs	3.5μs	16.8μs	0.343	0	0	41.95 KB
#5953	`WriteAndFlushEnrichedTraces`	net472	869μs	4.13μs	16.5μs	8.45	2.53	0.422	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.23μs	1.01ns	3.8ns	0.0146	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.73μs	1.47ns	5.71ns	0.0137	1.02 KB
master	`ExecuteNonQuery`	net472	1.98μs	1.89ns	7.31ns	0.157	987 B
#5953	`ExecuteNonQuery`	net6.0	1.17μs	0.952ns	3.69ns	0.0146	1.02 KB
#5953	`ExecuteNonQuery`	netcoreapp3.1	1.68μs	1.26ns	4.7ns	0.0135	1.02 KB
#5953	`ExecuteNonQuery`	net472	2.03μs	1.52ns	5.89ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5953

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0	1.156	1,383.61	1,196.69

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.23μs	0.999ns	3.87ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.63μs	0.555ns	2.15ns	0.013	976 B
master	`CallElasticsearch`	net472	2.43μs	1.72ns	6.66ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.38μs	1.31ns	4.91ns	0.0131	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.57μs	1.27ns	4.76ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.58μs	1.71ns	6.42ns	0.167	1.05 KB
#5953	`CallElasticsearch`	net6.0	1.29μs	2.94ns	11.4ns	0.0139	976 B
#5953	`CallElasticsearch`	netcoreapp3.1	1.53μs	1.23ns	4.26ns	0.0132	976 B
#5953	`CallElasticsearch`	net472	2.46μs	1.61ns	6.03ns	0.157	995 B
#5953	`CallElasticsearchAsync`	net6.0	1.2μs	0.567ns	2.2ns	0.0133	952 B
#5953	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	0.539ns	2.02ns	0.0138	1.02 KB
#5953	`CallElasticsearchAsync`	net472	2.6μs	1.9ns	7.35ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.26μs	0.661ns	2.47ns	0.0133	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.55μs	0.555ns	2ns	0.0131	952 B
master	`ExecuteAsync`	net472	1.78μs	1.47ns	5.71ns	0.145	915 B
#5953	`ExecuteAsync`	net6.0	1.32μs	1.43ns	5.35ns	0.0133	952 B
#5953	`ExecuteAsync`	netcoreapp3.1	1.66μs	0.633ns	2.37ns	0.0133	952 B
#5953	`ExecuteAsync`	net472	1.73μs	1.73ns	6.7ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.21μs	2.09ns	8.11ns	0.0315	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.09μs	2.27ns	8.81ns	0.0358	2.76 KB
master	`SendAsync`	net472	7.75μs	3.24ns	12.6ns	0.499	3.15 KB
#5953	`SendAsync`	net6.0	4.07μs	0.82ns	3.18ns	0.0305	2.22 KB
#5953	`SendAsync`	netcoreapp3.1	5.11μs	2.11ns	8.18ns	0.0358	2.76 KB
#5953	`SendAsync`	net472	7.8μs	2.67ns	10.3ns	0.498	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5953

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net6.0	1.112	1,483.18	1,649.85

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.48μs	0.708ns	2.65ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.38μs	1.1ns	4.12ns	0.0217	1.64 KB
master	`EnrichedLog`	net472	2.68μs	1.93ns	7.24ns	0.249	1.57 KB
#5953	`EnrichedLog`	net6.0	1.65μs	0.886ns	3.43ns	0.0234	1.64 KB
#5953	`EnrichedLog`	netcoreapp3.1	2.16μs	1.2ns	4.63ns	0.0225	1.64 KB
#5953	`EnrichedLog`	net472	2.69μs	1.68ns	6.04ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	166ns	621ns	0.0578	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	234ns	905ns	0	0	4.28 KB
master	`EnrichedLog`	net472	148μs	225ns	872ns	0.668	0.223	4.46 KB
#5953	`EnrichedLog`	net6.0	117μs	319ns	1.24μs	0.058	0	4.28 KB
#5953	`EnrichedLog`	netcoreapp3.1	121μs	173ns	670ns	0.0599	0	4.28 KB
#5953	`EnrichedLog`	net472	147μs	364ns	1.41μs	0.663	0.221	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.95μs	1.62ns	6.26ns	0.0313	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.09μs	1.07ns	4.01ns	0.0286	2.2 KB
master	`EnrichedLog`	net472	4.78μs	2.91ns	11.3ns	0.319	2.02 KB
#5953	`EnrichedLog`	net6.0	2.97μs	0.947ns	3.54ns	0.03	2.2 KB
#5953	`EnrichedLog`	netcoreapp3.1	4.1μs	3.65ns	14.1ns	0.0287	2.2 KB
#5953	`EnrichedLog`	net472	4.9μs	1.25ns	4.84ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.37μs	0.604ns	2.34ns	0.0164	0	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	1.38ns	5.35ns	0.0149	0	1.14 KB
master	`SendReceive`	net472	2.18μs	1.64ns	5.9ns	0.184	0.00109	1.16 KB
#5953	`SendReceive`	net6.0	1.41μs	0.788ns	2.95ns	0.0161	0	1.14 KB
#5953	`SendReceive`	netcoreapp3.1	1.72μs	0.624ns	2.42ns	0.0155	0	1.14 KB
#5953	`SendReceive`	net472	2.18μs	1.98ns	7.67ns	0.183	0.00109	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.81μs	0.997ns	3.86ns	0.0225	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.92μs	1.92ns	7.43ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	4.36μs	2.83ns	11ns	0.324	2.04 KB
#5953	`EnrichedLog`	net6.0	2.72μs	0.947ns	3.67ns	0.0218	1.6 KB
#5953	`EnrichedLog`	netcoreapp3.1	3.91μs	1.13ns	4.08ns	0.0215	1.65 KB
#5953	`EnrichedLog`	net472	4.41μs	2.68ns	10.4ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	396ns	0.124ns	0.463ns	0.00819	576 B
master	`StartFinishSpan`	netcoreapp3.1	606ns	0.414ns	1.6ns	0.00788	576 B
master	`StartFinishSpan`	net472	630ns	0.601ns	2.33ns	0.0918	578 B
master	`StartFinishScope`	net6.0	605ns	0.291ns	1.13ns	0.0097	696 B
master	`StartFinishScope`	netcoreapp3.1	717ns	0.508ns	1.97ns	0.00972	696 B
master	`StartFinishScope`	net472	841ns	0.845ns	3.27ns	0.104	658 B
#5953	`StartFinishSpan`	net6.0	393ns	0.265ns	1.03ns	0.00814	576 B
#5953	`StartFinishSpan`	netcoreapp3.1	572ns	0.458ns	1.78ns	0.00774	576 B
#5953	`StartFinishSpan`	net472	643ns	2.21ns	8.57ns	0.0918	578 B
#5953	`StartFinishScope`	net6.0	544ns	0.39ns	1.51ns	0.00975	696 B
#5953	`StartFinishScope`	netcoreapp3.1	719ns	0.976ns	3.65ns	0.00952	696 B
#5953	`StartFinishScope`	net472	852ns	0.652ns	2.53ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5953

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.145	578.90	662.96

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	579ns	0.354ns	1.37ns	0.00966	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	922ns	1.08ns	4.02ns	0.00931	696 B
master	`RunOnMethodBegin`	net472	1.05μs	0.801ns	3.1ns	0.104	658 B
#5953	`RunOnMethodBegin`	net6.0	663ns	0.399ns	1.49ns	0.00965	696 B
#5953	`RunOnMethodBegin`	netcoreapp3.1	908ns	0.672ns	2.6ns	0.00948	696 B
#5953	`RunOnMethodBegin`	net472	1.09μs	0.652ns	2.26ns	0.104	658 B

…artifacts (#5954) ## Summary of changes Exclude the `Datadog.Trace.BenchmarkDotNet` NuGet package from the v3 release artifacts ## Reason for change The v3 version of Datadog.Trace.BenchmarkDotNet requires custom-only instrumentation... which doesn't work in v3. Rather than ship something which breaks at runtime, we'll hold off shipping this for now, and figure out our solution later. The v2 version continues to work, the package is not heavily used, and it shouldn't be used with auto-instrumentation anyway, so this seems like the best short-term solution. ## Implementation details Remove the NuGet from the release artifacts. ## Test coverage For this build I'll check that it's not in the release artifacts. Otherwise there's nothing else to do. ## Other Details Favouring this over #5953 for now to avoid any future issues with mixing version conflict stuff

[CI Visibility] Ensure Datadog.Trace assembly is included in the Data…

245d69b

…dog.Trace.BenchmarkDotNet package

tonyredondo requested a review from a team as a code owner August 27, 2024 13:29

tonyredondo self-assigned this Aug 27, 2024

github-actions Bot added the area:ci-visibility label Aug 27, 2024

andrewlock approved these changes Aug 27, 2024

View reviewed changes

tonyredondo closed this Aug 27, 2024

andrewlock mentioned this pull request Aug 27, 2024

Don't include the Datadog.Trace.BenchmarkDotNet NuGet in the release artifacts #5954

Merged

Uh oh!

Conversation

tonyredondo commented Aug 27, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock commented Aug 27, 2024

Uh oh!

tonyredondo commented Aug 27, 2024

Uh oh!

andrewlock commented Aug 27, 2024

Uh oh!

datadog-ddstaging Bot commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

andrewlock commented Aug 27, 2024

Execution-Time Benchmarks Report ⏱️

Uh oh!

tonyredondo commented Aug 27, 2024

Uh oh!

tonyredondo commented Aug 27, 2024

Uh oh!

andrewlock commented Aug 27, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #5953

Raw results

Raw results

Faster 🎉 in #5953

Raw results

Raw results

Raw results

Slower ⚠️ in #5953

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5953

Raw results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-ddstaging Bot commented Aug 27, 2024 •

edited

Loading