[CI Visibility] Add vcpu_count tag to tslv events by tonyredondo · Pull Request #6055 · DataDog/dd-trace-dotnet

tonyredondo · 2024-09-19T13:02:05Z

Summary of changes

This PR adds the new _dd.host.vcpu_count tag to the test suite level visibility events in CI Visibility mode.

Reason for change

https://docs.google.com/document/d/1NhH9lg1MujAfeeSFQJY0Y-y6kxUDzqP0b7WtMA2FoxM/edit#heading=h.80nefriqhjyl

Test coverage

A new validation on the tests and update of the snapshots.

Other details

github-actions · 2024-09-19T13:09:05Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

1521 occurrences of :

+      _dd.host.vcpu_count: 2.0,

andrewlock · 2024-09-20T08:27:56Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6055) (11.059M)   : 0, 11059045
    master (11.059M)   : 0, 11058767
    benchmarks/2.9.0 (11.081M)   : 0, 11080577

    section Automatic
    This PR (6055) (7.383M)   : 0, 7382602
    master (7.245M)   : 0, 7244717
    benchmarks/2.9.0 (7.732M)   : 0, 7732233

    section Trace stats
    master (7.582M)   : 0, 7581684

    section Manual
    master (11.225M)   : 0, 11225037

    section Manual + Automatic
    This PR (6055) (6.755M)   : 0, 6755303
    master (6.771M)   : 0, 6770863

    section DD_TRACE_ENABLED=0
    master (10.172M)   : 0, 10172171

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6055) (9.676M)   : 0, 9675884
    master (9.615M)   : 0, 9614758
    benchmarks/2.9.0 (9.798M)   : 0, 9798067

    section Automatic
    This PR (6055) (6.654M)   : 0, 6654189
    master (6.423M)   : 0, 6422979

    section Trace stats
    master (6.857M)   : 0, 6856937

    section Manual
    master (9.533M)   : 0, 9533237

    section Manual + Automatic
    This PR (6055) (6.052M)   : 0, 6052297
    master (6.169M)   : 0, 6169272

    section DD_TRACE_ENABLED=0
    master (9.020M)   : 0, 9019554

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6055) (10.172M)   : 0, 10172399
    master (10.029M)   : 0, 10028834
    benchmarks/2.9.0 (10.067M)   : 0, 10067315

    section Automatic
    This PR (6055) (6.564M)   : 0, 6563586
    master (6.718M)   : 0, 6718458
    benchmarks/2.9.0 (7.552M)   : 0, 7552193

    section Trace stats
    master (7.357M)   : 0, 7356722

    section Manual
    master (9.593M)   : 0, 9592818

    section Manual + Automatic
    This PR (6055) (6.019M)   : 0, 6019243
    master (5.832M)   : 0, 5832157

    section DD_TRACE_ENABLED=0
    master (9.017M)   : 0, 9016737

andrewlock · 2024-09-20T08:30:21Z

Benchmarks Report for tracer 🐌

Benchmarks for #6055 compared to master:

3 benchmarks are faster, with geometric mean 1.169
1 benchmarks are slower, with geometric mean 1.126
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.71μs	43.9ns	322ns	0.0157	0.00785	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.78μs	52ns	290ns	0.0145	0.00484	0	5.61 KB
master	`StartStopWithChild`	net472	16μs	55.9ns	217ns	1.02	0.305	0.0963	6.07 KB
#6055	`StartStopWithChild`	net6.0	7.87μs	44.2ns	290ns	0.0158	0.00395	0	5.43 KB
#6055	`StartStopWithChild`	netcoreapp3.1	9.74μs	50.8ns	254ns	0.0188	0.00939	0	5.61 KB
#6055	`StartStopWithChild`	net472	15.8μs	43.4ns	162ns	1.02	0.314	0.102	6.07 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	467μs	1.18μs	4.56μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	638μs	227ns	788ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	822μs	588ns	2.28μs	0.411	3.3 KB
#6055	`WriteAndFlushEnrichedTraces`	net6.0	476μs	329ns	1.27μs	0	2.7 KB
#6055	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	292ns	1.13μs	0	2.7 KB
#6055	`WriteAndFlushEnrichedTraces`	net472	835μs	582ns	2.25μs	0.411	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	198μs	1.13μs	8.59μs	0.189	18.45 KB
master	`SendRequest`	netcoreapp3.1	221μs	1.13μs	7.5μs	0.206	20.61 KB
master	`SendRequest`	net472	0.00125ns	0.000624ns	0.00242ns	0	0 b
#6055	`SendRequest`	net6.0	192μs	1.01μs	5.44μs	0.191	18.45 KB
#6055	`SendRequest`	netcoreapp3.1	223μs	1.28μs	9.72μs	0.221	20.61 KB
#6055	`SendRequest`	net472	2.44E‑05ns	2.44E‑05ns	9.14E‑05ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	569μs	2.21μs	8.55μs	0.546	0	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	698μs	3.88μs	25.5μs	0.331	0	0	41.79 KB
master	`WriteAndFlushEnrichedTraces`	net472	890μs	2.96μs	11.1μs	8.45	2.53	0.422	53.3 KB
#6055	`WriteAndFlushEnrichedTraces`	net6.0	563μs	2.97μs	15.1μs	0.581	0	0	41.58 KB
#6055	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	668μs	2.3μs	8.59μs	0.336	0	0	41.76 KB
#6055	`WriteAndFlushEnrichedTraces`	net472	860μs	3.45μs	13.4μs	8.08	2.55	0.425	53.31 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.27μs	1.67ns	6.25ns	0.0139	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.78μs	1.3ns	4.85ns	0.0133	1.02 KB
master	`ExecuteNonQuery`	net472	2.02μs	1.56ns	5.82ns	0.156	987 B
#6055	`ExecuteNonQuery`	net6.0	1.3μs	1.12ns	4.33ns	0.0145	1.02 KB
#6055	`ExecuteNonQuery`	netcoreapp3.1	1.76μs	1.78ns	6.67ns	0.0138	1.02 KB
#6055	`ExecuteNonQuery`	net472	2.1μs	2.91ns	11.3ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.13μs	1.14ns	4.28ns	0.014	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.718ns	2.78ns	0.0137	976 B
master	`CallElasticsearch`	net472	2.6μs	1.69ns	6.34ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.27μs	1.19ns	4.62ns	0.0133	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.58μs	0.663ns	2.57ns	0.0142	1.02 KB
master	`CallElasticsearchAsync`	net472	2.66μs	1.27ns	4.9ns	0.166	1.05 KB
#6055	`CallElasticsearch`	net6.0	1.21μs	0.37ns	1.39ns	0.0134	976 B
#6055	`CallElasticsearch`	netcoreapp3.1	1.47μs	0.729ns	2.73ns	0.0133	976 B
#6055	`CallElasticsearch`	net472	2.41μs	1.68ns	6.51ns	0.158	995 B
#6055	`CallElasticsearchAsync`	net6.0	1.38μs	0.829ns	3.1ns	0.0131	952 B
#6055	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	0.966ns	3.74ns	0.0137	1.02 KB
#6055	`CallElasticsearchAsync`	net472	2.59μs	1.3ns	5.04ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.23μs	0.55ns	2.06ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.58μs	0.525ns	1.89ns	0.0127	952 B
master	`ExecuteAsync`	net472	1.75μs	0.736ns	2.85ns	0.145	915 B
#6055	`ExecuteAsync`	net6.0	1.25μs	0.509ns	1.9ns	0.0132	952 B
#6055	`ExecuteAsync`	netcoreapp3.1	1.66μs	0.829ns	3.1ns	0.0125	952 B
#6055	`ExecuteAsync`	net472	1.76μs	0.583ns	2.26ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.22μs	2.06ns	7.72ns	0.0317	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.07μs	2.6ns	10.1ns	0.038	2.76 KB
master	`SendAsync`	net472	7.78μs	1.74ns	6.74ns	0.498	3.15 KB
#6055	`SendAsync`	net6.0	4.32μs	1.71ns	6.4ns	0.0308	2.22 KB
#6055	`SendAsync`	netcoreapp3.1	5.05μs	9.63ns	37.3ns	0.0377	2.76 KB
#6055	`SendAsync`	net472	7.82μs	1.66ns	6.42ns	0.499	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.59μs	0.915ns	3.54ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.25μs	1.18ns	4.41ns	0.0226	1.64 KB
master	`EnrichedLog`	net472	2.57μs	0.896ns	3.47ns	0.249	1.57 KB
#6055	`EnrichedLog`	net6.0	1.65μs	1.03ns	3.85ns	0.0231	1.64 KB
#6055	`EnrichedLog`	netcoreapp3.1	2.27μs	1.13ns	4.24ns	0.0228	1.64 KB
#6055	`EnrichedLog`	net472	2.56μs	0.544ns	2.04ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	115μs	218ns	815ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	212ns	820ns	0	0	4.28 KB
master	`EnrichedLog`	net472	146μs	223ns	865ns	0.659	0.22	4.46 KB
#6055	`EnrichedLog`	net6.0	114μs	196ns	758ns	0.0566	0	4.28 KB
#6055	`EnrichedLog`	netcoreapp3.1	119μs	224ns	869ns	0	0	4.28 KB
#6055	`EnrichedLog`	net472	145μs	74.9ns	290ns	0.655	0.218	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.1μs	2.34ns	9.08ns	0.031	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.24μs	1.14ns	4.26ns	0.0292	2.2 KB
master	`EnrichedLog`	net472	4.91μs	2.12ns	8.21ns	0.319	2.02 KB
#6055	`EnrichedLog`	net6.0	2.98μs	1.14ns	4.28ns	0.0298	2.2 KB
#6055	`EnrichedLog`	netcoreapp3.1	4.49μs	1.18ns	4.42ns	0.0292	2.2 KB
#6055	`EnrichedLog`	net472	4.77μs	1.77ns	6.87ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.35μs	1.74ns	6.73ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.77μs	0.766ns	2.97ns	0.015	1.14 KB
master	`SendReceive`	net472	2.07μs	0.853ns	3.31ns	0.183	1.16 KB
#6055	`SendReceive`	net6.0	1.27μs	0.613ns	2.29ns	0.0159	1.14 KB
#6055	`SendReceive`	netcoreapp3.1	1.81μs	1.9ns	7.37ns	0.0152	1.14 KB
#6055	`SendReceive`	net472	2.14μs	2.51ns	9.74ns	0.184	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.58μs	0.882ns	3.41ns	0.022	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.89μs	5.3ns	20.5ns	0.0215	1.65 KB
master	`EnrichedLog`	net472	4.34μs	0.938ns	3.63ns	0.322	2.04 KB
#6055	`EnrichedLog`	net6.0	2.68μs	0.996ns	3.86ns	0.0215	1.6 KB
#6055	`EnrichedLog`	netcoreapp3.1	3.94μs	1.69ns	6.56ns	0.0214	1.65 KB
#6055	`EnrichedLog`	net472	4.42μs	1.76ns	6.59ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6055

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.126	395.78	445.70

Faster 🎉 in #6055

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.190	569.16	478.09
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.155	662.43	573.58

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	396ns	0.312ns	1.21ns	0.00801	576 B
master	`StartFinishSpan`	netcoreapp3.1	662ns	0.666ns	2.58ns	0.00792	576 B
master	`StartFinishSpan`	net472	624ns	0.728ns	2.82ns	0.0918	578 B
master	`StartFinishScope`	net6.0	569ns	0.591ns	2.29ns	0.00969	696 B
master	`StartFinishScope`	netcoreapp3.1	684ns	0.602ns	2.33ns	0.00936	696 B
master	`StartFinishScope`	net472	811ns	0.856ns	3.31ns	0.104	658 B
#6055	`StartFinishSpan`	net6.0	445ns	0.204ns	0.792ns	0.00803	576 B
#6055	`StartFinishSpan`	netcoreapp3.1	574ns	0.582ns	2.25ns	0.00774	576 B
#6055	`StartFinishSpan`	net472	646ns	0.235ns	0.911ns	0.0916	578 B
#6055	`StartFinishScope`	net6.0	479ns	0.686ns	2.66ns	0.00976	696 B
#6055	`StartFinishScope`	netcoreapp3.1	703ns	3.38ns	13.9ns	0.00953	696 B
#6055	`StartFinishScope`	net472	821ns	1.3ns	5.04ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6055

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.163	753.00	647.28

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	753ns	0.555ns	2.15ns	0.00982	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	908ns	0.796ns	3.08ns	0.0095	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.911ns	3.53ns	0.104	658 B
#6055	`RunOnMethodBegin`	net6.0	647ns	0.524ns	2.03ns	0.00979	696 B
#6055	`RunOnMethodBegin`	netcoreapp3.1	961ns	0.929ns	3.6ns	0.00958	696 B
#6055	`RunOnMethodBegin`	net472	1.11μs	1.12ns	4.33ns	0.104	658 B

bouwkast

👍

bouwkast · 2024-09-23T11:51:22Z

        AssertTargetSpanExists(targetTest, CommonTags.OSPlatform);
        AssertTargetSpanEqual(targetTest, CommonTags.OSVersion, CIVisibility.GetOperatingSystemVersion());
+        targetTest.Metrics[CommonTags.LogicalCpuCount].Should().Be(Environment.ProcessorCount);
+        targetTest.Metrics.Remove(CommonTags.LogicalCpuCount);


It seems this .Remove is done in most of these tests is the targetTest reused? (just curious)

is not reused nope. the test removes all the expected tags and then we assert that we no longer has any other tag in the test event.

Ahh makes sense thanks!

[CI Visibility] Add vcpu_count tag to tslv events

8ceb713

tonyredondo self-assigned this Sep 19, 2024

Merge branch 'master' into tony/civisibility-send-cpu-count

40e24ea

github-actions Bot added the area:ci-visibility label Sep 19, 2024

Snapshots update

ac13d0b

DataDog deleted a comment from github-actions Bot Sep 19, 2024

tonyredondo marked this pull request as ready for review September 19, 2024 13:34

tonyredondo requested review from a team as code owners September 19, 2024 13:34

tonyredondo added 2 commits September 19, 2024 16:34

update snapshots

9f95737

Merge branch 'master' into tony/civisibility-send-cpu-count

3f1ddf3

anmarchenko approved these changes Sep 20, 2024

View reviewed changes

tonyredondo added 2 commits September 20, 2024 09:30

Update selenium snapshot

41d0e45

Merge branch 'master' into tony/civisibility-send-cpu-count

2bd6260

DataDog deleted a comment from datadog-ddstaging Bot Sep 20, 2024

DataDog deleted a comment from andrewlock Sep 20, 2024

bouwkast approved these changes Sep 23, 2024

View reviewed changes

tonyredondo merged commit 0bcf842 into master Sep 23, 2024

tonyredondo deleted the tony/civisibility-send-cpu-count branch September 23, 2024 12:00

github-actions Bot added this to the vNext-v3 milestone Sep 23, 2024

andrewlock added the type:enhancement Improvement to an existing feature label Oct 23, 2024

Uh oh!

Conversation

tonyredondo commented Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Test coverage

Other details

Uh oh!

github-actions Bot commented Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Snapshots difference summary

Uh oh!

andrewlock commented Sep 20, 2024

Throughput/Crank Report ⚡

Uh oh!

andrewlock commented Sep 20, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6055

Faster 🎉 in #6055

Raw results

Faster 🎉 in #6055

Raw results

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

bouwkast Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

tonyredondo Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

bouwkast Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tonyredondo commented Sep 19, 2024 •

edited

Loading

github-actions Bot commented Sep 19, 2024 •

edited

Loading