[Tracer] Add tests for stats computation feature by zacharycmontoya · Pull Request #3047 · DataDog/dd-trace-dotnet

zacharycmontoya · 2022-08-03T21:40:28Z

Summary of changes

Adds testing for various requirements of the stats computation feature, as agreed upon by other tracers.

This is PR 1/3 in an attempt to break down the massive PR #2988

Reason for change

Implementation details

Test coverage

Adds new unit tests (StatsAggregatorTests) and integration tests (StatsTests)

Other details

…Operation, Type, HTTPStatusCode, and Synthetics will be stored in different buckets

…ount for service entry spans that are child spans

… support them. Since this has not yet been implemented, the test is skipped

… case is skipped because the feature is not yet implemented

… helper method

…ently fails

…o be computed

…al we can test

…mentation was correct

andrewlock · 2022-08-04T17:10:58Z

Benchmarks Report 🐌

Benchmarks for #3047 compared to master:

1 benchmarks are faster, with geometric mean 1.333
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net472	722μs	326ns	1.26μs	0.359	3.18 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	478μs	344ns	1.33μs	0	2.58 KB
#3047	`WriteAndFlushEnrichedTraces`	net472	717μs	757ns	2.93μs	0.357	3.18 KB
#3047	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	463μs	217ns	814ns	0	2.59 KB

Benchmarks.Trace.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #3047

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	1.333	2,208.74	1,657.47

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net472	2.21μs	2.52ns	9.44ns	0.237	0	1.49 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	1.8μs	2.11ns	8.16ns	0.0186	0	1.37 KB
master	`AllCycleMoreComplexBody`	net472	17.1μs	14.3ns	53.6ns	1.38	0.0171	8.75 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	14.2μs	17.1ns	63.8ns	0.107	0	7.85 KB
master	`BodyExtractorSimpleBody`	net472	277ns	0.284ns	1.02ns	0.0573	0	361 B
master	`BodyExtractorSimpleBody`	netcoreapp3.1	227ns	0.136ns	0.509ns	0.00376	0	272 B
master	`BodyExtractorMoreComplexBody`	net472	15.1μs	6.97ns	26.1ns	1.21	0.0151	7.62 KB
master	`BodyExtractorMoreComplexBody`	netcoreapp3.1	12μs	16.6ns	62.2ns	0.0902	0	6.75 KB
#3047	`AllCycleSimpleBody`	net472	1.66μs	1.77ns	6.87ns	0.237	0	1.49 KB
#3047	`AllCycleSimpleBody`	netcoreapp3.1	1.77μs	2.26ns	8.73ns	0.0183	0	1.37 KB
#3047	`AllCycleMoreComplexBody`	net472	17μs	14.1ns	52.8ns	1.38	0.0253	8.75 KB
#3047	`AllCycleMoreComplexBody`	netcoreapp3.1	13.8μs	14.4ns	52ns	0.104	0	7.85 KB
#3047	`BodyExtractorSimpleBody`	net472	260ns	0.803ns	3.11ns	0.0573	0	361 B
#3047	`BodyExtractorSimpleBody`	netcoreapp3.1	214ns	0.132ns	0.477ns	0.00373	0	272 B
#3047	`BodyExtractorMoreComplexBody`	net472	14.7μs	10.2ns	35.2ns	1.21	0.0147	7.62 KB
#3047	`BodyExtractorMoreComplexBody`	netcoreapp3.1	12.2μs	10ns	36.2ns	0.0906	0	6.75 KB

Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
master	`SendRequest`	netcoreapp3.1	177μs	225ns	872ns	0.266	20.33 KB
#3047	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#3047	`SendRequest`	netcoreapp3.1	178μs	105ns	406ns	0.266	20.33 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net472	1.57μs	0.513ns	1.92ns	0.126	0.000785	794 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.22μs	0.346ns	1.29ns	0.011	0	824 B
#3047	`ExecuteNonQuery`	net472	1.54μs	0.506ns	1.96ns	0.126	0.000772	794 B
#3047	`ExecuteNonQuery`	netcoreapp3.1	1.28μs	0.212ns	0.765ns	0.0111	0	824 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net472	2.28μs	0.705ns	2.73ns	0.159	1 KB
master	`CallElasticsearch`	netcoreapp3.1	1.43μs	0.613ns	2.3ns	0.0136	984 B
master	`CallElasticsearchAsync`	net472	2.43μs	0.81ns	3.14ns	0.18	1.14 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	1.49μs	0.281ns	1.09ns	0.0149	1.1 KB
#3047	`CallElasticsearch`	net472	2.21μs	0.837ns	3.24ns	0.159	1 KB
#3047	`CallElasticsearch`	netcoreapp3.1	1.38μs	0.882ns	3.18ns	0.0131	984 B
#3047	`CallElasticsearchAsync`	net472	2.3μs	0.698ns	2.7ns	0.181	1.14 KB
#3047	`CallElasticsearchAsync`	netcoreapp3.1	1.5μs	0.452ns	1.63ns	0.0149	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net472	2.34μs	5.11ns	19.8ns	0.199	1.26 KB
master	`ExecuteAsync`	netcoreapp3.1	1.59μs	2.72ns	10.2ns	0.0164	1.22 KB
#3047	`ExecuteAsync`	net472	2.44μs	1.98ns	7.65ns	0.2	1.26 KB
#3047	`ExecuteAsync`	netcoreapp3.1	1.64μs	0.737ns	2.85ns	0.0163	1.22 KB

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net472	4.88μs	8.01ns	31ns	0.393	2.48 KB
master	`SendAsync`	netcoreapp3.1	3.28μs	7.51ns	29.1ns	0.0324	2.36 KB
#3047	`SendAsync`	net472	5.02μs	1.62ns	6.29ns	0.393	2.48 KB
#3047	`SendAsync`	netcoreapp3.1	3.23μs	4.65ns	18ns	0.0319	2.36 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	2.89μs	0.935ns	3.5ns	0.263	1.66 KB
master	`EnrichedLog`	netcoreapp3.1	2.29μs	0.961ns	3.72ns	0.0239	1.73 KB
#3047	`EnrichedLog`	net472	2.99μs	1.25ns	4.69ns	0.263	1.66 KB
#3047	`EnrichedLog`	netcoreapp3.1	2.52μs	0.862ns	3.34ns	0.0238	1.73 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	146μs	93.9ns	364ns	0.66	0.22	4.5 KB
master	`EnrichedLog`	netcoreapp3.1	113μs	82.4ns	308ns	0.056	0	4.38 KB
#3047	`EnrichedLog`	net472	147μs	78.3ns	293ns	0.666	0.222	4.5 KB
#3047	`EnrichedLog`	netcoreapp3.1	113μs	70.1ns	272ns	0.0559	0	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	5.32μs	12.7ns	49.4ns	0.545	0.00262	3.43 KB
master	`EnrichedLog`	netcoreapp3.1	4.22μs	6.58ns	25.5ns	0.0512	0	3.8 KB
#3047	`EnrichedLog`	net472	5.36μs	4.53ns	17.6ns	0.544	0.00268	3.43 KB
#3047	`EnrichedLog`	netcoreapp3.1	4.3μs	2.81ns	10.5ns	0.0515	0	3.8 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net472	1.98μs	1.18ns	4.43ns	0.194	1.22 KB
master	`SendReceive`	netcoreapp3.1	1.61μs	0.467ns	1.75ns	0.0162	1.21 KB
#3047	`SendReceive`	net472	2.03μs	2.25ns	8.71ns	0.193	1.22 KB
#3047	`SendReceive`	netcoreapp3.1	1.65μs	0.791ns	2.96ns	0.0165	1.21 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	4.78μs	1.93ns	7.46ns	0.329	2.08 KB
master	`EnrichedLog`	netcoreapp3.1	4.12μs	1.73ns	6.48ns	0.0227	1.69 KB
#3047	`EnrichedLog`	net472	4.57μs	1.8ns	6.98ns	0.33	2.08 KB
#3047	`EnrichedLog`	netcoreapp3.1	4.12μs	1.93ns	7.21ns	0.0226	1.69 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net472	860ns	0.59ns	2.28ns	0.105	658 B
master	`StartFinishSpan`	netcoreapp3.1	733ns	0.251ns	0.906ns	0.00875	648 B
master	`StartFinishScope`	net472	1.05μs	0.702ns	2.63ns	0.117	738 B
master	`StartFinishScope`	netcoreapp3.1	881ns	0.501ns	1.87ns	0.0101	768 B
#3047	`StartFinishSpan`	net472	877ns	0.35ns	1.36ns	0.104	658 B
#3047	`StartFinishSpan`	netcoreapp3.1	776ns	0.133ns	0.515ns	0.0089	648 B
#3047	`StartFinishScope`	net472	1.01μs	0.253ns	0.981ns	0.117	738 B
#3047	`StartFinishScope`	netcoreapp3.1	899ns	1.39ns	4.82ns	0.0104	768 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net472	1.16μs	0.7ns	2.71ns	0.117	738 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.04μs	0.506ns	1.75ns	0.0104	768 B
#3047	`RunOnMethodBegin`	net472	1.22μs	0.519ns	2.01ns	0.117	738 B
#3047	`RunOnMethodBegin`	netcoreapp3.1	1.06μs	0.413ns	1.55ns	0.0101	768 B

andrewlock · 2022-08-04T17:47:31Z

Code Coverage Report 📊

✔️ Merging #3047 into master will not change line coverage
✔️ Merging #3047 into master will not change branch coverage
✔️ Merging #3047 into master will not change complexity

	master	#3047	Change
Lines	`16778` / `22859`	`16748` / `22859`
Lines %	`73%`	`73%`	`0%` ✔️
Branches	`9999` / `14184`	`9999` / `14184`
Branches %	`70%`	`70%`	`0%` ✔️
Complexity	`15245`	`15245`	`0` ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

	master	#3047	Change
Lines %	`73%`	`73%`	`0%` ✔️
Branches %	`70%`	`70%`	`0%` ✔️
Complexity	`15245`	`15245`	`0` ✔️

The following classes have significant coverage changes.

File	Line coverage change	Branch coverage change	Complexity change
Datadog.Trace.Ci.GitInfo	`-17%` ⛔	`-11%` ⛔	`0` ✔️
Datadog.Trace.Ci.CIVisibility	`7%` ✔️	`7%` ✔️	`0` ✔️

View the full reports for further details:

zacharycmontoya added 16 commits July 29, 2022 08:34

Add unit test to ensure that spans that differ by Resource, Service, …

3ea7111

…Operation, Type, HTTPStatusCode, and Synthetics will be stored in different buckets

Update unique key test

9b31d27

Update existing OnlyCollectsTopLevelSpansAndMeasuredSpans test to acc…

fd1fee3

…ount for service entry spans that are child spans

Separate error stats into its own unit test

6d1670a

Separate stats for measured spans into its own unit test

5564124

Implement stats test for relative error

86fbac7

Add unit test for disabling stats

be36a6a

Add test to ensure that stats are not received when the agent doesn't…

d7f70c1

… support them. Since this has not yet been implemented, the test is skipped

Add unit test for dropping P0 traces when stats is enabled. This test…

4ca5ce1

… case is skipped because the feature is not yet implemented

Consolidate a lot of the stats integration testing code into a common…

027d8c1

… helper method

Reorder test cases and append a test code to the cross-tracer test cases

450ceeb

Fix IsDisabledWhenIncompatibleAgentDetected test case so that it curr…

069c579

…ently fails

Add test that ensures stats are only computed once spans are finished

52e3b85

In StatsTests, explicitly state whether the test case expects stats t…

ba688ee

…o be computed

Manually revert the addition of the Dropping P0's test case

1978f56

Manually revert the addition of the endpoint discovery test case

5d80071

zacharycmontoya added the area:tests unit tests, integration tests label Aug 3, 2022

zacharycmontoya requested a review from a team as a code owner August 3, 2022 21:40

zacharycmontoya self-assigned this Aug 3, 2022

zacharycmontoya changed the title ~~Add tests from stats computation specs~~ [Tracer] Add tests for stats computation feature Aug 3, 2022

zacharycmontoya added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) and removed area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) labels Aug 3, 2022

zacharycmontoya mentioned this pull request Aug 3, 2022

[Tracer] Stats computation: Drop P0 traces #3048

Merged

3 tasks

This comment has been minimized.

Sign in to view

zacharycmontoya mentioned this pull request Aug 3, 2022

[Tracer] Add endpoint discovery and dropping of P0 traces to stats computation feature #2988

Closed

This comment has been minimized.

Sign in to view

pierotibou approved these changes Aug 4, 2022

View reviewed changes

Remove statsReceived variable, since the wait event is already a sign…

fed1dfd

…al we can test

This comment has been minimized.

Sign in to view

In StatsTests, make all stats wait events have the same timeout

249981a

Revert change to DenseStore, Kevin and I confirmed the original imple…

f4d05c9

…mentation was correct

bouwkast reviewed Aug 4, 2022

View reviewed changes

Comment thread tracer/test/Datadog.Trace.IntegrationTests/StatsTests.cs

andrewlock approved these changes Aug 8, 2022

View reviewed changes

Comment thread tracer/test/Datadog.Trace.IntegrationTests/StatsTests.cs

zacharycmontoya merged commit 32d8957 into master Aug 23, 2022

zacharycmontoya deleted the zach/stats/existing-tests branch August 23, 2022 18:34

github-actions Bot added this to the vNext milestone Aug 23, 2022

Uh oh!

Conversation

zacharycmontoya commented Aug 3, 2022

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

andrewlock commented Aug 4, 2022

Benchmarks Report 🐌

Benchmark details

Raw results

Faster 🎉 in #3047

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

andrewlock commented Aug 4, 2022

Code Coverage Report 📊

Datadog.Trace Breakdown ✔️

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants