[CI Visibility] - Add Intelligent Test Runner support in MSTest testing framework by tonyredondo · Pull Request #3341 · DataDog/dd-trace-dotnet

tonyredondo · 2022-10-11T13:50:39Z

Summary of changes

This PR adds support for Intelligent Test Runner in the MSTest testing framework integration.

Reason for change

It was the only CI Visibility supported testing framework that was lacking of Intelligent Test Runner support.

Implementation details

We create a new integration to replace the Executor instance of the tests we want to skip. The custom Executor will skip the MethodInfo.Invoke call and return that the test was skipped.

lucaspimentel · 2022-10-14T17:28:18Z

+                // the MethodInfo of the test
+                var executor = instance.TestMethodInfo.TestMethodOptions.Executor;
+                _customTestMethodAttribute ??= new CustomTestMethodAttribute(executor.GetType().Assembly);
+                instance.TestMethodInfo.TestMethodOptions.Executor = DuckType.CreateReverse(executor.GetType(), _customTestMethodAttribute);


zacharycmontoya

Looks great!

andrewlock · 2022-10-14T19:23:44Z

Benchmarks Report 🐌

Benchmarks for #3341 compared to master:

1 benchmarks are faster, with geometric mean 1.154
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net472	778μs	693ns	2.69μs	0.388	3.22 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	534μs	237ns	888ns	0	2.63 KB
#3341	`WriteAndFlushEnrichedTraces`	net472	790μs	509ns	1.9μs	0.393	3.22 KB
#3341	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	508μs	226ns	874ns	0	2.63 KB

Benchmarks.Trace.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #3341

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472	1.154	226.20	195.94

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net472	198ns	0.169ns	0.656ns	0.0727	0	457 B
master	`AllCycleSimpleBody`	netcoreapp3.1	268ns	0.139ns	0.537ns	0.00619	0	456 B
master	`AllCycleMoreComplexBody`	net472	226ns	0.318ns	1.19ns	0.0688	0	433 B
master	`AllCycleMoreComplexBody`	netcoreapp3.1	253ns	0.133ns	0.499ns	0.00592	0	432 B
master	`BodyExtractorSimpleBody`	net472	276ns	0.317ns	1.23ns	0.0574	0	361 B
master	`BodyExtractorSimpleBody`	netcoreapp3.1	245ns	0.147ns	0.55ns	0.0037	0	272 B
master	`BodyExtractorMoreComplexBody`	net472	15.6μs	8.36ns	30.2ns	1.21	0.0157	7.62 KB
master	`BodyExtractorMoreComplexBody`	netcoreapp3.1	12.5μs	4.78ns	18.5ns	0.087	0	6.75 KB
#3341	`AllCycleSimpleBody`	net472	197ns	0.059ns	0.221ns	0.0726	0	457 B
#3341	`AllCycleSimpleBody`	netcoreapp3.1	254ns	0.372ns	1.44ns	0.00626	0	456 B
#3341	`AllCycleMoreComplexBody`	net472	196ns	0.0368ns	0.133ns	0.0688	0	433 B
#3341	`AllCycleMoreComplexBody`	netcoreapp3.1	255ns	0.621ns	2.32ns	0.00593	0	432 B
#3341	`BodyExtractorSimpleBody`	net472	295ns	0.385ns	1.39ns	0.0574	0	361 B
#3341	`BodyExtractorSimpleBody`	netcoreapp3.1	230ns	0.0788ns	0.295ns	0.0037	0	272 B
#3341	`BodyExtractorMoreComplexBody`	net472	15.5μs	12.2ns	45.8ns	1.2	0.0155	7.62 KB
#3341	`BodyExtractorMoreComplexBody`	netcoreapp3.1	13.1μs	10.2ns	35.2ns	0.0911	0	6.75 KB

Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
master	`SendRequest`	netcoreapp3.1	180μs	157ns	608ns	0.27	20.68 KB
#3341	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#3341	`SendRequest`	netcoreapp3.1	181μs	250ns	937ns	0.181	20.68 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net472	1.85μs	0.917ns	3.18ns	0.155	0.000923	979 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.45μs	0.697ns	2.7ns	0.0131	0	968 B
#3341	`ExecuteNonQuery`	net472	1.9μs	0.563ns	2.11ns	0.155	0.00095	979 B
#3341	`ExecuteNonQuery`	netcoreapp3.1	1.45μs	2.76ns	10.7ns	0.0129	0	968 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net472	2.61μs	0.936ns	3.5ns	0.188	1.19 KB
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.646ns	2.42ns	0.0154	1.13 KB
master	`CallElasticsearchAsync`	net472	2.62μs	1.52ns	5.87ns	0.21	1.32 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	1.61μs	0.474ns	1.84ns	0.0169	1.25 KB
#3341	`CallElasticsearch`	net472	2.55μs	0.947ns	3.67ns	0.188	1.19 KB
#3341	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.359ns	1.39ns	0.0154	1.13 KB
#3341	`CallElasticsearchAsync`	net472	2.71μs	0.897ns	3.36ns	0.209	1.32 KB
#3341	`CallElasticsearchAsync`	netcoreapp3.1	1.56μs	0.862ns	3.34ns	0.0163	1.25 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net472	2.79μs	1.35ns	5.23ns	0.23	1.45 KB
master	`ExecuteAsync`	netcoreapp3.1	1.77μs	0.593ns	2.22ns	0.0186	1.38 KB
#3341	`ExecuteAsync`	net472	2.69μs	1.36ns	5.26ns	0.23	1.45 KB
#3341	`ExecuteAsync`	netcoreapp3.1	1.73μs	0.403ns	1.56ns	0.0182	1.38 KB

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net472	5.87μs	1.86ns	6.98ns	0.443	2.8 KB
master	`SendAsync`	netcoreapp3.1	3.62μs	2.54ns	9.51ns	0.0345	2.63 KB
#3341	`SendAsync`	net472	5.87μs	1.99ns	7.43ns	0.445	2.8 KB
#3341	`SendAsync`	netcoreapp3.1	3.68μs	0.939ns	3.39ns	0.0367	2.63 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	3.1μs	4ns	15.5ns	0.298	1.88 KB
master	`EnrichedLog`	netcoreapp3.1	2.47μs	1.13ns	4.39ns	0.0259	1.91 KB
#3341	`EnrichedLog`	net472	3.21μs	1.64ns	6.14ns	0.298	1.88 KB
#3341	`EnrichedLog`	netcoreapp3.1	2.47μs	0.914ns	3.42ns	0.0259	1.91 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	152μs	141ns	547ns	0.678	0.226	4.72 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	253ns	979ns	0.0603	0	4.55 KB
#3341	`EnrichedLog`	net472	152μs	99.3ns	385ns	0.687	0.229	4.72 KB
#3341	`EnrichedLog`	netcoreapp3.1	122μs	145ns	561ns	0.0612	0	4.55 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	5.86μs	2.77ns	10.7ns	0.578	0.00292	3.65 KB
master	`EnrichedLog`	netcoreapp3.1	4.46μs	8.48ns	32.8ns	0.0555	0	3.98 KB
#3341	`EnrichedLog`	net472	5.8μs	1.67ns	6.04ns	0.578	0.00291	3.65 KB
#3341	`EnrichedLog`	netcoreapp3.1	4.6μs	2.37ns	8.86ns	0.0549	0	3.98 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net472	2.37μs	2.36ns	8.52ns	0.223	1.4 KB
master	`SendReceive`	netcoreapp3.1	1.8μs	0.709ns	2.75ns	0.018	1.35 KB
#3341	`SendReceive`	net472	2.3μs	1.15ns	4.14ns	0.222	1.4 KB
#3341	`SendReceive`	netcoreapp3.1	1.87μs	0.781ns	2.92ns	0.0187	1.35 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	5.03μs	4.05ns	15.7ns	0.364	2.3 KB
master	`EnrichedLog`	netcoreapp3.1	4.41μs	1.73ns	6.23ns	0.0242	1.86 KB
#3341	`EnrichedLog`	net472	5.14μs	2.32ns	8.67ns	0.363	2.3 KB
#3341	`EnrichedLog`	netcoreapp3.1	4.28μs	0.965ns	3.34ns	0.0236	1.86 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net472	1.17μs	0.369ns	1.38ns	0.134	842 B
master	`StartFinishSpan`	netcoreapp3.1	944ns	0.311ns	1.12ns	0.0108	792 B
master	`StartFinishScope`	net472	1.45μs	0.481ns	1.8ns	0.146	923 B
master	`StartFinishScope`	netcoreapp3.1	1.05μs	0.225ns	0.811ns	0.0121	912 B
#3341	`StartFinishSpan`	net472	1.19μs	0.343ns	1.24ns	0.134	842 B
#3341	`StartFinishSpan`	netcoreapp3.1	916ns	0.243ns	0.943ns	0.0106	792 B
#3341	`StartFinishScope`	net472	1.37μs	0.185ns	0.667ns	0.146	923 B
#3341	`StartFinishScope`	netcoreapp3.1	1.11μs	0.462ns	1.66ns	0.0122	912 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net472	1.51μs	0.391ns	1.51ns	0.147	923 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.14μs	0.92ns	3.44ns	0.0124	912 B
#3341	`RunOnMethodBegin`	net472	1.47μs	0.729ns	2.73ns	0.146	923 B
#3341	`RunOnMethodBegin`	netcoreapp3.1	1.18μs	1.52ns	5.87ns	0.0122	912 B

andrewlock · 2022-10-14T21:27:30Z

Code Coverage Report 📊

✔️ Merging #3341 into master will not change line coverage
✔️ Merging #3341 into master will not change branch coverage
⛔ Merging #3341 into master will will increase complexity by 26

	master	#3341	Change
Lines	`19528` / `26976`	`19568` / `26998`
Lines %	`72%`	`72%`	`0%` ✔️
Branches	`11605` / `16776`	`11636` / `16800`
Branches %	`69%`	`69%`	`0%` ✔️
Complexity	`18148`	`18174`	`26` ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

	master	#3341	Change
Lines %	`72%`	`72%`	`0%` ✔️
Branches %	`69%`	`69%`	`0%` ✔️
Complexity	`18148`	`18174`	`26` ⛔

The following classes have significant coverage changes.

File	Line coverage change	Branch coverage change	Complexity change
Datadog.Trace.Ci.GitInfo	`-17%` ⛔	`-11%` ⛔	`0` ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.MsTestV2.MsTestIntegration	`-3%` ⚠️	`-11%` ⛔	`10` ⛔
Datadog.Trace.Debugger.PInvoke.DebuggerNativeMethods	`6%` ✔️	`0%` ✔️	`0` ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Http.HttpClient.WinHttpHandler.WinHttpHandlerIntegration	`100%` ✔️	`0%` ✔️	`0` ✔️

The following classes were added in #3341:

File	Line coverage	Branch coverage	Complexity
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.MsTestV2.SkipTestMethodExecutor	`0%`	`100%`	`2`
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.MsTestV2.TestMethodRunnerExecuteTestIntegration	`50%`	`50%`	`6`

View the full reports for further details:

[CIVisibility] Add Intelligent Test Runner support on MSTest

233865d

tonyredondo added the area:ci-visibility label Oct 11, 2022

tonyredondo self-assigned this Oct 11, 2022

tonyredondo requested a review from a team as a code owner October 11, 2022 13:50

tonyredondo changed the title ~~[CIVisibility] Add Intelligent Test Runner support in MSTest testing framework~~ [CIVisibility] - Add Intelligent Test Runner support in MSTest testing framework Oct 11, 2022

tonyredondo changed the title ~~[CIVisibility] - Add Intelligent Test Runner support in MSTest testing framework~~ [CI Visibility] - Add Intelligent Test Runner support in MSTest testing framework Oct 11, 2022

This comment has been minimized.

Sign in to view

Merge branch 'master' into tony/itr-on-mstestv2

8c45471

This comment has been minimized.

Sign in to view

tonyredondo added 2 commits October 13, 2022 15:23

Merge branch 'master' into tony/itr-on-mstestv2

5a70a17

Merge branch 'master' into tony/itr-on-mstestv2

71dcea9

This comment has been minimized.

Sign in to view

fix tests

40e5046

This comment has been minimized.

Sign in to view

lucaspimentel reviewed Oct 14, 2022

View reviewed changes

Comment thread .../Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/MsTestV2/CustomTestMethodAttribute.cs Outdated

lucaspimentel reviewed Oct 14, 2022

View reviewed changes

lucaspimentel approved these changes Oct 14, 2022

View reviewed changes

zacharycmontoya reviewed Oct 14, 2022

View reviewed changes

Comment thread .../Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/MsTestV2/CustomTestMethodAttribute.cs Outdated

zacharycmontoya reviewed Oct 14, 2022

View reviewed changes

Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/MsTestV2/ITestMethod.cs Outdated

zacharycmontoya approved these changes Oct 14, 2022

View reviewed changes

Changes from review.

0554177

tonyredondo merged commit aff3bed into master Oct 14, 2022

tonyredondo deleted the tony/itr-on-mstestv2 branch October 14, 2022 21:30

github-actions Bot added this to the vNext milestone Oct 14, 2022

Uh oh!

Conversation

tonyredondo commented Oct 11, 2022

Summary of changes

Reason for change

Implementation details

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

lucaspimentel Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock commented Oct 14, 2022

Benchmarks Report 🐌

Benchmark details

Raw results

Faster 🎉 in #3341

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

andrewlock commented Oct 14, 2022

Code Coverage Report 📊

Datadog.Trace Breakdown ✔️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants