Skip to content

[CI Visibility] Intelligent test runner#3075

Merged
tonyredondo merged 68 commits into
masterfrom
tony/itr-api-client
Sep 19, 2022
Merged

[CI Visibility] Intelligent test runner#3075
tonyredondo merged 68 commits into
masterfrom
tony/itr-api-client

Conversation

@tonyredondo

@tonyredondo tonyredondo commented Aug 12, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

Summary of changes

This PR introduces the new Intelligent Test Runner CI Visibility feature.

Reason for change

This is a new feature being developed by the CI Visibility team.

Implementation details

With the Git Metadata + Code Coverage the backend is able to tell the testing framework which tests should be skipped. This is done by changes in the testing framework instrumentation.

Test coverage

This is the first PR in this matter we are still testing internally and making changes to the backend. A proper test suite will come in subsequence PRs.

This feature is disabled by default and not ready to be GA.

@tonyredondo tonyredondo self-assigned this Aug 12, 2022
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
albertvaka
albertvaka reviewed Sep 12, 2022
View reviewed changes
Comment thread tracer/src/Datadog.Trace/LifetimeManager.cs Outdated
Comment thread tracer/src/Datadog.Trace/LifetimeManager.cs
Comment thread ...og.Trace/ClrProfiler/AutoInstrumentation/Testing/XUnit/XUnitTestRunnerRunAsyncIntegration.cs Outdated
Comment thread tracer/src/Datadog.Trace/LifetimeManager.cs
Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/NUnit/NUnitIntegration.cs Outdated
Comment thread tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs
Comment thread tracer/src/Datadog.Trace/Ci/CIVisibility.cs Outdated
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
albertvaka
albertvaka approved these changes Sep 13, 2022
View reviewed changes

@albertvaka albertvaka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
pierotibou
pierotibou approved these changes Sep 19, 2022
View reviewed changes

@pierotibou pierotibou left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a few nits

Comment thread tracer/src/Datadog.Trace/Ci/CIVisibility.cs Outdated
Comment thread tracer/src/Datadog.Trace/Ci/CIVisibility.cs Outdated
Comment thread tracer/src/Datadog.Trace/Ci/CIVisibility.cs Outdated
Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/NUnit/NUnitIntegration.cs
Comment thread tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs Outdated
Comment thread tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs Outdated
Comment thread tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs
@andrewlock

andrewlock commented Sep 19, 2022

Copy link
Copy Markdown
Member

Benchmarks Report 🐌

Benchmarks for #3075 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 717μs 325ns 1.21μs 0.359 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 469μs 207ns 773ns 0 0 0 2.58 KB
#3075 WriteAndFlushEnrichedTraces net472 719μs 663ns 2.57μs 0.357 0 0 3.18 KB
#3075 WriteAndFlushEnrichedTraces netcoreapp3.1 469μs 352ns 1.36μs 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 180ns 0.084ns 0.291ns 0.0676 9.02E-05 0 425 B
master AllCycleSimpleBody netcoreapp3.1 240ns 0.316ns 1.22ns 0.00591 0 0 424 B
master AllCycleMoreComplexBody net472 182ns 0.45ns 1.68ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 238ns 0.186ns 0.719ns 0.00552 0 0 400 B
master BodyExtractorSimpleBody net472 286ns 0.337ns 1.26ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 242ns 0.365ns 1.41ns 0.00377 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.4μs 10.1ns 39.1ns 1.2 0.0215 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.8μs 9.84ns 38.1ns 0.0887 0 0 6.75 KB
#3075 AllCycleSimpleBody net472 183ns 0.185ns 0.717ns 0.0675 0 0 425 B
#3075 AllCycleSimpleBody netcoreapp3.1 236ns 0.314ns 1.22ns 0.00589 0 0 424 B
#3075 AllCycleMoreComplexBody net472 185ns 0.19ns 0.736ns 0.0637 0 0 401 B
#3075 AllCycleMoreComplexBody netcoreapp3.1 234ns 0.183ns 0.711ns 0.00553 0 0 400 B
#3075 BodyExtractorSimpleBody net472 257ns 0.241ns 0.934ns 0.0574 0 0 361 B
#3075 BodyExtractorSimpleBody netcoreapp3.1 220ns 0.197ns 0.739ns 0.00376 0 0 272 B
#3075 BodyExtractorMoreComplexBody net472 14.7μs 15ns 58.3ns 1.21 0.0147 0 7.62 KB
#3075 BodyExtractorMoreComplexBody netcoreapp3.1 11.6μs 12.5ns 46.7ns 0.0919 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 179μs 281ns 1.09μs 0.269 0 0 20.58 KB
#3075 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3075 SendRequest netcoreapp3.1 180μs 236ns 914ns 0.272 0 0 20.59 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.8μs 1.07ns 3.85ns 0.15 0.000896 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.49μs 3.02ns 11.7ns 0.0126 0 0 936 B
#3075 ExecuteNonQuery net472 1.89μs 0.957ns 3.58ns 0.15 0.000944 0 947 B
#3075 ExecuteNonQuery netcoreapp3.1 1.48μs 2.33ns 8.74ns 0.0124 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.64μs 1.46ns 5.47ns 0.182 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.58μs 0.738ns 2.86ns 0.0148 0 0 1.1 KB
master CallElasticsearchAsync net472 2.69μs 2.01ns 7.79ns 0.205 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.61μs 0.572ns 2.14ns 0.0161 0 0 1.22 KB
#3075 CallElasticsearch net472 2.7μs 2.37ns 8.87ns 0.182 0 0 1.16 KB
#3075 CallElasticsearch netcoreapp3.1 1.59μs 1.37ns 5.29ns 0.0143 0 0 1.1 KB
#3075 CallElasticsearchAsync net472 2.8μs 1.82ns 7.03ns 0.205 0 0 1.29 KB
#3075 CallElasticsearchAsync netcoreapp3.1 1.67μs 0.652ns 2.53ns 0.0159 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.81μs 1.03ns 4ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.64μs 2.59ns 9.68ns 0.0176 0 0 1.34 KB
#3075 ExecuteAsync net472 2.74μs 4.13ns 16ns 0.225 0 0 1.42 KB
#3075 ExecuteAsync netcoreapp3.1 1.65μs 4.22ns 16.4ns 0.0182 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.65μs 7.88ns 30.5ns 0.438 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.64μs 5.39ns 20.2ns 0.035 0 0 2.6 KB
#3075 SendAsync net472 5.76μs 10.1ns 39ns 0.439 0 0 2.77 KB
#3075 SendAsync netcoreapp3.1 3.62μs 5.95ns 22.2ns 0.035 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.12μs 3.21ns 12.4ns 0.287 0.00154 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.57μs 0.843ns 3.04ns 0.0256 0 0 1.85 KB
#3075 EnrichedLog net472 3.25μs 4.12ns 15.9ns 0.288 0 0 1.81 KB
#3075 EnrichedLog netcoreapp3.1 2.56μs 2.26ns 8.74ns 0.0256 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 152μs 238ns 923ns 0.693 0.231 0 4.66 KB
master EnrichedLog netcoreapp3.1 118μs 129ns 500ns 0.0589 0 0 4.49 KB
#3075 EnrichedLog net472 153μs 251ns 971ns 0.689 0.23 0 4.67 KB
#3075 EnrichedLog netcoreapp3.1 118μs 251ns 971ns 0 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.73μs 17.2ns 66.6ns 0.569 0.00286 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.64μs 4.58ns 17.8ns 0.0537 0 0 3.91 KB
#3075 EnrichedLog net472 5.87μs 10.2ns 39.6ns 0.569 0.00286 0 3.59 KB
#3075 EnrichedLog netcoreapp3.1 4.47μs 14.4ns 53.8ns 0.0534 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.28μs 3.07ns 11.9ns 0.218 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.86μs 0.833ns 3.23ns 0.0177 0 0 1.32 KB
#3075 SendReceive net472 2.26μs 2.45ns 9.5ns 0.217 0 0 1.37 KB
#3075 SendReceive netcoreapp3.1 1.87μs 2.67ns 9.61ns 0.0176 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.08μs 1.71ns 6.61ns 0.353 0.00254 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.29μs 3.16ns 11.8ns 0.0235 0 0 1.8 KB
#3075 EnrichedLog net472 5.13μs 1.99ns 7.71ns 0.353 0 0 2.23 KB
#3075 EnrichedLog netcoreapp3.1 4.41μs 2.46ns 9.54ns 0.0243 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.11μs 0.81ns 3.14ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 977ns 0.404ns 1.46ns 0.0103 0 0 760 B
master StartFinishScope net472 1.46μs 0.953ns 3.69ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.03μs 0.383ns 1.43ns 0.0118 0 0 880 B
#3075 StartFinishSpan net472 1.09μs 0.858ns 3.32ns 0.129 0 0 810 B
#3075 StartFinishSpan netcoreapp3.1 900ns 0.358ns 1.34ns 0.0104 0 0 760 B
#3075 StartFinishScope net472 1.36μs 0.714ns 2.67ns 0.141 0 0 891 B
#3075 StartFinishScope netcoreapp3.1 1.09μs 0.388ns 1.45ns 0.012 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.44μs 0.704ns 2.73ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.22μs 0.391ns 1.41ns 0.0122 0 0 880 B
#3075 RunOnMethodBegin net472 1.42μs 0.624ns 2.25ns 0.141 0 0 891 B
#3075 RunOnMethodBegin netcoreapp3.1 1.31μs 0.452ns 1.69ns 0.012 0 0 880 B

@tonyredondo
tonyredondo merged commit c2d4413 into master Sep 19, 2022
@tonyredondo
tonyredondo deleted the tony/itr-api-client branch September 19, 2022 17:07
@github-actions github-actions Bot added this to the vNext milestone Sep 19, 2022
@andrewlock

andrewlock commented Sep 19, 2022

Copy link
Copy Markdown
Member

Code Coverage Report 📊

✔️ Merging #3075 into master will not change line coverage
✔️ Merging #3075 into master will will increase branch coverage by 1%
⛔ Merging #3075 into master will will increase complexity by 184

master #3075 Change
Lines 17903 / 24595 18239 / 24888
Lines % 73% 73% 0% ✔️
Branches 10421 / 15066 10720 / 15254
Branches % 69% 70% 1% ✔️
Complexity 16292 16476 184 ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3075 Change
Lines % 73% 73% 0% ✔️
Branches % 69% 70% 1% ✔️
Complexity 16292 16476 184 ⛔

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.Coverage.CoverageReporter -50% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Ci.Coverage.CoverageContextContainer -48% ⛔ -25% ⛔ 0 ✔️
Datadog.Trace.Ci.Coverage.CoverageEventHandler -35% ⛔ -12% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.XUnit.XUnitTestRunnerRunAsyncIntegration -25% ⛔ -20% ⛔ 6 ⛔
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.Common -24% ⛔ -19% ⛔ 37 ⛔
Datadog.Trace.Ci.Configuration.CIVisibilitySettings -15% ⛔ -2% ⚠️ 8 ⛔
Datadog.Trace.LifetimeManager -15% ⛔ 0% ✔️ 8 ⛔
Datadog.Trace.Ci.Coverage.DefaultCoverageEventHandler -10% ⛔ -8% ⛔ 0 ✔️
Datadog.Trace.Ci.CIVisibility -9% ⛔ -13% ⛔ 31 ⛔
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.NUnit.NUnitIntegration -3% ⚠️ -8% ⛔ 12 ⛔
...And 32 more

The following classes were added in #3075:

File Line coverage Branch coverage Complexity
Datadog.Trace.Ci.SkippableTest 0% 0% 3
Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.NUnit.NUnitWorkItemIntegration 38% 71% 14

View the full reports for further details:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pierotibou pierotibou pierotibou approved these changes
+1 more reviewer
@albertvaka albertvaka albertvaka approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tonyredondo tonyredondo

Labels

None yet

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

4 participants

@tonyredondo @andrewlock @albertvaka @pierotibou