[CI Visibility] Automatic Test Retries by tonyredondo · Pull Request #6061 · DataDog/dd-trace-dotnet

tonyredondo · 2024-09-23T11:36:35Z

Summary of changes

This PR adds the new automatic tests retries feature for all supported testing frameworks in CI Visibility mode.

Doc: https://docs.google.com/document/d/1Cy6n-CHu0QPV1_Jn9yLCa3aBvgm1JjviFDEq6oUzyuQ/edit#heading=h.1x66s9kgv57m

Reason for change

This is a new CI Visibility feature already implemented in other tracers, .NET was missing the feature.

Implementation details

Reuse part of the early flake detection code and adapt it for the flaky retries scenario according to the RFC.

Test coverage

A new suite of tests (MsTestV2RetriesTests.cs, NUnitRetriesTests.cs and XUnitRetriesTests.cs) with their samples apps were added to test this feature.

andrewlock

LGTM!

andrewlock · 2024-09-26T07:47:12Z

      "tracer\\test\\test-applications\\integrations\\Samples.MongoDB\\Samples.MongoDB.csproj",
      "tracer\\test\\test-applications\\integrations\\Samples.Msmq\\Samples.Msmq.csproj",
      "tracer\\test\\test-applications\\integrations\\Samples.MSTestTests2\\Samples.MSTestTests2.csproj",
+      "tracer\\test\\test-applications\\integrations\\Samples.MSTestTestsRetries\\Samples.MSTestTestsRetries.csproj",


Just a nit, but wondering if we need these duplicate samples, and we couldn't just reuse the MStestTests samples etc? 🤔

I think we could, but I didn't want to mess with the existing tests....

andrewlock · 2024-09-26T08:19:26Z

+            // 1 Module
+            testModules.Should().HaveCount(1);


No snapshots? 😉

no, tags contents are already covered by normal evp and early flake detection tests... I don't see the value of having them here as well compared to the complexity they add. For retries I think the presence of the events is enough.

Co-authored-by: Andrew Lock <[email protected]>

andrewlock · 2024-09-26T11:28:15Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6061) - mean (70ms)  : 67, 73
     .   : milestone, 70,
    master - mean (73ms)  : 66, 79
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (6061) - mean (1,109ms)  : 1064, 1154
     .   : milestone, 1109,
    master - mean (1,109ms)  : 1080, 1138
     .   : milestone, 1109,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6061) - mean (110ms)  : 106, 115
     .   : milestone, 110,
    master - mean (109ms)  : 106, 113
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6061) - mean (779ms)  : 734, 823
     .   : milestone, 779,
    master - mean (777ms)  : 753, 802
     .   : milestone, 777,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6061) - mean (93ms)  : 90, 96
     .   : milestone, 93,
    master - mean (94ms)  : 90, 99
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (6061) - mean (735ms)  : 719, 751
     .   : milestone, 735,
    master - mean (734ms)  : 713, 755
     .   : milestone, 734,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6061) - mean (191ms)  : 187, 195
     .   : milestone, 191,
    master - mean (191ms)  : 188, 193
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6061) - mean (1,193ms)  : 1166, 1220
     .   : milestone, 1193,
    master - mean (1,195ms)  : 1171, 1219
     .   : milestone, 1195,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6061) - mean (276ms)  : 271, 281
     .   : milestone, 276,
    master - mean (277ms)  : 273, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6061) - mean (939ms)  : 922, 956
     .   : milestone, 939,
    master - mean (936ms)  : 921, 951
     .   : milestone, 936,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6061) - mean (265ms)  : 260, 269
     .   : milestone, 265,
    master - mean (264ms)  : 261, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (6061) - mean (929ms)  : 911, 946
     .   : milestone, 929,
    master - mean (931ms)  : 902, 960
     .   : milestone, 931,

bouwkast

Looks good to me overall

Could you add the following to the CODEOWNERS?

/tracer/test/test-applications/integrations/Samples.XUnitRetriesTests/  @DataDog/ci-app-libraries-dotnet
/tracer/test/test-applications/integrations/Samples.NUnitTestsRetries/  @DataDog/ci-app-libraries-dotnet
/tracer/test/test-applications/integrations/Samples.MSTestTestsRetries/  @DataDog/ci-app-libraries-dotnet

[CI Visibility] Flaky test retry

9e435e2

tonyredondo self-assigned this Sep 23, 2024

github-actions Bot added the area:ci-visibility label Sep 23, 2024

tonyredondo added 2 commits September 23, 2024 16:16

Flaky retry for NUnit testing framework

89d34f7

Merge branch 'master' into tony/civisibility-flaky-test-retry

3db7f5d

DataDog deleted a comment from andrewlock Sep 23, 2024

DataDog deleted a comment from datadog-ddstaging Bot Sep 23, 2024

DataDog deleted a comment from andrewlock Sep 23, 2024

tonyredondo added 2 commits September 23, 2024 17:29

Flaky retry for MSTest testing framework

3da76ff

Merge branch 'master' into tony/civisibility-flaky-test-retry

2477263

DataDog deleted a comment from andrewlock Sep 23, 2024

Merge branch 'master' into tony/civisibility-flaky-test-retry

42cde60

DataDog deleted a comment from andrewlock Sep 23, 2024

tonyredondo added 5 commits September 24, 2024 10:53

test refactor

3d6d408

Fix debug logs.

49ca0b7

Merge branch 'master' into tony/civisibility-flaky-test-retry

6838051

Add XUnitRetries tests.

bf301bf

Merge branch 'master' into tony/civisibility-flaky-test-retry

bea62a6

DataDog deleted a comment from andrewlock Sep 24, 2024

tonyredondo added 4 commits September 24, 2024 17:31

Merge branch 'master' into tony/civisibility-flaky-test-retry

a4790d8

NUnitRetries tests, and refactor.

cc65431

Add Samples.MSTestTestsRetries test suite

571e860

Add Retries projects to the PackageVersions files

caa2c7f

DataDog deleted a comment from andrewlock Sep 24, 2024

Fix tests

2b17bb7

DataDog deleted a comment from andrewlock Sep 25, 2024

Fix tests

8459b5a

tonyredondo changed the title ~~[CI Visibility] Flaky test retry~~ [CI Visibility] Automatic Test Retries Sep 25, 2024

tonyredondo marked this pull request as ready for review September 25, 2024 09:19

tonyredondo requested review from a team as code owners September 25, 2024 09:19

anmarchenko reviewed Sep 25, 2024

View reviewed changes

Comment thread tracer/src/Datadog.Trace/Ci/CIVisibility.cs Outdated

anmarchenko approved these changes Sep 25, 2024

View reviewed changes

andrewlock approved these changes Sep 26, 2024

View reviewed changes

tonyredondo and others added 2 commits September 26, 2024 11:29

Apply suggestions from code review

4f18dd8

Co-authored-by: Andrew Lock <[email protected]>

changes from review.

c70d185

DataDog deleted a comment from andrewlock Sep 26, 2024

tonyredondo added 4 commits September 26, 2024 12:19

Changes from review

957e5d0

Changes from review

6cdb889

Changes from review

5d5ae1e

Merge branch 'master' into tony/civisibility-flaky-test-retry

496eb7d

bouwkast approved these changes Sep 26, 2024

View reviewed changes

tonyredondo added 3 commits September 26, 2024 14:10

Update CODEOWNERS

8b017e3

Fix xunit versions

fcd32ee

Merge branch 'master' into tony/civisibility-flaky-test-retry

163cf82

tonyredondo merged commit 5ae119d into master Sep 26, 2024

tonyredondo deleted the tony/civisibility-flaky-test-retry branch September 26, 2024 17:06

github-actions Bot added this to the vNext-v3 milestone Sep 26, 2024

andrewlock added the type:new-feature label Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI Visibility] Automatic Test Retries#6061

[CI Visibility] Automatic Test Retries#6061
tonyredondo merged 26 commits into
masterfrom
tony/civisibility-flaky-test-retry

tonyredondo commented Sep 23, 2024 •

edited

Loading

Uh oh!

Uh oh!

andrewlock left a comment

Uh oh!

andrewlock Sep 26, 2024

Uh oh!

tonyredondo Sep 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewlock Sep 26, 2024

Uh oh!

tonyredondo Sep 26, 2024

Uh oh!

andrewlock commented Sep 26, 2024 •

edited

Loading

Uh oh!

bouwkast left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

tonyredondo commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Uh oh!

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

tonyredondo Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewlock Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

tonyredondo Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

andrewlock commented Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tonyredondo commented Sep 23, 2024 •

edited

Loading

andrewlock commented Sep 26, 2024 •

edited

Loading