Skip to content

[CI Visibility] honor EFD retry mode over ATR#8288

Merged
tonyredondo merged 3 commits into
masterfrom
codex/fix-efd-atr-retry-precedence
Mar 18, 2026
Merged

[CI Visibility] honor EFD retry mode over ATR#8288
tonyredondo merged 3 commits into
masterfrom
codex/fix-efd-atr-retry-precedence

Conversation

@tonyredondo

@tonyredondo tonyredondo commented Mar 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Fix the combined EFD + ATR case across .NET test frameworks so retry metadata matches the retry executor that actually ran.

What Changed

  • NUnit: keep existing retry scheduling, but overwrite retry tags from the concrete retry behavior before closing the test span.
  • MSTest v2: keep existing retry scheduling, but derive retry tags from RetryState before the test span is closed.
  • xUnit v2/v3: add an explicit selected retry mode so new tests prefer EFD over ATR when both are enabled, and use that mode consistently for execution counts and retry tags.
  • EVP tests: add combined all_efd_with_atr scenarios for NUnit, MSTest v2, xUnit v2, and xUnit v3.
  • Snapshots: add verified snapshots for the new combined scenarios.

Framework Notes

  • NUnit and MSTest v2 were tag-correction fixes.
  • xUnit v2/v3 required executor-precedence changes so EFD wins over ATR for new tests.

Snapshot Check

The new all_efd_with_atr snapshots match the intended behavior across frameworks:

  • NUnit: 249 test events total, 216 retry events, all 216 tagged test.retry_reason: efd, 0 tagged atr.
  • MSTest v2 (post_2_2_4): 148 total, 126 retry events, all 126 tagged test.retry_reason: efd, 0 tagged atr.
  • xUnit v2: 124 total, 108 retry events, all 108 tagged test.retry_reason: efd, 0 tagged atr.
  • xUnit v3: 124 total, 108 retry events, all 108 tagged test.retry_reason: efd, 0 tagged atr.

Concrete examples from the snapshots:

  • SimplePassTest appears 10 times in every framework snapshot, which confirms EFD execution is active because ATR would not retry a passing test.
  • SimpleErrorParameterizedTest appears 30 times in NUnit/xUnit because 3 parameter sets each get the 10 EFD executions.
  • Final outcome tags still stay on the terminal event, and permanently failing tests still carry test.has_failed_all_retries: true.
  • For NUnit, xUnit v2, xUnit v3, and the locally verified MSTest post_2_2_4 path, the new all_efd_with_atr snapshots are byte-for-byte identical to the existing all_efd baselines, which is the expected outcome.

Verification

  • dotnet test tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/Datadog.Trace.ClrProfiler.IntegrationTests.csproj -c Release --no-build --framework net6.0 --filter "FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.NUnitEvpTests.EarlyFlakeDetection|FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.TcpXUnitEvpTests.EarlyFlakeDetection"
  • dotnet test tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/Datadog.Trace.ClrProfiler.IntegrationTests.csproj -c Release --no-build --framework net8.0 --filter "FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.XUnitEvpTestsV3.EarlyFlakeDetection"
  • dotnet test tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/Datadog.Trace.ClrProfiler.IntegrationTests.csproj -c Release --no-build --framework net9.0 --filter "FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.MsTestV2EvpTests.EarlyFlakeDetection"
  • dotnet test tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/Datadog.Trace.ClrProfiler.IntegrationTests.csproj -c Release --no-build --framework net9.0 --filter "FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.NUnitRetriesTests.FlakyRetries|FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.XUnitRetriesTests.FlakyRetries|FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.MsTestV2RetriesTests.FlakyRetries|FullyQualifiedName~Datadog.Trace.ClrProfiler.IntegrationTests.CI.XUnitRetriesTestsV3.FlakyRetries"

@pr-commenter

pr-commenter Bot commented Mar 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-18 13:57:46

Comparing candidate commit 64b83c9 in PR branch codex/fix-efd-atr-retry-precedence with baseline commit 2a80e17 in branch master.

Found 10 performance improvements and 9 performance regressions! Performance is the same for 159 metrics, 14 unstable metrics.

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟩 execution_time [-81.577ms; -81.301ms] or [-40.018%; -39.883%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟥 execution_time [+15.610ms; +21.765ms] or [+7.806%; +10.885%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟩 execution_time [-21.229ms; -15.241ms] or [-9.734%; -6.988%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

  • 🟥 throughput [-625.975op/s; -523.895op/s] or [-6.176%; -5.168%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟥 execution_time [+19.978ms; +20.421ms] or [+11.057%; +11.301%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

  • 🟥 execution_time [+98.585ms; +106.103ms] or [+100.472%; +108.133%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 throughput [-151.455op/s; -127.374op/s] or [-13.560%; -11.404%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+10.613ms; +15.369ms] or [+6.764%; +9.795%]
  • 🟥 throughput [-148.480op/s; -101.792op/s] or [-9.096%; -6.236%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 throughput [-342.086op/s; -257.195op/s] or [-20.535%; -15.439%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0

  • 🟩 execution_time [-180.469µs; -171.665µs] or [-11.443%; -10.884%]
  • 🟩 throughput [+77.850op/s; +81.480op/s] or [+12.278%; +12.850%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟩 execution_time [-16.307ms; -13.680ms] or [-9.514%; -7.981%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

  • 🟩 execution_time [-35.123ms; -24.394ms] or [-31.255%; -21.708%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

  • 🟩 execution_time [-23.479ms; -19.452ms] or [-10.869%; -9.005%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1

  • 🟥 execution_time [+10.767ms; +16.689ms] or [+5.396%; +8.364%]
  • 🟩 throughput [+49016.490op/s; +74100.273op/s] or [+6.029%; +9.114%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1

  • 🟩 throughput [+67784.913op/s; +95590.024op/s] or [+7.103%; +10.016%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

  • 🟩 execution_time [-19.398ms; -15.376ms] or [-9.073%; -7.192%]

@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Mar 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8288) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration71.33 ± (71.37 - 71.59) ms69.16 ± (69.17 - 69.43) ms-3.0%✅
.NET Framework 4.8 - Bailout
duration75.64 ± (75.53 - 75.77) ms72.92 ± (72.81 - 73.17) ms-3.6%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1058.55 ± (1058.16 - 1063.44) ms1045.26 ± (1051.51 - 1060.69) ms-1.3%✅
.NET Core 3.1 - Baseline
process.internal_duration_ms22.31 ± (22.27 - 22.35) ms21.94 ± (21.91 - 21.97) ms-1.7%✅
process.time_to_main_ms82.43 ± (82.29 - 82.57) ms80.28 ± (80.12 - 80.45) ms-2.6%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.90 ± (10.89 - 10.90) MB10.92 ± (10.92 - 10.93) MB+0.2%✅⬆️
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms22.26 ± (22.23 - 22.30) ms21.87 ± (21.84 - 21.89) ms-1.8%✅
process.time_to_main_ms84.15 ± (84.02 - 84.29) ms81.25 ± (81.08 - 81.41) ms-3.5%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.93 ± (10.93 - 10.94) MB10.95 ± (10.95 - 10.96) MB+0.2%✅⬆️
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms223.99 ± (222.77 - 225.21) ms219.46 ± (218.02 - 220.90) ms-2.0%✅
process.time_to_main_ms481.26 ± (480.48 - 482.04) ms475.18 ± (474.50 - 475.86) ms-1.3%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed47.15 ± (47.12 - 47.18) MB47.24 ± (47.21 - 47.26) MB+0.2%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.1%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms21.02 ± (20.99 - 21.06) ms20.83 ± (20.80 - 20.87) ms-0.9%✅
process.time_to_main_ms72.00 ± (71.84 - 72.17) ms69.78 ± (69.65 - 69.91) ms-3.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.62 ± (10.62 - 10.62) MB10.65 ± (10.65 - 10.65) MB+0.3%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms20.93 ± (20.89 - 20.97) ms20.82 ± (20.79 - 20.84) ms-0.5%✅
process.time_to_main_ms72.61 ± (72.48 - 72.74) ms70.97 ± (70.85 - 71.09) ms-2.3%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.73 ± (10.72 - 10.73) MB10.77 ± (10.77 - 10.77) MB+0.4%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms207.09 ± (205.55 - 208.63) ms202.34 ± (200.68 - 204.00) ms-2.3%✅
process.time_to_main_ms482.32 ± (481.68 - 482.96) ms474.22 ± (473.52 - 474.93) ms-1.7%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed49.09 ± (49.04 - 49.14) MB48.96 ± (48.90 - 49.02) MB-0.3%✅
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)-0.1%✅
.NET 8 - Baseline
process.internal_duration_ms19.26 ± (19.23 - 19.30) ms19.06 ± (19.03 - 19.09) ms-1.1%✅
process.time_to_main_ms71.03 ± (70.90 - 71.15) ms69.03 ± (68.91 - 69.15) ms-2.8%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.67 ± (7.66 - 7.67) MB7.69 ± (7.69 - 7.70) MB+0.4%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms19.28 ± (19.24 - 19.31) ms19.04 ± (19.01 - 19.07) ms-1.2%✅
process.time_to_main_ms72.66 ± (72.52 - 72.79) ms70.26 ± (70.15 - 70.36) ms-3.3%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.72 ± (7.71 - 7.73) MB7.78 ± (7.77 - 7.80) MB+0.9%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms161.71 ± (160.76 - 162.66) ms158.07 ± (157.15 - 158.99) ms-2.3%✅
process.time_to_main_ms459.59 ± (459.01 - 460.18) ms452.41 ± (451.89 - 452.94) ms-1.6%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed36.44 ± (36.43 - 36.46) MB36.56 ± (36.54 - 36.58) MB+0.3%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.2%✅⬆️

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration193.87 ± (193.82 - 194.73) ms199.48 ± (199.57 - 201.18) ms+2.9%✅⬆️
.NET Framework 4.8 - Bailout
duration196.84 ± (196.71 - 197.49) ms199.36 ± (199.58 - 200.49) ms+1.3%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1153.98 ± (1157.03 - 1166.64) ms1172.12 ± (1172.63 - 1181.03) ms+1.6%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms187.96 ± (187.65 - 188.27) ms191.24 ± (190.80 - 191.68) ms+1.7%✅⬆️
process.time_to_main_ms80.85 ± (80.61 - 81.10) ms82.12 ± (81.89 - 82.34) ms+1.6%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.18 ± (16.15 - 16.21) MB16.14 ± (16.12 - 16.17) MB-0.2%✅
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (20 - 20)+0.6%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms188.29 ± (187.93 - 188.64) ms190.06 ± (189.63 - 190.49) ms+0.9%✅⬆️
process.time_to_main_ms82.32 ± (82.18 - 82.45) ms83.24 ± (83.07 - 83.42) ms+1.1%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.26 ± (16.23 - 16.28) MB16.21 ± (16.18 - 16.25) MB-0.3%✅
runtime.dotnet.threads.count21 ± (20 - 21)21 ± (20 - 21)-0.3%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms396.19 ± (394.49 - 397.88) ms398.63 ± (396.94 - 400.31) ms+0.6%✅⬆️
process.time_to_main_ms474.53 ± (473.93 - 475.14) ms483.41 ± (482.42 - 484.40) ms+1.9%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed57.86 ± (57.70 - 58.01) MB58.03 ± (57.91 - 58.15) MB+0.3%✅⬆️
runtime.dotnet.threads.count30 ± (29 - 30)30 ± (30 - 30)+0.3%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms192.80 ± (192.38 - 193.22) ms197.92 ± (197.26 - 198.59) ms+2.7%✅⬆️
process.time_to_main_ms70.31 ± (70.15 - 70.47) ms72.05 ± (71.75 - 72.35) ms+2.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.18 ± (16.06 - 16.30) MB16.34 ± (16.32 - 16.36) MB+1.0%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)19 ± (19 - 19)+4.7%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms191.71 ± (191.43 - 191.99) ms195.50 ± (195.02 - 195.98) ms+2.0%✅⬆️
process.time_to_main_ms71.09 ± (71.01 - 71.17) ms72.22 ± (72.04 - 72.40) ms+1.6%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.17 ± (16.03 - 16.31) MB16.41 ± (16.38 - 16.44) MB+1.5%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)20 ± (20 - 20)+5.3%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms428.17 ± (426.41 - 429.94) ms435.92 ± (434.08 - 437.76) ms+1.8%✅⬆️
process.time_to_main_ms476.75 ± (475.94 - 477.56) ms486.28 ± (485.00 - 487.57) ms+2.0%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed60.31 ± (60.25 - 60.37) MB60.04 ± (59.95 - 60.14) MB-0.4%✅
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 31)+0.2%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms190.92 ± (190.53 - 191.32) ms192.66 ± (192.27 - 193.06) ms+0.9%✅⬆️
process.time_to_main_ms70.34 ± (70.13 - 70.55) ms71.10 ± (70.86 - 71.33) ms+1.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.71 ± (11.68 - 11.73) MB11.73 ± (11.71 - 11.75) MB+0.2%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)-0.2%✅
.NET 8 - Bailout
process.internal_duration_ms190.05 ± (189.61 - 190.49) ms194.25 ± (193.74 - 194.75) ms+2.2%✅⬆️
process.time_to_main_ms71.08 ± (70.91 - 71.25) ms72.65 ± (72.46 - 72.83) ms+2.2%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.80 ± (11.77 - 11.83) MB11.73 ± (11.71 - 11.75) MB-0.6%✅
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.7%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms353.81 ± (352.52 - 355.11) ms361.45 ± (359.89 - 363.01) ms+2.2%✅⬆️
process.time_to_main_ms453.97 ± (453.38 - 454.57) ms463.40 ± (462.56 - 464.25) ms+2.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed48.35 ± (48.32 - 48.38) MB48.45 ± (48.41 - 48.49) MB+0.2%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.1%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (69ms)  : 68, 71
    master - mean (71ms)  : 70, 73

    section Bailout
    This PR (8288) - mean (73ms)  : 71, 75
    master - mean (76ms)  : 75, 77

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (1,056ms)  : 987, 1126
    master - mean (1,061ms)  : 1023, 1099
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (108ms)  : 105, 111
    master - mean (111ms)  : 109, 114

    section Bailout
    This PR (8288) - mean (109ms)  : 107, 111
    master - mean (113ms)  : 111, 114

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (730ms)  : 714, 747
    master - mean (744ms)  : 724, 764
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (96ms)  : 93, 99
    master - mean (99ms)  : 96, 102

    section Bailout
    This PR (8288) - mean (97ms)  : 95, 99
    master - mean (99ms)  : 97, 101

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (711ms)  : 677, 744
    master - mean (723ms)  : 676, 769
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (95ms)  : 92, 98
    master - mean (98ms)  : 95, 100

    section Bailout
    This PR (8288) - mean (96ms)  : 95, 98
    master - mean (99ms)  : 98, 101

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (650ms)  : 624, 677
    master - mean (657ms)  : 634, 681
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (200ms)  : 188, 212
    master - mean (194ms)  : 189, 200

    section Bailout
    This PR (8288) - mean (200ms)  : 194, 206
    master - mean (197ms)  : 193, 201

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (1,177ms)  : 1115, 1238
    master - mean (1,162ms)  : 1090, 1233
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (282ms)  : 275, 290
    master - mean (277ms)  : 273, 282

    section Bailout
    This PR (8288) - mean (282ms)  : 277, 288
    master - mean (279ms)  : 273, 285

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (916ms)  : 874, 957
    master - mean (904ms)  : 873, 936
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (279ms)  : 265, 292
    master - mean (271ms)  : 265, 277

    section Bailout
    This PR (8288) - mean (276ms)  : 269, 283
    master - mean (271ms)  : 268, 274

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (956ms)  : 915, 998
    master - mean (938ms)  : 903, 973
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8288) - mean (274ms)  : 268, 280
    master - mean (271ms)  : 264, 278

    section Bailout
    This PR (8288) - mean (277ms)  : 267, 286
    master - mean (271ms)  : 265, 276

    section CallTarget+Inlining+NGEN
    This PR (8288) - mean (858ms)  : 825, 892
    master - mean (840ms)  : 810, 870
Loading

@tonyredondo tonyredondo changed the title fix(ci): honor EFD retry mode over ATR [CI Visibility] honor EFD retry mode over ATR Mar 11, 2026
@anmarchenko

anmarchenko commented Mar 12, 2026

Copy link
Copy Markdown
Member

E2E Test Report: SUCCESS ✅

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

  • Method: Local testing with peasy-net playground (xUnit, 319 tests)
  • Revision: 8a32baa
  • Artifact source: Azure DevOps build #197136 (runner-dotnet-tool artifact)
  • Mock backend: mockdog (local mock Datadog backend with pluggable scenarios)

Results

Test 1: Default scenario (baseline regression check)

Check Status
All 319 tests pass ✅
342 events (319 tests, 21 suites, 1 module, 1 session) ✅
Telemetry valid (1 session, 319 tests, 5 payloads) ✅
All 34 default assertions pass ✅

Test 2: Combined EFD + ATR scenario (PR-specific verification)

Scenario: early_flake_detection.enabled: true + flaky_test_retries_enabled: true with slow_test_retries: {5s: 2} and all tests treated as "new" (not in known tests list).

Check Status
EFD activates when both EFD and ATR enabled ✅
New tests detected (test.is_new: true) ✅
EFD retries executed (5 original + 5 retries = 10 spans) ✅
Retry reason is efd (not atr) on all retry spans ✅
Session tagged with test.early_flake.enabled: true ✅
Original executions have no retry tags ✅

Key Verification

The core fix was confirmed: when both EFD and ATR are enabled, retry spans for new tests carry test.retry_reason: efd, not atr. This was verified on 5 xUnit test methods from Peasy.Core.Tests.ServiceBaseTests, each showing exactly 1 retry with correct EFD tags.

Test Methodology

  1. Installed dd-trace .NET tool from Azure DevOps build #197136 artifacts (PR branch build)
  2. Ran peasy-net playground with default scenario — verified no regressions (all 34 assertions pass)
  3. Created combined EFD+ATR mockdog scenario with both features enabled
  4. Ran filtered subset of tests (5 xUnit test methods) against mockdog
  5. Inspected span tags via query-spans to confirm test.retry_reason: efd on all retries

This E2E test was performed by Shepherd — autonomous QA agent for Datadog Test Optimization

andrewlock
andrewlock approved these changes Mar 16, 2026
View reviewed changes

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as far as I can tell, though I didn't verify the logic extensively. Just a few (non-blocking) nits and suggestions for cleanup/de-dupe, feel free to take/ignore 😄

Comment thread ...ce/ClrProfiler/AutoInstrumentation/Testing/MsTestV2/TestMethodAttributeExecuteIntegration.cs Outdated
Comment thread ...er/AutoInstrumentation/Testing/XUnit/V3/XUnitTestMethodRunnerBaseRunTestCaseV3Integration.cs Outdated
Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/XUnit/TestCaseMetadata.cs Outdated
Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/XUnit/XUnitIntegration.cs Outdated
Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/XUnit/XUnitIntegration.cs Outdated
Comment thread tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/XUnit/XUnitIntegration.cs Outdated
Comment thread ...og.Trace/ClrProfiler/AutoInstrumentation/Testing/XUnit/XUnitTestRunnerRunAsyncIntegration.cs Outdated
Comment thread ...MsTestV2EvpTests.EarlyFlakeDetection_packageVersion=post_2_2_4_all_efd_with_atr.verified.txt
@tonyredondo
tonyredondo marked this pull request as ready for review March 16, 2026 17:05
@tonyredondo
tonyredondo requested review from a team as code owners March 16, 2026 17:05
@tonyredondo
tonyredondo force-pushed the codex/fix-efd-atr-retry-precedence branch from 12c5b58 to 64b83c9 Compare March 18, 2026 13:14
@tonyredondo
tonyredondo merged commit 31d5db8 into master Mar 18, 2026
139 checks passed
@tonyredondo
tonyredondo deleted the codex/fix-efd-atr-retry-precedence branch March 18, 2026 15:42
@github-actions github-actions Bot added this to the vNext-v3 milestone Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes
@andrewlock andrewlock andrewlock approved these changes

Assignees

No one assigned

Labels

area:ci-visibility

Projects

None yet

Milestone

3.40.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tonyredondo @anmarchenko @andrewlock