[CI Visibility] Avoid failing unfinished tests#6063
Conversation
Datadog ReportBranch report: ✅ 0 Failed, 364281 Passed, 2083 Skipped, 15h 56m 24.7s Total Time |
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6063) - mean (70ms) : 67, 72
. : milestone, 70,
master - mean (70ms) : 68, 72
. : milestone, 70,
section CallTarget+Inlining+NGEN
This PR (6063) - mean (1,096ms) : 1075, 1117
. : milestone, 1096,
master - mean (1,099ms) : 1081, 1117
. : milestone, 1099,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6063) - mean (108ms) : 106, 110
. : milestone, 108,
master - mean (108ms) : 105, 110
. : milestone, 108,
section CallTarget+Inlining+NGEN
This PR (6063) - mean (766ms) : 755, 777
. : milestone, 766,
master - mean (765ms) : 750, 780
. : milestone, 765,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6063) - mean (92ms) : 90, 94
. : milestone, 92,
master - mean (91ms) : 89, 93
. : milestone, 91,
section CallTarget+Inlining+NGEN
This PR (6063) - mean (726ms) : 708, 745
. : milestone, 726,
master - mean (721ms) : 705, 738
. : milestone, 721,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6063) - mean (190ms) : 187, 194
. : milestone, 190,
master - mean (190ms) : 187, 194
. : milestone, 190,
section CallTarget+Inlining+NGEN
This PR (6063) - mean (1,196ms) : 1170, 1221
. : milestone, 1196,
master - mean (1,193ms) : 1174, 1212
. : milestone, 1193,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6063) - mean (276ms) : 272, 279
. : milestone, 276,
master - mean (276ms) : 271, 280
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (6063) - mean (940ms) : 924, 956
. : milestone, 940,
master - mean (938ms) : 915, 961
. : milestone, 938,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6063) - mean (265ms) : 262, 268
. : milestone, 265,
master - mean (264ms) : 261, 268
. : milestone, 264,
section CallTarget+Inlining+NGEN
This PR (6063) - mean (926ms) : 902, 949
. : milestone, 926,
master - mean (921ms) : 905, 937
. : milestone, 921,
|
Throughput/Crank Report ⚡Throughput results for AspNetCoreSimpleController comparing the following branches/commits: Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red. Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards! gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6063) (10.947M) : 0, 10946829
master (11.116M) : 0, 11115802
benchmarks/2.9.0 (11.081M) : 0, 11080577
section Automatic
This PR (6063) (7.341M) : 0, 7341122
master (7.421M) : 0, 7420885
benchmarks/2.9.0 (7.732M) : 0, 7732233
section Trace stats
master (7.731M) : 0, 7731186
section Manual
master (11.004M) : 0, 11003658
section Manual + Automatic
This PR (6063) (6.766M) : 0, 6765987
master (6.830M) : 0, 6830012
section DD_TRACE_ENABLED=0
master (10.198M) : 0, 10198361
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6063) (9.614M) : 0, 9613788
benchmarks/2.9.0 (9.798M) : 0, 9798067
section Automatic
This PR (6063) (6.570M) : 0, 6569901
section Manual + Automatic
This PR (6063) (6.016M) : 0, 6015653
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6063) (10.182M) : 0, 10181840
master (9.921M) : 0, 9920740
benchmarks/2.9.0 (10.067M) : 0, 10067315
section Automatic
This PR (6063) (6.857M) : 0, 6857301
master (6.723M) : 0, 6723182
benchmarks/2.9.0 (7.552M) : 0, 7552193
section Trace stats
master (7.251M) : 0, 7251040
section Manual
master (9.883M) : 0, 9883308
section Manual + Automatic
This PR (6063) (6.424M) : 0, 6423568
master (6.144M) : 0, 6143502
section DD_TRACE_ENABLED=0
master (9.467M) : 0, 9466504
|
Benchmarks Report for tracer 🐌Benchmarks for #6063 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Slower
|
| Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 | 1.126 | 482.94 | 543.58 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | StartFinishSpan |
net6.0 | 406ns | 0.0939ns | 0.364ns | 0.00797 | 0 | 0 | 576 B |
| master | StartFinishSpan |
netcoreapp3.1 | 540ns | 0.288ns | 1.12ns | 0.00786 | 0 | 0 | 576 B |
| master | StartFinishSpan |
net472 | 610ns | 0.36ns | 1.39ns | 0.0916 | 0 | 0 | 578 B |
| master | StartFinishScope |
net6.0 | 483ns | 0.89ns | 3.45ns | 0.00977 | 0 | 0 | 696 B |
| master | StartFinishScope |
netcoreapp3.1 | 745ns | 0.526ns | 2.04ns | 0.00935 | 0 | 0 | 696 B |
| master | StartFinishScope |
net472 | 888ns | 0.845ns | 3.16ns | 0.104 | 0 | 0 | 658 B |
| #6063 | StartFinishSpan |
net6.0 | 404ns | 0.467ns | 1.68ns | 0.00818 | 0 | 0 | 576 B |
| #6063 | StartFinishSpan |
netcoreapp3.1 | 558ns | 1.15ns | 4.32ns | 0.00787 | 0 | 0 | 576 B |
| #6063 | StartFinishSpan |
net472 | 672ns | 0.319ns | 1.23ns | 0.0916 | 0 | 0 | 578 B |
| #6063 | StartFinishScope |
net6.0 | 544ns | 1.08ns | 4.17ns | 0.00982 | 0 | 0 | 696 B |
| #6063 | StartFinishScope |
netcoreapp3.1 | 679ns | 0.512ns | 1.77ns | 0.00944 | 0 | 0 | 696 B |
| #6063 | StartFinishScope |
net472 | 867ns | 0.727ns | 2.81ns | 0.104 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | RunOnMethodBegin |
net6.0 | 583ns | 0.579ns | 2.24ns | 0.00969 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
netcoreapp3.1 | 992ns | 1.12ns | 4.33ns | 0.00931 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
net472 | 1.11μs | 4.25ns | 16.5ns | 0.104 | 0 | 0 | 658 B |
| #6063 | RunOnMethodBegin |
net6.0 | 594ns | 0.357ns | 1.38ns | 0.00981 | 0 | 0 | 696 B |
| #6063 | RunOnMethodBegin |
netcoreapp3.1 | 1.02μs | 0.429ns | 1.61ns | 0.00935 | 0 | 0 | 696 B |
| #6063 | RunOnMethodBegin |
net472 | 1.1μs | 0.313ns | 1.13ns | 0.104 | 0 | 0 | 658 B |
Summary of changes
This PR changes the current behaviour of failing opened tests when the test process is being shutdown (SIGTERM).
With this change now those tests are marked as "Skipped" with the "test.skip_reason = Test is being closed due to test session shutdown."
Reason for change
Current behaviour is causing problem in CI when jobs gets cancelled automatically when a new commit appears, skewing the stats and results of the test sessions.
Implementation details
Changes in the method that closes the opened test events.