Skip to content

Skip flaky manual instrumentation tests#4949

Merged
andrewlock merged 1 commit into
masterfrom
andrew/skip-flaky-manual-tests
Dec 1, 2023
Merged

Skip flaky manual instrumentation tests#4949
andrewlock merged 1 commit into
masterfrom
andrew/skip-flaky-manual-tests

Conversation

@andrewlock

@andrewlock andrewlock commented Dec 1, 2023

Copy link
Copy Markdown
Member

Summary of changes

Skips the flaky tests

Reason for change

Their flaky, and we don't want to deal with it. The root cause looks related to Tracer lifetimes, but need to dig in further

Implementation details

Skip =

Test coverage

Less now

Other details

@andrewlock andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests labels Dec 1, 2023
@andrewlock
andrewlock requested a review from a team as a code owner December 1, 2023 18:14
@datadog-ddstaging

datadog-ddstaging Bot commented Dec 1, 2023 •
edited
Loading

Copy link
Copy Markdown

Datadog Report

Branch report: andrew/skip-flaky-manual-tests
Commit report: e801fb7

❄️ dd-trace-dotnet: 0 Failed, 1 New Flaky, 301117 Passed, 1023 Skipped, 30m 27.11s Wall Time

New Flaky Tests (1)

  • CanCreateDogStatsD_UDP_FromTraceAgentSettings - Datadog.Trace.Tests.DogStatsDTests - Last Failure

    Expand for error 
     Address already in use

@andrewlock

andrewlock commented Dec 1, 2023

Copy link
Copy Markdown
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4949) - mean (71ms)  : 63, 78
     .   : milestone, 71,
    master - mean (70ms)  : 62, 77
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4949) - mean (1,002ms)  : 984, 1019
     .   : milestone, 1002,
    master - mean (1,001ms)  : 986, 1017
     .   : milestone, 1001,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4949) - mean (106ms)  : 104, 108
     .   : milestone, 106,
    master - mean (106ms)  : 103, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4949) - mean (689ms)  : 671, 706
     .   : milestone, 689,
    master - mean (692ms)  : 676, 708
     .   : milestone, 692,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4949) - mean (90ms)  : 88, 93
     .   : milestone, 90,
    master - mean (90ms)  : 87, 93
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4949) - mean (660ms)  : 636, 684
     .   : milestone, 660,
    master - mean (663ms)  : 634, 692
     .   : milestone, 663,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4949) - mean (188ms)  : 186, 191
     .   : milestone, 188,
    master - mean (188ms)  : 184, 193
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4949) - mean (1,134ms)  : 1114, 1153
     .   : milestone, 1134,
    master - mean (1,135ms)  : 1109, 1161
     .   : milestone, 1135,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4949) - mean (272ms)  : 268, 276
     .   : milestone, 272,
    master - mean (272ms)  : 266, 277
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (4949) - mean (1,093ms)  : 1070, 1115
     .   : milestone, 1093,
    master - mean (1,092ms)  : 1063, 1122
     .   : milestone, 1092,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4949) - mean (261ms)  : 258, 264
     .   : milestone, 261,
    master - mean (262ms)  : 257, 267
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4949) - mean (1,063ms)  : 1035, 1091
     .   : milestone, 1063,
    master - mean (1,057ms)  : 1029, 1085
     .   : milestone, 1057,
Loading

@andrewlock

andrewlock commented Dec 1, 2023

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #4949 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.167
  • 1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.26μs 41.8ns 182ns 0.0253 0.0126 0 7.48 KB
master StartStopWithChild netcoreapp3.1 10.4μs 45.2ns 181ns 0.0305 0.0152 0 7.58 KB
master StartStopWithChild net472 17μs 59.1ns 229ns 1.34 0.354 0.101 7.95 KB
#4949 StartStopWithChild net6.0 8.71μs 48.3ns 317ns 0.0229 0.00915 0 7.48 KB
#4949 StartStopWithChild netcoreapp3.1 10.6μs 59.4ns 380ns 0.0209 0.0105 0 7.57 KB
#4949 StartStopWithChild net472 16.7μs 75.5ns 292ns 1.34 0.35 0.114 7.96 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 459μs 351ns 1.32μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 630μs 114ns 425ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 783μs 419ns 1.62μs 0.391 0 0 3.3 KB
#4949 WriteAndFlushEnrichedTraces net6.0 461μs 179ns 671ns 0 0 0 2.7 KB
#4949 WriteAndFlushEnrichedTraces netcoreapp3.1 630μs 280ns 1.08μs 0 0 0 2.7 KB
#4949 WriteAndFlushEnrichedTraces net472 772μs 299ns 1.16μs 0.386 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 43.7μs 19.6ns 75.8ns 0.0217 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 46.9μs 79.9ns 299ns 0 0 0 1.74 KB
master AllCycleSimpleBody net472 48.8μs 98.8ns 383ns 0.268 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 217μs 494ns 1.71μs 0.107 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 226μs 98.3ns 381ns 0.112 0 0 9.14 KB
master AllCycleMoreComplexBody net472 241μs 999ns 6.16μs 1.43 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 134ns 0.0456ns 0.176ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 181ns 0.0397ns 0.143ns 0.00374 0 0 272 B
master ObjectExtractorSimpleBody net472 153ns 0.117ns 0.453ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.83μs 1ns 3.76ns 0.0525 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.62μs 1.71ns 6.39ns 0.0508 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.43μs 1.75ns 6.55ns 0.602 0.00518 0 3.8 KB
#4949 AllCycleSimpleBody net6.0 44μs 61.7ns 239ns 0.0221 0 0 1.77 KB
#4949 AllCycleSimpleBody netcoreapp3.1 46.1μs 34.3ns 133ns 0.0229 0 0 1.74 KB
#4949 AllCycleSimpleBody net472 49.6μs 13.8ns 53.6ns 0.268 0 0 1.81 KB
#4949 AllCycleMoreComplexBody net6.0 214μs 204ns 764ns 0.107 0 0 9.25 KB
#4949 AllCycleMoreComplexBody netcoreapp3.1 224μs 398ns 1.54μs 0.111 0 0 9.14 KB
#4949 AllCycleMoreComplexBody net472 239μs 110ns 426ns 1.43 0 0 9.32 KB
#4949 ObjectExtractorSimpleBody net6.0 131ns 0.0562ns 0.218ns 0.00397 0 0 280 B
#4949 ObjectExtractorSimpleBody netcoreapp3.1 178ns 0.618ns 2.31ns 0.0037 0 0 272 B
#4949 ObjectExtractorSimpleBody net472 153ns 0.185ns 0.718ns 0.0446 0 0 281 B
#4949 ObjectExtractorMoreComplexBody net6.0 2.78μs 1.54ns 5.54ns 0.0532 0 0 3.78 KB
#4949 ObjectExtractorMoreComplexBody netcoreapp3.1 3.63μs 1.25ns 4.69ns 0.0491 0 0 3.69 KB
#4949 ObjectExtractorMoreComplexBody net472 3.4μs 2.49ns 9.33ns 0.602 0.00509 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 53.7μs 28.3ns 106ns 0.216 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 64μs 43.4ns 156ns 0.192 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 95.3μs 30.5ns 114ns 2.55 0.0946 0 16.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 114μs 255ns 1.11μs 0.285 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 136μs 233ns 902ns 0.271 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 165μs 136ns 526ns 3.59 0.167 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 106μs 58ns 225ns 0.42 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 131μs 712ns 3.97μs 0.398 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 186μs 764ns 2.96μs 5.28 0.364 0 33.67 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 172μs 753ns 2.92μs 0.528 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 212μs 384ns 1.49μs 0.523 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 259μs 538ns 2.08μs 6.39 0.383 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 99.4μs 25ns 90.3ns 0.417 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 133μs 152ns 587ns 0.434 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 181μs 71ns 275ns 5.15 0.361 0 32.63 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 165μs 77.3ns 289ns 0.524 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 206μs 1.05μs 4.57μs 0.495 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 257μs 127ns 494ns 6.22 0.519 0 39.2 KB
#4949 RunWaf(args=NestedMap (10)) net6.0 48.7μs 16.5ns 63.8ns 0.218 0 0 16.06 KB
#4949 RunWaf(args=NestedMap (10)) netcoreapp3.1 66.3μs 248ns 959ns 0.206 0 0 16.06 KB
#4949 RunWaf(args=NestedMap (10)) net472 99.8μs 25.3ns 91.2ns 2.54 0.0941 0 16.14 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 121μs 620ns 4.43μs 0.283 0 0 22.41 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 140μs 761ns 4.17μs 0.264 0 0 22.36 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [22]) net472 166μs 293ns 1.13μs 3.54 0.164 0 22.7 KB
#4949 RunWaf(args=NestedMap (100)) net6.0 104μs 43.5ns 169ns 0.418 0 0 32.76 KB
#4949 RunWaf(args=NestedMap (100)) netcoreapp3.1 129μs 683ns 3.41μs 0.434 0 0 33.33 KB
#4949 RunWaf(args=NestedMap (100)) net472 187μs 936ns 4.19μs 5.31 0.366 0 33.67 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 164μs 84.7ns 328ns 0.519 0 0 39.1 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 210μs 1.05μs 4.56μs 0.507 0 0 39.63 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [23]) net472 271μs 98.7ns 369ns 6.3 0.525 0 40.23 KB
#4949 RunWaf(args=NestedMap (20)) net6.0 99.3μs 28.8ns 108ns 0.422 0 0 32.18 KB
#4949 RunWaf(args=NestedMap (20)) netcoreapp3.1 134μs 711ns 3.69μs 0.453 0 0 32.3 KB
#4949 RunWaf(args=NestedMap (20)) net472 184μs 148ns 572ns 5.13 0.367 0 32.63 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 175μs 351ns 1.36μs 0.524 0 0 38.53 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 212μs 1.06μs 4.88μs 0.505 0 0 38.6 KB
#4949 RunWafWithAttack(args=Neste(...)tack) [22]) net472 258μs 145ns 542ns 6.17 0.525 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 174μs 172ns 665ns 0.173 0 0 18.25 KB
master SendRequest netcoreapp3.1 192μs 310ns 1.2μs 0.192 0 0 20.41 KB
master SendRequest net472 0.000168ns 7.01E‑05ns 0.000262ns 0 0 0 0 b
#4949 SendRequest net6.0 175μs 224ns 837ns 0.174 0 0 18.25 KB
#4949 SendRequest netcoreapp3.1 195μs 313ns 1.21μs 0.194 0 0 20.41 KB
#4949 SendRequest net472 0.000243ns 0.000103ns 0.0004ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 538μs 303ns 1.32μs 0.53 0 0 41.5 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 655μs 1.24μs 4.81μs 0.329 0 0 41.73 KB
master WriteAndFlushEnrichedTraces net472 816μs 3.89μs 16.1μs 8.17 2.45 0.408 53.23 KB
#4949 WriteAndFlushEnrichedTraces net6.0 541μs 289ns 1.12μs 0.546 0 0 41.55 KB
#4949 WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 189ns 653ns 0.334 0 0 41.74 KB
#4949 WriteAndFlushEnrichedTraces net472 840μs 3.56μs 13.3μs 8.17 2.45 0.408 53.23 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.03μs 0.64ns 2.48ns 0.0108 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.4μs 1.1ns 4.26ns 0.0102 0 0 768 B
master ExecuteNonQuery net472 1.66μs 0.868ns 3.13ns 0.116 0 0 730 B
#4949 ExecuteNonQuery net6.0 1.06μs 0.814ns 3.15ns 0.0106 0 0 768 B
#4949 ExecuteNonQuery netcoreapp3.1 1.49μs 2.38ns 9.23ns 0.0104 0 0 768 B
#4949 ExecuteNonQuery net472 1.6μs 1.17ns 4.54ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4949

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.135 1,203.28 1,366.26

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.2μs 0.402ns 1.5ns 0.0132 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.55μs 1.01ns 3.65ns 0.0125 0 0 936 B
master CallElasticsearch net472 2.45μs 0.46ns 1.78ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.4μs 0.692ns 2.68ns 0.0126 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.55μs 0.948ns 3.28ns 0.0129 0 0 984 B
master CallElasticsearchAsync net472 2.64μs 1.35ns 5.22ns 0.16 0 0 1.01 KB
#4949 CallElasticsearch net6.0 1.37μs 0.699ns 2.62ns 0.013 0 0 936 B
#4949 CallElasticsearch netcoreapp3.1 1.5μs 0.87ns 3.26ns 0.0128 0 0 936 B
#4949 CallElasticsearch net472 2.52μs 1.56ns 6.04ns 0.151 0 0 955 B
#4949 CallElasticsearchAsync net6.0 1.27μs 0.57ns 2.21ns 0.0129 0 0 912 B
#4949 CallElasticsearchAsync netcoreapp3.1 1.67μs 0.564ns 2.11ns 0.0134 0 0 984 B
#4949 CallElasticsearchAsync net472 2.59μs 0.582ns 2.25ns 0.161 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.24μs 0.612ns 2.37ns 0.0124 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.61μs 0.753ns 2.82ns 0.012 0 0 912 B
master ExecuteAsync net472 1.73μs 0.855ns 3.31ns 0.138 0 0 875 B
#4949 ExecuteAsync net6.0 1.31μs 1.09ns 4.08ns 0.0125 0 0 912 B
#4949 ExecuteAsync netcoreapp3.1 1.58μs 1.15ns 4.29ns 0.0127 0 0 912 B
#4949 ExecuteAsync net472 1.76μs 0.671ns 2.6ns 0.139 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.17μs 1.27ns 4.57ns 0.0285 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.99μs 2.39ns 8.94ns 0.0349 0 0 2.63 KB
master SendAsync net472 7.84μs 3.06ns 11.8ns 0.523 0 0 3.31 KB
#4949 SendAsync net6.0 3.93μs 1.71ns 6.4ns 0.0295 0 0 2.1 KB
#4949 SendAsync netcoreapp3.1 5.1μs 2.4ns 9.3ns 0.0357 0 0 2.63 KB
#4949 SendAsync net472 7.92μs 1.3ns 4.85ns 0.523 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #4949

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net6.0 43.91 KB 43.29 KB -624 B -1.42%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 61.5μs 874ns 8.7μs 0 0 0 43.44 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53μs 255ns 1.08μs 0 0 0 42.64 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 37.6μs 97.9ns 366ns 0 0 0 59.59 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 65μs 280ns 1.05μs 0 0 0 43.91 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 98μs 3.81μs 37.7μs 0 0 0 42.64 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 61.8μs 332ns 1.79μs 0 0 0 57.34 KB
#4949 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 61.7μs 800ns 7.96μs 0 0 0 43.44 KB
#4949 StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53.8μs 258ns 998ns 0 0 0 42.64 KB
#4949 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 37.9μs 135ns 488ns 0 0 0 59.7 KB
#4949 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 96.3μs 2.92μs 29.2μs 0 0 0 43.29 KB
#4949 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 99.9μs 3.81μs 37.9μs 0 0 0 42.64 KB
#4949 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 62.9μs 341ns 1.9μs 0 0 0 57.34 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.51μs 0.586ns 2.19ns 0.022 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.09μs 2.72ns 9.8ns 0.0218 0 0 1.57 KB
master EnrichedLog net472 2.53μs 2.03ns 7.86ns 0.238 0 0 1.5 KB
#4949 EnrichedLog net6.0 1.46μs 0.491ns 1.7ns 0.0219 0 0 1.57 KB
#4949 EnrichedLog netcoreapp3.1 2.27μs 0.612ns 2.21ns 0.0216 0 0 1.57 KB
#4949 EnrichedLog net472 2.54μs 2ns 7.75ns 0.238 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 266ns 1.03μs 0.0561 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 119μs 224ns 867ns 0.0596 0 0 4.21 KB
master EnrichedLog net472 147μs 217ns 839ns 0.659 0.22 0 4.39 KB
#4949 EnrichedLog net6.0 112μs 65.3ns 253ns 0.056 0 0 4.21 KB
#4949 EnrichedLog netcoreapp3.1 119μs 174ns 626ns 0.0592 0 0 4.21 KB
#4949 EnrichedLog net472 148μs 218ns 846ns 0.665 0.222 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.08μs 0.713ns 2.76ns 0.0293 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.33μs 7.01ns 27.1ns 0.0294 0 0 2.13 KB
master EnrichedLog net472 4.8μs 4.37ns 16.9ns 0.308 0 0 1.95 KB
#4949 EnrichedLog net6.0 2.93μs 0.797ns 3.09ns 0.0294 0 0 2.13 KB
#4949 EnrichedLog netcoreapp3.1 4.14μs 0.615ns 2.13ns 0.0291 0 0 2.13 KB
#4949 EnrichedLog net472 4.85μs 4.07ns 15.7ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.31μs 0.484ns 1.81ns 0.0155 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.73μs 1.29ns 5.01ns 0.0145 0 0 1.1 KB
master SendReceive net472 2.22μs 2.23ns 8.64ns 0.177 0 0 1.12 KB
#4949 SendReceive net6.0 1.32μs 0.941ns 3.64ns 0.0158 0 0 1.1 KB
#4949 SendReceive netcoreapp3.1 1.73μs 0.69ns 2.67ns 0.0146 0 0 1.1 KB
#4949 SendReceive net472 2.09μs 1.38ns 5.33ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.69μs 1.24ns 4.81ns 0.0215 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.88μs 3.13ns 12.1ns 0.021 0 0 1.58 KB
master EnrichedLog net472 4.21μs 2ns 7.74ns 0.312 0 0 1.97 KB
#4949 EnrichedLog net6.0 2.71μs 0.653ns 2.44ns 0.0217 0 0 1.53 KB
#4949 EnrichedLog netcoreapp3.1 3.94μs 2.07ns 8ns 0.0217 0 0 1.58 KB
#4949 EnrichedLog net472 4.28μs 1.77ns 6.87ns 0.31 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 504ns 0.124ns 0.466ns 0.00755 0 0 536 B
master StartFinishSpan netcoreapp3.1 696ns 0.912ns 3.53ns 0.00731 0 0 536 B
master StartFinishSpan net472 756ns 0.812ns 3.15ns 0.0852 0 0 538 B
master StartFinishScope net6.0 571ns 0.297ns 1.15ns 0.00923 0 0 656 B
master StartFinishScope netcoreapp3.1 802ns 0.264ns 0.953ns 0.00881 0 0 656 B
master StartFinishScope net472 996ns 1.22ns 4.74ns 0.0982 0 0 618 B
#4949 StartFinishSpan net6.0 481ns 0.62ns 2.4ns 0.00747 0 0 536 B
#4949 StartFinishSpan netcoreapp3.1 697ns 0.287ns 1.03ns 0.00707 0 0 536 B
#4949 StartFinishSpan net472 715ns 0.397ns 1.48ns 0.0853 0 0 538 B
#4949 StartFinishScope net6.0 557ns 0.111ns 0.417ns 0.00915 0 0 656 B
#4949 StartFinishScope netcoreapp3.1 830ns 0.378ns 1.41ns 0.00882 0 0 656 B
#4949 StartFinishScope net472 1.03μs 1.3ns 5.03ns 0.098 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4949

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.200 617.52 740.81

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 617ns 0.128ns 0.494ns 0.00904 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 966ns 0.532ns 2.06ns 0.00874 0 0 656 B
master RunOnMethodBegin net472 1.07μs 0.325ns 1.26ns 0.0978 0 0 618 B
#4949 RunOnMethodBegin net6.0 741ns 0.27ns 1.05ns 0.00926 0 0 656 B
#4949 RunOnMethodBegin netcoreapp3.1 896ns 1.02ns 3.96ns 0.00894 0 0 656 B
#4949 RunOnMethodBegin net472 1.09μs 0.298ns 1.15ns 0.0977 0 0 618 B

@andrewlock

andrewlock commented Dec 1, 2023

Copy link
Copy Markdown
Member Author

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4949) (11.106M)   : 0, 11106418
    master (11.119M)   : 0, 11118604
    benchmarks/2.9.0 (10.902M)   : 0, 10901856

    section Automatic
    This PR (4949) (7.606M)   : 0, 7606233
    master (7.595M)   : 0, 7595371
    benchmarks/2.9.0 (7.812M)   : 0, 7812244

    section Trace stats
    This PR (4949) (7.755M)   : 0, 7755326
    master (7.841M)   : 0, 7841183

    section Manual
    This PR (4949) (9.557M)   : 0, 9557390
    master (9.600M)   : 0, 9599547

    section Manual + Automatic
    This PR (4949) (7.141M)   : 0, 7140803
    master (7.183M)   : 0, 7183326

    section Version Conflict
    This PR (4949) (6.503M)   : 0, 6502740
    master (6.438M)   : 0, 6438384
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4949) (9.528M)   : 0, 9527738
    master (9.412M)   : 0, 9411881
    benchmarks/2.9.0 (9.771M)   : 0, 9771450

    section Automatic
    This PR (4949) (6.602M)   : 0, 6601923
    master (6.424M)   : 0, 6423890

    section Trace stats
    This PR (4949) (6.921M)   : 0, 6920959
    master (6.850M)   : 0, 6850126

    section Manual
    This PR (4949) (8.344M)   : 0, 8343892
    master (8.141M)   : 0, 8140796

    section Manual + Automatic
    This PR (4949) (6.196M)   : 0, 6195939
    master (6.211M)   : 0, 6211181

    section Version Conflict
    This PR (4949) (5.484M)   : 0, 5484035
    master (5.699M)   : 0, 5698571
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4949) (9.875M)   : 0, 9874967
    master (10.445M)   : 0, 10444942
    benchmarks/2.9.0 (10.877M)   : 0, 10876503

    section Automatic
    This PR (4949) (7.017M)   : 0, 7016708
    master (7.315M)   : 0, 7315496
    benchmarks/2.9.0 (8.110M)   : 0, 8109662

    section Trace stats
    This PR (4949) (7.534M)   : 0, 7533715
    master (7.403M)   : 0, 7402999

    section Manual
    This PR (4949) (8.917M)   : 0, 8916975
    master (8.693M)   : 0, 8692918

    section Manual + Automatic
    This PR (4949) (6.796M)   : 0, 6796285
    master (6.688M)   : 0, 6688178

    section Version Conflict
    This PR (4949) (6.092M)   : 0, 6092000
    master (6.092M)   : 0, 6091981
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.364M)   : 0, 7364231
    benchmarks/2.9.0 (7.803M)   : 0, 7803140

    section No attack
    master (1.957M)   : 0, 1956655
    benchmarks/2.9.0 (3.216M)   : 0, 3215869

    section Attack
    master (1.569M)   : 0, 1569292
    benchmarks/2.9.0 (2.464M)   : 0, 2463754

    section Blocking
    master (3.248M)   : 0, 3247710

    section IAST default
    master (6.433M)   : 0, 6433032

    section IAST full
    master (5.764M)   : 0, 5764259

    section Base vuln
    master (0.969M)   : 0, 969140

    section IAST vuln
    master (0.878M)   : 0, 877853
Loading

@andrewlock
andrewlock merged commit d5cd1a5 into master Dec 1, 2023
@andrewlock
andrewlock deleted the andrew/skip-flaky-manual-tests branch December 1, 2023 23:18
@github-actions github-actions Bot added this to the vNext milestone Dec 1, 2023
@andrewlock andrewlock mentioned this pull request Dec 7, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bouwkast bouwkast bouwkast approved these changes
@zacharycmontoya zacharycmontoya zacharycmontoya approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests

Projects

None yet

Milestone

2.43.0

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewlock @bouwkast @zacharycmontoya