Skip to content

Fix dotnet install in build#3348

Merged
andrewlock merged 2 commits into
masterfrom
andrew/exploration-tests
Oct 13, 2022
Merged

Fix dotnet install in build#3348
andrewlock merged 2 commits into
masterfrom
andrew/exploration-tests

Conversation

@andrewlock

@andrewlock andrewlock commented Oct 13, 2022

Copy link
Copy Markdown
Member

Summary of changes

  • Fix the broken exploration and execution benchmarks

Reason for change

In #3324, I changed the way we installed .NET Core 2.1 to resolve the constant build warnings from Azure Devops. Unfortunately, this broke stuff 🤷‍♂️

Implementation details

  • When installing .NET SDKs, install all of them the same way (using the dotnet-install script), so they're all in the same folder
  • Manually update the $PATH correctly
  • Don't install the x86 versions of the SDKs in exploration tests (not crucial I think, but not necessary)
  • Change the way execution benchmarks are run with timeit. For some reason, the spawned process wasn't working correctly (was not respecting the updated PATH to find the correct dotnet exe) when running the .exe directly. Running the .dll with dotnet fixes it. I have no idea why.

Test coverage

Ran several full builds, including the exploration tests

Other details

Uncovered some other issues:

  • I'm not convinced we're always running the exploration tests when we should (probably should run them on every merge to master?)
  • I don't think we ever run the comprehensive (minor) package version suite any more

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Oct 13, 2022
@andrewlock
andrewlock requested review from a team as code owners October 13, 2022 08:25
@andrewlock

andrewlock commented Oct 13, 2022

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #3348 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.114
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 787μs 594ns 2.3μs 0.393 0 0 3.22 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 506μs 563ns 2.18μs 0 0 0 2.63 KB
#3348 WriteAndFlushEnrichedTraces net472 786μs 675ns 2.62μs 0.391 0 0 3.22 KB
#3348 WriteAndFlushEnrichedTraces netcoreapp3.1 522μs 253ns 948ns 0 0 0 2.63 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 199ns 0.0442ns 0.165ns 0.0727 0 0 457 B
master AllCycleSimpleBody netcoreapp3.1 264ns 0.785ns 2.72ns 0.00628 0 0 456 B
master AllCycleMoreComplexBody net472 201ns 0.0299ns 0.104ns 0.0688 0 0 433 B
master AllCycleMoreComplexBody netcoreapp3.1 249ns 0.125ns 0.486ns 0.00584 0 0 432 B
master BodyExtractorSimpleBody net472 283ns 1.15ns 4.44ns 0.0574 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 236ns 0.0874ns 0.327ns 0.00367 0 0 272 B
master BodyExtractorMoreComplexBody net472 15.8μs 13ns 50.5ns 1.21 0.0157 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.42ns 12.8ns 0.0885 0 0 6.75 KB
#3348 AllCycleSimpleBody net472 196ns 0.0485ns 0.182ns 0.0726 0 0 457 B
#3348 AllCycleSimpleBody netcoreapp3.1 254ns 0.212ns 0.822ns 0.00628 0 0 456 B
#3348 AllCycleMoreComplexBody net472 195ns 0.0265ns 0.0917ns 0.0688 0 0 433 B
#3348 AllCycleMoreComplexBody netcoreapp3.1 251ns 0.151ns 0.565ns 0.00597 0 0 432 B
#3348 BodyExtractorSimpleBody net472 272ns 0.208ns 0.779ns 0.0573 0 0 361 B
#3348 BodyExtractorSimpleBody netcoreapp3.1 238ns 0.136ns 0.526ns 0.00371 0 0 272 B
#3348 BodyExtractorMoreComplexBody net472 15.7μs 15.9ns 61.7ns 1.21 0.0157 0 7.62 KB
#3348 BodyExtractorMoreComplexBody netcoreapp3.1 12.8μs 6.57ns 24.6ns 0.0902 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 181μs 225ns 872ns 0.271 0 0 20.68 KB
#3348 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3348 SendRequest netcoreapp3.1 182μs 274ns 1.06μs 0.184 0 0 20.68 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.75μs 0.822ns 3.19ns 0.155 0.000874 0 979 B
master ExecuteNonQuery netcoreapp3.1 1.42μs 1.31ns 5.08ns 0.0128 0 0 968 B
#3348 ExecuteNonQuery net472 1.78μs 0.508ns 1.9ns 0.155 0.000893 0 979 B
#3348 ExecuteNonQuery netcoreapp3.1 1.46μs 1.44ns 5.37ns 0.0131 0 0 968 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.52μs 0.66ns 2.56ns 0.188 0 0 1.19 KB
master CallElasticsearch netcoreapp3.1 1.51μs 0.516ns 1.86ns 0.0151 0 0 1.13 KB
master CallElasticsearchAsync net472 2.71μs 0.583ns 2.1ns 0.21 0 0 1.32 KB
master CallElasticsearchAsync netcoreapp3.1 1.66μs 0.422ns 1.58ns 0.0167 0 0 1.25 KB
#3348 CallElasticsearch net472 2.55μs 0.819ns 3.17ns 0.189 0 0 1.19 KB
#3348 CallElasticsearch netcoreapp3.1 1.52μs 1.04ns 3.91ns 0.0151 0 0 1.13 KB
#3348 CallElasticsearchAsync net472 2.6μs 0.66ns 2.55ns 0.21 0 0 1.32 KB
#3348 CallElasticsearchAsync netcoreapp3.1 1.66μs 0.4ns 1.44ns 0.0165 0 0 1.25 KB
Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #3348

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472 1.114 2,944.68 2,642.60

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.94μs 1.31ns 5.08ns 0.23 0 0 1.45 KB
master ExecuteAsync netcoreapp3.1 1.73μs 0.653ns 2.35ns 0.0182 0 0 1.38 KB
#3348 ExecuteAsync net472 2.64μs 1.35ns 5.24ns 0.23 0 0 1.45 KB
#3348 ExecuteAsync netcoreapp3.1 1.75μs 0.575ns 2.23ns 0.0185 0 0 1.38 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.93μs 1.8ns 6.72ns 0.443 0 0 2.8 KB
master SendAsync netcoreapp3.1 3.78μs 1.29ns 4.66ns 0.0361 0 0 2.63 KB
#3348 SendAsync net472 5.96μs 1.52ns 5.9ns 0.443 0 0 2.8 KB
#3348 SendAsync netcoreapp3.1 3.72μs 1.82ns 7.06ns 0.0354 0 0 2.63 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.19μs 3.1ns 12ns 0.298 0 0 1.88 KB
master EnrichedLog netcoreapp3.1 2.57μs 0.867ns 3.24ns 0.026 0 0 1.91 KB
#3348 EnrichedLog net472 3.11μs 3.06ns 11.5ns 0.298 0 0 1.88 KB
#3348 EnrichedLog netcoreapp3.1 2.44μs 0.868ns 3.36ns 0.0259 0 0 1.91 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 151μs 84.6ns 317ns 0.679 0.226 0 4.72 KB
master EnrichedLog netcoreapp3.1 115μs 137ns 514ns 0 0 0 4.55 KB
#3348 EnrichedLog net472 151μs 82.1ns 307ns 0.678 0.226 0 4.72 KB
#3348 EnrichedLog netcoreapp3.1 120μs 119ns 462ns 0.0596 0 0 4.55 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.66μs 1.27ns 4.42ns 0.578 0.00284 0 3.65 KB
master EnrichedLog netcoreapp3.1 4.28μs 2.51ns 9.38ns 0.0537 0 0 3.98 KB
#3348 EnrichedLog net472 5.8μs 1.23ns 4.42ns 0.58 0.0029 0 3.65 KB
#3348 EnrichedLog netcoreapp3.1 4.49μs 3.48ns 13ns 0.0558 0 0 3.98 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.38μs 2.29ns 8.87ns 0.223 0 0 1.4 KB
master SendReceive netcoreapp3.1 1.84μs 0.605ns 2.26ns 0.0185 0 0 1.35 KB
#3348 SendReceive net472 2.27μs 3.04ns 11.8ns 0.222 0 0 1.4 KB
#3348 SendReceive netcoreapp3.1 1.85μs 0.432ns 1.5ns 0.0175 0 0 1.35 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.18μs 2.13ns 7.67ns 0.363 0 0 2.3 KB
master EnrichedLog netcoreapp3.1 4.37μs 2.34ns 8.77ns 0.024 0 0 1.86 KB
#3348 EnrichedLog net472 5.1μs 2.4ns 8.98ns 0.362 0 0 2.3 KB
#3348 EnrichedLog netcoreapp3.1 4.19μs 1.33ns 4.79ns 0.0251 0 0 1.86 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.16μs 0.358ns 1.39ns 0.133 0 0 842 B
master StartFinishSpan netcoreapp3.1 968ns 0.36ns 1.35ns 0.0106 0 0 792 B
master StartFinishScope net472 1.49μs 0.257ns 0.928ns 0.146 0 0 923 B
master StartFinishScope netcoreapp3.1 1.08μs 0.372ns 1.39ns 0.0124 0 0 912 B
#3348 StartFinishSpan net472 1.18μs 0.727ns 2.82ns 0.134 0 0 842 B
#3348 StartFinishSpan netcoreapp3.1 972ns 0.568ns 2.2ns 0.0108 0 0 792 B
#3348 StartFinishScope net472 1.39μs 0.462ns 1.73ns 0.146 0 0 923 B
#3348 StartFinishScope netcoreapp3.1 1.14μs 0.688ns 2.67ns 0.0124 0 0 912 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.47μs 0.201ns 0.698ns 0.146 0 0 923 B
master RunOnMethodBegin netcoreapp3.1 1.17μs 0.403ns 1.51ns 0.0123 0 0 912 B
#3348 RunOnMethodBegin net472 1.49μs 0.778ns 3.01ns 0.146 0 0 923 B
#3348 RunOnMethodBegin netcoreapp3.1 1.18μs 0.463ns 1.73ns 0.0124 0 0 912 B

@andrewlock

andrewlock commented Oct 13, 2022

Copy link
Copy Markdown
Member Author

Code Coverage Report 📊

✔️ Merging #3348 into master will not change line coverage
✔️ Merging #3348 into master will not change branch coverage
⛔ Merging #3348 into master will will increase complexity by 22

master #3348 Change
Lines 19367 / 26705 19380 / 26721
Lines % 73% 73% 0% ✔️
Branches 11527 / 16615 11534 / 16629
Branches % 69% 69% 0% ✔️
Complexity 17944 17966 22 ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3348 Change
Lines % 73% 73% 0% ✔️
Branches % 69% 69% 0% ✔️
Complexity 17944 17966 22 ⛔

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.AppSec.Waf.Context -3% ⚠️ -10% ⛔ 0 ✔️
Datadog.Trace.AppSec.InstrumentationGateway 1% ✔️ 17% ✔️ 8 ⛔

View the full reports for further details:

gleocadie
gleocadie approved these changes Oct 13, 2022
View reviewed changes

@gleocadie gleocadie left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewlock
andrewlock merged commit 17fa324 into master Oct 13, 2022
@andrewlock
andrewlock deleted the andrew/exploration-tests branch October 13, 2022 10:04
@github-actions github-actions Bot added this to the vNext milestone Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gleocadie gleocadie gleocadie approved these changes
@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages

Projects

None yet

Milestone

2.18.0

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewlock @gleocadie @pierotibou