Skip to content

[Profiler] Download profiler debug symbols for integration tests#6070

Merged
gleocadie merged 1 commit into
masterfrom
gleocadie/add-profiler-symbols-for-integration-tests
Sep 24, 2024
Merged

[Profiler] Download profiler debug symbols for integration tests#6070
gleocadie merged 1 commit into
masterfrom
gleocadie/add-profiler-symbols-for-integration-tests

Conversation

@gleocadie

@gleocadie gleocadie commented Sep 24, 2024

Copy link
Copy Markdown
Collaborator

Summary of changes

Download profiler debug symbols for integration tests jobs.

Reason for change

When a test takes too much time, the profiler test infrastructure calls gdb to collect all threads callstack and print it in the console. (and generate a coredump)
Since binaries are stripped, the callstack are difficult to read. For the jobs that runs on CentOS 7, we cannot create a new image and install gdb (and tools) because of deprecation and mainly because all the CentOS 7 packages are gone.

Implementation details

Download the profiler native debug symbols when setting up the tests.

Test coverage

Other details

@gleocadie
gleocadie requested a review from a team as a code owner September 24, 2024 12:12
@github-actions github-actions Bot added area:builds project files, build scripts, pipelines, versioning, releases, packages area:profiler Issues related to the continous-profiler labels Sep 24, 2024
@andrewlock

andrewlock commented Sep 24, 2024

Copy link
Copy Markdown
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6070) - mean (71ms)  : 67, 75
     .   : milestone, 71,
    master - mean (70ms)  : 67, 73
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6070) - mean (1,109ms)  : 1086, 1132
     .   : milestone, 1109,
    master - mean (1,105ms)  : 1067, 1143
     .   : milestone, 1105,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6070) - mean (113ms)  : 106, 120
     .   : milestone, 113,
    master - mean (109ms)  : 105, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6070) - mean (789ms)  : 742, 835
     .   : milestone, 789,
    master - mean (766ms)  : 751, 782
     .   : milestone, 766,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6070) - mean (92ms)  : 89, 95
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6070) - mean (726ms)  : 710, 742
     .   : milestone, 726,
    master - mean (727ms)  : 707, 747
     .   : milestone, 727,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6070) - mean (192ms)  : 178, 207
     .   : milestone, 192,
    master - mean (191ms)  : 187, 195
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6070) - mean (1,197ms)  : 1172, 1222
     .   : milestone, 1197,
    master - mean (1,202ms)  : 1165, 1240
     .   : milestone, 1202,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6070) - mean (275ms)  : 271, 280
     .   : milestone, 275,
    master - mean (276ms)  : 270, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6070) - mean (939ms)  : 919, 960
     .   : milestone, 939,
    master - mean (938ms)  : 918, 958
     .   : milestone, 938,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6070) - mean (265ms)  : 262, 269
     .   : milestone, 265,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6070) - mean (934ms)  : 910, 958
     .   : milestone, 934,
    master - mean (922ms)  : 906, 939
     .   : milestone, 922,
Loading

@datadog-ddstaging

datadog-ddstaging Bot commented Sep 24, 2024 •
edited
Loading

Copy link
Copy Markdown

Datadog Report

Branch report: gleocadie/add-profiler-symbols-for-integration-tests
Commit report: 83efaca
Test service: dd-trace-dotnet

✅ 0 Failed, 361089 Passed, 2048 Skipped, 15h 31m 25.11s Total Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

  • Baseline - Samples.FakeDbCommand.windows.netcoreapp31.json.scenarios 113.04ms (+4.57ms, +4%) - Details

@andrewlock

andrewlock commented Sep 24, 2024

Copy link
Copy Markdown
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6070 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.145
  • 2 benchmarks are slower, with geometric mean 1.135
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.88μs 43.3ns 256ns 0.0162 0.00812 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.88μs 55.7ns 390ns 0.0189 0.00945 0 5.61 KB
master StartStopWithChild net472 15.8μs 52.8ns 205ns 1.02 0.3 0.103 6.07 KB
#6070 StartStopWithChild net6.0 7.87μs 44.5ns 312ns 0.0154 0.00771 0 5.43 KB
#6070 StartStopWithChild netcoreapp3.1 9.97μs 54.8ns 364ns 0.0249 0.00997 0 5.62 KB
#6070 StartStopWithChild net472 16.2μs 49.3ns 185ns 1.01 0.292 0.0946 6.05 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 484μs 485ns 1.88μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 417ns 1.56μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 835μs 498ns 1.93μs 0.417 0 0 3.3 KB
#6070 WriteAndFlushEnrichedTraces net6.0 481μs 260ns 1.01μs 0 0 0 2.7 KB
#6070 WriteAndFlushEnrichedTraces netcoreapp3.1 625μs 240ns 931ns 0 0 0 2.7 KB
#6070 WriteAndFlushEnrichedTraces net472 839μs 563ns 2.18μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 207μs 1.21μs 11.1μs 0.202 0 0 18.45 KB
master SendRequest netcoreapp3.1 237μs 1.35μs 9.76μs 0.215 0 0 20.61 KB
master SendRequest net472 0.00304ns 0.00115ns 0.00444ns 0 0 0 0 b
#6070 SendRequest net6.0 209μs 1.32μs 13μs 0.21 0 0 18.45 KB
#6070 SendRequest netcoreapp3.1 236μs 1.37μs 12.1μs 0.217 0 0 20.61 KB
#6070 SendRequest net472 0.000307ns 0.000238ns 0.00089ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 577μs 3.1μs 17.8μs 0.566 0 0 41.57 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 686μs 3.28μs 14.3μs 0.347 0 0 41.64 KB
master WriteAndFlushEnrichedTraces net472 875μs 3.4μs 12.7μs 8.09 2.25 0.45 53.33 KB
#6070 WriteAndFlushEnrichedTraces net6.0 560μs 2.07μs 7.45μs 0.556 0 0 41.61 KB
#6070 WriteAndFlushEnrichedTraces netcoreapp3.1 702μs 2.32μs 8.37μs 0.351 0 0 41.7 KB
#6070 WriteAndFlushEnrichedTraces net472 859μs 3.64μs 14.1μs 8.19 2.59 0.431 53.34 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.34μs 1.32ns 4.95ns 0.0142 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.71μs 1.13ns 4.39ns 0.0137 0 0 1.02 KB
master ExecuteNonQuery net472 2.13μs 2.16ns 8.36ns 0.156 0 0 987 B
#6070 ExecuteNonQuery net6.0 1.26μs 2.26ns 8.75ns 0.014 0 0 1.02 KB
#6070 ExecuteNonQuery netcoreapp3.1 1.75μs 1.75ns 6.8ns 0.013 0 0 1.02 KB
#6070 ExecuteNonQuery net472 2.01μs 2.12ns 7.93ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.22μs 3.91ns 14.6ns 0.0139 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.53μs 0.951ns 3.56ns 0.0128 0 0 976 B
master CallElasticsearch net472 2.46μs 1.66ns 6.41ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.17μs 1.32ns 4.93ns 0.0136 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.68μs 0.823ns 3.08ns 0.0134 0 0 1.02 KB
master CallElasticsearchAsync net472 2.55μs 1.79ns 6.95ns 0.166 0 0 1.05 KB
#6070 CallElasticsearch net6.0 1.15μs 0.321ns 1.2ns 0.0138 0 0 976 B
#6070 CallElasticsearch netcoreapp3.1 1.44μs 0.44ns 1.65ns 0.0134 0 0 976 B
#6070 CallElasticsearch net472 2.5μs 0.761ns 2.74ns 0.157 0 0 995 B
#6070 CallElasticsearchAsync net6.0 1.24μs 1.46ns 5.65ns 0.0131 0 0 952 B
#6070 CallElasticsearchAsync netcoreapp3.1 1.6μs 1.07ns 3.87ns 0.0136 0 0 1.02 KB
#6070 CallElasticsearchAsync net472 2.62μs 1.38ns 4.98ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6070

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0 1.120 1,225.47 1,372.35

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.23μs 0.572ns 2.14ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.61μs 2.86ns 11.1ns 0.0127 0 0 952 B
master ExecuteAsync net472 1.81μs 0.978ns 3.79ns 0.145 0 0 915 B
#6070 ExecuteAsync net6.0 1.37μs 2.34ns 9.06ns 0.0131 0 0 952 B
#6070 ExecuteAsync netcoreapp3.1 1.62μs 0.533ns 1.92ns 0.0122 0 0 952 B
#6070 ExecuteAsync net472 1.85μs 0.427ns 1.65ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.26μs 1.23ns 4.6ns 0.0298 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.07μs 4.13ns 16ns 0.0354 0 0 2.76 KB
master SendAsync net472 7.85μs 1.34ns 5.2ns 0.497 0 0 3.15 KB
#6070 SendAsync net6.0 4.23μs 1.34ns 5.03ns 0.0318 0 0 2.22 KB
#6070 SendAsync netcoreapp3.1 5.04μs 2.06ns 7.71ns 0.0353 0 0 2.76 KB
#6070 SendAsync net472 7.8μs 1.46ns 5.64ns 0.5 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.56μs 0.743ns 2.88ns 0.0227 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.2μs 1.15ns 4.46ns 0.022 0 0 1.64 KB
master EnrichedLog net472 2.6μs 1.54ns 5.96ns 0.249 0 0 1.57 KB
#6070 EnrichedLog net6.0 1.59μs 0.98ns 3.67ns 0.0229 0 0 1.64 KB
#6070 EnrichedLog netcoreapp3.1 2.32μs 0.95ns 3.43ns 0.0221 0 0 1.64 KB
#6070 EnrichedLog net472 2.66μs 2.25ns 8.7ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 120ns 464ns 0.0566 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 119μs 106ns 411ns 0.0598 0 0 4.28 KB
master EnrichedLog net472 145μs 67.6ns 244ns 0.656 0.219 0 4.46 KB
#6070 EnrichedLog net6.0 115μs 151ns 585ns 0 0 0 4.28 KB
#6070 EnrichedLog netcoreapp3.1 121μs 154ns 595ns 0.0602 0 0 4.28 KB
#6070 EnrichedLog net472 149μs 221ns 856ns 0.667 0.222 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.09μs 0.928ns 3.59ns 0.0308 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.41μs 1.9ns 7.37ns 0.0287 0 0 2.2 KB
master EnrichedLog net472 4.81μs 1.3ns 4.85ns 0.32 0 0 2.02 KB
#6070 EnrichedLog net6.0 3.15μs 1.06ns 3.96ns 0.0301 0 0 2.2 KB
#6070 EnrichedLog netcoreapp3.1 4.18μs 1.67ns 6.45ns 0.0292 0 0 2.2 KB
#6070 EnrichedLog net472 4.75μs 1.46ns 5.64ns 0.321 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.36μs 0.7ns 2.62ns 0.0156 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.73μs 1.22ns 4.58ns 0.0155 0 0 1.14 KB
master SendReceive net472 2.17μs 1.16ns 4.32ns 0.183 0.00109 0 1.16 KB
#6070 SendReceive net6.0 1.38μs 0.731ns 2.83ns 0.0159 0 0 1.14 KB
#6070 SendReceive netcoreapp3.1 1.73μs 0.767ns 2.97ns 0.0157 0 0 1.14 KB
#6070 SendReceive net472 2.11μs 0.985ns 3.41ns 0.183 0.00104 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.77μs 10.3ns 40.1ns 0.022 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.77μs 1.36ns 5.08ns 0.0209 0 0 1.65 KB
master EnrichedLog net472 4.33μs 1.67ns 6.46ns 0.323 0 0 2.04 KB
#6070 EnrichedLog net6.0 2.7μs 0.753ns 2.92ns 0.0216 0 0 1.6 KB
#6070 EnrichedLog netcoreapp3.1 3.99μs 1.85ns 6.93ns 0.022 0 0 1.65 KB
#6070 EnrichedLog net472 4.37μs 2.59ns 9.68ns 0.322 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6070

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.150 407.09 468.29

Faster 🎉 in #6070

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.140 758.39 665.05

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 407ns 0.361ns 1.4ns 0.00818 0 0 576 B
master StartFinishSpan netcoreapp3.1 594ns 1.01ns 3.93ns 0.00798 0 0 576 B
master StartFinishSpan net472 654ns 1.56ns 5.84ns 0.0917 0 0 578 B
master StartFinishScope net6.0 477ns 0.235ns 0.911ns 0.00966 0 0 696 B
master StartFinishScope netcoreapp3.1 759ns 1.06ns 4.09ns 0.00938 0 0 696 B
master StartFinishScope net472 888ns 1.04ns 4.01ns 0.105 0 0 658 B
#6070 StartFinishSpan net6.0 469ns 0.677ns 2.62ns 0.0081 0 0 576 B
#6070 StartFinishSpan netcoreapp3.1 557ns 1.2ns 4.64ns 0.00794 0 0 576 B
#6070 StartFinishSpan net472 680ns 0.42ns 1.63ns 0.0917 0 0 578 B
#6070 StartFinishScope net6.0 490ns 1.06ns 3.82ns 0.00987 0 0 696 B
#6070 StartFinishScope netcoreapp3.1 665ns 0.82ns 3.07ns 0.00958 0 0 696 B
#6070 StartFinishScope net472 881ns 0.182ns 0.705ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6070

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.149 675.74 588.02

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 676ns 0.514ns 1.99ns 0.00988 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 982ns 0.697ns 2.61ns 0.00936 0 0 696 B
master RunOnMethodBegin net472 1.11μs 0.875ns 3.39ns 0.105 0 0 658 B
#6070 RunOnMethodBegin net6.0 588ns 0.268ns 1.04ns 0.00977 0 0 696 B
#6070 RunOnMethodBegin netcoreapp3.1 923ns 0.582ns 2.26ns 0.00922 0 0 696 B
#6070 RunOnMethodBegin net472 1.14μs 0.59ns 2.28ns 0.104 0 0 658 B

@gleocadie
gleocadie merged commit 03b9ea7 into master Sep 24, 2024
@gleocadie
gleocadie deleted the gleocadie/add-profiler-symbols-for-integration-tests branch September 24, 2024 14:16
@github-actions github-actions Bot added this to the vNext-v3 milestone Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewlock andrewlock andrewlock approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages area:profiler Issues related to the continous-profiler

Projects

None yet

Milestone

3.4.0

Development

Successfully merging this pull request may close these issues.

2 participants

@gleocadie @andrewlock