[backport][build] Build tracer with ReadyToRun (5962 => v2)#6005
Merged
duncanista merged 6 commits intoSep 10, 2024
Conversation
duncanista
commented
Sep 6, 2024
duncanista
added
type:backport
Member
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6005) - mean (73ms) : 64, 82
. : milestone, 73,
master - mean (70ms) : 67, 72
. : milestone, 70,
section CallTarget+Inlining+NGEN
This PR (6005) - mean (1,037ms) : 1016, 1058
. : milestone, 1037,
master - mean (1,087ms) : 1054, 1119
. : milestone, 1087,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6005) - mean (109ms) : 105, 114
. : milestone, 109,
master - mean (110ms) : 104, 116
. : milestone, 110,
section CallTarget+Inlining+NGEN
This PR (6005) - mean (720ms) : 697, 743
. : milestone, 720,
master - mean (762ms) : 745, 779
. : milestone, 762,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6005) - mean (92ms) : 90, 94
. : milestone, 92,
master - mean (93ms) : 91, 95
. : milestone, 93,
section CallTarget+Inlining+NGEN
This PR (6005) - mean (675ms) : 653, 697
. : milestone, 675,
master - mean (714ms) : 696, 732
. : milestone, 714,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6005) - mean (191ms) : 188, 195
. : milestone, 191,
master - mean (190ms) : 186, 193
. : milestone, 190,
section CallTarget+Inlining+NGEN
This PR (6005) - mean (1,117ms) : 1095, 1138
. : milestone, 1117,
master - mean (1,160ms) : 1135, 1184
. : milestone, 1160,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6005) - mean (275ms) : 271, 280
. : milestone, 275,
master - mean (276ms) : 272, 280
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (6005) - mean (885ms) : 867, 903
. : milestone, 885,
master - mean (920ms) : 895, 946
. : milestone, 920,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6005) - mean (265ms) : 261, 269
. : milestone, 265,
master - mean (265ms) : 261, 270
. : milestone, 265,
section CallTarget+Inlining+NGEN
This PR (6005) - mean (869ms) : 845, 894
. : milestone, 869,
master - mean (905ms) : 885, 926
. : milestone, 905,
|
Datadog ReportBranch report: ✅ 0 Failed, 339829 Passed, 1769 Skipped, 14h 28m 11.15s Total Time ⌛ Performance Regressions vs Default Branch (16)
|
Member
Benchmarks Report for tracer 🐌Benchmarks for #6005 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️
|
| Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 | 1.187 | 483.04 | 407.03 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | StartFinishSpan |
net6.0 | 483ns | 0.305ns | 1.18ns | 0.008 | 0 | 0 | 576 B |
| master | StartFinishSpan |
netcoreapp3.1 | 570ns | 0.541ns | 2.02ns | 0.00771 | 0 | 0 | 576 B |
| master | StartFinishSpan |
net472 | 567ns | 0.914ns | 3.54ns | 0.0916 | 0 | 0 | 578 B |
| master | StartFinishScope |
net6.0 | 521ns | 0.592ns | 2.29ns | 0.00968 | 0 | 0 | 696 B |
| master | StartFinishScope |
netcoreapp3.1 | 688ns | 0.537ns | 2.08ns | 0.00937 | 0 | 0 | 696 B |
| master | StartFinishScope |
net472 | 863ns | 0.867ns | 3.36ns | 0.105 | 0 | 0 | 658 B |
| #6005 | StartFinishSpan |
net6.0 | 407ns | 0.406ns | 1.57ns | 0.00801 | 0 | 0 | 576 B |
| #6005 | StartFinishSpan |
netcoreapp3.1 | 596ns | 0.982ns | 3.67ns | 0.00765 | 0 | 0 | 576 B |
| #6005 | StartFinishSpan |
net472 | 619ns | 0.664ns | 2.57ns | 0.0918 | 0 | 0 | 578 B |
| #6005 | StartFinishScope |
net6.0 | 554ns | 0.3ns | 1.16ns | 0.00969 | 0 | 0 | 696 B |
| #6005 | StartFinishScope |
netcoreapp3.1 | 742ns | 0.512ns | 1.92ns | 0.00917 | 0 | 0 | 696 B |
| #6005 | StartFinishScope |
net472 | 855ns | 1.42ns | 5.12ns | 0.105 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | RunOnMethodBegin |
net6.0 | 599ns | 0.311ns | 1.2ns | 0.00988 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
netcoreapp3.1 | 872ns | 0.785ns | 3.04ns | 0.00932 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
net472 | 1.11μs | 0.921ns | 3.57ns | 0.104 | 0 | 0 | 658 B |
| #6005 | RunOnMethodBegin |
net6.0 | 609ns | 0.378ns | 1.46ns | 0.0097 | 0 | 0 | 696 B |
| #6005 | RunOnMethodBegin |
netcoreapp3.1 | 959ns | 0.325ns | 1.26ns | 0.00918 | 0 | 0 | 696 B |
| #6005 | RunOnMethodBegin |
net472 | 1.13μs | 0.809ns | 3.13ns | 0.104 | 0 | 0 | 658 B |
Member
Throughput/Crank Report:zap:Throughput results for AspNetCoreSimpleController comparing the following branches/commits: Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red. Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards! gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6005) (10.933M) : 0, 10933220
master (11.264M) : 0, 11263908
benchmarks/2.9.0 (11.151M) : 0, 11151473
section Automatic
This PR (6005) (7.412M) : 0, 7411821
master (7.396M) : 0, 7395721
benchmarks/2.9.0 (7.902M) : 0, 7902288
section Trace stats
master (7.755M) : 0, 7754697
section Manual
This PR (6005) (9.581M) : crit ,0, 9580695
master (10.894M) : 0, 10894391
section Manual + Automatic
This PR (6005) (7.027M) : 0, 7027467
master (6.833M) : 0, 6832568
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6005) (9.692M) : 0, 9692000
master (9.689M) : 0, 9688722
benchmarks/2.9.0 (9.808M) : 0, 9808052
section Automatic
This PR (6005) (6.506M) : 0, 6505861
master (6.686M) : 0, 6685743
section Trace stats
master (6.609M) : 0, 6609162
section Manual
This PR (6005) (8.343M) : crit ,0, 8343119
master (9.754M) : 0, 9753866
section Manual + Automatic
This PR (6005) (6.182M) : 0, 6181925
master (6.225M) : 0, 6225281
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6005) (10.242M) : 0, 10242068
master (10.194M) : 0, 10193815
benchmarks/2.9.0 (10.095M) : 0, 10094883
section Automatic
This PR (6005) (6.789M) : 0, 6789140
master (6.868M) : 0, 6867533
benchmarks/2.9.0 (7.494M) : 0, 7494302
section Trace stats
master (7.421M) : 0, 7420558
section Manual
This PR (6005) (8.689M) : crit ,0, 8689312
master (10.247M) : 0, 10246979
section Manual + Automatic
This PR (6005) (6.588M) : 0, 6587699
master (6.274M) : 0, 6273947
|
andrewlock
changed the title
andrewlock
approved these changes
Sep 9, 2024
andrewlock
left a comment
andrewlock
left a comment
Member
There was a problem hiding this comment.
LGTM in general, just a few comments/questions about the release primarily 🙂
Comment on lines
+180
to
+182
| - if: '$CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+(-prerelease)?$/' # Manually triggered as artifacts are from the Github release | ||
| when: manual | ||
| allow_failure: false |
Member
There was a problem hiding this comment.
I'm not entirely sure how to think about this 🤔 Should we be manually triggering this when we do a release? Because it doesn't actually do anything special for releases, right (in contrast to the SSI artifacts where we take a different path for releases).
If it's not doing anything different, should we just skip for the tag instead? I'm not entirely sure the best option 🤔
Contributor
Author
There was a problem hiding this comment.
So, the release is not really automated from the .NET repo side, but on a downstream on our side. To not cause any accidental releases for layers.
This manual trigger is more if someone wants to release a testing AWS Lambda layer for .NET, which is why I decided to have it manual. As it'd be triggering our downstream pipeline on every push for every branch
I just realized this is in the download-serverless-arfiact and not in aws-lambda-layer.
But we do need it for the tag, since we want to make sure we can get the artifacts directly from the specific tag, which then will be used from a manual trigger downstream.
Member
There was a problem hiding this comment.
But we do need it for the tag, since we want to make sure we can get the artifacts directly from the specific tag, which then will be used from a manual trigger downstream.
The thing is, what does "directly from the specific tag" mean 😅 It's "special" like this for the download-ssi-artifacts because we use the artifacts directly from the GitHub release. But we're not doing anything like that here. It's not using the tag for anything in your script, so it's not getting "the artifacts directly from the specific tag". It's always using the COMMIT_SHA I believe. Which is not a big problem I think, it just reiterates that the "tag" logic here is superfluous I think 😃
It's not a big deal, as it all depends on downstream processes tbh, but was just flagging it in general, because it may cause issues for you down the line 🙂
Comment on lines
+196
to
+198
| trigger: | ||
| project: DataDog/dd-trace-dotnet-aws-lambda-layer | ||
| strategy: depend |
Member
There was a problem hiding this comment.
Just to be clear, this doesn't actually publish a new layer, right, it just creates it?
Contributor
Author
There was a problem hiding this comment.
This creates and publishes a new layer for Sandbox environments, for Production, more steps have to be done on our downstream pipeline, which won't be accidentally triggered from here!
duncanista
deleted the
jordan.gonzalez/backport/ready-to-run/linux-deb
branch
September 10, 2024 03:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick
5c5bec1b893b52c2364a2bf2b66c58a92e6d7d835962
Summary of changes
Allows tracer publishing to be compiled with
ReadyToRun
to improve Serverless workloads init duration.
Reason for change
It has showcased a 500ms init duration improvement for AWS Lambda.
Potentially could be used for other workloads in the future.
Implementation details
Followed #4573 and
ReadyToRun
docs.
Test coverage
Other details
Increases tracer size by 3x.