-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Labels
PGOApplied if there were any profile guided optimization updates in the observed interval.Applied if there were any profile guided optimization updates in the observed interval.arch-arm64branch-refs/heads/mainkind-microos-linuxperf-improvementruntime-coreclr
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 3e5517beb897faf4592d23f036446561da1e5c23 |
| Compare | 0533792282ba5546e10cb5c3d10748aff46ddc1a |
| Diff | Diff |
Improvements in Exceptions.Handling
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ThrowAndCatch - Duration of single invocation | 178.09 μs | 137.24 μs | 0.77 | 0.46 | False | |||||
| ThrowAndCatch_ManyCatchBlocks - Duration of single invocation | 200.19 μs | 135.89 μs | 0.68 | 0.36 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Exceptions.Handling*'Details
Payloads
Histogram
Exceptions.Handling.ThrowAndCatch(kind: ReflectionSoftware)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 137.2443082315163 < 178.38886739410003.
IsChangePoint: Marked as a change because one of 4/8/2022 2:40:20 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 5.493051663467603 (T) = (0 -144709.21474880382) / Math.Sqrt((459639543.50362307 / (299)) + (60558153.23931924 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.09151647731392606 = (159286.55956351123 - 144709.21474880382) / 159286.55956351123 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionSoftware)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 135.88835321906353 < 189.12500181201315.
IsChangePoint: Marked as a change because one of 4/8/2022 2:40:20 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 15.261202368787597 (T) = (0 -138343.09820499978) / Math.Sqrt((500851545.2415736 / (299)) + (2616435.919967634 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.1323773167760695 = (159450.7622725372 - 138343.09820499978) / 159450.7622725372 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 3e5517beb897faf4592d23f036446561da1e5c23 |
| Compare | 0533792282ba5546e10cb5c3d10748aff46ddc1a |
| Diff | Diff |
Improvements in Microsoft.Extensions.DependencyInjection.TimeToFirstService
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Transient - Duration of single invocation | 19.74 μs | 17.19 μs | 0.87 | 0.06 | False | |||||
| Singleton - Duration of single invocation | 19.93 μs | 17.60 μs | 0.88 | 0.06 | False | |||||
| Singleton - Duration of single invocation | 19.68 μs | 17.97 μs | 0.91 | 0.08 | False | |||||
| Scoped - Duration of single invocation | 21.01 μs | 18.89 μs | 0.90 | 0.04 | False | |||||
| Singleton - Duration of single invocation | 19.93 μs | 17.53 μs | 0.88 | 0.06 | False | |||||
| Scoped - Duration of single invocation | 21.07 μs | 18.84 μs | 0.89 | 0.07 | False | |||||
| Singleton - Duration of single invocation | 19.72 μs | 17.76 μs | 0.90 | 0.06 | False | |||||
| Scoped - Duration of single invocation | 21.08 μs | 19.00 μs | 0.90 | 0.09 | False | |||||
| Transient - Duration of single invocation | 19.37 μs | 17.14 μs | 0.88 | 0.06 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.DependencyInjection.TimeToFirstService*'Details
Payloads
Histogram
Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "ILEmit")
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.187070024595965 < 18.548053560155118.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 36.05107076720448 (T) = (0 -17498.342493605916) / Math.Sqrt((371525.11458023137 / (299)) + (41062.882259744605 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.1268857373042857 = (20041.29727486103 - 17498.342493605916) / 20041.29727486103 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Expressions")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.603181839007544 < 18.70145217958402.
IsChangePoint: Marked as a change because one of 3/18/2022 7:38:50 AM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 23.283670684304365 (T) = (0 -17811.10270683597) / Math.Sqrt((336196.0135842124 / (299)) + (112204.16902273285 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.1221258983897214 = (20288.903242691842 - 17811.10270683597) / 20288.903242691842 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Runtime")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.973728042024835 < 18.653811883958515.
IsChangePoint: Marked as a change because one of 3/18/2022 1:59:24 AM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 19.28748721574007 (T) = (0 -18029.67547330024) / Math.Sqrt((353946.8399368647 / (299)) + (139756.450349394 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.11195801147382291 = (20302.728594199547 - 18029.67547330024) / 20302.728594199547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Expressions")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.891674214289324 < 20.0380591865709.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 57.691265456114 (T) = (0 -19039.669275490145) / Math.Sqrt((382122.16323418764 / (299)) + (9475.76898387818 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.12292416487626694 = (21708.122049450983 - 19039.669275490145) / 21708.122049450983 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "ILEmit")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.526823101817524 < 18.92236482904492.
IsChangePoint: Marked as a change because one of 3/18/2022 2:15:20 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 58.29102626506418 (T) = (0 -17714.872052853923) / Math.Sqrt((330575.5860110873 / (299)) + (9533.075737284435 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.12749989964528982 = (20303.57594876154 - 17714.872052853923) / 20303.57594876154 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "ILEmit")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.83900167476038 < 20.244983613584.
IsChangePoint: Marked as a change because one of 3/18/2022 1:59:24 AM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 18.141402899256132 (T) = (0 -19237.845471632947) / Math.Sqrt((405703.7387961346 / (299)) + (197718.04102508974 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.11591467702617689 = (21760.168359001884 - 19237.845471632947) / 21760.168359001884 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Singleton(Mode: "Dynamic")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.75608424597276 < 18.65047475657848.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 27.11249704668036 (T) = (0 -17875.125913270425) / Math.Sqrt((334765.6166177906 / (299)) + (74342.04807290909 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.11865208561403992 = (20281.57736746234 - 17875.125913270425) / 20281.57736746234 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Scoped(Mode: "Dynamic")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 19.00322544563955 < 20.037165961730448.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 23.8627474547782 (T) = (0 -19215.69487926475) / Math.Sqrt((434460.8046669556 / (299)) + (108998.9054549722 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.11689710837085308 = (21759.29335234727 - 19215.69487926475) / 21759.29335234727 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.DependencyInjection.TimeToFirstService.Transient(Mode: "Runtime")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.140703491132378 < 18.491480644835573.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 23.340792544231437 (T) = (0 -17581.398567453554) / Math.Sqrt((352303.1776981717 / (299)) + (105726.7170832878 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.12118924092757948 = (20005.898182232788 - 17581.398567453554) / 20005.898182232788 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 3e5517beb897faf4592d23f036446561da1e5c23 |
| Compare | 0533792282ba5546e10cb5c3d10748aff46ddc1a |
| Diff | Diff |
Improvements in Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| CreateInstance - Duration of single invocation | 2.68 μs | 1.99 μs | 0.74 | 0.48 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark*'Details
Payloads
Histogram
Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark.CreateInstance
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9921491510865452 < 2.6459349244007577.
IsChangePoint: Marked as a change because one of 4/7/2022 2:11:06 AM, 5/9/2022 9:57:04 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 41.60724521394111 (T) = (0 -2043.1054846403565) / Math.Sqrt((116689.74443120719 / (299)) + (2982.151543826062 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.34371233405079393 = (3113.1249155587275 - 2043.1054846403565) / 3113.1249155587275 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
PGOApplied if there were any profile guided optimization updates in the observed interval.Applied if there were any profile guided optimization updates in the observed interval.arch-arm64branch-refs/heads/mainkind-microos-linuxperf-improvementruntime-coreclr











