-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in System.Tests.Perf_UInt64
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryFormat - Duration of single invocation | 16.22 ns | 3.47 ns | 0.21 | 0.82 | False | |||||
| TryFormat - Duration of single invocation | 66.64 ns | 41.92 ns | 0.63 | 0.41 | True | |||||
| TryFormat - Duration of single invocation | 25.06 ns | 9.95 ns | 0.40 | 0.78 | False | |||||
| ToString - Duration of single invocation | 35.24 ns | 28.92 ns | 0.82 | 0.58 | False | |||||
| ToString - Duration of single invocation | 85.85 ns | 68.36 ns | 0.80 | 0.48 | False | |||||
| ToString - Duration of single invocation | 13.09 ns | 3.54 ns | 0.27 | 0.83 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*'Details
Payloads
Histogram
System.Tests.Perf_UInt64.TryFormat(value: 0)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.4665684696318304 < 14.028606844601768.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 21.266340115464686 (T) = (0 -2.57192927689081) / Math.Sqrt((8.711951755108869 / (299)) + (4.757408586640993 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.7867349395731131 = (12.059777967111204 - 2.57192927689081) / 12.059777967111204 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt64.TryFormat(value: 18446744073709551615)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 41.91975302634436 < 63.571802047728966.
IsChangePoint: Marked as a change because one of 1/11/2022 6:41:22 PM, 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 62.467283467108025 (T) = (0 -42.97236417381113) / Math.Sqrt((16.44035103608032 / (290)) + (3.5456136688105215 / (28))) is greater than 1.9674995188240387 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (290) + (28) - 2, .975) and 0.38362844228545506 = (69.71827891142337 - 42.97236417381113) / 69.71827891142337 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Tests.Perf_UInt64.TryFormat(value: 12345)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.95276842171851 < 22.049936238156256.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 23.197629177318678 (T) = (0 -8.94652077951926) / Math.Sqrt((10.040396931827939 / (275)) + (7.346370346563901 / (28))) is greater than 1.9678765312856872 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (275) + (28) - 2, .975) and 0.5863581592891736 = (21.628664943916295 - 8.94652077951926) / 21.628664943916295 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt64.ToString(value: 12345)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 28.918512355551762 < 33.35829453426279.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 30.595087762649904 (T) = (0 -29.018131182520893) / Math.Sqrt((29.524225448516383 / (299)) + (0.33443931987097253 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.2596855393661085 = (39.19703413286595 - 29.018131182520893) / 39.19703413286595 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt64.ToString(value: 18446744073709551615)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 68.35994018325134 < 77.226127744924.
IsChangePoint: Marked as a change because one of 1/27/2022 11:49:03 PM, 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 43.1325674281997 (T) = (0 -67.1638575213719) / Math.Sqrt((58.32764880014823 / (299)) + (2.0790268220890233 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.24997005739675038 = (89.54823495213469 - 67.1638575213719) / 89.54823495213469 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt64.ToString(value: 0)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.5439418410955996 < 9.328477804713275.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 14.492154853291531 (T) = (0 -4.545899375837662) / Math.Sqrt((5.625824517841812 / (299)) + (2.275923221023555 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.5021453051505455 = (9.130976212270708 - 4.545899375837662) / 9.130976212270708 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in System.Tests.Perf_Int64
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryFormat - Duration of single invocation | 73.64 ns | 43.09 ns | 0.59 | 0.45 | False | |||||
| TryFormat - Duration of single invocation | 21.84 ns | 13.86 ns | 0.63 | 0.73 | False | |||||
| ToString - Duration of single invocation | 88.55 ns | 72.61 ns | 0.82 | 0.47 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int64*'Details
Payloads
Histogram
System.Tests.Perf_Int64.TryFormat(value: 9223372036854775807)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 43.08853281035844 < 60.752758808970654.
IsChangePoint: Marked as a change because one of 1/28/2022 4:34:00 PM, 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 56.949381621800754 (T) = (0 -41.239425601221704) / Math.Sqrt((16.388150087727134 / (299)) + (5.315770234673443 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.4058429929163168 = (69.40829630813964 - 41.239425601221704) / 69.40829630813964 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryFormat(value: 12345)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.863692250415282 < 21.82158734640099.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 17.879463712784954 (T) = (0 -11.3278117417261) / Math.Sqrt((9.773724256259978 / (299)) + (12.011001114325884 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.5174729558628449 = (23.47601420347798 - 11.3278117417261) / 23.47601420347798 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.ToString(value: 9223372036854775807)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 72.61198840971196 < 85.22773409594217.
IsChangePoint: Marked as a change because one of 1/28/2022 9:02:40 AM, 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 41.65058829694537 (T) = (0 -70.05438928433128) / Math.Sqrt((26.478729788444145 / (299)) + (7.722696953128098 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.26410288284755606 = (95.19590123604091 - 70.05438928433128) / 95.19590123604091 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in System.Tests.Perf_Version
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToString2 - Duration of single invocation | 75.09 ns | 65.37 ns | 0.87 | 0.51 | False | |||||
| ToString4 - Duration of single invocation | 100.12 ns | 87.65 ns | 0.88 | 0.50 | False | |||||
| ToString3 - Duration of single invocation | 88.69 ns | 80.57 ns | 0.91 | 0.45 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Version*'Details
Payloads
Histogram
System.Tests.Perf_Version.ToString2
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 65.36856615508697 < 70.90750268598526.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 12.569728449318905 (T) = (0 -65.6509830588634) / Math.Sqrt((24.355234602100786 / (275)) + (6.824289301555473 / (28))) is greater than 1.9678765312856872 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (275) + (28) - 2, .975) and 0.0993975827089026 = (72.89674311149817 - 65.6509830588634) / 72.89674311149817 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToString4
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 87.65206635608342 < 93.65231997840058.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 17.05433188576439 (T) = (0 -82.66660460152009) / Math.Sqrt((38.737021869579124 / (299)) + (18.756542150702074 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.15573132716330482 = (97.91504441798718 - 82.66660460152009) / 97.91504441798718 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToString3
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 80.56997386336843 < 83.62590549092006.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 13.91601619719142 (T) = (0 -75.09268908011379) / Math.Sqrt((31.73953649621583 / (291)) + (18.320504383826623 / (28))) is greater than 1.9674756580274855 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (291) + (28) - 2, .975) and 0.1393515564782871 = (87.25129249387797 - 75.09268908011379) / 87.25129249387797 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in GuardedDevirtualization.TwoClassInterface
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Call - Duration of single invocation | 5.72 ns | 4.08 ns | 0.71 | 0.41 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'GuardedDevirtualization.TwoClassInterface*'Details
Payloads
Histogram
GuardedDevirtualization.TwoClassInterface.Call(testInput: pB = 0.90)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.084067636278195 < 5.486973504155973.
IsChangePoint: Marked as a change because one of 4/22/2022 1:40:09 PM, 5/1/2022 10:22:11 AM, 5/5/2022 12:13:02 AM, 5/12/2022 5:52:45 AM falls between 5/3/2022 7:02:53 AM and 5/12/2022 5:52:45 AM.
IsImprovementStdDev: Marked as improvement because 4.855352520355236 (T) = (0 -4.493065192247391) / Math.Sqrt((1.117472670289252 / (299)) + (0.6272157914660806 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.1487248689599573 = (5.278041174253502 - 4.493065192247391) / 5.278041174253502 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository












