[Perf] Changes at 5/5/2022 2:36:12 AM #5277
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in System.Tests.Perf_Version
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryFormat2 - Duration of single invocation | 13.11 ns | 10.16 ns | 0.77 | 0.13 | False | |||||
| ToString4 - Duration of single invocation | 44.36 ns | 38.92 ns | 0.88 | 0.07 | False | |||||
| TryFormat3 - Duration of single invocation | 20.74 ns | 16.29 ns | 0.79 | 0.11 | False | |||||
| ToStringL - Duration of single invocation | 101.38 ns | 89.37 ns | 0.88 | 0.07 | False | |||||
| TryFormatL - Duration of single invocation | 76.03 ns | 65.22 ns | 0.86 | 0.06 | False | |||||
| TryFormat4 - Duration of single invocation | 26.37 ns | 21.46 ns | 0.81 | 0.11 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Version*'Details
Payloads
Histogram
System.Tests.Perf_Version.TryFormat2
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.156135017358991 < 12.447406190546666.
IsChangePoint: Marked as a change because one of 1/26/2022 3:56:56 PM, 1/28/2022 4:34:00 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 22.510585171556585 (T) = (0 -10.324428237524652) / Math.Sqrt((1.5698904276058796 / (299)) + (0.18561638180247708 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.2287677839474392 = (13.386925523377027 - 10.324428237524652) / 13.386925523377027 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToString4
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 38.92177348386976 < 42.55216623919884.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 44.72902395538345 (T) = (0 -39.23482344830163) / Math.Sqrt((7.155384659295994 / (299)) + (0.07412194461353507 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.1631056981814828 = (46.88145607282413 - 39.23482344830163) / 46.88145607282413 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.TryFormat3
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 16.288200611767465 < 19.827425186559.
IsChangePoint: Marked as a change because one of 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 50.8099671040967 (T) = (0 -16.0467298079918) / Math.Sqrt((1.2323254423582228 / (299)) + (0.08513733638032756 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.2423272263065603 = (21.178971140494525 - 16.0467298079918) / 21.178971140494525 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToStringL
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 89.36785235665651 < 96.38995578879334.
IsChangePoint: Marked as a change because one of 1/28/2022 4:34:00 PM, 3/18/2022 7:38:50 AM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 67.86321986714229 (T) = (0 -89.23583131227322) / Math.Sqrt((11.65016132916743 / (299)) + (0.25317746651988143 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.15371956539839224 = (105.44475290189308 - 89.23583131227322) / 105.44475290189308 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.TryFormatL
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 65.21725445608126 < 72.33431117516427.
IsChangePoint: Marked as a change because one of 1/28/2022 4:34:00 PM, 4/12/2022 3:59:36 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 94.16074508910437 (T) = (0 -65.4158891893674) / Math.Sqrt((4.129270327855717 / (299)) + (0.10572957990622883 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.17381550383164343 = (79.17830641067515 - 65.4158891893674) / 79.17830641067515 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.TryFormat4
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 21.46265896789983 < 25.658046188219583.
IsChangePoint: Marked as a change because one of 1/26/2022 8:19:47 PM, 1/28/2022 4:34:00 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 56.7644478842839 (T) = (0 -21.40206300726871) / Math.Sqrt((1.5281403845192194 / (299)) + (0.07645121954761293 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.21427151328388816 = (27.238496973320753 - 21.40206300726871) / 27.238496973320753 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in System.Tests.Perf_Int32
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToStringHex - Duration of single invocation | 17.45 ns | 15.26 ns | 0.87 | 0.26 | False | |||||
| TryFormat - Duration of single invocation | 13.91 ns | 11.03 ns | 0.79 | 0.02 | False | |||||
| TryFormat - Duration of single invocation | 6.55 ns | 4.32 ns | 0.66 | 0.37 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Int32*'Details
Payloads
Histogram
System.Tests.Perf_Int32.ToStringHex(value: 4)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 15.261530250227624 < 16.61835971652599.
IsChangePoint: Marked as a change because one of 2/14/2022 3:33:45 PM, 2/16/2022 2:51:06 AM, 4/18/2022 10:15:32 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 13.920474878504404 (T) = (0 -15.166156958738908) / Math.Sqrt((1.6836108603997342 / (299)) + (0.10859077683128955 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.09600440257331966 = (16.776804004257308 - 15.166156958738908) / 16.776804004257308 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int32.TryFormat(value: 2147483647)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.029851874419347 < 13.214466432229184.
IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 1/28/2022 4:34:00 PM, 4/12/2022 3:59:36 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 63.63453617994623 (T) = (0 -11.164317494904457) / Math.Sqrt((0.8012942391056729 / (299)) + (0.008062722104031176 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.2454142038465779 = (14.795292399904294 - 11.164317494904457) / 14.795292399904294 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int32.TryFormat(value: 12345)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.324528211001206 < 6.073923060322746.
IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 1/28/2022 4:34:00 PM, 4/12/2022 3:59:36 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 31.500212790849563 (T) = (0 -4.448839389404454) / Math.Sqrt((0.5064794325630185 / (299)) + (0.07933646142108555 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.3779068289457573 = (7.151403674573599 - 4.448839389404454) / 7.151403674573599 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | 6111bc7fcfc5445e8d48b2321edb5ba2b1461ae5 |
| Compare | d66d72dcc26cb1443f3ed3df92a2d91c17b2f7c4 |
| Diff | Diff |
Improvements in System.Tests.Perf_UInt32
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| TryFormat - Duration of single invocation | 13.39 ns | 10.30 ns | 0.77 | 0.02 | False | |||||
| TryFormat - Duration of single invocation | 6.08 ns | 3.84 ns | 0.63 | 0.36 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt32*'Details
Payloads
Histogram
System.Tests.Perf_UInt32.TryFormat(value: 4294967295)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.297431457959025 < 12.710938672574922.
IsChangePoint: Marked as a change because one of 1/28/2022 4:34:00 PM, 4/12/2022 3:59:36 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 47.054350158260185 (T) = (0 -10.69694106147835) / Math.Sqrt((0.8891379884219709 / (299)) + (0.038239323939367166 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.24939283743453275 = (14.251051142274935 - 10.69694106147835) / 14.251051142274935 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_UInt32.TryFormat(value: 12345)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.8358603579981123 < 5.783137009819102.
IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 1/28/2022 4:34:00 PM, 4/12/2022 3:59:36 PM, 5/5/2022 12:13:02 AM, 5/9/2022 6:32:22 AM falls between 4/30/2022 6:29:51 PM and 5/9/2022 6:32:22 AM.
IsImprovementStdDev: Marked as improvement because 40.00688019448571 (T) = (0 -3.7955359025325808) / Math.Sqrt((0.6434628528094238 / (299)) + (0.04406003631921618 / (14))) is greater than 1.9676211333067681 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (14) - 2, .975) and 0.43416586821958236 = (6.707859581728994 - 3.7955359025325808) / 6.707859581728994 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository