-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | b098b6ffb9274fd9f9fd6db943e291ce9909af84 |
| Compare | 9d84900aedd34a7d15e2fb8b92614f0429919061 |
| Diff | Diff |
Regressions in System.Tests.Perf_Single
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToStringWithFormat - Duration of single invocation | 280.44 ns | 301.33 ns | 1.07 | 0.07 | False | |||||
| TryParse - Duration of single invocation | 92.64 ns | 101.20 ns | 1.09 | 0.01 | True | |||||
| TryParse - Duration of single invocation | 94.60 ns | 105.55 ns | 1.12 | 0.02 | True | |||||
| Parse - Duration of single invocation | 55.41 ns | 61.30 ns | 1.11 | 0.17 | False | |||||
| Parse - Duration of single invocation | 91.58 ns | 98.98 ns | 1.08 | 0.03 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Single*'Details
Payloads
Histogram
System.Tests.Perf_Single.ToStringWithFormat(value: 12345, format: "F50")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 301.3275753663291 > 293.216352696102.
IsChangePoint: Marked as a change because one of 6/1/2022 9:42:28 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -10.49815622599495 (T) = (0 -301.11228467256154) / Math.Sqrt((37.425865380307755 / (31)) + (57.66549688765729 / (38))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (38) - 2, .025) and -0.061065212626035335 = (283.78301454944256 - 301.11228467256154) / 283.78301454944256 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Single.TryParse(value: "3.4028235E+38")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 101.19993338398353 > 96.26737779006918.
IsChangePoint: Marked as a change because one of 5/30/2022 10:25:35 AM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -34.821637732616765 (T) = (0 -100.40305073252196) / Math.Sqrt((0.8011819049067009 / (19)) + (1.0536938843521206 / (51))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (51) - 2, .025) and -0.09520860667480774 = (91.67481895285535 - 100.40305073252196) / 91.67481895285535 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Tests.Perf_Single.TryParse(value: "-3.4028235E+38")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 105.54556012489527 > 99.18096343671321.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -38.87700820738145 (T) = (0 -104.32846958055814) / Math.Sqrt((0.8952174729561101 / (20)) + (0.8076337887781272 / (50))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (50) - 2, .025) and -0.10128539666551359 = (94.7333633011436 - 104.32846958055814) / 94.7333633011436 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.Tests.Perf_Single.Parse(value: "12345")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.29601597166671 > 58.18371736430522.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -11.124918512825896 (T) = (0 -64.54790360457493) / Math.Sqrt((6.863970588663012 / (20)) + (3.4067048794975814 / (51))) is less than -1.9949454151066512 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (51) - 2, .025) and -0.12404798365617772 = (57.42450904508606 - 64.54790360457493) / 57.42450904508606 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Single.Parse(value: "3.4028235E+38")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 98.9755246387004 > 96.12754731083413.
IsChangePoint: Marked as a change because one of 5/30/2022 12:57:05 PM, 6/7/2022 9:00:42 AM falls between 5/29/2022 5:15:01 PM and 6/7/2022 9:00:42 AM.
IsRegressionStdDev: Marked as regression because -26.492454971702756 (T) = (0 -100.56076251062125) / Math.Sqrt((0.8138973113942265 / (20)) + (2.8280045514436707 / (50))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (50) - 2, .025) and -0.0895120620702221 = (92.29889783830573 - 100.56076251062125) / 92.29889783830573 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository




