Regressions in System.Text.Perf_Utf8Encoding and System.Text.Tests.Perf_Encoding #80247
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.25094 |
| Baseline | 279fb0436f475fbc35ffeff68330f970ee77831a |
| Compare | 8d66372138fca4c9c73eef07c1632f13a0887d8e |
| Diff | Diff |
Regressions in System.Text.Perf_Utf8Encoding
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| GetByteCount - Duration of single invocation | 38.55 μs | 58.79 μs | 1.52 | 0.63 | False | |||||
| GetBytes - Duration of single invocation | 203.89 μs | 254.92 μs | 1.25 | 0.60 | False | |||||
| GetByteCount - Duration of single invocation | 31.05 μs | 59.27 μs | 1.91 | 0.63 | False | |||||
| GetByteCount - Duration of single invocation | 71.94 μs | 127.82 μs | 1.78 | 0.63 | False | |||||
| GetByteCount - Duration of single invocation | 23.92 μs | 47.40 μs | 1.98 | 0.65 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'Details
Payloads
Histogram
System.Text.Perf_Utf8Encoding.GetByteCount(Input: Greek)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 58.78677252650176 > 38.56174266572224.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -24.642880356633054 (T) = (0 -61272.045484563685) / Math.Sqrt((3231635.7490745448 / (13)) + (13770696.509863114 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.6375680776935077 = (37416.487484822326 - 61272.045484563685) / 37416.487484822326 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishMostlyAscii)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 254.92176315789467 > 202.5638441984954.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -11.030456260848213 (T) = (0 -249154.62405981254) / Math.Sqrt((201147770.39900446 / (13)) + (151574382.4788075 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.2699330956401716 = (196195.0790283279 - 249154.62405981254) / 196195.0790283279 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Chinese)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.265270161290324 > 35.61435114979619.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -23.49290089307371 (T) = (0 -58515.49681304273) / Math.Sqrt((4190721.2612498812 / (13)) + (15601297.696510985 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.7287340583031703 = (33848.755701891074 - 58515.49681304273) / 33848.755701891074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: EnglishMostlyAscii)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 127.81981534090906 > 80.15821677714646.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -20.88937159330011 (T) = (0 -122573.92458679804) / Math.Sqrt((6202223.574572832 / (13)) + (89470089.16861711 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.6107971293310409 = (76095.19681581666 - 122573.92458679804) / 76095.19681581666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Perf_Utf8Encoding.GetByteCount(Input: Cyrillic)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 47.402465206185575 > 26.441110882146138.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -24.18386114944223 (T) = (0 -47817.66947918986) / Math.Sqrt((3340879.0019299146 / (13)) + (12035498.973049158 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.8820978500452556 = (25406.579938466042 - 47817.66947918986) / 25406.579938466042 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.25094 |
| Baseline | 279fb0436f475fbc35ffeff68330f970ee77831a |
| Compare | 8d66372138fca4c9c73eef07c1632f13a0887d8e |
| Diff | Diff |
Regressions in System.Text.Tests.Perf_Encoding
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| GetBytes - Duration of single invocation | 39.97 ns | 50.39 ns | 1.26 | 0.65 | False | |||||
| GetChars - Duration of single invocation | 43.40 ns | 51.90 ns | 1.20 | 0.65 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'Details
Payloads
Histogram
System.Text.Tests.Perf_Encoding.GetBytes(size: 16, encName: "ascii")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.38990552918499 > 42.86550871073043.
IsChangePoint: Marked as a change because one of 10/17/2022 3:02:22 AM, 12/21/2022 10:31:08 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -5.127555412582627 (T) = (0 -44.637560236603335) / Math.Sqrt((8.958820198634074 / (13)) + (10.93650399530871 / (20))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (20) - 2, .025) and -0.1464031449234489 = (38.93705319482878 - 44.637560236603335) / 38.93705319482878 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "utf-8")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.8996410785766 > 48.899269298461725.
IsChangePoint: Marked as a change because one of 12/22/2022 9:36:43 AM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -3.395306959489965 (T) = (0 -54.93801804432714) / Math.Sqrt((20.718999452564333 / (17)) + (12.97011407759705 / (16))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (16) - 2, .025) and -0.09654163952613115 = (50.101169042762955 - 54.93801804432714) / 50.101169042762955 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository