Skip to content

[Perf] ubuntu 18.04/x64 : Regression on 8/24/2022 9:42:22 AM #8255

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline affc0fc7a79812b92445eb474c87533e32b35127
Compare 86412566424353b2d9b32d629fe7efb66346ac38
Diff Diff

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithFormat - Duration of single invocation 218.55 ns 234.14 ns 1.07 0.07 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Single*'
Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Single.ToStringWithFormat(value: 12345, format: "F50")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 234.1446869837515 > 230.4713008400628.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 8/24/2022 2:39:26 AM, 9/1/2022 1:45:20 PM falls between 8/23/2022 11:46:46 PM and 9/1/2022 1:45:20 PM.
IsRegressionStdDev: Marked as regression because -11.933212438244922 (T) = (0 -232.26003690893282) / Math.Sqrt((15.735418770881013 / (17)) + (10.960431002755968 / (46))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (46) - 2, .025) and -0.058681740518831856 = (219.3860798950857 - 232.26003690893282) / 219.3860798950857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline affc0fc7a79812b92445eb474c87533e32b35127
Compare 86412566424353b2d9b32d629fe7efb66346ac38
Diff Diff

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithFormat - Duration of single invocation 226.19 ns 244.63 ns 1.08 0.11 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Double*'
Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.ToStringWithFormat(value: 12345, format: "F50")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 244.62571816903343 > 237.16361016675182.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 8/24/2022 2:39:26 AM, 9/1/2022 1:45:20 PM falls between 8/23/2022 11:46:46 PM and 9/1/2022 1:45:20 PM.
IsRegressionStdDev: Marked as regression because -15.193521699282348 (T) = (0 -249.00495672680145) / Math.Sqrt((16.942290263584578 / (17)) + (32.025812827859845 / (46))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (46) - 2, .025) and -0.08623411523858518 = (229.23691424671276 - 249.00495672680145) / 229.23691424671276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline affc0fc7a79812b92445eb474c87533e32b35127
Compare 86412566424353b2d9b32d629fe7efb66346ac38
Diff Diff

Regressions in System.Net.Tests.Perf_WebUtility

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Decode_NoDecodingRequired - Duration of single invocation 62.09 ns 74.63 ns 1.20 0.03 False
Decode_DecodingRequired - Duration of single invocation 96.85 ns 110.53 ns 1.14 0.02 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Net.Tests.Perf_WebUtility*'
Details

Payloads

Baseline
Compare

Histogram

System.Net.Tests.Perf_WebUtility.Decode_NoDecodingRequired


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.62563026608555 > 64.47159203333908.
IsChangePoint: Marked as a change because one of 8/24/2022 2:39:26 AM, 9/1/2022 1:45:20 PM falls between 8/23/2022 11:46:46 PM and 9/1/2022 1:45:20 PM.
IsRegressionStdDev: Marked as regression because -81.17770889130645 (T) = (0 -74.5405181097264) / Math.Sqrt((0.33999685947579533 / (17)) + (0.3264314994569256 / (47))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (47) - 2, .025) and -0.21767967119876525 = (61.21521108777626 - 74.5405181097264) / 61.21521108777626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.53292997551272 > 101.58797140367841.
IsChangePoint: Marked as a change because one of 8/24/2022 2:39:26 AM, 9/1/2022 1:45:20 PM falls between 8/23/2022 11:46:46 PM and 9/1/2022 1:45:20 PM.
IsRegressionStdDev: Marked as regression because -43.27633389439559 (T) = (0 -110.45661554543427) / Math.Sqrt((0.8805203234443126 / (17)) + (2.3892855263197803 / (46))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (46) - 2, .025) and -0.14441323069648826 = (96.51812176114963 - 110.45661554543427) / 96.51812176114963 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline affc0fc7a79812b92445eb474c87533e32b35127
Compare 86412566424353b2d9b32d629fe7efb66346ac38
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 1.57 μs 1.73 μs 1.10 0.11 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.Stack(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7310674038827107 > 1.6456527502399736.
IsChangePoint: Marked as a change because one of 8/5/2022 8:01:14 PM, 8/9/2022 6:41:10 AM, 8/24/2022 2:39:26 AM, 9/1/2022 1:45:20 PM falls between 8/23/2022 11:46:46 PM and 9/1/2022 1:45:20 PM.
IsRegressionStdDev: Marked as regression because -19.3358534849793 (T) = (0 -1752.1004995086198) / Math.Sqrt((813.1735585008805 / (17)) + (1566.7266962826457 / (45))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (45) - 2, .025) and -0.11151714224393733 = (1576.3144201010423 - 1752.1004995086198) / 1576.3144201010423 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions