Skip to content

Regressions in SequenceEqual #67596

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 4019e83878a81465f6e42e8502b53bc5d1752f81
Compare 019bc7f143ae5840fe7d0e8e720e4854b1a91c26
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsSuffix_DifferentLastChar - Duration of single invocation 16.64 ns 19.98 ns 1.20 0.29 False
IsPrefix_FirstHalf - Duration of single invocation 12.03 ns 13.96 ns 1.16 0.18 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'
Details

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, Ordinal, False))


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.97911225341915 > 17.44699084664059.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/28/2022 7:37:41 AM, 4/5/2022 2:02:34 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 2:02:34 AM.
IsRegressionStdDev: Marked as regression because -13.002365424661688 (T) = (0 -19.115487599893818) / Math.Sqrt((0.32787265954056094 / (19)) + (0.5051995568270351 / (36))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (36) - 2, .025) and -0.13677715168127164 = (16.8155100334506 - 19.115487599893818) / 16.8155100334506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, Ordinal, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.957168843212287 > 12.539578215438409.
IsChangePoint: Marked as a change because one of 2/4/2022 6:00:25 PM, 3/2/2022 3:17:47 PM, 3/30/2022 4:48:39 AM, 4/5/2022 2:02:34 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 2:02:34 AM.
IsRegressionStdDev: Marked as regression because -10.00018437580746 (T) = (0 -14.18687111136129) / Math.Sqrt((0.34624076266092135 / (28)) + (0.5333189236045973 / (27))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (27) - 2, .025) and -0.1445935477736601 = (12.39468031158839 - 14.18687111136129) / 12.39468031158839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 4019e83878a81465f6e42e8502b53bc5d1752f81
Compare 019bc7f143ae5840fe7d0e8e720e4854b1a91c26
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SequenceEqual - Duration of single invocation 12.39 ns 14.71 ns 1.19 0.13 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.SequenceEqual(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.712574604126564 > 13.072225077291382.
IsChangePoint: Marked as a change because one of 1/28/2022 6:30:08 PM, 3/11/2022 8:42:43 PM, 3/17/2022 6:54:53 PM, 3/28/2022 7:37:41 AM, 4/5/2022 2:02:34 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 2:02:34 AM.
IsRegressionStdDev: Marked as regression because -8.217728367436298 (T) = (0 -13.685223290119147) / Math.Sqrt((0.21000824790973555 / (18)) + (0.39593923483020577 / (36))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (18) + (36) - 2, .025) and -0.09938774412884786 = (12.448040614608892 - 13.685223290119147) / 12.448040614608892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 4019e83878a81465f6e42e8502b53bc5d1752f81
Compare 019bc7f143ae5840fe7d0e8e720e4854b1a91c26
Diff Diff

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 54.44 ns 67.42 ns 1.24 0.17 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringEquality*'
Details

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, Ordinal))


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 67.42402506213939 > 57.61935682181894.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 3/17/2022 6:54:53 PM, 3/28/2022 7:37:41 AM, 4/5/2022 2:02:34 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 2:02:34 AM.
IsRegressionStdDev: Marked as regression because -24.152920412669175 (T) = (0 -71.68631198538739) / Math.Sqrt((2.5467050998996177 / (19)) + (13.176893030965825 / (36))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (36) - 2, .025) and -0.31277837319955654 = (54.606560748461 - 71.68631198538739) / 54.606560748461 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtime

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions