Skip to content

[Perf] Changes at 5/28/2022 1:41:21 AM #5607

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 2a6613530b19fafb34c2c1dd731f7e4672d3f53c
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 42.01 ms 47.49 ms 1.13 0.01 False
Count - Duration of single invocation 41.17 ms 45.62 ms 1.11 0.01 False
Count - Duration of single invocation 231.77 μs 253.39 μs 1.09 0.01 False
Count - Duration of single invocation 11.90 ms 16.75 ms 1.41 0.01 False
Count - Duration of single invocation 21.69 ms 25.09 ms 1.16 0.01 False
Count - Duration of single invocation 11.90 ms 16.69 ms 1.40 0.02 False
Count - Duration of single invocation 9.68 ms 14.14 ms 1.46 0.06 False
Count - Duration of single invocation 1.97 ms 2.16 ms 1.09 0.00 True

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{L}", Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 47.49373333333334 > 44.16333395833333.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -25.823696923532644 (T) = (0 -47767521.58333334) / Math.Sqrt((564176417200.65 / (50)) + (399313633722.2267 / (15))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (15) - 2, .025) and -0.11763323545021047 = (42739890.03565324 - 47767521.58333334) / 42739890.03565324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\p{Ll}", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.619445 > 43.174713749999995.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -41.69313213811837 (T) = (0 -45939707.05128205) / Math.Sqrt((276024421547.8084 / (50)) + (91761029816.27515 / (13))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (13) - 2, .025) and -0.11332386605066629 = (41263560.81294262 - 45939707.05128205) / 41263560.81294262 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 253.38996960486327 > 243.6993125.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -34.35576102353137 (T) = (0 -252746.74552683305) / Math.Sqrt((2250489.9093127907 / (49)) + (5140152.737177277 / (15))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (15) - 2, .025) and -0.09258102083445281 = (231329.97984332466 - 252746.74552683305) / 231329.97984332466 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.749198888888888 > 12.4504695.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -68.62335470018509 (T) = (0 -16832811.37883598) / Math.Sqrt((7844497644.202676 / (49)) + (75517512772.36484 / (15))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (15) - 2, .025) and -0.41608269212643667 = (11886884.482402135 - 16832811.37883598) / 11886884.482402135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.091336666666667 > 22.825660375.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -51.48526013158499 (T) = (0 -25376219.88888889) / Math.Sqrt((58078979384.77318 / (50)) + (56887712917.645454 / (15))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (15) - 2, .025) and -0.1665933183764931 = (21752413.19246032 - 25376219.88888889) / 21752413.19246032 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[^\n]*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.685900888888888 > 12.455041375.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -118.36744543845774 (T) = (0 -16752596.668376068) / Math.Sqrt((36180846514.71715 / (50)) + (14244914155.489752 / (15))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (15) - 2, .025) and -0.40651759272569365 = (11910691.167332768 - 16752596.668376068) / 11910691.167332768 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.144534814814813 > 10.164228800395257.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -72.59141441185093 (T) = (0 -14155371.323260073) / Math.Sqrt((15692422789.547808 / (49)) + (44317596582.43899 / (13))) is less than -2.0002978220134566 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (13) - 2, .025) and -0.4559604854653995 = (9722359.54517357 - 14155371.323260073) / 9722359.54517357 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.157654550724638 > 2.0498066959755032.
IsChangePoint: Marked as a change because one of 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -43.511489676202785 (T) = (0 -2172272.4674796513) / Math.Sqrt((243204052.47117305 / (49)) + (349714805.22875917 / (15))) is less than -1.998971517032675 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (15) - 2, .025) and -0.11921317869845952 = (1940892.502718563 - 2172272.4674796513) / 1940892.502718563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 2a6613530b19fafb34c2c1dd731f7e4672d3f53c
Compare 748dfb78ef87b38599b89e85d3201cdb9f9c0b33
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 5.06 ms 8.95 ms 1.77 0.03 True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\s?#]+[^\s?#]+(?:?[^\s#])?(?:#[^\s])?", Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.954105357142858 > 5.295534534285714.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsRegressionStdDev: Marked as regression because -97.6897327827504 (T) = (0 -8902021.698715664) / Math.Sqrt((8878969343.112051 / (50)) + (20157558939.178204 / (15))) is less than -1.9983405425199077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (15) - 2, .025) and -0.7483740122402754 = (5091600.330588921 - 8902021.698715664) / 5091600.330588921 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions