Skip to content

[Perf] Changes at 6/7/2022 3:01:09 PM #6130

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 0971e5e22794fc7d7d0744376be2f452e26159ad
Compare 17d6d50eb9d74905f83e709093d1cc32489226ee
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 3.77 ms 3.20 ms 0.85 0.05 False
Count - Duration of single invocation 83.00 μs 71.38 μs 0.86 0.01 False
Count - Duration of single invocation 207.48 μs 183.09 μs 0.88 0.05 False
Count - Duration of single invocation 2.08 ms 1.90 ms 0.92 0.12 False
Count - Duration of single invocation 11.40 ms 5.75 ms 0.50 0.08 False
Count - Duration of single invocation 24.37 ms 19.42 ms 0.80 0.05 False
Count - Duration of single invocation 3.77 ms 3.23 ms 0.86 0.06 False
Count - Duration of single invocation 14.34 ms 8.42 ms 0.59 0.11 False
Count - Duration of single invocation 13.98 ms 8.33 ms 0.60 0.17 False
Count - Duration of single invocation 164.87 μs 147.35 μs 0.89 0.01 False
Count - Duration of single invocation 4.36 ms 3.61 ms 0.83 0.10 False
Count - Duration of single invocation 1.87 ms 1.48 ms 0.79 0.03 False
Count - Duration of single invocation 11.06 ms 8.38 ms 0.76 0.03 False
Count - Duration of single invocation 7.30 ms 5.93 ms 0.81 0.02 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes\s+\w+", Options: NonBacktracking)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.199027863247864 < 3.55064875.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 100.3173114672644 (T) = (0 -3195342.715821248) / Math.Sqrt((6718761276.070281 / (299)) + (21123097.87668557 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.13076737156411525 = (3676050.128917749 - 3195342.715821248) / 3676050.128917749 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 71.37593375214163 < 79.33546607824223.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/11/2022 7:02:26 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 57.55799410530936 (T) = (0 -71816.68306240285) / Math.Sqrt((12341306.618810702 / (299)) + (253967.49093380987 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.14827430578421247 = (84319.02847374692 - 71816.68306240285) / 84319.02847374692 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 183.08695992179864 < 198.63616796507162.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 18.616511827957094 (T) = (0 -180588.73160501654) / Math.Sqrt((80290394.38323839 / (299)) + (20747516.51045543 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.08201129698309366 = (196722.17208286352 - 180588.73160501654) / 196722.17208286352 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9036979166666668 < 1.95657509765625.
IsChangePoint: Marked as a change because one of 4/13/2022 5:30:39 PM, 5/9/2022 5:23:46 PM, 5/24/2022 7:55:26 PM, 6/6/2022 10:51:35 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 7.563674611694471 (T) = (0 -1907499.6613641898) / Math.Sqrt((16468025341.77603 / (299)) + (8562635071.521588 / (51))) is greater than 1.9668042230483112 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (51) - 2, .975) and 0.05589994412860983 = (2020442.2714535233 - 1907499.6613641898) / 2020442.2714535233 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.749009157509158 < 10.828340684065935.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 18.419953185348252 (T) = (0 -5790086.7337872535) / Math.Sqrt((4119026555380.6543 / (299)) + (1511368177.2524383 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.27212811926633507 = (7954815.795262051 - 5790086.7337872535) / 7954815.795262051 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 19.418846153846154 < 22.843148208333336.
IsChangePoint: Marked as a change because one of 4/25/2022 9:16:47 PM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 9.213622267805754 (T) = (0 -19469490.626756962) / Math.Sqrt((4006779685782.945 / (299)) + (80390707331.743 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.055247850632349896 = (20608040.574227277 - 19469490.626756962) / 20608040.574227277 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\w+\s+Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.22815922077922 < 3.5875719249999998.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 75.33630967338652 (T) = (0 -3226832.787375046) / Math.Sqrt((10714777212.296322 / (299)) + (41624725.28770018 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.12406360028215635 = (3683866.532335535 - 3226832.787375046) / 3683866.532335535 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.424738666666666 < 13.620979666666665.
IsChangePoint: Marked as a change because one of 4/19/2022 10:22:09 AM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 15.204595899015477 (T) = (0 -8379050.716386973) / Math.Sqrt((5853567635765.7 / (299)) + (5701623503.498569 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.2030308669506545 = (10513645.220269995 - 8379050.716386973) / 10513645.220269995 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[^\n]*", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.330201333333333 < 13.428961651785714.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 16.413149041490495 (T) = (0 -8351715.362380088) / Math.Sqrt((5501914882570.705 / (299)) + (4147845956.557764 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.21091087625587346 = (10583995.028029624 - 8351715.362380088) / 10583995.028029624 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 147.35072327044028 < 156.66570428716355.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 124.78093878820846 (T) = (0 -146988.92403361763) / Math.Sqrt((6223394.920037474 / (299)) + (119991.35665970553 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.11537706315888606 = (166159.9738285083 - 146988.92403361763) / 166159.9738285083 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\s[a-zA-Z]{0,12}ing\s", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.606279152731327 < 4.146635472222222.
IsChangePoint: Marked as a change because one of 5/25/2022 3:58:08 AM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 84.05739425918891 (T) = (0 -3623277.759953868) / Math.Sqrt((12240794272.575794 / (299)) + (312906072.8966819 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.13874067584163177 = (4206953.304679254 - 3623277.759953868) / 4206953.304679254 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.4801576331360946 < 1.7771793086419752.
IsChangePoint: Marked as a change because one of 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 27.678587128123166 (T) = (0 -1483261.7811769806) / Math.Sqrt((11464686697.194645 / (299)) + (571512730.4759716 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.11823562287174119 = (1682152.0801369704 - 1483261.7811769806) / 1682152.0801369704 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "\b\w+n\b", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.37897511111111 < 10.659096758116881.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 42.38559236702495 (T) = (0 -8330594.24569443) / Math.Sqrt((56297250463.33345 / (299)) + (156402565644.0658 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.23936714039489068 = (10952188.221291605 - 8330594.24569443) / 10952188.221291605 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-zA-Z]+ing", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.928414074074074 < 6.931512754901961.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 181.22176310133668 (T) = (0 -5929325.762889135) / Math.Sqrt((11548836659.116009 / (299)) + (893557425.2618588 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.1906560805490289 = (7326089.219168249 - 5929325.762889135) / 7326089.219168249 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.19042
Baseline 0971e5e22794fc7d7d0744376be2f452e26159ad
Compare 17d6d50eb9d74905f83e709093d1cc32489226ee
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 166.82 ns 125.65 ns 0.75 0.02 False
IsMatch - Duration of single invocation 166.51 ns 124.72 ns 0.75 0.02 False
IsMatch - Duration of single invocation 254.37 ns 192.00 ns 0.75 0.03 False
IsMatch - Duration of single invocation 180.60 ns 139.45 ns 0.77 0.02 False
IsMatch - Duration of single invocation 266.08 ns 212.38 ns 0.80 0.02 False
IsMatch - Duration of single invocation 415.81 ns 250.66 ns 0.60 0.04 False
IsMatch - Duration of single invocation 158.18 ns 121.45 ns 0.77 0.03 False
IsMatch - Duration of single invocation 139.22 ns 118.75 ns 0.85 0.05 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: NonBacktracking)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 125.6538349675918 < 159.69198112608206.
IsChangePoint: Marked as a change because one of 5/26/2022 5:29:00 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 60.820302480787305 (T) = (0 -126.61108013306269) / Math.Sqrt((40.64710415387611 / (299)) + (3.717576615714195 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.18469777803370532 = (155.2934319591452 - 126.61108013306269) / 155.2934319591452 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 9, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 124.71935211610057 < 157.02658025914585.
IsChangePoint: Marked as a change because one of 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 57.44332493131082 (T) = (0 -124.09803820288037) / Math.Sqrt((48.37606710266199 / (299)) + (6.258680746910505 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.2042130683128307 = (155.9438001071176 - 124.09803820288037) / 155.9438001071176 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 4, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 192.00025235247412 < 241.4234265768619.
IsChangePoint: Marked as a change because one of 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 12.582061153450798 (T) = (0 -195.01815350005825) / Math.Sqrt((489.8909470911328 / (299)) + (22.521761547107594 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.08664866280208755 = (213.51931678159914 - 195.01815350005825) / 213.51931678159914 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 10, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 139.44762661386176 < 172.79416750824794.
IsChangePoint: Marked as a change because one of 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 62.186968853921634 (T) = (0 -133.81585637408295) / Math.Sqrt((56.708488809921974 / (299)) + (7.724977380124337 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.22022132284512247 = (171.6074833724965 - 133.81585637408295) / 171.6074833724965 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 5, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 212.38116237620125 < 253.10312068399628.
IsChangePoint: Marked as a change because one of 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 10.568571517495386 (T) = (0 -215.68835458443252) / Math.Sqrt((390.1946151638185 / (299)) + (12.829740757309835 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.05842018176850257 = (229.0707069205717 - 215.68835458443252) / 229.0707069205717 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 3, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 250.6589208069955 < 310.17736555629074.
IsChangePoint: Marked as a change because one of 4/19/2022 10:22:09 AM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 17.776052554715093 (T) = (0 -253.01094578880986) / Math.Sqrt((523.6468565245852 / (299)) + (74.65528951765015 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.11598753373035547 = (286.2074410064208 - 253.01094578880986) / 286.2074410064208 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 8, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 121.44511957548526 < 149.59922708700375.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/26/2022 5:29:00 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 43.41744517506338 (T) = (0 -122.33002618349737) / Math.Sqrt((55.92134472491179 / (299)) + (4.834067482088015 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.16263272804012252 = (146.08885524888154 - 122.33002618349737) / 146.08885524888154 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 7, Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 118.7451921095207 < 132.1608648042323.
IsChangePoint: Marked as a change because one of 4/18/2022 2:31:51 PM, 5/26/2022 5:29:00 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 20.63069315335657 (T) = (0 -117.89903075220381) / Math.Sqrt((38.212781114420395 / (299)) + (4.222244466686052 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.0767977639374319 = (127.70661307650197 - 117.89903075220381) / 127.70661307650197 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 0971e5e22794fc7d7d0744376be2f452e26159ad
Compare 17d6d50eb9d74905f83e709093d1cc32489226ee
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 7.23 ms 3.39 ms 0.47 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'
Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\s?#]+[^\s?#]+(?:?[^\s#])?(?:#[^\s])?", Options: NonBacktracking)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.3888847747747746 < 7.040600057143836.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 14.807188075674786 (T) = (0 -3383597.5212792824) / Math.Sqrt((2121434419971.4568 / (299)) + (700454454.1055566 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.2695600829341669 = (4632273.5686066365 - 3383597.5212792824) / 4632273.5686066365 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 0971e5e22794fc7d7d0744376be2f452e26159ad
Compare 17d6d50eb9d74905f83e709093d1cc32489226ee
Diff Diff

Improvements in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CheckArrayIsInterfaceNo - Duration of single invocation 682.25 μs 361.98 μs 0.53 0.30 False
ObjObjrefValueType - Duration of single invocation 167.11 μs 139.22 μs 0.83 0.47 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf*'
Details

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 361.9835773601398 < 647.7625276633089.
IsChangePoint: Marked as a change because one of 5/9/2022 11:56:12 AM, 5/16/2022 12:27:40 PM, 6/4/2022 9:47:11 PM, 6/7/2022 9:37:06 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 4.966228235074528 (T) = (0 -404591.53054902726) / Math.Sqrt((25861879308.466637 / (299)) + (5616994131.3142805 / (43))) is greater than 1.9669657340893643 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (43) - 2, .975) and 0.15316542320471582 = (477769.2617135946 - 404591.53054902726) / 477769.2617135946 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.ObjObjrefValueType

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 139.22333645336965 < 178.61369460011397.
IsChangePoint: Marked as a change because one of 5/21/2022 9:00:51 AM, 6/8/2022 3:33:18 AM, 6/14/2022 8:14:23 AM falls between 6/5/2022 6:02:14 PM and 6/14/2022 8:14:23 AM.
IsImprovementStdDev: Marked as improvement because 12.243516651057822 (T) = (0 -150517.07693408697) / Math.Sqrt((910754839.7534016 / (299)) + (424479131.8404118 / (38))) is greater than 1.967070609662329 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (38) - 2, .975) and 0.234714667906507 = (196680.99024234098 - 150517.07693408697) / 196680.99024234098 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions