-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.25094 |
| Baseline | 7f7c6a229b691bd060ee1bf4aeb2cea4c5e52168 |
| Compare | 748dfb78ef87b38599b89e85d3201cdb9f9c0b33 |
| Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 3.13 ms | 3.44 ms | 1.10 | 0.51 | False | |||||
| Count - Duration of single invocation | 3.20 ms | 3.68 ms | 1.15 | 0.78 | False | |||||
| Count - Duration of single invocation | 21.83 ms | 35.93 ms | 1.65 | 0.47 | True |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Details
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: NonBacktracking)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.440952895522388 > 3.275119675788426.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -5.716394021394091 (T) = (0 -3606195.9445370575) / Math.Sqrt((51026108488.539795 / (20)) + (62699593488.91942 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.1357606713716952 = (3175137.1881734002 - 3606195.9445370575) / 3175137.1881734002 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\s+\w+", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.678907700000001 > 3.3302789960755814.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -7.639942511245829 (T) = (0 -3962396.029700359) / Math.Sqrt((69876168679.46613 / (20)) + (139473531192.4456 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.2457434821447741 = (3180747.952121229 - 3962396.029700359) / 3180747.952121229 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: NonBacktracking)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.927161428571424 > 22.6417172625.
IsChangePoint: Marked as a change because one of 5/27/2022 6:33:44 PM, 6/2/2022 10:01:42 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 10:01:42 AM.
IsRegressionStdDev: Marked as regression because -22.792117523507144 (T) = (0 -35185649.18909775) / Math.Sqrt((1595063305126.3992 / (20)) + (5529427005405.243 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.6303029124882452 = (21582277.08456691 - 35185649.18909775) / 21582277.08456691 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

