[Perf] Linux/x64: 38 Regressions on 4/9/2023 11:24:15 AM #15699
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 1411364699b5784040e86f76cb3db8200f6a2c8c |
| Compare | 86b48d7c6f081c12dcc9c048fb53de1b78c9966f |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SerializeToUtf8Bytes - Duration of single invocation | 1.44 ms | 1.73 ms | 1.20 | 0.06 | False | |||||
| SerializeToWriter - Duration of single invocation | 854.43 μs | 1.10 ms | 1.29 | 0.11 | False | |||||
| SerializeToString - Duration of single invocation | 928.36 μs | 1.20 ms | 1.30 | 0.12 | False | |||||
| SerializeObjectProperty - Duration of single invocation | 917.53 μs | 1.20 ms | 1.30 | 0.10 | False | |||||
| SerializeObjectProperty - Duration of single invocation | 1.47 ms | 1.79 ms | 1.21 | 0.09 | False | |||||
| SerializeToString - Duration of single invocation | 1.44 ms | 1.82 ms | 1.27 | 0.05 | False | |||||
| SerializeToWriter - Duration of single invocation | 1.41 ms | 1.71 ms | 1.21 | 0.08 | False | |||||
| SerializeToUtf8Bytes - Duration of single invocation | 889.05 μs | 1.15 ms | 1.29 | 0.08 | False |
_1.png)
_2.png)
_3.png)
_4.png)
_5.png)
_6.png)
_7.png)
_8.png)
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>*'Payloads
Histogram
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToUtf8Bytes(Mode: Reflection)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7334409737103176 > 1.5106449220028406.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -19.30336757285662 (T) = (0 -1760532.8299248866) / Math.Sqrt((764249190.579451 / (43)) + (1911901496.1484249 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.23002579589065036 = (1431297.4864483236 - 1760532.8299248866) / 1431297.4864483236 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToWriter(Mode: SourceGen)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1024863930555557 > 901.3729586266448.
IsChangePoint: Marked as a change because one of 3/10/2023 11:41:11 AM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -24.259682282447997 (T) = (0 -1123007.4771714562) / Math.Sqrt((418743005.78988874 / (43)) + (628950525.1830385 / (6))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (6) - 2, .025) and -0.3007620807913441 = (863345.7984016974 - 1123007.4771714562) / 863345.7984016974 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToString(Mode: SourceGen)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2038464321428572 > 965.5757336914063.
IsChangePoint: Marked as a change because one of 2/13/2023 6:54:14 PM, 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -45.77574660563786 (T) = (0 -1182657.0629807692) / Math.Sqrt((439787795.285117 / (42)) + (154765105.23673412 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.2835820233641032 = (921372.4105306315 - 1182657.0629807692) / 921372.4105306315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeObjectProperty(Mode: SourceGen)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1962798625 > 960.3113865820311.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 3/10/2023 4:14:40 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -33.13704484439174 (T) = (0 -1187492.0079490452) / Math.Sqrt((307815688.6058221 / (43)) + (395338337.9171594 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.2863453064862063 = (923151.8177594243 - 1187492.0079490452) / 923151.8177594243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeObjectProperty(Mode: Reflection)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7883439810185184 > 1.554131670706731.
IsChangePoint: Marked as a change because one of 3/10/2023 4:14:40 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -16.830382284852913 (T) = (0 -1763616.5070400883) / Math.Sqrt((381766522.5330156 / (42)) + (2052467067.4214847 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.1989310502871279 = (1470990.7684997614 - 1763616.5070400883) / 1470990.7684997614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToString(Mode: Reflection)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.822725446986607 > 1.548973780778846.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -15.867382020776489 (T) = (0 -1796059.0779696947) / Math.Sqrt((622096772.6900558 / (42)) + (2879577254.4005203 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.2230645613881314 = (1468490.817795617 - 1796059.0779696947) / 1468490.817795617 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToWriter(Mode: Reflection)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7108328703703704 > 1.4829773012784087.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -31.906590112677556 (T) = (0 -1684949.1084665533) / Math.Sqrt((1117112246.5810547 / (42)) + (359277974.52884793 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.20070805174327352 = (1403296.2517576392 - 1684949.1084665533) / 1403296.2517576392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Json.Serialization.Tests.WriteJson<MyEventsListerViewModel>.SerializeToUtf8Bytes(Mode: SourceGen)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.148270192628205 > 935.54041484375.
IsChangePoint: Marked as a change because one of 2/16/2023 3:28:45 AM, 3/10/2023 4:14:40 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -55.43342176315234 (T) = (0 -1144866.63070382) / Math.Sqrt((498266766.82401186 / (43)) + (62833427.953192644 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.281341170824307 = (893490.8647064772 - 1144866.63070382) / 893490.8647064772 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 1411364699b5784040e86f76cb3db8200f6a2c8c |
| Compare | 86b48d7c6f081c12dcc9c048fb53de1b78c9966f |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.Tests.Perf_DateTime
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToString - Duration of single invocation | 360.58 ns | 599.99 ns | 1.66 | 0.16 | True | |||||
| ToString - Duration of single invocation | 130.53 ns | 185.18 ns | 1.42 | 0.25 | False | |||||
| ToString - Duration of single invocation | 370.78 ns | 610.66 ns | 1.65 | 0.13 | True | |||||
| ToString - Duration of single invocation | 358.66 ns | 569.28 ns | 1.59 | 0.15 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_DateTime*'Payloads
Histogram
System.Tests.Perf_DateTime.ToString(format: null)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 599.9888331743136 > 377.39721273111485.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -24.322655055724343 (T) = (0 -601.8257350283385) / Math.Sqrt((151.86787165916476 / (43)) + (668.1806228612865 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6725035176669795 = (359.8352581451318 - 601.8257350283385) / 359.8352581451318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_DateTime.ToString(format: "o")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 185.17668575963293 > 136.5719478509588.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -26.270893599250268 (T) = (0 -178.7206589105019) / Math.Sqrt((26.735841185961704 / (43)) + (21.936082818519164 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.3983300571957398 = (127.81006743780834 - 178.7206589105019) / 127.81006743780834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Tests.Perf_DateTime.ToString(format: "s")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 610.6603128857563 > 389.66698580220293.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -18.132431961749482 (T) = (0 -561.851156518624) / Math.Sqrt((150.855263664341 / (42)) + (702.6031308587686 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.49044704835091574 = (376.96821040390284 - 561.851156518624) / 376.96821040390284 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Tests.Perf_DateTime.ToString(format: "G")
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 569.2818855012036 > 363.7887397283544.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -24.932149334810706 (T) = (0 -601.256363345744) / Math.Sqrt((123.23113972028878 / (43)) + (659.2188534817575 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.6905721339501865 = (355.65259314954517 - 601.256363345744) / 355.65259314954517 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 1411364699b5784040e86f76cb3db8200f6a2c8c |
| Compare | 86b48d7c6f081c12dcc9c048fb53de1b78c9966f |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.Text.Json.Tests.Perf_DateTimes
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| WriteDateTimes - Duration of single invocation | 8.46 ms | 15.49 ms | 1.83 | 0.09 | True | |||||
| WriteDateTimes - Duration of single invocation | 7.36 ms | 15.01 ms | 2.04 | 0.11 | True | |||||
| WriteDateTimes - Duration of single invocation | 8.37 ms | 15.63 ms | 1.87 | 0.06 | True | |||||
| WriteDateTimes - Duration of single invocation | 7.27 ms | 15.21 ms | 2.09 | 0.11 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_DateTimes*'Payloads
Histogram
System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: True, SkipValidation: False)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.494484431111115 > 8.8966080478125.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -18.580510834604535 (T) = (0 -16015703.269991279) / Math.Sqrt((101391066844.01012 / (42)) + (1096545253234.7323 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -0.8611475292152594 = (8605284.115625264 - 16015703.269991279) / 8605284.115625264 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: False, SkipValidation: True)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.006017768888888 > 7.5158398423076935.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -66.5256048402972 (T) = (0 -15111659.094716337) / Math.Sqrt((61407129919.42048 / (42)) + (91044348727.34425 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -1.1255148638444077 = (7109646.397571625 - 15111659.094716337) / 7109646.397571625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: True, SkipValidation: True)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.629066666666668 > 8.826791964843752.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -25.617238810746446 (T) = (0 -16247207.360770974) / Math.Sqrt((14463057281.009233 / (43)) + (652324797797.6385 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.9312130303363065 = (8412954.503492368 - 16247207.360770974) / 8412954.503492368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
System.Text.Json.Tests.Perf_DateTimes.WriteDateTimes(Formatted: False, SkipValidation: False)
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.213145820833333 > 7.70143438125.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -34.920054594525325 (T) = (0 -15340341.68857448) / Math.Sqrt((37287439335.00616 / (42)) + (371548600378.466 / (7))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (7) - 2, .025) and -1.1222916700892094 = (7228196.719977541 - 15340341.68857448) / 7228196.719977541 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 1411364699b5784040e86f76cb3db8200f6a2c8c |
| Compare | 86b48d7c6f081c12dcc9c048fb53de1b78c9966f |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.Text.Tests.Perf_StringBuilder
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Append_ValueTypes - Duration of single invocation | 6.30 μs | 6.82 μs | 1.08 | 0.10 | False | |||||
| Append_ValueTypes_Interpolated - Duration of single invocation | 8.45 μs | 9.78 μs | 1.16 | 0.12 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'Payloads
Histogram
System.Text.Tests.Perf_StringBuilder.Append_ValueTypes
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.817022412823868 > 6.409512357608278.
IsChangePoint: Marked as a change because one of 2/12/2023 3:00:19 AM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -10.694344103194501 (T) = (0 -6969.682413049682) / Math.Sqrt((19648.708733229687 / (43)) + (37512.4582635502 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.1325244710812947 = (6154.111978167918 - 6969.682413049682) / 6154.111978167918 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
System.Text.Tests.Perf_StringBuilder.Append_ValueTypes_Interpolated
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.778222395506385 > 8.870198687098206.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 3/10/2023 8:06:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -38.543168387420515 (T) = (0 -9749.466830629566) / Math.Sqrt((28780.18582165279 / (43)) + (3108.6513994967463 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.15196008518214554 = (8463.372087313252 - 9749.466830629566) / 8463.372087313252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 18.04 |
| Queue | TigerUbuntu |
| Baseline | 1411364699b5784040e86f76cb3db8200f6a2c8c |
| Compare | 86b48d7c6f081c12dcc9c048fb53de1b78c9966f |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SystemTextJson_SourceGen_ - Duration of single invocation | 1.30 ms | 1.60 ms | 1.23 | 0.07 | False | |||||
| SystemTextJson_Reflection_ - Duration of single invocation | 1.38 ms | 1.72 ms | 1.25 | 0.06 | False |
_1.png)
_2.png)
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>*'Payloads
Histogram
MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.SystemTextJson_SourceGen_
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6022623655913977 > 1.3721657819213418.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -16.95924254476948 (T) = (0 -1600796.5826501288) / Math.Sqrt((1284416405.5164037 / (41)) + (1656159385.6116073 / (7))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (7) - 2, .025) and -0.20978955255254847 = (1323202.5183823006 - 1600796.5826501288) / 1323202.5183823006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
MicroBenchmarks.Serializers.Json_ToStream<MyEventsListerViewModel>.SystemTextJson_Reflection_
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7246685162037043 > 1.4672714498997175.
IsChangePoint: Marked as a change because one of 3/10/2023 8:06:53 PM, 3/22/2023 10:50:22 PM, 3/23/2023 11:51:53 PM, 4/9/2023 4:09:45 AM, 4/11/2023 1:13:25 AM falls between 3/30/2023 4:24:24 AM and 4/11/2023 1:13:25 AM.
IsRegressionStdDev: Marked as regression because -22.448551132325488 (T) = (0 -1691423.309335212) / Math.Sqrt((531703895.71895164 / (43)) + (1054205304.6076332 / (7))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (7) - 2, .025) and -0.20398913058443383 = (1404849.3182942362 - 1691423.309335212) / 1404849.3182942362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository