Skip to content

[Perf] Regressions from flow graph reordering change #70159

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 6.16 μs 6.74 μs 1.09 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.737065667380443 > 6.456760085057447.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -41.18520937234114 (T) = (0 -6702.308505913001) / Math.Sqrt((1380.3406537166666 / (29)) + (1275.045897158201 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.09234589955871175 = (6135.701620357264 - 6702.308505913001) / 6135.701620357264 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Memory.Constructors<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadOnlySpanFromArray - Duration of single invocation 0.11 ns 1.14 ns 10.40 0.69 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Constructors&lt;Byte&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Memory.Constructors<Byte>.ReadOnlySpanFromArray


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1434632033857692 > 0.1891852574345431.
IsChangePoint: Marked as a change because one of 2/1/2022 4:18:38 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -19.990188823533767 (T) = (0 -0.9257845523005314) / Math.Sqrt((0.002826405902611865 / (29)) + (0.011484976696724828 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -4.006572236919883 = (0.18491385093248702 - 0.9257845523005314) / 0.18491385093248702 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 457.25 ns 498.06 ns 1.09 0.00 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<String>.ImmutableArray(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 498.0561399282693 > 480.0857621193237.
IsChangePoint: Marked as a change because one of 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -596.4214656864929 (T) = (0 -498.1333366046504) / Math.Sqrt((0.11315161591925187 / (29)) + (0.005949614232756872 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.08799380264675961 = (457.8457481952956 - 498.1333366046504) / 457.8457481952956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.Concurrent.IsEmpty<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 11.28 ns 14.68 ns 1.30 0.21 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.IsEmpty&lt;Int32&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.IsEmpty<Int32>.Queue(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.683828767887277 > 11.919290558200084.
IsChangePoint: Marked as a change because one of 2/3/2022 8:57:20 PM, 3/15/2022 12:53:39 AM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -11.07923240330845 (T) = (0 -14.93100806697449) / Math.Sqrt((0.3966814480625782 / (29)) + (0.6743232996875372 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.28349057113305665 = (11.633126415407554 - 14.93100806697449) / 11.633126415407554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline 486b4d1a36aef5bbe6a77bb2c3412772e712961e
Compare afacf6a99e1b98331af958caec638bd93dc46ffe
Diff Diff

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 6.24 μs 7.09 μs 1.14 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.094232775331843 > 6.614450639204545.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/30/2022 4:33:23 PM, 6/2/2022 5:13:37 AM falls between 5/24/2022 3:42:52 PM and 6/2/2022 5:13:37 AM.
IsRegressionStdDev: Marked as regression because -48.80556660224299 (T) = (0 -7130.090187117306) / Math.Sqrt((4198.404554838031 / (29)) + (1019.9229245797691 / (9))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (9) - 2, .025) and -0.12355503582863395 = (6346.008837794749 - 7130.090187117306) / 6346.008837794749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

category:performance
theme:benchmarks

Metadata

Metadata

Assignees

Labels

area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIruntime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions