Allocate 2 CPUs per microbenchmark item#8663
Conversation
|
BenchmarksBenchmark execution time: 2026-05-21 12:06:40 Comparing candidate commit 01ce1d3 in PR branch Some scenarios are present only in baseline or only in candidate runs. If you didn't create or remove some scenarios in your branch, this maybe a sign of crashed benchmarks 💥💥💥 Scenarios present only in baseline:
Found 7 performance improvements and 4 performance regressions! Performance is the same for 51 metrics, 10 unstable metrics, 84 known flaky benchmarks, 42 flaky benchmarks without significant changes.
|
bbaf7b0 to
593595e
Compare
|
Running stability experiments. Once we verify flakiness was reduced, we can merge :) |
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8663) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (74ms) : 71, 78
master - mean (75ms) : 71, 79
section Bailout
This PR (8663) - mean (78ms) : 76, 79
master - mean (79ms) : 76, 83
section CallTarget+Inlining+NGEN
This PR (8663) - mean (1,113ms) : 1054, 1171
master - mean (1,109ms) : 1052, 1165
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (115ms) : 109, 121
master - mean (117ms) : 111, 122
section Bailout
This PR (8663) - mean (114ms) : 112, 117
master - mean (115ms) : 112, 118
section CallTarget+Inlining+NGEN
This PR (8663) - mean (790ms) : 763, 818
master - mean (797ms) : 766, 828
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (101ms) : 98, 105
master - mean (102ms) : 98, 106
section Bailout
This PR (8663) - mean (102ms) : 99, 106
master - mean (105ms) : 99, 111
section CallTarget+Inlining+NGEN
This PR (8663) - mean (947ms) : 911, 983
master - mean (947ms) : 904, 989
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (103ms) : 96, 109
master - mean (100ms) : 96, 103
section Bailout
This PR (8663) - mean (101ms) : 96, 106
master - mean (103ms) : 98, 108
section CallTarget+Inlining+NGEN
This PR (8663) - mean (823ms) : 784, 862
master - mean (823ms) : 783, 864
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (200ms) : 194, 206
master - mean (199ms) : 194, 204
section Bailout
This PR (8663) - mean (204ms) : 198, 210
master - mean (203ms) : 197, 208
section CallTarget+Inlining+NGEN
This PR (8663) - mean (1,208ms) : 1156, 1261
master - mean (1,199ms) : 1156, 1241
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (289ms) : 281, 296
master - mean (289ms) : 280, 298
section Bailout
This PR (8663) - mean (290ms) : 280, 299
master - mean (289ms) : 281, 298
section CallTarget+Inlining+NGEN
This PR (8663) - mean (967ms) : 945, 989
master - mean (967ms) : 947, 986
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (280ms) : 271, 288
master - mean (280ms) : 273, 288
section Bailout
This PR (8663) - mean (280ms) : 273, 286
master - mean (281ms) : 273, 288
section CallTarget+Inlining+NGEN
This PR (8663) - mean (1,159ms) : 1121, 1197
master - mean (1,161ms) : 1117, 1204
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8663) - mean (276ms) : 268, 285
master - mean (282ms) : 271, 294
section Bailout
This PR (8663) - mean (278ms) : 269, 287
master - mean (281ms) : 275, 287
section CallTarget+Inlining+NGEN
This PR (8663) - mean (1,039ms) : 992, 1086
master - mean (1,036ms) : 998, 1074
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Stability analysis summary for 10 pipelines triggered on the latest commit 593595eStability Analysis ReportTotal pipelines: 10 Pipelines: 114416104, 114416110, 114416113, 114416119, 114416125, 114416129, 114416133, 114416138, 114416141, 114416144 CPU: Intel(R) Xeon(R) Platinum 8259CL Stability ResultsSignificant impact threshold: 2.0%, t-test alpha: 0.05
Significant Impact FP Rate: 1.1% T-test FP Rate: 17.3% Max Significant Impact Bound: 16.73% p95 Significant Impact Bound: 2.75% |
Stability analysis summary for .NET 6.0 allocated_mem on the latest commit 593595eStability Analysis Report: allocated_mem, net6.0Total pipelines: 10 Pipelines: 114416104, 114416110, 114416113, 114416119, 114416125, 114416129, 114416133, 114416138, 114416141, 114416144 CPU: Intel(R) Xeon(R) Platinum 8259CL Commit: 593595e (branch: augusto/update-microbenchmark-params) Filter: net6.0, allocated_mem Stability ResultsSignificant impact threshold: 2.0%, t-test alpha: 0.05
Summary
|
|
Non-zero FP allocated_mem scenarios (10 pipelines, commit 593595e)
|
|
To use Codex here, create a Codex account and connect to github. |
There was a problem hiding this comment.
Pull request overview
This PR updates the Windows microbenchmarks bp-runner and GitLab pipeline configuration to allocate 2 CPUs per parallel microbenchmark item, reduce variance (notably for .NET 6.0), and rebalance experiment layout to fit a 2×24-core NUMA topology.
Changes:
- Increased
cpus_per_itemfrom 1 → 2 across all microbenchmark steps and expanded CPU affinity masks accordingly. - Renamed and rebalanced trace experiments to
trace-0(NUMA 0) andtrace-1(NUMA 1) with a 9/11 benchmark split. - Merged
otel-instr-apiandotel-apiinto a singleotelexperiment with sequential steps, and added a GitLab variable gate (RUN_OTEL_API_BENCHMARKS_ON_PR) controlling PR-time execution behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
.gitlab/benchmarks/microbenchmarks/bp-runner.windows.yml |
Reworked experiment layout/naming, updated CPU masks, and set cpus_per_item: 2 for trace and OTel steps. |
.gitlab/benchmarks/microbenchmarks.yml |
Added RUN_OTEL_API_BENCHMARKS_ON_PR and updated PR_NUMBER export logic to optionally allow OTel Api benchmarks outside master. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
RUN_OTEL_API_BENCHMARKS_ON_PRto"false"before mergingSummary of changes
cpus_per_itemfrom 1 to 2 for all microbenchmark experimentstracetotrace-0,trace-unstabletotrace-1. Using "unstable" doesn't reflect the split we're currently doing across NUMA nodes.otel-instr-apiandotel-apiinto singleotelexperiment with two sequential stepsReason for change
Allocating 2 CPUs per benchmark item should reduce variance for .NET 6.0 benchmarks.
Implementation details
48-core bare metal instance (2 NUMA nodes × 24 cores, HT disabled):
trace-0(NUMA 0): 9 items × 2 CPUs = 18 cores (0-17)trace-1(NUMA 1): 11 items × 2 CPUs = 22 cores (0-21)otel(NUMA 0): 3 items × 2 CPUs = 6 cores (18-23), InstrumentedApi then Api sequentiallyTest coverage
Other details