Update microbenchmark design to reduce flakiness#8300
Conversation
Phase 1 of modernizing dd-trace-dotnet microbenchmarks: - Add bp-runner.windows.yml with single dry-run benchmark - Add instance.yml provision file for Windows instances - Add run-benchmarks-test job using bp-infra with --env-regex - Uses test AMI with bp-runner from dev branch The new job runs automatically on augusto/fix-microbenchmarks branch and manually on other branches. It validates the new infrastructure by running a single SpanBenchmark.StartFinishSpan with --job dry.
Replace the shell-script-based benchmark execution with bp-infra and bp-runner. This uses the instance.yml provision file and bp-runner.windows.yml config directly from dd-trace-dotnet repo.
bp-runner passes $CodeSrc and $BaselineOrCandidate as function params, not environment variables.
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8300) and master.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 189.63 ± (189.56 - 190.28) ms | 211.55 ± (211.17 - 212.10) ms | +11.6% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 193.19 ± (193.12 - 193.51) ms | 216.95 ± (216.62 - 217.64) ms | +12.3% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1136.69 ± (1141.07 - 1150.90) ms | 1236.93 ± (1234.11 - 1241.31) ms | +8.8% | ❌⬆️ |
Full Metrics Comparison
FakeDbCommand
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 72.29 ± (72.25 - 72.61) ms | 76.57 ± (76.52 - 76.96) ms | +5.9% | ✅⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 76.69 ± (76.48 - 76.84) ms | 80.52 ± (80.29 - 80.69) ms | +5.0% | ✅⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1062.49 ± (1063.55 - 1069.14) ms | 1110.69 ± (1109.77 - 1115.09) ms | +4.5% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 22.40 ± (22.36 - 22.44) ms | 23.11 ± (23.05 - 23.16) ms | +3.2% | ✅⬆️ |
| process.time_to_main_ms | 83.81 ± (83.62 - 84.00) ms | 88.90 ± (88.71 - 89.09) ms | +6.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.89 ± (10.89 - 10.90) MB | 10.92 ± (10.92 - 10.93) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 12 ± (12 - 12) | 12 ± (12 - 12) | +0.0% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 22.31 ± (22.27 - 22.35) ms | 22.90 ± (22.85 - 22.96) ms | +2.7% | ✅⬆️ |
| process.time_to_main_ms | 84.90 ± (84.70 - 85.10) ms | 89.80 ± (89.59 - 90.01) ms | +5.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.87 ± (10.86 - 10.87) MB | 10.95 ± (10.95 - 10.96) MB | +0.8% | ✅⬆️ |
| runtime.dotnet.threads.count | 13 ± (13 - 13) | 13 ± (13 - 13) | +0.0% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 224.53 ± (223.23 - 225.84) ms | 232.73 ± (231.53 - 233.92) ms | +3.7% | ✅⬆️ |
| process.time_to_main_ms | 515.80 ± (514.61 - 516.98) ms | 544.20 ± (542.79 - 545.60) ms | +5.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 47.88 ± (47.84 - 47.92) MB | 47.83 ± (47.80 - 47.86) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +1.3% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 20.91 ± (20.87 - 20.94) ms | 21.86 ± (21.81 - 21.91) ms | +4.6% | ✅⬆️ |
| process.time_to_main_ms | 71.75 ± (71.60 - 71.89) ms | 77.23 ± (77.05 - 77.42) ms | +7.6% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.61 ± (10.60 - 10.61) MB | 10.64 ± (10.64 - 10.65) MB | +0.4% | ✅⬆️ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 20.95 ± (20.91 - 20.98) ms | 21.70 ± (21.64 - 21.76) ms | +3.6% | ✅⬆️ |
| process.time_to_main_ms | 72.97 ± (72.79 - 73.15) ms | 78.21 ± (78.03 - 78.38) ms | +7.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.70 ± (10.70 - 10.71) MB | 10.76 ± (10.76 - 10.77) MB | +0.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 389.19 ± (387.12 - 391.26) ms | 385.79 ± (383.66 - 387.91) ms | -0.9% | ✅ |
| process.time_to_main_ms | 515.46 ± (514.56 - 516.36) ms | 545.94 ± (544.71 - 547.17) ms | +5.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 49.84 ± (49.81 - 49.87) MB | 49.86 ± (49.83 - 49.89) MB | +0.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.1% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 19.26 ± (19.23 - 19.29) ms | 20.24 ± (20.19 - 20.28) ms | +5.1% | ✅⬆️ |
| process.time_to_main_ms | 71.85 ± (71.67 - 72.03) ms | 77.00 ± (76.83 - 77.18) ms | +7.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.68 ± (7.68 - 7.69) MB | 7.68 ± (7.68 - 7.69) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 19.12 ± (19.08 - 19.15) ms | 20.17 ± (20.12 - 20.23) ms | +5.5% | ✅⬆️ |
| process.time_to_main_ms | 72.23 ± (72.10 - 72.37) ms | 77.63 ± (77.45 - 77.81) ms | +7.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.73 ± (7.72 - 7.73) MB | 7.74 ± (7.74 - 7.75) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 306.86 ± (304.63 - 309.09) ms | 306.54 ± (304.14 - 308.94) ms | -0.1% | ✅ |
| process.time_to_main_ms | 477.29 ± (476.54 - 478.04) ms | 505.37 ± (504.62 - 506.12) ms | +5.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 37.03 ± (37.00 - 37.05) MB | 37.08 ± (37.05 - 37.12) MB | +0.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 27 ± (27 - 27) | 27 ± (27 - 27) | -1.0% | ✅ |
HttpMessageHandler
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 189.63 ± (189.56 - 190.28) ms | 211.55 ± (211.17 - 212.10) ms | +11.6% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 193.19 ± (193.12 - 193.51) ms | 216.95 ± (216.62 - 217.64) ms | +12.3% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1136.69 ± (1141.07 - 1150.90) ms | 1236.93 ± (1234.11 - 1241.31) ms | +8.8% | ❌⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 184.99 ± (184.71 - 185.26) ms | 207.92 ± (207.44 - 208.40) ms | +12.4% | ✅⬆️ |
| process.time_to_main_ms | 79.28 ± (79.10 - 79.46) ms | 90.60 ± (90.34 - 90.85) ms | +14.3% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.07 ± (16.04 - 16.09) MB | 15.97 ± (15.95 - 15.98) MB | -0.6% | ✅ |
| runtime.dotnet.threads.count | 20 ± (19 - 20) | 20 ± (20 - 20) | +1.8% | ✅⬆️ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 184.18 ± (183.98 - 184.39) ms | 209.90 ± (209.39 - 210.42) ms | +14.0% | ✅⬆️ |
| process.time_to_main_ms | 80.62 ± (80.54 - 80.70) ms | 93.03 ± (92.76 - 93.29) ms | +15.4% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.99 ± (15.86 - 16.12) MB | 15.94 ± (15.92 - 15.96) MB | -0.3% | ✅ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 21 ± (21 - 21) | +5.0% | ✅⬆️ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 388.46 ± (387.03 - 389.89) ms | 419.25 ± (417.54 - 420.95) ms | +7.9% | ✅⬆️ |
| process.time_to_main_ms | 501.31 ± (500.27 - 502.35) ms | 550.14 ± (548.73 - 551.55) ms | +9.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 57.82 ± (57.61 - 58.03) MB | 58.77 ± (58.74 - 58.81) MB | +1.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 30 ± (30 - 30) | +1.4% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 189.48 ± (189.23 - 189.74) ms | 215.71 ± (215.20 - 216.22) ms | +13.8% | ✅⬆️ |
| process.time_to_main_ms | 69.07 ± (68.96 - 69.18) ms | 80.50 ± (80.22 - 80.77) ms | +16.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.84 ± (15.67 - 16.01) MB | 16.19 ± (16.17 - 16.21) MB | +2.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 18 ± (18 - 18) | 20 ± (19 - 20) | +8.5% | ✅⬆️ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 188.90 ± (188.71 - 189.09) ms | 215.84 ± (215.25 - 216.43) ms | +14.3% | ✅⬆️ |
| process.time_to_main_ms | 70.06 ± (70.00 - 70.12) ms | 81.73 ± (81.46 - 82.00) ms | +16.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.71 ± (15.54 - 15.88) MB | 16.23 ± (16.21 - 16.25) MB | +3.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 19 ± (18 - 19) | 21 ± (21 - 21) | +11.2% | ✅⬆️ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 601.33 ± (598.64 - 604.02) ms | 600.70 ± (598.24 - 603.17) ms | -0.1% | ✅ |
| process.time_to_main_ms | 502.36 ± (501.71 - 503.00) ms | 557.24 ± (556.10 - 558.38) ms | +10.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 61.55 ± (61.45 - 61.64) MB | 61.24 ± (61.14 - 61.34) MB | -0.5% | ✅ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 31 ± (31 - 31) | +1.8% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 187.22 ± (186.97 - 187.48) ms | 213.71 ± (213.14 - 214.29) ms | +14.1% | ✅⬆️ |
| process.time_to_main_ms | 68.72 ± (68.55 - 68.89) ms | 79.41 ± (79.13 - 79.68) ms | +15.6% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.85 ± (11.82 - 11.88) MB | 11.60 ± (11.58 - 11.61) MB | -2.2% | ✅ |
| runtime.dotnet.threads.count | 18 ± (18 - 18) | 19 ± (19 - 19) | +4.8% | ✅⬆️ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 186.28 ± (186.10 - 186.47) ms | 212.61 ± (212.13 - 213.08) ms | +14.1% | ✅⬆️ |
| process.time_to_main_ms | 69.44 ± (69.38 - 69.51) ms | 80.62 ± (80.34 - 80.90) ms | +16.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.69 ± (11.58 - 11.79) MB | 11.68 ± (11.65 - 11.70) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 18 ± (18 - 19) | 20 ± (20 - 20) | +9.2% | ✅⬆️ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 521.37 ± (518.81 - 523.93) ms | 548.64 ± (541.53 - 555.74) ms | +5.2% | ✅⬆️ |
| process.time_to_main_ms | 461.52 ± (460.83 - 462.20) ms | 512.63 ± (511.56 - 513.71) ms | +11.1% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 50.69 ± (50.67 - 50.72) MB | 51.01 ± (50.92 - 51.10) MB | +0.6% | ✅⬆️ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 30 ± (30 - 30) | +1.3% | ✅⬆️ |
Comparison explanation
Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (77ms) : 74, 80
master - mean (72ms) : 70, 75
section Bailout
This PR (8300) - mean (80ms) : 78, 83
master - mean (77ms) : 75, 78
section CallTarget+Inlining+NGEN
This PR (8300) - mean (1,112ms) : 1074, 1151
master - mean (1,066ms) : 1027, 1106
FakeDbCommand (.NET Core 3.1)
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (119ms) : 116, 122
master - mean (113ms) : 110, 116
section Bailout
This PR (8300) - mean (120ms) : 117, 123
master - mean (114ms) : 111, 117
section CallTarget+Inlining+NGEN
This PR (8300) - mean (815ms) : 794, 837
master - mean (778ms) : 755, 801
FakeDbCommand (.NET 6)
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (106ms) : 102, 110
master - mean (99ms) : 96, 102
section Bailout
This PR (8300) - mean (107ms) : crit, 103, 110
master - mean (100ms) : 97, 102
section CallTarget+Inlining+NGEN
This PR (8300) - mean (960ms) : 927, 993
master - mean (933ms) : 900, 966
FakeDbCommand (.NET 8)
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (105ms) : 102, 109
master - mean (99ms) : 95, 102
section Bailout
This PR (8300) - mean (106ms) : crit, 103, 109
master - mean (99ms) : 97, 101
section CallTarget+Inlining+NGEN
This PR (8300) - mean (843ms) : 806, 879
master - mean (814ms) : 778, 850
HttpMessageHandler (.NET Framework 4.8)
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (212ms) : 207, 217
master - mean (190ms) : 186, 193
section Bailout
This PR (8300) - mean (217ms) : crit, 212, 223
master - mean (193ms) : 191, 195
section CallTarget+Inlining+NGEN
This PR (8300) - mean (1,238ms) : crit, 1185, 1290
master - mean (1,146ms) : 1072, 1220
HttpMessageHandler (.NET Core 3.1)
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (308ms) : 298, 318
master - mean (272ms) : 268, 276
section Bailout
This PR (8300) - mean (313ms) : crit, 303, 323
master - mean (272ms) : 270, 275
section CallTarget+Inlining+NGEN
This PR (8300) - mean (1,012ms) : crit, 981, 1042
master - mean (919ms) : 892, 946
HttpMessageHandler (.NET 6)
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (305ms) : 296, 315
master - mean (266ms) : 262, 271
section Bailout
This PR (8300) - mean (307ms) : crit, 300, 313
master - mean (267ms) : 264, 269
section CallTarget+Inlining+NGEN
This PR (8300) - mean (1,191ms) : 1144, 1238
master - mean (1,133ms) : 1093, 1173
HttpMessageHandler (.NET 8)
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8300) - mean (304ms) : 297, 311
master - mean (265ms) : 262, 269
section Bailout
This PR (8300) - mean (304ms) : crit, 298, 310
master - mean (265ms) : 262, 268
section CallTarget+Inlining+NGEN
This PR (8300) - mean (1,101ms) : crit, 989, 1212
master - mean (1,016ms) : 966, 1066
The .NET SDK is installed at C:\dotnet on the AMI, but it's not in PATH by default for the bp-runner execution context.
The test AMI has .NET 3.1, 6.0, and 10.0 installed but not 8.0. Change the benchmark target framework to net6.0.
Add Phase 2.5: S3 artifact persistence for microbenchmarks - Add upload_results provision step in instance.yml using Write-S3Object - Add S3 fetch step in microbenchmarks.yml after bp-infra launch - Update _latest on master branch for baseline comparison
Output format changes from: candidate-Benchmarks.Trace.SpanBenchmark-report-full-compressed-0.json To: candidate.Trace.SpanBenchmark.json Strips 'Benchmarks.' prefix and '-report(-full)?(-compressed)?' suffix.
When running parallel benchmarks, multiple dotnet run commands were trying to rebuild simultaneously, causing build failures. Adding --no-build uses the pre-built output from how_to_fetch_release.
The --no-build flag was failing because only the tracer was being built, not the benchmarks project itself. Now we build both in how_to_fetch_release so parallel runs can use --no-build successfully.
- Add GetBenchmarkProjectsWithSettings() to share project list between targets - Add independent BuildBenchmarks target to pre-build benchmark projects - Add NoBenchmarkBuild parameter to skip building in RunBenchmarks - Add BenchmarkArtifactsDirectory parameter for configurable output - Make EnsureCleanDirectory conditional (only in sequential mode) - Update bp-runner.windows.yml to use Nuke build targets: - how_to_fetch_release: build.cmd BuildTracerHome BuildBenchmarks - how_to_run_benchmarks: build.cmd RunBenchmarks --no-benchmark-build
Parallel bp-runner invocations race to compile/load Nuke's _build.dll, causing file lock errors. Add retry with 3 attempts and 5s delay.
Run benchmark executables directly instead of via Nuke to avoid _build.dll lock contention during parallel bp-runner invocations. Add analysis scripts (fetch-results, analyze-results, post-pr-comment, upload-to-bp-ui) copied from benchmarking-platform and simplified.
- Fix BenchmarkDotNet runtime arguments (array instead of string) - Use correct runtime names (netcoreapp31, net60 instead of dotted) - Add fetch/analyze/upload steps to run-benchmarks job
Run `dotnet build/_build/bin/Debug/_build.dll RunBenchmarks` instead of `build.cmd` to avoid _build.dll file lock when parallel bp-runner invocations try to recompile Nuke's build project simultaneously.
…tion - bp-runner now calls run-benchmarks.ps1 instead of Nuke DLL - Removes runtime (-r) argument, uses BenchmarkDotNet defaults - Each parallel run gets unique artifacts directory via PARALLEL_INDEX - Bypasses Nuke entirely to avoid .nuke/build.schema.json file locks
Without -r flag, BenchmarkDotNet fails to generate benchmark harnesses properly. Add back the runtime targets (net472, netcoreapp3.1, net6.0) to match Build.cs RunBenchmarks behavior.
BenchmarksBenchmark execution time: 2026-04-07 13:35:03 Comparing candidate commit 2ab96a5 in PR branch Found 32 performance improvements and 38 performance regressions! Performance is the same for 214 metrics, 4 unstable metrics.
|
Use AppContext.BaseDirectory instead of Directory.GetCurrentDirectory() for locating benchmark files. On .NET Framework 4.7.2, when BenchmarkDotNet spawns a separate process, the working directory may differ from the exe location, causing WAF rules to not be found. Also removes BenchmarkCategory from AgentWriterBenchmark to exclude WriteEnrichedTraces from PR/master benchmark runs.
| "-m", | ||
| "-f", $Filter, | ||
| "--allCategories", $Category, | ||
| "--iterationTime", "200", # Increased to 500 ms for flaky benchmarks |
There was a problem hiding this comment.
This line seems kind of contradictory
|
There are several files that should not be commited:
|
|
|
||
| Write-Output "=== Benchmarks completed for filter: $env:PARALLEL_ITEM ===" | ||
|
|
||
| .how_to_run_benchmarks_batch1: &how_to_run_benchmarks_batch1 | |
There was a problem hiding this comment.
This one is nearly identical to how_to_run_benchmarks_batch0. They could probably be refactored, something like:
experiments:
- name: batch-0
steps:
- name: batch-0
environment:
BATCH_PREFIX: "b0"
how_to_run_benchmarks: *how_to_run_benchmarks
- name: batch-1
steps:
- name: batch-1
environment:
BATCH_PREFIX: "b1"
how_to_run_benchmarks: *how_to_run_benchmarks
|
Here is a comparison of memory allocation measurements using these benchmarks for 10c5701 and 10c5701^: That PR added 8 bytes. And the median byte delta, when positive, is +8, as expected.
Click to show table.
|
andrewlock
left a comment
There was a problem hiding this comment.
I mostly looked at the Nuke changes, I take your word for it on the iteration changes etc, and 🙈 on the gitlab yaml and scripts 😅
FWIW, ideally we wouldn't have any of those scripts. That's why we have Nuke, to avoid having bash in YAML 😉 But we can always convert them later anyway
|
@andrewlock, just to be extra extra extra sure, I ran stability tests again after these last changes (ran 10 separate The false positive rate was 2.1%, which is consistent with our previous results. Addressing some remaining comments:
Absolutely. If we could implement everything in Nuke and BenchmarkDotNet alone, I would. But the design fixes here require running benchmarks in parallel, otherwise CI times get even longer. Parallel execution requires bp-runner, which means using scripts. These scripts originated from the benchmarking-platform repo. @NachoEchevarria and I decided to move them here so there's no switching between repos to understand or debug the pipeline. We can move them back if you both prefer, but having everything in one place has been much easier to work with.
There's no need to take my word for it! We've ran extensive tests to validate everything is working as expected 😅 Once again, thx for the review! |
commit d5d94dd7eb8e5ec080ffbe3b5b7d89aafe95fef9 Author: Augusto de Oliveira <[email protected]> Date: Wed Apr 8 06:58:34 2026 +0200 Update microbenchmark design to reduce flakiness (#8300) ## Summary of changes - Update the benchmark design to reduce flakiness. - See [other details for flakiness reduction results](#other-details). - Move benchmarking from [benchmarking-platform @ dd-trace-dotnet/micro](https://github.com/DataDog/benchmarking-platform/tree/dd-trace-dotnet/micro) to this repo. - This is why we have +1k lines (sorry!). - Include `BytesAllocatedPerOperation` as `allocated_mem` into our benchmarking results. ## Reason for change [APMSP-2291 Identify and fix flaky dd-trace-dotnet microbenchmark scenarios](https://datadoghq.atlassian.net/browse/APMSP-2291) ## Implementation details ### Benchmark design changes (what reduced flakiness) 1. Run every benchmark 10 times (`launchCount`). - This is important to mitigate inter-run variance, which can be caused by things such as varying memory placement across runs. 2. Manually set warmup iterations to 10 (`warmupCount`). - On average, the number of automatically determined warmup iterations is ~15 for the current benchmarking setup. - I got this down to 10 because: - I needed to make benchmarks faster. - On average, every iteration runs the benchmark method for hundreds of thousands of times, so I hypothesized that 5 less warmup iterations would not affect results significantly. - We were already doing 10 runs. I hypothesized that stability would come from different process samples, not from a deeper warmup within a single process. 3. Manually set actual iterations to 10 (`iterationCount`). - On average, the number of automatically determined warmup iterations is ~35 for the current benchmarking setup. Some (`CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces`, `StringConcatAspectBenchmark`, `StringConcatBenchmark`) go up to 120. - I got this down to 10 because of the same reasons described on change 2. 4. Kept a "base" `iterationTime` of 200 ms, but set `iterationTime` on flaky benchmarks to 500 ms. - Increasing the `iterationTime` for flaky benchmarks helps immensely. We're able to capture much more variation into our samples once we fit more operations into the iteration. I tried many configurations, this turned out to be the minimum we need. ### Benchmark execution changes (what allowed me to implement the benchmark design changes) 1. Moved code from [benchmarking-platform @ dd-trace-dotnet/micro](https://github.com/DataDog/benchmarking-platform/tree/dd-trace-dotnet/micro) to here. - This is just easier to understand, and doesn't require changing two repos to update benchmarks. - ⚠️ We just have to move the `ami.yaml` to build the AMIs with which we run microbenchmarks. But there's some preliminary work to avoid dd-octo-sts failures on this repo. Captured on [APMSP-2736](https://datadoghq.atlassian.net/browse/APMSP-2736). 2. Used `bp-runner` to run the benchmarks, since it allows us to run microbenchmarks in parallel on isolated CPU sets. - This was crucial to allow us to run 10 launches. The current benchmarking job takes 50 minutes (20 min build + 30 min benchmarks). By running things 10 times sequentially it would take us 320 minutes. 3. Split benchmarks into two batches: flaky ones and non-flaky ones. I moved the non-flaky ones to CPUs in NUMA node 1 because Windows often runs system processes on the first NUMA node, which can affect results. ### Benchmark class/Build.cs changes 1. `Build.cs`: - Added a `BuildBenchmarks` target to build all benchmark executables in one go (needed for `bp-runner` parallel execution). 2. **`AppSecWafBenchmark.cs` / `AppSecBenchmarkUtils.cs`**: Fixed path resolution to make .NET Framework 4.7.2 benchmarks run in parallel. - Changed `Directory.GetCurrentDirectory()` and `Environment.CurrentDirectory` to `AppContext.BaseDirectory`. - BenchmarkDotNet spawns a separate process for each benchmark run, and on .NET Framework the working directory doesn't match the executable directory. This caused WAF initialization to fail with "No valid rules found" because the rules file and native libraries couldn't be located. - Added `[IterationTime(500)]` attribute to benchmarks identified as flaky (e.g., `AspNetCoreBenchmark`). ## Test coverage - [x] CI job succeeds: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1524793394 - [x] Results are uploaded to S3: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1524793394#L3875 - [x] Results are uploaded to the BP UI: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1524793394#L4200 and https://benchmarking.us1.prod.dog/benchmarks?projectId=9&ciJobDateStart=1771416292448&ciJobDateEnd=1774008292448&gitBranch=augusto%2Ffix-microbenchmarks&benchmarkGroupPipelineId=103686524&benchmarkGroupSha=ce6c9b12 ## Other details <!-- Fixes #{issue} --> ### How did we measure flakiness? 1. Run the `run-benchmarks` job for the same commit 10 times. 2. Cross-validate results. - For every benchmark and metric (`execution_time`, `throughtput`, and `allocated_mem`✨), we compute the **_false positive rate_** and the **_max confidence interval difference_**. - The **_false positive rate_** is how often the benchmark system incorrectly flags a "regression" when comparing identical code. - The **_max confidence interval difference_** is the largest spurious difference observed across all 45 comparisons of `run-benchmarks` results when comparing identical code. ### How much did flakiness drop? - The false positive rate dropped from 5.7% to 3.8%. - The maximum CI difference dropped from 56% to 16.66%. See the [per-benchmark/metric breakdown](https://docs.google.com/spreadsheets/d/103hjAaXOBSbwSHfD-nhBoM23j8dLdeenRMGAS2TeBGU/edit?usp=sharing) This graph shows intuitively the distribution of FP rates and max CI differences for all comparisons. **The more weight the distribution has closer to 0, the better.** Memory allocation results are already pretty stable, so it's hard to see a clear win there. <img width="1213" height="601" alt="image" src="https://github.com/user-attachments/assets/b95e8d0d-1092-48e0-8ff7-ceda61174a63" /> ### How were flaky benchmarks determined (necessary to choose a higher `iterationTime` for them)? 1. Started with existing ignored list @NachoEchevarria and I created. 2. Ran stability experiments and refined based on data (again, [per-benchmark/metric breakdown](https://docs.google.com/spreadsheets/d/103hjAaXOBSbwSHfD-nhBoM23j8dLdeenRMGAS2TeBGU/edit?usp=sharing)) - AppSec benchmarks (`AppSecWafBenchmark`, `AppSecBodyBenchmark`): Moved to stable batch. With the new design, most FP rates are under 5%, only one outlier at 20% for the `netcoreapp3.1` runtime. - `AspNetCoreBenchmark`: Consistently over 20% FP even with optimized design. - `AgentWriterBenchmark`: Added to flaky batch. Most methods are stable, but `WriteAndFlushEnrichedTraces netcoreapp3.1` is VERY problematic. <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> [APMSP-2736]: https://datadoghq.atlassian.net/browse/APMSP-2736?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ commit 275c3a4b037764154277a7cea2a275ab788751d7 Author: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com> Date: Tue Apr 7 13:47:42 2026 -0400 [Version Bump] 3.42.0 (#8419) The following files were found to be modified (as expected) - [x] docs/CHANGELOG.md - [x] .azure-pipelines/ultimate-pipeline.yml - [x] profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/CMakeLists.txt - [x] profiler/src/ProfilerEngine/Datadog.Profiler.Native.Windows/Resource.rc - [x] profiler/src/ProfilerEngine/Datadog.Profiler.Native/dd_profiler_version.h - [x] profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/CMakeLists.txt - [x] profiler/src/ProfilerEngine/ProductVersion.props - [x] shared/src/Datadog.Trace.ClrProfiler.Native/CMakeLists.txt - [x] shared/src/Datadog.Trace.ClrProfiler.Native/Resource.rc - [x] shared/src/msi-installer/WindowsInstaller.wixproj - [x] shared/src/native-src/version.h - [x] tracer/build/artifacts/dd-dotnet.sh - [x] tracer/build/_build/Build.cs - [x] tracer/samples/AutomaticTraceIdInjection/MicrosoftExtensionsExample/MicrosoftExtensionsExample.csproj - [x] tracer/samples/AutomaticTraceIdInjection/Log4NetExample/Log4NetExample.csproj - [x] tracer/samples/AutomaticTraceIdInjection/NLog40Example/NLog40Example.csproj - [x] tracer/samples/AutomaticTraceIdInjection/NLog45Example/NLog45Example.csproj - [x] tracer/samples/AutomaticTraceIdInjection/NLog46Example/NLog46Example.csproj - [x] tracer/samples/AutomaticTraceIdInjection/SerilogExample/SerilogExample.csproj - [x] tracer/samples/ConsoleApp/Alpine3.10.dockerfile - [x] tracer/samples/ConsoleApp/Alpine3.9.dockerfile - [x] tracer/samples/ConsoleApp/Debian.dockerfile - [x] tracer/samples/OpenTelemetry/Debian.dockerfile - [x] tracer/samples/WindowsContainer/Dockerfile - [x] tracer/src/Datadog.Trace.ClrProfiler.Managed.Loader/Startup.cs - [x] tracer/src/Datadog.Tracer.Native/CMakeLists.txt - [x] tracer/src/Datadog.Tracer.Native/dd_profiler_constants.h - [x] tracer/src/Datadog.Tracer.Native/Resource.rc - [x] tracer/src/Directory.Build.props - [x] tracer/src/Datadog.Trace/TracerConstants.cs @DataDog/apm-dotnet Co-authored-by: link04 <[email protected]> commit 84efe2bfa576e95bbb385bbeb5832ee5ccc51648 Author: Raphaël Vandon <[email protected]> Date: Fri Apr 3 07:00:29 2026 +0100 maybe fix macos smoketests (#8413) ## Summary of changes ## Reason for change ## Implementation details ## Test coverage ## Other details <!-- Fixes #{issue} --> <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> commit 09325736fb3fb2c626d25e94b8e970a7fa02459f Author: Zach Montoya <[email protected]> Date: Thu Apr 2 15:54:55 2026 -0700 [Tracing] Update mapping from Datadog to OTLP spans (#8385) ## Summary of changes Add core Datadog semantics as span attributes. This ensures that the core semantics are identified when the Datadog Agent / backend receives OTLP spans and translates them into Datadog spans that power the backend. ## Reason for change We should keep the same Datadog experience regardless of the tracing protocol used. These changes facilitate that. ## Implementation details Updates `Datadog.Trace.OpenTelemetry.OtlpMapper.EmitAttributesFromSpan` to add several span attributes sourced from `Datadog.Trace.Span` properties: - `Span.ServiceName` => `OtlpSpan.Attributes["service.name"]` - `Span.OperationName` => `OtlpSpan.Attributes["operation.name"]` - `Span.ResourceName` => `OtlpSpan.Attributes["resource.name"]` - `Span.Type` => `OtlpSpan.Attributes["span.type"]` ## Test coverage - Unit tests: Added unit tests for `Datadog.Trace.OpenTelemetry.OtlpMapper` which handles the mapping from Datadog-specific concepts into OTLP attributes - Integration tests: Updated the snapshot for `OpenTelemetrySdkTests` ## Other details N/A commit 32c92b2a10d2299f612d41a89e34c16f18ae5273 Author: Lucas Pimentel <[email protected]> Date: Thu Apr 2 15:57:58 2026 -0400 [Tests] Use `EnvironmentRestorer` to prevent env var leaks (#8388) ## Summary of changes Add `[EnvironmentRestorer]` attribute to test classes/methods that call `Environment.SetEnvironmentVariable` without proper cleanup, preventing environment variable leaks between tests. ## Reason for change Several test files were setting environment variables without restoring them, which could cause flaky or order-dependent test failures due to leaked state. ## Implementation details - **`EnvironmentRestorerAttribute`**: Allow usage on methods (`AttributeTargets.Method`) in addition to classes - **`TelemetrySettingsAgentlessSettingsTests`**: Replace manual `IDisposable` save/restore with class-level `[EnvironmentRestorer]` - **`AzurePerformanceCountersListenerTests`**: Add class-level `[EnvironmentRestorer("WEBSITE_COUNTERS_CLR")]` (previously had no restore at all) - **`LegacyCommandLineArgumentsTests.SetCi`**: Add method-level `[EnvironmentRestorer("TF_BUILD")]`, remove manual try/finally - **`ConfigureCiCommandTests.ConfigureCi`**: Add method-level `[EnvironmentRestorer("GITHUB_ENV")]`, remove manual save/restore - `AutodetectCi` keeps its manual try/finally because it clears all environment variables, which the attribute can't handle ## Test coverage Existing tests cover the affected functionality. No new tests needed — this is a cleanup of test infrastructure. ## Other details > *"I tried to leak an environment variable once, but it was restored before anyone noticed."* — Claude 🤖 --------- Co-authored-by: Andrew Lock <[email protected]> commit 5686c1c6af5f1e8fa3724aa82ddbd7f29ba90b43 Author: Andrew Lock <[email protected]> Date: Thu Apr 2 20:12:30 2026 +0100 Add potential fix for duck typing derived types (#8410) ## Summary of changes Potentially fixes a `FieldAccessException` in some cases ## Reason for change We have seen stack traces like the following in production: ```bash Error : Error creating or populating span. System.FieldAccessException at REDACTED at Datadog.Trace.ClrProfiler.ScopeFactory.CreateInactiveOutboundHttpSpan(Tracer tracer, String httpMethod, Uri requestUri, IntegrationId integrationId, HttpTags& tags, TraceId traceId, UInt64 spanId, Nullable`1 startTime, Boolean addToTraceContext) ``` After a bunch of 🤖 noodling and testing (and based on @zacharycmontoya idea), proved that we can repro this stack (sort of, assuming inlining) if we _derive_ from `System.Uri`. The error happens because we apply `IgnoresAccessChecksToAttribute` to the target type (i.e. the derived type) during ducktyping, but it _needs_ to be applied to the `targetField.DeclaringType`. ## Implementation details Updated the `UseDirectAccessTo` calls to point to the type that "owns" the field, instead of the current instance. ## Test coverage Added a test confirming the error, and then showing it passed ## Other details https://app.datadoghq.com/cases/APMLP-35 commit ef71061988a026654c7e784732980f8d468499ad Author: Andrew Lock <[email protected]> Date: Thu Apr 2 19:13:17 2026 +0100 Revert YARP exclusion (#8407) ## Summary of changes Reverts including YARP spans in the ignore handler ## Reason for change We were a bit hasty including this in #8370 ## Implementation details Remove it again ## Test coverage Explicitly disabling OTel in the test instead for now. There's nothing _wrong_ with the span, just inconsistency between versions. We could consider having different snapshots, but I'll do that as part of a follow up, as its important to get this in fast commit bd1574507e3588ee7743ee4befcccf0357956f3d Author: Steven Bouwkamp <[email protected]> Date: Thu Apr 2 11:07:30 2026 -0400 Skip some errors from being sent to telemetry (#8003) ## Summary of changes Skips a few errors that are `Ignored` in Error Tracking and shouldn't really be sent to telemetry as we can't act on them. ## Reason for change Nothing we can really do about these. ## Implementation details Marked as `ErrorSkipTelemetry` ## Test coverage N/A ## Other details <!-- Fixes #{issue} --> I guess one thing we do lose here is whether we _want_ to see how often these happen? <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> commit 291ef8e6f8c27b2bdc422dfd218888a4e07a2c7c Author: Lucas Pimentel <[email protected]> Date: Thu Apr 2 11:01:47 2026 -0400 [CI Tools] Add prerequisite validation to AzureDevOps helper scripts (#8409) ## Summary of changes Add a `Test-Prerequisites` function to `AzureDevOpsHelpers.psm1` that validates all required CLI tools are installed, authenticated, and properly configured before running Azure DevOps build analysis or retry scripts. ## Reason for change When users run `Get-AzureDevOpsBuildAnalysis.ps1` or `Retry-AzureDevOpsFailedStages.ps1` directly (not via the Claude skill), missing or misconfigured prerequisites produce unhelpful error messages. The Claude skill had guidance for these scenarios, but the scripts themselves only checked if `az` and `gh` were installed. ## Implementation details New `Test-Prerequisites` function checks (in order): 1. **`az` CLI installed** — with install links for Windows/macOS 2. **`azure-devops` extension** — with `az extension add` command 3. **`az` authenticated** — with `az login` guidance (including MFA hint) 4. **Subscription logging** — logs current subscription at `Verbose` level for troubleshooting 5. **`gh` CLI installed** — only when needed for PR-based resolution 6. **`gh` authenticated** — scoped to `github.com` only, so GHES auth issues don't block `Resolve-BuildId` (the shared entry point for both scripts) now delegates to `Test-Prerequisites` instead of inline `Get-Command` checks. Additionally, `Invoke-AzDevOpsApi` now includes a troubleshooting tip in its error message suggesting the user check and switch Azure subscriptions if needed. ## Test coverage Tested manually: - [x] `az` not installed - [x] `az` installed without `azure-devops` extension - [x] `az` installed but not authenticated - [x] `gh` not installed - [x] `gh` installed but not authenticated ## Other details The subscription is not validated upfront because `az devops invoke --org <url> --detect false` targets the org URL directly, so the subscription doesn't strictly control API routing. However, the wrong subscription can affect token permissions, so `Invoke-AzDevOpsApi` now suggests checking subscriptions in its error message. > *"I validate your prerequisites so you don't have to validate your life choices."* — Claude 🤖 --------- Co-authored-by: Copilot <[email protected]> commit 94d461d7e1a32e617ef723764c6c5275c19fcef3 Author: Raphaël Vandon <[email protected]> Date: Thu Apr 2 16:00:53 2026 +0100 scrub process tags from debugger snapshots (#8406) ## Summary of changes follow up on https://github.com/DataDog/dd-trace-dotnet/pull/8296 debugger itests didn't run in the PR so we missed the fact that we need to apply the same scrubbing we did on other tests for those too ## Reason for change ## Implementation details ## Test coverage ## Other details <!-- Fixes #{issue} --> <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> commit a57df1c432ebf06bec5c71b9aa2a3581b4cd9090 Author: gh-worker-campaigns-3e9aa4[bot] <244854796+gh-worker-campaigns-3e9aa4[bot]@users.noreply.github.com> Date: Thu Apr 2 11:45:53 2026 +0000 chore(ci) update one-pipeline (#8403) This pull request updates one-pipeline to a newer version. Recent changes: Add CA certificates to SSI images using BusyBox, to prevent TLS failures (See https://github.com/DataDog/libdatadog-build/pull/194) Some of these changes may have already applied depending on your previous version of one-pipeline. See the libdatadog-build repository for all changes Co-authored-by: gh-worker-campaigns-3e9aa4[bot] <244854796+gh-worker-campaigns-3e9aa4[bot]@users.noreply.github.com> commit b68393db419d884599a96f4a989da06e4638d1a4 Author: Raphaël Vandon <[email protected]> Date: Thu Apr 2 10:33:42 2026 +0100 switch process tags on by default (#8296) ## Summary of changes all features have been implemented, we can now GA this to all customers ## Reason for change ## Implementation details depends on https://github.com/DataDog/dd-trace-dotnet/pull/8061 https://github.com/DataDog/dd-trace-dotnet/pull/8163 https://github.com/DataDog/dd-trace-dotnet/pull/8295 theoretically, https://github.com/DataDog/dd-trace-dotnet/pull/8282 as well ## Test coverage ## Other details <!-- Fixes #{issue} --> I took the liberty to refactor the asserts in AgentWriterTest to use actual asserts instead of asserting through the mock's `Verify`, so that we get more actionable errors when the test fails <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> commit 57de0e2dfa7ecc06ca1608bd5fd2041f9fa8d76a Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Apr 2 08:52:55 2026 +0000 Bump the gh-actions-packages group across 2 directories with 5 updates (#8404) Bumps the gh-actions-packages group with 4 updates in the / directory: [actions/setup-dotnet](https://github.com/actions/setup-dotnet), [DataDog/dd-octo-sts-action](https://github.com/datadog/dd-octo-sts-action), [slackapi/slack-github-action](https://github.com/slackapi/slack-github-action) and [github/codeql-action](https://github.com/github/codeql-action). Bumps the gh-actions-packages group with 1 update in the /.github/actions/publish-debug-symbols directory: [actions/setup-go](https://github.com/actions/setup-go). Updates `actions/setup-dotnet` from 5.1.0 to 5.2.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/setup-dotnet/releases">actions/setup-dotnet's releases</a>.</em></p> <blockquote> <h2>v5.2.0</h2> <h2>What's changed</h2> <h3>Enhancements</h3> <ul> <li>Add support for workloads input by <a href="https://github.com/gowridurgad"><code>@gowridurgad</code></a> in <a href="https://redirect.github.com/actions/setup-dotnet/pull/693">actions/setup-dotnet#693</a></li> <li>Add support for optional architecture input for cross-architecture .NET installs by <a href="https://github.com/priya-kinthali"><code>@priya-kinthali</code></a> in <a href="https://redirect.github.com/actions/setup-dotnet/pull/700">actions/setup-dotnet#700</a></li> </ul> <h3>Dependency Updates</h3> <ul> <li>Upgrade fast-xml-parser from 4.4.1 to 5.3.6 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/setup-dotnet/pull/671">actions/setup-dotnet#671</a></li> <li>Upgrade minimatch from 3.1.2 to 3.1.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/setup-dotnet/pull/705">actions/setup-dotnet#705</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-dotnet/compare/v5...v5.2.0">https://github.com/actions/setup-dotnet/compare/v5...v5.2.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/actions/setup-dotnet/commit/c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7"><code>c2fa09f</code></a> Bump minimatch from 3.1.2 to 3.1.5 (<a href="https://redirect.github.com/actions/setup-dotnet/issues/705">#705</a>)</li> <li><a href="https://github.com/actions/setup-dotnet/commit/02574b18e2dc57a218ee4e11ba1e1603c67236e8"><code>02574b1</code></a> Add support for optional architecture input for cross-architecture .NET insta...</li> <li><a href="https://github.com/actions/setup-dotnet/commit/16c7b3c2fa55a0e394467d22512b84fda46adf63"><code>16c7b3c</code></a> Bump fast-xml-parser from 4.4.1 to 5.3.6 (<a href="https://redirect.github.com/actions/setup-dotnet/issues/671">#671</a>)</li> <li><a href="https://github.com/actions/setup-dotnet/commit/131b410979e0b49e2162c0718030257b22d6dc2c"><code>131b410</code></a> Add support for workloads input (<a href="https://redirect.github.com/actions/setup-dotnet/issues/693">#693</a>)</li> <li>See full diff in <a href="https://github.com/actions/setup-dotnet/compare/v5.1.0...c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7">compare view</a></li> </ul> </details> <br /> Updates `DataDog/dd-octo-sts-action` from 1.0.3 to 1.0.4 <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a"><code>96a2546</code></a> Fix typo in Readme (<a href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/18">#18</a>)</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/9691c26e1de0f1f26e1e8708c5c34b4f64e43f5f"><code>9691c26</code></a> Merge pull request <a href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/14">#14</a> from DataDog/improve/parse-jwt-claims</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/b98b59d08d3575cbda7001bddfe86633787536e8"><code>b98b59d</code></a> Merge pull request <a href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/13">#13</a> from DataDog/improve/fetch-error-url-logging</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/e7953d4e870e933635e6afa9172b3957b568c417"><code>e7953d4</code></a> Merge pull request <a href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/15">#15</a> from DataDog/improve/ci-workflow-hardening</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/e47344e9570a80d3a7d333a339ace4a5e88b7646"><code>e47344e</code></a> Merge pull request <a href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/16">#16</a> from DataDog/improve/bump-node24</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/5a7a632cb3be2334cd1515df9c74eb3103942b50"><code>5a7a632</code></a> Bump Node.js runtime from node20 to node24</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/260fcf964ad38660b2abc359216586af9d31a05d"><code>260fcf9</code></a> Add parseJwtClaims() function with tests, replace fragile inline parsing</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/371c4d81ebd5ed74dfcc7bb2ab234d9f1e30fe65"><code>371c4d8</code></a> Harden CI workflows with least-privilege permissions and credential controls</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/1fc658893bed0edd73a7e284f6266e3fc4bdc93e"><code>1fc6588</code></a> Include URL in fetchWithRetry error messages</li> <li><a href="https://github.com/DataDog/dd-octo-sts-action/commit/0b31f95da950c7562ef40f6447086e75515897ce"><code>0b31f95</code></a> Harden CI workflows with least-privilege permissions and credential controls</li> <li>Additional commits viewable in <a href="https://github.com/datadog/dd-octo-sts-action/compare/v1.0.3...96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a">compare view</a></li> </ul> </details> <br /> Updates `slackapi/slack-github-action` from 2.1.1 to 3.0.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/slackapi/slack-github-action/releases">slackapi/slack-github-action's releases</a>.</em></p> <blockquote> <h2>Slack GitHub Action v3.0.1</h2> <h2>What's Changed</h2> <p>Alongside the breaking changes of <a href="https://github.com/slackapi/slack-github-action/releases/tag/v3.0.0"><code>@v3.0.0</code></a> and a <a href="https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/">new technique</a> to run Slack CLI commands, we tried the wrong name to publish to the GitHub Marketplace 🐙 This action is now noted as <a href="https://github.com/marketplace/actions/the-slack-github-action"><strong>The Slack GitHub Action</strong></a> in listings 🎶 ✨</p> <h3>:art: Maintenance</h3> <ul> <li>chore: use a unique title for marketplace in <a href="https://redirect.github.com/slackapi/slack-github-action/pull/576">slackapi/slack-github-action#576</a> - Thanks <a href="https://github.com/zimeg"><code>@zimeg</code></a>!</li> <li>chore(release): tag version 3.0.1 in <a href="https://redirect.github.com/slackapi/slack-github-action/pull/577">slackapi/slack-github-action#577</a> - Thanks <a href="https://github.com/zimeg"><code>@zimeg</code></a>!</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/slackapi/slack-github-action/compare/v3.0.0...v3.0.1">https://github.com/slackapi/slack-github-action/compare/v3.0.0...v3.0.1</a></p> <h2>Slack GitHub Action v3.0.0</h2> <blockquote> <p>The <code>@v3.0.0</code> release had a hiccup on publish and we recommend using <a href="https://github.com/slackapi/slack-github-action/releases/tag/v3.0.1"><strong><code>@v3.0.1</code></strong></a> or a more recent version when updating! Oops!</p> </blockquote> <p>🎽 <strong>Running Slack CLI commands and the active Node runtime, both included in this release</strong> 👟 ✨</p> <h3>⚠️ Breaking change: Node.js 24 the runtime</h3> <p>This major version updates the GitHub Actions required runtime to <a href="https://nodejs.org/en/about/previous-releases"><strong>Node.js 24</strong>.</a> Most <a href="https://github.com/actions/runner-images?tab=readme-ov-file#software-and-image-support">GitHub-hosted runners</a> already include this, but self-hosted runners may need to be updated ahead of <a href="https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/">planned deprecations of Node 20 on GitHub Actions runners</a>.</p> <h3>📺 Enhancement: Run Slack CLI commands</h3> <p>This release introduces a new technique for running <a href="https://docs.slack.dev/tools/slack-cli">Slack CLI</a> commands directly in GitHub Actions workflows. Use this to install the latest version (or a specific one) of the CLI and execute commands like <code>deploy</code> for merges to main, <code>manifest validate</code> with tests, and other <a href="https://docs.slack.dev/tools/slack-cli/reference/commands/slack">commands</a>.</p> <p>Gather a token using the following CLI command to store with repo secrets, then get started with an example below:</p> <pre><code>$ slack auth token </code></pre> <h3>🧪 Validate an app manifest on pull requests</h3> <p>Check that your app manifest is valid before merging changes:</p> <p>🔗 <a href="https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/validate-a-manifest">https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/validate-a-manifest</a></p> <pre lang="yaml"><code>- name: Validate the manifest uses: slackapi/slack-github-action/[email protected] with: command: "manifest validate --app ${{ vars.SLACK_APP_ID }}" token: ${{ secrets.SLACK_SERVICE_TOKEN }} </code></pre> <h3>🚀 Deploy your app on push to main</h3> <p>Automate deployments whenever changes land on your main branch:</p> <p>🔗 <a href="https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/deploy-an-app">https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/deploy-an-app</a></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/slackapi/slack-github-action/commit/af78098f536edbc4de71162a307590698245be95"><code>af78098</code></a> Release</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/add1a00063f351e4c0e55c3703da81637f03a8be"><code>add1a00</code></a> chore(release): tag version 3.0.1 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/577">#577</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/2bc9e7a4cd10f4d06ef49b8fa8a11efdc7fb891b"><code>2bc9e7a</code></a> chore: use a unique title for marketplace (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/576">#576</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/c5d43dad17bba7ebd47486137b9ab6936fd6bbf4"><code>c5d43da</code></a> chore(release): tag version 3.0.0 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/575">#575</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/963b9796dcc3184602a0aefe2f052d034027bfaf"><code>963b979</code></a> build(deps): bump <code>@slack/web-api</code> from 7.14.1 to 7.15.0 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/574">#574</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/90b7328a4cea35bd9dc6fc64d7f70e772d6d5876"><code>90b7328</code></a> build(deps): bump <code>@slack/logger</code> from 4.0.0 to 4.0.1 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/573">#573</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/e45cb891a61f925570820f137980df2028625fec"><code>e45cb89</code></a> feat: support slack cli commands with composite action inputs (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/560">#560</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/0aed2c2a70fe17c67bfd489b5dc3d9b410f69f79"><code>0aed2c2</code></a> build(deps): bump https-proxy-agent from 7.0.6 to 8.0.0 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/572">#572</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/4795f96c2818074349810cac0abc3bf5437bdc2c"><code>4795f96</code></a> build(deps-dev): bump sinon from 21.0.1 to 21.0.2 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/571">#571</a>)</li> <li><a href="https://github.com/slackapi/slack-github-action/commit/bd9e2ce619554772120b8cfcbbc7fe4bd2d42a2f"><code>bd9e2ce</code></a> build(deps): bump actions/setup-node from 6.2.0 to 6.3.0 (<a href="https://redirect.github.com/slackapi/slack-github-action/issues/569">#569</a>)</li> <li>Additional commits viewable in <a href="https://github.com/slackapi/slack-github-action/compare/91efab103c0de0a537f72a35f6b8cda0ee76bf0a...af78098f536edbc4de71162a307590698245be95">compare view</a></li> </ul> </details> <br /> Updates `github/codeql-action` from 4.34.1 to 4.35.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action's releases</a>.</em></p> <blockquote> <h2>v4.35.1</h2> <ul> <li>Fix incorrect minimum required Git version for <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis</a>: it should have been 2.36.0, not 2.11.0. <a href="https://redirect.github.com/github/codeql-action/pull/3781">#3781</a></li> </ul> <h2>v4.35.0</h2> <ul> <li>Reduced the minimum Git version required for <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis</a> from 2.38.0 to 2.11.0. <a href="https://redirect.github.com/github/codeql-action/pull/3767">#3767</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.1">2.25.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3773">#3773</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <ul> <li>The Git version 2.36.0 requirement for improved incremental analysis now only applies to repositories that contain submodules. <a href="https://redirect.github.com/github/codeql-action/pull/3789">#3789</a></li> <li>Python analysis on GHES no longer extracts the standard library, relying instead on models of the standard library. This should result in significantly faster extraction and analysis times, while the effect on alerts should be minimal. <a href="https://redirect.github.com/github/codeql-action/pull/3794">#3794</a></li> </ul> <h2>4.35.1 - 27 Mar 2026</h2> <ul> <li>Fix incorrect minimum required Git version for <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis</a>: it should have been 2.36.0, not 2.11.0. <a href="https://redirect.github.com/github/codeql-action/pull/3781">#3781</a></li> </ul> <h2>4.35.0 - 27 Mar 2026</h2> <ul> <li>Reduced the minimum Git version required for <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis</a> from 2.38.0 to 2.11.0. <a href="https://redirect.github.com/github/codeql-action/pull/3767">#3767</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.1">2.25.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3773">#3773</a></li> </ul> <h2>4.34.1 - 20 Mar 2026</h2> <ul> <li>Downgrade default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.24.3">2.24.3</a> due to issues with a small percentage of Actions and JavaScript analyses. <a href="https://redirect.github.com/github/codeql-action/pull/3762">#3762</a></li> </ul> <h2>4.34.0 - 20 Mar 2026</h2> <ul> <li>Added an experimental change which disables TRAP caching when <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis</a> is enabled, since improved incremental analysis supersedes TRAP caching. This will improve performance and reduce Actions cache usage. We expect to roll this change out to everyone in March. <a href="https://redirect.github.com/github/codeql-action/pull/3569">#3569</a></li> <li>We are rolling out improved incremental analysis to C/C++ analyses that use build mode <code>none</code>. We expect this rollout to be complete by the end of April 2026. <a href="https://redirect.github.com/github/codeql-action/pull/3584">#3584</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.0">2.25.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3585">#3585</a></li> </ul> <h2>4.33.0 - 16 Mar 2026</h2> <ul> <li> <p>Upcoming change: Starting April 2026, the CodeQL Action will skip collecting file coverage information on pull requests to improve analysis performance. File coverage information will still be computed on non-PR analyses. Pull request analyses will log a warning about this upcoming change. <a href="https://redirect.github.com/github/codeql-action/pull/3562">#3562</a></p> <p>To opt out of this change:</p> <ul> <li><strong>Repositories owned by an organization:</strong> Create a custom repository property with the name <code>github-codeql-file-coverage-on-prs</code> and the type "True/false", then set this property to <code>true</code> in the repository's settings. For more information, see <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">Managing custom properties for repositories in your organization</a>. Alternatively, if you are using an advanced setup workflow, you can set the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable to <code>true</code> in your workflow.</li> <li><strong>User-owned repositories using default setup:</strong> Switch to an advanced setup workflow and set the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable to <code>true</code> in your workflow.</li> <li><strong>User-owned repositories using advanced setup:</strong> Set the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable to <code>true</code> in your workflow.</li> </ul> </li> <li> <p>Fixed <a href="https://redirect.github.com/github/codeql-action/issues/3555">a bug</a> which caused the CodeQL Action to fail loading repository properties if a "Multi select" repository property was configured for the repository. <a href="https://redirect.github.com/github/codeql-action/pull/3557">#3557</a></p> </li> <li> <p>The CodeQL Action now loads <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">custom repository properties</a> on GitHub Enterprise Server, enabling the customization of features such as <code>github-codeql-disable-overlay</code> that was previously only available on GitHub.com. <a href="https://redirect.github.com/github/codeql-action/pull/3559">#3559</a></p> </li> <li> <p>Once <a href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">private package registries</a> can be configured with OIDC-based authentication for organizations, the CodeQL Action will now be able to accept such configurations. <a href="https://redirect.github.com/github/codeql-action/pull/3563">#3563</a></p> </li> <li> <p>Fixed the retry mechanism for database uploads. Previously this would fail with the error "Response body object should not be disturbed or locked". <a href="https://redirect.github.com/github/codeql-action/pull/3564">#3564</a></p> </li> <li> <p>A warning is now emitted if the CodeQL Action detects a repository property whose name suggests that it relates to the CodeQL Action, but which is not one of the properties recognised by the current version of the CodeQL Action. <a href="https://redirect.github.com/github/codeql-action/pull/3570">#3570</a></p> </li> </ul> <h2>4.32.6 - 05 Mar 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.24.3">2.24.3</a>. <a href="https://redirect.github.com/github/codeql-action/pull/3548">#3548</a></li> </ul> <h2>4.32.5 - 02 Mar 2026</h2> <ul> <li>Repositories owned by an organization can now set up the <code>github-codeql-disable-overlay</code> custom repository property to disable <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis for CodeQL</a>. First, create a custom repository property with the name <code>github-codeql-disable-overlay</code> and the type "True/false" in the organization's settings. Then in the repository's settings, set this property to <code>true</code> to disable improved incremental analysis. For more information, see <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">Managing custom properties for repositories in your organization</a>. This feature is not yet available on GitHub Enterprise Server. <a href="https://redirect.github.com/github/codeql-action/pull/3507">#3507</a></li> <li>Added an experimental change so that when <a href="https://redirect.github.com/github/roadmap/issues/1158">improved incremental analysis</a> fails on a runner — potentially due to insufficient disk space — the failure is recorded in the Actions cache so that subsequent runs will automatically skip improved incremental analysis until something changes (e.g. a larger runner is provisioned or a new CodeQL version is released). We expect to roll this change out to everyone in March. <a href="https://redirect.github.com/github/codeql-action/pull/3487">#3487</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/c10b8064de6f491fea524254123dbe5e09572f13"><code>c10b806</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/3782">#3782</a> from github/update-v4.35.1-d6d1743b8</li> <li><a href="https://github.com/github/codeql-action/commit/c5ffd0683786820677d054e3505e1c5bb4b8c227"><code>c5ffd06</code></a> Update changelog for v4.35.1</li> <li><a href="https://github.com/github/codeql-action/commit/d6d1743b8ec7ecd94f78ad1ce4cb3d8d2ba58001"><code>d6d1743</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/3781">#3781</a> from github/henrymercer/update-git-minimum-version</li> <li><a href="https://github.com/github/codeql-action/commit/65d2efa7333ad65f97cc54be40f4cd18630f884c"><code>65d2efa</code></a> Add changelog note</li> <li><a href="https://github.com/github/codeql-action/commit/2437b20ab31021229573a66717323dd5c6ce9319"><code>2437b20</code></a> Update minimum git version for overlay to 2.36.0</li> <li><a href="https://github.com/github/codeql-action/commit/ea5f71947c021286c99f61cc426a10d715fe4434"><code>ea5f719</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/3775">#3775</a> from github/dependabot/npm_and_yarn/node-forge-1.4.0</li> <li><a href="https://github.com/github/codeql-action/commit/45ceeea896ba2293e10982f871198d1950ee13d6"><code>45ceeea</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/3777">#3777</a> from github/mergeback/v4.35.0-to-main-b8bb9f28</li> <li><a href="https://github.com/github/codeql-action/commit/24448c98434f429f901d27db7ddae55eec5cc1c4"><code>24448c9</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/7c510606312e5c68ac8b27c009e5254f226f5dfa"><code>7c51060</code></a> Update changelog and version after v4.35.0</li> <li><a href="https://github.com/github/codeql-action/commit/b8bb9f28b8d3f992092362369c57161b755dea45"><code>b8bb9f2</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/3776">#3776</a> from github/update-v4.35.0-0078ad667</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/38697555549f1db7851b81482ff19f1fa5c4fedc...c10b8064de6f491fea524254123dbe5e09572f13">compare view</a></li> </ul> </details> <br /> Updates `actions/setup-go` from 6.3.0 to 6.4.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/setup-go/releases">actions/setup-go's releases</a>.</em></p> <blockquote> <h2>v6.4.0</h2> <h2>What's Changed</h2> <h3>Enhancement</h3> <ul> <li>Add go-download-base-url input for custom Go distributions by <a href="https://github.com/gdams"><code>@gdams</code></a> in <a href="https://redirect.github.com/actions/setup-go/pull/721">actions/setup-go#721</a></li> </ul> <h3>Dependency update</h3> <ul> <li>Upgrade minimatch from 3.1.2 to 3.1.5 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/actions/setup-go/pull/727">actions/setup-go#727</a></li> </ul> <h3>Documentation update</h3> <ul> <li>Rearrange README.md, add advanced-usage.md by <a href="https://github.com/priyagupta108"><code>@priyagupta108</code></a> in <a href="https://redirect.github.com/actions/setup-go/pull/724">actions/setup-go#724</a></li> <li>Fix Microsoft build of Go link by <a href="https://github.com/gdams"><code>@gdams</code></a> in <a href="https://redirect.github.com/actions/setup-go/pull/734">actions/setup-go#734</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/gdams"><code>@gdams</code></a> made their first contribution in <a href="https://redirect.github.com/actions/setup-go/pull/721">actions/setup-go#721</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-go/compare/v6...v6.4.0">https://github.com/actions/setup-go/compare/v6...v6.4.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/actions/setup-go/commit/4a3601121dd01d1626a1e23e37211e3254c1c06c"><code>4a36011</code></a> docs: fix Microsoft build of Go link (<a href="https://redirect.github.com/actions/setup-go/issues/734">#734</a>)</li> <li><a href="https://github.com/actions/setup-go/commit/8f19afcc704763637be6b1718da0af52ca05785d"><code>8f19afc</code></a> feat: add go-download-base-url input for custom Go distributions (<a href="https://redirect.github.com/actions/setup-go/issues/721">#721</a>)</li> <li><a href="https://github.com/actions/setup-go/commit/27fdb267c15a8835f1ead03dfa07f89be2bb741a"><code>27fdb26</code></a> Bump minimatch from 3.1.2 to 3.1.5 (<a href="https://redirect.github.com/actions/setup-go/issues/727">#727</a>)</li> <li><a href="https://github.com/actions/setup-go/commit/def8c394e3ad351a79bc93815e4a585520fe993b"><code>def8c39</code></a> Rearrange README.md, add advanced-usage.md (<a href="https://redirect.github.com/actions/setup-go/issues/724">#724</a>)</li> <li>See full diff in <a href="https://github.com/actions/setup-go/compare/4b73464bb391d4059bd26b0524d20df3927bd417...4a3601121dd01d1626a1e23e37211e3254c1c06c">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit b40c001f54fe95c5307316b80a10a46cdcbeebbd Author: Andrew Lock <[email protected]> Date: Wed Apr 1 16:13:39 2026 +0100 Add hardcoded limits for OCI and libinjection packages (#8382) ## Summary of changes Adds .NET-specific OCI and libinjection package size limits, instead of relying on the "global" limits ## Reason for change https://github.com/DataDog/dd-trace-dotnet/pull/8351 bumped the global package size limitations for OCI and lib-injection images, but @lloeki flagged that this can lead to large regressions in size to slip through. Given that these packages are quite size-sensitive, this is sub-optimal. ## Implementation details This PR uses the work in https://github.com/DataDog/libdatadog-build/pull/171 and https://github.com/DataDog/libdatadog-build/pull/174 to set size-limits based on [his example](https://github.com/DataDog/dd-trace-rb/blob/af17de097795fc3b0053f47d1436f308e0d5f92e/.gitlab-ci.yml#L17-L18). It adds the package size override variables, limiting both to 40MB. ## Test coverage Right now we have the following sizes: lib-injection images: - `linux-amd64`: 38MB - `linux-arm64`: 35MB OCI images: - `linux-amd64`: 30MB - `linux-arm64`: 28MB - `win-amd64`: 35MB So a 40MB limit across the board seems reasonable to me. You can see these limits being applied in [the Gitlab run](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1545330315) commit e72f8e8d2dc624d3ee711a78a76f113727adc0f7 Author: Andrew Lock <[email protected]> Date: Wed Apr 1 16:10:45 2026 +0100 Delete the static analysis workflow (#8393) ## Summary of changes Delete the static analysis workflow ## Reason for change Nobody uses it and it's a pain to migrate commit 954cad2f4c12a203a503401bc21774ac2ed2cc0f Author: Steven Bouwkamp <[email protected]> Date: Wed Apr 1 10:47:35 2026 -0400 Pin floating pre-release versions of SQLitePCLRaw.* dependencies in samples (#8390) ## Summary of changes Pins two different projects that were using floating versions of PackageReferences to `SQLitePCLRaw.bundle_e_sqlite3` and `SQLitePCLRaw.core`. These appeared to link to these versions, I bumped them to be stable - https://www.nuget.org/packages/SQLitePCLRaw.core/2.1.6-pre20230809203314 - https://www.nuget.org/packages/SQLitePCLRaw.bundle_e_sqlite3/2.1.6-pre20230809203314 ## Reason for change We don't want any floating dependencies ## Implementation details Searched here: https://github.com/search?q=repo%3ADataDog%2Fdd-trace-dotnet%20%2FVersion%3D%22%5B%5E%22%5D*%5C*%2F&type=code ## Test coverage This is the test, hopefully this works ## Other details <!-- Fixes #{issue} --> <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> commit 5f7ed4de8dc1152a8326d40b0ad95fb6593e6e67 Author: Steven Bouwkamp <[email protected]> Date: Wed Apr 1 09:48:16 2026 -0400 Configure `GeneratePackageVersions` to support a "cooldown" on dependencies (#8371) ## Summary of changes > default cooldown period is 2 days Adds a configurable cooldown period to the `GeneratePackageVersions` to support remediation efforts for follow up after #incident-51602. To use supply the optional parameter `--PackageVersionCooldownDays X` where `X` is some number of days. The current period is at the moment is going to be 2 days and is the default now when the overall `GeneratePackageVersion` target is ran. Additionally, this is overridden to 0 if `--IncludePackages` is supplied (this is commonly used when working on a singular package locally). After running the tool a "cooldown" report is generated, this file will contain packages that we see have a newer version, but will not incorporate into the test, the fallback version that falls within the cooldown period is provided. This file content will show up in the output of future test package version bump PRs. Note that this is for _automated_ updates, so if it sees something already updated it will honor it. Here's an example output ran locally with 14 days set: ``` ## Package Version Cooldown Report The following versions were published less than **14 days** ago and have been overridden. These require manual review before inclusion. | Package | Integration | Overridden Version | Published | Age (days) | Using Instead | |---------|-------------|--------------------|-----------|------------|---------------| | AWSSDK.Core | AwsSdk | 4.0.3.22 | 2026-03-25 | 0 | 4.0.3.21 | | AWSSDK.S3 | AwsS3 | 4.0.19.2 | 2026-03-25 | 0 | 4.0.19.1 | | StackExchange.Redis | StackExchangeRedis | 2.12.8 | 2026-03-25 | 0 | 2.12.4 | ``` ## Reason for change In https://github.com/DataDog/dd-trace-dotnet/pull/8364 and #incident-51602 all automated dependency updaters to be disabled temporarily, to re-enable we need to supply a 2 day "cooldown" to any version that we update to (in other words the version of the NuGet must be published for at least 2 days before we can update to it). ## Implementation details I made Claude do this 🤖 - NuGetPackageHelper now captures the Published date from IPackageSearchMetadata via a new VersionWithDate record (previously discarded) - NuGetVersionCache stores the new {Version, Published} format - PackageVersionGenerator.ApplyCooldown filters selected versions after LatestMajors/LatestMinors/LatestSpecific selection: - Versions outside the cooldown window pass through unchanged - Versions at or below the baseline (derived from supported_versions.json MaxVersionTestedInclusive) are kept even if within cooldown -- no downgrades - Versions above the baseline and within cooldown are overridden to the best available fallback - CooldownReport collects overridden versions and renders a markdown table saved to tracer/build/cooldown_report.md - The GitHub Actions workflow reads the report and appends it to the auto-bump PR body - Honeypot IntegrationGroups.cs fixes: MSTest.TestFramework now maps to itself, Hangfire.Core maps to Hangfire.Core (was Hangfire), OpenFeature mapping moved to Datadog.FeatureFlags.OpenFeature Passing `--IncludePackages` will override the cooldown to 0 ## Test coverage I ran `GeneratePackageVersions --PackageVersionCooldownDays 14` locally seems good enough IMO (also ran without, with different days etc) ``` [WRN] GeneratePackageVersi: 3 package version(s) were excluded due to the 14-day cooldown period [WRN] GeneratePackageVersi: AWSSDK.Core 4.0.3.22 overridden (published 2026-03-25, using: 4.0.3.21) [WRN] GeneratePackageVersi: AWSSDK.S3 4.0.19.2 overridden (published 2026-03-25, using: 4.0.19.1) [WRN] GeneratePackageVersi: StackExchange.Redis 2.12.8 overridden (published 2026-03-25, using: 2.12.4) ``` ## Other details <!-- Fixes #{issue} --> The workflow file (`auto_bump_test_package_versions.yml`) will be re-enabled with this PR <!-- ⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> commit 07039f099ad278d0d6b397ffd7d92cea62d07cd8 Author: Andrew Lock <[email protected]> Date: Wed Apr 1 09:47:34 2026 +0100 Set `DD_TRACE_OTEL_ENABLED=true` by default for all integration…
…ks (#8424) ## Summary of changes - Enable PR-level regression gates on microbenchmarks (`check-big-regressions`). - Enable release-level gates on macrobenchmarks (`check-slo-breaches`). - Replace `IGNORED_BENCHMARKS_REGEX` with `FLAKY_BENCHMARKS_REGEX` on microbenchmarks. - **TL;DR: `FLAKY_BENCHMARKS_REGEX` is used by internal tooling to avoid gating on flaky microbenchmarks. `IGNORED_BENCHMARKS_REGEX` was a workaround.** - `FLAKY_BENCHMARKS_REGEX` are excluded from gating decisions by `benchmark_analyzer`. - `FLAKY_BENCHMARKS_REGEX` is set on the `run-benchmarks` job, where `benchmark_analyzer` convert marks flaky benchmarks. Downstream jobs (`check-big-regressions`) read the flaky flag from the converted results. - Remove `check-big-regressions-no-ignored-benchmarks` (no longer needed). ## Reason for change [APMSP-2291 Identify and fix flaky dd-trace-dotnet microbenchmark scenarios](https://datadoghq.atlassian.net/browse/APMSP-2291) [APMSP-2292 Setup performance quality gates on dd-trace-dotnet](https://datadoghq.atlassian.net/browse/APMSP-2292) With #8300 reducing microbenchmark flakiness and with the ability to set `FLAKY_BENCHMARKS_REGEX` to make `benchmark_analyzer` ignore flaky benchmarks, we can now enforce the gates without false-positive noise for microbenchmarks. ## Implementation details - Flaky microbenchmarks on `FLAKY_BENCHMARKS_REGEX` were determined by stability tests. They are summarized on #8300. I also added other benchmarks @NachoEchevarria and I had classified as flaky before. - Removed the shell script logic that renamed ignored benchmarks with the `ignored.` prefix, `benchmark_analyzer` now handles this via `FLAKY_BENCHMARKS_REGEX`. ## Test coverage The most important bit to verify is whether we'd have flaky gates on micro/macrobenchmarks. We won't. Because: - On microbenchmarks, the regression check threshold is 20%, which is way bigger than variation for stable benchmarks. - Our stability tests (see #8300) show that variation for stable benchmarks stay mostly under 20%. For flaky benchmarks, this variation can be higher, but we ignore flaky benchmarks by filtering them on `FLAKY_BENCHMARKS_REGEX`. - On macrobenchmarks, SLO checks on macrobenchmarks are all passing for a month without issue: <img width="697" height="409" alt="image" src="https://github.com/user-attachments/assets/97793111-3744-4e2a-9720-04e8ffade07a" /> ## Other details As a reminder, if bypassing is necessary, see https://datadoghq.atlassian.net/wiki/x/8YFzMwE
## Summary of changes - Set `DD_CIVISIBILITY_AGENTLESS_ENABLED=1` on the benchmark process in `.gitlab/benchmarks/microbenchmarks/scripts/run-benchmarks.ps1` when `DD_API_KEY` is present, and default `DD_SITE=datadoghq.com`. - Change the SSM parameter name in `.gitlab/benchmarks/microbenchmarks.yml` from `ci.dd-trace-dotnet.dd_api_key` to `ci.dd-trace-dotnet.dd_api_key-prod` to match the key the previous pipeline was using. ## Reason for change Microbenchmark results stopped being reported to Datadog on April 8, 2026, after #8300 moved the Windows microbenchmark runner from `benchmarking-platform@dd-trace-dotnet/micro` into this repo. The `Datadog.Trace.BenchmarkDotNet` exporter ships results to Datadog via CI Visibility, and the old flow enabled agentless mode (`DD_CIVISIBILITY_AGENTLESS_ENABLED=1`) with a production SSM key — two details that were not carried over in the rewrite, so the exporter silently failed to reach intake. Cross-reference with the old script: `benchmarking-platform/run-benchmarks.ps1:128` (`$env:DD_CIVISIBILITY_AGENTLESS_ENABLED=1`) and `benchmarking-platform/steps/run-windows-benchmarks.sh:9` (`ci.${CI_PROJECT_NAME}.dd_api_key-prod`). ## Implementation details - `run-benchmarks.ps1` now guards the agentless toggle on `DD_API_KEY` being set, so local/dev invocations without a key continue to behave as before (warn and skip agentless) rather than attempting — and failing — to hit intake. - No changes to `Build.cs`, tracer code, or the benchmark projects. This is purely a CI config fix. ## Test coverage - Verified the execution order in `bp-runner.windows.yml`: the env vars are set in `run-benchmarks.ps1` after `BuildTracerHome`/`BuildBenchmarks` and before the BDN CLI is invoked — i.e., inherited by every BDN child process. ## Other details
## Summary of changes - Enable OpenTelemetry benchmarks which were turned off on #8300. - Use 1 CPU per benchmark instead of 2 to fit OpenTelemetry benchmarks. - Reduce `launchCount` from 10 to 5. - Flakiness changes. - This increases flakiness overall because of execution time/throughput metrics. - The false positive (FP) rate went from ~4% to 9%. **- Memory allocation metrics remain stable.** - The FP rate considering allocation metrics is 6%. There are some big outliers¹, but 85% of the benchmarks have a 0% FP rate. By marking them as flaky, we can have very effective gates on the remaining benchmarks. - Absolute value changes. - This might lead to changes in absolute values on certain benchmarks (mean delta of execution time and throughput is -11%² and -6%). - **But once again, memory allocation metrics remain stable (mean delta is of 0.1%).** - This reduced CI job time in 10 minutes (from ~70 minutes to ~60 minutes). I was expecting a bigger change. - This might be because on 1 CPU the benchmarks are slower? - Maybe reverting to 10 launches and having stabler benchmarks would be better. But if we only care about memory allocation metrics, it is already stable. And we send half of the data over the wire to the BP UI, which leads to less curl retries. <details> <summary>¹ Worst offenders when it comes to allocated memory false positives</summary> | Scenario | FP Rate | Max Bound | |----------|---------|-----------| | StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1 | **68.9%** | 14.69% | | OTel Api ActivityBenchmark.StartSpan_GetContext_Sampled net6.0 | 55.6% | 5.93% | | OTel Api ActivityBenchmark.StartSpan_UpdateName_Sampled net6.0 | 55.6% | 5.14% | | OTel Api TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net6.0 | 55.6% | 6.01% | | OTel Api ActivityBenchmark.StartSpan_AddEvent_Sampled net6.0 | 53.3% | 4.90% | | OTel Api ActivityBenchmark.StartSpan_SetAttributes_Sampled net6.0 | 53.3% | 3.28% | | OTel Api ActivityBenchmark.StartSpan_SetStatus_Sampled net6.0 | 53.3% | 5.93% | | OTel Api TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net6.0 | 53.3% | 5.93% | | OTel Api TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net6.0 | 53.3% | 8.41% | | OTel Api TracerBenchmark.StartActiveSpan net6.0 | 53.3% | 6.90% | | OTel Api TracerBenchmark.StartRootSpan net6.0 | 53.3% | 8.41% | | OTel Api TracerBenchmark.StartSpan_GetCurrentSpan net6.0 | 53.3% | 6.90% | </details> ² I have no idea why both execution time and throughput went down. I would imagine they would have inverse behaviors? Maybe this is because of some BenchmarkDotNet internal? Anyways, execution time and throughput are not looking super reliable for gating in any case. ## Reason for change Restore OpenTelemetry benchmarks missing after bp-runner migration, and improve CI job performance. ## Implementation details ## Test coverage - Ran stability tests for all benchmarks after changes. - If necessary, we can go into details on the PR conversation. But the main conclusions are listed on the "Summary of changes" section. - Results for this branch on the CIApp APM .NET Benchmarks dashboard: https://app.datadoghq.com/dashboard/9eu-mfe-5ad?fromUser=true&refresh_mode=paused&tpl_var_branch%5B0%5D=%22augusto%2Fenable-otel-benchmarks%22&tpl_var_runtime%5B0%5D=%2A&from_ts=1777887699376&to_ts=1778159028000&live=false

Summary of changes
BytesAllocatedPerOperationasallocated_meminto our benchmarking results.Reason for change
APMSP-2291 Identify and fix flaky dd-trace-dotnet microbenchmark scenarios
Implementation details
Benchmark design changes (what reduced flakiness)
launchCount).warmupCount).iterationCount).CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces,StringConcatAspectBenchmark,StringConcatBenchmark) go up to 120.iterationTimeof 200 ms, but setiterationTimeon flaky benchmarks to 500 ms.iterationTimefor flaky benchmarks helps immensely. We're able to capture much more variation into our samples once we fit more operations into the iteration. I tried many configurations, this turned out to be the minimum we need.Benchmark execution changes (what allowed me to implement the benchmark design changes)
ami.yamlto build the AMIs with which we run microbenchmarks. But there's some preliminary work to avoid dd-octo-sts failures on this repo. Captured on APMSP-2736.bp-runnerto run the benchmarks, since it allows us to run microbenchmarks in parallel on isolated CPU sets.Benchmark class/Build.cs changes
Build.cs:BuildBenchmarkstarget to build all benchmark executables in one go (needed forbp-runnerparallel execution).AppSecWafBenchmark.cs/AppSecBenchmarkUtils.cs: Fixed path resolution to make .NET Framework 4.7.2 benchmarks run in parallel.Directory.GetCurrentDirectory()andEnvironment.CurrentDirectorytoAppContext.BaseDirectory.[IterationTime(500)]attribute to benchmarks identified as flaky (e.g.,AspNetCoreBenchmark).Test coverage
Other details
How did we measure flakiness?
run-benchmarksjob for the same commit 10 times.execution_time,throughtput, andallocated_mem✨), we compute the false positive rate and the max confidence interval difference.run-benchmarksresults when comparing identical code.How much did flakiness drop?
See the per-benchmark/metric breakdown
This graph shows intuitively the distribution of FP rates and max CI differences for all comparisons. The more weight the distribution has closer to 0, the better. Memory allocation results are already pretty stable, so it's hard to see a clear win there.
How were flaky benchmarks determined (necessary to choose a higher
iterationTimefor them)?AppSecWafBenchmark,AppSecBodyBenchmark): Moved to stable batch. With the new design, most FP rates are under 5%, only one outlier at 20% for thenetcoreapp3.1runtime.AspNetCoreBenchmark: Consistently over 20% FP even with optimized design.AgentWriterBenchmark: Added to flaky batch. Most methods are stable, butWriteAndFlushEnrichedTraces netcoreapp3.1is VERY problematic.